With touchscreens firmly established on the factory floor and iPad apps getting all industrial, it is only a matter of time before the next consumer electronics breakthroughs—gesture-based “natural user interfaces” like those used for Microsoft Xbox 360 and Nintendo Wii gaming consoles—make their way to machine control. Development kits are letting industrial machine interface developers experiment and innovate.  

Bryan Brown is a staff engineer at Applied Instrument Technologies, an Upland Calif.-based maker of process analytical instruments for refining, petrochemical, pharmaceutical and steel industries. His workday life focuses on product development with an emphasis on electronic and software design. But after a few hours of tinkering at home, he was demonstrating a first step toward gesture-based industrial machine control using a Kinect controller for the Microsoft XBox 360 connected to an Opto 22 SNAP-PAC-R2 programmable automation controller.

“This first pass at Kinect-based machine control is very simplistic in nature,” explained Brown. “I needed something to control, so I hacked a flashlight to serve as a basic indicator for the demonstration. I have already received a fair amount of ribbing for applying hundreds of dollars worth of technology to turn a flashlight on and off, but I hope most people get the point.”

The software Brown used is based on the Microsoft Research Kinect for Windows software development kit (SDK) and the SNAP PAC.NET OptoMMP Messaging Toolkit . The latter lets .NET developers quickly create an application for Ethernet-based Opto 22 products, such as SNAP PAC controllers. The SNAP-PAC-R2 controller provides control, communication and input/output (I/O) processing in a one compact rack-mounted package.

Natural user interfaces
“Combining gesture-based control and speech recognition, which are both supported by the Kinect sensor and SDK, opens the door to many sophisticated home and business automation applications,” said Brown. “I also envision applications for controlling industrial equipment, particularly when operating in hazardous areas."

Field service technicians, for example, have limited access to equipment such as process analyzers that are deployed in potentially explosive environments. Depending on the area classification, interface electronics must use intrinsically safe (IS) barriers and other safeguards to prevent ignition of a hazardous atmosphere. “A device similar to the Kinect would provide a hands-off sensor that could be placed, along with an industrial computer and monitor, inside a purged enclosure behind an ATEX-rated window/bezel. The technician would not need to interact with an IS-protected keyboard or touchscreen interface, but rather use gestures for simple control and monitoring of the instrument,” Brown explained.

A beta version of the Kinect for Windows SDK can be downloaded at no cost for the development of non-commercial applications. It’s designed to let developers experiment with natural user interfaces (NUIs) that include depth sensing, human motion tracking and voice recognition. The Kinect for Windows SDK beta works with Windows 7 and includes drivers, application programming interfaces (APIs) for raw sensor streams and NUIs, installation documents, and resource materials. Although it is designed for non-commercial purposes only, a commercial version is expected to be available at a later date.

Development how-to
Brown started by modifying the SkeletonViewer sample application included in the Kinect SDK beta download. “First, the OptoMMP toolkit was added as a resource to the SkeletonViewer project, and then code was added to capture left- and right-hand jointID information,” he said. “These coordinates are passed to a new method that translates hand position to digital control commands that get sent to the PAC-R2 controller.” All coding was done in C# using Microsoft Visual Studio 2010.

“When executed, the program displays the typical skeleton, depth stream and video stream images of the original SkeletonViewer application. However, the video stream now has the superimposed user interface objects for positioning the body (ellipse) and controlling the digital outputs (rectangles),” said Brown.

A video demonstrating the result is available online. In it, Brown stands in front of the Kinect sensor and his computer screen shows him standing inside an ellipse. When he raises his arm, his hand is seen going outside the bounds of the ellipse and into a rectangle. This turns on the flashlight.

Brown has also experimented with entertainment applications, such as controlling a radio-controlled boat with a Kinect, a Measurement Computing data acquisition module, and some custom interface electronics. A video of this project is also available.  

More gesture software
Others interested in gesture-based machine control may want to check out GestureBuilder software from Movea, a French maker of motion-enabled control technologies and sensors with a U.S. subsidiary in Milpitas, Calif. Part of Movea’s SmartMotion portfolio, GestureBuilder ...