AI-based tool creates simple interfaces for virtual and augmented reality

A paper published in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, by researchers in Carnegie Mellon University’s Human-Computer Interaction Institute, introduces EgoTouch, a tool that uses artificial intelligence to control AR/VR interfaces by touching the skin with a finger.

The team wanted to ultimately design a control that would provide tactile feedback using only the sensors that come with a standard AR/VR headset.

OmniTouch, a previous method developed by Chris Harrison, an associate professor in the HCII and director of the Future Interfaces Group, got close. But that method required a special, clunky, depth-sensing camera. Vimal Mollyn, a Ph.D. student advised by Harrison, had the idea to use a machine learning algorithm to train normal cameras to recognize touching.

“Try taking your finger and see what happens when you touch your skin with it. You’ll notice that there are these shadows and local skin deformations that only occur when you’re touching the skin,” Mollyn said. “If we can see these, then we can train a machine learning model to do the same, and that’s essentially what we did.”

Mollyn collected the data for EgoTouch by using a custom touch sensor that ran along the underside of the index finger and the palm. The sensor collected data on different types of touch at different forces while staying invisible to the camera. The model then learned to correlate the visual features of shadows and skin deformities to touch and force without human annotation.

The team broadened its training data collection to include 15 users with different skin tones and hair densities and gathered hours of data across many situations, activities and lighting conditions.

EgoTouch can detect touch with more than 96% accuracy and has a false positive rate of around 5%. It recognizes pressing down, lifting up and dragging. The model can also classify whether a touch was light or hard with 98% accuracy.

“That can be really useful for having a right-click functionality on the skin,” Mollyn said.

Detecting variations in touch could enable developers to mimic touchscreen gestures on the skin. For example, a smartphone can recognize scrolling up or down a page, zooming in, swiping right, or pressing and holding on an icon. To translate this to a skin-based interface, the camera needs to recognize the subtle differences between the type of touch and the force of touch.

Accuracies were about the same across diverse skin tones and hair densities, and at different areas on the hand and forearm like the front of arm, back of arm, palm and back of hand. The system did not perform well on bony areas like the knuckles.

“It’s probably because there wasn’t as much skin deformation in those areas,” Mollyn said. “As a user interface designer, what you can do is avoid placing elements on those regions.”

Mollyn is exploring ways to use night vision cameras and nighttime illumination to enable the EgoTouch system to work in the dark. He’s also collaborating with researchers to extend this touch-detection method to surfaces other than the skin.

“For the first time, we have a system that just uses a camera that is already in all the headsets. Our models are calibration free, and they work right out of the box,” said Mollyn. “Now we can build off prior work on on-skin interfaces and actually make them real.”