A sign of the times: smart glove translates sign language into text

Engineers from the University of California, San Diego have developed a smart glove that translates the American Sign Language (ASL) alphabet into text and controls a virtual hand to mimic sign language gestures. Known as ‘The Language of Glove’, the device was built for less than $100 using stretchable and printable electronics that are inexpensive, commercially available and easy to assemble.

Current methods for tracking human body positions include cameras as well as optical systems involving infrared emitters and receivers. Both can yield good results but the former uses a lot of power, while emitters and receivers are expensive and immovable. Wearable sensor systems avoid these constraints and gloves in particular are intuitive human-machine interfaces. With this in mind, researchers led by Darren Lipomi set out to build a glove that decodes the ASL alphabet and then wirelessly transmits the text to electronic devices.

Lipomi and his team built the device using a leather athletic glove and adhered nine stretchable sensors to the back at the knuckles — two on each finger and one on the thumb. The sensors are made of thin strips of a silicon-based polymer coated with a conductive carbon paint. The sensors are secured onto the glove with copper tape. Stainless steel thread connects each of the sensors to a low-power, custom-made printed circuit board that’s attached to the back of the wrist.

The sensors change their electrical resistance when stretched or bent, allowing them to code for different letters of the ASL alphabet based on the positions of all nine knuckles. A straight or relaxed knuckle is encoded as ‘0’ and a bent knuckle is encoded as ‘1’. When signing a particular letter, the glove creates a nine-digit binary key that translates into that letter. For example, the code for the letter ‘A’ (thumb straight, all other fingers curled) is ‘011111111’, while the code for ‘B’ (thumb bent, all other fingers straight) is ‘100000000’. Engineers equipped the glove with an accelerometer and pressure sensor to distinguish between letters like ‘I’ and ‘J’, whose gestures are different but which generate the same nine-digit code. The PCB on the glove converts the nine-digit key into a letter and then transmits the signals via Bluetooth to a smartphone or computer screen.

Writing in the journal PLOS ONE, the researchers revealed that the wearable electronic glove determined all 26 letters of the ASL alphabet accurately. Based on fatigue studies of the sensors, the system will translate ASL letters accurately after the knuckles are bent maximally 1000 times. The researchers also found that data from the glove could generate an accurate virtual display: when a real hand in the glove made the ASL gestures that spelt ‘UCSD’, a virtual hand mimicked them accurately.

“We’ve innovated a low-cost and straightforward design for smart wearable devices using off-the-shelf components,” said Lipomi. “Our work could enable other researchers to develop similar technologies without requiring costly materials or complex fabrication methods.”

Moving forward, the team is looking to develop the next version of the glove — one that’s endowed with the sense of touch. The aim is to make a glove that could control a virtual or robotic hand and then send tactile sensations back to the user’s own hand, opening up possibilities for a variety of applications — from virtual and augmented reality to telesurgery, technical training and defence.

“Our ultimate goal is to make this a smart glove that in the future will allow people to use their hands in virtual reality, which is much more intuitive than using a joystick and other existing controllers,” said Timothy O’Connor, first author on the study. “This could be better for games and entertainment but more importantly for virtual training procedures in medicine, for example, where it would be advantageous to actually simulate the use of one’s hands.”

Image credit: Timothy O’Connor/UC San Diego Jacobs School of Engineering.