Knock knock: object interaction with a smartphone

Researchers from the Korea Advanced Institute of Science and Technology (KAIST) have developed a new technology that identifies objects and executes actions following a simple knock against a smartphone. The software is powered by machine learning that analyses sounds, vibrations and other reactions.

What separates ‘Knocker’ from existing technology is the sensor fusion of sound and motion. Previously, object identification used either computer vision technology with cameras or hardware such as RFID (radio-frequency identification) tags, but these solutions all have their limitations. For computer vision technology, users need to take pictures of every item, and the technology will not work well in poor lighting situations. Using hardware meanwhile leads to additional costs and labour burdens.

Knocker, on the other hand, can identify objects even in dark environments only with a smartphone, without requiring any specialised hardware or using a camera. Knocker utilises the smartphone’s built-in sensors — such as a microphone, an accelerometer and a gyroscope — to capture a unique set of responses generated when the phone is knocked against an object. Machine learning is used to analyse these responses and classify and identify objects.

“This new technology does not require any specialised sensor or hardware,” said Professor Sung-Ju Lee, who led the KAIST research team. “It simply uses the built-in sensors on smartphones and takes advantage of the power of machine learning. It’s a software solution that everyday smartphone users could immediately benefit from.”

The researchers have confirmed the applicability of Knocker technology using 23 everyday objects such as books, laptop computers, water bottles and bicycles. In noisy environments such as a busy cafe or on the side of a road, it achieved 83% identification accuracy. In a quiet indoor environment, the accuracy rose to 98%.

The team believes Knocker will open a new paradigm of object interaction. For instance, by knocking on an empty water bottle, a smartphone can automatically order new water bottles from a merchant app. When integrated with IoT devices, knocking on a bed’s headboard before going to sleep could turn off the lights and set an alarm. A paper on the technology, presented at the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2019) in London last month, demonstrates 15 application cases for the technology.

“This technology enables users to conveniently interact with their favourite objects,” Prof Lee said.

Image caption: Professor Sung-Ju Lee (far right) demonstrates ‘Knocker’ with his students.

Please follow us and share on Twitter and Facebook. You can also subscribe for FREE to our weekly newsletter and bimonthly magazine.