Publication
AirScript - Creating Documents in Air
Ayushman Dash; Amit Sahu; Rajveer Shringi; John Gamboa; Muhammad Zeshan Afzal; Muhammad Imran Malik; Andreas Dengel; Sheraz Ahmed
In: ICDAR. International Conference on Document Analysis and Recognition (ICDAR), IEEE, 2017.
Abstract
This paper presents a novel approach, called
AirScript, for creating, recognizing and visualizing documents in
air. We present a novel algorithm, called 2-DifViz, that converts
the hand movements in air (captured by a Myo-armband worn
by a user) into a sequence of x; y coordinates on a 2D Cartesian
plane, and visualizes them on a canvas. Existing sensor-based
approaches either do not provide visual feedback or represent
the recognized characters using prefixed templates. In contrast,
AirScript stands out by giving freedom of movement to the user,
as well as by providing a real-time visual feedback of the written
characters, making the interaction natural. AirScript provides
a recognition module to predict the content of the document
created in air. To do so, we present a novel approach based on
deep learning, which uses the sensor data and the visualizations
created by 2-DifViz. The recognition module consists of a Convolutional
Neural Network (CNN) and two Gated Recurrent Unit
(GRU) Networks. The output from these three networks is fused
to get the final prediction about the characters written in air.
AirScript can be used in highly sophisticated environments like
a smart classroom, a smart factory or a smart laboratory, where
it would enable people to annotate pieces of texts wherever they
want without any reference surface. We have evaluated AirScript
against various well-known learning models (HMM, KNN, SVM,
etc.) on the data of 12 participants. Evaluation results show
that the recognition module of AirScript largely outperforms
all of these models by achieving an accuracy of 91.7% in a
person independent evaluation and a 96.7% accuracy in a person
dependent evaluation.