Trajectory Reconstruction of sensor based data using ma- chine learning for the application of handwriting recogni- tion

Master's Thesis

Trajectory Reconstruction of sensor based data using machine learning for the application of handwriting recognition


Mohamad Wehbi, M.Sc., Dr. Dario Zanca


11 / 2020 – 04 / 2021


IMU (Inertial measurement unit) odometry is the use of sensor based data to compute how the
position of an object changes over time[1]. It is mostly used in addition to visual odometry for the
field of robotics[2]. IMU odometry has gained popularity in applications movement tracking and
trajectory reconstruction using sensor data extracted from mobile phones or other IMU sensors[3,
4, 5].
The process of double integrating acceleration data produces large drift errors that tend to
decrease accuracy over time as the errors from the sensors accumulate. Several machine learning
models and algorithms have been developed to tackle this loss in accuracy using different extracted
features from the data[6, 7]. [8] developed an end-to-end tracking framework for IMU based data
using raw data extracted from a mobile phone. These models were applied over different datasets
of IMU based data, and tested over large scale tracking applications such as movement over large
distances. No models have been developed or tested over smaller scale applications, such as single
hand movement of an individual.
In the domain of online handwriting recognition, the use of specific writing surfaces is still the
dominant approach among different applications. Tablet based applications make use of a tablet
grid system with S-pens to apply real-time recognition[9]. Other products include grid based
notebooks (smartpads) that trace writing onto a given tablet[10]. Such applications are limited
by the writing surface, limiting both the product’s usability and the user’s productivity.
In this thesis, we study the eciency of available state of the art IMU odometry algorithms
and models for the application of online handwriting. We make use of raw sensor data collected by
the Digipen, a sensor equipped pen provided by STABILO, and develop a model for reconstructing
the tip trajectory of the pen. We build upon a previous thesis in which a proof of concept was
established using ground truth data collected by writing on a tablet using the Digipen and applying
basic convolutional and recurrent neural networks to trace sensor data back onto the tablet. In
this work, we apply available IMU odemetry models [6, 7, 8] over the collected data. We also
consider different data collection and augmentation methods that would allow a better mapping
from sensor data into ground truth labels.