New Paper: A Sense of Quality for Augmented Reality Assisted Process Guidance

In this paper, we investigate and present the possible benefits of combining a very powerful Augmented Reality (AR) Headset (Microsoft Hololens) with the Machine Learning (ML) powered tooltracking system. The ongoing automation of modern production processes requires novel human-computer interaction concepts that support employees in dealing with the unstoppable increase in time pressure, cognitive load, and the required fine-grained and process-specific knowledge. AR systems support employees by guiding and teaching work processes. Such systems still lack a precise process quality analysis (monitoring), which is, however, crucial to close gaps in the quality assurance of industrial processes.

We combine inertial sensors, i.e., our tooltracking system, mounted on work tools, with AR headsets to enrich modern assistance systems with a sense of process quality. For this purpose, we develop a ML classifier that predicts quality metrics from a 9-degrees of freedom inertial measurement unit, while we simultaneously guide and track the work processes with a HoloLens AR system. In our user study, 6 test subjects perform typical assembly tasks with our system. We evaluate the tracking accuracy of the system based on a precise optical reference system and evaluate the classification of each work step quality based on the collected ground truth data. Our evaluation shows a tracking accuracy of fast dynamic movements of 4.92 mm and our classifier predicts the actions carried out with mean F1 value of 93.8% on average.

This work will be presented at the 22nd International Symposium on Mixed and Augmented Reality (ISMAR 2020, 9th-13th November 2020). It was supported by the Bavarian Ministry of Economic Affairs, Infrastructure, Energy, and Technology as part of the Bavarian project Leistungszentrum Elektroniksysteme (LZE) and through the Center for Analytics-Data-Applications (ADA-Center) within the framework of “BAYERN DIGITAL II”.