Publication
Context-Aware Worker Assistance System in Augmented Reality using Semantically Zoomable Digital Twin
Snehal Walunj; Parsha Pahlevannejad; Ali Karnoub; Christiane Plociennik; Martin Ruskowski
In: 2025 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA). IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA-2025), June 2-5, Duisburg, Germany, Pages 72-78, IEEE, 7/2025.
Abstract
Manual maintenance and repair tasks are frequently demanding and time-consuming, necessitating workers to pinpoint problems and remember intricate procedures for their resolution in contemporary industrial environments. To address these challenges we propose an innovative assistance system that leverages a semantically zoomable Digital Twin (DT) of our factory environment to provide context-sensitive assistance to factory workers thereby helping them make informed decisions in maintenance and repair tasks. Our system aids workers by providing situation-specific guidance in an intuitive user interface in Augmented Reality (AR) glasses based on dynamic inputs such as user feedback, spatial marker tracking and registration in AR, and object detection. Based on these dynamic cues the user can zoom into the multi-layered 3D DT in Unity scene and access the necessary visualization relevant to that situation. This assistance in troubleshooting and repair procedures could potentially reduce their cognitive load and minimize time and errors. A preliminary study (N=6) is carried out to provide an initial understanding of the impact of situation-awareness and the semantic zoom-based visualizations as assistance feature and demonstrate its usability of the assistance system.