Project

GreifbAR

Tangible reality - skilful interaction of user hands and fingers with real tools in mixed reality worlds

Tangible reality - skilful interaction of user hands and fingers with real tools in mixed reality worlds

The goal of the GreifbAR project is to make mixed reality (MR) worlds, including virtual (VR) and augmented reality ("AR"), tangible and graspable by allowing users to interact with real and virtual objects with their bare hands. Hand accuracy and dexterity is paramount for performing precise tasks in many fields, but the capture of hand-object interaction in current MR systems is woefully inadequate. Current systems rely on hand-held controllers or capture devices that are limited to hand gestures without contact with real objects. GreifbAR solves this limitation by introducing a sensing system that detects both the full hand grip including hand surface and object pose when users interact with real objects or tools. This sensing system will be integrated into a mixed reality training simulator that will be demonstrated in two relevant use cases: industrial assembly and surgical skills training. The usability and applicability as well as the added value for training situations will be thoroughly analysed through user studies.

Partners

NMY - Mixed Reality Communication GmbH Hanauer Landstraße 188, 60314 Frankfurt am Main

Charité – Universitätsmedizin Berlin Charitéplatz 1, 10117 Berlin

Universität Passau vertreten durch den Präsidenten, Prof. Dr. Ulrich Bartosch Innstraße 40, 94032 Passau Wissenschaftliche Leitung: Prof. Dr. Susanne Mayr Lehrstuhl für Psychologie mit Schwerpunkt Mensch - Maschine - Interaktion

Sponsors

BMBF - Federal Ministry of Education and Research

16SV8732

BMBF - Federal Ministry of Education and Research

Share project:

Keyfacts

Publications about the project

Ahmed Tawfik Aboukhadra; Muhammad Jameel Nawaz Malik; Ahmed Elhayek; Nadia Robertini; Didier Stricker

In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE Winter Conference on Applications of Computer Vision (WACV-2023), January 3-7, Waikoloa, Hawaii, USA, Pages 1001-1010, IEEE, 2023.

To the publication

Yongzhi Su; Mahdi Saleh; Torben Fetzer; Jason Raphael Rambach; Nassir Navab; Benjamin Busam; Didier Stricker; Federico Tombari

In: IEEE/CVF. International Conference on Computer Vision and Pattern Recognition (CVPR-2022), June 19-24, New Orleans, Louisiana, USA, IEEE/CVF, 2022.

To the publication

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz