Cognitive Workflow Capturing and Rendering with On-Body Sensor-Networks

Cognitive Workflow Capturing and Rendering with On-Body Sensor-Networks

  • Duration:

Augmented and virtual reality are becoming more and more common in systems for user assistance, educational simulators, novel games, and the whole range of applications in between. Technology to automatically capture, recognize, and render human activities is essential for all these applications. The aim of COGNITO is to bring this technology a big step forward.

COGNITO is a European project with activities covering the whole chain from low-level sensor fusion to workflow analysis and assistive visualization. Novel techniques are developed for analyzing, learning, and recording workflows, and then to use the acquired information in the way most suited for the user.

The project emphasizes how the hands are used to interact with objects and tools in the environment. This is an important component needed for making the technology useful in industrial applications. The workflow capturing in COGNITO is built upon the development of a on-body sensor network of miniature inertial and vision sensors. With the sensor network it is possible to accurately track limb motions, and even the fine motor skills of the hands using a wrist mounted camera. This information is then used to identify and classify workflow patterns in the captured movements. In the next step this is used for user monitoring and to develop new interaction paradigms for user-adaptive information presentation.

The focus of the Augmented Vision department is to develop the visual-inertial sensor network and provide the first level of information abstraction from it. This involves developing sensor fusion algorithms to estimate the limb motions and to use the wrist camera to provide detailed hand reconstructions. Augmented Vision also takes part in classifying workflows needed for user monitoring.


  • Centre National de la Recherche Scientifiques (CNRS)
  • Trivisio Prototyping GmbH
  • Center for Computer Graphics (CCG) &
  • Technology-Initiative SmartFactory KL .


EU - European Union

EU - European Union

Publications about the project

Lazaros Nalpantidis; Renaud Detry; Dima Damen; Gabriele Bleser-Taetz; Maya Cakmak; Mustafa Suphi Erden

In: Journal of Intelligent & Robotic Systems (JIRS), Vol. June, Pages 1-3, Springer Netherlands, 6/2015.

To the publication

Gabriele Bleser-Taetz; Dima Damen; Ardhendu Behera; Gustaf Hendeby; Katharina Mura; Markus Miezal; Andrew Gee; Nils Petersen; Gustavo Maçães; Hugo Domingues; Dominic Gorecky; Luis Almeida; Walterio Mayol-Cuevas; Andrew Calway; Anthony G. Cohn; David C. Hogg; Didier Stricker

In: Gianluca Bontempi (Hrsg.). PLoS ONE, Vol. 10, No. 6, Pages 1-41, PLOS, 6/2015.

To the publication

Gabriele Bleser-Taetz; Luis Almeida; Ardhendu Behera; Andrew Calway; Anthony Cohn; Dima Damen; Hugo Domingues; Andrew Gee; Dominic Gorecky; David Hogg; Michael Kraly; Gustavo Maçães; Frederic Marin; Walterio Mayol-Cuevas; Markus Miezal; Katharina Mura; Nils Petersen; Nicolas Vignais; Luis Paulo Santos; Gerrit Spaas; Didier Stricker

German Research Center for Artificial Intelligence, DFKI Research Reports (RR), Vol. 13-02, 2/2013.

To the publication

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz