Skip to main content Skip to main navigation



Sustained Emotionally coloured Machine-human Interaction using Nonverbal Expression

Sustained Emotionally coloured Machine-human Interaction using Nonverbal Expression

  • Duration:

The aim of the SEMAINE project is to build a Sensitive Artificial Listener – a multimodal dialogue system with the social interaction skills needed for a sustained conversation with a human user. The system will emphasise “soft” communication skills, i.e. non-verbal, social and emotional perception, interaction and behaviour capabilities. The Sensitive Artificial Listener paradigm involves only very limited verbal capabilities, but has been shown to be suited for prolonged human-machine interaction. In this paradigm, we will build a real-time, robust interactive system perceiving a human user's facial expression, gaze, and voice, and engaging with the user through an Embodied Conversational Agent's body, face and voice. The agent will exhibit audiovisual listener feedback in real time while the user is speaking, and will take the user's feedback into account while the agent is speaking. The agent will pursue different dialogue strategies depending on the user's state; it will learn to interpret the user's non-verbal behaviour and adapt its own behaviour accordingly. Data to train system components will be collected initially using a Wizard-of-Oz setup, later on using the autonomous system at increasing levels of maturity. Some of the data will be released to the research community.

Consortium partners are involved in several standardisation initiatives related to building socially competent interactive agents; these standards will be co-shaped by work in SEMAINE.


  • Deutsches Forschungszentrum für Künstliche Intelligenz DFKI GmbH, Saarbrücken, Germany (Coordinator)
  • Queen's University Belfast, UK
  • Imperial College of Science, Technology and Medicine, London, UK
  • Universiteit Twente, Netherlands
  • Université Paris VIII, France
  • Technische Universität München, Germany


EU - European Union


EU - European Union

Publications about the project

Marc Schröder; Marcela Charfuelan Oliva; Sathish Pammi; Ingmar Steiner

In: 12th Annual Conference of the International Speech Communication Association. Conference in the Annual Series of Interspeech Events (INTERSPEECH-2011), 12th, August 28-31, Florence, Italy, ISCA, 8/2011.

To the publication

Gary McKeown; Michel Valstar; Roddy Cowie; Maja Pantic; Marc Schröder

In: IEEE Transactions on Affective Computing (TAC), Vol. 99, Pages 1-1, IEEE, 7/2011.

To the publication

Marc Schröder; Hannes Pirker; Myriam Lamolle; Felix Burkhardt; Christian Peter; Enrico Zovato

In: Paolo Petta; Roddy Cowie; Catherine Pelachaud (Hrsg.). Emotion-Oriented Systems - The Humaine Handbook. Pages 367-386, Springer, 2011.

To the publication