Multilinguality and Language Technology

Modern NLP models and LLMs have specific flaws, despite being highly performant: First, they are black boxes: Parameters of proprietary models are not accessible at all; and even non-proprietary models are largely opaque in the sense that it is unclear where exactly specific knowledge is encoded in potentially billions of parameters. Second, there is a tendency to always increase the size of LLMs and training data to improve performance, which is especially problematic for domains or languages with fewer resources.

The E&E group of DFKI’s Research Department Multilinguality and Language Technology works on transparent and efficient NLP models. Our objective is to make the parameters and behaviour of LLMs more explainable and understandable to both end users and researchers. We try to improve LLMs with regard to data consumption, e.g. for domains or languages where data is scarce, by using structured data, new learning techniques, or other modalities; and in terms of model size, e.g. for settings where powerful hardware is not available.

We are involved in Twinning projects, where we provide knowledge transfer both on research topics and project management to newly established research institutions across Europe. We are involved in European procurement projects focusing on language resources, such as the European Language Resource Coordination and the Language Data Space.

• Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization
  Cennet Oguz; Pascal Denis; Emmanuel Vincent; Simon Ostermann; Josef van Genabith
  In: Houda Bouamor; Juan Pino; Kalika Bali (Hrsg.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, Pages 8099-8110, Association for Computational Linguistics, 2023.
• Investigating the Encoding of Words in BERT's Neurons Using Feature Textualization
  Tanja Bäumel; Soniya Vijayakumar; Josef van Genabith; Günter Neumann; Simon Ostermann
  In: Yonatan Belinkov; Sophie Hao; Jaap Jumelet; Najoung Kim; Arya McCarthy; Hosein Mohebbi (Hrsg.). Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP-2023), Singapore, Pages 261-270, Association for Computational Linguistics, 2023.
• InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations
  Nils Feldhus; Qianli Wang; Tatiana Anikina; Sahil Chopra; Cennet Oguz; Sebastian Möller
  In: Houda Bouamor; Juan Pino; Kalika Bali (Hrsg.). Findings of the Association for Computational Linguistics: EMNLP 2023. Conference on Empirical Methods in Natural Language Processing (EMNLP-2023), December 6-10, Singapore, Singapore, Association for Computational Linguistics, 12/2023.
• ELRC White Paper: Sustainable Language Data Sharing to Support Language Equality in Multilingual Europe
  Aivars Berzins; Khalid Choukri; Maria Giagkou; Andrea Lösch; Eileen Marra; Hélène Mazo; Stelios Piperidis; Mickaël Rigault; Lilli Smal; Josef van Genabith; et al.
  ISBN 978-3-943853-05-6, OVD.eu, Saarbrücken, 2019.