Publikation
Identification of Important LLM Test Criteria for State Media Authorities
Stefan Schaffer; Niko Kleer; Pascal Lessel; Michael Feld; Julien Armin Bauer; Ina Goedert
In: Proceedings of the Mensch Und Computer 2025. Mensch und Computer (MuC-2025), Chemnitz, Germany, MuC '25, ISBN 9798400715822, Association for Computing Machinery, 2025.
Zusammenfassung
The increasing deployment of Foundation Models, particularly Large Language Models (LLMs), in novel AI services presents significant challenges for media regulation and societal discourse. These models, often operating as “black boxes”, raise concerns about transparency, bias, and their impact on media diversity and public opinion. Supervisory authorities with the legal mandate for regulating and supervising media, such as State Media Authorities (SMA) in Germany, face the challenge of assessing the legal conformity and the broader impact of LLM-based AI systems on opinion-forming. One of the goals of an SMA is the development of a test laboratory for the automatic evaluation of LLM-based AI systems. An important element of such a test laboratory is the criteria on the basis of which an LLM is to be assessed. This paper introduces important criteria in the categories Freedom, Legal Conformity, and Discrimination, Diversity, Inclusion for evaluating LLMs from the viewpoint of an SMA.