Publication
Software doping analysis for human oversight
Sebastian Biewer; Kevin Baum; Sarah Sterz; Holger Hermanns; Sven Hetmank; Markus Langer; Anne Lauber-Rönsberg; Franz Lehr
In: Formal Methods in System Design, Vol. 62, Pages 1-50, Springer, 4/2024.
Abstract
This article introduces a framework that is meant to assist in mitigating societal risks that software can pose. Concretely, this encompasses facets of software doping as well as unfairness
and discrimination in high-risk decision-making systems. The term software doping refers to
software that contains surreptitiously added functionality that is against the interest of the user.
A prominent example of software doping are the tampered emission cleaning systems that
were found in millions of cars around the world when the diesel emissions scandal surfaced.
The first part of this article combines the formal foundations of software doping analysis with
established probabilistic falsification techniques to arrive at a black-box analysis technique
for identifying undesired effects of software. We apply this technique to emission cleaning
systems in diesel cars but also to high-risk systems that evaluate humans in a possibly unfair
or discriminating way. We demonstrate how our approach can assist humans-in-the-loop to
make better informed and more responsible decisions. This is to promote effective human
oversight, which will be a central requirement enforced by the European Union’s upcoming
AI Act. We complement our technical contribution with a juridically, philosophically, and
psychologically informed perspective on the potential problems caused by such systems.