Skip to main content Skip to main navigation

ICML 2025: DFKI research warns of deceptive explainability in AI systems

| Knowledge & Business Intelligence | Data Management & Analysis | Data Science and its Applications

‘X-hacking’ is the term used by DFKI researchers to describe a risk in the field of explainable artificial intelligence (XAI) that has received little attention to date. At this year's International Conference on Machine Learning (ICML), one of the world's most important conferences for machine learning, the team from the Data Science and its Applications research department is presenting a systematic analysis of this phenomenon for the first time - and appealing for a critically considered use of AutoML tools.

© DFKI
At ICML 2025: New study on ‘X-Hacking’ shows risks of automated model selection - and pleads for reflected use of AutoML in research and practice.
Prof. Sebastian Vollmer, head of research department Data Science and its Applications at DFKI

‘At a time when AI explains decisions but does not always understand them, we as scientists must take responsibility for the depth of these explanations - and for their limitations.’

Prof. Sebastian Vollmer, head of research department Data Science and its Applications at DFKI

What happens when AI systems make correct predictions but give completely different reasons for how they arrive at this result? Can users then not simply choose the explanation that best fits their desired narrative? The DFKI team led by Prof Sebastian Vollmer (Rahul Sharma, Sumantrak Mukherjee, Andrea Šipka, Eyke Hüllermeier, Sergey Redyuk and David Antony Selby) investigated precisely this problem and identified a structural risk to the trustworthiness of AI with the term X-hacking.

X-hacking: When AutoML plausibly deceives

The term X-hacking, based on the p-hacking known from statistics, describes two central mechanisms:

  • Cherry-Picking: from a large number of similarly good models, the one whose explanation best supports the desired result is specifically selected.
  • Directed search: AutoML systems not only optimise the prediction performance, but also specifically find models with certain explanation patterns - an often underestimated risk.

The problem is that feature importance - i.e. the weighting of input features - can differ drastically, even if the models deliver almost identically good results. This is particularly sensitive in fields of application such as medical research or social science, where explainable models often form the basis for critical decisions.

‘The explainability of a model can become an illusion, especially when there are many plausible but contradictory models to choose from.’, says David Antony Selby, researcher at Data Science and its Applications.

What is behind AutoML - the core of the problem?

AutoML (Automated Machine Learning) stands for automated processes for the development, selection and optimisation of ML models. Software tools take over many tasks that were previously reserved for experienced ML engineers, such as the selection of suitable model architectures, preprocessing steps and parameter tuning.

 

© FB: Data Science and its Applications

Especially in data-intensive fields such as medicine, industry or social research, AutoML tools promise faster development, lower access barriers and reproducible results. However, it is precisely this automation that makes it difficult to trace the origins of modelling decisions - a critical aspect of explainable AI. The best-known AutoML frameworks include auto-sklearn, Google Cloud AutoML, H2O.ai und Microsoft Azure AutoML.

Solution approach: Honest explainability through transparency

The DFKI team is deliberately not proposing any technical control mechanisms, but rather a scientifically reflected practice based on transparency and methodological diligence. The following recommendations take centre stage:

The scientific poster on X-Hacking is presented during the poster presentation (16 July) at the ICML in Canada.

1. explanation histograms: 

show the distribution of model explanations across all valid models and help to recognise outliers immediately.

 

2. complete pipeline documentation: 

not only should the result be disclosed, but also the entire search space of models, data pre-processing and evaluation metrics.

 

3. interdisciplinary training: 

Specialist disciplines using AutoML should be aware of the methodological risks and not just trust the software.

Prof. Sebastian Vollmer, head of research department Data Science and its Applications at DFKI

‘The goal is a scientific culture that focuses not only on accuracy, but also on honesty in explainability.’

Prof. Sebastian Vollmer, head of research department Data Science and its Applications at DFKI

Trustworthy AI as a DFKI focus

The ICML 2025 study emphasises DFKI's research approach of making artificial intelligence not only powerful, but also transparent and socially trustworthy. In the context of the strategic focus ‘Trustworthy AI’, this work is an example of how scientific excellence and methodological responsibility can be combined.

Share this post:

Contact:

Prof. Dr. Sebastian Vollmer

Head research department Data Science and its Applications, DFKI

Press contact:

Jeremy Gob

Editor & PR-Officer, DFKI

Further information: