Publikation
NTCIR-18 MedNLP-CHAT Determining Medical, Ethical and Legal Risks in Patient-Doctor Conversations: Task Overview
Eiji Aramaki; Shoko Wakamiya; Shuntaro Yada; Shohei Hisada; Tomohiro Nishiyama; Lenard Paulo Velasco Tamayo; Jingnan Xiao; Axalia Levenchaud; Pierre Zweigenbaum; Christoph Otto; Jerycho Pasniczek; Philippe Thomas; Nathan Pohl; Wiebke Duettmann; Lisa Raithel; Roland Roller
In: Proceedings of the NTCIR-18 Conference. Conference on Evaluation of Information Access Technologies (NTCIR), NII, 2025.
Zusammenfassung
This paper presents an overview of the Medical Natural Language Processing for AI Chat (MedNLP-CHAT) task, conducted as part of the shared task at NTCIR-18. Recently, medical chatbot services have emerged as a promising solution to address the shortage of medical and healthcare professionals. However, the potential risks associated with these chatbots remain insufficiently understood. Given this context, we designed the MedNLP-CHAT task to evaluate medical chatbots from multiple risk perspectives, including medical, legal, and ethical aspects. In this shared task, participants were required to analyze a given medical question along with the corresponding chatbot response and determine whether the response posed a potential medical, legal, or ethical risk (binary classification). Nine teams participated in this task applying different approaches, yielding valuable insights.