A Workflow Manager for Complex NLP and Content Curation Pipelines

Julian Moreno Schneider; Peter Bourgonje; Florian Kintzel; Georg Rehm

In: Georg Rehm; Kalina Bontcheva; Khalid Choukri; Jan Hajic; Stelios Piperidis; Andrejs Vasiljevs (Hrsg.). Proceedings of the 1st International Workshop on Language Technology Platforms (IWLTP 2020, co-located with LREC 2020). International Workshop on Language Technology Platforms (IWLTP-2020), located at LREC, Marseille, France, Pages 73-80, ISBN 979-10-95546-64-1, ELRA, 5/2020.


We present a workflow manager for the flexible creation and customisation of NLP processing pipelines. The workflow manager addresses challenges in interoperability across various different NLP tasks and hardware-based resource usage. Based on the four key principles of generality, flexibility, scalability and efficiency, we present the first version of the workflow manager by providing details onits custom definition language, explaining the communication components and the general system architecture and setup. We currently implement the system, which is grounded and motivated by real-world industry use cases in several innovation and transfer projects.


Weitere Links

IWLTP-2020-Moreno-Schneider-et-al-final.pdf (pdf, 559 KB )

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence