Publikation
MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding
Nils Rethmeier; Barbara Plank
In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019). Annual Meeting of the Association for Computational Linguistics (ACL-2019), The 57th Annual Meeting of the Association for Computational Linguistics (ACL), located at ACL, July 28 - August 2, Florence, Italy, Pages 49-54, Association for Computational Linguistics, 7/2019.
Zusammenfassung
Word embeddings have undoubtedly revolutionized NLP. However, pretrained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, self-supervised post-processing method that constructs task-specialized word representations by picking from a menu of reconstructing transformations to yield improved end-task performance (MORTY). The method is complementary to recent state-of-the-art approaches to inductive transfer via fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY on a broad range of setups, including different word embedding methods, corpus sizes and end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized embeddings that better fit end-tasks.
Projekte
- BigMedilytics - Big Data for Medical Analytics
- DEEPLEE - Tiefes Lernen für End-to-End-Anwendungen in der Sprachtechnologie