Generating Affective Captions using Concept And Syntax Transition Networks

Tushar Karayil, Philipp Blandfort, Damian Borth, Andreas Dengel

In: Proceedings of the 2016 ACM on Multimedia Conference. ACM International Conference on Multimedia (ACM MM-16) October 15-19 Amsterdam Netherlands ACM 10/2016.


The area of image captioning i.e. the automatic generation of short textual descriptions of images has experienced much progress recently. However, image captioning approaches often only focus on describing the content of the image without any emotional or sentimental dimension which is common in human captions. This paper presents an approach for image captioning designed specifically to incorporate emotions and feelings into the caption generation process. The presented approach consists of a Deep Convolutional Neural Network (CNN) for detecting Adjective Noun Pairs in the image and a graphical network architecture called "Concept And Syntax Transition (CAST)" network for generating sentences from these detected concepts.


Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence