Conditional GANs for Image Captioning with Sentiments

Tushar Karayil, Federico Raue, Jörn Hees, Andreas Dengel

In: 2019 International Conference on Artificial Neural Networks. International Conference on Artificial Neural Networks (ICANN-2019) 28th International Conference on Artificial Neural Networks. September 17-19 Munich Germany Springer 2019.


The area of automatic image captioning has witnessed much progress recently. However, generating captions with sentiment, which is a common dimension in human generated captions, still remains a challenge. This work presents a generative approach that combines sentiment (positive/negative) and variation for caption generation. The presented approach consists of a Generative Adversarial Network which takes as input, an image and a binary vector indicating the sentiment of the caption to be generated. We evaluate our model quantitatively on the state-of-the-art image caption dataset and qualitatively using a crowd-sourcing platform. Our results, along with human evaluation prove that we competitively succeed in the task of creating variations and sentiment in image captions.


German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz