Skip to main content Skip to main navigation


SEGA: Instructing Diffusion using Semantic Dimensions

Manuel Brack; Felix Friedrich; Dominik Hintersdorf; Lukas Struppek; Patrick Schramowski; Kristian Kersting
In: Thirty-seventh Conference on Neural Information Processing Systems. Neural Information Processing Systems (NeurIPS), Pages 0-10, Curran Associates, Inc. 2023.


Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user's intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) allows for subtle and extensive edits, changes in composition and style, as well as optimizing the overall artistic conception. We demonstrate SEGA's effectiveness on a variety of tasks and provide evidence for its versatility and flexibility.

Weitere Links