Manara - Qatar Research Repository
Browse

Improving text-to-image generation with object layout guidance

Download (1.35 MB)
journal contribution
posted on 2022-11-22, 21:13 authored by Jezia Zakraoui, Moutaz Saleh, Somaya Al-Maadeed, Jihad Mohammed Jaam

The automatic generation of realistic images directly from a story text is a very challenging problem, as it cannot be addressed using a single image generation approach due mainly to the semantic complexity of the story text constituents. In this work, we propose a new approach that decomposes the task of story visualization into three phases: semantic text understanding, object layout prediction, and image generation and refinement. We start by simplifying the text using a scene graph triple notation that encodes semantic relationships between the story objects. We then introduce an object layout module to capture the features of these objects from the corresponding scene graph. Specifically, the object layout module aggregates individual object features from the scene graph as well as averaged or likelihood object features generated by a graph convolutional neural network. All these features are concatenated to form semantic triples that are then provided to the image generation framework. For the image generation phase, we adopt a scene graph image generation framework as stage-I, which is refined using a StackGAN as stage-II conditioned on the object layout module and the generated output image from stage-I. Our approach renders object details in high-resolution images while keeping the image structure consistent with the input text. To evaluate the performance of our approach, we use the COCO dataset and compare it with three baseline approaches, namely, sg2im, StackGAN and AttnGAN, in terms of image quality and user evaluation. According to the obtained assessment results, our object layout guidance-based approach significantly outperforms the abovementioned baseline approaches in terms of the accuracy of semantic matching and realism of the generated images representing the story text sentences.

Other Information

Published in: Multimedia Tools and Applications
License: https://creativecommons.org/licenses/by/4.0
See article on publisher's website: http://dx.doi.org/10.1007/s11042-021-11038-0

History

Language

  • English

Publisher

Springer Science and Business Media LLC

Publication Year

  • 2021

Institution affiliated with

  • Qatar University

Usage metrics

    Manara - Qatar Research Repository

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC