text2image

This repository includes the implementation for Text to Image Generation with Semantic-Spatial Aware GAN

This repo is not completely.

Network Structure

The structure of the spatial-semantic aware convolutional network (SSACN) is shown as below

Requirements

• python 3.6+
• pytorch 1.0+
• numpy
• matplotlib
• opencv

Or install full requirements by running:

pip install -r requirements.txt

TODO

• instruction to prepare dataset
• remove all unnecessary files
• add link to download our pre-trained model
• clean code including comments
• instruction for training
• instruction for evaluation

Prepare data

1. Download the preprocessed metadata for birds coco and save them to data/
2. Download the birds image data. Extract them to data/birds/
3. Download coco dataset and extract the images to data/coco/

Pre-trained text encoder

1. Download the pre-trained text encoder for CUB and save it to DAMSMencoders/bird/inception/
2. Download the pre-trained text encoder for coco and save it to DAMSMencoders/coco/inception/

Trained model

you can download our trained models from our onedrive repo

Start training

See opts.py for the options.

Evaluation

Performance

You will get the scores close to below after training under xe loss for xxxxx epochs:

Qualitative Results

Some qualitative results on coco and birds dataset from different methods are shown as follows:

The predicted mask maps on different stages are shown as as follows:

Reference

If you find this repo helpful in your research, please consider citing our paper:

@article{liao2021text,
  title={Text to Image Generation with Semantic-Spatial Aware GAN},
  author={Liao, Wentong and Hu, Kai and Yang, Michael Ying and Rosenhahn, Bodo},
  journal={arXiv preprint arXiv:2104.00567},
  year={2021}
}

The code is released for academic research use only. For commercial use, please contact Wentong Liao.

Acknowledgements

This implementation borrows part of the code from DF-GAN.