A Tensorflow implementation of the Text Conditioned Auxiliary Classifier Generative Adversarial Network for Generating Images from text descriptions

Overview

TAC-GAN

This is the official Tensorflow implementation of TAC-GAN model presented in https://arxiv.org/abs/1703.06412.

Text Conditioned Auxiliary Classifier Generative Adversarial Network, (TAC-GAN) is a text to image Generative Adversarial Network (GAN) for synthesizing images from their text descriptions. TAC-GAN builds upon the AC-GAN by conditioning the generated images on a text description instead of on a class label. In the presented TAC-GAN model, the input vector of the Generative network is built based on a noise vector and another vector containing an embedded representation of the textual description. While the Discriminator is similar to that of the AC-GAN, it is also augmented to receive the text information as input before performing its classification.

For embedding the textual descriptions of the images into vectors we used skip-thought vectors

The following is the architecture of the TAC-GAN model

Prerequisites

Some important dependencies are the following and the rest can be installed using the requirements.txt

  1. Python 3.5
  2. Tensorflow 1.2.0
  3. Theano 0.9.0 : for skip thought vectors
  4. scikit-learn : for skip thought vectors
  5. NLTK 3.2.1 : for skip thought vectors

It is recommended to use a virtual environment for running this project and installing the required dependencies in it by using the requirements.txt file.

The project has been tested on a Ubuntu 14.04 machine with an 12 GB NVIDIA Titen X GPU

1. Setup and Run

1.1. Clone the Repository

git clone https://github.com/dashayushman/TAC-GAN.git
cd TAC-GAN

1.2. Download the Dataset

The model presented in the paper was trained on the flowers dataset. This To train the TAC-GAN on the flowers dataset, first, download the dataset by doing the following,

  1. Download the flower images from here. Extract the 102flowers.tgz file and copy the extracted jpg folder to Data/datasets/flowers

  2. Download the captions from here. Extract the downloaded file, copy the text_c10 folder and paste it in Data/datasets/flowers directory

  3. Download the pretrained skip-thought vectors model from here and copy the downloaded files to Data/skipthoughts

NB: It is recommended to keep all the images in an SSD if available. This makes the batch loading and processing operation faster.

1.3. Data Preprocessing

Extract the skip-thought features for the captions and prepare the dataset for training by running the following script

python dataprep.py --data_dir=Data --dataset=flowers

This script will create a set of pickled files in the datet directory which will be used during training. The following are the available flags for data preparation:

FLAG VALUE TYPE DEFAULT VALUE DESCRIPTION
data_dir str Data The data directory
dataset str flowers Dataset to use. For Eg., "flowers"

1.4. Training

To train TAC-GAN with the default hyper parameters run the following script

python train.py --dataset="flowers" --model_name=TAC-GAN

While training, you can montor samples generated by the model in the Data/training/TAC_GAN/samples directory. Notice that a directory is created according to the "model_name" taht you provide. This directory contains all the data related to a particular experiment. This can also be considered as an "experiment name" too.

The following flags can be set to change the hyperparameters of the network.

FLAG VALUE TYPE DEFAULT VALUE DESCRIPTION
z-dim int 100 Number of dimensions of the Noise vector
t_dim int 256 Number of dimensions for the latent representation of the text embedding.
batch_size int 64 Mini-Batch Size
image_size int 128 Batch size to use during training.
gf_dim int 64 Number of conv filters in the first layer of the generator.
df_dim int 64 Number of conv filters in the first layer of the discriminator.
caption_vector_length int 4800 Length of the caption vector embedding (vector generated using skip-thought vectors model).
n_classes int 102 Number of classes
data_dir String Data Data directory
learning_rate float 0.0002 Learning rate
beta1 float 0.5 Momentum for Adam Update
epochs int 200 Maximum number of epochs to train
save_every int 30 Save model and samples after this many number.of iterations
resume_model Boolean False To Load the pre-trained model
data_set String flowers Which dataset to use: "flowers"
model_name String model_1 Name of the model: Can be anything
train bool True This is True while training and false otherwise. Used for batch normalization

We used the following script (hyper-parameters) in for the results that we show in our paper

python train.py --t_dim=100 --image_size=128 --data_set=flowers --model_name=TAC_GAN --train=True --resume_model=True --z_dim=100 --n_classes=102 --epochs=400 --save_every=20 --caption_vector_length=4800 --batch_size=128

1.5. Monitoring

While training, you can monitor the updates on the terminal as well as by using tensorboard

1.5.1 The Terminal:

Terminal log

1.5.1 Tensorboard:

You can use the following script to start tensorboard and visualize realtime changes:

tensorboard --logdir=Data/training/TAC_GAN/summaries

Tensorboard

2. Generating Images for the text in the dataset

Once you have trained the model for certain epochs you can generate images for all the text descriptions in the dataset use the following script. This will create a synthetic dataset with images generated by the generator.

python train.py --data_set=flowers --epochs=100 --output_dir=Data/synthetic_dataset --checkpoints_dir=Data/training/TAC_GAN/checkpoints

Notice that the checkpoints directory is ls created automatically created inside the model directory after you run the training script.

This script will create the following directory structure:

Data
  |__synthetic_dataset
        |___ds
             |___train
             |___val

the train directory will contain all the images generated from the text descriptions of the images in the training set and the same goes for the val directory.

3. Generating Images from any Text

To generate images from any text, do the following

3.1 Add Text Descriptions:

Write your text descriptions in a file or use the example file Data/text.txt that we have provided in the Data directory. The text description file should contain one text description per line. For example,

a flower with red petals which are pointed
many pointed petals
A yellow flower

3.2 Extract Skip-Thought Vectors:

Run the following script for extracting the Skip-Thought vectors for the text descriptions

python encode_text.py --caption_file=Data/text.txt --data_dir=Data

This script will create a pickle file called Data/enc_text.pkl with features extracted from the text descriptions.

3.3 Generate Images:

To generate images for the text descriptions, run the following script,

python generate_images.py --data_set=flowers --checkpoints_dir=Data/training/TAC_GAN/checkpoints --images_per_caption=30 --data_dir=Data

This will create a directory Data/images_generated_from_text/ with a folder corresponding to every row of the text.txt file. Each of these folders will contain images for that text.

The following are the parameters you need to set, in case you have used different parameters for training the model.

FLAG VALUE TYPE DEFAULT VALUE DESCRIPTION
z-dim int 100 Number of dimensions of the Noise vector
t_dim int 256 Number of dimensions for the latent representation of the text embedding.
batch_size int 64 Mini-Batch Size
image_size int 128 Batch size to use during training.
gf_dim int 64 Number of conv filters in the first layer of the generator.
df_dim int 64 Number of conv filters in the first layer of the discriminator.
caption_vector_length int 4800 Length of the caption vector embedding (vector generated using skip-thought vectors model).
n_classes int 102 Number of classes
data_dir String Data Data directory
learning_rate float 0.0002 Learning rate
beta1 float 0.5 Momentum for Adam Update
images_per_caption int 30 Maximum number of images that you want to generate for each of the text descriptions
data_set String flowers Which dataset to use: "flowers"
checkpoints_dir String /tmp Path to the checkpoints directory which will be used to generate the images

4. Evaluation

We have used two metrics for evaluating TAC-GAN,

  1. Inception-Scope
  2. MS-SSIM score

The links are from where we adapted the code for evaluating TAC-GAN. Before evaluating the model, generate a synthetic dataset by referring to Section 6

4.1 Inception Score

To calculate the inception score, use the following script,

python inception_score.py --output_dir=Data/synthetic_dataset --data_dir=Data --n_images=30000 --image_size=128

This will create a collection of all the generated images in Data/synthetic_dataset/ds_inception and show the inception score on the terminal.

The following are the set of available parameters/flags

FLAG VALUE TYPE DEFAULT VALUE DESCRIPTION
output_dir str Data/ds_inception Directory to dump all the images for calculating the inception score
data_dir str Data/synthetic_dataset/ds The root directory of the synthetic dataset
n_images int 30000 Number of images to consider for calculating inception score
image_size int 128 Size of the image to consider for calculating inception score

4.2 MS-SSIM

To calculate the MS-SSIM score, use the following script,

python inception_score.py --output_dir=Data --data_dir=Data --dataset=flowers --syn_dataset_dir=Data/synthetic_datset/ds

This will create a Data/msssim.tsv tab separated file. The data in this file is structured as follows


     
       
     

     
    
   

Once you have generated the msssim.tsv file, you can use the following script to generate a figure to compare the MS-SSIM score of the images in the real dataset with the images in the synthetic dataset belonging to the same class,

python utility/plot_msssim.py --input_file=Data/msssim.tsv --output_file=Data/msssim

This will create Data/msssim.pdf, which is the .pdf file of the generated figure.

5. Generate Interpolated Images

In our paper we show the effect of interpolating the noise and the text embedding vectors on the generated image. Images are randomply selected and their text descriptions are used to generate synthetic images. The following sub-sections will elaborate on how to do it and which scripts will help you in doing it.

5.1 Z (Noise) Interpolation

For interpolating the noise vector and generating images, use the following scripts

python z_interpolation.py --output_dir=Data/synthetic_dataset --data_set=flowers --checkpoints_dir=Data/training/TAC_GAN/checkpoints --n_images=500

This will generate the interpolated images in Data/synthetic_dataset/z_interpolation/.

5.1 T (Text Embedding) Interpolation

For interpolating the text embedding vectors and generating images, use the following scripts

python t_interpolation.py --output_dir=Data/synthetic_dataset --data_set=flowers --checkpoints_dir=Data/training/TAC_GAN/checkpoints --n_images=500

This will generate the interpolated images in Data/synthetic_dataset/t_interpolation/.

NOTE: Both the above mentioned scripts have the same flags/arguments, which are the following,

FLAG VALUE TYPE DEFAULT VALUE DESCRIPTION
z-dim int 100 Number of dimensions of the Noise vector
t_dim int 256 Number of dimensions for the latent representation of the text embedding.
batch_size int 64 Mini-Batch Size
image_size int 128 Batch size to use during training.
gf_dim int 64 Number of conv filters in the first layer of the generator.
df_dim int 64 Number of conv filters in the first layer of the discriminator.
caption_vector_length int 4800 Length of the caption vector embedding (vector generated using skip-thought vectors model).
n_classes int 102 Number of classes
data_dir String Data Data directory
learning_rate float 0.0002 Learning rate
beta1 float 0.5 Momentum for Adam Update
data_set str flowers The dataset to use: "flowers"
output_dir String Data/synthetic_dataset The directory in which the t_interpolated images will be generated
checkpoints_dir String /tmp Path to the checkpoints directory which will be used to generate the images
n_interp int 100 The factor difference between each interpolation (Should ideally be a multiple of 10)
n_images int 500 Number of images to randomply sample for generating interpolation results

6. References

TAC-GAN

If you find this code usefull, then please use the following BibTex to cite our work.

@article{dash2017tac,
  title={TAC-GAN-Text Conditioned Auxiliary Classifier Generative Adversarial Network},
  author={Dash, Ayushman and Gamboa, John Cristian Borges and Ahmed, Sheraz and Afzal, Muhammad Zeshan and Liwicki, Marcus},
  journal={arXiv preprint arXiv:1703.06412},
  year={2017}
}

Oxford-102 Flowers Dataset

If you use the Oxford-102 Flowers Dataset, then please cite their work using the following BibTex.

@InProceedings{Nilsback08,
   author = "Nilsback, M-E. and Zisserman, A.",
   title = "Automated Flower Classification over a Large Number of Classes",
   booktitle = "Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing",
   year = "2008",
   month = "Dec"
}

Skip-Thought

If you use the Skip-Thought model in your work like us, then please cite their work using the following BibTex

@article{kiros2015skip,
  title={Skip-Thought Vectors},
  author={Kiros, Ryan and Zhu, Yukun and Salakhutdinov, Ruslan and Zemel, Richard S and Torralba, Antonio and Urtasun, Raquel and Fidler, Sanja},
  journal={arXiv preprint arXiv:1506.06726},
  year={2015}
}

Code

We have referred to the text-to-image and DCGAN-tensorflow repositories for developing our code, and we are extremely thankful to them.

Owner
Ayushman Dash
I am a research scientist, musician and composer. I have an affinity towards Machine Learning, Deep Neural Networks and I wish to add more to the community.
Ayushman Dash
A Python library for generating new text from existing samples.

ReMarkov is a Python library for generating text from existing samples using Markov chains. You can use it to customize all sorts of writing from birt

8 May 17, 2022
Text-to-Image generation

Generate vivid Images for Any (Chinese) text CogView is a pretrained (4B-param) transformer for text-to-image generation in general domain. Read our p

THUDM 1.3k Dec 29, 2022
Model Agnostic Interpretability for Multiple Instance Learning

MIL Model Agnostic Interpretability This repo contains the code for "Model Agnostic Interpretability for Multiple Instance Learning". Overview Executa

Joe Early 10 Dec 17, 2022
Clairvoyance: a Unified, End-to-End AutoML Pipeline for Medical Time Series

Clairvoyance: A Pipeline Toolkit for Medical Time Series Authors: van der Schaar Lab This repository contains implementations of Clairvoyance: A Pipel

van_der_Schaar \LAB 89 Dec 07, 2022
Predicting lncRNA–protein interactions based on graph autoencoders and collaborative training

Predicting lncRNA–protein interactions based on graph autoencoders and collaborative training Code for our paper "Predicting lncRNA–protein interactio

zhanglabNKU 1 Nov 29, 2022
Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification

Dealing With Misspecification In Fixed-Confidence Linear Top-m Identification This repository is the official implementation of [Dealing With Misspeci

0 Oct 25, 2021
Generate vibrant and detailed images using only text.

CLIP Guided Diffusion From RiversHaveWings. Generate vibrant and detailed images using only text. See captions and more generations in the Gallery See

Clay M. 401 Dec 28, 2022
Self-Supervised CNN-GCN Autoencoder

GCNDepth Self-Supervised CNN-GCN Autoencoder GCNDepth: Self-supervised monocular depth estimation based on graph convolutional network To be published

53 Dec 14, 2022
Pytoydl: A toy deep learning framework built upon numpy.

Documents: https://pytoydl.readthedocs.io/zh/latest/ Pytoydl A toy deep learning framework built upon numpy. You can star this repository to keep trac

28 Dec 10, 2022
A production-ready, scalable Indexer for the Jina neural search framework, based on HNSW and PSQL

🌟 HNSW + PostgreSQL Indexer HNSWPostgreSQLIndexer Jina is a production-ready, scalable Indexer for the Jina neural search framework. It combines the

Jina AI 25 Oct 14, 2022
PyTorch code of my ICDAR 2021 paper Vision Transformer for Fast and Efficient Scene Text Recognition (ViTSTR)

Vision Transformer for Fast and Efficient Scene Text Recognition (ICDAR 2021) ViTSTR is a simple single-stage model that uses a pre-trained Vision Tra

Rowel Atienza 198 Dec 27, 2022
A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

Yunxia Zhao 3 Dec 29, 2022
SemiNAS: Semi-Supervised Neural Architecture Search

SemiNAS: Semi-Supervised Neural Architecture Search This repository contains the code used for Semi-Supervised Neural Architecture Search, by Renqian

Renqian Luo 21 Aug 31, 2022
Code and models used in "MUSS Multilingual Unsupervised Sentence Simplification by Mining Paraphrases".

Multilingual Unsupervised Sentence Simplification Code and pretrained models to reproduce experiments in "MUSS: Multilingual Unsupervised Sentence Sim

Facebook Research 81 Dec 29, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

32 Dec 26, 2022
The codes I made while I practiced various TensorFlow examples

TensorFlow_Exercises The codes I made while I practiced various TensorFlow examples About the codes I didn't create these codes by myself, but re-crea

Terry Taewoong Um 614 Dec 08, 2022
Readings for "A Unified View of Relational Deep Learning for Polypharmacy Side Effect, Combination Therapy, and Drug-Drug Interaction Prediction."

Polypharmacy - DDI - Synergy Survey The Survey Paper This repository accompanies our survey paper A Unified View of Relational Deep Learning for Polyp

AstraZeneca 79 Jan 05, 2023
As-ViT: Auto-scaling Vision Transformers without Training

As-ViT: Auto-scaling Vision Transformers without Training [PDF] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou In ICLR 2

VITA 68 Sep 05, 2022
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis Andreas Bl

CompVis Heidelberg 36 Dec 25, 2022
Trax — Deep Learning with Clear Code and Speed

Trax — Deep Learning with Clear Code and Speed Trax is an end-to-end library for deep learning that focuses on clear code and speed. It is actively us

Google 7.3k Dec 26, 2022