4 Methodologies in Text to Image Tools ｜DAFTCRAFT ENGINEER BLOG

Let’s see in detail about all 4 methodologies.

Autoregressive model

This is the model that generates images from natural language descriptions. This model iteratively draws patches on a canvas, while attending to the relevant words in the description. After training on Microsoft COCO, this model is compared with several baseline generative models on image generation and retrieval tasks. 

GAN(Generative Adversarial Networks)

Two papers ”AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks” and “Cross-Modal Contrastive Learning for Text-to-Image Generation ”showed a promise of using GAN for creating images from text.

GAN is a machine learning model in which 2 neural networks compete with each other to become more accurate in their result or predictions. There are 2 neural networks:

1. Generator
2. Discriminator

The generator is the convolutional neural network and the Discriminator is the deconvolutional neural network. The generator’s goal is to artificially manufacture output that can be easily mistaken for real data. The goal of the discriminator is to identify which data is artificially created, which means the generator USS the latent space and by using the latent space it creates the content, and that content is given to the discriminator to validate or predict for its correctness and the discriminator uses the real dataset and real content in order to generate the result.

There will be a back propagation between the result, generator, generated content, and the discriminator. There will be a loop in Generator and the generator will start producing more accurate results in the later stages of the loop. 

VQ-VAE Transformer

It is introduced in the paper ”Zero-Shot Text-to-Image Generation” and “Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors”. To understand how it actually works, let’s look at the DALL-E mini first. This model takes images from three different datasets.

1. Conceptual 12M datasets (Which contains 12 million images and caption pairs).
2. Conceptual caption dataset(Which contains 3 million image and caption pairs).
3. Open AI subset of YFCC100M(Which contains 15 million image and caption pairs but only 2 million is taken from that because of the storage space limitations).

In this model, the images are taken by the VQ-GAN encoder and converted into a sequence of tokens. The BART encoder takes image captions. The output of the BART encoder and the encoded image is fed through the BART decoder which is an autoregressive model whose goal is to predict on next token. The sequence of image token decodes with the help of VQ-GEN decoder and finally, the output is chosen from the clip based on the requirement.  

Defusion model

These are generative models, which means they are used to generate data similar data on which they are framed.

Fundamentally they work by destroying the training data through the successive addition of Gozian noise after learning to recover the data by recovering the noising process. Let’s see talk about Google’s IMAGEN. It uses a large amount of frozen text encoder to encode text into the text embeddings. The conditional diffusion model maps the text into 64X64 images. then it utilizes the text, conditional super-resolution model, to upsample the image from 64X64 to 256X256 image and again there will be a work of super-resolution model which gain upsample the image from 256X256 to 1024 X 1024 as a final result.