OpenAI: Look at our awesome image generator! Google: Hold my Shiba Inu

The AI world is still figuring out how to deal with the amazing show of prowess that is DALL-E 2’s ability to draw/paint/imagine just about anything… but OpenAI isn’t the only one working on something like that. Google Research has rushed to publicize a similar model it’s been working on — which it claims is even better.

Imagen (get it?) is a text-to-image diffusion-based generator built on large transformer language models that… okay, let’s slow down and unpack that real quick.

Text-to-image models take text inputs like “a dog on a bike” and produce a corresponding image, something that has been done for years but recently has seen huge jumps in quality and accessibility.

Part of that is using diffusion techniques, which basically start with a pure noise image and slowly refine it bit by bit until the model thinks it can’t make it look any more like a dog on a bike than it already does. This was an improvement over top-to-bottom generators that could get it hilariously wrong on first guess, and others that could easily be led astray.

The other part is improved language understanding through large language models using the transformer approach, the technical aspects of which I won’t (and can’t) get into here, but it and a few other recent advances have led to convincing language models like GPT-3 and others.

Image Credits: Google Research

Imagen starts by generating a small (64×64 pixels) image and then does two “super resolution” passes on it to bring it up to 1024×1024. This isn’t like normal upscaling, though, as AI super-resolution creates new details in harmony with the smaller image, using the original as a basis.

Say for instance you have a dog on a bike and the dog’s eye is 3 pixels across in the first image. Not a lot of room for expression! But on the second image, it’s 12 pixels across. Where does the detail needed for this come from? Well, the AI knows what a dog’s eye looks like, so it generates more detail as it draws. Then this happens again when the eye is done again, but at 48 pixels across. But at no point did the AI have to just pull 48 by whatever pixels of dog eye out of its… let’s say magic bag. Like many artists, it started with the equivalent of a rough sketch, filled it out in a study, then really went to town on the final canvas.

This isn’t unprecedented, and in fact artists working with AI models use this technique already to create pieces that are much larger than what the AI can handle in one go. If you split a canvas into several pieces, and super-resolution all of them separately, you end up with something much larger and more intricately detailed; you can even do it repeatedly. An interesting example from an artist I know: