Next step in image and video generation: Consistent Self-Attention

Amazing steps toward consistent story boards and video generation, which Sora and all the other video models currently lack: To create subject-consistent images to describe a story, they incorporate a Consistent Self-Attention into the pre-trained text-to-image diffusion model. They split a story text into several prompts and generate images using these prompts in a batch. The Consistent Self-Attention the builds connections among multiple images in a batch for subject consistency across the story board.

https://storydiffusion.github.io/

0 comments