NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis

In this paper, we present NUWA-Infinity, a generative model for infinitevisual synthesis, which is defined as the task of generating arbitrarily-sizedhigh-resolution images or long-duration videos. An autoregressive overautoregressive generation mechanism is proposed to deal with this variable-sizegeneration task, where a global patch-level autoregressive model considers thedependencies between patches, and a local token-level autoregressive modelconsiders dependencies between visual tokens within each patch. A NearbyContext Pool (NCP) is introduced to cache-related patches already generated asthe context for the current patch being generated, which can significantly savecomputation costs without sacrificing patch-level dependency modeling. AnArbitrary Direction Controller (ADC) is used to decide suitable generationorders for different visual synthesis tasks and learn order-aware positionalembeddings. Compared to DALL-E, Imagen and Parti, NUWA-Infinity can generatehigh-resolution images with arbitrary sizes and support long-duration videogeneration additionally. Compared to NUWA, which also covers images and videos,NUWA-Infinity has superior visual synthesis capabilities in terms of resolutionand variable-size generation. The GitHub link ishttps://github.com/microsoft/NUWA. The homepage link ishttps://nuwa-infinity.microsoft.com.