One-sentence Summary

Cantina Labs, USC, and UIUC researchers propose OrbitQuant, a data-agnostic weight-activation quantizer that normalizes activations via a randomized permuted block-Hadamard (RPBH) rotation to a fixed distribution for a single Lloyd–Max codebook across timesteps and modalities, absorbs the rotation into weights offline, and sets state-of-the-art low-bit post-training quantization on image and video diffusion transformers including FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, while pushing image diffusion transformers to $W_2A_4$W2​A4​ with usable generation quality.

Key Contributions

• OrbitQuant introduces a data-agnostic weight-activation quantizer that operates in a normalized, rotated basis, where a single Lloyd–Max codebook serves all timesteps, prompts, and layers of a given input dimension.
• The rotation is absorbed into the weights offline, cancels inside each linear layer, and leaves only a forward RPBH rotation on the activations at runtime. The random permutation of the RPBH rotation ensures well-behaved rotated marginals at low bit-width without calibration.
• Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, the same recipe transfers unchanged from image to video without per-modality tuning. It sets the state of the art for PTQ on GenEval and VBench at several low-bit settings and, at W2A4, is the only method that produces usable images where prior PTQ baselines collapse to noise.

Introduction

Diffusion Transformers (DiTs) now power state-of-the-art image and video generation, but their compute-bound inference makes low-bit weight-and-activation quantization essential. Existing post-training quantization (PTQ) approaches inherit from large language models and rely on calibration data to absorb outlier shifts; however, DiT activations drift across timesteps, prompts, and classifier-free guidance branches, breaking the static-statistics assumption and demanding new calibration for every checkpoint or modality. The authors introduce OrbitQuant, a calibration-free PTQ framework that rotates activations into a common normalized basis where their coordinates follow a fixed, known marginal distribution, regardless of the input. A single offline Lloyd–Max codebook then uniformly quantizes all activations across denoising steps. The rotation is folded into the weight rows so it cancels inside each linear layer, leaving only a lightweight randomized permuted block-Hadamard (RPBH) transform at runtime. OrbitQuant achieves state-of-the-art results on both image and video DiTs without any calibration data, and it remains usable at W2A4 bit-widths where prior methods collapse.

Method

The authors propose OrbitQuant, a calibration-free quantization framework that replaces per-input range calibration with a distributional quantizer applied in a shared, rotated, and normalized basis. Because weights and activations are quantized in the same basis, the rotation cancels out in the matrix product, leaving only a forward rotation on the activation at runtime.

The framework operates through two distinct stages: offline weight quantization and online activation quantization. For a linear layer with input dimension $d$d, the authors apply a shared rotation $\Pi_d$Πd​. During the offline stage, the weight matrix is rotated into this basis as $\mathbf{W}' = \mathbf{W} \boldsymbol{\Pi}_d^\top$W′=WΠd⊤​. Each row of $\mathbf{W}'$W′ is then decomposed into a magnitude $r_i' = \|\mathbf{w}_i'\|_2$ri′​=∥wi′​∥2​ and a unit direction $\tilde{\mathbf{w}}_i' = \mathbf{w}_i' / r_i'$w~i′​=wi′​/ri′​. The direction is quantized using a Lloyd-Max codebook, and the magnitude is re-attached, yielding $\hat{\mathbf{W}}' = \text{diag}(\mathbf{r}') \cdot \hat{Q}_{b_w}^{(d)}(\tilde{\mathbf{W}}')$W^′=diag(r′)⋅Q^​bw​(d)​(W~′). The row-norm vector is stored in BF16, adding negligible overhead.

At inference time, each incoming activation $\mathbf{x}$x is rotated by $\Pi_d$Πd​ before entering the layer, producing $\mathbf{x}' = \boldsymbol{\Pi}_d \mathbf{x}$x′=Πd​x. This is split into a magnitude $s = \|\mathbf{x}'\|_2$s=∥x′∥2​ and a unit direction $\tilde{\mathbf{x}}' = \mathbf{x}' / (s + \varepsilon)$x~′=x′/(s+ε). The direction is quantized with the Lloyd-Max quantizer $\hat{Q}_{b_a}^{(d)}$Q^​ba​(d)​ and rescaled by $s$s, resulting in $\hat{\mathbf{x}}' = s \cdot \hat{Q}_{b_a}^{(d)}(\tilde{\mathbf{x}}')$x^′=s⋅Q^​ba​(d)​(x~′). Since the weight absorbs $\Pi_d^\top$Πd⊤​ and the activation applies $\Pi_d$Πd​, the two cancel in the product, allowing the quantized layer to compute $\hat{\mathbf{W}}' \hat{\mathbf{x}}' \approx \mathbf{W} \mathbf{x}$W^′x^′≈Wx without an inverse rotation at runtime.

To make the rotation efficient, the authors realize $\Pi_d$Πd​ as a randomized permuted block-Hadamard (RPBH) transform. It is defined as $\boldsymbol{\Pi}_d = \text{blkdiag}(\mathbf{H}_h \mathbf{D}_1, \dots, \mathbf{H}_h \mathbf{D}_{d/h}) \cdot \mathbf{P}_\pi$Πd​=blkdiag(Hh​D1​,…,Hh​Dd/h​)⋅Pπ​, where $\mathbf{H}_h$Hh​ is a Walsh-Hadamard matrix, $\mathbf{D}_i$Di​ are Rademacher sign diagonals, and $\mathbf{P}_\pi$Pπ​ is a uniform random permutation matrix. The leading permutation $\mathbf{P}_\pi$Pπ​ is crucial as it spreads coordinates across blocks, ensuring that the variance of each coordinate of $\Pi_d \tilde{\mathbf{x}}$Πd​x~ stays close to $1/d$1/d. This keeps the marginal distribution close to $\mathcal{N}(0, 1/d)$N(0,1/d), making the Lloyd-Max codebook near-optimal.

Because the rotated coordinates consistently follow the fixed marginal $f_d$fd​, the authors run Lloyd-Max on $f_d$fd​ offline to obtain a single codebook $\mathcal{C}_d$Cd​ per dimension. This data-agnostic codebook serves every timestep, prompt, layer, and weight row of dimension $d$d, completely eliminating the need for input statistics collection or calibration. As demonstrated in the figure, raw activations deviate sharply from $f_d$fd​, but after the RPBH rotation, both weights and activations match the target distribution as closely as a dense Haar rotation, allowing a single codebook to fit all operands effectively.

Experiment

Evaluation on image and video diffusion transformers shows OrbitQuant achieving state-of-the-art post-training quantization quality, with near-lossless W4A4 performance that can surpass full precision on GenEval and VBench, and robust W2A4 behavior where baselines collapse. OrbitQuant also exhibits the lowest quantization overhead in latency comparisons, and qualitative samples confirm its fidelity even at very low bit-widths. Ablation studies support the use of a randomized permuted block Hadamard rotation and INT4 AdaLN projections for optimal accuracy–efficiency trade-offs.

OrbitQuant achieves lossless or better-than-FP16 GenEval performance at W4A4 on two of three image diffusion transformers and trails by a tiny margin on the third, surpassing all post-training quantization baselines to set a new state of the art. At W2A4, the advantage grows: while rotation and smoothing methods collapse to near-zero scores on every model, OrbitQuant remains functional, retaining most of its quality on the FLUX models and being the sole method that produces meaningful results on Z-Image-Turbo. OrbitQuant exceeds FP16 on GenEval Overall for FLUX.1-schnell and Z-Image-Turbo at W4A4, outperforming every PTQ baseline. At W2A4, competing methods degrade to near-zero scores, whereas OrbitQuant stays usable and uniquely delivers meaningful scores on Z-Image-Turbo.

On video generation benchmarks, OrbitQuant outperforms other post-training quantization methods at both W4A6 and W4A4 precision. It achieves the highest Overall Consistency on Wan 2.1-1.3B and CogVideoX-2B, leading on most individual quality dimensions, while competing methods degrade more sharply at the lower bit-width. At W4A6 on Wan 2.1-1.3B, OrbitQuant attains the best Overall Consistency (24.35), ahead of the next-best method SVDQuant (23.26). OrbitQuant leads in Imaging Quality, Aesthetic Quality, Dynamic Degree, Background Consistency, Subject Consistency, and Scene at W4A6 on Wan 2.1-1.3B. Motion Smoothness is the only W4A6 metric where OrbitQuant is not the top method; it ranks second (97.76) behind SmoothQuant (98.01). At W4A4, OrbitQuant stays closest to full precision on most dimensions and again ranks first in Overall Consistency on both Wan 2.1-1.3B and CogVideoX-2B, while baselines lose ground.

At W4A4 all rotation methods perform within noise, but at lower bit-widths RPBH yields the best GenEval scores while matching the speed of other fast Hadamard transforms. The dense Haar rotation is an order of magnitude slower because it cannot exploit a fast kernel. RPBH achieves the highest GenEval Overall at W3A3 and W2A4, outperforming Block-RHT, Full RHT, and Haar. Removing the random permutation degrades low-bit performance (Block-RHT), showing the permutation's role in spreading outliers for codebook compatibility. Structured Hadamard transforms (Full RHT, Block-RHT, RPBH) run over 25× faster than the dense Haar rotation, with RPBH adding only a small overhead over Block-RHT.

OrbitQuant sets a new state of the art in post-training quantization for diffusion models, achieving lossless or better-than-FP16 image quality at 4-bit weights and activations and remaining the sole functional method at 2-bit weights where all baselines collapse. On video generation, it leads overall consistency and most quality dimensions at both W4A6 and W4A4 precision. Ablation on rotation transforms reveals that randomized permutation Hadamard (RPBH) yields the best low-bit performance by effectively spreading outliers, with only a small speed overhead over other fast structured transforms.