One-sentence Summary

Researchers from the University of Southern California and Yale University propose KRONQ, a post-training quantization framework that leverages gradient covariance under the Kronecker-factored Hessian approximation to enable bidirectional incoherence processing and a novel sensitivity metric for mixed-precision allocation, achieving 7.93 perplexity on 2-bit weight-only LLaMA-3-70B where GPTQ and GPTAQ diverge.

Key Contributions

• KRONQ is a post-training quantization framework that incorporates gradient covariance into the quantization pipeline, enabling bidirectional incoherence processing that extends input-side random rotation to the output dimension and reduces weight magnitude variance across both dimensions.
• It introduces a new inter-layer mixed-precision sensitivity metric computed from the product of gradient and activation Hessian traces, which resolves ambiguities in sublayers that share identical input statistics and supports optimal bit-width allocation.
• Experiments on LLaMA-2 and LLaMA-3 models from 7B to 70B in weight-only and weight-and-activation settings at 2/3/4-bit show consistent state-of-the-art performance. The largest improvements occur at 2-bit, where alternative methods diverge or produce degenerate quantizations, and KRONQ achieves 7.93 perplexity on LLaMA-3-70B.

Introduction

Large language models with hundreds of billions of parameters demand compression for practical deployment. Post-training quantization (PTQ) reduces memory and latency by converting weights and activations to low-bit representations, and among PTQ approaches, compensation-based methods like GPTQ rely on the input activation covariance to approximate the Hessian when correcting rounding errors. However, these methods ignore the output-side gradient structure, implicitly assuming equal importance across output channels and missing the gradient covariance term in the Kronecker-factored Hessian. The authors introduce KRONQ, which augments the quantization objective with the gradient covariance H_G obtained from a single backward pass. This addition enables bidirectional incoherence processing and a joint Hessian-trace sensitivity metric for inter-layer mixed precision, while algebraically canceling in the weight update so that the efficient GPTAQ solver remains unchanged. KRONQ achieves state-of-the-art accuracy on LLaMA-2 and LLaMA-3 at 2- to 4-bit, with the largest gains at ultra-low precisions.

Method

The authors propose KRONQ, a post-training quantization (PTQ) framework that addresses the limitations of existing compensation-based methods like GPTQ and GPTAQ. Standard methods rely solely on the input activation covariance $H_X$HX​ as a proxy for the full Hessian, implicitly assuming the output-side gradient covariance $H_G$HG​ is an identity matrix. However, output channels exhibit substantial variation in gradient magnitude. As shown in the figure below:

The normalized diagonal entries of the gradient covariance $H_G$HG​ for Q, K, V, and O projections vary by orders of magnitude, revealing heterogeneous output-side sensitivity. To capture this missing factor, the authors leverage the Kronecker-factored approximation (K-FAC).

For a linear layer $y = Wx$y=Wx, the per-sample gradient factorizes as $\frac{\partial \mathcal{L}}{\partial W} = gx^\top$∂W∂L​=gx⊤, where $g = \partial \mathcal{L} / \partial y$g=∂L/∂y. The empirical Fisher approximation of the full Hessian $H$H is given by:

Applying the K-FAC independence assumption $x \perp\!\!\!\perp g$x⊥⊥g, the expectation factorizes into $H \approx H_X \otimes H_G$H≈HX​⊗HG​, where $H_X = \mathbb{E}[xx^\top]$HX​=E[xx⊤] and $H_G = \mathbb{E}[gg^\top]$HG​=E[gg⊤]. This reduces the complexity from $O(d_{in}^2 d_{out}^2)$O(din2​dout2​) to $O(d_{in}^2 + d_{out}^2)$O(din2​+dout2​). Substituting this into the standard layer-wise PTQ objective yields the Kronecker-factored quantization objective:

where $\Delta W = W - \widehat{W}$ΔW=W−W. Building on GPTAQ, the authors further correct for input drift by incorporating asymmetric calibration, resulting in the final objective:

Under the column-wise Optimal Brain Surgeon (OBS) update applied to this objective, the weight compensation after quantizing column $p$p reduces to:

where $\delta_p$δp​ is the scaled quantization error and $P$P is the GPTAQ asymmetric correction matrix.

Quantization error is reduced when the proxy Hessian is incoherent, meaning its eigenvectors are not aligned with the coordinate axes. While previous methods applied incoherence processing to $H_X$HX​, the authors investigate whether $H_G$HG​ is also coherent. This motivates Bidirectional Incoherence Processing (BiIP). As shown in the figure below:

The incoherence measure $\mu(H_G)/\sqrt{d_{out}}$μ(HG​)/dout​​ reaches up to 0.99 before processing but drops below 0.10 after incoherence processing, confirming that output-side rotation effectively incoherences $H_G$HG​. Extending input-side diagonal rescaling to both column and row directions, the authors apply:

where the output-side $S_G$SG​ term is novel. Subsequently, orthogonal transforms are applied:

instantiating $U$U and $V$V as randomized Hadamard transforms. This ensures both $H_G$HG​ and $H_X$HX​ become incoherent. The figure demonstrates that while incoherencing $H_X$HX​ alone leaves the output-channel coefficient of variation ($CV_{out}$CVout​) nearly unchanged, BiIP reduces both $CV_{in}$CVin​ and $CV_{out}$CVout​ significantly.

To allocate a bit budget across sublayers, the authors rank them according to a sensitivity metric. Under the second-order approximation, the expected quantization loss for layer $\ell$ℓ is proportional to $\epsilon_\ell^2 \cdot \text{tr}(H^{(\ell)})$ϵℓ2​⋅tr(H(ℓ)). Using the Kronecker approximation $H \approx H_X \otimes H_G$H≈HX​⊗HG​, this extends naturally to both factors:

The authors define the KRONQ sensitivity score as $s_\ell = \text{tr}(H_G^{(\ell)}) \cdot \text{tr}(H_X^{(\ell)})$sℓ​=tr(HG(ℓ)​)⋅tr(HX(ℓ)​), computed after BiIP, and allocate higher bit-widths to sublayers with larger $s_\ell$sℓ​. As shown in the figure below:

The sublayer rankings under $\text{tr}(H_G) \cdot \text{tr}(H_X)$tr(HG​)⋅tr(HX​) differ substantially from those under $\text{tr}(H_X)$tr(HX​) alone. Within a transformer attention block, the query (Q), key (K), and value (V) projections share the same input, so $H_X^{(Q)} = H_X^{(K)} = H_X^{(V)}$HX(Q)​=HX(K)​=HX(V)​. The joint score breaks this degeneracy via $H_G$HG​, which differs across Q, K, and V as they receive different downstream gradients, yielding strictly better perplexity and bit-width tradeoffs.

Experiment

KRONQ is evaluated on LLaMA models across weight-only, group, and weight-and-activation quantization regimes against GPTQ and GPTAQ, using perplexity and zero-shot accuracy on reasoning benchmarks. The method consistently achieves the lowest perplexity and highest accuracy, with the largest gains at ultra-low bit-widths where activation-covariance-only methods fail, and it successfully quantizes LLaMA-3-70B where baselines diverge. These improvements generalize to newer models and harder benchmarks, while mixed-precision allocation using a joint input-output sensitivity score further optimizes the perplexity-bit tradeoff. Ablations confirm that output-side curvature, diagonal rescaling, and bidirectional incoherence processing are all essential, and the approach delivers significant memory savings and decoding speedups.

At 4-bit per-channel weight-only quantization, several methods like OSTQuant and SpinQuant achieve perplexity close to the FP16 baseline on LLaMA models, while the text reports that KRONQ achieves the lowest perplexity across all bit-widths, with the largest gains at 2 and 3 bits where other approaches degrade severely. Zero-shot accuracy improvements mirror the perplexity trends, with KRONQ consistently outperforming baselines on all reasoning benchmarks. Among the listed 4-bit methods, OSTQuant records the lowest WikiText-2 perplexity on LLaMA-2-7B (5.64) and the highest average zero-shot accuracy on PiQA, ArcC, and WG (64.0). KRONQ delivers the most significant perplexity reductions at W2 and W3, where activation-covariance-only methods fail, and is the only method that produces valid quantizations on LLaMA-3-70B at those bit-widths.

Under 2-bit weight-only group quantization, KRONQ delivers the lowest perplexity and highest zero-shot accuracy on LLaMA-2-7B, reaching 7.61 perplexity compared to 274.0 for GPTQ and 11.06 for OmniQuant, with an average accuracy of 53.7. The text indicates that KRONQ also maintains its lead over GPTQ and GPTAQ on LLaMA-2-13B. On LLaMA-2-7B, GPTQ collapses to 274.0 perplexity under group quantization, whereas KRONQ achieves 7.61, outperforming OmniQuant (11.06) and GPTAQ (23.19). KRONQ attains an average zero-shot accuracy of 53.7 on LLaMA-2-7B, surpassing OmniQuant (48.3) and GPTQ (39.3), with gains across all listed reasoning benchmarks.

KRONQ consistently achieves the lowest perplexity and highest average zero-shot accuracy under weight-and-activation quantization at W2A4 across LLaMA-2 and LLaMA-3 models. The largest perplexity reduction occurs on LLaMA-2-7B, where KRONQ lowers perplexity from 36.74 to 9.38 compared to GPTQ. Accuracy gains mirror the perplexity improvements, with KRONQ outperforming both GPTQ and GPTAQ on all benchmarks. KRONQ reduces perplexity on LLaMA-2-7B from 36.74 (GPTQ) to 9.38, a 74% relative drop. On LLaMA-3-8B, KRONQ's average zero-shot accuracy reaches 51.0, surpassing GPTAQ by 8.8 points.

Across four recent model families, KRONQ achieves the lowest WikiText-2 perplexity at both W4 and W2. The margin widens dramatically at W2, where baseline methods often suffer catastrophic degradation while KRONQ remains robust, and in one case GPTAQ even underperforms GPTQ while KRONQ still excels. KRONQ's perplexity advantage over GPTQ and GPTAQ widens sharply at 2-bit quantization, with gains often exceeding an order of magnitude on newer models. On DeepSeek-R1-Distill-Llama-8B at W2, GPTAQ perplexity is worse than GPTQ, yet KRONQ achieves a far lower perplexity, highlighting the complementary benefit of its output-side correction.

KRONQ consistently outperforms GPTQ and GPTAQ across four hard reasoning benchmarks for both DeepSeek-R1-Distill-Llama-8B and Gemma-3-12B-IT under weight-only 4-bit quantization. The most dramatic gains occur on generative coding tasks, where KRONQ nearly doubles the next best score on both models. Substantial improvements on math and knowledge benchmarks confirm that the method generalizes well beyond perplexity. KRONQ achieves the highest accuracy on every model–benchmark combination. On LiveCodeBench, KRONQ nearly doubles the score of the next best method for both DeepSeek-R1-Distill-Llama-8B and Gemma-3-12B-IT. On the AIME-2024 math benchmark, KRONQ more than doubles the accuracy of GPTQ and GPTAQ on Gemma-3-12B-IT. KRONQ's advantage over GPTAQ is largest on tasks requiring complex reasoning and generation, such as code and math.

The evaluation spans weight-only, group, and weight-and-activation quantization across LLaMA, DeepSeek, and Gemma models, measuring perplexity and zero-shot accuracy on reasoning benchmarks. KRONQ consistently achieves the lowest perplexity and highest accuracy, with the advantage becoming most pronounced at 2- and 3-bit widths where competing methods like GPTQ and GPTAQ often collapse or degrade severely. The method generalizes well beyond perplexity, delivering substantial gains on hard reasoning tasks including code generation and math, and remains robust even on newer model families where baseline approaches fail.