One-sentence Summary

Quantized Side Tuning (QST), proposed by researchers at Carnegie Mellon University and Tsinghua University among other institutions, combines 4-bit weight quantization with a separate side network that avoids backpropagation through the LLM, reducing all three sources of memory overhead (model weights, optimizer states, and intermediate activations) by up to $2.3\times$2.3× and accelerating finetuning by up to $3\times$3×.

Key Contributions

• QST provides a memory-efficient LLM finetuning framework that quantizes LLM weights to 4-bit and trains a separate side network on LLM hidden states, reducing weight memory and avoiding backpropagation through the LLM to lower activation memory.

• QST further reduces optimizer-state memory by using low-rank adaptors and gradient-free downsample modules to shrink the trainable parameters required for task-specific adaptation.

• Experiments show that QST reduces total finetuning memory by up to 2.3× and speeds up finetuning by up to 3× while achieving comparable performance to state-of-the-art methods, with up to 7× total memory reduction compared with full finetuning.

Introduction

Large language models are increasingly adapted to specialized applications, but fine-tuning very large models is memory intensive, with full fine-tuning sometimes requiring hundreds of gigabytes of memory. Existing parameter-efficient fine-tuning methods reduce trainable parameters, yet they often still store most intermediate activations for backpropagation, while methods like QLoRA reduce weight and optimizer memory but do not adequately address activation memory, especially with larger batches. Prior side-network approaches also remain difficult to scale to very large models because the base LLM still consumes substantial memory. The authors propose Quantized Side-Tuning, which quantizes the LLM to 4-bit and trains a separate side network that consumes downsampled hidden states from the LLM, thereby reducing weight, activation, and optimizer memory while speeding up fine-tuning and maintaining competitive performance.

Method

The authors build Quantized Side Tuning around two coupled mechanisms: low-bit storage of the pretrained language model weights and a parallel trainable side network that recovers the precision loss caused by quantization. In the first mechanism, the original 16-bit weights are mapped to a 4-bit dtype. For a tensor $X^{16bit}$X16bit, the authors compute

$$\hat{X}^{4bit} = \operatorname{round}\left( \frac{M_{4bit}}{\operatorname{Absmax}(X^{16bit})} X^{16bit} \right) = \operatorname{round}\left( c^{16bit} \cdot X^{16bit} \right),$$X^4bit=round(Absmax(X16bit)M4bit​​X16bit)=round(c16bit⋅X16bit),

where $\hat{X}^{4bit}$X^4bit is the quantized tensor, $M_{4bit}$M4bit​ is the maximum representable value of the 4-bit dtype, and $c^{16bit}$c16bit is the quantization scale. The method supports both NF4 and FP4, with NF4 treated as an information-theoretically motivated dtype that balances the number of values assigned to each quantization bin. Dequantization is then defined as

$$\operatorname{dequant}(c^{16bit}, \hat{X}^{4bit}) = \frac{\hat{X}^{4bit}}{c^{16bit}} \approx X^{16bit}.$$dequant(c16bit,X^4bit)=c16bitX^4bit​≈X16bit.

A major source of error in this scalar-scale formulation is the presence of outlier values, which can stretch the scale and leave many quantization bins underused. To mitigate this, the authors reshape $X \in \mathbb{R}^{b \times h}$X∈Rb×h into a one-dimensional array and partition it into

$$n = \frac{b \times h}{B}$$n=Bb×h​

contiguous blocks, each of size $B$B. Each block is quantized independently with its own scale, which improves robustness to outliers. Because storing many scales can introduce additional memory overhead, the quantization scales are themselves stored using an 8-bit floating format. For a linear layer, the forward computation therefore uses dequantized weights:

$$Y^{16bit} = \operatorname{dequant}\left( \operatorname{dequant}(c_2^{16bit}, c_1^{8bit}), W^{4bit} \right) X^{16bit}.$$Y16bit=dequant(dequant(c216bit​,c18bit​),W4bit)X16bit.

This design keeps the base language model in 4-bit memory while still using 16-bit computation during the forward pass, reducing weight storage and improving execution efficiency on modern accelerators.

To explain why the side network is needed, the authors decompose training memory into three main components. M1 is the weight memory of the LLM, M2 is the optimizer state, which for Adam is several times the size of the trainable parameters, and M3 is the memory occupied by intermediate activations during backpropagation. Quantized weights reduce M1, but if gradients still flow through the full LLM computation graph, activation memory M3 can remain large. The side network is introduced to reduce all three components.

The per-layer connection between the quantized LLM branch and the trainable side branch is shown below.

The side network $g$g is a lightweight version of the quantized LLM $f$f. Its hidden-state and weight dimensions are $r$r times smaller than those of $f$f. At the $i$i-th layer, the side hidden state is computed from the hidden state of the quantized LLM as

$$h_{g_i}^{16bit} = (1 - \beta_i) \operatorname{downsample}_i(h_{f_i}^{16bit}) + \beta_i h_{g_{i-1}}^{16bit},$$hgi​16bit​=(1−βi​)downsamplei​(hfi​16bit​)+βi​hgi−1​16bit​,

where

$$\beta_i = \operatorname{sigmoid}(\gamma_i)$$βi​=sigmoid(γi​)

and $\gamma_i$γi​ is a learnable zero-initialized scalar. The gate $\beta_i$βi​ controls how much information is taken from the downsampled output of the quantized LLM versus the previous side network state. For the first layer, the output of the embedding layer and the downsampled embedding layer initialize $h_{f_0}^{16bit}$hf0​16bit​ and $h_{g_0}^{16bit}$hg0​16bit​.

The downsample module reduces the high-dimensional hidden state of $f$f to the low-dimensional hidden state of $g$g. A direct linear projection would require too many trainable parameters, especially when the LLM has many layers and large hidden dimensions. The authors therefore use parameter-efficient or parameter-free downsampling alternatives. LoRA replaces a full projection with a factorized projection,

$$W = L_1 L_2,$$W=L1​L2​,

where $W \in \mathbb{R}^{d_{in} \times d_{out}}$W∈Rdin​×dout​, $L_1 \in \mathbb{R}^{d_{in} \times d_r}$L1​∈Rdin​×dr​, and $L_2 \in \mathbb{R}^{d_r \times d_{out}}$L2​∈Rdr​×dout​. Adapter is similar but adds a nonlinear function between $L_1$L1​ and $L_2$L2​. MaxPooling and AvgPooling introduce no extra trainable parameters. Among these choices, the Adapter-based downsample performs best in their empirical analysis.

After the side network has produced its final hidden state, QST aligns the side output with the full LLM output dimension and combines it with the last hidden state of the quantized LLM. The final representation sent to the language modeling head is

$$h_N^{16bit} = \alpha h_{f_N}^{16bit} + (1 - \alpha) h_{g_N}^{16bit},$$hN16bit​=αhfN​16bit​+(1−α)hgN​16bit​,

where $\alpha$α is a learnable parameter initialized to $1$1. This initialization preserves the pretrained model’s starting point at the beginning of fine-tuning. Since the pretrained LLM remains fixed as a 4-bit tensor, task adaptation can be achieved by updating only the side network rather than redeploying the full model.

During training, the authors update only the parameters of the side network $g$g. The gradient of the loss with respect to $g$g is computed without explicitly differentiating the quantized backbone weights in $f$f. In other words, the method avoids backpropagating through the 4-bit weight storage path of the frozen LLM. This reduces the number of trainable parameters, shrinks the optimizer state, and limits the activation memory associated with a full backward pass through the backbone.

Experiment

The experiments evaluate QST across OPT and LLaMA-2 models from small to large sizes on GLUE, MMLU, and chatbot generation, comparing it with QLoRA, LST, LoRA, and Adapter. The results show that QST achieves substantially lower memory usage and trainable parameters while preserving competitive accuracy, with some gains over QLoRA on larger models and MMLU. Additional analyses indicate that QST remains robust across batch sizes, sequence lengths, and model sizes, trains faster, and benefits from carefully chosen reduction factors, NF4 quantization, and adapter-based downsampling. On a GPT-4 judged MT-Bench chatbot task, QST improves training efficiency and final response quality over QLoRA, including surpassing the original LLaMA-2-70B.

On the GLUE benchmark, QST has the lowest memory footprint and very few trainable parameters while remaining close to the average accuracy of stronger memory-heavy methods. For larger OPT models, QST offers a better accuracy-efficiency tradeoff than QLoRA, lowering memory and trainable parameters with only a modest drop in average accuracy. The method also appears more stable across computation data types than QLoRA, which can fail on some GLUE datasets under FP16. QST keeps accuracy competitive while using less memory and fewer trainable parameters than QLoRA, LST, LoRA, and Adapter. For larger OPT models, QST gives a better accuracy-memory tradeoff than QLoRA with a small average accuracy loss. QST remains stable across BF16 and FP16 computation settings, while QLoRA can fail to finetune some GLUE datasets.

Across the MMLU 5-shot finetuning results, QST matches or slightly improves accuracy compared with QLoRA while using substantially less memory on the tested OPT and LLaMA-2 models. The accuracy difference is generally small overall, with the clearest gain for QST on LLaMA-2-13B and modest gains on several OPT models. QST reduces memory footprint relative to QLoRA across OPT and LLaMA-2 model scales. QST maintains competitive MMLU accuracy, showing its strongest relative accuracy gain on LLaMA-2-13B.

The reported training throughput results show that computational cost per token increases for every method as model size grows from 7B to 70B. QST consistently achieves the lowest cost per token and provides roughly a 2.5 times speedup over the baselines. LST has the highest cost, especially at larger model sizes, while QLoRA is slightly more expensive than LoRA and Adapter due to additional LoRA components. QST gives the best training throughput across all tested model sizes. LST has the worst throughput and becomes especially costly on the largest model. Adding more LoRA components makes QLoRA slightly less efficient than LoRA and Adapter.

Experiments on LLaMA-2 models compare FP4 and NF4 for 4-bit finetuning on the MMLU benchmark. NF4 consistently achieves slightly higher accuracy than FP4 across model sizes and is selected as the default 4-bit data type. NF4 improves accuracy over FP4 for the 7B, 13B, and 70B LLaMA-2 models. The average benefit of NF4 over FP4 is small but consistent enough to motivate its default use.

Under FP16 computation, QST remains stable and achieves substantially higher MRPC and QNLI scores than QLoRA. The comparison also indicates that QLoRA can be unstable with FP16, including failed finetuning attempts on some tasks. QST performs consistently well with FP16 and outperforms QLoRA on the reported tasks. QLoRA shows reduced robustness under FP16, with much lower scores and reported finetuning instability.

QST is evaluated on GLUE and MMLU benchmarks and compared with QLoRA, LST, LoRA, and Adapter to assess its accuracy, memory efficiency, and stability. The results show that QST uses substantially less memory and fewer trainable parameters while maintaining competitive accuracy, and it provides the best training throughput across model sizes with about a 2.5 times speedup over baselines. Additional experiments support NF4 over FP4 as the default 4-bit format and show that QST remains more robust than QLoRA under FP16 computation, where QLoRA can become unstable or fail.