One-sentence Summary

UNC Charlotte, DreamSoul, University of Minnesota, and Intel Corporation propose GISP (Global Iterative Structured Pruning), a post-training method that replaces task-agnostic layerwise reconstruction with global first-order loss-based importance scores aggregated at the structure level using block-wise normalization, adopts an iterative pruning schedule that stabilizes accuracy at higher sparsity without intermediate fine-tuning, forms nested subnetworks supporting a prune-once-deploy-many workflow, and defines structural importance directly with respect to a target loss to enable task-specific adaptation, yielding consistent WikiText-2 perplexity reductions and downstream accuracy gains across Llama2-7B/13B, Llama3-8B, and Mistral-0.3-7B with especially strong improvements at 40–50% sparsity.

Key Contributions

• GISP is a global iterative structured pruning method for LLMs that removes attention heads and MLP channels using first-order loss-based importance scores aggregated at the structure level with block-wise normalization, tying pruning to a target loss instead of relying only on layerwise reconstruction.

• Its iterative schedule forms nested subnetworks that support a prune-once, deploy-many workflow across sparsity levels, while task-aware calibration can use perplexity for language modeling or a margin-based objective for decision-style tasks.

• Experiments on Llama2-7B/13B, Llama3-8B, and Mistral-0.3-7B show lower WikiText-2 perplexity and improved downstream accuracy, especially at 40% to 50% sparsity, and GSM8K task-aligned calibration improves exact-match accuracy for DeepSeek R1-Distill-Llama-3-8B and Qwen3-8B.

Introduction

Pruning compresses large language models by removing redundant parameters, and structured pruning is especially useful because it produces compact, hardware-friendly architectures that run efficiently on constrained devices. Prior work mostly uses local pruning, which decomposes model compression into layer-wise reconstruction problems; this is simple and efficient, but it is task-agnostic and often fails to improve downstream performance even when calibration data are task-specific. Non-uniform local variants reduce this weakness, but they add algorithmic complexity and still rely on layer-level proxies rather than the full model objective. The authors leverage global pruning, defining importance with respect to a model-level loss, and propose GISP, a global iterative structured pruning framework for attention heads and MLP channels. GISP uses iterative, ratio-scheduled pruning to stabilize performance at high sparsity, reveals nested subnetworks that support a prune-once, deploy-many workflow, and directly incorporates task-aware objectives for stronger downstream accuracy.

Method

The proposed method centers on structured global pruning. Instead of pruning each layer independently, it estimates the importance of attention heads and MLP channels with respect to a model-level loss, ranks all candidate structures together, and removes the least important ones under a global budget. The authors first analyze a naive one-shot version and then identify why it becomes unstable at high sparsity. This analysis motivates their final method, Global Iterative Structured Pruning, or GISP.

The one-shot procedure starts from a calibration dataset

$$D = \{(x_i, y_i)\}_{i=1}^{N}.$$D={(xi​,yi​)}i=1N​.

For each weight element, the method computes a first-order importance score derived from the model-level loss. In local notation, the sensitivity of a weight element $w$w can be written as

$$I_w = \left|\langle \nabla_w L, w \rangle\right|.$$Iw​=∣⟨∇w​L,w⟩∣.

These element-level scores are then aggregated into structure-level scores. A structure is either one attention head or one MLP channel. Because different structures contain different numbers of weight elements, the authors normalize the aggregated scores so that all structures can be compared fairly. They also observe that attention blocks and MLP blocks can have substantially different score magnitudes, so normalization is applied separately within these two structural families. After normalization, the method globally ranks all attention heads and MLP channels, selects the lowest-scoring structures, and prunes them under the target pruning ratio $\rho$ρ.

This one-shot setting is evaluated without post-pruning fine-tuning. Fine-tuning after every pruning step would be prohibitively expensive for large language models, so the method operates as a post-training pruning procedure. The one-shot strategy can be effective at low pruning ratios, but its performance can degrade sharply at high sparsity. The authors attribute this limitation to the fact that removing many structures at once increases the risk of eliminating weights or units that remain important after the model has already changed.

To stabilize high-ratio pruning, the authors decompose the pruning process into multiple smaller pruning steps. Given a target pruning ratio $\rho$ρ and a predefined number of iterations $n$n, GISP applies a small incremental ratio $\rho_k$ρk​ at iteration $k$k. A linear scheduler controls the cumulative pruning budget across iterations. In a common instantiation, the cumulative target can be written as

$$\rho_k = \rho \frac{k}{n},$$ρk​=ρnk​,

so that each step removes a comparable number of structures while the total pruning ratio gradually reaches the final target. This iterative procedure leverages feedback from the already-pruned model at each step, allowing the method to refine which structures are truly redundant after earlier pruning decisions.

As shown in the framework diagram below:

The framework organizes each pruning iteration into three interacting stages. First, the scheduled target pruning ratio determines how much structure needs to be removed at the current step. Second, the method computes structured weight importance using the current model state and applies normalization so that attention heads and MLP channels can be compared within their respective structural groups. Third, the method performs global ranking over all candidate structures and prunes the lowest-ranked ones. The pruned model from the current iteration is then passed to the next iteration, where importance is recomputed on the updated network.

This iterative design produces a nested sequence of subnetworks. At each iteration, GISP removes new attention heads and MLP channels from the model that has already been pruned in previous steps. Consequently, an intermediate model at a lower pruning ratio is naturally a parent subnetwork of later, sparser models. This nested property gives GISP a once-for-all capability. If the method is run once toward a high target ratio, the intermediate models along the pruning trajectory can serve directly as deployable models at lower sparsity levels. The authors show that these intermediate models can achieve perplexity values close to those obtained by running separate, shorter pruning procedures for each target ratio. This is particularly valuable for practical deployment because one pruning run can yield an entire accuracy-versus-sparsity frontier rather than a single fixed subnetwork. The cost of the iterative trajectory is therefore amortized across many models, even though each individual pruning run may take longer than a one-shot method.

The method also extends naturally to task-specific pruning. Because GISP estimates importance with respect to a model-level objective rather than a layer-wise reconstruction loss, the objective can be replaced with a downstream task objective. In the general form, the importance score becomes

$$I_W = \left|\langle \nabla_W L_{\mathrm{task}}, W \rangle\right|.$$IW​=∣⟨∇W​Ltask​,W⟩∣.

For text generation or language modeling, $L_{\mathrm{task}}$Ltask​ can be token-level cross entropy:

$$L = -\frac{1}{N} \sum_{i=1}^{N} \log p(x_i \mid x_{<i}),$$L=−N1​i=1∑N​logp(xi​∣x<i​),

where $p(x_i \mid x_{<i})$p(xi​∣x<i​) is the probability of token $x_i$xi​ given the preceding context. This objective preserves generation quality and perplexity behavior.

For multiple option question answering or other decision-oriented tasks, minimizing perplexity on correct answers alone may be insufficient. In such tasks, the model often ranks candidate question-answer pairs by loss or perplexity and selects the candidate with the best score. If pruning reduces the loss of incorrect candidates faster than it reduces the loss of the correct candidate, the decision boundary can shift and accuracy can fall even when the correct answer’s loss remains favorable in absolute terms. To address this, the authors define a margin-based importance score. Let $L_+$L+​ denote the average loss on positive labels and $L_-$L−​ denote the average loss on negative labels. For a weight element $W_i^j$Wij​, the importance becomes

$$I_{W_i^j} = \left| \left( \frac{\partial L_+}{\partial W_i^j} - \frac{\partial L_-}{\partial W_i^j} \right) W_i^j \right|.$$IWij​​=​(∂Wij​∂L+​​−∂Wij​∂L−​​)Wij​​.

This score prioritizes weights that help preserve the loss gap between correct and incorrect candidates. In this way, GISP remains a post-training, structure-aware pruning framework while becoming aligned with the actual task objective rather than only with generic language-model reconstruction.

Experiment

Experiments evaluate GISP on several LLMs using text-generation perplexity, commonsense reasoning, arithmetic generation, ablations, and calibration robustness tests. The studies show that iterative global pruning stabilizes aggressive structured pruning, yields smoother subnetwork trajectories, and outperforms representative local baselines while enabling a once-for-all family of sparsity levels from a single pruning run. Additional analyses confirm GISP's task-aware behavior, better adaptation after fine-tuning, and robustness to limited or varied calibration data.

The reported results show that local pruning accuracy falls as pruning ratios increase, with larger degradation at higher pruning ratios. Using a task informed calibration dataset gives only limited gains for local pruning, and the benefit varies by method rather than producing a consistent improvement. Local pruning does not gain much from task informed calibration. Higher pruning ratios reduce CMQA accuracy noticeably. The effect of calibration data differs across methods.

The evaluation compares one-shot global pruning with Wanda-sp on WikiText-2 and PTB perplexity across increasing pruning ratios. One-shot global pruning shows lower perplexity than Wanda-sp at lower and moderate sparsity levels, but its advantage becomes less consistent as pruning becomes more severe. The increasing perplexity with higher pruning ratios indicates that this one-shot approach works best within a limited sparsity range. One-shot global pruning outperforms the compared Wanda-sp baseline at lower pruning ratios on both datasets. Perplexity rises rapidly as the pruning ratio increases, showing reduced stability under aggressive pruning.

GISP has a longer total pruning time than the compared local pruning methods because it performs iterative pruning. However, because a single GISP run produces many usable sparsity levels, its amortized time per deployable subnetwork is lower than the alternatives. The intermediate models from the pruning trajectory are presented as effective subnetworks, reducing the need for separate pruning runs at each sparsity level. GISP produces substantially more usable sparsity levels than the compared pruning methods. Despite a longer total runtime, GISP achieves a lower amortized pruning cost per subnetwork. A single GISP pruning run can yield useful subnetworks across multiple sparsity levels.

The comparison shows that GISP generally preserves text-generation quality better than the baselines, yielding lower WikiText2 perplexity across tested models and pruning ratios. It also improves commonsense reasoning accuracy on CMQA, with the advantage becoming more pronounced at higher sparsity levels. These trends suggest that task-specific calibration helps maintain both language-modeling ability and downstream reasoning performance after pruning. GISP tends to have lower perplexity than the compared pruning methods on Llama2 models. GISP achieves higher CMQA reasoning accuracy across model sizes and pruning ratios. The advantage is especially notable at higher pruning levels.

Across Llama3-8B and Mistral-0.3, the proposed pruning method gives the best downstream accuracy among the compared methods at the shown sparsity while remaining competitive on WikiText2 perplexity. The reported results say its advantages are strongest at higher pruning levels, indicating better preservation of task-relevant capability as models are thinned. At the shown 20 percent pruning ratio, the method achieves higher accuracy than Wanda-sp, OWL, ShortGPT, and LLM-Pruner on Llama3-8B, and higher accuracy than the available baselines on Mistral-0.3. For perplexity, it is better than Wanda-sp, OWL, and ShortGPT, although LLM-Pruner is slightly lower on Llama3-8B, showing that accuracy is the main consistent advantage.

The experiments evaluate pruning behavior across settings such as local pruning, one-shot global pruning, perplexity, and reasoning tasks, showing that performance generally degrades as sparsity increases while calibration effects vary by method. The proposed GISP method is shown to produce multiple usable subnetworks from a single pruning run, improving amortized efficiency and better preserving language modeling quality and reasoning performance than the compared baselines. Overall, the results indicate that the method is most advantageous at higher pruning levels, with stronger gains in downstream task accuracy than in perplexity alone.