Command Palette
Search for a command to run...
작은 LLMs: Pruning 대 Scratch부터의 Training
작은 LLMs: Pruning 대 Scratch부터의 Training
Yufeng Xu Taiming Lu Kunjun Li Jiachen Zhu Mingjie Sun Zhuang Liu
초록
가지치기(Pruning)는 강력하고 경량화된 소규모 언어 모델(Small Language Models)을 구축하기 위한 지름길로서 가능성을 보여준다. 본 연구에서는 두 가지 제어된 토큰 일치(Token-matched) 설정 하에서 Llama-3.1-8B 모델에 대해 6가지 방법을 적용하여 0.5~0.8의 가지치기 비율을 적용하며 이러한 가능성을 검증하였다. 이 6가지 방법은 깊이(Depth), 폭(Width), 그리고 희소성(Sparse) 등 다양한 세분화 수준(Granularities)을 포괄한다.첫째, 동일한 학습 토큰 예산(Training Token Budget) 내에서 가지치기된 초기화(Pruned Initialization)는 무작위 초기화(Random Initialization)를 지속적으로 상회하는 성능을 보였다. 이는 부모 모델이 강력한 시작점을 제공한다는 사실을 시사하나, 학습 토큰 예산이 증가하거나 가지치기 비율이 높아질수록 그 우위차는 점차 축소되며, 본 연구에서 검토한 최고 가지치기 비율에서는 거의 소멸하는 경향을 나타냈다.둘째, 전 Pipeline이 소비하는 전체 토큰 예산을 무작정 학습(Training from Scratch)에 사용하는 상황에서는 더욱 미세한 세분화 수준(Fine Granularity)에서의 가지치기가 여전히 우위를 유지하는 반면, 조 coarse 한 구조적 가지치기(Coarser Structured Pruning)는 이를 따라잡거나 오히려 앞설 수 있다. 이는 부모 모델이 전수하는 지식은 추가적인 학습 토큰만으로는 완전히 복원되지 않으며, 오직 미세한 세분화 수준에서만 이러한 이점이 발생함을 나타낸다.종합하면, 본 연구 결과는 명확한 제언을 도출한다. 대규모 사전 학습 모델이 존재하고 학습 토큰 예산이 제한적인 상황에서는 가지치기가 무작정 학습보다 유리하다. 반면, 학습 예산이 제한되지 않은 상황에서는 조 coarse 한 가지치기의 경우 무작정 학습이 경쟁력이 있을 수 있으므로, 대규모 사전 학습된 부모 모델이 항상 필수적인 것은 아니다.본 연구의 코드는 github.com/zlab-princeton/pruning-vs-scratch에서 확인 가능하다.
One-sentence Summary
Comparing six pruning methods spanning depth, width, and sparse granularities on Llama-3.1-8B at pruning ratios of 0.5–0.8 under two token-matched budgets, Princeton researchers reveal that pruned initialization significantly outperforms random initialization when training tokens are limited, though this advantage narrows as the budget grows and the pruning ratio rises, nearly vanishing at the highest ratio, and that when training from scratch receives the full pipeline budget, finer granularities still benefit while coarser structured pruning is matched or surpassed, offering a clear guide that a large parent model is necessary only when training tokens are limited.
Key Contributions
- Six pruning methods spanning depth, width, and sparse granularities are compared on Llama-3.1-8B at pruning ratios 0.5–0.8 under two token-matched retraining conditions.
- Given the same retraining token budget, pruned models consistently outperform randomly initialized models, though the advantage shrinks as the token budget grows and as the pruning ratio increases, nearly vanishing at ratio 0.8.
- When scratch training receives the full pipeline token budget, fine-grained sparse pruning still provides a benefit while coarser structured pruning can be matched or exceeded. Therefore, pruning is recommended when training tokens are limited, but training from scratch is competitive for coarse-grained pruning when tokens are abundant.
Introduction
As large language models scale, pruning becomes critical for reducing inference costs, but prior work rarely examines how pruning granularity and retraining data interact under token-limited budgets. The authors address this gap by systematically evaluating six pruning methods across depth, width, and sparse granularities within a continual pretraining framework, revealing nuanced trade-offs between compression rate, accuracy retention, and data efficiency.
Experiment
This work frames pruning as an initialization strategy and compares it against training from scratch under two token-matched settings, using Llama-3.1-8B as the parent model and six pruning methods spanning structured and sparse granularities. Under an equal training token budget, pruned initialization consistently beats random initialization, though this advantage shrinks as the pruning ratio increases and nearly vanishes at the most aggressive compression. When scratch training is granted the full token budget of the entire pruning pipeline, it catches up to coarse structured pruning, but fine-grained sparse pruning retains its lead, showing that parent knowledge can transfer benefits that additional training tokens alone cannot recover.
The authors evaluate Minitron depth pruning initialization against training from scratch across varying retraining token budgets. Results demonstrate that pruning initialization consistently yields lower loss and perplexity along with higher downstream accuracy compared to random initialization. However, this performance advantage diminishes as the retraining token budget increases. Pruning initialization consistently outperforms random initialization across all evaluated token budgets. The performance gap between pruned and scratch-trained models narrows as the retraining token budget grows. Initializing from a released model checkpoint yields performance comparable to or slightly better than using a custom pretrained checkpoint.
The authors compare pruning initialization against training from scratch under equal training token budgets and equal total token budgets. Results show that pruning initialization consistently outperforms random initialization when the training token budget is equal. However, when the scratch model is trained with the full token budget of the pruning pipeline, it becomes competitive with structured pruning methods, while finer sparse pruning methods retain their advantage. Under an equal training token budget, the pruned initialization consistently achieves lower perplexity and higher average accuracy than training from scratch across all evaluated methods. When given the full token budget, training from scratch becomes competitive with or surpasses coarser structured pruning methods, effectively closing the performance gap. Finer granularity sparse pruning maintains an accuracy advantage over training from scratch even when the total token budget is matched, unlike coarser structured methods.
The authors compare pruning initialization against training from scratch under equal training token and equal total token budgets across various pruning methods and granularities. Results show that pruning initialization consistently outperforms random initialization when training token budgets are equal, though this advantage diminishes for coarser granularities when the total token budget is matched. Finer-grained sparse pruning methods retain a performance edge over training from scratch even when the scratch model is trained on the full token budget of the pruning pipeline. Pruning initialization consistently yields lower perplexity and higher downstream accuracy than training from scratch under an equal training token budget across all evaluated methods. When given the full token budget of the pruning pipeline, training from scratch matches or surpasses coarser structured pruning methods but struggles to catch up to finer sparse pruning methods. Finer pruning granularities provide a stronger initialization advantage and better preserve the parent model capabilities compared to coarser structured approaches.
The authors evaluate the FLAP pruning method as an initialization strategy compared to training from scratch across various retraining token budgets. Results indicate that initializing with a pruned model consistently outperforms random initialization in terms of perplexity and downstream accuracy. However, the performance advantage of pruning diminishes as the retraining token budget increases, showing that additional training data helps close the gap between the two approaches. Pruned initialization consistently achieves lower perplexity and higher average accuracy than training from scratch across all evaluated token budgets. The performance advantage of pruning over scratch training diminishes as the number of retraining tokens increases. Initializing from a publicly released large model checkpoint yields slightly better results than the authors' custom pretraining pipeline.
The authors compare pruning large language models against training from scratch, utilizing an architecture search to define target model configurations at a standard pruning ratio. Results indicate that pruning initialization consistently outperforms random initialization under equal training budgets, acting as a token-efficient shortcut. However, this advantage diminishes with higher compression, and while coarse structured pruning can be matched by extended training, fine-grained sparse pruning retains its edge. Pruning initialization provides a strong starting point that beats random initialization under equal training token budgets. The advantage of pruning shrinks as the pruning ratio increases and the target model becomes smaller. Fine-grained sparse pruning outperforms training from scratch even when the scratch model is trained on the full token budget of the pruning pipeline.
Across multiple experimental setups comparing pruning initialization to training from scratch, pruning consistently yields lower perplexity and higher accuracy under equal retraining token budgets, though its advantage diminishes as the retraining token count increases or the pruning granularity becomes coarser. When training from scratch is allowed the full token budget of the pruning pipeline, it becomes competitive with or surpasses coarser structured methods, while finer sparse pruning approaches retain a clear performance edge. Utilizing a publicly released model checkpoint for initialization performs comparably to or slightly better than a custom pretrained checkpoint, and the overall findings confirm that pruning transfers valuable parent model knowledge, with the benefit most pronounced for fine-grained sparsity and limited retraining data.