Densing Law

Densing Law is a new concept about Large Language Models (LLMs) proposed by Professor Liu Zhiyuan's team from the Natural Language Processing Laboratory of Tsinghua University in December 2024. The relevant paper results are "Densing Law of LLMs".

The Densing Law describes the exponential growth of the power density of large language models (LLMs) over time. Power density is defined as the ratio of the effective parameter size of a given LLM to the actual parameter size, where the effective parameter size refers to the number of parameters of the reference model required to achieve the same performance as the target model. This law reveals the performance and efficiency of LLMs at different scales and provides a new perspective to evaluate and optimize the development of LLMs.

By analyzing 29 widely used open source large models, the research team found that the maximum capacity density of LLM increases exponentially over time, doubling approximately every 3.3 months (about 100 days). According to the density law, the model reasoning overhead decreases exponentially over time. From January 2023 to date, the reasoning cost of the GPT-3.5 level model has been reduced by 266.7 times.

The density law emphasizes the importance of finding a balance between model performance and efficiency, especially in the face of the challenges of increasing computing resource requirements and environmental impact. In addition, this law also points out that existing model compression methods, such as pruning and distillation, usually cannot improve the density of compressed models, indicating that more effective model compression algorithms are needed to improve the density of small models.