Depth Up-Scaling (DUS)

Depth Up-Scaling (DUS) is a term used in the paper “SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling ” proposed in the journal, which includes deep expansion and continuous pre-training. Compared with other LLM upgrade methods using expert mixtures, DUS does not require complex changes for effective training and reasoning. The research team demonstrated through experiments that DUS is simple and effective, and can expand small LLMs to high-performance LLMs.