**SkyPile-150B is a comprehensive large-scale Chinese dataset designed specifically for pre-training of large language models.**It is sourced from a large number of publicly accessible Chinese Internet web pages. The dataset uses strict filtering, extensive deduplication, and thorough sensitive data filtering to ensure its quality. In addition, researchers also use advanced tools such as fastText and BERT to filter out low-quality data. The public part of the SkyPile-150B dataset contains approximately 233 million web pages, each of which contains more than 1,000 Chinese characters on average. The dataset contains a total of approximately 150 billion tokens and 620 GB of plain text data.

Citation

@misc{wei2023skywork, title={Skywork: A More Open Bilingual Foundation Model}, author={Tianwen Wei and Liang Zhao and Lichang Zhang and Bo Zhu and Lijie Wang and Haihua Yang and Biye Li and Cheng Cheng and Weiwei Lü and Rui Hu and Chenxia Li and Liu Yang and Xilin Luo and Xuejie Wu and Lunan Liu and Wenjun Cheng and Peng Cheng and Jianhao Zhang and Xiaoyu Zhang and Lei Lin and Xiaokun Wang and Yutuan Ma and Chuanhai Dong and Yanqi Sun and Yifu Chen and Yongyi Peng and Xiaojuan Liang and Shuicheng Yan and Han Fang and Yahui Zhou} year={2023}, eprint={2310.19341}, archivePrefix={arXiv}, primaryClass={cs.CL} }