SkyPile-150B Comprehensive Large-Scale Chinese Dataset

SkyPile-150B is a comprehensive large-scale Chinese dataset designed specifically for pre-training of large language models.It is sourced from a large number of publicly accessible Chinese Internet web pages. The dataset uses strict filtering, extensive deduplication, and thorough sensitive data filtering to ensure its quality. In addition, researchers also use advanced tools such as fastText and BERT to filter out low-quality data.

The public part of the SkyPile-150B dataset contains approximately 233 million web pages, each of which contains more than 1,000 Chinese characters on average. The dataset contains a total of approximately 150 billion tokens and 620 GB of plain text data.