**ChineseWebText is the latest and largest Chinese dataset, containing 1.42 TB of data.**Each text is assigned a quality score, making it easier for large language model researchers to select data based on new quality thresholds. A cleaner subset containing 600 GB of Chinese text with quality exceeding 90% is also released here. This directory contains the ChineseWebText dataset and the EvalWeb toolchain for processing CommonCrawl data.

Citation

@misc{chen2023chinesewebtext, title={ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model}, author={Jianghao Chen and Pu Jian and Tengxiao Xi and Dongyi Yi and Qianlong Du and Chenglin Ding and Guibo Zhu and Chengqing Zong and Jinqiao Wang and Jiajun Zhang}, year={2023}, eprint={2311.01149}, archivePrefix={arXiv}, primaryClass={cs.CL} }