InfinityInstruct is a large-scale, high-quality, open-source instruction fine-tuning dataset project launched by Beijing Academy of Artificial Intelligence (BAAI). The goal of the project is to develop a dataset containing millions of instructions to support instruction tracing capabilities of large language models, thereby improving model performance. This version is the InfinityInstruct-3M instruction dataset, and the final version is expected to be released at the end of June.

Features of InfinityInstruct include:

1. Large-scale datasets：The project plans to release tens of millions of command data, and 3 million Chinese and English command data have been released in the first phase.
2. High quality screening：The Zhiyuan Research Institute conducts field analysis and quality screening on existing open source data to ensure the high value of the data, and augments the data in areas where it is lacking.
3. Open Source Community Contributions: During the dataset construction process, the open source community provided a large amount of instruction data, including datasets from multiple sources, such as OpenHermes-2.5, UltraInteract_sft, CodeBagel, etc.
4. Risk Assessment and Data Generation: The project team is currently conducting risk assessment and data generation and expects to release the final version containing 10 million instructions by the end of June.
5. Performance Improvements: The current open source 3 million instruction data set has demonstrated SFT (Supervised Fine-Tuning) data capabilities that surpass existing data sets such as Mistral and Openhermes.
6. Future Outlook: It is expected that after the data volume increases to tens of millions, the dialogue model trained based on the instruction fine-tuning dataset will be able to reach the level of GPT-4. The development and release of the InfinityInstruct dataset is of great significance for promoting the research and application of large language models. It provides rich instruction data for large models, which helps improve the model's ability to understand and execute instructions. At the same time, its open source nature also promotes collaboration and knowledge sharing in the AI community.

Citation

@misc{li2025infinityinstructscalinginstruction,
title={Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models},
author={Jijie Li and Li Du and Hanyu Zhao and Bo-wen Zhang and Liangdong Wang and Boyan Gao and Guang Liu and Yonghua Lin},
year={2025},
eprint={2506.11116},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.11116},
}
@misc{du2025infinityinstructsubject,
title={Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report},
author={Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan},
year={2025},
eprint={2507.06968},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2507.06968},
}
@article{zhao2024iidoptimizinginstructionlearning,
title={Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency},
author={Hanyu Zhao and Li Du and Yiming Ju and Chengwei Wu and Tengfei Pan},
year={2024},
eprint={2409.07045},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.07045},
}
@misc{zhang2024inifinitymath,
title={InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic Mathematical Reasoning},
author={Bo-Wen Zhang and Yan Yan and Lin Li and Guang Liu},
year={2024},
eprint={2408.07089},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2408.07089},
}