GF-Minecraft Game Video Dataset

The GF-Minecraft dataset is a high-quality action-annotated video dataset jointly created by the University of Hong Kong and Kuaishou Technology in 2025. It is mainly used for the research of generative game engines. The related paper results are "GameFactory: Creating New Games with Generative Interactive Videos".

The dataset uses Minecraft as a data collection platform due to its comprehensive API, diverse open world environment, and extensive action space. 70 hours of game videos were collected and action annotations were performed by executing predefined random action sequences. To enhance diversity, 3 biomes (forest, plains, desert), 3 weather conditions (sunny, rainy, thunderstorm), and 6 time periods (e.g. sunrise, noon, midnight) were preconfigured, generating more than 2k video clips. Each clip contains 2k frames and is accompanied by a text description generated by the multimodal language model MiniCPM-V.

This dataset is designed for motion-controlled video generation and meets the following three key requirements:

1. Customizable actions: Facilitates large-scale, low-cost data collection.
2. Unbiased action sequence: Ensure the diversity of action combinations and coverage of low-probability events.
3. Diverse scenarios: Contains text descriptions to capture the physical dynamics of a particular scene.