The Sports-1M dataset contains more than 1 million videos, which are obtained through YouTube URLs specified by the authors.

Since the dataset was compiled, approximately 7% (as of 2016) of the videos have been removed by YouTube uploaders. However, the dataset still contains over a million videos, covering 487 sports-related categories, with 1,000 to 3,000 videos per category. By analyzing the textual metadata associated with the videos (e.g., tags, descriptions), the authors used the YouTube Topic API to automatically label the videos into the 487 sports categories. Approximately 5% of the videos were labeled with more than one category.