SA-V: Meta Builds the Largest Video Segmentation Dataset

* This dataset supports online use.Click here to jump.

The SA-V dataset is a large-scale video segmentation dataset built by Meta in 2024. It is used to train and evaluate Meta Segment Anything Model 2 (SAM 2). This dataset is very large in scale and diversity, containing about 51,000 real-world videos and 643K spatiotemporal masklet annotations, which is about 50 times larger than other similar datasets.

The SA-V dataset was constructed using an iterative process in which human annotators interactively annotated masklets in videos using the SAM 2 model, and then this newly annotated data was used to update and train the SAM 2 model. This approach not only improved the efficiency of data collection, but also helped build a more accurate and diverse dataset.

In addition, the videos in the SA-V dataset come from 47 different countries, covering diverse geographies and real-world scenes, which provides rich visual content for the model to learn and generalize. The annotations in the dataset include not only the entire object, but also parts of the object, such as a person’s hat, as well as challenging instances when the object is occluded, disappears, and reappears.

The release of this dataset, coupled with the open-sourcing of the SAM 2 model, provides researchers and developers with powerful tools to explore new applications and innovations in areas such as video editing, mixed reality, robotics, autonomous driving, and video content understanding.

Dataset structure

– Training segmentation: Videos are encoded in MP4, each file is about 8G, sav_000.tar – sav_055.tar . Masklet uses COCO run-length encoding (RLE) format (list of lists), where the outer list is located above the video frame.

– Val/Test segmentation: Video frames are in JPEG format, each file is about 16G, sav_val.tar and sav_test.tar. Masklets are in PNG format.