One-sentence Summary

Apple and Carnegie Mellon University introduce TIC-CLIP, the first webscale Time-Continual benchmarks (TIC-DataComp with over $12.7\text{B}$12.7B timestamped image-text pairs, TIC-YFCC, and TIC-RedCaps) to expose temporal robustness gaps, showing that CLIP loses approximately $8\%$8% zero-shot accuracy on 2021–2022 retrieval tasks, and propose a rehearsal-based continual training approach that reduces compute by $2.5\times$2.5× compared to retraining from scratch.

Key Contributions

• Introduces the first set of webscale Time-Continual (TiC) benchmarks for vision-language models, including TIC-DataComp, TIC-YFCC, and TIC-RedCaps, with TIC-DataComp containing over 12.7B timestamped image-text pairs spanning 2014 to 2022.

• Curates dynamic evaluation tasks from these benchmarks to measure temporal robustness, showing that OpenAI's CLIP model trained on data up to 2020 loses roughly 8% zero-shot accuracy on retrieval tasks from 2021 to 2022 compared with more recently trained models.

• Demonstrates that a rehearsal-based continual training approach, which continues from the last checkpoint and replays old data, achieves performance competitive with retraining from scratch while reducing compute by roughly 2.5 to 2.7 times across dataset scales from 11M samples to 3B.

Introduction

Large multimodal foundation models like CLIP are trained once on static web-scale datasets, yet the real world continually generates new data, causing their performance to degrade over time. Prior work in continual learning has largely relied on synthetic benchmarks with artificial task boundaries that lack natural evolution, while CLIP-specific continual learning methods focus on downstream task adaptation rather than adapting to the evolving data distribution itself. Retraining these models from scratch is computationally prohibitive, such as the ViT-g-14 model requiring 240K A100 GPU hours. To address this, the authors introduce TIC-DataComp, a benchmark for time-continual training of CLIP models by appending crawl time information to existing datasets like CommonPool, YFCC, and RedCaps. Their key contribution is a cumulative method that warm starts from the latest checkpoint and replays all old data, which achieves performance competitive to an oracle that retrains from scratch while being 2.7 times more computationally efficient. This highlights the critical need for models to evolve alongside dynamic data distributions and exposes the limitations of static benchmarks like ImageNet for evaluating temporal robustness.

Dataset

The authors introduce TIC-CLIP, a benchmark for continual training of vision-language models using contrastive language image pretraining. Unlike static datasets such as WiT or DataComp, TIC-CLIP streams image-text pairs sequentially over time. The authors construct three time-continual datasets and define dynamic evaluation tasks to measure how models perform as data evolves.

Dataset composition and sources

• TIC-DataComp: the largest public image-text dataset, containing 12.7 billion image-text pairs sourced from Common Crawl. The authors augment each pair with crawl-time metadata by retaining the earliest snapshot in which it appears, giving monthly timestamps spanning 2014 to 2022. Experiments consolidate data to yearly granularity.
• TIC-YFCC: based on the YFCC15M subset selected by OpenAI, containing images with natural text captions from Flickr. It covers years 2008 to 2014 and uses original upload timestamps for yearly splits.
• TIC-RedCaps: 12 million image-caption pairs from a manually curated set of subreddits, covering 2011 to 2020. The authors use post creation timestamps to create yearly splits.

Filtering and scale details

• For TIC-DataComp, the authors experiment with medium, large, and xlarge sizes from the CommonPool subset. They use Basic filtering for all scales and Bestpool filtering, which involves an off-the-shelf CLIP model, only at xlarge scale. The paper notes that Bestpool filtering can bias selected data toward older time steps.
• TIC-YFCC retains only images whose captions contain natural text, following the original OpenAI filtering.
• TIC-RedCaps uses the original subreddit selection with no additional filtering described.

Evaluation tasks

The authors create both dynamic and static evaluation tasks.

• Dynamic retrieval: samples image-text pairs across different timestamps and measures retrieval performance given an image or text query. This is built for all three datasets.
• Dynamic classification: TIC-DataComp-Net, constructed from ImageNet classes in CommonPool. Examples are kept when the caption contains exactly one ImageNet synset and when the similarity between the synset definition and the caption exceeds 0.5 using a sentence embedding model. Unlike LAIONNet, the authors do not filter by CLIP similarity scores to avoid bias.
• Static tasks: 28 standard tasks, including ImageNet and its distribution shifts, VTAB, and Flickr30k retrieval, evaluated zero-shot.

Metrics and usage

• For each time step, a model is trained and evaluated on data from all time steps, producing a performance matrix. The authors report in-domain performance, backward transfer, and forward transfer, computed from the diagonal, lower triangle, and upper triangle of the matrix. Revised metrics are presented in the appendix to mitigate bias in backward transfer.
• Extracted evaluation pairs are removed from training sets, and all dynamic evaluations are zero-shot. The benchmark aims to maximize static task performance while optimizing dynamic task performance across distribution shift and forgetting.

Method

The authors introduce TIC-CLIP, a benchmark and experimental protocol designed for the continual training of vision-language foundation models, specifically focusing on the Contrastive Language-Image Pretraining (CLIP) objective. Unlike conventional approaches that utilize static datasets, this framework operates in a dynamic environment where image-text data arrives sequentially over time. The pipeline encompasses data aggregation, continual training under strict resource constraints, and a dual evaluation strategy involving both standard and dynamic tasks.

Refer to the framework diagram below for the overall workflow.

Data Aggregation and Streaming Protocol The foundation of this framework relies on time-continual datasets: TIC-DataComp, TIC-YFCC, and TIC-RedCaps. These datasets are constructed by augmenting existing large-scale image-text corpora with auxiliary time information, such as crawl timestamps or post creation dates. The data is revealed to the learner in a streaming fashion, typically aggregated at the granularity of years. For instance, TIC-DataComp spans from 2014 to 2022, while TIC-YFCC and TIC-RedCaps cover shorter, specific ranges.

To manage the influx of data, the system employs a streaming protocol where new data batches are combined with a replay buffer containing historical data. This aggregation step is critical for mitigating catastrophic forgetting. However, the framework imposes a memory budget that restricts the retention of old data due to potential data expiration policies, while allowing the retention of the last model checkpoint.

Continual Training Strategies The core of the methodology involves training a deployable model at each time step $t$t under a fixed compute budget, quantified in Multiply-Accumulate Operations (MACs). The authors evaluate several training strategies to balance the trade-off between plasticity (learning new data) and stability (retaining old knowledge):

1. Oracle: This serves as an upper-bound baseline where a model is trained from scratch on all data received up to time $t$t using a compute budget of $t \times C$t×C. While prohibitively expensive, it represents the ideal performance achievable with unlimited resources.
2. Cumulative: This approach trains the model on the union of all data up to time $t$t (new data plus replayed old data) using the fixed compute budget $C$C. It is initialized from the checkpoint of the previous time step ($t-1$t−1). To manage the replay buffer, the authors propose several sampling strategies:
   • -All: Replays all previous data.
   • -Exp: Uses a buffer of size $D$D and reduces the amount of old data by half at each step (e.g., retaining $D/2$D/2 of older data and $D/2$D/2 of slightly older data).
   • -Equal: Splits the buffer size $D$D equally among all previous years.
3. Sequential: The model is trained only on the new data arriving at time $t$t, initialized from the best checkpoint of the previous step. This method tests the model's ability to adapt without explicit replay.
4. Restart: This method trains from scratch (random initialization) on all data up to time $t$t but is constrained to the fixed compute budget $C$C. It helps isolate the effects of forward transfer and loss of plasticity.
5. Patching: Leveraging sequential patching techniques, this method initializes training from a patched model of the previous step. It applies weight interpolation between the patched model from time $t-1$t−1 and the current model, optimizing mixing coefficients based on retrieval performance on previous tasks.
6. Learning with Forgetting (LwF): This approach trains only on new data but introduces a KL divergence penalty. This penalty constrains the current model's image-text similarity matrix to remain close to that of the previous checkpoint, preserving knowledge of past distributions.

The training process utilizes a cyclic learning rate schedule. Specifically, it employs a cosine decay schedule for each sequential run, resulting in a cyclic pattern. This was found to be effective as the increased training loss observed early in subsequent runs decreases rapidly compared to training from scratch.

Evaluation Protocol The framework distinguishes between standard static evaluation and proposed dynamic evaluation tasks. Static evaluation assesses performance on established benchmarks like ImageNet, ObjectNet, and Flickr30k, which do not change over time. In contrast, dynamic evaluation tasks are designed to measure the model's robustness to time-evolving data distributions. These tasks include backward transfer (performance on past tasks), ID evaluation (in-distribution data at current time $t$t), and forward transfer (performance on future tasks at time $t+1$t+1). This dual evaluation ensures a comprehensive assessment of the model's adaptability and stability in a non-stationary environment.

Experiment

The authors constructed time-continual benchmarks (TIC-DataComp, TIC-YFCC, TIC-RedCaps) with temporal metadata and introduced dynamic retrieval and classification tasks alongside 28 static tasks to evaluate continually trained CLIP models under a streaming protocol. Training sequentially with replay of all past data (Cumulative-All) closely matched the Oracle that retrains from scratch on dynamic tasks while saving up to 4x compute, though gaps on static tasks appeared at larger scales. Methods with limited replay buffers stayed competitive with full replay, while no-replay sequential training showed strong forward transfer but significant losses on static and backward transfer tasks. Dynamic evaluation tasks revealed distribution shifts that static benchmarks miss, such as emerging concepts and evolving categories, and provided complementary signal for model selection.

This table outlines continual training methods for CLIP models, varying in data replay size, initialization from previous checkpoints, and compute budgets. Methods like Cumulative use replay data and start from the last checkpoint, while Restart re-trains from scratch on all data. The methods are evaluated against an Oracle baseline that uses the full compute budget. Cumulative methods replay data from prior steps and initialize from the last checkpoint, with compute budget C per step. Oracle trains from scratch on all accumulated data with a large compute budget t×C, serving as an upper-bound reference. Restart uses all data up to time t but starts from random initialization, while Sequential trains only on current data from the last checkpoint. Patching initializes from a patched combination of previous checkpoints and trains on current data with budget C.

Replaying all past data with Cumulative-All achieves near-Oracle performance on dynamic tasks while using significantly less compute, though gaps appear on larger static benchmarks. Sequential training without replay shows strong forward transfer at scale but underperforms on static and backward transfer tasks. Additional strategies like patching and LwF improve over Sequential but still lag behind Cumulative-All. Cumulative-All stays within 1% of Oracle on dynamic evaluations while cutting compute by up to 4 times. Sequential training excels at forward transfer on large-scale data but drops sharply on static and backward transfer tasks. Patching and LwF boost backward transfer over Sequential, yet Cumulative-All outperforms them on all metrics without extra compute.

On ImageNet continual training, Cumulative-All maintains accuracy close to the Oracle even as the number of training splits increases. The performance gap remains below 0.9% even with up to 8 splits, indicating that replaying all past data is a feasible and efficient strategy. Cumulative-All stays within 0.9% of Oracle accuracy across up to 8 splits. Accuracy declines only slightly with more splits, from 80.9 at 1 split to 80.0 at 8 splits. Results support replaying all past data as a computationally efficient alternative to retraining from scratch.

The experiments compare continual training methods for CLIP, including cumulative replay, sequential training, restart, and patching, against an Oracle baseline that retrains on all accumulated data with full compute. Replaying all past data consistently approaches Oracle performance on dynamic tasks while using significantly less compute, though it shows some gaps on larger static benchmarks, and it outperforms alternatives like sequential training, which exhibits strong forward transfer but weak backward transfer, and patching or LwF, which improve backward transfer but still fall short. On ImageNet across multiple training splits, cumulative replay maintains accuracy close to the Oracle with only slight degradation as the number of splits increases, confirming it as an efficient and feasible alternative to full retraining.