One-sentence Summary

Researchers at The Chinese University of Hong Kong propose A-RSI, an adaptive task refresh method for zero-data multimodal self-improvement that monitors reward variation and accuracy in training logs to refresh a task pool when the learning signal declines, uses model-authored programs to generate independently checkable multimodal tasks, and raises the eleven-benchmark average from 57.77 to 61.05 with 800 fresh tasks.

Key Contributions

• A-RSI monitors reward variation and accuracy in training logs to refresh a task pool when its learning signal declines, without consulting validation or test scores.
• Tasks are generated from a static library of model-authored programs with program-computed reference answers, so answer correctness does not depend on model voting; the task-generation contract, reward and refresh properties, and generated answers are verified or audited, with no external training examples or human-written answer labels.
• On the eleven-benchmark suite, A-RSI raises the Macro11 average from 57.77 to 61.05 with 800 fresh tasks; matched comparisons show difficulty-based, controller-based, and rollout-based selectors do not outperform random replacement, and length-budget audits indicate some tight-budget gains reflect learning to finish.

Introduction

Recursive self-improvement systems need both useful experience and a way to recognize when that experience stops helping. Recent work has emphasized what to generate next, such as harder tasks or tasks near the model’s competence, but it often overlooks a separate decision: when to replace the current training pool. Keeping a stale pool wastes updates, while refreshing too often consumes new tasks unnecessarily. Self-generated targets from methods like majority voting can also turn model errors into training labels. The authors address this by introducing A-RSI, which uses training-log signals such as the decline in prompts with within-group reward variation to decide when to draw fresh tasks. New tasks come from model-written programs with independently computed answers, separating the refresh decision from answer correctness and avoiding reliance on validation or test scores.

Dataset

Dataset construction and sources

• The authors build a programmatic task library $T$T from seed task types in geometry, charts, tables, and counting.
• A base model generates reusable problem structures and parameterized programs. Each template specifies admissible parameters, the target quantity, sampling procedures, image drawing, question composition, and answer computation.
• During generation, a template $\tau$τ and latent parameters $z$z produce an image, question, and reference answer:

$$(I,q,a)=G_{\tau}(z)$$(I,q,a)=Gτ​(z)

• The learner receives only $(I,q)$(I,q). The reference answer $a$a is used for scoring.

Dataset composition and scale

• Generator groups cover geometry, charts, tables, and counting.
• The audited consumed replacement pools sample 200 tasks without replacement from each of five supported generator groups, totaling 1,000 audited tasks. These five groups cover geometry, charts, tables, and counting.
• Exact per-generator allocations and reservoir sizes are reported in Appendix Q.2.
• The initial replacement pool comes from earlier geometry and chart batches with model-assisted question rewriting and checks.
• In one comparison, the active pool changes from 500 to 400 tasks during replacement.

Processing and filtering

• Candidates are admitted to the eligible set $\mathcal{C}$C only if they pass generator-specific construction checks $V_{\tau}$Vτ​, freshness checks $F$F, contamination screening $C$C, and are not in the diagnostic holdout $H$H:

$$x\in\mathcal{C}\implies V_{\tau}(x)=1,\quad F(x)=1,\quad C(x)=1,\quad x\notin H$$x∈C⟹Vτ​(x)=1,F(x)=1,C(x)=1,x∈/H

• Generator-specific checks test the sampled configuration, answer computation, rendering constraints, and ambiguity constraints.
• Freshness and contamination screens reject flagged candidates.
• Held-out diagnostic items are excluded from training.
• Retained records associate each image, question, and answer with a generator identity and seed for reconstruction.
• Image hashes are recorded and checked.
• Audited image files must decode, meet minimum-dimension checks, and be nonblank.
• The generation contract requires one shared underlying configuration:

$$I=R_{\tau}(s_{\tau}(z)),\quad q=Q_{\tau}(s_{\tau}(z)),\quad a=E_{\tau}(s_{\tau}(z))$$I=Rτ​(sτ​(z)),q=Qτ​(sτ​(z)),a=Eτ​(sτ​(z))

• Independent solvers agree with all 1,000 sampled reference answers. Training gold/solution values agree with selected source records, and image-file hashes agree with recorded hashes on all 1,000 audited tasks.
• Replaying frozen generators reproduces 800 whole images byte for byte. For the remaining 200 chart tasks, plot pixels and category/numeric-axis labels agree, but whole-image reproduction is unresolved because of font differences.

How the data is used

• The eligible set $\mathcal{C}$C supplies replacement pools for continuation training.
• Training uses the generated image-question pairs $(I,q)$(I,q) as input, and responses are scored against the program-computed reference answer $a$a.
• Program-computed supervision remains subject to the correctness of the template and its answer executor.
• The authors compare fresh-pool training with original-pool continuation. At matched progress, fresh-pool training scores 60.40 against 58.35 on the Macro11 format-corrected benchmark, while the active pool changes from 500 to 400 tasks.
• The dataset construction and validity checks focus on verifiable data properties, but the paper notes that these checks do not certify every visual or linguistic property of the generated tasks.

Method

The authors propose A-RSI, a framework that supports continued multimodal reinforcement learning through a programmatic task pipeline and a training-log controller. The overall workflow is organized into three interconnected phases: pre-training preparation, active training, and dynamic pool management.

Before training begins, the system constructs a fixed library of executable task templates. For each template, the model specifies admissible parameters, conditions, and procedures for sampling values, drawing images, composing questions, and computing answers. The pipeline then executes these frozen templates to generate candidate instances. Each candidate undergoes rigorous admission checks, including generator-specific validation, freshness screening, and contamination checks, to ensure data quality. Retained records are organized into disjoint replacement pools under fixed family quotas, forming a reservoir of eligible tasks that remains fixed throughout the training process.

During the training phase, only the Solver policy is optimized. For each sampled task from the active pool, the model generates multiple responses. The authors employ Group Relative Policy Optimization (GRPO) to update the policy based on a composite reward function:

$$r(y, a) = r_{\mathrm{ans}}(y, a) + \lambda_{\mathrm{fmt}} r_{\mathrm{fmt}}(y) + \lambda_{\mathrm{stop}} r_{\mathrm{stop}}(y)$$r(y,a)=rans​(y,a)+λfmt​rfmt​(y)+λstop​rstop​(y)

where $r_{\mathrm{ans}}$rans​ evaluates the correctness of the extracted answer against the program-computed reference, $r_{\mathrm{fmt}}$rfmt​ enforces a required boxed-answer format, and $r_{\mathrm{stop}}$rstop​ penalizes abnormal decoding termination. The coefficients $\lambda_{\mathrm{fmt}}$λfmt​ and $\lambda_{\mathrm{stop}}$λstop​ remain fixed throughout training.

To prevent the model from overfitting to a static dataset, the framework incorporates a training-log controller that monitors pool saturation. The controller tracks the number of distinct prompts yielding nonzero relative advantages, denoted as $U_w$Uw​, alongside the response accuracy $a_w$aw​. A response group with identical total rewards provides no policy-gradient signal, indicating that the model has exhausted the learning potential of the current prompts. At the end of each accepted training window, the controller compares these signals against their values from $\ell$ℓ windows earlier. The decision to refresh the pool, $\delta_w$δw​, is triggered if the minimum observation period is met and either the nonzero advantage count drops below a retention threshold or the accuracy increases sufficiently:

$$\delta_{w} = \begin{cases} 1, & n_{w} \geq n_{\min} \text{ and } \big(U_{w} \leq \rho U_{w - \ell} \text{ or } a_{w} - a_{w - \ell} \geq \eta\big), \\ 0, & \text{otherwise}. \end{cases}$$δw​={1,0,​nw​≥nmin​ and (Uw​≤ρUw−ℓ​ or aw​−aw−ℓ​≥η),otherwise.​

When $\delta_w = 1$δw​=1, the system replaces the entire active pool with $B$B unused tasks drawn without replacement from the pre-built reservoir, adhering to fixed family quotas. Crucially, this replacement does not reset the model parameters; the Solver continues training from the latest accepted weights, ensuring that the learning experience accumulates across windows while the data distribution remains fresh and challenging.

Experiment

The experiments evaluate a zero-data RL training loop for multimodal models by comparing refresh strategies on Qwen and MiMo across eleven benchmarks. They validate that refreshing the training pool when a saturation detector fires and then sampling random unused tasks sustains learning with fewer new tasks than fixed periodic or continuous refresh, while keeping the old pool eventually stalls. Ablations show replacement-selection refinements do not improve over random sampling, and reward checks find that auxiliary rewards cannot reverse correctness rankings though a parser vulnerability exists. Qualitative case studies further confirm that later checkpoints can finish previously truncated responses and correct previously wrong answers.

Related self-improvement systems differ in how they source tasks, supervise training, and update data. Most compared methods change training data continuously or in step with model ability, whereas A-RSI keeps a fixed library of task-generating programs and replaces the whole pool only when its own training logs indicate saturation. A-RSI also differs from fixed-data selection methods by refilling with random programmatic tasks rather than selecting from a static collection. Most compared methods update training data continuously or co-evolve it with the model, while A-RSI uses a saturation signal from training logs to trigger a full pool refresh. Task supervision differs widely across systems, including code execution, model labels, majority vote, game rewards, and dataset labels. ThinkLite-VL selects a challenging subset from a fixed task collection one time, whereas A-RSI draws random programmatic tasks after saturation and does not adapt its task-generation rules during training. A-RSI's training gain is less sensitive to a doubled generation budget than MM-Zero v5's gain, suggesting the improvement is not mainly due to completing within the evaluation limit.

For the listed Qwen3-VL-8B-Instruct benchmarks, later training pools generally improve over the starting model, with VisNumBench and MathVision showing the largest gains. Improvements are uneven across tasks and not strictly monotonic, as several benchmarks dip between adjacent windows or peak before the final recorded window, while MathVerse changes only slightly. VisNumBench and MathVision show the largest overall improvements from the starting model to later recorded windows. MathVerse remains relatively flat, and MMSI shows only modest net movement. Progress is non-monotonic in places, with temporary dips in benchmarks such as VisNumBench and MMMU-Pro (10). All listed final scores exceed the starting model, but MathVerse and MMMU-Pro (4) peak earlier.

For the MiMo-VL-7B-SFT model evaluated at 8192 tokens, most benchmarks improve over the starting checkpoint through training, with the clearest gains on MathVerse and MMMU-Pro. MMSI is the exception and remains below its initial score across all displayed windows. The trajectory is not monotonic, with later checkpoints sometimes regressing relative to earlier ones in the same pool. MathVerse, MathVision, and both MMMU-Pro variants improve over the starting model in every displayed training window. MMSI is the only benchmark that stays below its starting score across all shown checkpoints. Within the last displayed pool, the first checkpoint performs best on most benchmarks, while the added continuation often regresses.

A-RSI, which combines training-log-triggered refresh with random fresh-task selection, reaches the highest final eleven-benchmark score while using the smallest new-task budget along with frontier refresh. No-refresh continuation plateaus and halts early, and alternative refresh policies either use more new tasks or score lower. The comparison supports the implemented fresh-pool policy in this run, though it does not isolate whether the trigger timing beats a fixed schedule. A-RSI achieves the highest final score and, together with frontier refresh, uses the smallest number of new tasks, but scores well above that frontier variant. No-refresh continuation plateaus and fails to produce a final checkpoint, while all refresh arms continue improving. Continuous refresh narrows the gap but requires 3,200 new tasks, four times the budget of A-RSI, and still trails at the final checkpoint. With frontier selection held fixed, detector-triggered refresh trails periodic refresh slightly, so the advantage of A-RSI is not shown to come from better timing alone.

The every-window random refresh control achieves the highest Macro11 score among replacement-selection variants at 8192 tokens. Failure-prioritized, learnable-band, and always-wrong sampling all score lower than this control, with the largest gaps for failure-prioritized and always-wrong. Ladder and always-increase controllers also underperform the control, though by smaller margins; these variants share control runs and training histories, so the comparisons are not independent replications. All listed replacement-selection variants score below the every-window random refresh control. Failure-prioritized and always-wrong sampling show larger drops than the ladder and always-increase controllers.

The experiments compare A-RSI with self-improvement baselines that vary task sourcing, supervision, and refresh policies across Qwen3-VL-8B-Instruct and MiMo-VL-7B-SFT benchmarks. Later training pools generally improve over starting checkpoints, but gains are uneven and non-monotonic, with some benchmarks peaking before the final window or regressing. A-RSI attains the highest final overall score while using the smallest new-task budget, whereas no-refresh continuation plateaus and alternative refresh policies require more tasks or score lower. Replacement-selection ablations show that simple random refresh outperforms failure-prioritized, learnable-band, and controller-based sampling variants.