One-sentence Summary

Researchers from the Chinese Academy of Sciences propose Ability-aware Environment Selection (AES) and Hierarchical Difficulty Curriculum (HDC), which organizes curriculum learning via harness weakening and state-scale progression, to construct training environment distributions with superior diversity and difficulty structure, demonstrating significant improvements in multimodal agent learning over mere environment scaling.

Key Contributions

• Empirical analysis reveals that simply increasing the number of multimodal environments can degrade agent performance, due to amplified gradient conflicts and multimodal-specific bottlenecks in visual state extraction and world modeling.
• Introduces Ability-aware Environment Selection (AES) to construct diverse environment sets.
• Proposes Hierarchical Difficulty Curriculum (HDC), which organizes difficulty through harness weakening and state-scale progression; experiments demonstrate that AES and HDC effectively improve multimodal agent training.

Introduction

Training multimodal agents increasingly relies on dynamic environments, but the authors find that simply scaling the number of environments does not reliably improve performance. Their experiments show that naive scaling causes instability: mixed multimodal training amplifies negative transfer and gradient conflicts compared to text-symbolic counterparts, and adding more environment types can even degrade results. Existing work concentrates on sample-level quality, ensuring each environment is executable, while ignoring distribution-level factors like diversity and difficulty structure. The authors tackle both dimensions: they propose Ability-aware Environment Selection (AES), which builds meta-ability profiles from agent trajectories to select a high-coverage, low-redundancy set of environments, and Hierarchical Difficulty Curriculum (HDC), which addresses multimodal bottlenecks through harness weakening and state-scale progression. Combined, AES and HDC produce large average performance gains, showing that designing effective environment distributions is critical for multimodal agent training.

Dataset

The authors construct a set of meta-ability profiles to represent the skill demands of 200 interactive environments. This dataset is built in three stages:

• Trajectory collection For each environment, 40 agent trajectories are recorded—20 from Qwen3‑VL‑4B and 20 from Gemini‑3‑Flash. The two models differ in capability, so the trajectory pool contains both successful strategies and failure patterns, revealing ability requirements from complementary perspectives. This yields 8,000 trajectories across all environments.

• Atomic ability segmentation A strong annotation model (GPT‑5) decomposes every trajectory into a sequence of atomic abilities—interpretable behavior units such as “identifying the target position.” A unified prompt with few‑shot examples controls the granularity, and manual checks filter out low‑quality segmentations (e.g., overly coarse labels like “perception”).

• Aggregation into meta‑ability profiles GPT‑5 merges semantically equivalent atomic abilities across all trajectories for a given environment. Frequencies of each meta‑ability and transition frequencies between them are computed, forming an ability graph. Low‑frequency and unstable abilities are removed. The result is a meta‑ability profile that lists the environment’s meta‑abilities, transition edges, and their frequencies. Each ability is further classified as core (stable, task‑critical) or soft (occasional or weakly related). In total, 200 profiles are built, covering 72 distinct core meta‑abilities.

The complete set of profiles is used to select a diverse subset of training environments. By maximizing coverage of the 72 core abilities while minimizing redundancy, the authors can assemble a varied and efficient training pool without manual inspection of environments.

Method

The authors propose a framework to evaluate and design effective training environment distributions for multimodal agents, focusing on two key dimensions: diversity and difficulty structure. The overall framework is illustrated below.

Diversity: Ability-aware Environment Selection (AES)

To ensure the environment set covers broad abilities while avoiding redundancy and conflicts, the authors characterize environment diversity from the perspective of agent ability diversity. They first construct meta-ability profiles for each environment. For every environment, they collect agent trajectories from models with different capability levels to capture both successful strategies and failure patterns. An annotation model then decomposes these trajectories into a sequence of atomic abilities, which are interpretable behavior units meaningful for solving tasks. These atomic abilities are aggregated to build a meta-ability profile $P_e$Pe​ for each environment $e$e, containing meta-abilities, transition edges, and their frequencies. Meta-abilities are further categorized into core abilities, which are stable and directly related to task completion, and soft abilities.

Based on these profiles, the authors propose Ability-aware Environment Selection (AES). For each candidate environment $e$e, its coverage set $C(e)$C(e) is constructed from core and soft abilities, with core abilities assigned higher weights. Given the current selected environment set $S$S, the new coverage brought by $e$e is defined as:

$$\text{NewCoverage}(e, S) = \sum_{a \in C(e) \setminus C(S)} w_a$$NewCoverage(e,S)=a∈C(e)∖C(S)∑​wa​

where $w_a$wa​ denotes the weight of ability $a$a.

To mitigate redundancy and optimization conflicts during joint training, the authors define redundancy and conflict scores. Redundancy is measured by the similarity between meta-ability profiles:

$$\text{Redundancy}(e, S) = \max_{e' \in S} \text{sim}(P_e, P_{e'})$$Redundancy(e,S)=e′∈Smax​sim(Pe​,Pe′​)

Conflict is estimated by the gradient cosine similarity between environments:

$$\text{Conflict}(e, S) = \max_{e' \in S} \max(0, -\cos(g_e, g_{e'}))$$Conflict(e,S)=e′∈Smax​max(0,−cos(ge​,ge′​))

The gain of a candidate environment $e$e is then calculated as:

$$\text{Gain}(e \mid S) = \lambda_1 \text{NewCoverage}(e, S) - \lambda_2 \text{Redundancy}(e, S) - \lambda_3 \text{Conflict}(e, S)$$Gain(e∣S)=λ1​NewCoverage(e,S)−λ2​Redundancy(e,S)−λ3​Conflict(e,S)

The algorithm iteratively selects the environment with the highest gain until all core behaviors are covered. This process effectively preserves broad ability coverage with a compact subset of environments, as demonstrated by the coverage accumulation curve below.

Difficulty: Hierarchical Difficulty Curriculum (HDC)

To address the bottleneck where multimodal agents struggle with raw visual inputs early in training, the authors design textual harnesses as training scaffolds. These harnesses include textual observations, text states, text hints, and rule descriptions, providing auxiliary information to help the model extract states and understand rules.

The authors propose a Hierarchical Difficulty Curriculum (HDC) that organizes training along two difficulty axes: harness weakening and state-scale progression. Harness weakening serves as the outer curriculum, gradually removing auxiliary scaffolds to force the model to rely on raw visual observations. State-scale difficulty, referring to the scale complexity of an environment instance, serves as the inner curriculum.

For each environment $e$e, the curriculum state is maintained as a tuple $(r_e, \ell_e, u_e)$(re​,ℓe​,ue​), where $r_e$re​ is the current harness frontier and $[\ell_e, u_e]$[ℓe​,ue​] is the state-scale sampling window. During training, a harness level $h$h is sampled from an environment-specific distribution $D_e(h \mid r_e)$De​(h∣re​), which assigns probability $p_{\text{cur}}$pcur​ to the current frontier and distributes the remaining probability over earlier levels with exponential decay:

$$w_h = \exp(-\alpha(r_e - h))$$wh​=exp(−α(re​−h)) $$D_e(h \mid r_e) = (1 - p_{\text{cur}}) \frac{w_h}{\sum_{j < r_e} w_j}, \quad h < r_e$$De​(h∣re​)=(1−pcur​)∑j<re​​wj​wh​​,h<re​

Similarly, a state-scale level $s$s is sampled uniformly from a sliding window $[\ell_e, u_e]$[ℓe​,ue​], where $\ell_e = \max(0, u_e - \Delta d)$ℓe​=max(0,ue​−Δd).

The curriculum update is hierarchical. When the model reaches a scale advancement threshold $\tau_s$τs​, the inner curriculum advances ($u_e \leftarrow u_e + 1$ue​←ue​+1). Once the target state-scale level is reached and the harness advancement threshold $\tau_h$τh​ is satisfied, the outer curriculum advances ($r_e \leftarrow r_e + 1$re​←re​+1), and the state-scale is reset. This ensures that within each harness level, state-scale complexity increases, and after achieving sufficient competence, the harness is weakened and the scale progression restarts.

Experiment

The main experiments on Qwen3-VL models show that the diversity-aware environment selection (AES) significantly outperforms both random sampling and using all environments, demonstrating that larger environment pools can introduce redundancy and conflict that hinder training. Combining AES with difficulty-aware curriculum learning (HDC) further enhances performance, indicating that diversity and difficulty are complementary dimensions for effective multimodal environment training. Ablation studies confirm that redundancy control, conflict control, and both harness weakening and state-scale progression are essential for achieving robust gains.

Mixed-environment training causes a larger performance drop in multimodal settings than in text-symbolic settings. The text-symbolic version shows only a minor decline, whereas the multimodal version suffers a substantial degradation, indicating that multimodal environments are more prone to negative transfer. In text-symbolic environments, mixed training reduces performance by just 1.3% compared to single-environment models. In multimodal environments, the same mixed training leads to a 10.7% drop, more than eight times the decline observed in the text-symbolic case. Even when task logic is unchanged, multimodal observations amplify optimization conflicts between environments, resulting in stronger negative transfer.

Without any textual harness, the model achieves only low success rates. Providing explicit text state descriptions yields the largest improvement, followed by textual observations, while rule descriptions offer a smaller but still positive gain. Across all harness types, single-turn performance consistently exceeds multi-turn. Text state and text observation harnesses deliver the strongest performance improvements over the no-harness baseline. All harness types outperform having no auxiliary information, with text state being the most effective. Single-turn success rates are consistently higher than multi-turn rates for every harness condition.

Selecting a small set of 30 environments with the AES method, which maximizes meta-ability coverage while controlling redundancy and conflict, yields substantially higher relative gains in both in-distribution and out-of-distribution success rates compared to using all available environments or random selection. Adding difficulty‑aware curriculum learning (HDC) further boosts these gains, with the combination of AES and HDC achieving the strongest overall performance. The improvements come primarily in the target environments, while performance on general visual‑language benchmarks remains nearly unchanged. On the 4B model, AES achieves relative gains of 150.6% on in-distribution and 40.3% on out-of-distribution environments, far surpassing the 80.1% and 7.5% gains from training on all environments. Combining AES with HDC produces the best results, with an average relative gain of 143.2% across ID/OOD groups and two model scales, demonstrating that diversity and difficulty curricula complement each other. Removing conflict control from AES causes the out-of-distribution relative gain to drop from 40.3% to only 2.8%, showing its critical role in robust environment selection.

The full AES method achieves a 40.3% relative gain over the base model on out-of-distribution environments. Removing conflict control causes the improvement to nearly vanish (only 2.8% gain), while ablating redundancy control reduces it to 25.3%. Both components are crucial, with conflict control having a larger impact on performance. Removing conflict control collapses the relative gain from 40.3% to just 2.8%, showing that filtering out conflicting environments is essential. Dropping redundancy control lowers the gain to 25.3% and causes the method to select 41 environments instead of 30, indicating that redundant environments reduce training efficiency.

Ablation of the two HDC axes shows that both harness weakening and state-scale progression individually improve over the AES baseline. Harness weakening yields a larger gain than state-scale progression. Combining both axes produces the highest relative gain, indicating they are complementary and jointly enhance multimodal training. Harness-only curriculum delivers a larger average relative gain than scale-only curriculum. The combined curriculum achieves the best performance, confirming that the two difficulty components complement each other.

The experiments evaluate multimodal agent training by comparing text-symbolic and visual observation setups, testing textual auxiliary harnesses, and proposing a selective environment sampling method (AES) with difficulty-aware curriculum learning (HDC). Mixed-environment training causes disproportionately larger performance drops in multimodal settings due to observation-driven conflicts, while providing explicit text state descriptions yields the greatest success rate gains. AES, which maximizes meta-ability coverage while controlling redundancy and conflict, substantially boosts in-distribution and out-of-distribution performance, and conflict control proves critical for generalization. Combining environment selection with a curriculum that progressively weakens harnesses and increases visual complexity further enhances results, showing that diversity and difficulty curricula complement each other.