One-sentence Summary

Researchers from Leiden University, Renmin University of China, Alibaba Group, and the Institute of Computing Technology, Chinese Academy of Sciences introduce USERARENA, a real-world e-commerce interactive benchmark with 64,729 session trajectories from 11,010 users spanning 566,102 items and 13 action types, and propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent on student-generated rollouts to achieve the strongest overall fidelity among evaluated simulators and closely reproduce production A/B policy effects.

Key Contributions

• The paper introduces USERARENA, a real-world e-commerce interactive benchmark and environment containing 64,729 long-horizon session trajectories from 11,010 users, spanning 566,102 items and 13 action types, with a five-level evaluation protocol and offline validation against a controlled production A/B test.
• The work proposes On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent and dense teacher targets on student-generated rollouts to address sparse rewards and exposure bias in long-horizon user simulation.
• Experiments show that OPTD achieves the strongest overall simulator fidelity under the USERARENA evaluation protocol and most closely reproduces the production policy effect in offline simulation.

Introduction

Recent advances in language and multimodal agents enable agentic recommendation, where a policy makes a sequence of context-dependent decisions. Evaluating and optimizing such policies requires a scalable recommendation world model that simulates both the shopping environment and user responses, reducing reliance on costly online experiments. Prior resources offer behavioral data and interactive evaluation but rarely provide long-horizon trajectories that align real user profiles, visual observations, product context, and actions across repeated sessions. Evaluation also tends to measure step-wise next-action agreement rather than long-horizon behavioral fidelity, and it seldom validates whether simulator-based policy comparisons agree with production A/B outcomes. To address these gaps, the authors introduce UserArena, an interactive benchmark with a production-grounded shopping environment and real-user trajectories from a large e-commerce platform, along with a five-level evaluation protocol. They also propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent and dense teacher supervision on student-generated rollouts to improve faithful long-horizon user simulation.

Dataset

The authors introduce USERARENA as a production-grounded e-commerce interaction benchmark built from real-user shopping trajectories. The dataset description focuses on its composition, scale, processing, and evaluation use.

• Overall composition and scale: The dataset contains 11,010 users, 64,729 sessions, 577,670 actions, and 566,102 products. It includes 6 user attributes and records 73,717 item clicks across 41 product categories.

• Source and interaction design: Data comes from a production shopping environment that reconstructs recommendation-feed and item-detail pages. The action space has 13 action types, including feed navigation, product inspection, add-to-cart, and purchase. Each recorded decision aligns the shopper’s profile, visual observation, visible product context, and action.

• Visual processing: Recommendation feeds are pre-rendered as full pages. During simulation, the visible viewport is cropped from the pre-rendered full feed as the simulator scrolls. Clicking an exposed item opens an interactive detail page reconstructed from that item’s pre-rendered views and recorded product content.

• Long-horizon subset: For cross-session evaluation, USERARENA organizes 59,378 chronologically ordered sessions from 5,768 users into long-horizon trajectories. These trajectories average 10.29 sessions and 86.36 actions per user.

• Evaluation use: The dataset is used for trajectory-level evaluation rather than only next-action agreement. It also supports offline simulation comparisons with controlled production A/B outcomes for the same recommendation policy pair.

Method

The authors propose On-Policy Trajectory Distillation (OPTD) to model long-horizon shopping behavior by extracting trajectory-level intent and using it to provide dense, token-level supervision to a student policy. This approach addresses the train-test mismatch inherent in standard supervised fine-tuning and the coarse credit assignment of reinforcement learning methods.

As shown in the figure below:

To align training with autoregressive inference, the authors treat the student policy $\pi_\theta$πθ​ as the generator and optimize it using on-policy sampling. At each training step, a real state $x_t$xt​ is drawn from the empirical distribution, and a complete response $y_t$yt​ is sampled from the current policy:

This exposes the model to self-generated prefixes during training, mitigating the discrepancy between teacher-forced training and actual inference.

To guide the student toward behavioral patterns spanning an entire session, the framework introduces trajectory-level intent supervision. A large language model acts as an extractor $E$E to map the complete training trajectory $\tau_u$τu​ to an abstract intent summary $z_u = E(\tau_u)$zu​=E(τu​). This structured record captures trajectory-wide preferences and decision tendencies without revealing concrete actions or future states. A frozen teacher model with parameters $\phi$ϕ then computes an intent-conditioned next-token distribution:

The student never directly observes $z_u$zu​; it only receives guidance through the token distributions produced by the teacher.

To prevent the student from over-relying on training-only privileged information, the authors implement a per-token trust-region regularization. They construct an unconditioned reference distribution $q_{t,k}^0(v) = P_\phi(v \mid x_t, y_{t,<k})$qt,k0​(v)=Pϕ​(v∣xt​,yt,<k​) by querying the teacher without the intent context. The final target distribution is formed by interpolating the two distributions in log-probability space:

For each token, the algorithm selects the maximum interpolation coefficient $\lambda_{t,k}^\star \in [0, \lambda_{\max}]$λt,k⋆​∈[0,λmax​] such that the Kullback-Leibler divergence $D_{\mathrm{KL}}(q_{t,k}^\lambda \| q_{t,k}^0)$DKL​(qt,kλ​∥qt,k0​) remains within a predefined budget $\epsilon$ϵ. This ensures the intent guidance is adaptively bounded at every generation step.

The overall training objective combines a distillation term with a grounding term:

The distillation loss $\mathcal{L}_{\mathrm{dist}}$Ldist​ matches the regularized target $q_{t,k}^{\lambda_{t,k}^\star}$qt,kλt,k⋆​​ at every generated token using the generalized Jensen-Shannon divergence, which remains bounded even under large student-target disagreement. The grounding loss $\mathcal{L}_{\mathrm{GT}}$LGT​ applies standard teacher-forced cross-entropy on the recorded response $y_t^\star$yt⋆​ to provide step-level supervision. Together, these losses effectively combine session-wide intent with state-level behavioral grounding.

Experiment

The USERARENA benchmark reconstructs a production-grounded shopping environment with fine-grained real-user trajectories and evaluates simulators across five fidelity levels, from next-action parsing to long-horizon outcomes. In controlled comparisons using the same base model, OPTD outperforms text-based and GUI-based simulators on most behavioral fidelity measures in both single-session and cross-session settings. Production A/B validation further shows that OPTD most closely reproduces real-user policy effects, while ablations confirm that each training component contributes and human evaluation rates its generated trajectories as more human-like than strong baselines.

Across representative interactive recommendation benchmarks, most provide real user traces, but coverage of visual context, full action space, long-horizon histories, and production A/B validation is uneven. Long-horizon coverage and production A/B validation are especially rare, and no other listed benchmark combines all five capabilities. USERARENA is the only system that includes real traces, visual context, a full action space, long-horizon behavior, and production A/B validation. Real user traces are present in nearly all listed benchmarks, with A/B Agent as the only exception. Visual context and a full action space appear together in OPeRA, USB, A/B Agent, and USERARENA. Long-horizon coverage is uncommon, appearing only in OmniBehavior and USERARENA. Production A/B validation is absent from all other listed benchmarks and appears only in USERARENA.

OPTD performs best overall in both single-session and cross-session evaluations. In single-session, it ranks first on all reported L1-L4 metrics, while in cross-session it ranks first on five of eight metrics and second on two more, with especially strong sequential and personalized population fidelity. Other simulators show narrower strengths, such as STA on selected local and outcome metrics and Customer-R1 on local prediction. OPTD ranks first across all eight single-session L1-L4 metrics, covering local, sequential, personalized population, and outcome fidelity. In cross-session evaluation, OPTD achieves the strongest sequential behavioral fidelity and the highest personalized population fidelity. STA leads macro-F1 and CTR error in cross-session, while A/B Agent stays competitive on ACR error but shows weaker transfer on replay and rollout valid-action rates.

In production A/B validation of a ranking-policy pair, OPTD most closely reproduces the online change in page-view click-through rate, achieving the smallest effect error among the evaluated simulators. Most other user simulators substantially underestimate the policy effect, while GPT-6-Sol overestimates it. Simulator effect ratios diverge widely from the recorded online outcome, indicating large differences in behavioral calibration. OPTD has the lowest effect error among all compared simulators, making it the most aligned with the production A/B outcome. GPT-6-Sol is the only compared simulator with an effect ratio above one, overestimating the online response. Several simulators, including AlignUSER and Qwen3.5-2B, produce near-zero effect ratios, indicating severe underestimation of the policy effect. The spread of effect ratios shows that simulator choice strongly affects whether policy comparisons match real user behavior.

The benchmark comparison evaluates coverage of real user traces, visual context, full action space, long-horizon behavior, and production A/B validation, and finds that only USERARENA combines all five capabilities while long-horizon and production validation remain rare elsewhere. In simulator fidelity experiments, OPTD performs best overall in both single-session and cross-session settings, leading all single-session metrics and most cross-session metrics, while other simulators show narrower strengths. Production A/B validation of a ranking-policy pair further shows that OPTD most closely reproduces the online change in page-view click-through rate and achieves the smallest effect error, whereas most simulators substantially underestimate the policy effect and one overestimates it.