One-sentence Summary

Xiaohongshu Dots Studio and Evolvent AI introduce VibeLifeBench, a benchmark of 200 multi-week everyday-life tasks in a simulated world of 22 mock services where agents must be proactive, detect silent changes, and maintain consistency under implicit constraints, with evaluation revealing that seven frontier LLMs score low, underscoring the gap between current assistants and real-life proactive agents.

Key Contributions

• The paper introduces VibeLifeBench, a benchmark of 200 long-horizon tasks across ten everyday-life domains, where each task unfolds over a multi-week timeline in a simulated world of 22 mock services that evolves independently and silently, requiring agents to act proactively and maintain a coherent plan.
• A stage-aware, weighted scoring methodology is presented that jointly measures end-state correctness, the timeliness of proactive actions, and the faithful propagation of unannounced world changes, using only the agent's observable traces.
• Experiments on seven frontier models show that all score low on the benchmark, and the tasks, environments, and evaluation framework will be open-sourced to support research on persistent, proactive life agents.

Introduction

While large language model agents are increasingly deployed in professional settings like coding and office work, everyday life assistance remains underexplored, despite its importance for ordinary users. Prior benchmarks concentrate on passive execution of explicit tasks in static environments, failing to capture proactivity, adaptation to a dynamically changing world, and long-horizon coherence across multi-week lifecycles. The authors introduce VibeLifeBench, a benchmark of 200 tasks spanning ten everyday-life domains, each structured as a multi-week simulated living world with 22 mock service backends and 288 tool interfaces. The world evolves autonomously through scripted events, requiring agents to proactively re-inspect changes, maintain evolving plans, and decide when to act or notify the user. Stage-aware scoring with over 12,000 checks evaluates both end-state correctness and proactive behavior, revealing that even the strongest frontier models achieve only a 32.5 average@3, indicating a substantial gap for reliable, long-term life assistance.

Dataset

The authors introduce VibeLifeBench, a curated evaluation suite of 200 long‑horizon tasks that measure how well an agent assistant can stay proactive, trustworthy, and persistent over weeks of simulated time. Here is a breakdown of the dataset’s composition, key statistics, and how it is used.

What the dataset contains

• Each task is a self‑contained directory that bundles five components:
  • Initial world state – seed data for a set of 22 mock services, creating a realistic starting point (e.g., bookings, accounts, calendar items).
  • Event timeline – a sequence of stages, each containing timestamped events (user messages, notifications, world observations, and silent mutations).
  • Enabled services – a subset of the 22 mock backends the task activates (e.g., email, banking, travel booking).
  • Persona and workspace – background files that define the user’s identity, preferences, constraints, and authorization boundaries.
  • Scoring criteria – a collection of weighted, deterministic checks that inspect the agent’s output artifacts (end state of services, files, emails, replies) and never peek at its internal reasoning.

Sources and construction

• Tasks are handcrafted from everyday‑life scenarios (a trip abroad, a rental dispute, a renovation). They are spread evenly across 10 domains (20 tasks each), including travel, finance, shopping, health, and career.
• A uniform pipeline turns each scenario into a reproducible test:
  1. Scenario & persona – fix the user’s identity, implicit constraints, safety red lines, and authorization limits.
  2. Timeline authoring – unroll the story into a stage‑organized event timeline. A large share of changes are silent mutations (1,483 total) that alter the world without triggering a turn, forcing the agent to re‑inspect proactively.
  3. Environment instantiation – seed the needed services per task with ample distractors, making all key entities discoverable only through tool calls.
  4. Scoring criteria – author weighted checks that demand concrete evidence (value computation, state change) rather than keyword matching, grouped into per‑stage, cross‑stage, and final tiers.

Key statistics

• Task horizon: median simulated duration 29 days (range up to ~111 days), compressed into a median of 24 stages.
• Events: 7,453 total events across the suite, median 36 per task. Composition: 30.1% user messages, 25.8% notifications/reminders, 24.1% world observations, 19.9% silent mutations (1,483 events). Overall, 69.9% of events are environment‑driven, not user‑prompted.
• Services: 22 mock backends expose 288 tool interfaces. A task uses a median of 7 services (up to 12). General services like email (198/200 tasks), calendar (195), and notes (162) are near‑universal; domain‑specific backends (banking, flight booking, visa, etc.) appear in relevant clusters.
• Scoring: 12,261 weighted checks, median 58 per task (range 31 to 135). Per‑stage checks make up 80.8% of the count, but cross‑stage and final checks carry 26.8% of the total weight, heavily penalizing safety violations, budget breaches, or information leaks.

How the dataset is used

• VibeLifeBench is a pure evaluation benchmark; it does not provide training data or a train/test split. The 200 tasks are released as a fixed test set.
• An agent policy is run through the entire timeline of a task. At each turn it may issue tool calls, write to workspace files, or send replies. Its observable trace (the end state of the services, files, emails, and reply text) is then compared against the deterministic scoring checks.
• The overall score is the fraction of weighted checks passed, reported on a 0–100 scale. This rewards partial competence over the long horizon and ensures that a single critical failure (e.g., leaking personal data) cannot be masked by many routine successes.

Processing and reproducibility

• All tasks share the same 22 mock service backends, each of which boots from seed data on a cold start. The world simulation is fully offline, deterministic, and reproducible.
• The construction deliberately compresses quiet intervals into sparse stages, so a multi‑week task can be evaluated in a manageable number of checkpoints while preserving the long‑term memory and proactivity requirements.
• Persona and workspace files embed implicit constraints and authorization boundaries that the agent must infer and uphold; tempting shortcuts are planted to test compliance.

Method

The VibeLifeBench framework is built on the premise that a task is not a single prompt but a world advancing on a clock. An agent is placed in a persistent situation, equipped with a set of tools and one or more personas to serve, and time begins to move forward. The underlying world state changes whether or not the agent pays attention, and the agent must decide when to act, when to re-inspect, and when to stay silent. This design gives rise to three core commitments: the world advances silently to measure proactivity; tasks embed implicit constraints and safety red lines to test trustworthiness; and shared, reproducible service backends with per-task initial data reconcile realism with determinism.

Each task is formalized as a five-tuple

where the components are:

• $W_0$W0​: the initial world state, determined by seed data provided for each enabled service backend.
• $\mathcal{E} = \{e_1, e_2, \dots\}$E={e1​,e2​,…}: an event timeline, grouped into stages and ordered by timestamp within a stage.
• $\mathcal{K} \subseteq \{k_1, \dots, k_{22}\}$K⊆{k1​,…,k22​}: the set of service capabilities (tool backends) available to the agent.
• $P$P: the persona and workspace, including the user’s identity, preferences, authorization policy, and any background material given at the start.
• $R = \{(c_i, w_i)\}_{i=1}^m$R={(ci​,wi​)}i=1m​: the scoring criteria, a collection of weighted deterministic predicates over the observable world.

A run advances the agent policy through the timeline slot by slot. Let $W_j$Wj​ be the world state after processing the $j$j-th dispatched item. The state evolves as

where $a_j$aj​ is the sequence of actions the agent takes in that turn (tool calls, file writes, replies) and $H_{j-1}$Hj−1​ is its history. For a silent mutation, no turn is produced, $a_j = \emptyset$aj​=∅, and the world changes solely because of the event itself. The agent’s output is not a single answer but the entire observable trace: the end state of backend services, workspace files, notes, calendar, sent emails, and replies at each stage. Scoring is based exclusively on these artifacts.

The world is simulated by 22 mock service backends, representing the everyday applications an agent would interact with. They are divided into general services (email, calendar, notes, notification hub) and domain services (banking, travel booking, e-commerce, logistics, legal search, etc.). Each service is a self-contained product backend that defines its own data model, boots from seed data on a cold start, and exposes a stable set of tools. Together they provide 288 tool interfaces. A task only configures its own scenario data and event pacing; the backends themselves remain unchanged across tasks, ensuring reproducibility while keeping the evaluation fully offline and deterministic.

The event timeline is organized into stages, each a checkpoint at which the agent may need to act or the scorer inspects the world. Events are dispatched in timestamp order and come in four kinds: user messages, external world advisories, notifications, and silent mutations. The first three kinds open a turn and capture the agent’s reply; a mutation, in contrast, is applied directly to the relevant service state without any accompanying message or notification. This gap is the core mechanism for measuring proactivity. A mutation changes the world silently, and later stages depend on that change. Only an agent that persistently re-inspects the world on its own can detect and respond to the discrepancy in time. The benchmark embeds 1,483 such background mutations across tasks.

Scoring is performed by evaluating the weighted checks in $R$R against the final world state and produced artifacts. Checks are deterministic predicates that never read the model’s hidden reasoning. They are organized into three tiers: per-stage checks that reward timely behavior, cross-stage checks that enforce episode-long constraints (budget caps, safety red lines), and final checks that inspect the end state the agent leaves behind. The score is the fraction of total check weight earned, scaled to 0–100. Weights are deliberately uneven: a single critical failure such as leaking personal information or breaching the hard budget carries far more weight than a cosmetic oversight, so an agent cannot mask unsafe actions through routine task completion.

The test set is constructed through a uniform pipeline that translates everyday scenarios into objective, hard-to-game tasks. The pipeline proceeds through four stages:

1. Scenario and persona design: each task starts from a real-life situation (e.g., a trip abroad, a rental dispute). A persona is fixed, along with its workspace containing identity, preferences, and implicit constraints, including safety red lines.
2. Timeline authoring: the scenario is unrolled into a stage-organized event timeline. The four event kinds are interleaved, with a substantial share of world changes arranged as silent mutations to force proactive re-inspection.
3. Environment instantiation: per-task seed data is prepared for each enabled service. The initial state is seeded with distractors to prevent shallow guessing, and all key entities are discoverable through tool calls, not hard-coded into scoring.
4. Scoring criteria authoring: stage-aware, weighted checks are written so that each check makes a discriminating judgment (e.g., computing a concrete value, verifying a real state change) and cannot be fooled by the mere appearance of a keyword.

This pipeline ensures that each task is realistic, that proactivity and trustworthiness are essential for a high score, and that the evaluation remains fully objective and reproducible across 200 diverse tasks, all sharing the same 22 backends.

Experiment

The evaluation uses VibeLifeBench, a benchmark of 200 long-horizon life tasks where agents must persist state, react to unannounced world changes, and satisfy stage-aware checks over multiple runs. Experiments across seven frontier models reveal that all perform poorly, with even the strongest failing to consistently maintain cross-stage artifacts or proactively adapt to a dynamic environment, highlighting a fundamental gap between single-turn tool use and the persistent, proactive behavior needed for real-world life assistance.

Existing agent benchmarks for coding, office, and web tasks do not require proactive decision-making or dynamic environments, and only a few partially address long-horizon planning. VibeLifeBench specifically measures these missing dimensions, and experiments show that current models struggle with initiating actions unprompted, maintaining persistent state, and adapting to independently evolving worlds. Performance also degrades over time in long-horizon tasks, and no model covers all everyday life domains reliably. Across all evaluated models, proactivity scores are low, indicating that agents rarely decide on their own when to act without prompting. Persistence and bookkeeping, which require maintaining cross-stage artifacts, also score low, showing that models tend to respond passively and once. When the environment changes independently, agents often fail to re-inspect and propagate those changes, with propagation and recovery scores sitting alongside proactivity and persistence at the bottom. Every model's pass rate declines from the first third to the last third of a long-horizon task, with drops of 10 to 15 percentage points. No model is competent across all ten life domains, and the consistent easy-to-hard ordering across models highlights the breadth challenge of real-world life assistance. Existing benchmarks are marked as not satisfied for proactive and living-world aspects, and only partially satisfied for long-horizon, contrasting with VibeLifeBench's design.

The experiment defines four event types: user messages, world observations, and notifications all open an agent turn, while mutations silently change the world state without triggering a turn. This distinction shapes agent behavior—agents must actively re-inspect the world to detect and propagate mutations, a capability that current models lack, contributing to low proactivity and persistence scores. Mutations are background state changes that do not trigger an agent turn, requiring the agent to re-inspect the world on its own initiative. Agents routinely miss mutations because they do not re-inspect the world when not explicitly prompted, making adaptation to a living world a core weakness. Proactivity and persistence scores are low across all models, with agents responding passively and rarely maintaining cross-stage, auditable artifacts. Even when mutations are detected, models struggle to propagate the change into their plans, limiting their ability to handle dynamic environments.

The scoring criteria evaluate agents on five dimensions: tool use, backend state, persistent artifacts, reply consistency, and cross-stage coherence. Empirical analysis shows that models struggle most with persistent artifacts and cross-stage consistency, often failing to produce durable records, re-inspect the world proactively, or maintain constraints over long task horizons. Checks that require a persistent artifact pass almost never, indicating that models rarely produce the cross-stage-linked files and notes the scoring criteria demand. The cross-stage consistency dimension, which joins artifacts like a running ledger total and a safety red line that must never be reversed, exposes a widespread failure to maintain auditable state across stages. Proactivity and persistence are low across all models, as many checks hinge on the agent re-inspecting the world without a prompt and propagating mutations into its plan. Every model exhibits a substantial drop in pass rate on checks in the later stages of a task, driven by the difficulty of sustaining constraints rather than by task length alone.

VibeLifeBench tasks simulate life scenarios lasting between 20 and 48 days, with each task broken into many stages and dozens of events. The finance domain requires a median of 94 checks per task, far more than any other domain, highlighting the demand for persistent, cross-stage bookkeeping. Across all domains, tasks recruit 5 to 8 distinct services, showing that agents must coordinate multiple tools throughout a long horizon. The career domain has the longest median simulated horizon at 48 days, while finance requires a median of 94 checks, emphasizing the need for long-horizon coherence and persistent audit trails. All domains feature a median of 24 to 28 stages and 30 to 44 events, indicating that difficulty stems from multi-stage coordination rather than simply from task duration.

Claude Opus 5 achieves the highest average score and generates the most output tokens, but greater resource expenditure does not guarantee top performance. GPT-5.5 ranks second despite using the fewest output tokens and context, while Gemini 3.5 Flash reads the most context and takes the most turns yet lands mid-pack. The results indicate that how effort is spent, rather than the sheer volume of tokens or interactions, drives success. Claude Opus 5 leads with the highest average score and the largest output token count, tool calls, and turns per run. GPT-5.5 places second with the smallest output and context budgets among the top models, yet makes the most tool calls. Gemini 3.5 Flash reads the most context and takes the most turns but scores only in the middle of the pack. DeepSeek-V4-Pro achieves a score of 21.1 on the lowest context budget, showing a frugal yet steady profile.

VibeLifeBench evaluates agents on long-horizon everyday life scenarios in which the world evolves independently through silent mutations, requiring proactive re-inspection and persistent cross-stage bookkeeping. Experiments reveal that current models consistently fail to initiate actions without prompting, maintain durable artifacts, and propagate world changes, and their performance degrades over time regardless of domain. No model covers all life domains reliably, and success depends more on efficient reasoning than on the volume of tokens or interactions, underscoring the gap these benchmarks fill in measuring proactivity, persistence, and dynamic adaptation.