One-sentence Summary

MatrAIx introduces a population-scale simulated-user evaluation infrastructure with Persona 8B's 8.3 billion persona records (1,290 categorical dimensions) and four interactive environments, enabling testing of AI systems and digital products across 1,010 tasks in over 25 domains, where persona agents powered by Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5 achieved $91.5\%$91.5% persona adherence in a 400-trial controlled study and human-grounded personas were validated by human and LLM judges.

Key Contributions

• Persona 8B contains 8.3 billion persona records across 1,290 categorical dimensions, generated by sampling from a dependency graph that preserves cross-attribute correlations or by deriving records from human-authored profiles. A quality-filtered coreset of approximately one million personas is released, comprising 599,847 human-grounded and 400,000 synthetic records.

• The MatrAIx Playground runs persona agents in four interactive environments (Survey, AI Chatbot, Web, and App) to capture behaviors such as hesitation after a price increase, willingness to continue after an AI assistant fails, and latency tolerance.

• A library of 1,010 application tasks spanning more than 25 domains (Commerce, Software, Finance, Healthcare) is provided. Validation across 18,189 trials with three persona-agent models includes a controlled adherence study (91.5% correct) and a human evaluation yielding a mean extraction quality score of 4.135 out of 5 for human-grounded personas.

Introduction

Human evaluation is critical for capturing real user experiences, but it is expensive and slow, while offline benchmarks are scalable yet measure only task outcomes without modeling how diverse users formulate requests, interact, and react. Prior persona-based simulation efforts often lack full evaluation pipelines from population sampling to interactive trials, and existing simulators may not preserve cross-attribute dependencies or reliably adhere to assigned behaviors. The authors introduce MatrAIx, a population-scale infrastructure that combines an 8.3-billion-record persona dataset, four interactive evaluation environments (Survey, AI Chatbot, Web, App), and a library of over 1,000 reusable tasks across 25+ domains, enabling large-scale testing with heterogeneous user profiles and exposing subgroup-level variation that aggregate scores miss.

Dataset

The authors introduce Persona 8B, a population-scale persona dataset containing 8.3 billion persona records designed to power the MatrAIx simulated-user evaluation infrastructure. The dataset is not intended to reconstruct real individuals but provides a shared schema for describing human variation, querying records, and sampling evaluation cohorts. A persona record becomes a persona agent when paired with a model, and evaluations then assign sampled agents to tasks.

Schema and composition

• All records share 1,290 categorical dimensions grouped into five areas: background, psychology, capability, behavior, and lifestyle.
• Each dimension uses a finite set of values (e.g., age bracket, English proficiency level, risk tolerance).
• The schema is informed by public sources covering demographics, economics, education, labor, health, values, and technology use (e.g., UN World Population Prospects, World Bank indicators, ILOSTAT, public surveys, developer ecosystem statistics).

Synthetic persona generation

• To avoid incoherent profiles that result from independent sampling, the authors build a dependency-aware probabilistic model using a directed acyclic graph (DAG) over the 1,290 dimensions.
• Each dimension is sampled conditionally on its parents, with edges added only when a source directly reports the conditional relationship.
• For a given dimension, a candidate value is scored by combining a population prior, a parent-dependent adjustment, and a binary compatibility mask. This separates adjusting for likely contexts from enforcing hard constraints (e.g., a persona with English as primary language cannot have proficiency “None”).
• Synthetic records are generated by forward sampling in topological order, starting from root attributes with categorical priors.

Human-grounded records

• To complement the synthetic population, the authors map real-world evidence into the same 1,290-dimensional schema. Only source-supported assignments are filled; unsupported dimensions remain null.
• Six sources are used:
  • Wikipedia biographies
  • Amazon Reviews (grouped by reviewer)
  • Stack Overflow Developer Survey (mapped via deterministic crosswalk)
  • General Social Survey (GSS) responses (coded answers mapped directly)
  • PRISM Alignment (coded demographics and self-descriptions)
  • MatrAIx Persona Survey (355 consented self-reports collected through social media and university lists, with no direct identifiers)
• Free-text content from Wikipedia, Amazon Reviews, and PRISM Alignment is processed using LLM-based constrained extraction.

Quality control and the public 1M coreset

• Synthetic records are filtered for cross-attribute conflicts; human-grounded records are checked for unsupported assignments and provenance.
• Human-grounded records are deduplicated with exact hashing and MinHash-based fuzzy detection. Synthetic records are deduplicated by comparing 14 high-information attributes; records with identical values across all 14 are treated as duplicates and only one is kept.
• Because human-grounded sources are not population-representative, the authors calibrate a public 1M coreset by selecting synthetic records so that the combined set approximates published statistics for age bracket, region, gender identity, and urbanicity. Missing fields are not imputed.
• The final deterministic coreset contains 599,847 human-grounded records and 400,000 synthetic records. This split is a release design choice, not an estimate of a real-world ratio.

How the dataset is used

• Persona 8B is the foundation of the MatrAIx simulated-user evaluation infrastructure. It is not used for model training.
• In evaluation, individual persona records are paired with a model to create persona agents. Tasks are then assigned to sampled cohorts of these agents, enabling large-scale, population-like testing of model behavior.

Method

The authors develop a population-scale persona dataset and an evaluation infrastructure that together enable running realistic, controlled simulations of human behavior. The methodology consists of three connected components: a dependency-aware probabilistic model for synthetic persona generation, a simulation execution framework, and a formal task specification contract.

Synthetic Persona Generation with DAG Sampling

Generating coherent synthetic personas requires preserving the complex dependencies among demographic and behavioral attributes. Simply matching marginal distributions would produce implausible profiles, such as individuals with mismatched age–education or region–language combinations. To address this, the authors construct a directed acyclic graph (DAG) over the 1,290 persona dimensions, where each edge encodes a conditional dependency for which a direct source exists. Let $G = (\mathcal{D}, E)$G=(D,E) be the graph and $\mathrm{Pa}(i)$Pa(i) the parent nodes of dimension $X_i$Xi​. The proposal distribution then factorizes as

This factorization makes local dependence assumptions explicit, conditioning each attribute only on its relevant predecessors. For root dimensions, the local conditional probability distribution (CPD) is a categorical prior $\pi_i(\nu)$πi​(ν) derived from population-wide statistics. For a non-root dimension, a candidate value $\nu$ν is scored by combining the prior with a parent-dependent adjustment and a compatibility mask:

The adjustment term $r_i$ri​ uses source-informed likelihood ratios to upweight values that are common in the given parent context and downweight rare ones. The binary mask $m_i \in \{0,1\}$mi​∈{0,1} enforces hard compatibility constraints: impossible combinations (e.g., English proficiency “None” when primary language is English) receive a zero mask and are removed, while atypical but plausible values remain eligible. Normalizing the scores yields the final CPD. Synthetic personas are generated by forward sampling in topological order $\tau = (\tau_1, \dots, \tau_d)$τ=(τ1​,…,τd​):

Root attributes are drawn from their grounded priors, and each downstream attribute is drawn from its normalized local CPD, ensuring that every draw uses the context on which it depends. This approach produces a coherent synthetic population while preserving rare but valid profiles. The base priors and dependency edges are derived from trusted sources, and the graph construction is carefully reviewed to avoid inventing non-existent joint distributions.

Simulation Configuration and Execution

Once personas are available, the evaluation infrastructure orchestrates simulations through a centralized playground. A simulation begins with a population query and an application task specification. The playground records the eligible persona pool, sampling procedure, task version, agent interface, and underlying model in a run manifest. It then launches independent trials in parallel, one per persona. Each trial is represented as a tuple $\tau = \langle \pi, \theta, \alpha, \mu, \sigma \rangle$τ=⟨π,θ,α,μ,σ⟩, where persona $\pi$π performs task $\theta$θ through agent interface $\alpha$α using model $\mu$μ and seed $\sigma$σ. The trial produces a canonical artifact bundle $A = \alpha(\pi, \theta; \mu)$A=α(π,θ;μ) containing the persona’s submission and, where applicable, the interaction trajectory and environment state. A task-owned verifier maps this bundle to typed findings $V_\theta(A)$Vθ​(A). Trials are stateless and thus fully parallelizable. The manifest preserves both the requested population and the realized cohort, while the typed findings keep persona fidelity, product outcomes, and execution failures distinct, preventing any conflation between task success and human validity.

Application Task Specification

Each simulation task is defined by a portable contract that names the evaluation target (e.g., an AI model, chatbot, website, or native application), the persona cohort, the user-facing scenario and objective, and the evidence that counts as a meaningful outcome. This separation ensures that product credentials and scoring rules never leak into persona instructions. The task also specifies which artifacts to retain and how they will be evaluated. Survey tasks produce structured answers and rationales; AI chatbot tasks record conversations and post-run feedback; web tasks preserve pages viewed, actions taken, and final submissions; app tasks can capture exported files, permission changes, and final application state. A task-specific verifier then converts these artifacts into structured findings. Programmatic checks assess directly observable outcomes, while human or LLM judges evaluate interpretive properties using recorded prompts and rubrics. Reports summarize completion rates, outcome distributions, uncertainty, and subgroup differences, all while retaining links to the underlying traces. This design keeps product outcomes, simulated-user behavior, and persona fidelity in separate analytical channels, enabling rigorous, reproducible evaluation of AI systems against diverse human populations.

Experiment

The MatrAIx evaluation infrastructure provides four environments (Survey, AI Chatbot, Web, App) and a task library of over 1,000 specifications to simulate user interactions with products. Validation experiments confirm that tasks execute reliably, persona effects are recoverable across models when the attribute is task-relevant, declared behaviors are followed in the vast majority of trials, and human-grounded persona records achieve high extraction quality. These results support end-to-end execution, persona adherence, and multidimensional reporting, with the persona-agent model reported as a key configuration factor.

The Persona 8B schema comprises 1,290 categorical dimensions organized into five thematic groups. Lifestyle attributes form the largest group, while Behavior and Interaction attributes form the smallest, and each group draws on distinct grounding sources ranging from population statistics to validated psychological instruments. Lifestyle dimensions are the most numerous (387), and Behavior and Interaction dimensions are the fewest (124). Background attributes are grounded in population statistics and household surveys, whereas Psychology attributes rely on validated instruments and values surveys. Capability attributes, covering domain expertise and programming, are grounded in occupational taxonomies and technology/developer surveys. Grounding sources vary in kind and strength across groups, and do not provide direct population estimates for every value.

The public Persona 1M coreset contains 599,847 human-grounded records from six extraction sources, led by Wikipedia (323,438), and 400,000 synthetic records chosen to calibrate the combined set toward population marginal distributions for age, region, gender identity, and urbanicity. The human-grounded portion is not population-representative on its own, and the 60/40 split is a release design choice. Wikipedia extraction is the largest human-grounded source, providing 323,438 of the 599,847 human-grounded records. Synthetic records account for 400,000 of the coreset, bringing the total to roughly 1 million records and aiming to improve demographic alignment.

The task library contains 1,010 unique specifications across four environments. Survey tasks dominate with 621 entries, followed by AI Chatbot with 371, while Web and App together contribute only 18 tasks. The anchor domains of Commerce, Software, Finance, and Healthcare are well-covered in Survey, whereas AI Chatbot tasks are heavily concentrated in the 'Other' category spanning over 25 additional domains. Validation runs used two tasks per environment, with App tasks limited to far fewer personas due to higher native interaction costs. Survey tasks represent 61% of the library and are evenly distributed across the four anchor domains. AI Chatbot tasks are almost entirely in the 'Other' domain, which aggregates more than 25 different domains. Web and App environments are sparsely represented, with only 12 and 6 tasks respectively. Validation executed two tasks per environment; App tasks used only 20–24 personas per model compared to roughly 1,000 for the other environments.

In a controlled behavioral adherence test, persona-driven agents expressed or suppressed the intended attribute in 91.5% of 400 trials across four environments. Adherence was high across most attributes and environments, though the App environment showed lower consistency, particularly for politeness, storytelling, and humor. Survey, Chatbot, and Web environments each met the success threshold for 9 of 10 attributes, while App met it for only 6 of 10. Overall, 33 of 40 attribute-by-environment cells achieved at least four out of five successes in both arms. App environment struggled most, with cog-politeness receiving a 5/10 and cog-storytelling a 6/10. Survey, Chatbot, and Web environments each demonstrated strong adherence across 9 of 10 attributes.

The Persona 8B schema organizes 1,290 categorical dimensions into five thematic groups with varied grounding sources, while the Persona 1M coreset combines human-grounded and synthetic records to achieve better demographic alignment. The task library contains 1,010 specifications across four environments, heavily dominated by survey and AI chatbot tasks. A controlled behavioral adherence test shows that persona-driven agents reliably express intended attributes in 91.5% of trials across environments, though the App environment exhibits lower consistency for social attributes like politeness, storytelling, and humor.