One-sentence Summary

Researchers from Meta, KAIST, and Korea University present WearableQA, a benchmark of 4,084 multiple-choice questions built from 200 users’ longitudinal wearable data, featuring 16 reasoning types and a dual-grounding framework that combines literature-grounded and population-grounded patterns to preserve authentic physiological distributions, revealing that LLMs achieve only 19.6%–72.9% accuracy, leaving health reasoning over real-world wearables far from solved.

Key Contributions

• WearableQA is a benchmark of 4,084 multiple-choice questions built from 200 real users’ longitudinal wearable time series, blood biomarkers, and demographics, preserving device noise and inter-individual variability; evaluation of 14 LLMs yields accuracy from 19.6% to 72.9% (chance 10%), demonstrating effective differentiation of model capabilities.
• A diagnostic taxonomy of 16 question types is defined along two axes: data versus health reasoning and single- versus cross-signal reasoning, enabling fine-grained assessment of distinct reasoning skills over longitudinal health records.
• A dual-grounding framework constructs reliable questions at scale by combining literature-grounded physiological findings with statistically validated population-grounded patterns, ensuring questions capture meaningful real-world relationships.

Introduction

Wearable sensors now enable continuous tracking of heart rate, sleep, and activity, and large language models (LLMs) are increasingly used to interpret these signals for personalized health. However, prior benchmarks for evaluating LLM reasoning on wearable data rely on synthetic or simulated signals, failing to capture the noise, inter-individual variability, and health context of real-world measurements. The authors introduce WearableQA, a benchmark of 4,084 multiple-choice questions built from 200 real users' longitudinal wearable records, blood biomarkers, and demographics. It uses a dual-grounding framework combining literature-derived and population-validated patterns to generate deterministic ground truth, and evaluates models along two axes: data reasoning versus health reasoning, and single-signal versus cross-signal reasoning.

Dataset

The authors construct WearableQA, a benchmark for evaluating wearable health reasoning. It is built from real-world user data and biomedical literature, then used to test models on multiple-choice questions that require interpreting longitudinal wearable signals and blood panels.

Dataset composition and sources

• User cohort: 200 real users sampled from a larger in-the-wild wearable dataset, chosen for demographic diversity (age, sex, BMI, ethnicity). Each user contributes up to 500 days of daily-aggregated records.
• Signals: 16 daily wearable metrics across four domains: cardio-fitness (resting heart rate, heart-rate variability, VO₂max), activity/energy (steps, active calorie burn, BMR calories, exercise count/duration/average heart rate, METs), sleep (duration, efficiency, deep-sleep %, REM %, bedtime regularity), and average stress level.
• Blood panel: 17 biomarkers per user (e.g., insulin, HbA1c, glucose, lipids).
• Cohort-reference percentiles: For five core metrics (resting heart rate, steps, heart-rate variability, sleep duration, sleep efficiency), percentiles are computed over a larger real-world cohort to enable peer comparisons.
• Questions: 4,084 multiple-choice items, each with 10 options (chance = 10%), answer positions balanced. Questions are organized along a 2×2 taxonomy: data reasoning vs. health reasoning, and single-metric vs. cross-metric grounding. Additionally, each question is labeled by its ground-truth source: literature-grounded or population-grounded.

Key details for each subset

• Data reasoning (2.7k questions): Requires computing over raw sensor measurements (trends, anomalies, excursions, recovery times, cross-signal relationships). Answers are determined directly from the numbers in the wearable records.
• Health reasoning (1.4k questions): Requires clinical and physiological interpretation (prognostic framing, recommendations, differential reasoning, phenotyping, concordance). Answers layer domain knowledge on top of the signals.
• Single-metric (1.7k questions): Focuses on one signal in isolation.
• Multi-signal (2.4k questions): Requires integrating two or more signals (e.g., coupling between activity and resting heart rate).
• Literature-grounded questions: Derived from 11 anchor papers (peer-reviewed, observational, full-text verified, supported by at least two independent same-direction studies). Each relationship is recorded as a structured card with DOI/PMID, thresholds, and explicit limitations. No study-specific cut-offs are imported.
• Population-grounded questions: Patterns mined directly from the cohort (28-day windows) and promoted only after passing statistical-consistency gates: effect size (strong correlation |ρ|≥0.5, winner margin ≥0.15), robustness (bootstrap resampling, leave-one-out, dead-band rejection), and authenticity (cross-user null test with FDR ≤ 0.20).

How the paper uses the data The benchmark is used for evaluation only; no training split is defined. Models are presented with a user’s longitudinal wearable trajectory (and optionally blood panel and demographics) and must answer the multiple-choice question. The deterministic ground truth is computed from the same measurements, enabling reproducible scoring.

Processing details

• Discover-then-label approach: A reasoning objective is first identified from literature or population patterns. Then a deterministic program computes the answer on each user’s real measurements.
• Computation primitives: Reusable operations (e.g., lagged correlation, threshold flags) are composed into an explicit computation graph, guaranteeing identical inputs yield identical answers.
• Answer construction: For literature-grounded questions, personalized baselines (e.g., 1.5 SD above 28-day mean) are used; users with undefined or ambiguous criteria are discarded. For population-grounded questions, the answer follows the discovered relationship in the user’s own trajectory.
• Distractor construction: For data reasoning, distractors include opposite trends, adjacent windows, or incorrect magnitudes drawn from the cohort distribution. For health reasoning, distractors are clinically plausible alternative interpretations inconsistent with the individual’s measurements. All options are verified so that only the gold answer satisfies the computation.
• Validation: Computational validity is ensured by deterministic labeling and statistical gates. Quality control includes duplicate removal, answer balancing, distractor verification, and shortcut auditing (e.g., removing predictable metric pairs like active burn with steps, and revising generation procedures to prevent models from answering without the time series).

Method

The authors construct the WearableQA benchmark through a dual grounding strategy that ensures every question is anchored either in published biomedical evidence or in statistically validated patterns discovered directly from a large wearable cohort. This design reconciles two goals: probing clinically meaningful reasoning and reflecting physiological behaviors that genuinely appear in real-world recordings. For both sources, the answer is not asserted a priori but re-derived from the underlying user measurements, keeping the benchmark faithful to the population and resistant to memorization.

Literature-grounded questions draw from a curated pool of peer-reviewed consumer-wearable studies. Only observational studies where both the wearable exposure and the clinical outcome are computable on the cohort are retained, yielding a set of anchor papers read at full-text depth. To guard against spurious findings, each retained relationship must be supported by at least two independent, same-direction studies verified against primary records. Each relationship is recorded as a structured card containing identifiers, verbatim thresholds, cohort details, and an explicit statement of what the source does not establish, preventing over-claiming. Study-specific cutoffs are deliberately not imported, as they rarely transfer across populations.

Population-grounded questions mine physiological patterns directly from the cohort, targeting a 28-day window to surface temporal trends, anomalous episodes, recovery dynamics, cross-signal relationships, and higher-order multi-signal interactions. A candidate pattern is promoted to a question only after passing a sequence of statistical-consistency gates across three axes. First, effect size requires a strong, well-separated target: a coupling must exceed a minimum correlation magnitude $| \rho | \geq 0.5$∣ρ∣≥0.5 with consistent sign in both halves of the window, and for comparative questions the winning option must beat the runner-up by a margin of at least 0.15. Second, robustness demands stability under perturbation: the labeled outcome must persist under bootstrap resampling of days (reproducing in ≥ 80% of 30 resamples), under leave-one-out removal of any single day, and across a range of baseline-window choices, with near-boundary cases inside a dead-band rejected. Third, authenticity is enforced via a cross-user null test: each statistic is recomputed on deliberately mismatched user pairings, and the ratio of null to observed prevalence is used as an empirical false-discovery rate; a pattern is retained only at $\mathrm{FDR} \leq 0.20$FDR≤0.20 (positive predictive value ≥ 80%). Together, literature-grounded and population-grounded questions provide complementary coverage, pairing established clinical knowledge with emergent physiological behavior from real-world data.

For labeling, the authors adopt a discover-then-label approach. A reasoning objective is first discovered from either a literature finding or a population-grounded pattern. Its answer is then determined by executing a deterministic program over the individual’s real measurements. The reasoning is expressed as a composition of reusable computation primitives drawn from a shared library. For example, a strongest-pair question uses lagged_correlation(max_lag=3) and argmax_pair over lags of up to ±3 days; a blood-state question applies threshold_flags to biomarkers and aggregates the flags. Each instance is represented as an explicit computation graph, guaranteeing that identical inputs always yield identical answers and enabling fully reproducible evaluation.

The selected primitives are executed on each user’s longitudinal wearable trajectory. For literature-grounded questions, the computation instantiates the published physiological relationship using personalized baselines and cohort-relative statistics (e.g., resting heart rate elevation ≥ 1.5 standard deviations above the user’s recent 28-day mean). A scenario is admitted only when the data deterministically satisfies the criterion; ambiguous or undefined cases are discarded. For population-grounded questions, the answer follows directly from the discovered relationship in the user’s own trajectory. In both cases, the ground truth is computed entirely from the underlying measurements.

Distractors are constructed differently for the two reasoning types. For data-reasoning questions, each template defines a distractor category (e.g., opposite trend direction, adjacent time window, or an incorrect magnitude drawn from the cohort distribution). The same executable computation evaluates all ten options, and an item is retained only when the gold option uniquely satisfies it. For health-reasoning questions, distractors are derived from the grounded finding itself: clinically plausible but incorrect interpretations of the same physiological relationship that are inconsistent with the individual’s measurements.

Experiment

The evaluation assesses 14 large language models on WearableQA, a benchmark of 4,084 multiple-choice questions derived from real-world wearable and health data of 200 users, spanning data versus health reasoning and single versus cross-signal complexity. Performance varies widely, with the best proprietary model reaching 72.9% accuracy, while data reasoning and cross-signal tasks remain particularly challenging for most systems. Chain-of-thought prompting reduces positional bias and improves accuracy, especially for proprietary models, and providing Python tool access yields substantial gains, indicating that explicit computation helps bridge the gap between textual reasoning and quantitative analysis of longitudinal health signals.

Proprietary models substantially outperform open-source alternatives on WearableQA, with Gemini 3.1 Pro achieving the highest overall accuracy and a wide lead over the next-best model. Most models find health reasoning easier than data reasoning, but Gemini 3.1 Pro uniquely reverses this trend, excelling on data reasoning. Single-signal questions are generally easier than cross-signal ones for proprietary models, yet Gemini 3.1 Pro performs comparably on both. Gemini 3.1 Pro leads all models with 72.9% accuracy, outpacing the second-best proprietary model by more than 12 percentage points. Data reasoning is consistently harder than health reasoning for most models, with gaps exceeding 25 points for some, while Gemini 3.1 Pro is the only model to achieve higher data reasoning accuracy. Cross-signal questions reduce performance for most proprietary models, but Gemini 3.1 Pro shows nearly identical accuracy on single-signal and cross-signal subsets. The strongest open-source model, Gemma-4-26B-A4B, reaches 42.5% accuracy, trailing the top proprietary model by over 30 points.

When assessing signal associations, several models perform substantially better on definitional pairs, which are answerable from prior knowledge, than on empirical pairs that require data-driven inference, even though both sets share comparable relationship strength. For instance, Gemini-2.5-Pro, GPT-5.4, and GPT-4o show large gaps exceeding 35 points, while Claude Opus 4.6, Claude Sonnet 4, and Gemini 3.1 Pro exhibit only minor differences of about 4 points. This pattern reveals that some models rely heavily on familiar signal relationships rather than recovering user-specific associations from longitudinal measurements. Gemini-2.5-Pro, GPT-5.4, and GPT-4o show large performance gaps favoring definitional pairs, indicating a strong reliance on prior knowledge. Claude Opus 4.6, Claude Sonnet 4, and Gemini 3.1 Pro perform similarly on both pair types, suggesting they can infer empirical relationships from data almost as effectively as definitional ones.

Textual representations of time-series data yield comparable moderate accuracy, whereas visual plots cause a large drop in performance. Providing summary statistics offers a small improvement, but the most substantial gain comes from agentic Python tool access, which boosts accuracy by over 20 points. Allowing the model to select among multiple representations does not add further benefit. Row-wise, column-wise, CSV, and Markdown formats all achieve similar overall accuracy, with CSV and Markdown augmented with statistics performing slightly best. Visual representations, whether separate per-signal plots or range-grouped plots, reduce overall accuracy by more than 14 points compared to text baselines, and adding charts to CSV does not recover the loss. Enabling Python tool access in an agentic setting improves overall accuracy from the row-wise baseline by a large margin, but multi-view representation selection provides no additional gain over fixed representation.

Ablating the longitudinal history sharply reduces performance on history-dependent questions, while window-scoped accuracy remains largely unchanged. Removing all wearable time series causes a catastrophic drop across all question types, confirming that the model grounds answers in the provided measurements rather than relying on demographic or cohort priors. Removing only the history context cuts history-dependent accuracy by nearly half, while window-scoped accuracy stays stable. Withholding all time-series inputs collapses overall accuracy to 17.3% and window-scoped accuracy to 13.7%, demonstrating the model's dependence on the wearable data.

The experiments evaluate large language models on WearableQA, a benchmark for reasoning over longitudinal wearable sensor data, comparing proprietary and open-source models, signal association types, data representations, and the role of input context. Proprietary models substantially outperform open-source ones, with Gemini 3.1 Pro uniquely reversing the common trend of health reasoning being easier than data reasoning and handling cross-signal questions as well as single-signal ones. Analysis of signal associations shows that some models rely heavily on prior knowledge for definitional pairs while others infer empirical relationships from data, and agentic Python tool access provides the largest accuracy gain among data representations. Ablation studies confirm that models ground their answers in the provided time-series data, as removing history sharply reduces history-dependent accuracy and withholding all wearable data causes a catastrophic performance collapse.