One-sentence Summary

A student from Mendon High School proposes a multimodal eye-to-hand prediction framework that reformulates the task as a machine translation problem solved with a sequence-to-sequence model, and demonstrates that prediction accuracy differs significantly between autistic and typically developing children on VR-collected data, providing quantitative evidence for restricted and repetitive behaviors in ASD.

Key Contributions

• Introduces a multimodal behavioral analysis approach that links eye movement and hand motion data collected from a VR system, extending prior methods that often focus on a single behavioral modality.
• Frames eye-to-hand prediction as a machine translation problem and uses a sequence-to-sequence model to translate noisy, irregular eye movement sequences into hand motion sequences, yielding more accurate next-hand-location prediction than alignment-based alternatives.
• Uses a VR gameplay dataset of autistic and typically developing children to show high prediction accuracy and a significant prediction-accuracy difference between groups, with more accurate hand-motion prediction for autistic participants providing quantitative evidence related to restricted and repetitive behaviors.

Introduction

Autism Spectrum Disorder is a developmental condition that can affect learning, social interaction, communication, and behavior, and researchers are increasingly using virtual reality to collect fine-grained behavioral data from children. Traditional diagnosis often relies on questionnaires, which may miss subtle sensory and behavioral patterns, while existing data-driven studies usually analyze only one behavioral signal at a time. The authors note that multimodal behavioral data is challenging because it is noisy and poorly aligned across sensing channels, making it hard to model relationships between signals such as eye movement and hand motion. To address this, the authors leverage an analogy to machine translation and propose a behavioral data translation framework that predicts hand motion sequences from eye movement data. Their results show that this approach outperforms alternative designs and reveals clearer behavioral differences between autistic and typically developing children, with autistic hand motion being more accurately predictable, offering quantitative evidence that may support later intervention and assessment.

Method

The authors formulate the analysis of multimodal eye and hand fixation data as a sequence to sequence translation problem. In this formulation, a temporally ordered gaze sequence is translated into a corresponding hand fixation sequence. The underlying data are collected from a virtual classroom interface that records attention related behavior through coordinated eye tracking and hand movement capture. The system observes fixation events over the whole task session. Each fixation is represented as a tuple:

where $x_i$xi​ and $y_i$yi​ denote the spatial location on a normalized unit screen, and $t_i^{s}$tis​ and $t_i^{e}$tie​ denote the fixation start time and end time. The spatial coordinates are continuous values inside the unit square:

The data collection environment is illustrated below. Refer to the figure for the multimodal interface used to produce the fixation sequences. The interface combines a task panel with background classroom activity and task related panels, providing an ecological setting in which gaze and hand behavior can be recorded as aligned temporal streams.

The first modeling design uses recurrent sequence to sequence models based on long short term memory or gated recurrent unit cells. The model takes the eye fixation sequence as the source sequence. To support the translation, the input representation includes both the spatial gaze positions and the fixation start and end times. Thus, the recurrence learns temporal context from the gaze trajectory and maps it into the target hand fixation sequence:

Conceptually, the recurrent encoder or recurrent processing pathway summarizes the evolving gaze sequence, while the sequence generation pathway emits the hand fixation sequence. The temporal features allow the model to distinguish brief and sustained fixations, which may have different consequences for subsequent hand movement.

The second design adds a spatial module. The screen is partitioned into functional regions, including a main task region, a facilitation region, and a reference region. For a selected region, the eye input is filtered to retain only fixations whose coordinates fall inside that region. A gated recurrent unit sequence to sequence model then predicts hand fixations that also lie inside the same region. This produces a region conditioned eye to hand mapping. The design restricts the learned translation to local gaze to movement relationships and enables analysis of how attention in a particular screen region governs corresponding hand behavior.

Experiment

Experiments trained recurrent models on fixed train-test splits and compared GRU and LSTM variants with different eye fixation inputs to predict hand fixation sequences, including a region-based GRU design. The results show that the GRU generally performs best and that predictions are substantially more reliable for autistic individuals than for typically developing individuals, especially in task-focused and reference regions. These findings suggest that gaze-hand coordination in autism is more regular and predictable, providing quantitative support for patterns associated with restricted and repetitive behaviors.

Across GRU and LSTM variants, prediction error is lower for autistic hand fixation sequences than for typically developing sequences, with statistically significant differences. The GRU models, especially the version using fixation location only, show the strongest overall performance in this table. Adding start and end time features does not consistently improve accuracy. All model variants perform better on autistic predictions than on typically developing predictions. GRU predictions generally achieve lower error than LSTM predictions. Using fixation location with start and end times does not yield the lowest error for either group.

Prediction errors differ across regions, with autistic individuals showing notably lower errors in the main task region. The facilitation region shows little difference between autistic and typically developing predictions, while the reference region reverses the pattern and shows lower errors for typically developing individuals. These trends suggest that predictive accuracy is highest for autistic behavior during the main task and that reference-area behavior may be more consistent in the typically developing group. Autistic behavior is predicted more accurately in the main task region. Autistic and typically developing errors are similar in the facilitation region. Typically developing predictions are more accurate in the reference region.

The experiments evaluate GRU and LSTM models for predicting fixation behavior in autistic and typically developing sequences. Overall, GRU models outperform LSTM variants, with fixation location alone yielding the strongest results and temporal features offering limited additional benefit. Region-level analyses further show that autistic behavior is predicted more accurately in the main task region, performance is comparable in the facilitation region, and typically developing behavior is predicted more accurately in the reference region.