One-sentence Summary

This study introduces a hierarchical convolutional neural network that pairs a lightweight gesture detector with a ResNeXt-101 classifier and employs a sliding window approach to enable real-time dynamic hand gesture detection and classification, achieving state-of-the-art offline classification accuracies of 94.04% and 83.82% for the depth modality on the EgoGesture and NVIDIA Dynamic Hand Gesture datasets while utilizing Levenshtein distance to evaluate detection consistency.

Key Contributions

• The paper introduces a hierarchical architecture that adapts offline convolutional neural networks for real-time operation through a sliding window mechanism. This design decouples gesture detection using a lightweight CNN from classification via a deep CNN to satisfy memory and power constraints while maintaining online efficiency.
• The study proposes Levenshtein distance as an evaluation metric to measure single-time gesture activations by simultaneously tracking misclassifications, duplicate detections, and missed events. This approach ensures that each performed gesture triggers exactly one recognition event, addressing the continuous stream processing requirement.
• Experiments on the EgoGesture and NVIDIA Dynamic Hand Gesture datasets demonstrate that a ResNeXt-101 classifier achieves state-of-the-art offline accuracies of 94.04% and 83.82% on the depth modality. Real-time evaluation confirms substantial early detections with performance comparable to offline operation, and the associated code and pretrained models are publicly released.

Introduction

Vision-based hand gesture recognition provides a natural, wearable-free interface for human-computer interaction, making it highly valuable for real-time applications. However, prior research typically prioritizes offline classification accuracy while overlooking critical deployment constraints such as strict memory and power budgets, unpredictable gesture boundaries, and the requirement for single-time activations. The authors address these limitations by introducing a hierarchical architecture that efficiently adapts offline convolutional neural networks for continuous video streams. By pairing a lightweight 3D CNN detector with a deeper classifier that activates only during recognized gestures, the system conserves computational resources. The pipeline further leverages a sliding window approach, weighted score filtering to resolve early classification ambiguity, and the Levenshtein distance metric to rigorously evaluate temporal detection, ultimately delivering state-of-the-art accuracy on standard benchmarks.

Dataset

• Dataset composition and sources: The authors utilize two multimodal datasets for egocentric hand gesture recognition. EgoGesture contains 83 static and dynamic gesture classes recorded across six indoor and outdoor environments. nvGesture provides 25 gesture classes designed for human-computer interfaces, captured in an indoor car simulator using multiple sensors and viewpoints.

• Subset details: EgoGesture follows a subject-based 3:1:1 split, yielding 1,239 training, 411 validation, and 431 testing videos. This corresponds to 14,416, 4,768, and 4,977 labeled gesture samples respectively. nvGesture consists of 1,532 weakly segmented videos that include non-gesture intervals. It is divided 7:3 into 1,050 training and 482 testing videos, with each test clip containing exactly one gesture.

• Data usage and training strategy: All models undergo initial pretraining on the Jester dataset before fine-tuning on the target datasets. For final test set evaluations, the authors combine the EgoGesture training and validation splits. Both datasets are evaluated offline for classification and detection, with depth-modality networks consistently selected over RGB for superior background suppression and hand motion isolation.

• Processing and frame handling: The authors experiment with varying input window sizes, testing frame counts from 8 to 32. While larger windows improve offline accuracy, they opt for an 8-frame window for real-time detection to more robustly capture gesture start and end points. Depth sensors are leveraged to filter out environmental motion, allowing the networks to concentrate on discriminative hand features. The nvGesture subset relies on weak temporal segmentation, meaning precise gesture boundaries are not enforced during model training.

Method

The authors leverage a two-model hierarchical architecture designed to enable real-time gesture recognition using state-of-the-art 3D CNN models efficiently. This framework operates through a sliding window mechanism applied to an incoming video stream, where a detector model acts as a switch to activate a classifier when a gesture is detected. As shown in the figure below, the workflow processes sequences of frames with different lengths: the detector operates on a shorter sequence of size $n$n, while the classifier processes a longer sequence of size $m$m, where $n \ll m$n≪m. The stride $s$s for the sliding window is consistent across both models, and the authors select $s=1$s=1 to ensure no gesture is missed, despite higher computational cost. The detector and classifier models are fed overlapping frames, with the detector queue positioned at the beginning of the classifier queue to allow immediate activation upon gesture onset. In addition to the detector and classifier, the system incorporates a post-processing module and a single-time activation mechanism to refine decisions.

The detector is designed to be robust, accurate in detecting true positives, and lightweight for continuous operation. It utilizes a ResNet-10 architecture constructed from ResNet blocks, as illustrated in the figure below, with minimal feature sizes to limit parameters to approximately 862K. To enhance recall and reduce false positives, the detector is trained with a weighted cross-entropy loss, assigning class weights of 1 for no gesture and 3 for gesture, which achieves high recall rates on the EgoGesture and nvGesture datasets. Post-processing is applied to the detector's output, where a counter tracks consecutive no-gesture predictions to determine when to deactivate the classifier. The classifier, in contrast, is not constrained by size and employs more complex 3D CNN architectures such as C3D and ResNeXt-101, both of which are pre-trained on the Jester dataset and fine-tuned on EgoGesture and nvGesture to mitigate overfitting and accelerate training.

The training process employs stochastic gradient descent with Nesterov momentum, weight decay, and data augmentation techniques, including random cropping, scaling, spatial elastic displacement, and temporal frame selection. Images are normalized to the 0-1 scale using dataset statistics, and during inference, center cropping is applied to maintain consistency. The post-processing module enhances the detector's reliability by maintaining a queue of recent raw softmax probabilities and applying filtering. Among average, exponentially-weighted average, and median filtering, median filtering is selected for its superior performance. The single-time activation mechanism ensures that only one decision is made per gesture, leveraging a two-level control strategy: early detection when confidence exceeds a threshold during the gesture's nucleus phase, and late detection when the classifier deactivates. Early detection is enabled by a sigmoid-like weight function applied to class scores, calculated as $w_j = \frac{1}{1 + \exp^{-0.2 \times (j - t)}}$wj​=1+exp−0.2×(j−t)1​, where $t = \left| \frac{\mu}{4 \times s} \right|$t=​4×sμ​​ and $\mu$μ is the mean gesture duration. This weighting emphasizes the nucleus part of gestures, reducing ambiguity from preparation phases. Confidence for early detection is determined by the difference between the top two weighted average scores, triggering a decision if it exceeds a threshold $\tau_{early}$τearly​. If no early detection occurs, a late detection is made when the highest score surpasses a fixed threshold $\tau_{late}$τlate​ of 0.15. Algorithm 1 formalizes this process, ensuring single-time activations by managing the active state and applying the described mechanisms.

Experiment

The proposed approach was evaluated on the EgoGesture and NVIDIA Dynamic Hand Gestures datasets to assess its gesture classification accuracy, early detection capabilities, and real-time processing efficiency. Experimental results indicate that the model achieves robust recognition performance while significantly outperforming uniform averaging through a weighted early detection strategy. By calibrating detection thresholds, the system effectively balances latency and accuracy, enabling substantially faster responses with only marginal performance trade-offs. Additionally, the architecture maintains high computational throughput, confirming its practical viability for real-time deployment.

The authors evaluate a two-model approach on a dataset using different modalities, comparing performance metrics such as recall, precision, and f1-score. The results show that depth-based features achieve higher performance compared to RGB-based features across all metrics. Depth-based features achieve higher recall, precision, and f1-score compared to RGB-based features. The f1-score for depth-based features is substantially higher than for RGB-based features. Precision and recall are both significantly better for depth-based features than for RGB-based features.

The authors compare the performance of different models and input frame lengths across RGB and depth modalities. Results show that increasing the number of input frames generally improves accuracy, and the ResNeXt-101 model achieves higher performance than C3D, particularly with depth input. Accuracy improves with longer input sequences for both models and modalities. ResNeXt-101 outperforms C3D across all input lengths and modalities. Depth modality consistently achieves higher accuracy than RGB, especially with ResNeXt-101.

The authors compare the performance of two deep learning architectures, ResNeXt-101 and ResNet-10, by analyzing their layer-wise output sizes and configurations. The results indicate that ResNeXt-101 produces higher-dimensional feature maps at multiple layers, suggesting a more complex and potentially more expressive model compared to ResNet-10. ResNeXt-101 generates higher-dimensional feature maps than ResNet-10 across multiple layers. ResNeXt-101 maintains higher feature dimensionality throughout the network compared to ResNet-10. The final layers of ResNeXt-101 produce significantly more features than ResNet-10, indicating a larger capacity model.

The authors evaluate the performance of their approach on two datasets, presenting results for different modalities in terms of recall, precision, and f1-score. The results show that the depth modality achieves higher scores across all metrics compared to the RGB modality. The depth modality outperforms the RGB modality in recall, precision, and f1-score. The depth modality achieves the highest f1-score among the evaluated modalities. The RGB modality shows lower performance compared to the depth modality across all evaluation metrics.

The authors evaluate the performance of a ResNet-10 model across different input frame counts and modalities, comparing RGB and Depth inputs. Results show that increasing the number of input frames improves accuracy for both modalities, with Depth consistently outperforming RGB. The highest accuracy is achieved with 24-frame input in the Depth modality. Increasing input frame count improves accuracy for both RGB and Depth modalities. Depth modality consistently achieves higher accuracy than RGB across all input frame counts. The highest accuracy is observed with 24-frame input in the Depth modality.

The experiments evaluate multiple deep learning architectures across RGB and depth modalities while varying input sequence lengths to assess their impact on recognition performance. Results consistently demonstrate that depth-based representations significantly outperform RGB inputs across all tested configurations and datasets. Furthermore, extending the input sequence length yields improved accuracy, and the ResNeXt-101 architecture proves more expressive and robust than both C3D and ResNet-10. Ultimately, the findings highlight the superior efficacy of depth modality and the benefits of increased model capacity for improving recognition performance.