Our system maps OpenXR hand-tracking input to gesture predictions with a Transformer encoder.
OpenXR hand tracking + Machine Learning (Transformers) for real-time VR gesture recognition
Our system maps OpenXR hand-tracking input to gesture predictions with a Transformer encoder.
We present a real-time hand gesture recognition system built on hand-tracking data captured through the OpenXR standard in Unity. Unlike video- or electromyography-based approaches, our method operates directly on 3D hand-joint positions and wrist rotation angles, which are embedded and processed by a Transformer encoder that models temporal dependencies within short frame windows. Normalizing joints relative to the palm makes recognition robust to hand orientation and size. We also outline how the approach can be extended toward detecting the flow of movement — the transitions between gestures — as future work.
Reported on a custom OpenXR-collected dataset. Public-benchmark (SHREC'17) evaluation is in progress.
Per-gesture accuracy before and after broadening coverage of hand rotations, sizes, and joint distances.
| Method | Venue | 14-class | 28-class |
|---|---|---|---|
| ST-GCN | AAAI 2018 | 92.7 | 87.7 |
| Shift-GCN | CVPR 2020 | 95.5 | 89.4 |
| CTR-GCN | ICCV 2021 | 96.1 | 94.4 |
| TD-GCN | IEEE TMM 2023 | 97.02 | 95.36 |
| DSTSA-GCN | 2025 | 97.74 | 95.37 |
| Our method (this work) | OpenXR + Transformer | TBD | TBD |
Published accuracies of representative skeleton-based methods, as compiled by Cui et al. (2025). Our SHREC'17 numbers are pending and will replace the TBD row.
@inproceedings{rezayani2027openxr,
title = {Real-Time Hand Gesture Recognition for OpenXR
Using Transformer-Based Machine Learning},
author = {Rezayani, Salar and Butler, Russell},
booktitle = {Proc. IEEE Int. Conf. on AI and Extended/Virtual Reality (AIxVR)},
year = {2027},
note = {To appear / under review}
}