Gesture Quest

Gesture Quest is a bridge between technology and accessibility: an educational game that makes learning sign language engaging and intuitive. Built by Team Questure, six graduate students at CMU's Entertainment Technology Center, it combines machine learning and computer vision into an interactive experience. As one of the programmers, I owned the external input system: computer vision and ML for precise hand detection and gesture recognition, processed in real time and passed to the Unity game engine. Gesture Quest has since been adopted by the CMU Physical Intelligence Lab to study how humans master complex movements.

Note: Want to play in your favorite browser instead of downloading it? There's a WebGL version at Gesture Quest Web.

Real-time hand tracking with MediaPipe

I used Google MediaPipe for real-time hand-landmark detection, chosen for its fast inference and efficiency. Its tracking runs on two models: a palm detection model (a Single-Shot Detector that finds the hand in each webcam frame and crops it) and a hand-landmark model (a CNN regression model that predicts handedness and precise landmark positions). I paired this with OpenCV for landmark collection and preprocessing, feeding the data into our gesture-classification model.

Data collection and preprocessing for ASL

With camera processing, hand detection, and landmark extraction in place, I built a data-collection module for the ASL alphabet. To optimize performance while keeping accuracy, I used only 2D landmarks: fewer features, still enough precision for the alphabet. The pipeline extracted, normalized, and structured the landmark data: I normalized relative to frame size to account for hand position and scale, flattened it into a structured format, and logged it to CSV for training.

To improve robustness, I packaged the collection module into a standalone tool and set it up on every teammate's machine so we could train on a variety of hand shapes, sizes, and signing styles (the whole team essentially donated their hands to science). We validated the dataset with a subject-matter expert too: garbage in, garbage out, so we made sure the data going in was clean.

Model training and integration

I trained the static-signs model in TensorFlow with a simple, effective neural network. I carefully split the data into training, validation, and test sets and tuned layers and epochs. The aim wasn't just accuracy but generalization across different hands. To guard against overfitting I added early-stopping callbacks, set random seeds for reproducibility, and monitored training with TensorBoard.

For integration, we prioritized smooth gameplay over absolute precision, a common real-time trade-off. I quantized the trained model to TensorFlow Lite for fast on-device inference, keeping the game fluid and responsive.

Bringing it into Unity

To connect the model to Unity, we broadcast data over UDP: the Python script handles the webcam feed, detects the hand, runs inference, and sends results to the game. Given the timeline, it was the quickest solution. With more time and clearer client data requirements, a dedicated server or Dockerized setup would have offered a more scalable, data-friendly foundation.

Two more to go: dynamic signs J and Z

The last two letters, J and Z, refuse to sit still: both need motion, so I trained a separate model for dynamic signs. I designed a representation where a queue of 21 frames captures each landmark's movement relative to the first frame. Subtle finger differences (using the ring finger instead of the pinky for J) caused misclassifications, so I added a validation step: treat the first frame as a static sign to confirm the correct hand shape before analyzing the motion. For this time-series data I used an LSTM, which captured the sequential nature of the gestures well enough for the two signs.

Reflection

This was my first time working directly with a research-facility client, and it taught me a lot about both the technical and collaborative sides of development: adapting quickly to client needs, managing time under tight constraints, and delivering something both functional and scalable. With more time I'd have pushed further on model performance, a more sophisticated server architecture, and a broader dataset. I'm proud of the outcome, and clear on what I'd do better next time.