Gesture Quest Web

Gesture Quest Web is a WebGL port of Gesture Quest, rebuilt to run the ASL hand-tracking game entirely inside a browser. The client I'd worked with on the original reached out and asked me to bring the game to the web, both to make it far more accessible and to make it easier to gather hand-landmark and motor-skill progression data at scale. I took it on solo.

Note: Happy to walk through the full-stack source and the game project, or share them on request.

Where Gesture Quest left off

Before diving in, a quick recap of what we were working with. Gesture Quest shipped as two separate programs: the game itself, built in Unity, and a tracker program, built as a Windows executable (.exe) using Nuitka to compile the Python tracker into a standalone binary. That executable, along with the model and label-index files, lived in the same directory as the Unity build, and Unity connected to it to receive the recognized-gesture data. There were good reasons for that architecture at the time, trust me.

The problem: none of that works on the web. And shipping a separate tracker executable compiled for every operating system under the sun would have been a maintenance nightmare. This one called for a serious renovation.

The renovation: a browser-native tracker

After a good amount of research into the best way to approach this, I settled on a third-party component called react-unity-webgl to act as the bridge between a WebGL Unity build and a tracker module running directly in the browser. With the stack spec'd out, I started stripping out and rewriting the tracker, moving it from Python to TypeScript. Since I was now in React, I also got to break up the sprawling main.py into focused hooks in separate .ts files, split along the stages a frame passes through from a hand being recognized to a sign or letter coming out: a useWebcam hook, a useHandLandmarker hook, and a useSignClassifier hook.

Next came the model. I tried to convert our existing model straight to TensorFlow.js, but converting directly from both the .keras and .tflite files gave me nothing but trouble. So I rewrote the trainer in JavaScript and retrained a fresh .tfjs model on the enormous pile of training data my former teammates had collected. Their hands, once again, donated to science.

Wiring Unity to the browser

Back in Unity, I ripped out the old UDP connection code and replaced it with the react-unity-webgl bridge, setting up functions that could be called in both directions between the React frontend and the game. The React side could now stream landmarks and hand position and rotation data into the game, and the game could call back into React to do things like open a camera view or go fullscreen from an in-game button. As a nice side effect, keeping the raw webcam feed in React meant the client never had to touch sensitive data like a player's face or location. The game only ever sees abstract landmarks.

Of course, nothing is ever quite that simple.

A game that streams itself

When I first built the Unity WebGL target, the browser refused to show the game properly. Inspecting it, I found the build was simply too large for a browser tab: the standalone version loaded every asset up front, from the first frame to the last. Since the standalone build was already doing some dynamic loading, and since we needed a server for the landmark data anyway, I asked the obvious question: why not let that same server host the assets and own the game's configuration too?

So I set up a Django backend with API endpoints that serve config .json files and the Unity asset bundles (.unity3d files, though in hindsight I should have used Addressables) for the game to load on demand. The WebGL build became a thin shell: on load, it makes a simple API request for its config and the matching assets, then pulls them into the game. That slimmed the build dramatically, and it left the client the flexibility to add new letters entirely from the server, without ever rebuilding the game.

Trimming memory and shipping it

Streaming the assets solved the size problem but surfaced two more: loading still took too long, and holding every asset in memory ate more of the browser's budget than I liked. So I added explicit loading and releasing of resources in the WebGL build, tucked neatly behind stage changes and scene transitions, and made sure only the assets needed next stayed resident. That broke one enormous load into a series of small ones and kept memory lean, which meant smoother gameplay and more headroom for the research data collection.

With the game working end to end between the WebGL build and the full-stack app, I deployed it on AWS, using S3 and EC2 for the React and TypeScript frontend and the Django backend. I also set up lightweight deployment workflows to make future updates painless: an automatic client-side deploy on every push to GitHub, plus a shell script that ships server-side changes with a single word typed at the prompt (deploy).

Research Quest

With the game live on the web, I turned to the researchers themselves and built a tool I called Research Quest: a modified build of the game that reads extra researcher-defined configs, a GUI dashboard that lets researchers generate those configs and set up different experiments without touching code, and a database that automatically pulls collected data off the server for analysis. In practice that meant maintaining several tailored branches of the game to match the experiments they wanted to run. Once those branches were done, I took on a different gig and handed the remaining data-collection and deployment work off. Before I did, I documented everything: the development guidelines, the system architecture, and the deployment steps, so whoever picked it up next could carry the work forward without me.

Reflection

This project was a marathon of full-stack problem-solving, and I did the whole thing on my own: the tracker rewrite, the model retraining, the Unity WebGL work, the backend, the deployment, and the researcher tooling. I'm genuinely proud of how much ground one person covered, and of a design that turned a rigid two-program desktop app into a flexible, server-driven web experience. If there's a lesson to carry forward, it's less about the code than about scope and handoff: building something this broad solo is deeply rewarding, and documenting relentlessly so the work can outlive your involvement is what makes it last.