Data and AI
Real-Time Hand Gesture Gamepad
A Python producer classifies 21-point hand landmarks into four gestures and streams them over a Unix socket to a C process that posts real macOS key events. A gesture holds its key down until the gesture ends, so it acts like a held gamepad button, and a synthetic replay mode runs the full pipeline with no camera.
Overview
Install
# 1. build the C consumer
make
# 2. start the producer (binds the socket, then waits for a consumer)
.venv10projects/bin/python producer.py --source replay
# 3. in another terminal, start the consumer
./keybridge
producer.py --source live tries the webcam via MediaPipe’s HandLandmarker
task first, and automatically falls back to --source replay if the camera
can’t be opened (no device, no permission), so the same command works
whether or not a camera is available. --source replay (the default) skips
the camera entirely and streams a hand-crafted 21-landmark sequence that
cycles through all four gestures, so the whole pipeline is runnable with
zero external permissions.
Useful flags: --fps, --hold (frames per gesture in replay), --cycles,
--socket (defaults to /tmp/hand_gesture_gamepad.sock).
Self-checks: python gesture_classify.py asserts the classifier against
hand-built landmark sets for each gesture.
Docs
The full documentation covers how it works, configuration and results.