Real-Time Hand Gesture Gamepad

Hand landmarks -> heuristic gesture classifier -> Unix socket -> C process that fires real macOS keyboard events. Four gestures act as four gamepad buttons.

Gesture	Action	Key
open palm (4 fingers extended)	accelerate	Up arrow
fist (0 fingers extended)	brake	Down arrow
point left (only index, tip left of wrist)	steer left	Left arrow
point right (only index, tip right of wrist)	steer right	Right arrow

Architecture

producer.py                         keybridge.c
  landmark source                     Unix socket client
  (live webcam+MediaPipe,     JSON     reads NDJSON lines
   or synthetic replay)  --> socket -->  tracks gesture changes
  gesture_classify.py                  CGEventCreateKeyboardEvent
  (shared, real code path)             + CGEventPost (keyDown/keyUp)

producer.py binds a Unix domain socket, waits for keybridge to connect, then for every frame runs gesture_classify.classify() on a 21-point hand landmark list and sends {"frame", "ts", "gesture", "confidence"}\n. The classifier is the same function regardless of where the landmarks came from. Only the frame source changes between live and replay.

keybridge.c connects to that socket, reads lines, maps gesture to a key, and posts a keyDown when a gesture starts and a keyUp when it ends (so a held gesture behaves like a held button, not a key repeated every frame). Every attempted key press is logged regardless of whether the OS actually delivered it.

Build & run

# 1. build the C consumer
make

# 2. start the producer (binds the socket, then waits for a consumer)
.venv10projects/bin/python producer.py --source replay

# 3. in another terminal, start the consumer
./keybridge

producer.py --source live tries the webcam via MediaPipe’s HandLandmarker task first, and automatically falls back to --source replay if the camera can’t be opened (no device, no permission), so the same command works whether or not a camera is available. --source replay (the default) skips the camera entirely and streams a hand-crafted 21-landmark sequence that cycles through all four gestures, so the whole pipeline is runnable with zero external permissions.

Useful flags: --fps, --hold (frames per gesture in replay), --cycles, --socket (defaults to /tmp/hand_gesture_gamepad.sock).

Self-checks: python gesture_classify.py asserts the classifier against hand-built landmark sets for each gesture.

Captured run (synthetic replay, real producer -> real consumer)

Producer (producer.py --source replay --fps 15 --hold 8 --cycles 2):

[producer] listening on /tmp/hand_gesture_gamepad.sock, waiting for consumer to connect...
[producer] consumer connected, streaming gestures
[producer] frame=000 gesture=open_palm    conf=0.90
...
[producer] frame=063 gesture=point_right  conf=0.85
[producer] done, 64 frames sent

Consumer (./keybridge), same run:

[keybridge] connecting to /tmp/hand_gesture_gamepad.sock ...
[keybridge] connected, reading gesture events
>>> KEYDOWN key=UpArrow    (gesture=open_palm, action=accelerate)
    frame=001 gesture=open_palm    (holding UpArrow)
    ...
>>> KEYUP   key=UpArrow    (gesture=open_palm, action=accelerate)
>>> KEYDOWN key=DownArrow  (gesture=fist, action=brake)
    ...
>>> KEYUP   key=DownArrow  (gesture=fist, action=brake)
>>> KEYDOWN key=LeftArrow  (gesture=point_left, action=steer_left)
    ...
>>> KEYUP   key=LeftArrow  (gesture=point_left, action=steer_left)
>>> KEYDOWN key=RightArrow (gesture=point_right, action=steer_right)
    ...
>>> KEYUP   key=RightArrow (gesture=point_right, action=steer_right)
[keybridge] stream closed. 64 events, 8 key presses.

(Full logs cycle through the four gestures twice, 64 frames / 8 key transitions total. The excerpt above is trimmed for length.)

--source live was also exercised in this sandbox: OpenCV reported not authorized to capture video, producer.py caught that and fell back to replay automatically, and the consumer still fired the correct keyDown/keyUp sequence. This proves the fallback path, not just the demo path.

Simplified vs full spec

• No live camera / no Accessibility permission in this sandbox. The demo runs on synthetic landmark data and CGEventPost calls that are logged whether or not the OS actually delivers them (no Accessibility grant here). Add when: running on a Mac with a camera and this app added to System Settings > Privacy & Security > Accessibility and Camera.
• Heuristic finger-extension classifier instead of a trained CNN. gesture_classify.py compares fingertip vs. PIP-joint y-coordinates, which is standard practice for landmark-based gesture recognition and needs no training data. Add when: gestures beyond simple finger counting/pointing are needed (e.g. distinguishing OK-sign from thumbs-up): train a small sklearn KNN/SVM on landmark feature vectors.
• User-space CGEventPost instead of a kernel HID driver. A true virtual HID device needs a signed DriverKit extension (Apple developer account, notarization, kernel entitlements), infeasible in a sandbox. Add when: an unsigned/no-permission-dialog input path is required: ship a DriverKit HID driver extension instead.

Support

Submit issues, questions, or bug reports to the GitHub issue tracker: https://github.com/ADM1SH/hand-gesture-gamepad/issues

Roadmap

• Done: Core architecture and baseline implementation.
• Done: Functional verification and test coverage.
• To do: Add two-hand gesture chord combinations
• To do: Support dynamic sensitivity and deadzone calibration

Contributing

Contributions are welcome.

1. Fork the repository.
2. Create a feature branch: git checkout -b feature/improvement.
3. Commit your changes: git commit -m "feat: enhance functionality".
4. Push to the branch: git push origin feature/improvement.
5. Open a Pull Request.

Authors and Acknowledgment

• Adam Anwar (ADM1SH) - Lead architect and developer.
• Authored by Adam Anwar. Uses Google MediaPipe Hands and macOS CoreGraphics event taps.

License

Licensed under the MIT License. See LICENSE for details.

Project Status

Real-time computer vision gamepad bridge.