Computer vision · Embedded
Hands-free control of a wall-projected thinking surface, so the interface moves onto the wall and the laptop stops being the thing that keeps me in a chair.
I already work standing. I pace when I think, I run meetings in a room rather than at a desk, and I keep a whiteboard four metres wide at home because ideas make more sense to me laid out next to each other than stacked in a window. The laptop is the part that does not fit. It is where the power and the compute live, and it is also the thing that pulls me back into a chair.
So the goal is not a gadget. It is to keep the laptop as the engine and move the interface onto the wall: a projected surface I can walk up to, point at and rearrange, closer to working a physical board than driving an application.
On scope, since this is easy to misread: it is not about generating content. Voice and AI agents already do that well. This is the opposite problem, an external surface for thinking visually, navigated by hand.
Hand tracking runs on MediaPipe through an 850nm IR camera with automatic IR-cut switching, which turned out to matter more than the resolution did. A normal webcam degrades badly in a dark projection room. The IR camera stays smooth. A Wiimote handles coarse navigation and clicking through a keyboard-emulation layer.
The next version replaces the Wiimote with a wrist-worn band built on an ESP32 and a BNO055 IMU, streaming orientation over Bluetooth. It is worth being precise about what it replaces. The IMU takes the Wiimote's slot, a worn controller with motion and buttons. It does not replace the camera, which still supplies absolute position and always will, because an IMU drifts. The two are complementary rather than competing, and the Wiimote keeps its own job driving the music visualizer.
None of this is plug and play. It needs firmware on the ESP32 to read the sensor and a receiver on the laptop to interpret it, which is roughly the same amount of work the camera pipeline took.
The Wiimote's personalization lived in a config screen and could only emit what that screen offered. Here the receiver script is the personalization layer, so a gesture maps to whatever can be expressed in code: press a key, move the cursor, call an API, run a script. The gesture vocabulary is mine to invent rather than to pick from a dropdown.
That also makes profiles possible. The same band means different things depending on the mode: flick to advance a slide when presenting, flick to move between atoms inside a molecule page, flick to switch windows at the desktop. One device, several vocabularies, switched deliberately.
The wrist controller is the interesting part to build, which is exactly why it deserves scrutiny. Most of the actual benefit, being able to work while moving, comes from the projector and a voice layer, and I am most of the way there already. The IMU is a precision accent on top of that, not the thing that unlocks it. Naming that keeps the project honest about which parts are useful and which parts are just fun to solder.
Embedded firmware, computer vision, sensor fusion and interaction design in one build. Evidence that I work where software meets physical hardware rather than only at the API, and that I own the whole interpretation stack rather than configuring someone else's. It is also the most personal thing here, a tool shaped to fit how my own mind works.