AI Controller — Run Linux with an Xbox Controller

What it does

🎙️ Push-to-Talk Dictation

Hold Right Trigger, speak, release. Groq Whisper transcribes your voice and types it into the focused window — like a walkie-talkie for your desktop.

⌨️ Floating Keyboard

Press View for a GTK on-screen keyboard that sends keystrokes without stealing focus. Style modes and pinned snippets included.

🖱️ Mouse + Scroll

Left stick moves the cursor. Right stick scrolls. No mouse required.

🗺️ HUD Legend

Press Guide to show your current button mappings as a floating overlay. Never forget what a button does.

🔄 Smart Profiles

AntiMicroX profiles auto-switch between desktop, browser, and YouTube TV based on the focused window.

🔊 Voice Responses

The FastAPI voice bridge can speak answers back with edge-tts when you ask it something.

Who it's for

Couch or bed computing — when a keyboard and mouse are out of reach.
Limited mobility — one-handed, low-dexterity, or limited reach.
Voice-first users — people who prefer dictation over typing.
HTPC / living-room PCs — run Linux from the sofa with a controller.

This is not "download this instead of that." AI Controller does something most people don't have at all: a controller-native desktop experience with voice control.

Get started in 30 seconds

git clone https://github.com/ebey317/ai-controller.git
cd ai-controller
bash install.sh

Plug in your Xbox controller, open the launcher, hold Right Trigger, and talk.

Tech stack

Layer	Technology	Purpose
Button mapping	AntiMicroX	Controller → keyboard/mouse events
Push-to-talk	Python + pynput	F13 hotkey listener, audio capture
Voice bridge	FastAPI + uvicorn	STT + TTS on :8002
On-screen keyboard	Python + GTK3	Floating keyboard, xdotool keystrokes
HUD legend	Python + GTK3	Button-map overlay
Profile switching	Bash + xdotool	Window-focus watcher
TTS	edge-tts	Cloud text-to-speech
Service management	systemd user units	Start/stop/autostart

Watch it work

Live screenshots and demo GIFs are in the repo under media/demo/.

Screenshot 1 — Terminal + floating keyboard + controller legend in action.

Demo GIFs — Push-to-talk, keyboard toggle, scrolling, and profile switching.