Case study

Kaka: a desktop assistant that acts, but asks first

A personal AI assistant for Windows that I talk to in Hinglish or Bengali (Banglish). It answers by voice, shows a HUD and actually does things on my PC.

89-second narrated walkthrough made from Kaka's real screens (AI voiceover, not a live screen recording).

The problem

I wanted an assistant that does more than chat: open apps, make calls, record the screen and answer in my own mix of Hindi and English, without leaving my keyboard. And it must never act on something risky without my OK.

Architecture

Shell: Python app with a PyQt6 HUD (a full reactor-core view and a minimal particle sphere).
Voice: Gemini Live for hearing and speaking, with a fallback text pipeline.
Tools: 60 tool definitions across 55 action modules: apps, Spotify, VS Code, reminders, notes, timers, WhatsApp and Discord calls, GPU screen recording, file tools, deep research.
Remote: my own Discord bot calls the same tools, so phone and voice share one brain.
Presence and security: OpenCV face greeting, and a thief-alarm photo sent to my phone if someone else opens the PC.
Resilience: a chain of 7 AI providers (Gemini, Groq, Mistral, NVIDIA NIM, OpenRouter, Cohere, Cloudflare/SambaNova) plus custom slots, so one busy provider never stops it.
Permission boundary: before any send, delete, buy, post or install, Kaka asks first.
Shipping: a Setup installer with Repair/Update, so fixes reach my PC without a reinstall.

By the numbers

60tool definitions
55action modules
7AI providers in the chain
~1 svoice reply
163Python files

The hardest bug

After I added WhatsApp calls, Kaka kept saying it could not find apps it had just opened. The cause was not UI Automation, it was Kaka itself: it treated every process it launched as “itself” and hid them from its own detection.

Fix: I separated Kaka's own process tree from apps it only launched, then added a check that the right window is focused and the text is really in the box before it asks me to confirm a send.

What I learned

Measure before guessing. A timing log showed voice was fine (about 1 second) but the WhatsApp call took about 80 seconds because every step re-read the whole UI tree. A small targeted read of just the header fixed the slow part.

Screens

Full HUD: reactor core, system monitor, activity log

Code and setup

Source and install steps are in the repository README: github.com/abhinaba-chanda/Kaka (CC BY-NC-SA 4.0).

Status: the features above run on my own Windows PC. It needs your own free API keys to run elsewhere.