// IRIS Desktop / Capabilities
This is the real inventory — every capability documented here is implemented and working. Voice, vision, sixty-plus tools, macros, gestures, proactive intelligence, and the guardrail pipeline that keeps it all accountable.
Voice system
IRIS runs a complete speech state machine — idle → listening → hearing → processing → speaking — with native audio capture, speaker verification, and streaming playback.
| Subsystem | What it does |
|---|---|
| Wake words | Hey IRIS Hi IRIS Okay IRIS Yo IRIS — enrolled owners can skip the wake word; strangers always need it. 60 seconds of silence puts her to sleep. |
| Speech-to-text | Whisper via the backend with echo cancellation and noise suppression, plus a native Rust/cpal capture path that bypasses WebView audio issues entirely. Speaker confidence scoring detects who's talking. |
| Text-to-speech | Streaming TTS over WebSocket → Socket.IO → gapless chunked playback with 20ms overlaps. Falls back to HTTP TTS automatically if streaming drops. |
| Voice activity detection | Frequency-domain thresholding with ~500ms silence detection, min/max recording bounds, and a post-speech cooldown so IRIS never hears her own voice. |
| Native audio engine | Rust-level capture with RMS-based VAD, echo reference tracking, per-device selection, and automatic default device detection. |
Voice commands
| Domain | Say things like |
|---|---|
| Apps | Open Chrome Close Notepad Launch VS Code |
| Web | Search for Python tutorials Go to github.com |
| Media | Play music Next track Volume up |
| System | Lock my computer Sleep Show desktop |
| Files | Read that file Clear my downloads |
| Display | Set brightness to 50% Dim the screen |
| Network | Turn off WiFi Enable Bluetooth |
| Mouse & keyboard | Click there Type hello world Press Ctrl+S |
| Search Gmail for invoice Download attachments | |
| Calendar | What's on my calendar? Schedule a meeting |
| Windows | Move to top right Jump to monitor 2 Focus mode |
| Weather | What's the weather? |
Tool registry
Every tool is registered in a typed registry with an assigned risk level, then executed through the safety pipeline below. Backend tools load dynamically at startup and refresh at runtime.
Guardrail system
Tools are categorized by danger level, and behavior adapts accordingly — from instant execution to simulated preview plus explicit human approval.
| Risk level | Example tools | Behavior |
|---|---|---|
| Low | open_url, web_search, type_text | Execute immediately |
| Medium | edit_file, safe run_command | Log, then execute |
| High | delete_file, risky run_command | Require explicit approval |
| Critical | delete_folder, shutdown-class actions | Approval + pre-flight simulation, always |
Before anything dangerous happens, IRIS previews the impact: affected files and folders, total size of data at risk, blocked protected paths, and reversibility assessment.
High and critical actions wait for a human yes. Denial cancels outright — no retries, no second guesses behind your back.
Timestamp, tool, risk level, your original command, understood intent, approval status, execution result, and conversation context — logged for every medium-plus action.
Screen annotation
IRIS draws directly on your display — cyan annotations that auto-fade after five seconds, targeted through a resolution-calibrated 10×10 grid (columns A–J, rows 1–10).
| Type | Example | Use it for |
|---|---|---|
| Circle | Circle that button | Highlight exactly the UI element she's acting on |
| Arrow | Point from here to there | Show relationships between two targets |
| Highlight | Highlight this area | Region emphasis with animated pulse |
| Text | Label this "click here" | Floating labels anchored to grid cells |
| Pointer | Show me where | Animated attention pointer |
The same grid powers click targeting and AI-directed actions — one calibrated coordinate system shared by vision, voice, and input. Debug overlay on Ctrl+Shift+G.
Spatial memory
Your entire work context — open apps and window positions — saved under a name and restored on command.
Snap everything into mission-ready arrangements across one or many monitors.
IRIS becomes fully click-through — visible but untouchable by your mouse — until you call her back.
The Gauntlet
Macros chain up to thirty-plus action types — apps, shell, git, media, input, windows, notifications, voice — behind a single voice trigger with exact, alias, and fuzzy matching. Define them in YAML:
name: deploy trigger: "deploy" aliases: ["ship it", "send it"] description: "Push code to production" steps: - action: run_command params: command: "git push" - action: notify params: title: "Deployed" body: "Code pushed successfully"
And with Watch-Me-Learn pattern detection, IRIS proposes macros herself: when she sees you repeat the same workflow three days running, she offers to automate it.
JARVIS mode
IRIS monitors system state, calendar, and error patterns — then speaks up only when it matters. Every trigger has configurable thresholds, cooldowns, quiet hours, and randomized natural phrasing.
| Trigger | Response | Cooldown |
|---|---|---|
| Battery < 20% | "Heads up — your battery is getting low." | 5 minutes |
| Battery < 10% | "Warning — battery is critically low!" | 5 minutes |
| Idle > 2 hours | "You've been at it a while. Maybe take a break?" | 30 minutes |
| Same error 3× | "I've noticed that happening repeatedly. Want help troubleshooting?" | 5 minutes |
| Meeting in 5 min | "You have '[meeting]' starting soon." | Per event |
| Email watch rules | Proactive notification when watched senders or subjects arrive | Per rule |
Presence & gesture control
MediaPipe face and hand tracking (21-point skeleton) runs locally at ~12 FPS with debounced triggers. Fourteen gestures map to twenty action types, with preset bundles for minimal, media, presentation, and accessibility setups.
| Gesture | Action | Hold time |
|---|---|---|
| 👋 Open palm | Start listening | 500 ms |
| 👍 Thumbs up | Confirm | 300 ms |
| 👎 Thumbs down | Cancel | 300 ms |
| 🤣 Shush | Interrupt speech instantly | — |
| ✋🏼 Point | Take screenshot | 800 ms |
| ✊ Fist | Mute toggle | 500 ms |
| ✨🏿 Swipes | Track skip / volume up / down | — |
| 🖐 Wave | Say hello | — |
Face detection adds presence awareness — she greets you when you sit down and tracks attention direction. Everything is opt-in and toggleable in settings.
Clipboard intelligence
Passive clipboard monitoring with priority-ordered pattern recognition suggests context-aware actions — and warns you before secrets leak somewhere they shouldn't.
AWS, OpenAI, Anthropic, GitHub keys; SSNs; credit cards; private keys. Detected secrets are masked and flagged with a paste-carefully warning.
UPS, FedEx, USPS, Amazon, DHL tracking numbers become one-click package tracking. Flight numbers surface live status. Airport codes resolve instantly.
URLs preview or archive on request; emails compose; JIRA tickets, ServiceNow INC/REQ/CHG numbers, and Salesforce IDs get recognized in flight.
BTC/ETH/SOL addresses, UUIDs, IPv4/IPv6/CIDR, SemVer, Docker images, Git refs, JSON, SQL, regexes, hex colors, base64 — catalogued on sight.
ISO 8601, cron expressions, durations, timezones, USD/EUR/GBP amounts and percentages — parsed into actionable suggestions.
VINs, license plates, ISBNs, order numbers, GPS coordinates, MAC addresses, mDNS hostnames — the long tail, covered.
Email & calendar
| Command | What IRIS does |
|---|---|
| Search Gmail for [query] | Searches your inbox and summarizes results aloud |
| Download attachments from [query] | Pulls matching attachments down to a folder of your choosing |
| Watch for email from [sender] | Creates a durable watch rule with proactive notification |
| Search Outlook for [query] | Same power for Outlook mailboxes |
| What do I have this week? | Reads back your upcoming schedule |
| Schedule a meeting with [name] | Creates the event — which then feeds JARVIS reminders |
| Find my meeting about [topic] | Full-text search across events |
Beyond the orb
Voice-opened web dashboards for glanceable monitoring.
A resizable side stage for rich output: live HTML/CSS/JS previews, Mermaid diagrams, and Recharts visualizations rendered straight from conversation.
STL, GLB, and GLTF models in their own window — pinch to zoom, spread to zoom out, palm to rotate, fist to reset. Gesture-controlled 3D from your webcam.
Underneath it all sits backend-native memory: Lethe for personal continuity and Mnemosyne for shared knowledge — so IRIS picks up tomorrow exactly where today left off. See the architecture page for how it works.