// IRIS Desktop / Capabilities

Everything she can do.

This is the real inventory — every capability documented here is implemented and working. Voice, vision, sixty-plus tools, macros, gestures, proactive intelligence, and the guardrail pipeline that keeps it all accountable.

#voice #commands #tools #guardrails #annotation #workspaces #gauntlet #jarvis #gestures #clipboard #email-calendar #surfaces

Voice system

A full-duplex voice that never talks over you.

IRIS runs a complete speech state machine — idle → listening → hearing → processing → speaking — with native audio capture, speaker verification, and streaming playback.

Subsystem What it does
Wake words Hey IRIS Hi IRIS Okay IRIS Yo IRIS — enrolled owners can skip the wake word; strangers always need it. 60 seconds of silence puts her to sleep.
Speech-to-text Whisper via the backend with echo cancellation and noise suppression, plus a native Rust/cpal capture path that bypasses WebView audio issues entirely. Speaker confidence scoring detects who's talking.
Text-to-speech Streaming TTS over WebSocket → Socket.IO → gapless chunked playback with 20ms overlaps. Falls back to HTTP TTS automatically if streaming drops.
Voice activity detection Frequency-domain thresholding with ~500ms silence detection, min/max recording bounds, and a post-speech cooldown so IRIS never hears her own voice.
Native audio engine Rust-level capture with RMS-based VAD, echo reference tracking, per-device selection, and automatic default device detection.

Voice commands

Natural language in. Real actions out.

Domain Say things like
Apps Open Chrome Close Notepad Launch VS Code
Web Search for Python tutorials Go to github.com
Media Play music Next track Volume up
System Lock my computer Sleep Show desktop
Files Read that file Clear my downloads
Display Set brightness to 50% Dim the screen
Network Turn off WiFi Enable Bluetooth
Mouse & keyboard Click there Type hello world Press Ctrl+S
Email Search Gmail for invoice Download attachments
Calendar What's on my calendar? Schedule a meeting
Windows Move to top right Jump to monitor 2 Focus mode
Weather What's the weather?

Tool registry

Sixty-plus tools. One guardrailed pipeline.

Every tool is registered in a typed registry with an assigned risk level, then executed through the safety pipeline below. Backend tools load dynamically at startup and refresh at runtime.

App control

launch_app close_app open_app close_application

Web

web_search open_url goto search

Input synthesis

type_text hotkey press_key_combo copy / paste / select_all

Mouse

move_mouse click double_click right_click drag

Files

read_file write_file edit_file delete_file clear_downloads

Media, volume & brightness

media_play_pause media_next set_volume set_brightness

Gmail + Outlook

gmail_search gmail_download outlook_search outlook_download email_watch

Calendar

calendar_today calendar_upcoming calendar_create calendar_search calendar_watch

Capture & annotate

take_screenshot annotate_circle annotate_arrow annotate_highlight annotate_text

System

set_wifi set_bluetooth show_notification get_system_stats

Clipboard

copy_to_clipboard read_clipboard

3D & surfaces

launch_holopoint open_hub canvas render

Guardrail system

Every action earns its execution.

Tools are categorized by danger level, and behavior adapts accordingly — from instant execution to simulated preview plus explicit human approval.

Risk level Example tools Behavior
Low open_url, web_search, type_text Execute immediately
Medium edit_file, safe run_command Log, then execute
High delete_file, risky run_command Require explicit approval
Critical delete_folder, shutdown-class actions Approval + pre-flight simulation, always

Screen annotation

She points at things. On your actual screen.

IRIS draws directly on your display — cyan annotations that auto-fade after five seconds, targeted through a resolution-calibrated 10×10 grid (columns A–J, rows 1–10).

Type Example Use it for
Circle Circle that button Highlight exactly the UI element she's acting on
Arrow Point from here to there Show relationships between two targets
Highlight Highlight this area Region emphasis with animated pulse
Text Label this "click here" Floating labels anchored to grid cells
Pointer Show me where Animated attention pointer

The same grid powers click targeting and AI-directed actions — one calibrated coordinate system shared by vision, voice, and input. Debug overlay on Ctrl+Shift+G.

Spatial memory

Workspaces, War Room layouts, Ghost Mode.

💾

Workspace snapshots

Your entire work context — open apps and window positions — saved under a name and restored on command.

  • Save this workspace as coding
  • Load writing
  • What workspaces do I have?
🎭

War Room layouts

Snap everything into mission-ready arrangements across one or many monitors.

  • Focus mode — one app centered, rest minimized
  • Coding layout — IDE left, terminal right
  • War room — full multi-monitor coverage
👻

Ghost Mode

IRIS becomes fully click-through — visible but untouchable by your mouse — until you call her back.

  • Go ghost
  • Come back
  • Toggle visibility with Ctrl+Shift+I

The Gauntlet

One word. Thirty steps.

Macros chain up to thirty-plus action types — apps, shell, git, media, input, windows, notifications, voice — behind a single voice trigger with exact, alias, and fuzzy matching. Define them in YAML:

~/.iris/macros/deploy.yaml
name: deploy
trigger: "deploy"
aliases: ["ship it", "send it"]
description: "Push code to production"
steps:
  - action: run_command
    params:
      command: "git push"
  - action: notify
    params:
      title: "Deployed"
      body: "Code pushed successfully"

And with Watch-Me-Learn pattern detection, IRIS proposes macros herself: when she sees you repeat the same workflow three days running, she offers to automate it.

JARVIS mode

Proactive intelligence that knows when to speak.

IRIS monitors system state, calendar, and error patterns — then speaks up only when it matters. Every trigger has configurable thresholds, cooldowns, quiet hours, and randomized natural phrasing.

Trigger Response Cooldown
Battery < 20% "Heads up — your battery is getting low." 5 minutes
Battery < 10% "Warning — battery is critically low!" 5 minutes
Idle > 2 hours "You've been at it a while. Maybe take a break?" 30 minutes
Same error 3× "I've noticed that happening repeatedly. Want help troubleshooting?" 5 minutes
Meeting in 5 min "You have '[meeting]' starting soon." Per event
Email watch rules Proactive notification when watched senders or subjects arrive Per rule

Presence & gesture control

Hands-free, literally.

MediaPipe face and hand tracking (21-point skeleton) runs locally at ~12 FPS with debounced triggers. Fourteen gestures map to twenty action types, with preset bundles for minimal, media, presentation, and accessibility setups.

Gesture Action Hold time
👋 Open palm Start listening 500 ms
👍 Thumbs up Confirm 300 ms
👎 Thumbs down Cancel 300 ms
🤣 Shush Interrupt speech instantly —
✋🏼 Point Take screenshot 800 ms
✊ Fist Mute toggle 500 ms
✨🏿 Swipes Track skip / volume up / down —
🖐 Wave Say hello —

Face detection adds presence awareness — she greets you when you sit down and tracks attention direction. Everything is opt-in and toggleable in settings.

Clipboard intelligence

Fifty-plus entity types, understood in real time.

Passive clipboard monitoring with priority-ordered pattern recognition suggests context-aware actions — and warns you before secrets leak somewhere they shouldn't.

🔒

Sensitive data shield

AWS, OpenAI, Anthropic, GitHub keys; SSNs; credit cards; private keys. Detected secrets are masked and flagged with a paste-carefully warning.

📦

Shipping & travel

UPS, FedEx, USPS, Amazon, DHL tracking numbers become one-click package tracking. Flight numbers surface live status. Airport codes resolve instantly.

🌐

Web & enterprise

URLs preview or archive on request; emails compose; JIRA tickets, ServiceNow INC/REQ/CHG numbers, and Salesforce IDs get recognized in flight.

🧩

Crypto & tech

BTC/ETH/SOL addresses, UUIDs, IPv4/IPv6/CIDR, SemVer, Docker images, Git refs, JSON, SQL, regexes, hex colors, base64 — catalogued on sight.

📅

Dates & money

ISO 8601, cron expressions, durations, timezones, USD/EUR/GBP amounts and percentages — parsed into actionable suggestions.

🚗

Identifiers

VINs, license plates, ISBNs, order numbers, GPS coordinates, MAC addresses, mDNS hostnames — the long tail, covered.

Email & calendar

Your inbox and schedule, spoken to.

Command What IRIS does
Search Gmail for [query] Searches your inbox and summarizes results aloud
Download attachments from [query] Pulls matching attachments down to a folder of your choosing
Watch for email from [sender] Creates a durable watch rule with proactive notification
Search Outlook for [query] Same power for Outlook mailboxes
What do I have this week? Reads back your upcoming schedule
Schedule a meeting with [name] Creates the event — which then feeds JARVIS reminders
Find my meeting about [topic] Full-text search across events

Beyond the orb

Hubs, Canvas, and Holopoint.

📺

IRIS Hubs

Voice-opened web dashboards for glanceable monitoring.

  • Pull up the baby monitor
  • Show security cameras
  • Open system dashboard
  • Open the music player
🖌

Canvas window

A resizable side stage for rich output: live HTML/CSS/JS previews, Mermaid diagrams, and Recharts visualizations rendered straight from conversation.

🚀

Holopoint viewer

STL, GLB, and GLTF models in their own window — pinch to zoom, spread to zoom out, palm to rotate, fist to reset. Gesture-controlled 3D from your webcam.

Underneath it all sits backend-native memory: Lethe for personal continuity and Mnemosyne for shared knowledge — so IRIS picks up tomorrow exactly where today left off. See the architecture page for how it works.

All of this, open at the core.

Read the tool registry, the guardrail pipeline, and the macro engine for yourself.

github.com/solsticeaistudio/iris-desktop-oss

ask iris
IRIS ONLINE
Drop files here Images, PDFs, text files