From de5fa4aecdd2dc56ffeb0a70334101f33678f34b Mon Sep 17 00:00:00 2001 From: weishu Date: Wed, 5 Aug 2026 06:46:26 +0800 Subject: [PATCH] docs: update voice assistant guide for multi-backend support --- docs/guide/voice-assistant.md | 108 +++++++++++++++++++++++++++------- 1 file changed, 86 insertions(+), 22 deletions(-) diff --git a/docs/guide/voice-assistant.md b/docs/guide/voice-assistant.md index 58afcf6c..59e7196e 100644 --- a/docs/guide/voice-assistant.md +++ b/docs/guide/voice-assistant.md @@ -1,16 +1,24 @@ # Voice input and assistant -Control your AI coding agent with voice using the built-in voice assistant powered by ElevenLabs Conversational AI. +Control your AI coding agent with your voice. The built-in voice assistant supports three backends: **ElevenLabs** Conversational AI, **Gemini Live**, and **Qwen Realtime**. Pick whichever provider you have credentials for — the hub detects what's configured and the web app lets you switch. For speech-to-text without a spoken assistant, open **Settings → Voice**, choose **Dictation**, then select a configured provider. Dictation records until you tap the microphone again, inserts the transcript into the composer, and never sends it automatically. Standard mode is the default. Realtime mode shows a live transcript while you speak and inserts the final result when you stop. Provider credentials are read only from the hub's startup environment: ```bash -# Pick any providers you use -export OPENAI_API_KEY="..." # gpt-transcribe / gpt-live-transcribe -export ELEVENLABS_API_KEY="..." # scribe_v2 / scribe_v2_realtime -export DEEPGRAM_API_KEY="..." # nova-3 standard / realtime +# Voice assistant backends (any of these enables the assistant) +export ELEVENLABS_API_KEY="..." # ElevenLabs ConvAI +export GEMINI_API_KEY="..." # Gemini Live (GOOGLE_API_KEY also works) +export DASHSCOPE_API_KEY="..." # Qwen Realtime (QWEN_API_KEY also works) + +# Optional: pin the hub's default assistant backend +export VOICE_BACKEND="elevenlabs" # elevenlabs | gemini-live | qwen-realtime + +# Dictation transcription providers (pick any you use) +export OPENAI_API_KEY="..." # gpt-transcribe / gpt-live-transcribe +export ELEVENLABS_API_KEY="..." # scribe_v2 / scribe_v2_realtime +export DEEPGRAM_API_KEY="..." # nova-3 standard / realtime export GROQ_API_KEY="..." # whisper-large-v3 # Or an OpenAI-compatible local server such as Speaches @@ -20,7 +28,7 @@ export TRANSCRIPTION_API_KEY="..." # optional ``` Restart the hub after changing credentials. API keys are not entered or stored in the web app. -Realtime OpenAI, ElevenLabs, and Deepgram sessions receive only short-lived credentials minted by the hub. Eligible desktop browsers with the on-device `SpeechRecognition` API expose **Browser on-device** as a realtime-only provider. HAPI checks the selected language pack when dictation starts and never falls back from that option to browser-hosted recognition. Mobile and unknown browser environments fail closed because this API is experimental and some Android WebViews expose unsafe partial implementations. +Realtime OpenAI, ElevenLabs, and Deepgram dictation sessions receive only short-lived credentials minted by the hub. Gemini Live and Qwen Realtime assistant sessions connect through hub-side WebSocket proxies, so those API keys never reach the browser either. Eligible desktop browsers with the on-device `SpeechRecognition` API expose **Browser on-device** as a realtime-only dictation provider. HAPI checks the selected language pack when dictation starts and never falls back from that option to browser-hosted recognition. Mobile and unknown browser environments fail closed because this API is experimental and some Android WebViews expose unsafe partial implementations. ## Overview @@ -34,45 +42,69 @@ The assistant bridges voice communication with your active coding agent (Claude ## Prerequisites -- Voice assistant: an [ElevenLabs](https://elevenlabs.io) account with API access -- Dictation: at least one configured provider above, or an OpenAI-compatible local server +You need API credentials for at least one assistant backend: + +- **ElevenLabs** - an [ElevenLabs](https://elevenlabs.io) account with API access +- **Gemini Live** - a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey) +- **Qwen Realtime** - a DashScope API key from [Alibaba Cloud Model Studio](https://www.alibabacloud.com/help/en/model-studio/get-api-key) + +Dictation needs at least one configured transcription provider from the list above, or an OpenAI-compatible local server. ## Setup -### 1. Get an API Key +### ElevenLabs 1. Sign up or log in at [elevenlabs.io](https://elevenlabs.io) 2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys) in your account settings 3. Create a new API key and copy it - -### 2. Configure the Hub - -Set the environment variable before starting the hub: +4. Set the environment variable before starting the hub: ```bash export ELEVENLABS_API_KEY="your-api-key" hapi hub --relay ``` -The hub automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use. +The hub automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use. When you pick a non-default voice, the hub creates a dedicated per-voice agent named `Hapi Voice Assistant [voice:]` so the selection always takes effect. -### 3. (Optional) Custom Agent - -If you want to use your own ElevenLabs agent instead of the auto-created one: +To use your own ElevenLabs agent instead of the auto-created one: ```bash export ELEVENLABS_AGENT_ID="your-agent-id" ``` +To map specific voices to your own agents (overrides per-voice auto-creation): + +```bash +export ELEVENLABS_VOICE_AGENT_MAP='{"": ""}' +``` + +### Gemini Live + +```bash +export GEMINI_API_KEY="your-api-key" # or GOOGLE_API_KEY +hapi hub --relay +``` + +### Qwen Realtime + +```bash +export DASHSCOPE_API_KEY="your-api-key" # or QWEN_API_KEY +hapi hub --relay +``` + +When more than one backend is configured, the hub default is ElevenLabs unless you set `VOICE_BACKEND`. Users can override the backend per browser in **Settings → Voice**. + ## Usage ### Starting a Voice Session 1. Open a session in the web app -2. Click the **microphone button** in the composer (or the send button when empty) +2. Click the **microphone button** in the composer (the send button shows a mic icon when the composer is empty) 3. Grant microphone permission when prompted 4. Start speaking +While the assistant is connected, a voice pill in the status bar shows its state (Connecting, Active, Muted, Error), and you can mute the microphone without ending the session. + ### Voice Commands | Say this | What happens | @@ -83,6 +115,22 @@ export ELEVENLABS_AGENT_ID="your-agent-id" | "No" / "Deny" / "Cancel" | Denies pending permission requests | | Direct questions | The voice assistant answers itself if it can | +## Settings + +Everything user-facing lives under **Settings → Voice**: + +- **Voice mode** - Voice assistant (two-way conversation) or Dictation (speech-to-text only) +- **Voice backend** - ElevenLabs, Gemini Live, or Qwen Realtime (shown when the hub has more than one configured) +- **Voice Language** - Auto-detect or a specific language; shared by the assistant and dictation +- **Voice** - Pick the assistant's voice. ElevenLabs lists your account voices (including clones) with audio previews; Gemini Live and Qwen Realtime offer their built-in voice catalogs +- **Opening** (assistant) - "Greet me" for a simple hello, or "Brief me" for a spoken summary of recent agent activity when you connect +- **Response length** (assistant) - Brief, Balanced, or Detailed answers + +**Settings → Voice → Advanced** additionally offers: + +- **Persona & instructions** - Rename/rebrand the assistant and shape its character and speaking style (preset or custom text) +- **How it sounds** - ElevenLabs tuning sliders (stability, style, speed, similarity boost, speaker boost) and Gemini's affective dialog option + ## How It Works ### Context Synchronization @@ -98,18 +146,26 @@ You don't need to ask for status updates - the assistant proactively summarizes ### Tools -The voice assistant has two tools to interact with your coding agent: +The voice assistant has two tools to interact with your coding agent, on every backend: 1. **messageCodingAgent** - Forwards your requests to the active agent 2. **processPermissionRequest** - Handles permission approvals and denials ### Architecture +ElevenLabs sessions stream audio over WebRTC directly to ElevenLabs; the hub only mints short-lived conversation tokens: + ``` Browser → WebRTC → ElevenLabs ConvAI → Voice Assistant → HAPI Hub → Coding Agent ``` -The voice connection uses WebRTC for low-latency audio streaming. The HAPI hub provides conversation tokens and handles authentication. +Gemini Live and Qwen Realtime sessions connect over WebSocket to a hub-side proxy, which injects the API key and session configuration server-side: + +``` +Browser → WebSocket → HAPI Hub proxy → Gemini Live / Qwen Realtime → Coding Agent +``` + +The voice connection uses WebRTC (ElevenLabs) or WebSocket (Gemini Live, Qwen Realtime) for low-latency audio streaming. The HAPI hub provides tokens and proxies and handles authentication; provider API keys never reach the browser. ## Tips @@ -124,7 +180,15 @@ The voice connection uses WebRTC for low-latency audio streaming. The HAPI hub p Set `ELEVENLABS_API_KEY` in your environment and restart the hub. -### "Failed to get microphone permission" +### "Gemini API key not configured" + +Set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) in your environment and restart the hub. + +### "DashScope API key not configured" + +Set `DASHSCOPE_API_KEY` (or `QWEN_API_KEY`) in your environment and restart the hub. + +### "Microphone permission denied" - Check browser permissions for microphone access - Ensure no other app is using the microphone @@ -137,7 +201,7 @@ If voice cannot start on a Xiaomi/MIUI device, or the browser cannot request mic ### Voice not responding - Verify the session is connected (green dot in status bar) -- Check that voice status shows "connecting" or connected state +- Check that the voice status pill shows "Connecting..." or the active state - Ensure you have a stable internet connection ### "Failed to create ElevenLabs agent automatically"