wushenghua/hapi
1
0
Fork 0
mirror of https://github.com/wu736139669/hapi.git synced 2026-08-05 06:24:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: update voice assistant guide for multi-backend support

Browse Source
This commit is contained in:
weishu
2026-08-05 06:46:26 +08:00
parent b31c5a8e76
commit de5fa4aecd
1 changed files with 86 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/voice-assistant.md
+83 -19
View File
@@ -1,13 +1,21 @@
# Voice input and assistant
Control your AI coding agent with voice using the built-in voice assistant powered by ElevenLabs Conversational AI.
Control your AI coding agent with your voice. The built-in voice assistant supports three backends: **ElevenLabs** Conversational AI, **Gemini Live**, and **Qwen Realtime**. Pick whichever provider you have credentials for — the hub detects what's configured and the web app lets you switch.
For speech-to-text without a spoken assistant, open **Settings → Voice**, choose **Dictation**, then select a configured provider. Dictation records until you tap the microphone again, inserts the transcript into the composer, and never sends it automatically. Standard mode is the default. Realtime mode shows a live transcript while you speak and inserts the final result when you stop.
Provider credentials are read only from the hub's startup environment:
```bash
# Pick any providers you use
# Voice assistant backends (any of these enables the assistant)
export ELEVENLABS_API_KEY="..." # ElevenLabs ConvAI
export GEMINI_API_KEY="..." # Gemini Live (GOOGLE_API_KEY also works)
export DASHSCOPE_API_KEY="..." # Qwen Realtime (QWEN_API_KEY also works)
# Optional: pin the hub's default assistant backend
export VOICE_BACKEND="elevenlabs" # elevenlabs | gemini-live | qwen-realtime
# Dictation transcription providers (pick any you use)
export OPENAI_API_KEY="..." # gpt-transcribe / gpt-live-transcribe
export ELEVENLABS_API_KEY="..." # scribe_v2 / scribe_v2_realtime
export DEEPGRAM_API_KEY="..." # nova-3 standard / realtime
@@ -20,7 +28,7 @@ export TRANSCRIPTION_API_KEY="..." # optional
```
Restart the hub after changing credentials. API keys are not entered or stored in the web app.
Realtime OpenAI, ElevenLabs, and Deepgram sessions receive only short-lived credentials minted by the hub. Eligible desktop browsers with the on-device `SpeechRecognition` API expose **Browser on-device** as a realtime-only provider. HAPI checks the selected language pack when dictation starts and never falls back from that option to browser-hosted recognition. Mobile and unknown browser environments fail closed because this API is experimental and some Android WebViews expose unsafe partial implementations.
Realtime OpenAI, ElevenLabs, and Deepgram dictation sessions receive only short-lived credentials minted by the hub. Gemini Live and Qwen Realtime assistant sessions connect through hub-side WebSocket proxies, so those API keys never reach the browser either. Eligible desktop browsers with the on-device `SpeechRecognition` API expose **Browser on-device** as a realtime-only dictation provider. HAPI checks the selected language pack when dictation starts and never falls back from that option to browser-hosted recognition. Mobile and unknown browser environments fail closed because this API is experimental and some Android WebViews expose unsafe partial implementations.
## Overview
@@ -34,45 +42,69 @@ The assistant bridges voice communication with your active coding agent (Claude
## Prerequisites
- Voice assistant: an [ElevenLabs](https://elevenlabs.io) account with API access
- Dictation: at least one configured provider above, or an OpenAI-compatible local server
You need API credentials for at least one assistant backend:
- **ElevenLabs** - an [ElevenLabs](https://elevenlabs.io) account with API access
- **Gemini Live** - a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey)
- **Qwen Realtime** - a DashScope API key from [Alibaba Cloud Model Studio](https://www.alibabacloud.com/help/en/model-studio/get-api-key)
Dictation needs at least one configured transcription provider from the list above, or an OpenAI-compatible local server.
## Setup
### 1. Get an API Key
### ElevenLabs
1. Sign up or log in at [elevenlabs.io](https://elevenlabs.io)
2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys) in your account settings
3. Create a new API key and copy it
### 2. Configure the Hub
Set the environment variable before starting the hub:
4. Set the environment variable before starting the hub:
```bash
export ELEVENLABS_API_KEY="your-api-key"
hapi hub --relay
```
The hub automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use.
The hub automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use. When you pick a non-default voice, the hub creates a dedicated per-voice agent named `Hapi Voice Assistant [voice:<id>]` so the selection always takes effect.
### 3. (Optional) Custom Agent
If you want to use your own ElevenLabs agent instead of the auto-created one:
To use your own ElevenLabs agent instead of the auto-created one:
```bash
export ELEVENLABS_AGENT_ID="your-agent-id"
```
To map specific voices to your own agents (overrides per-voice auto-creation):
```bash
export ELEVENLABS_VOICE_AGENT_MAP='{"<voice-id>": "<agent-id>"}'
```
### Gemini Live
```bash
export GEMINI_API_KEY="your-api-key" # or GOOGLE_API_KEY
hapi hub --relay
```
### Qwen Realtime
```bash
export DASHSCOPE_API_KEY="your-api-key" # or QWEN_API_KEY
hapi hub --relay
```
When more than one backend is configured, the hub default is ElevenLabs unless you set `VOICE_BACKEND`. Users can override the backend per browser in **Settings → Voice**.
## Usage
### Starting a Voice Session
1. Open a session in the web app
2. Click the **microphone button** in the composer (or the send button when empty)
2. Click the **microphone button** in the composer (the send button shows a mic icon when the composer is empty)
3. Grant microphone permission when prompted
4. Start speaking
While the assistant is connected, a voice pill in the status bar shows its state (Connecting, Active, Muted, Error), and you can mute the microphone without ending the session.
### Voice Commands
| Say this | What happens |
@@ -83,6 +115,22 @@ export ELEVENLABS_AGENT_ID="your-agent-id"
| "No" / "Deny" / "Cancel" | Denies pending permission requests |
| Direct questions | The voice assistant answers itself if it can |
## Settings
Everything user-facing lives under **Settings → Voice**:
- **Voice mode** - Voice assistant (two-way conversation) or Dictation (speech-to-text only)
- **Voice backend** - ElevenLabs, Gemini Live, or Qwen Realtime (shown when the hub has more than one configured)
- **Voice Language** - Auto-detect or a specific language; shared by the assistant and dictation
- **Voice** - Pick the assistant's voice. ElevenLabs lists your account voices (including clones) with audio previews; Gemini Live and Qwen Realtime offer their built-in voice catalogs
- **Opening** (assistant) - "Greet me" for a simple hello, or "Brief me" for a spoken summary of recent agent activity when you connect
- **Response length** (assistant) - Brief, Balanced, or Detailed answers
**Settings → Voice → Advanced** additionally offers:
- **Persona & instructions** - Rename/rebrand the assistant and shape its character and speaking style (preset or custom text)
- **How it sounds** - ElevenLabs tuning sliders (stability, style, speed, similarity boost, speaker boost) and Gemini's affective dialog option
## How It Works
### Context Synchronization
@@ -98,18 +146,26 @@ You don't need to ask for status updates - the assistant proactively summarizes
### Tools
The voice assistant has two tools to interact with your coding agent:
The voice assistant has two tools to interact with your coding agent, on every backend:
1. **messageCodingAgent** - Forwards your requests to the active agent
2. **processPermissionRequest** - Handles permission approvals and denials
### Architecture
ElevenLabs sessions stream audio over WebRTC directly to ElevenLabs; the hub only mints short-lived conversation tokens:
```
Browser → WebRTC → ElevenLabs ConvAI → Voice Assistant → HAPI Hub → Coding Agent
```
The voice connection uses WebRTC for low-latency audio streaming. The HAPI hub provides conversation tokens and handles authentication.
Gemini Live and Qwen Realtime sessions connect over WebSocket to a hub-side proxy, which injects the API key and session configuration server-side:
```
Browser → WebSocket → HAPI Hub proxy → Gemini Live / Qwen Realtime → Coding Agent
```
The voice connection uses WebRTC (ElevenLabs) or WebSocket (Gemini Live, Qwen Realtime) for low-latency audio streaming. The HAPI hub provides tokens and proxies and handles authentication; provider API keys never reach the browser.
## Tips
@@ -124,7 +180,15 @@ The voice connection uses WebRTC for low-latency audio streaming. The HAPI hub p
Set `ELEVENLABS_API_KEY` in your environment and restart the hub.
### "Failed to get microphone permission"
### "Gemini API key not configured"
Set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) in your environment and restart the hub.
### "DashScope API key not configured"
Set `DASHSCOPE_API_KEY` (or `QWEN_API_KEY`) in your environment and restart the hub.
### "Microphone permission denied"
- Check browser permissions for microphone access
- Ensure no other app is using the microphone
@@ -137,7 +201,7 @@ If voice cannot start on a Xiaomi/MIUI device, or the browser cannot request mic
### Voice not responding
- Verify the session is connected (green dot in status bar)
- Check that voice status shows "connecting" or connected state
- Check that the voice status pill shows "Connecting..." or the active state
- Ensure you have a stable internet connection
### "Failed to create ElevenLabs agent automatically"