docs: add voice assistant documentation

2026-08-05 06:24:37 +00:00 · 2026-01-19 18:48:13 +08:00
parent 8f5b773b23
commit e9db67e18b
6 changed files with 160 additions and 0 deletions
@@ -22,6 +22,7 @@ export default defineConfig({
      { text: 'Installation', link: '/guide/installation' },
      { text: 'PWA', link: '/guide/pwa' },
      { text: 'How it Works', link: '/guide/how-it-works' },
+      { text: 'Voice Assistant', link: '/guide/voice-assistant' },
      { text: 'Why HAPI', link: '/guide/why-hapi' },
      { text: 'FAQ', link: '/guide/faq' }
    ],
@@ -89,6 +89,10 @@ In the session view, tap the "Files" tab to:

 Yes. Open any session and use the chat interface to send messages directly to the AI agent.

+### How do I use voice control?
+
+Set `ELEVENLABS_API_KEY`, open a session in the web app, and click the microphone button. See [Voice Assistant](./voice-assistant.md).
+
 ## Security

 ### Is my data safe?
@@ -113,6 +113,8 @@ On first run, HAPI:
 | `HAPI_HOME` | `~/.hapi` | Config directory path |
 | `DB_PATH` | `~/.hapi/hapi.db` | Database file path |
 | `CORS_ORIGINS` | - | Allowed CORS origins |
+| `ELEVENLABS_API_KEY` | - | ElevenLabs API key for voice |
+| `ELEVENLABS_AGENT_ID` | Auto-created | Custom ElevenLabs agent ID |
 </details>

 ## CLI setup
@@ -220,6 +222,20 @@ With the runner running:
 - You can spawn sessions remotely from the web app
 - Sessions persist even when the terminal is closed

+### Voice assistant setup
+
+Enable voice control:
+
+1. Get an API key from [elevenlabs.io](https://elevenlabs.io/app/settings/api-keys)
+2. Set the environment variable:
+
+```bash
+export ELEVENLABS_API_KEY="your-api-key"
+hapi server --relay
+```
+
+See [Voice Assistant](./voice-assistant.md) for usage details.
+
 ### Security notes

 - Keep tokens secret and rotate if needed
@@ -0,0 +1,128 @@
+# Voice Assistant
+
+Control your AI coding agent with voice using the built-in voice assistant powered by ElevenLabs Conversational AI.
+
+## Overview
+
+The voice assistant lets you:
+
+- **Talk to your agent** - Ask questions, give instructions, and request code changes hands-free
+- **Approve permissions by voice** - Say "yes" or "no" to approve or deny permission requests
+- **Monitor progress** - Receive spoken updates when tasks complete or errors occur
+
+The assistant bridges voice communication with your active coding agent (Claude Code, Codex, or Gemini), relaying your requests and summarizing responses in natural speech.
+
+## Prerequisites
+
+An [ElevenLabs](https://elevenlabs.io) account with API access
+
+## Setup
+
+### 1. Get an API Key
+
+1. Sign up or log in at [elevenlabs.io](https://elevenlabs.io)
+2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys) in your account settings
+3. Create a new API key and copy it
+
+### 2. Configure the Server
+
+Set the environment variable before starting the server:
+
+```bash
+export ELEVENLABS_API_KEY="your-api-key"
+hapi server --relay
+```
+
+The server automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use.
+
+### 3. (Optional) Custom Agent
+
+If you want to use your own ElevenLabs agent instead of the auto-created one:
+
+```bash
+export ELEVENLABS_AGENT_ID="your-agent-id"
+```
+
+## Usage
+
+### Starting a Voice Session
+
+1. Open a session in the web app
+2. Click the **microphone button** in the composer (or the send button when empty)
+3. Grant microphone permission when prompted
+4. Start speaking
+
+### Voice Commands
+
+| Say this | What happens |
+|----------|--------------|
+| "Ask Claude to..." / "Have it..." | Sends your request to the coding agent |
+| "Refactor the auth module" | Coding requests are forwarded automatically |
+| "Yes" / "Allow" / "Go ahead" | Approves pending permission requests |
+| "No" / "Deny" / "Cancel" | Denies pending permission requests |
+| Direct questions | The voice assistant answers itself if it can |
+
+## How It Works
+
+### Context Synchronization
+
+The voice assistant automatically receives updates when:
+
+- You focus on a session (full history is loaded)
+- The agent sends messages or uses tools
+- Permission requests arrive
+- Tasks complete
+
+You don't need to ask for status updates - the assistant proactively summarizes relevant changes.
+
+### Tools
+
+The voice assistant has two tools to interact with your coding agent:
+
+1. **messageCodingAgent** - Forwards your requests to the active agent
+2. **processPermissionRequest** - Handles permission approvals and denials
+
+### Architecture
+
+```
+Browser → WebRTC → ElevenLabs ConvAI → Voice Assistant → HAPI Server → Coding Agent
+```
+
+The voice connection uses WebRTC for low-latency audio streaming. The HAPI server provides conversation tokens and handles authentication.
+
+## Tips
+
+- **Be specific** - Clear, complete requests get better results
+- **Wait for completion** - The assistant stays silent while the agent works, then summarizes results
+- **Use natural language** - No special command syntax needed
+- **Keep sessions focused** - One active session at a time for clearest context
+
+## Troubleshooting
+
+### "ElevenLabs API key not configured"
+
+Set `ELEVENLABS_API_KEY` in your environment and restart the server.
+
+### "Failed to get microphone permission"
+
+- Check browser permissions for microphone access
+- Ensure no other app is using the microphone
+- Try refreshing the page
+
+### Voice not responding
+
+- Verify the session is connected (green dot in status bar)
+- Check that voice status shows "connecting" or connected state
+- Ensure you have a stable internet connection
+
+### "Failed to create ElevenLabs agent automatically"
+
+- Verify your API key is valid
+- Check your ElevenLabs account has available quota
+- Try setting a custom `ELEVENLABS_AGENT_ID`
+
+### Poor audio quality
+
+- Use a headset to avoid echo
+- Reduce background noise
+- Check your internet connection stability