From e9db67e18bab4c9304352222416bff00c6414c33 Mon Sep 17 00:00:00 2001 From: weishu Date: Mon, 19 Jan 2026 18:48:13 +0800 Subject: [PATCH] docs: add voice assistant documentation --- README.md | 2 + docs/.vitepress/config.ts | 1 + docs/guide/faq.md | 4 ++ docs/guide/installation.md | 16 +++++ docs/guide/voice-assistant.md | 128 ++++++++++++++++++++++++++++++++++ server/README.md | 9 +++ 6 files changed, 160 insertions(+) create mode 100644 docs/guide/voice-assistant.md diff --git a/README.md b/README.md index 59069c96..121755be 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ Run official Claude Code / Codex / Gemini sessions locally and control them remo - **AFK Without Stopping** - Step away from your desk? Approve AI requests from your phone with one tap. - **Your AI, Your Choice** - Claude Code, Codex, Gemini—different models, one unified workflow. - **Terminal Anywhere** - Run commands from your phone or browser, directly connected to the working machine. +- **Voice Control** - Talk to your AI agent hands-free using the built-in voice assistant. ## Demo @@ -33,6 +34,7 @@ For self-hosted options (Cloudflare Tunnel, Tailscale), see [Installation](docs/ - [App](docs/guide/pwa.md) - [How it Works](docs/guide/how-it-works.md) +- [Voice Assistant](docs/guide/voice-assistant.md) - [Why HAPI](docs/guide/why-hapi.md) - [FAQ](docs/guide/faq.md) diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 1bbdb556..15562e64 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -22,6 +22,7 @@ export default defineConfig({ { text: 'Installation', link: '/guide/installation' }, { text: 'PWA', link: '/guide/pwa' }, { text: 'How it Works', link: '/guide/how-it-works' }, + { text: 'Voice Assistant', link: '/guide/voice-assistant' }, { text: 'Why HAPI', link: '/guide/why-hapi' }, { text: 'FAQ', link: '/guide/faq' } ], diff --git a/docs/guide/faq.md b/docs/guide/faq.md index bd9bca9b..62d57340 100644 --- a/docs/guide/faq.md +++ b/docs/guide/faq.md @@ -89,6 +89,10 @@ In the session view, tap the "Files" tab to: Yes. Open any session and use the chat interface to send messages directly to the AI agent. +### How do I use voice control? + +Set `ELEVENLABS_API_KEY`, open a session in the web app, and click the microphone button. See [Voice Assistant](./voice-assistant.md). + ## Security ### Is my data safe? diff --git a/docs/guide/installation.md b/docs/guide/installation.md index 4346e723..e0d02994 100644 --- a/docs/guide/installation.md +++ b/docs/guide/installation.md @@ -113,6 +113,8 @@ On first run, HAPI: | `HAPI_HOME` | `~/.hapi` | Config directory path | | `DB_PATH` | `~/.hapi/hapi.db` | Database file path | | `CORS_ORIGINS` | - | Allowed CORS origins | +| `ELEVENLABS_API_KEY` | - | ElevenLabs API key for voice | +| `ELEVENLABS_AGENT_ID` | Auto-created | Custom ElevenLabs agent ID | ## CLI setup @@ -220,6 +222,20 @@ With the runner running: - You can spawn sessions remotely from the web app - Sessions persist even when the terminal is closed +### Voice assistant setup + +Enable voice control: + +1. Get an API key from [elevenlabs.io](https://elevenlabs.io/app/settings/api-keys) +2. Set the environment variable: + +```bash +export ELEVENLABS_API_KEY="your-api-key" +hapi server --relay +``` + +See [Voice Assistant](./voice-assistant.md) for usage details. + ### Security notes - Keep tokens secret and rotate if needed diff --git a/docs/guide/voice-assistant.md b/docs/guide/voice-assistant.md new file mode 100644 index 00000000..694e8d48 --- /dev/null +++ b/docs/guide/voice-assistant.md @@ -0,0 +1,128 @@ +# Voice Assistant + +Control your AI coding agent with voice using the built-in voice assistant powered by ElevenLabs Conversational AI. + +## Overview + +The voice assistant lets you: + +- **Talk to your agent** - Ask questions, give instructions, and request code changes hands-free +- **Approve permissions by voice** - Say "yes" or "no" to approve or deny permission requests +- **Monitor progress** - Receive spoken updates when tasks complete or errors occur + +The assistant bridges voice communication with your active coding agent (Claude Code, Codex, or Gemini), relaying your requests and summarizing responses in natural speech. + +## Prerequisites + +An [ElevenLabs](https://elevenlabs.io) account with API access + +## Setup + +### 1. Get an API Key + +1. Sign up or log in at [elevenlabs.io](https://elevenlabs.io) +2. Go to [API Keys](https://elevenlabs.io/app/settings/api-keys) in your account settings +3. Create a new API key and copy it + +### 2. Configure the Server + +Set the environment variable before starting the server: + +```bash +export ELEVENLABS_API_KEY="your-api-key" +hapi server --relay +``` + +The server automatically creates a "Hapi Voice Assistant" agent in your ElevenLabs account on first use. + +### 3. (Optional) Custom Agent + +If you want to use your own ElevenLabs agent instead of the auto-created one: + +```bash +export ELEVENLABS_AGENT_ID="your-agent-id" +``` + +## Usage + +### Starting a Voice Session + +1. Open a session in the web app +2. Click the **microphone button** in the composer (or the send button when empty) +3. Grant microphone permission when prompted +4. Start speaking + +### Voice Commands + +| Say this | What happens | +|----------|--------------| +| "Ask Claude to..." / "Have it..." | Sends your request to the coding agent | +| "Refactor the auth module" | Coding requests are forwarded automatically | +| "Yes" / "Allow" / "Go ahead" | Approves pending permission requests | +| "No" / "Deny" / "Cancel" | Denies pending permission requests | +| Direct questions | The voice assistant answers itself if it can | + +## How It Works + +### Context Synchronization + +The voice assistant automatically receives updates when: + +- You focus on a session (full history is loaded) +- The agent sends messages or uses tools +- Permission requests arrive +- Tasks complete + +You don't need to ask for status updates - the assistant proactively summarizes relevant changes. + +### Tools + +The voice assistant has two tools to interact with your coding agent: + +1. **messageCodingAgent** - Forwards your requests to the active agent +2. **processPermissionRequest** - Handles permission approvals and denials + +### Architecture + +``` +Browser → WebRTC → ElevenLabs ConvAI → Voice Assistant → HAPI Server → Coding Agent +``` + +The voice connection uses WebRTC for low-latency audio streaming. The HAPI server provides conversation tokens and handles authentication. + +## Tips + +- **Be specific** - Clear, complete requests get better results +- **Wait for completion** - The assistant stays silent while the agent works, then summarizes results +- **Use natural language** - No special command syntax needed +- **Keep sessions focused** - One active session at a time for clearest context + +## Troubleshooting + +### "ElevenLabs API key not configured" + +Set `ELEVENLABS_API_KEY` in your environment and restart the server. + +### "Failed to get microphone permission" + +- Check browser permissions for microphone access +- Ensure no other app is using the microphone +- Try refreshing the page + +### Voice not responding + +- Verify the session is connected (green dot in status bar) +- Check that voice status shows "connecting" or connected state +- Ensure you have a stable internet connection + +### "Failed to create ElevenLabs agent automatically" + +- Verify your API key is valid +- Check your ElevenLabs account has available quota +- Try setting a custom `ELEVENLABS_AGENT_ID` + +### Poor audio quality + +- Use a headset to avoid echo +- Reduce background noise +- Check your internet connection stability diff --git a/server/README.md b/server/README.md index d15f93b4..662fe7f4 100644 --- a/server/README.md +++ b/server/README.md @@ -24,6 +24,11 @@ See `src/configuration.ts` for all options. - `TELEGRAM_BOT_TOKEN` - Token from @BotFather. - `HAPI_PUBLIC_URL` - Public HTTPS URL for Telegram Mini App access. Also used to derive default CORS origins for the web app. +### Optional (Voice) + +- `ELEVENLABS_API_KEY` - ElevenLabs API key for voice assistant. +- `ELEVENLABS_AGENT_ID` - Custom ElevenLabs agent ID (auto-created if not set). + ### Optional - `HAPI_LISTEN_HOST` - HTTP bind address (default: 127.0.0.1). @@ -100,6 +105,10 @@ See `src/web/routes/` for all endpoints. - `GET /api/events` - SSE stream for live updates. +### Voice (`src/web/routes/voice.ts`) + +- `POST /api/voice/token` - Get ElevenLabs conversation token. + ### CLI (`src/web/routes/cli.ts`) - `POST /cli/sessions` - Create/load session.