4 Commits

Include renames

Author	SHA1	Message	Date
3 people	a812a51dd7	feat(voice): backend voice picker + advanced controls behind disclosure (#742) (#743) ... * feat(voice): voice personality, picker catalog, and prompt layer foundation - voicePickerCatalog.ts: per-backend voice lists for Gemini and Qwen with resolve helpers (resolveGeminiLiveVoice, resolveQwenRealtimeVoice) - voicePersonality.ts: VoicePersonalityPreferences schema, presets, composed system prompt with identity/character/response-length layers - voicePromptLayers.ts: buildResolvedVoiceSystemPrompt, preset delivery snippets - voiceSystemPromptParam.ts: hub-side base64url decode for ?systemPrompt= - voicePickerPreferences.ts, voicePersonalitySession.ts: browser-side encode, decode, and storage helpers - useVoicePersonality: React hook for preferences persistence via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): preset delivery included when non-balanced preset selected; restore test typecheck - isDefaultVoicePersonality: add preset check so warm/calm/direct presets trigger the delivery snippet instead of being treated as default - web/tsconfig.json: remove test file exclusion from typecheck (restoring strict coverage of test code); fix resulting type error in mock declaration via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): include use_speaker_boost in ElevenLabs TTS override payload The checkbox persisted the pref but ttsDiffersFromDefault and buildElevenLabsTtsOverride both omitted it, so the setting was never sent to the agent. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * test(voice): update speaker_boost test to assert it IS included in override The previous test asserted use_speaker_boost was omitted; now it's correctly included in the TTS payload. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): authorize use_speaker_boost in ElevenLabs override schema Add use_speaker_boost to both the VoiceAgentConfig tts override type and the buildVoiceAgentConfig() platform_settings so the field is accepted by the ElevenLabs agent runtime. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): propagate full language code through composed prompt, not just zh getDefaultVoiceSystemPrompt and resolveComposedVoiceSystemPrompt were filtering language to zh-only before passing to composeVoiceAgentPrompt. Now append buildVoiceLanguageBlock(language) after composition so French, Spanish, Japanese etc. reach Gemini/Qwen sessions correctly. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): only append language block when language explicitly set Building language block unconditionally when no language is given caused getDefaultVoiceSystemPrompt() to diverge from VOICE_SYSTEM_PROMPT. Only append the block when a code is explicitly provided. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): always include language block for Gemini/Qwen in composed prompt When auto-detect is on (language=undefined), the composed prompt sent via hub proxy was losing the language auto-detect instruction because the block was only added when language was explicitly set. Now: ElevenLabs skips the block (has its own language field); Gemini/Qwen always include it — undefined produces the auto-detect block, an explicit code produces the appropriate language instruction. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> --------- Co-authored-by: HAPI <noreply@hapi.run>	2026-06-05 21:43:04 +08:00
4 people	59c29e8423	feat(voice): pluggable voice backend with Gemini Live & Qwen Realtime (#692) ... * feat(voice): pluggable voice backend with Gemini Live & Qwen Realtime Rebased from Overbaker/hapi#401 onto current main. Adds a pluggable voice backend architecture that extends the existing ElevenLabs integration: - Gemini 2.5 Live (gemini-live): Google real-time audio via WebSocket with full function calling (messageCodingAgent, processPermissionRequest) - Qwen Realtime (qwen-realtime): Alibaba DashScope via hub WebSocket proxy (browser cannot set Authorization header directly) - VoiceBackendSession: dynamic backend selector with React.lazy loading, gates voice button until backend module is registered - Hub WS proxies: JWT-authenticated /api/voice/gemini-ws and /api/voice/qwen-ws endpoints in Bun.serve, with message queueing during upstream connect to prevent dropped setup frames - AudioWorklet pipeline: inline Blob URL recorder, 24 kHz PCM player, serial tool call execution, AudioContext created in user gesture for mobile - Backend discovery: GET /voice/backend + POST /voice/gemini-token / POST /voice/qwen-token hub routes; frontend auto-detects active backend Merge notes: - Rebased 135 upstream commits cleanly; HappyComposer keeps upstream's configurable enter-behavior setting (supersedes hard-coded Ctrl+Enter) - Converted gemini test files from bun:test to vitest (web package uses vitest) - All 221 hub tests and 636 web tests pass; TypeScript clean * fix(voice): restore user mic mute state after Gemini turn completes turnComplete handler was unconditionally calling setMuted(false), which re-enabled the mic track even when the user had manually muted. Now restores to state.micMuted instead. * fix(voice): remove hard-coded Chinese language from Gemini backend buildGeminiLiveConfig was appending VOICE_CHINESE_LANGUAGE_BLOCK which forced Gemini to always respond in Mandarin regardless of user locale. Gemini now uses the neutral base prompt and responds in the language the user speaks to it, consistent with the ElevenLabs behaviour. * fix(voice): reset modelSpeaking in cleanup to unblock mic on restart If the session closes while Gemini is mid-speech, cleanup() left state.modelSpeaking=true. The next startSession() would then drop all mic audio in sendAudioChunk() until a model turn eventually flipped the flag — effectively deaf until page reload. * fix(voice): guard stale close handlers in Gemini and Qwen sessions ws.onclose operated on module-level state.ws, not the socket that fired the event. A rapid stop/restart could cause the old socket's onclose to call cleanup() after the new socket was assigned, tearing down the live session. Guard with `if (state.ws !== ws) return` before cleanup. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): remove hard-coded Chinese language from Qwen backend Matches the Gemini fix — both backends now use VOICE_SYSTEM_PROMPT without the Chinese language block, giving consistent English-default behaviour across all non-ElevenLabs backends. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * feat(voice): proactive/reactive toggle in voice settings Adds a "Proactive voice" toggle (default: off = reactive) to the Voice Assistant settings section. Reactive (default): initial context and agent-ready events are fed silently; the assistant waits for the user to speak first. Proactive: original behaviour — Gemini/Qwen narrate context on connect and speak unprompted when the agent finishes a task. ElevenLabs is also affected via onReady sending a user message rather than a silent update. Covers all three backends uniformly. localStorage key: hapi-voice-proactive. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): normalize WS close codes, drop barrel re-exports, fix SSE visibility - hub/server.ts: add toClientCloseCode() to normalize reserved upstream close codes (1005/1006/1015) to 1011 before forwarding to browser; abnormal upstream drops (1006) would otherwise throw on clientWs.close() and leave the browser socket open - realtime/index.ts: remove static GeminiLiveVoiceSession and QwenVoiceSession barrel exports; VoiceBackendSession lazy-imports both, so barrel re-exports created static dependencies that defeated the intended code-split - App.tsx: gate global useVisibilityReporter on !sessionEventSubscription so the always-on SSE connection does not suppress native Web Push notifications for sessions the user is not currently viewing via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): respect language setting in Gemini/Qwen; fix voice-start toggle label - buildGeminiLiveConfig() now accepts optional language param; appends VOICE_CHINESE_LANGUAGE_BLOCK only when language === 'zh' - GeminiLiveVoiceSession passes config.language through - QwenVoiceSession conditionally builds basePrompt from language setting - Fixes silent no-op when user selects Chinese in voice settings on Gemini/Qwen backends (was ElevenLabs-only) - Rename voice-start toggle label to 'Start voice session with summary' - Fix description: clarifies the choice is about session-open behaviour (summary vs greeting), not ongoing narration via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): send greeting trigger in reactive mode for Gemini Gemini Live has no built-in first-message like ElevenLabs agents do; without an explicit turnComplete:true it sits silently. In reactive mode (default, toggle off) now sends a greeting instruction after any silent context feed so Gemini introduces itself and invites the user to speak. Proactive mode is unchanged: the context summary is the opening speech. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): suppress Gemini self-identification and context leak in greeting - VOICE_SYSTEM_PROMPT: explicit instruction never to call itself Gemini, Google, or any underlying model/provider name — always HAPI - Greeting trigger text: instruct to greet as HAPI only, suppress model name and any reference to context/recent activity in the opening line via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): address code review findings — error handling, proxy, audio Gemini + Qwen client: - onerror now sets setupDone/sessionReady and nulls state.ws before calling reject(), so the stale-close guard trips in onclose and prevents a duplicate statusCallback('error') on WS failure Gemini client: - Proactive mode with no initialContext now falls through to the greeting trigger instead of sitting silently - Remove unused handleBargeIn callback (dead code) Qwen client: - Add input_audio_sample_rate: 16000 to session.update so PCM rate is declared explicitly rather than relying on DashScope's default Hub proxy: - Remove no-op ternary in Gemini flush loop and message handler (typeof x === 'string' ? x : x); use upstream.send(msg) directly - Qwen onerror now calls upstreamMap.delete() before closing client, eliminating the stale map entry window - Align Qwen hub fallback model string with QWEN_REALTIME_MODEL constant ('qwen3-omni-flash-realtime') via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): trailing-slash WS URL, Qwen session.update schema hub/voice.ts: - Replace string-concat WS URL construction with buildVoiceWsUrl() which uses URL API to set protocol/pathname cleanly — fixes double-slash when HAPI_PUBLIC_URL has a trailing slash (would silently skip the proxy route) QwenVoiceSession.tsx: - Wrap tool definitions in {type:'function', function:{...}} as required by Qwen-Omni realtime schema — previous flat shape caused session.update rejection before audio capture could start - Use pcm16/pcm24 audio formats matching DashScope spec; remove input_audio_sample_rate (encoded in format name) via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): await audio capture before setMuted; sanitize upstream close codes GeminiLiveVoiceSession + QwenVoiceSession: - startAudioCapture() is now async and awaits recorder.start() before calling setMuted() — previously setMuted ran before getUserMedia resolved so a session restarted while muted would open the mic anyway - statusCallback('connected') now fires after audio is ready - setMuted() called unconditionally (not just when true) to correctly apply saved state in either direction hub/src/web/server.ts: - Both Gemini and Qwen close() handlers now pass the client code through toClientCloseCode() before forwarding to upstream — prevents reserved codes (e.g. 1006) from causing WebSocket.close() to throw and leave the upstream session open until provider timeout - Reason string capped at 123 bytes (WebSocket protocol limit) via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): wrap startAudioCapture in try/catch to propagate mic errors An unhandled rejection inside the async onmessage callback does not propagate to the outer startSession Promise — the UI hangs on 'connecting' and the provider socket stays partially open. Wrapping the await in try/catch calls cleanup()/statusCallback('error')/reject() so failures surface correctly. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): propagate backend discovery failure instead of silently falling back to ElevenLabs fetchVoiceBackend no longer catches errors and defaults to 'elevenlabs' — any network or server failure now throws so VoiceBackendSession can surface it via onStatusChange('error', ...) rather than silently mounting the wrong backend. VoiceBackendSession also resets backend state to null when api changes, so a stale ElevenLabs registration from a prior discovery cannot persist into a new session. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): throw on unrecognised backend value instead of silently falling back to ElevenLabs Unknown backend strings (future values, typos) now throw rather than defaulting to elevenlabs, closing the narrow remaining form of the original misrouting bug. Also removes the unnecessary `as VoiceBackendResponse` cast. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): add Qwen greeting/proactive trigger; fix socket buffer for base64 uploads Qwen session.updated handler now sends the same proactive summary or greeting trigger that Gemini does — previously it started silently in both proactive and reactive modes. maxHttpBufferSize raised to 68 MiB to account for base64 expansion: 50 MiB decoded files become ~66.7 MiB as base64 JSON, so the previous 55 MiB ceiling would disconnect uploads above ~41 MiB before they reached the CLI. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): replace unsupported conversation.item.create with session.update for Qwen text Qwen's realtime API only supports conversation.item.create for function_call_output. Sending it with type:'message' for greetings/context was invalid and could fail before the user spoke. sendTextMessage and sendContextualUpdate now update session instructions via session.update (accumulating context into the system prompt) and trigger response.create only when a spoken reply is needed — matching Qwen's supported client event surface. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): guard session.updated re-entry and reset config on session start session.updated now returns early after the first ack — subsequent session.update calls (instruction appends) also echo session.updated but must not re-trigger audio capture or the greeting path. currentSessionConfig is now reset to null at the top of startSession so a stale config from a failed previous session cannot leak into the new one. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): assert wsUrl presence for Gemini proxy connections Without this guard, a missing wsUrl in the hub token response would silently attempt to connect directly to Google with "proxied" as the API key — producing a confusing auth failure instead of a clear error. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): correct Qwen audio formats and default voice DashScope realtime API accepts only 'pcm' for both input and output audio formats. The pcm16/pcm24 values caused session.update rejection before audio capture could start, leaving the Qwen backend unusable. Also updates the default voice from Mia (not in the qwen3-omni-flash- realtime voice list) to Cherry, which is documented as supported. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): close AudioContext on failed voice session start Failed token fetch, microphone denial, or WebSocket error during setup left state.playbackContext open. Each failure path now calls cleanup() before throwing/rejecting, preventing AudioContext leaks on mobile browsers with hard limits on concurrent contexts. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * chore: restore non-voice files to upstream/main state Reverts changes to files that shouldn't differ from upstream: - .gitignore: remove fork-only AGENTS.local.md entry - web/src/App.tsx: restore dual-subscription SSE pattern (scope-aware) - web/src/hooks/useSSE.ts: restore SSEScope/scope parameter - web/src/hooks/useSSE.test.ts: restore (was accidentally deleted) - web/src/lib/appSseSubscriptions.ts: restore (was accidentally deleted) - web/src/lib/appSseSubscriptions.test.ts: restore (was accidentally deleted) - hub/src/sync/syncEngine.ts: restore (off-topic change) * fix(voice): harden Gemini and Qwen WS proxies against client abuse Hub sends HAPI-owned Gemini setup on proxy connect and rejects client setup frames. Qwen proxy always uses QWEN_REALTIME_MODEL instead of a client query parameter. Shared buildGeminiLiveSetupMessage() keeps wire format in one place. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(voice): harden Qwen proxy — hub-owned setup, client frame allowlist Mirror the Gemini proxy security model for Qwen: - Hub sends initial session.update (voice/tools/instructions) on upstream connect so the browser cannot override config fields. - Proxy message() now calls isQwenSafeClientFrame() and closes the connection (1008) if a client session.update touches any field other than 'instructions' (blocks tool/voice/modality overrides). - QwenVoiceSession no longer sends session.update on session.created; it waits for the hub-relayed session.updated and then sends only instruction-only updates for context/proactive content. - Language passed as query param (?language=zh) so hub builds the correct Chinese system prompt without a client-supplied session.update. - buildQwenSessionUpdateMessage() and isQwenSafeClientFrame() added to @hapi/protocol/voice; 9 new unit tests cover filter edge cases. * fix(voice): respect Qwen session.created→session.update protocol ordering DashScope requires session.update to be sent AFTER session.created is received, not immediately on WebSocket open. Previously the hub sent session.update in upstream.onopen, which violated this ordering and risked the config being processed in an uninitialized session context. Add pendingSetupMap to buffer the hub-owned session.update payload. The onmessage handler now relays session.created to the browser first, then immediately sends the pending session.update to DashScope — matching the protocol ordering the old browser-side code used (which waited for session.created before sending session.update). Also remove maxHttpBufferSize from the socket.io Engine config. That setting is unrelated to voice backends; upstream/main had no such limit set and it is not introduced by this PR. * fix(voice): use Realtime tool shape for Qwen session.update (not chat-completions) Qwen Realtime session.update expects tools as flat objects: { type: 'function', name, description, parameters } The previous code used the chat-completions shape: { type: 'function', function: { name, description, parameters } } DashScope may reject session.update or silently ignore tools with the nested shape, causing tool calls to fail at runtime. Fix applied in buildQwenSessionUpdateMessage(); test updated to assert flat shape and that no nested `function` key is present. * fix(voice): update Qwen Realtime model, voice, and endpoint for intl service Live-tested against DashScope international API: - Model: qwen3-omni-flash-realtime → qwen3.5-omni-flash-realtime (previous model ID did not exist on DashScope) - Default voice: Cherry → Tina (confirmed from session.created response on qwen3.5-omni-flash-realtime) - Default WS base: dashscope.aliyuncs.com → dashscope-intl.aliyuncs.com (international accounts use the -intl endpoint; China endpoint rejects international API keys; QWEN_REALTIME_WS_URL env var still overrides) * fix(voice): correct Qwen text injection and generalise language handling Two dogfooding fixes verified against live Qwen Realtime session: sendTextMessage: switch from instruction-injection to conversation.item.create Qwen Realtime requires a user conversation item before response.create. The previous approach (updateInstructions + response.create) produced "input messages do not contain elements with role user" errors. Now sends {type:message, role:user, content:[{type:input_text}]} then response.create. sendContextualUpdate is unchanged (instruction-only, no response trigger). Language handling: replace zh-only branch with buildVoiceLanguageBlock() Previously, only language='zh' added any instruction; all other languages (including English) sent no language block, causing Qwen to drift to Chinese. buildVoiceLanguageBlock() now covers three cases: - 'zh'/'zh-*': existing Chinese block (unchanged) - explicit code ('en','es','fr',...): "Always respond in [Language]" - undefined/auto: "Detect the user's language and maintain it" Applied to buildGeminiLiveConfig, buildQwenSessionUpdateMessage, and the client-side currentInstructions mirror in QwenVoiceSession. Also removes the Gemini hub proxy's zh-only filter, which was discarding explicit language selections other than Chinese. * fix(hub): gate Gemini client frames until upstream setupComplete Hub sends its owned setup on upstream open, then waits for Google's setupComplete acknowledgment before flushing queued client frames. isGeminiSetupCompleteFrame() detects the {"setupComplete":{}} message; message() queues instead of forwarding while pendingMap is live. Addresses the repeated Major finding from bot review on PR #743. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(hub): cap Gemini setup-window pending queue at 1 MiB An authenticated client could flood the queue between upstream.onopen and Google's setupComplete acknowledgment. Add pendingBytesMap tracking and close with 1009 if the budget is exceeded. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(gemini): pass all language codes to hub proxy, not just zh Language selection for French, Spanish, Japanese etc. was silently dropped — only 'zh' was forwarded as a query param. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> * fix(voice): expand LANGUAGE_NAMES to cover full ElevenLabs language set Codes like 'no', 'da', 'fi', 'pt-br', 'bg', 'ro', 'cs', 'el', 'ms', 'tl', 'uk', 'hu', 'hr', 'sk' were falling through to raw-code prompts ("Always respond in no"). Now resolve to proper display names. via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> --------- Co-authored-by: HAPI <noreply@hapi.run> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-04 17:56:52 +08:00
weishu	9be0a96af4	feat: add web vibration api fallback for haptic feedback ... Replace no-op browser haptic feedback with Web Vibration API support, allowing haptic feedback when running outside Telegram Mini App. Also remove stale react-syntax-highlighter type reference from tsconfig.	2025-12-18 12:53:11 +08:00
weishu	b4654acb92	init	2025-12-16 15:03:50 +08:00