* feat(cli): cross-flavor inline image display via MCP and ACP Share display_image prompt across MCP-bridge flavors (Cursor, Gemini, Kimi, Codex, Claude, OpenCode), auto-approve the tool in buildHapiMcpBridge, handle ACP image content blocks, and harden generated-image registration with content sniffing. Closes tiann/hapi#956 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): render generated-image cards reliably in chat Keep object URLs stable across refetch, upscale tiny inline images, fetch generated-image bytes with cache no-store (avoid empty 304 bodies), and load hapiMcpUrl from per-session API in hapi-display-image tooling. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(cli+web): display_video MCP for inline mp4/webm (#956) Add display_video alongside display_image, video MIME sniffing with avif guard, web GeneratedImageCard video player, and hapi-display-image auto-routing. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(cli+web): cross-flavor display_video parity with images (#956) Share display_video prompts across MCP-bridge flavors, auto-approve the tool, register mp4/webm via path sniffing, render inline video in web on the existing generated-image RPC path, and restore robust media card fetch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ACP image ordering and inline media source provenance Flush buffered assistant text before async generated_image emit from ACP image blocks (PR #958 review Major). Add optional source metadata on generated-image wire messages (ingress, flavor, toolCallId, toolName) for MCP, ACP, and Codex tool-result paths. Seeds artifact-event follow-up #966. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli+web): address PR #958 review Majors on media order and stale blobs Queue ACP session updates and await async image registration before later events; clear GeneratedImageCard blob state when imageId changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): await ACP queue after late-drain before turn_complete Straggler session/update during drainLateBuffers can queue async image registration; re-await sessionUpdateQueue so generated_image is not emitted after turn_complete. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): route AVIF ftyp brands to display_image in helper Match server-side detectImageMimeType so .avif files are not sent to display_video and rejected as unsupported video. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(#956): agent inline-media doctor and discovery fixes - hapi doctor inline-media: probe bridges, print per-session inline commands - Expose hapiMcpUrl on session list summaries (stops false "no MCP" scans) - Helper script: match cursorSessionId prefixes; HAPI_SESSION_ID path-only mode - ACP bridge prompt: shell fallback + HAPI session id vs agent id rule Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): allow immutable cache for generated media blobs Drop cache: no-store on generated-image fetch so browser can reuse hub immutable responses; on 304 re-read via force-cache (#927, PR review). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(#956): Cursor native MCP overlay; drop user-turn bridge prepend Cursor ACP ignores session/new mcpServers. Write .cursor/mcp.json and run agent mcp enable hapi instead. Remove HAPI_MCP_BRIDGE_PROMPT from user turns on ACP remotes; enrich MCP tool descriptions for discovery. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): preserve non-HAPI mcp.json keys on Cursor overlay cleanup Cleanup only removes or restores the hapi MCP entry instead of rewriting the full pre-session snapshot, so concurrent edits to other servers survive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): handle generated_image in Grok ACP launcher switch Upstream Grok launcher exhaustiveness broke after AgentMessage gained generated_image for cross-flavor inline media. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): rebase fallout for display_video + OpenCode skill lookup Gate display_video in the STDIO bridge, restore OpenCode first-prompt TITLE_INSTRUCTION (skill_lookup), and update tool-list test expectations. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): leave user-owned hapi MCP entry alone on overlay cleanup Only undo mcpServers.hapi when it still matches the exact entry this session installed; concurrent Cursor/user edits of that key survive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): exact-match auto-approve for display_image and display_video Move media tools off substring name/id hints onto the exact-name set so forged lookalike tools are not approved in default permission mode. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): drop dead Cursor bridge prompt; put guidance in MCP descriptions Cursor must not get a user-turn media prepend (prompt-taint). Remove unused HAPI_MCP_BRIDGE_PROMPT_CURSOR and embed DISPLAY_*_PROMPT_CURSOR in the display_image/display_video MCP tool descriptions instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): require user approval for display_image and display_video Those tools read arbitrary local paths into chat; keep them on MCP approval_mode prompt and out of default-mode auto-approve exact names. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: re-trigger Codex PR review after provider 503 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ignore URI-only ACP image blocks that read local disk Passive ACP agentMessageChunk handling must not load file:// or bare paths; local media goes through prompt-gated display_image/display_video. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): allowlist MP4 ftyp brands for video sniffing Reject HEIC/HEIF and other non-video ISO-BMFF containers instead of treating every non-AVIF ftyp as video/mp4. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): keep Cursor ACP startup if MCP overlay fails Wrap installCursorMcpOverlay so a malformed project .cursor/mcp.json cannot abort the session; continue without inline media tools. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail doctor inline-media when checks fail Exit non-zero whenever required checks fail, even if an active hapiMcpUrl bridge is present. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): expect display_video when change_title is disabled Native ACP title mode still exposes display_image and display_video; update startHappyServer test after rebase onto 0.23.4. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): keep ACP title sync synchronous outside media queue session_info_update title forwarding (#1028) must not wait on the async message-handler queue used for inline media ordering. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): sniff media headers only; advertise OpenCode display_video Read 16 bytes for detectMediaTool instead of the whole file, and include hapi_display_video in OPENCODE_NATIVE_TOOL_INSTRUCTION for remote ACP. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): show Cursor generate_image inline in HAPI chat cursor/generate_image only emitted a tool card; register filePath or base64 imageData into generatedImages and emit generated_image so the web chat card renders (issue #956 / swear01 report). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ignore path-only Cursor generate_image reads Path-only filePath registration bypassed permission-gated display_image / display_video MCP tools. Keep base64 imageData only; local paths must go through MCP approval. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): gate inline media base64 length before decode Reject oversized ACP/Cursor base64 payloads by character count so the CLI never allocates past the 25 MB generated-image cap. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): require EBML DocType webm for inline video sniff Bare EBML magic matches Matroska/MKV too; only accept DocType webm. Also restore annotated Playwright cursor in annotatedVideoUseOption. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): read 128-byte header for WebM DocType sniff detectMediaTool only loaded 16 bytes, so EBML DocType webm was often missing and valid WebM files fell through to display_image. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): compare generated-image source by value in reconcile Wire normalization allocates a fresh source object each pass; reference equality forced media-card recomputation on every reload/SSE refresh. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): inject Cursor MCP enable for overlay unit tests installCursorMcpOverlay always spawned `agent mcp enable`; tests now pass a noop so the suite never shells out to a real Cursor binary. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): shell-quote doctor inline-media helper command Paths and session prefixes with spaces/metacharacters broke the copied snippet; JSON.stringify each interpolated argument. Co-authored-by: Cursor <cursoragent@cursor.com> * test(cli): fix doctor inline-media quote path expectation Repo root from scriptPath is three levels up (cli/), not the parent of cli. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli/web): per-session Cursor MCP overlay id and bound tiny-image scale Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): redact generate_image base64 from logs and fix doctor MCP ids Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): harden inline-media doctor for packaged installs and hub headers Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): lock Cursor mcp.json updates and bound ACP media filenames Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): preserve mcp.json mode and token-scoped overlay locks Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): publish Cursor MCP lock owners via link(2) and treat EPERM as alive Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail closed on stale MCP locks; keep concurrent mcp.json top-level keys Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): roll back Cursor MCP overlay when agent mcp enable fails Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): await ACP session queue in suppressUpdatesDuring tests #958 queues handleUpdate for media registration; upstream compact tests assumed sync delivery after restore. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): capture ACP handler at enqueue; keep display_video manual Close two Major review findings on #958: suppress queue leak after restore, and Claude --allowedTools auto-approving local-path video. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: write through symlinked mcp.json; lazy-load inline video Preserve user Cursor MCP symlinks on atomic overlay writes, and require explicit Load video before fetching large generated-video blobs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): use valid TerminalToolDisplayMode in media card test Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): require unique session prefix in display helper Reject ambiguous prefix matches so images/videos cannot land in the wrong HAPI chat when multiple agent session ids share a prefix. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): always cleanup Cursor MCP overlay on teardown Run overlay cleanup in finally so cancelAll/disconnect failures cannot leave a dead hapi-<sessionId> entry in .cursor/mcp.json. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): handle Copilot generated_image; refuse MCP symlinks Unblock typecheck after Antigravity/Copilot merge, and fail closed when .cursor/mcp.json or .cursor is a project-controlled symlink. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: abort restore deliveryMode; prune dead Cursor MCP overlays Unblock web typecheck after steer merge, and recover orphaned hapi-* mcp.json entries via HAPI_MCP_OVERLAY_PID ownership stamps. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): recover dead-PID Cursor MCP overlay locks Token-matched unlock so a crash mid-lock no longer permanently disables inline media; keep live-owner waits identity-safe. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): serialize Cursor MCP stale-lock recovery Acquire an exclusive recovery lock before token-matched unlink so two recoverers cannot remove a successor's live mcp.json lock. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail closed on stale Cursor MCP overlay locks Withdraw racy auto-recovery: pathname check-then-unlink/rename can steal a successor lock. Stale locks throw with an explicit rm hint instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): drop duplicate deliveryMode on abort restore Merge left both steer and queue; keep queue so retries after abort do not re-bind to a later Pi turn. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): bound inline media reads on open fd Close TOCTOU between pathname size check and readFile for display_image / display_video and registerGeneratedImageFromPath. Also preserve non-PID env edits on Cursor MCP overlay cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): repair happyMcpStdioBridge test syntax after merge Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): write Cursor MCP overlay to ~/.cursor, not the project Keep ephemeral hapi-<sessionId> bridges out of the checked-out tree so agents cannot git-add a live loopback URL. Tests inject mcpConfigDir. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): point Cursor MCP diagnostics at ~/.cursor/mcp.json Co-authored-by: Cursor <cursoragent@cursor.com> * feat(media): add audio and file display --------- Co-authored-by: HeavyGee <133152184+heavygee@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Debian <heavygee@oos-linux.in.lockhouse>
hapi-hub
Telegram bot + HTTP API + realtime updates for hapi hub.
What it does
- Telegram bot for notifications and the Mini App entrypoint.
- HTTP API for sessions, messages, permissions, machines, and files.
- Server-Sent Events stream for live updates in the web app.
- Socket.IO channel for CLI connections.
- Serves the web app from
web/distor embedded assets in the single binary. - Persists state in SQLite.
Configuration
See src/configuration.ts for all options.
Required
CLI_API_TOKEN- Base shared secret used by CLI and web login. Clients append:<namespace>for isolation. Auto-generated on first run if not set.
Optional (Telegram)
TELEGRAM_BOT_TOKEN- Token from @BotFather.HAPI_PUBLIC_URL- Public HTTPS URL for Telegram Mini App access. Also used to derive default CORS origins for the web app.
Optional (Voice)
Dictation and voice-assistant provider keys can also be added from Settings → Voice (stored in settings.json under providerCredentials; env vars still win when set at process start).
ELEVENLABS_API_KEY- ElevenLabs API key for voice assistant + dictation.ELEVENLABS_AGENT_ID- Custom ElevenLabs agent ID (auto-created if not set).GEMINI_API_KEY/GOOGLE_API_KEY- Gemini Live voice assistant.DASHSCOPE_API_KEY/QWEN_API_KEY- Qwen Realtime voice assistant.OPENAI_API_KEY- OpenAI dictation (gpt-transcribe/gpt-live-transcribe).DEEPGRAM_API_KEY- Deepgram dictation (nova-3, standard and realtime).GROQ_API_KEY- Groq dictation (whisper-large-v3).TRANSCRIPTION_BASE_URLandTRANSCRIPTION_MODEL- OpenAI-compatible/local transcription endpoint and model.TRANSCRIPTION_API_KEY- Optional bearer token for the OpenAI-compatible endpoint.
Optional
HAPI_LISTEN_HOST- HTTP bind address (default: 127.0.0.1).HAPI_LISTEN_PORT- HTTP port (default: 3006).CORS_ORIGINS- Comma-separated origins, or*.HAPI_HOME- Data directory (default: ~/.hapi).DB_PATH- SQLite database path (default: HAPI_HOME/hapi.db).TELEGRAM_NOTIFICATION- Enable/disable Telegram notifications (default: true).HAPI_RELAY_API- Relay API domain (default: relay.hapi.run).HAPI_RELAY_AUTH- Explicit relay auth key. By default the hub obtains and persists an individually revocable key from the relay. A persisted key rejected with HTTP 403 is discarded and reissued once; an explicitly configured environment key must be updated manually.HAPI_RELAY_FORCE_TCP- Force TCP relay mode (true/1).VAPID_SUBJECT- Contact email/URL for Web Push.
Running
Binary (single executable):
export TELEGRAM_BOT_TOKEN="..."
export CLI_API_TOKEN="shared-secret"
export HAPI_PUBLIC_URL="https://your-domain.example"
hapi hub
hapi server remains supported as an alias.
If you only need web + CLI, you can omit TELEGRAM_BOT_TOKEN.
To enable Telegram, set TELEGRAM_BOT_TOKEN and HAPI_PUBLIC_URL, start the hub, open /app
in the bot chat, and bind the Mini App with CLI_API_TOKEN:<namespace> when prompted.
From source:
bun install
bun run dev:hub
HTTP API
See src/web/routes/ for all endpoints.
Authentication (src/web/routes/auth.ts)
POST /api/auth- Get JWT token (Telegram initData orCLI_API_TOKEN[:namespace]).POST /api/bind- Bind a Telegram account using initData +CLI_API_TOKEN:<namespace>.
Sessions (src/web/routes/sessions.ts)
GET /api/sessions- List all sessions.GET /api/sessions/:id- Get session details.POST /api/sessions/:id/abort- Abort session.POST /api/sessions/:id/switch- Switch session to remote mode.POST /api/sessions/:id/resume- Resume inactive session.POST /api/sessions/:id/upload- Upload file (base64, max 50MB).POST /api/sessions/:id/upload/delete- Delete uploaded file.POST /api/sessions/:id/archive- Archive active session.PATCH /api/sessions/:id- Rename session.DELETE /api/sessions/:id- Delete inactive session.GET /api/sessions/:id/slash-commands- List slash commands.GET /api/sessions/:id/skills- List skills.POST /api/sessions/:id/permission-mode- Set permission mode.POST /api/sessions/:id/model- Set model preference.POST /api/sessions/:id/effort- Set Claude effort preference.
Messages (src/web/routes/messages.ts)
GET /api/sessions/:id/messages- Get messages (paginated).POST /api/sessions/:id/messages- Send message.
Permissions (src/web/routes/permissions.ts)
POST /api/sessions/:id/permissions/:requestId/approve- Approve permission.POST /api/sessions/:id/permissions/:requestId/deny- Deny permission.
Machines (src/web/routes/machines.ts)
GET /api/machines- List online machines.POST /api/machines/:id/spawn- Spawn new session on machine.POST /api/machines/:id/paths/exists- Check if path exists.
Usage (src/web/routes/usage.ts)
GET /api/usage/summary- Get cache-aware token usage for the owner namespace (range=7d|30d|all).
Git/Files (src/web/routes/git.ts)
GET /api/sessions/:id/git-status- Git status.GET /api/sessions/:id/git-diff-numstat- Diff summary.GET /api/sessions/:id/git-diff-file- File-specific diff.GET /api/sessions/:id/file- Read file content.GET /api/sessions/:id/files- File search with ripgrep.
Events (src/web/routes/events.ts)
GET /api/events- SSE stream for live updates.POST /api/visibility- Report client visibility state.
Voice (src/web/routes/voice.ts)
POST /api/voice/token- Get ElevenLabs conversation token.GET /api/voice/transcription/providers- List configured providers and supported modes.POST /api/voice/transcription- Transcribe a bounded recording.POST /api/voice/transcription/realtime-token- Mint a short-lived OpenAI, ElevenLabs, or Deepgram credential.
Push Notifications (src/web/routes/push.ts)
GET /api/push/vapid-public-key- Get VAPID public key.POST /api/push/subscribe- Subscribe to push notifications.DELETE /api/push/subscribe- Unsubscribe.
CLI (src/web/routes/cli.ts)
POST /cli/sessions- Create/load session.GET /cli/sessions/:id- Get session by ID.POST /cli/machines- Create/load machine.GET /cli/machines/:id- Get machine by ID.
Socket.IO
See src/socket/handlers/cli.ts for event handlers.
Namespace: /cli
Client events (CLI to hub)
message- Send message to session.update-metadata- Update session metadata.update-state- Update agent state.session-alive- Keep session active.session-ready- Cursor ACPsession/load(ornewSession) succeeded; hub defers merge/dedup until this arrives on reopen.session-end- Mark session ended.machine-alive- Keep machine online.rpc-register- Register RPC handler.rpc-unregister- Unregister RPC handler.
Terminal events (web to hub)
terminal:create- Open terminal for session.terminal:write- Send input.terminal:resize- Resize dimensions.terminal:close- Close terminal.
Hub events (hub to clients)
update- Broadcast session/message updates.rpc-request- Incoming RPC call.
See src/socket/rpcRegistry.ts for RPC routing.
Telegram Bot
See src/telegram/bot.ts for bot implementation.
Commands
/start- Welcome message with Mini App link./app- Open Mini App.
Features
- Permission request notifications with approve/deny buttons.
- Session ready notifications.
- Deep links to Mini App sessions.
See src/telegram/callbacks.ts for button handlers.
Core Logic
See src/sync/syncEngine.ts for the main session/message manager:
- In-memory session cache with versioning.
- Message pagination and retrieval.
- Permission approval/denial.
- RPC method routing via Socket.IO.
- Event publishing to SSE and Telegram.
- Git operations and file search.
- Activity tracking and timeouts.
Storage
See src/store/index.ts for SQLite persistence:
- Sessions with metadata and agent state.
- Messages with pagination support.
- Machines with runner state.
- Todo extraction from messages.
- Users table for Telegram bindings (includes namespace).
Message content is stored via src/store/contentCodec.ts: oversized strings
inside agent messages (giant tool output) are head+tail truncated at ingest,
and payloads ≥256 chars are zstd-compressed (TEXT = plaintext JSON, BLOB =
zstd). User prompts are never truncated — queued rows are delivered to the CLI
verbatim.
Maintenance scripts (run with the hub stopped before swapping files):
scripts/compact-db.ts— retroactively truncate + compress + VACUUM an existing DB into a new file (the source is only opened read-only).scripts/cleanup-sessions.ts— bulk-delete sessions by message count, path glob, or first-message pattern.
Source structure
src/web/- HTTP service and routes.src/socket/- Socket.IO setup and handlers.src/socket/handlers/cli/- Modular CLI handlers.src/telegram/- Telegram bot.src/sync/- Core session/message logic.src/store/- SQLite persistence.src/sse/- Server-Sent Events.src/config/- Configuration loading and generation.src/notifications/- Push and Telegram notifications.src/visibility/- Client visibility tracking.
Security model
Access is controlled by:
- Telegram initData verification plus bound Telegram users (bound via
CLI_API_TOKEN:<namespace>). CLI_API_TOKENbase secret for CLI and browser access (namespace is appended by clients).
Transport security depends on HTTPS in front of the hub.
Build for deployment
From the repo root:
bun run build:hub
bun run build:web
The hub build output is hub/dist/index.js, and the web assets are in web/dist.
Networking notes
- Telegram Mini Apps require HTTPS and a public URL. If the hub has no public IP, use Cloudflare Tunnel or Tailscale and set
HAPI_PUBLIC_URLto the HTTPS endpoint. - If the web app is hosted on a different origin, set
CORS_ORIGINS(orHAPI_PUBLIC_URL) to include that static host origin.
Standalone web hosting
The web UI can be hosted separately from the hub (for example on GitHub Pages or Cloudflare Pages):
- Build and deploy
web/distfrom the repo root. - Set
CORS_ORIGINS(orHAPI_PUBLIC_URL) to the static host origin. - Open the static site, click the Hub button on the login screen, and enter the hapi hub origin.
Leaving the hub override empty preserves the default same-origin behavior when the hub serves the web assets directly.