mirror of
https://github.com/wu736139669/hapi.git
synced 2026-10-07 19:06:11 +00:00
* feat(cli): cross-flavor inline image display via MCP and ACP Share display_image prompt across MCP-bridge flavors (Cursor, Gemini, Kimi, Codex, Claude, OpenCode), auto-approve the tool in buildHapiMcpBridge, handle ACP image content blocks, and harden generated-image registration with content sniffing. Closes tiann/hapi#956 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): render generated-image cards reliably in chat Keep object URLs stable across refetch, upscale tiny inline images, fetch generated-image bytes with cache no-store (avoid empty 304 bodies), and load hapiMcpUrl from per-session API in hapi-display-image tooling. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(cli+web): display_video MCP for inline mp4/webm (#956) Add display_video alongside display_image, video MIME sniffing with avif guard, web GeneratedImageCard video player, and hapi-display-image auto-routing. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(cli+web): cross-flavor display_video parity with images (#956) Share display_video prompts across MCP-bridge flavors, auto-approve the tool, register mp4/webm via path sniffing, render inline video in web on the existing generated-image RPC path, and restore robust media card fetch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ACP image ordering and inline media source provenance Flush buffered assistant text before async generated_image emit from ACP image blocks (PR #958 review Major). Add optional source metadata on generated-image wire messages (ingress, flavor, toolCallId, toolName) for MCP, ACP, and Codex tool-result paths. Seeds artifact-event follow-up #966. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli+web): address PR #958 review Majors on media order and stale blobs Queue ACP session updates and await async image registration before later events; clear GeneratedImageCard blob state when imageId changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): await ACP queue after late-drain before turn_complete Straggler session/update during drainLateBuffers can queue async image registration; re-await sessionUpdateQueue so generated_image is not emitted after turn_complete. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): route AVIF ftyp brands to display_image in helper Match server-side detectImageMimeType so .avif files are not sent to display_video and rejected as unsupported video. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(#956): agent inline-media doctor and discovery fixes - hapi doctor inline-media: probe bridges, print per-session inline commands - Expose hapiMcpUrl on session list summaries (stops false "no MCP" scans) - Helper script: match cursorSessionId prefixes; HAPI_SESSION_ID path-only mode - ACP bridge prompt: shell fallback + HAPI session id vs agent id rule Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): allow immutable cache for generated media blobs Drop cache: no-store on generated-image fetch so browser can reuse hub immutable responses; on 304 re-read via force-cache (#927, PR review). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(#956): Cursor native MCP overlay; drop user-turn bridge prepend Cursor ACP ignores session/new mcpServers. Write .cursor/mcp.json and run agent mcp enable hapi instead. Remove HAPI_MCP_BRIDGE_PROMPT from user turns on ACP remotes; enrich MCP tool descriptions for discovery. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): preserve non-HAPI mcp.json keys on Cursor overlay cleanup Cleanup only removes or restores the hapi MCP entry instead of rewriting the full pre-session snapshot, so concurrent edits to other servers survive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): handle generated_image in Grok ACP launcher switch Upstream Grok launcher exhaustiveness broke after AgentMessage gained generated_image for cross-flavor inline media. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): rebase fallout for display_video + OpenCode skill lookup Gate display_video in the STDIO bridge, restore OpenCode first-prompt TITLE_INSTRUCTION (skill_lookup), and update tool-list test expectations. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): leave user-owned hapi MCP entry alone on overlay cleanup Only undo mcpServers.hapi when it still matches the exact entry this session installed; concurrent Cursor/user edits of that key survive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): exact-match auto-approve for display_image and display_video Move media tools off substring name/id hints onto the exact-name set so forged lookalike tools are not approved in default permission mode. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): drop dead Cursor bridge prompt; put guidance in MCP descriptions Cursor must not get a user-turn media prepend (prompt-taint). Remove unused HAPI_MCP_BRIDGE_PROMPT_CURSOR and embed DISPLAY_*_PROMPT_CURSOR in the display_image/display_video MCP tool descriptions instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): require user approval for display_image and display_video Those tools read arbitrary local paths into chat; keep them on MCP approval_mode prompt and out of default-mode auto-approve exact names. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: re-trigger Codex PR review after provider 503 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ignore URI-only ACP image blocks that read local disk Passive ACP agentMessageChunk handling must not load file:// or bare paths; local media goes through prompt-gated display_image/display_video. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): allowlist MP4 ftyp brands for video sniffing Reject HEIC/HEIF and other non-video ISO-BMFF containers instead of treating every non-AVIF ftyp as video/mp4. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): keep Cursor ACP startup if MCP overlay fails Wrap installCursorMcpOverlay so a malformed project .cursor/mcp.json cannot abort the session; continue without inline media tools. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail doctor inline-media when checks fail Exit non-zero whenever required checks fail, even if an active hapiMcpUrl bridge is present. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): expect display_video when change_title is disabled Native ACP title mode still exposes display_image and display_video; update startHappyServer test after rebase onto 0.23.4. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): keep ACP title sync synchronous outside media queue session_info_update title forwarding (#1028) must not wait on the async message-handler queue used for inline media ordering. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): sniff media headers only; advertise OpenCode display_video Read 16 bytes for detectMediaTool instead of the whole file, and include hapi_display_video in OPENCODE_NATIVE_TOOL_INSTRUCTION for remote ACP. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): show Cursor generate_image inline in HAPI chat cursor/generate_image only emitted a tool card; register filePath or base64 imageData into generatedImages and emit generated_image so the web chat card renders (issue #956 / swear01 report). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ignore path-only Cursor generate_image reads Path-only filePath registration bypassed permission-gated display_image / display_video MCP tools. Keep base64 imageData only; local paths must go through MCP approval. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): gate inline media base64 length before decode Reject oversized ACP/Cursor base64 payloads by character count so the CLI never allocates past the 25 MB generated-image cap. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): require EBML DocType webm for inline video sniff Bare EBML magic matches Matroska/MKV too; only accept DocType webm. Also restore annotated Playwright cursor in annotatedVideoUseOption. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): read 128-byte header for WebM DocType sniff detectMediaTool only loaded 16 bytes, so EBML DocType webm was often missing and valid WebM files fell through to display_image. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): compare generated-image source by value in reconcile Wire normalization allocates a fresh source object each pass; reference equality forced media-card recomputation on every reload/SSE refresh. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): inject Cursor MCP enable for overlay unit tests installCursorMcpOverlay always spawned `agent mcp enable`; tests now pass a noop so the suite never shells out to a real Cursor binary. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): shell-quote doctor inline-media helper command Paths and session prefixes with spaces/metacharacters broke the copied snippet; JSON.stringify each interpolated argument. Co-authored-by: Cursor <cursoragent@cursor.com> * test(cli): fix doctor inline-media quote path expectation Repo root from scriptPath is three levels up (cli/), not the parent of cli. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli/web): per-session Cursor MCP overlay id and bound tiny-image scale Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): redact generate_image base64 from logs and fix doctor MCP ids Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): harden inline-media doctor for packaged installs and hub headers Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): lock Cursor mcp.json updates and bound ACP media filenames Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): preserve mcp.json mode and token-scoped overlay locks Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): publish Cursor MCP lock owners via link(2) and treat EPERM as alive Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail closed on stale MCP locks; keep concurrent mcp.json top-level keys Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): roll back Cursor MCP overlay when agent mcp enable fails Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): await ACP session queue in suppressUpdatesDuring tests #958 queues handleUpdate for media registration; upstream compact tests assumed sync delivery after restore. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): capture ACP handler at enqueue; keep display_video manual Close two Major review findings on #958: suppress queue leak after restore, and Claude --allowedTools auto-approving local-path video. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: write through symlinked mcp.json; lazy-load inline video Preserve user Cursor MCP symlinks on atomic overlay writes, and require explicit Load video before fetching large generated-video blobs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): use valid TerminalToolDisplayMode in media card test Co-authored-by: Cursor <cursoragent@cursor.com> * fix(scripts): require unique session prefix in display helper Reject ambiguous prefix matches so images/videos cannot land in the wrong HAPI chat when multiple agent session ids share a prefix. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): always cleanup Cursor MCP overlay on teardown Run overlay cleanup in finally so cancelAll/disconnect failures cannot leave a dead hapi-<sessionId> entry in .cursor/mcp.json. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): handle Copilot generated_image; refuse MCP symlinks Unblock typecheck after Antigravity/Copilot merge, and fail closed when .cursor/mcp.json or .cursor is a project-controlled symlink. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: abort restore deliveryMode; prune dead Cursor MCP overlays Unblock web typecheck after steer merge, and recover orphaned hapi-* mcp.json entries via HAPI_MCP_OVERLAY_PID ownership stamps. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): recover dead-PID Cursor MCP overlay locks Token-matched unlock so a crash mid-lock no longer permanently disables inline media; keep live-owner waits identity-safe. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): serialize Cursor MCP stale-lock recovery Acquire an exclusive recovery lock before token-matched unlink so two recoverers cannot remove a successor's live mcp.json lock. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail closed on stale Cursor MCP overlay locks Withdraw racy auto-recovery: pathname check-then-unlink/rename can steal a successor lock. Stale locks throw with an explicit rm hint instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(web): drop duplicate deliveryMode on abort restore Merge left both steer and queue; keep queue so retries after abort do not re-bind to a later Pi turn. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): bound inline media reads on open fd Close TOCTOU between pathname size check and readFile for display_image / display_video and registerGeneratedImageFromPath. Also preserve non-PID env edits on Cursor MCP overlay cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): repair happyMcpStdioBridge test syntax after merge Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): write Cursor MCP overlay to ~/.cursor, not the project Keep ephemeral hapi-<sessionId> bridges out of the checked-out tree so agents cannot git-add a live loopback URL. Tests inject mcpConfigDir. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): point Cursor MCP diagnostics at ~/.cursor/mcp.json Co-authored-by: Cursor <cursoragent@cursor.com> * feat(media): add audio and file display --------- Co-authored-by: HeavyGee <133152184+heavygee@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Debian <heavygee@oos-linux.in.lockhouse>
221 lines
8.3 KiB
JavaScript
221 lines
8.3 KiB
JavaScript
#!/usr/bin/env bun
|
|
/**
|
|
* Post a local file to a HAPI session via display_image / display_video / display_media MCP.
|
|
*
|
|
* Uses session.metadata.hapiMcpUrl (published at MCP server start) so we hit the MCP
|
|
* endpoint, not the session hook server on another loopback port in the same process.
|
|
*
|
|
* Usage:
|
|
* # inside a wrapped session (self-targets via $HAPI_SESSION_ID — no list):
|
|
* bun scripts/tooling/hapi-display-image.mjs <media-path> [title]
|
|
* # explicit self:
|
|
* bun scripts/tooling/hapi-display-image.mjs self <media-path> [title]
|
|
* # explicit other session:
|
|
* bun scripts/tooling/hapi-display-image.mjs <session-id-prefix> <media-path> [title]
|
|
*
|
|
* Self-resolution (tiann/hapi#1119): $HAPI_SESSION_ID → GET /api/sessions/:id directly.
|
|
* Picks the strict image/video tool when recognized, else display_media.
|
|
* Prefer the MCP tools when available; this script is the shell fallback.
|
|
*/
|
|
|
|
import { closeSync, openSync, readSync, readFileSync, lstatSync } from 'node:fs'
|
|
import { Client } from '@modelcontextprotocol/sdk/client/index.js'
|
|
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'
|
|
|
|
const HAPI_HOST = process.env.HAPI_HOST ?? 'http://localhost:3006'
|
|
const SETTINGS = process.env.HAPI_SETTINGS ?? `${process.env.HOME}/.hapi/settings.json`
|
|
|
|
const SELF_TOKENS = new Set(['self', '@self', '@me', 'current', '-'])
|
|
|
|
function isFile(p) {
|
|
try {
|
|
return lstatSync(p).isFile()
|
|
} catch {
|
|
return false
|
|
}
|
|
}
|
|
|
|
function sessionMatchesPrefix(session, prefix) {
|
|
if (typeof session.id === 'string' && session.id.startsWith(prefix)) {
|
|
return true
|
|
}
|
|
const meta = session.metadata ?? {}
|
|
const agentIds = [
|
|
meta.agentSessionId,
|
|
meta.cursorSessionId,
|
|
meta.codexSessionId,
|
|
meta.claudeSessionId,
|
|
meta.geminiSessionId,
|
|
meta.opencodeSessionId,
|
|
meta.kimiSessionId,
|
|
]
|
|
return agentIds.some((id) => typeof id === 'string' && id.startsWith(prefix))
|
|
}
|
|
|
|
function readHeader(path, length = 16) {
|
|
const fd = openSync(path, 'r')
|
|
try {
|
|
const head = Buffer.alloc(length)
|
|
const bytesRead = readSync(fd, head, 0, head.length, 0)
|
|
return head.subarray(0, bytesRead)
|
|
} finally {
|
|
closeSync(fd)
|
|
}
|
|
}
|
|
|
|
function detectMediaTool(path) {
|
|
// EBML DocType can sit well past the first 16 bytes; match generatedImages scan window.
|
|
const head = readHeader(path, 128)
|
|
if (head.length >= 12 && head.subarray(4, 8).toString('ascii') === 'ftyp') {
|
|
const brand = head.subarray(8, 12).toString('ascii')
|
|
if (brand === 'avif' || brand === 'avis') return 'display_image'
|
|
return 'display_media'
|
|
}
|
|
if (head.length >= 4 && head[0] === 0x1a && head[1] === 0x45 && head[2] === 0xdf && head[3] === 0xa3) {
|
|
// EBML is shared by WebM and Matroska — only route DocType "webm" to video.
|
|
const limit = Math.min(head.length, 128)
|
|
for (let i = 4; i + 3 < limit; i += 1) {
|
|
if (head[i] !== 0x42 || head[i + 1] !== 0x82) continue
|
|
const sizeByte = head[i + 2]
|
|
if ((sizeByte & 0x80) === 0) continue
|
|
const len = sizeByte & 0x7f
|
|
if (len === 0 || i + 3 + len > limit) continue
|
|
const docType = head.subarray(i + 3, i + 3 + len).toString('ascii')
|
|
if (docType === 'webm') return 'display_video'
|
|
break
|
|
}
|
|
return 'display_media'
|
|
}
|
|
if (head.length >= 8 && head.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) return 'display_image'
|
|
if (head.length >= 3 && head[0] === 0xff && head[1] === 0xd8 && head[2] === 0xff) return 'display_image'
|
|
if (head.length >= 6 && ['GIF87a', 'GIF89a'].includes(head.subarray(0, 6).toString('ascii'))) return 'display_image'
|
|
if (head.length >= 12 && head.subarray(0, 4).toString('ascii') === 'RIFF' && head.subarray(8, 12).toString('ascii') === 'WEBP') return 'display_image'
|
|
return 'display_media'
|
|
}
|
|
|
|
// Arg shapes (backward compatible):
|
|
// <media> [title] → self-target current session
|
|
// <self-token> <media> [title] → self-target, explicit
|
|
// <session-id-prefix> <media> [title] → explicit session
|
|
const args = process.argv.slice(2)
|
|
let sessionArg
|
|
let imagePath
|
|
let title
|
|
if (args.length > 0 && isFile(args[0]) && !SELF_TOKENS.has(args[0])) {
|
|
sessionArg = null
|
|
imagePath = args[0]
|
|
title = args[1]
|
|
} else {
|
|
sessionArg = args[0]
|
|
imagePath = args[1]
|
|
title = args[2]
|
|
}
|
|
|
|
if (!imagePath) {
|
|
console.error('usage: hapi-display-image.mjs [<session-id-prefix>|self] <media-path> [title]')
|
|
console.error(' or: HAPI_SESSION_ID=<uuid> hapi-display-image.mjs <media-path> [title]')
|
|
process.exit(2)
|
|
}
|
|
|
|
if (!isFile(imagePath)) {
|
|
console.error(`not a file: ${imagePath}`)
|
|
process.exit(2)
|
|
}
|
|
|
|
const token = process.env.CLI_API_TOKEN ?? JSON.parse(readFileSync(SETTINGS, 'utf8')).cliApiToken
|
|
if (!token) {
|
|
console.error('missing CLI_API_TOKEN env and no cliApiToken in settings')
|
|
process.exit(2)
|
|
}
|
|
const authRes = await fetch(`${HAPI_HOST}/api/auth`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({ accessToken: token }),
|
|
})
|
|
if (!authRes.ok) {
|
|
console.error('auth failed', authRes.status)
|
|
process.exit(3)
|
|
}
|
|
const { token: jwt } = await authRes.json()
|
|
const authHeaders = { Authorization: `Bearer ${jwt}` }
|
|
|
|
async function fetchSessionDetail(sessionId) {
|
|
const detailRes = await fetch(`${HAPI_HOST}/api/sessions/${encodeURIComponent(sessionId)}`, {
|
|
headers: authHeaders,
|
|
})
|
|
if (!detailRes.ok) {
|
|
return null
|
|
}
|
|
const detailBody = await detailRes.json()
|
|
return detailBody.session ?? detailBody
|
|
}
|
|
|
|
async function listSessions() {
|
|
const sessionsRes = await fetch(`${HAPI_HOST}/api/sessions?limit=500`, {
|
|
headers: authHeaders,
|
|
})
|
|
const sessionsBody = await sessionsRes.json()
|
|
return sessionsBody.sessions ?? sessionsBody
|
|
}
|
|
|
|
let session
|
|
const wantsSelf = !sessionArg || SELF_TOKENS.has(sessionArg)
|
|
const hapiSessionId = process.env.HAPI_SESSION_ID?.trim()
|
|
|
|
if (wantsSelf) {
|
|
if (!hapiSessionId) {
|
|
console.error(
|
|
'cannot self-resolve session: $HAPI_SESSION_ID is not set. '
|
|
+ 'Pass an explicit <session-id-prefix>, or run inside a HAPI-wrapped agent session.',
|
|
)
|
|
process.exit(4)
|
|
}
|
|
// Preferred path (#1119): direct GET, no /api/sessions list.
|
|
session = await fetchSessionDetail(hapiSessionId)
|
|
if (!session) {
|
|
console.error(`GET /api/sessions/${hapiSessionId} failed (HAPI_SESSION_ID set but hub has no such row)`)
|
|
process.exit(4)
|
|
}
|
|
} else {
|
|
// Explicit id/prefix: full uuid → direct GET; otherwise list + prefix match
|
|
// (HAPI id or agent session ids such as cursorSessionId).
|
|
const looksFull = /^[0-9a-f-]{36}$/i.test(sessionArg)
|
|
if (looksFull) {
|
|
session = await fetchSessionDetail(sessionArg)
|
|
}
|
|
if (!session) {
|
|
const sessions = await listSessions()
|
|
const matches = sessions.filter((candidate) => sessionMatchesPrefix(candidate, sessionArg))
|
|
if (matches.length !== 1) {
|
|
console.error(
|
|
matches.length === 0
|
|
? `no session for prefix ${sessionArg} (use HAPI session id from /sessions/<uuid>, not cursorSessionId alone)`
|
|
: `ambiguous session prefix ${sessionArg} (${matches.length} matches); use a full HAPI session id`,
|
|
)
|
|
process.exit(4)
|
|
}
|
|
const listed = matches[0]
|
|
// List summaries may omit hapiMcpUrl; detail fetch always has it when present.
|
|
session = await fetchSessionDetail(listed.id) ?? listed
|
|
}
|
|
}
|
|
|
|
const mcpUrl = session.metadata?.hapiMcpUrl
|
|
if (!mcpUrl) {
|
|
console.error('session has no hapiMcpUrl metadata (restart session CLI after MCP server start)')
|
|
process.exit(5)
|
|
}
|
|
|
|
console.error(`hapi-display-image: session=${session.id} mcp=${mcpUrl}`)
|
|
|
|
const mediaTool = detectMediaTool(imagePath)
|
|
const client = new Client({ name: 'hapi-display-image', version: '1.0.0' }, { capabilities: {} })
|
|
const transport = new StreamableHTTPClientTransport(new URL(mcpUrl))
|
|
await client.connect(transport)
|
|
const result = await client.callTool({
|
|
name: mediaTool,
|
|
arguments: { path: imagePath, title: title ?? undefined },
|
|
})
|
|
await client.close()
|
|
console.log(JSON.stringify(result, null, 2))
|