mirror of
https://github.com/wu736139669/hapi.git
synced 2026-11-08 01:43:07 +00:00
fix(runner,hub): cross-platform orphan reap on archive / tracking loss (#1911)
* fix(runner,hub): reap detached driver CLI on archive and tracking loss Archive always verifies via stopRunnerSession; runner stopSession scans argv orphans and awaits webhook-timeout tree-kill so PPID=1 children cannot accumulate after KillSession miss or map drop. Fixes #1910 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): keep --hapi-session-id reap-only; test orphan kill path Claude/cursor swallow the stamp without bootstrap reuse, and reapRunnerSpawnedOrphans is unit-tested as the stopSession map-miss path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): address #1911 CI and bot Majors Use Vitest in orphanReap tests, add Pi session.on mock, refuse archive when machine StopSession is unreachable, trust full tree-kill results, and argv-sweep orphans after every mapped stop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): gate late-webhook kill on timeout orphans; fail closed on scan error Track webhook-timeout PIDs separately so recovered shared roots are not killed, and treat ps-list failures as still_alive rather than empty. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): keep shared Codex wrapper alive when archiving one root StopSession must detach a single shared root without tree-killing the wrapper PID while sibling roots remain, including after KillSession has already marked the runtime inactive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): keep shared Codex siblings after restart tracking loss Before persisted-PID or argv orphan kills, consult the durable runtime registry for active sibling roots on the same wrapper PID so archiving one root after a runner restart cannot tree-kill live shared sessions. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): deliver hub-authored archive metadata to reconnecting CLIs When KillSession misses and archive is written via markSessionArchivedFromHub, emit Socket.IO update-session to the session room. On reconnect, ApiSessionClient reconciles GET /cli/sessions/:id metadata so hub-archived still fires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): argv-scan orphans even when shared siblings remain Remove session-wide early returns that skipped orphan reaping whenever any shared wrapper had active siblings. Keep only the PID-specific filter so older untracked CLIs with the same HAPI id are still reaped. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): protect tracked shared wrappers when runtime registry is empty Argv orphan sweeps must exclude in-memory shared wrapper PIDs that still host sibling roots, not only durable registry siblings — otherwise a missing runtime record lets the sweep tree-kill the wrapper while archiving one root. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): adopt untracked nonshared CLIs after runner restart When a runner-spawned Claude webhook arrives with no TrackedSession (typical after restart mid-bootstrap), durably adopt the PID instead of ignoring it so StopSession can reap. Keep killing only nonshared timeout orphans; never kill shared Codex wrappers on late webhooks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): keep recovered shared wrappers for registry siblings After restart adoption tracks only a newly reported Codex root, archiving it must consult the durable registry before tree-killing the wrapper PID so older active sibling roots survive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub): do not archive on KillSession ack when StopSession is unknown KillSession returns before cleanupAndExit finishes. When the runner cannot find the PID (unknown), wait for the session KillSession handler to drop before archiving; refuse if the CLI socket remains reachable. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): confirm KillSession pid via StopSession before archive KillSession now returns the CLI pid. When session-id StopSession is unknown, archive confirms that OS pid (without treating socket loss as exit) and still refuses when confirmation stays unknown or still_alive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner,hub): tree-kill adopted CLIs and require PID start markers Adopted post-restart runner sessions now use killProcessTreeByPid when no ChildProcess handle exists. KillSession carries a process-start marker so raw PID confirmation cannot tree-kill a reused OS pid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner,hub): verify adopted PID markers and hub-archive after stop Adopted sessions compare the persisted start marker before tree-kill and drop stale tracking on mismatch. After confirmed StopSession, hub-authors archive metadata when the CLI did not finish its flush. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): refuse unconfirmed shared/adopted stops Keep shared wrappers when siblings remain, but return unknown unless runtimeControl already ended the root. Adopted PIDs without a start marker no longer tree-kill by tracked PID alone. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): require inactive binding before sibling-kept archive Siblings alone must not flip StopSession to stopped on retry, while an acknowledged inactive Codex registry binding remains stop proof. Handle PID- markers before any tracked kill so reuse cannot nuke strangers. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): distinguish orphan reap from unknown keep-wrapper base Pass unknown into finishWithOrphanSweep so a successful argv reap stays stopped instead of being overwritten by an absent registry binding. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): honor orphan reap when adopted PID lacks a start marker finishWithOrphanSweep with an unknown base lets a successful argv kill report stopped instead of forcing unknown after the marker probe fails. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub): confirm archive via metadata.hostPid tombstone When StopSession(sessionId) is unknown and KillSession supplies no confirmable pid+marker, probe PID-<metadata.hostPid>. already_gone allows archive; still_alive/unknown still refuse (#1911 Peer #1820). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): escalate Windows taskkill to /F like SIGTERM→SIGKILL Soft taskkill /T is refused on win32 console trees; orphanReap left orphans still_alive. waitForProcessToDie now force-escalates on both platforms so cross-platform reaping matches. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): widen process.kill mock signal to string | number Match @types/node process.kill(pid, signal?: string | number) so the Windows taskkill escalation mocks typecheck on tip 54f141368. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): tasklist liveness + immediate /F after soft taskkill refuse Windows orphan reap still failed in 5ms on Teemo: soft taskkill refused but escalate waited on an unreliable process.kill(0) probe. Prefer tasklist for win32 liveness and escalate to /F as soon as the PID is still live after soft kill. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): coerce win32 ps-list PIDs and escalate soft taskkill to /F String PIDs from ps-list made Number.isFinite fail so orphan kill returned still_alive without calling taskkill. Coerce PIDs, prefer tasklist liveness, and escalate soft taskkill to /F like SIGTERM→SIGKILL. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): Windows orphan scan via CIM CommandLine + taskkill /F escalate ps-list/fastlist is missing CommandLine and often missing from single-exe bundles, so Teemo orphan reap returned scan_failed (logged as live PIDs). List via Win32_Process CIM, coerce PIDs, escalate soft taskkill to /F. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): grace-wait after successful soft taskkill before /F Escalate immediately only when soft taskkill is refused. When soft succeeds but the PID is still draining, honor the existing grace period before force (Codex Major on #1911). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): re-verify start marker before orphan argv tree-kill Capture getProcessStartMarker after the argv match list, re-check immediately before killTree, and skip on mismatch/null-while-alive so PID reuse cannot tree-kill an unrelated process on the orphan fallback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub): preallocate HAPI id before fresh machine spawn Fresh POST /machines/:id/spawn omitted existingSessionId, so argv never got a reap stamp. Preallocate the hub row (fork/clear pattern) and pass that id so a runner restart before the first webhook remains reapeable. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): close three post-prealloc unreapable paths Pass --hapi-session-id into Claude/cursor/kimi/copilot bootstrap so the preallocated hub row is used. Treat null marker re-check while alive as still_alive (not stopped). Keep the preallocated stub when spawn-fail StopSession cannot confirm the child is gone. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): adopt-stub via reservedSessionId, not existingSessionId `--hapi-session-id` binds a fresh create bootstrap to a hub-preallocated UUID (getOrCreateSession id) without taking the reopen path. Non-UUID stamps stay reap-only to match hub create schema. Restores integration spawn invariants broken by mapping hapi→existingSessionId. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): close Opus cold-read B1–B3 for #1911 B1: capture start markers from the same process snapshot as argv match (POSIX ps lstart / Win32 CreationDate); re-check before kill. Continue other orphans when one target is unconfirmed. B2: Windows killProcessTreeByPid verifies every collected descendant, not root-only — taskkill /T exit 0 is signalled, not gone. B3: runner sets childStarted:false on pre-exec rejection; hub deletes the preallocated stub without StopSession. Post-exec ambiguity still keeps the row until stop confirms. Terminal archives no longer refuse when no runner is connected. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): explicit adopt for hub-preallocated session stubs Fresh machine spawn preallocates with tag machine-spawn:<uuid>; CLI create with a different tag and the same id 409'd on identity conflict. Add adoptPreallocatedSession (hub-internal stub check) and POST /cli/sessions { adopt: true, id } so create binds the reserved row and overwrites tag/metadata without guessing the hub tag. Tests fail-first on the real prealloc→CLI create path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): preserve snapshot markers + unify Windows CIM start markers Runner stopSession was still on findOrphans (PID-only), discarding same-snapshot markers and reopening the PID-reuse kill window. Switch both call sites to findTargets. Force UTC ISO 'o' CreationDate for list and single-PID probe so ConvertTo-Json cannot disagree with getProcessStartMarker on WinPS 5.1. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): exercise production orphan sweep wiring in tests Extract findStopSessionOrphanTargets as the stopSession discovery path both run.ts sweeps call. Add tests that drive that helper end-to-end (not an injected findOrphans seam), assert run.ts source wiring, and cover CIM-list ISO vs probe ISO agreement plus the broken WinPS shape mismatch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): never delete live spawn on RPC timeout; release stub tags Ambiguous machine-spawn failures (timeout / unset childStarted) no longer call stopRunnerSession or deleteSession — that path killed healthy late boots and CASCADE-wiped transcripts. Pre-exec delete still requires a real machine-spawn stub (store tag check). Metadata updates release the stub tag so reopen flavors are not permanently adoptable; adopt refuses archived stubs. POSIX pgrep ENOENT fails closed instead of root-only verify. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): parseable Windows tree scan + fail closed on CIM failure Space-joined PowerShell for collectWindowsProcessTree was a WinPS parse error ($seen=@{} $bfs=@()), so every scan fell open to [root] and B2 false-stopped again. Join with newlines; return scan_failed on error/empty; killProcessTreeByPid refuses root-only verify. Assert on generated command text (Teemo-confirmed) so mocks cannot hide the next malformed script. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): reopen stamps existing-session-id; scan-fail signals root Opus Critical on #1911: runner always stamps --existing-session-id for known hub ids so Claude/Kimi/Copilot reopen does not hit adopt-stub 409. On pgrep/CIM tree-scan failure still signal the verified root but never claim stopped. Archive when both KillSession and machine RPC are missing. Empty Win process-list stdout and strict Codex registry reads fail closed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): OK: sentinel for Windows tree scan; adopt CAS CIM failure under SilentlyContinue printed root-only with exit 0 — byte-identical to a healthy childless tree, so scan_failed never fired and killProcessTreeByPid false-stopped (#1911 Overseer B1). Require ErrorActionPreference=Stop, trap exit 1, and an OK: success sentinel; parse anything else as scan_failed. Same Stop/trap on the machine-wide list command. Execute the generated scripts under Docker pwsh in CI when the image is present. Treat pgrep status=null as scan_failed; coarsen Windows tasklist poll; wrap adoptPreallocatedSession in a transaction with stub-tag CAS. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): reservedSessionId for fresh prealloc; signal Win survivors Fresh machine-spawn stubs must not stamp --existing-session-id — that sends Codex down reopen without a thread binding (#1911 bot Major) and re-broke local HTTP non-UUID hints as reopen (round-4). Hub passes prealloc as reservedSessionId; buildCliArgs stamps --hapi-session-id for reserved/sessionId and --existing-session-id only for live reopen. Codex parses --hapi-session-id and adopts via bootstrapSession. Windows tree kill now taskkills surviving pre-scan PIDs after /T misses broken intermediate links. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): fail-loud Docker pwsh CI; scope Codex registry; stub leak CI skipped all four windowsProcessTree.pwsh tests on 48bb (image inspect without pull). Pull the image in test.yml and throw under CI when missing. Scope strict runtime-registry failure to Codex stop contexts so one bad JSON cannot block archive machine-wide. Stamp childStarted:false on machine RPC outside_workspace_roots. Add Codex reservedSessionId prealloc and runSharedCodex regression coverage for the #1911 Critical. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): single-use reservedSessionId; skip UUID HTTP hints Consume the machine-spawn reservation on first prepare() so a second Codex root/fork cannot re-adopt a consumed stub (409). Stop stamping UUID local-HTTP sessionId as --hapi-session-id (adopt hard-fail). Update stale hub spawn mocks for the reservedSessionId arity. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub): archive never-started machine-spawn stubs on unknown Ambiguous keep-stubs (startedBy=runner, no hostPid, still machine-spawn tagged) made StopSession return unknown forever while the runner was online, so archive threw permanently. Treat unknown + prealloc stub + no hostPid as already_gone. Also reject spawn success when the runner reports a different id than the preallocated stub. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): refuse dual-RPC-miss archive; generation-check Win survivors Both KillSession and StopSession missing is not proof a detached CLI exited — keep the row unconfirmed until the runner reconnects. Windows tree kill now re-checks process-start markers before individually signalling survivors so PID reuse is not killed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): refuse archived reopen in bootstrapExistingSession Hub-archived rows could be resurrected when a late-booting child hit --existing-session-id: buildSessionMetadata stamped lifecycleState=running and updateMetadata overwrote the archive. Adopt already rejected archived stubs; reopen now matches. Intentional revive still goes through hub reopenSession first. Correct stub-hatch comment (no post-archive argv sweep). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): reject un-archive in updateSessionMetadata CAS Hub-side transactional guard (Discovery #1911 M1): refuse metadata writes that clear archived lifecycle unless allowUnarchive (reopen clear). Client stops infinite updateMetadata backoff when mismatch applies archived. Clear archive immediately before spawn so Pi/PTY attempt snapshots stay valid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(hub,cli): success+preserve hub archive; M3 reservedSessionId split Replace version-mismatch refuse (livelock in CAS backoff) with success + merge-preserve of hub-archived fields unless allowUnarchive. Ack path routes through noteHubArchived so the CLI exits (criterion 6). Clear- before-spawn remains the authorized revive. Thread --hapi-session-id to reservedSessionId for shared-parser flavors (adopt≠reopen). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): fail-closed orphan ps signal; PID- registry; spawn timeouts Overseer B1: signalled ps (status null) is scan_failed, not empty table. B2: unreadable registry on PID-* stops returns null (no soft []). B3: 10s timeout on process/orphanReap spawn.sync. Narrow bootstrap archived belt to archivedBy=hub to match hub merge-preserve. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli,hub): hubArchived latch; PTY archiveSnapshot durability Criterion 6: noteHubArchived latches hubArchived so late registerKillSessionHandler registrations still EXIT (EventEmitter does not replay). Mirror Pi: persist archiveSnapshot on ptyResumeAttempt across clear-before-spawn and quarantine, restore on failed-resume cleanup (#1911 Discovery Majors). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): Codex hub-archived exit; AC6 latch test production order C2: SharedCodexRoot registers hub-archived (and latch check) beside its private KillSession — Codex never called registerKillSessionHandler. C1: keep latch+subscribe; test write-then-register order Discovery proved. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): type Codex hub-archived end mock for tsc vi.fn(async () => {}) inferred zero-arg calls; RootHost['end'] signature unblocks end.mock.calls[0][0] (TS2493). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): catch hub-archived Codex host.end rejection void this.host.end() discarded archive-RPC failures as unhandled rejections; Codex has no unhandledRejection handler, so Node can exit the shared wrapper and kill sibling roots mid-turn. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
HeavyGee
@@ -38,6 +38,8 @@ export class ApiClient {
|
||||
|
||||
async getOrCreateSession(opts: {
|
||||
id?: string
|
||||
/** Bind a hub-preallocated stub (requires id). */
|
||||
adopt?: boolean
|
||||
tag: string
|
||||
metadata: Metadata
|
||||
state: AgentState | null
|
||||
@@ -56,6 +58,7 @@ export class ApiClient {
|
||||
`${configuration.apiUrl}/cli/sessions`,
|
||||
{
|
||||
id: opts.id,
|
||||
...(opts.adopt === true ? { adopt: true } : {}),
|
||||
tag: opts.tag,
|
||||
metadata: opts.metadata,
|
||||
agentState: opts.state,
|
||||
|
||||
@@ -746,6 +746,35 @@ describe('ApiMachineClient SpawnHappySession handler', () => {
|
||||
client.shutdown()
|
||||
}
|
||||
})
|
||||
|
||||
it('returns childStarted:false on outside_workspace_roots so hub deletes the stub', async () => {
|
||||
const machine = makeMachine('machine-spawn-outside')
|
||||
const client = new ApiMachineClient('cli-token', machine, [workspaceRoot])
|
||||
const spawnSession = vi.fn()
|
||||
|
||||
client.setRPCHandlers({
|
||||
spawnSession,
|
||||
stopSession: vi.fn(async () => 'stopped' as const),
|
||||
requestShutdown: vi.fn()
|
||||
})
|
||||
|
||||
try {
|
||||
const result = await callSpawnHappySession(client, machine.id, {
|
||||
directory: '/tmp/definitely-outside-roots',
|
||||
agent: 'claude',
|
||||
})
|
||||
|
||||
expect(result).toEqual({
|
||||
type: 'error',
|
||||
errorMessage: "Directory is outside this machine's workspace roots",
|
||||
code: 'outside_workspace_roots',
|
||||
childStarted: false,
|
||||
})
|
||||
expect(spawnSession).not.toHaveBeenCalled()
|
||||
} finally {
|
||||
client.shutdown()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('ApiMachineClient keepAlive lifecycle', () => {
|
||||
|
||||
@@ -70,7 +70,10 @@ export { normalizeWindowsDriveRoot } from './machinePathPolicy'
|
||||
|
||||
type MachineRpcHandlers = {
|
||||
spawnSession: (options: SpawnSessionOptions) => Promise<SpawnSessionResult>
|
||||
stopSession: (sessionId: string) => Promise<'stopped' | 'already_gone' | 'still_alive'>
|
||||
stopSession: (
|
||||
sessionId: string,
|
||||
opts?: { processStartMarker?: string }
|
||||
) => Promise<'stopped' | 'already_gone' | 'still_alive' | 'unknown'>
|
||||
requestShutdown: () => void
|
||||
}
|
||||
|
||||
@@ -401,7 +404,7 @@ export class ApiMachineClient {
|
||||
|
||||
setRPCHandlers({ spawnSession, stopSession, requestShutdown }: MachineRpcHandlers): void {
|
||||
this.rpcHandlerManager.registerHandler(RPC_METHODS.SpawnHappySession, async (params: any) => {
|
||||
const { directory, sessionId, existingSessionId, resumeSessionId, machineId, approvedNewDirectoryCreation, agent, model, effort, modelReasoningEffort, yolo, permissionMode, serviceTier, collaborationMode, copilotAgentMode, token, sessionType, worktreeName, startingMode, forkSession } = params || {}
|
||||
const { directory, sessionId, existingSessionId, reservedSessionId, resumeSessionId, machineId, approvedNewDirectoryCreation, agent, model, effort, modelReasoningEffort, yolo, permissionMode, serviceTier, collaborationMode, copilotAgentMode, token, sessionType, worktreeName, startingMode, forkSession } = params || {}
|
||||
|
||||
if (!directory) {
|
||||
throw new Error('Directory is required')
|
||||
@@ -413,6 +416,8 @@ export class ApiMachineClient {
|
||||
type: 'error',
|
||||
errorMessage: 'Directory is outside this machine\'s workspace roots',
|
||||
code: 'outside_workspace_roots',
|
||||
// Pre-exec: no OS child — hub must delete the prealloc stub (#1911).
|
||||
childStarted: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -420,6 +425,7 @@ export class ApiMachineClient {
|
||||
directory,
|
||||
sessionId,
|
||||
existingSessionId,
|
||||
reservedSessionId,
|
||||
resumeSessionId,
|
||||
machineId,
|
||||
approvedNewDirectoryCreation,
|
||||
@@ -456,12 +462,15 @@ export class ApiMachineClient {
|
||||
})
|
||||
|
||||
this.rpcHandlerManager.registerHandler(RPC_METHODS.StopSession, async (params: any) => {
|
||||
const { sessionId } = params || {}
|
||||
const { sessionId, processStartMarker } = params || {}
|
||||
if (!sessionId) {
|
||||
throw new Error('Session ID is required')
|
||||
}
|
||||
|
||||
const status = await stopSession(sessionId)
|
||||
const status = await stopSession(
|
||||
sessionId,
|
||||
typeof processStartMarker === 'string' ? { processStartMarker } : undefined
|
||||
)
|
||||
return { status }
|
||||
})
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ const socketHarness = vi.hoisted(() => ({
|
||||
trigger: (event: string, ...args: any[]) => void
|
||||
triggerConnect: () => void
|
||||
triggerConnectError: () => void
|
||||
emitWithAckImpl: (event: string, ...args: unknown[]) => Promise<unknown>
|
||||
}>
|
||||
}))
|
||||
|
||||
@@ -28,7 +29,8 @@ vi.mock('socket.io-client', () => ({
|
||||
listeners: new Map<string, Array<(...args: any[]) => void>>(),
|
||||
trigger: () => {},
|
||||
triggerConnect: () => {},
|
||||
triggerConnectError: () => {}
|
||||
triggerConnectError: () => {},
|
||||
emitWithAckImpl: async () => ({})
|
||||
}
|
||||
state.trigger = (event: string, ...args: any[]) => {
|
||||
for (const listener of state.listeners.get(event) ?? []) {
|
||||
@@ -62,8 +64,10 @@ vi.mock('socket.io-client', () => ({
|
||||
state.emitted.push({ event, args })
|
||||
return socket
|
||||
},
|
||||
emitWithAck: async () => ({}),
|
||||
timeout: () => ({ emitWithAck: async () => ({}) }),
|
||||
emitWithAck: async (event: string, ...args: unknown[]) => state.emitWithAckImpl(event, ...args),
|
||||
timeout: () => ({
|
||||
emitWithAck: async (event: string, ...args: unknown[]) => state.emitWithAckImpl(event, ...args)
|
||||
}),
|
||||
connect: () => {
|
||||
state.connectCalls += 1
|
||||
if (state.connectImmediately) {
|
||||
@@ -76,7 +80,10 @@ vi.mock('socket.io-client', () => ({
|
||||
return socket
|
||||
}
|
||||
}
|
||||
Object.assign(socket, { volatile: socket })
|
||||
Object.assign(socket, {
|
||||
volatile: socket,
|
||||
io: { opts: { reconnection: true } }
|
||||
})
|
||||
socketHarness.sockets.push(state)
|
||||
return socket
|
||||
}
|
||||
@@ -410,6 +417,189 @@ describe('ApiSessionClient lazy materialization', () => {
|
||||
expect(socket.connectCalls).toBeGreaterThan(0)
|
||||
client.close()
|
||||
})
|
||||
|
||||
it('emits hub-archived from update-session metadata (#1910)', async () => {
|
||||
socketHarness.sockets.length = 0
|
||||
axiosHarness.get.mockResolvedValue({ data: { messages: [] } })
|
||||
const session = createSession({
|
||||
namespace: 'default',
|
||||
metadata: { path: '/tmp', host: 'h', flavor: 'claude' },
|
||||
metadataVersion: 1
|
||||
})
|
||||
const client = new ApiSessionClient('token', session)
|
||||
const socket = socketHarness.sockets[0]
|
||||
if (!socket) throw new Error('expected socket')
|
||||
|
||||
let archived = false
|
||||
client.on('hub-archived', () => { archived = true })
|
||||
|
||||
socket.trigger('update', {
|
||||
body: {
|
||||
t: 'update-session',
|
||||
sid: session.id,
|
||||
metadata: {
|
||||
version: 2,
|
||||
value: {
|
||||
path: '/tmp',
|
||||
host: 'h',
|
||||
flavor: 'claude',
|
||||
lifecycleState: 'archived',
|
||||
archivedBy: 'hub',
|
||||
archiveReason: 'Archived from hub (CLI unreachable)'
|
||||
}
|
||||
},
|
||||
agentState: null
|
||||
}
|
||||
})
|
||||
|
||||
expect(archived).toBe(true)
|
||||
expect(client.getMetadata()?.lifecycleState).toBe('archived')
|
||||
client.close()
|
||||
})
|
||||
|
||||
it('stops metadata CAS when hub returns archived on version-mismatch (#1911 M1)', async () => {
|
||||
socketHarness.sockets.length = 0
|
||||
axiosHarness.get.mockResolvedValue({ data: { messages: [] } })
|
||||
const session = createSession({
|
||||
namespace: 'default',
|
||||
metadata: { path: '/tmp', host: 'h', flavor: 'claude', lifecycleState: 'running' },
|
||||
metadataVersion: 1
|
||||
})
|
||||
const client = new ApiSessionClient('token', session)
|
||||
const socket = socketHarness.sockets[0]
|
||||
if (!socket) throw new Error('expected socket')
|
||||
|
||||
let ackCalls = 0
|
||||
let archived = false
|
||||
client.on('hub-archived', () => { archived = true })
|
||||
socket.emitWithAckImpl = async (event) => {
|
||||
if (event !== 'update-metadata') return {}
|
||||
ackCalls += 1
|
||||
return {
|
||||
result: 'version-mismatch',
|
||||
version: 2,
|
||||
metadata: {
|
||||
path: '/tmp',
|
||||
host: 'h',
|
||||
flavor: 'claude',
|
||||
lifecycleState: 'archived',
|
||||
archivedBy: 'hub',
|
||||
archiveReason: 'Archived from hub'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
client.updateMetadata((meta) => ({ ...meta, lifecycleState: 'running', hostPid: 42 }))
|
||||
|
||||
await vi.waitFor(() => {
|
||||
expect(client.getMetadata()?.lifecycleState).toBe('archived')
|
||||
expect(archived).toBe(true)
|
||||
})
|
||||
await new Promise((r) => setTimeout(r, 50))
|
||||
expect(ackCalls).toBe(1)
|
||||
client.close()
|
||||
})
|
||||
|
||||
it('exits via hub-archived on success+merge-preserve ack (#1911 criterion 6)', async () => {
|
||||
socketHarness.sockets.length = 0
|
||||
axiosHarness.get.mockResolvedValue({ data: { messages: [] } })
|
||||
const session = createSession({
|
||||
namespace: 'default',
|
||||
metadata: { path: '/tmp', host: 'h', flavor: 'claude', lifecycleState: 'running' },
|
||||
metadataVersion: 1
|
||||
})
|
||||
const client = new ApiSessionClient('token', session)
|
||||
const socket = socketHarness.sockets[0]
|
||||
if (!socket) throw new Error('expected socket')
|
||||
|
||||
let ackCalls = 0
|
||||
let archived = false
|
||||
client.on('hub-archived', () => { archived = true })
|
||||
socket.emitWithAckImpl = async (event) => {
|
||||
if (event !== 'update-metadata') return {}
|
||||
ackCalls += 1
|
||||
// Hub merge-preserved archive fields and returned success.
|
||||
return {
|
||||
result: 'success',
|
||||
version: 2,
|
||||
metadata: {
|
||||
path: '/tmp',
|
||||
host: 'h',
|
||||
flavor: 'claude',
|
||||
lifecycleState: 'archived',
|
||||
archivedBy: 'hub',
|
||||
archiveReason: 'Archived from hub',
|
||||
hostPid: 42
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
client.updateMetadata((meta) => ({ ...meta, lifecycleState: 'running', hostPid: 42 }))
|
||||
|
||||
await vi.waitFor(() => {
|
||||
expect(archived).toBe(true)
|
||||
expect(client.getMetadata()?.lifecycleState).toBe('archived')
|
||||
})
|
||||
await new Promise((r) => setTimeout(r, 50))
|
||||
expect(ackCalls).toBe(1)
|
||||
client.close()
|
||||
})
|
||||
|
||||
it('reconciles hub-archived metadata on reconnect (#1910)', async () => {
|
||||
socketHarness.sockets.length = 0
|
||||
axiosHarness.get.mockResolvedValue({
|
||||
data: {
|
||||
session: {
|
||||
metadataVersion: 1,
|
||||
metadata: { path: '/tmp', host: 'h', flavor: 'claude' }
|
||||
},
|
||||
messages: []
|
||||
}
|
||||
})
|
||||
|
||||
const client = new ApiSessionClient('token', createSession({
|
||||
namespace: 'default',
|
||||
metadata: { path: '/tmp', host: 'h', flavor: 'claude' },
|
||||
metadataVersion: 1
|
||||
}))
|
||||
const socket = socketHarness.sockets[0]
|
||||
if (!socket) throw new Error('expected socket')
|
||||
|
||||
let archived = false
|
||||
client.on('hub-archived', () => { archived = true })
|
||||
|
||||
// Establish first connection so hasConnectedOnce is true.
|
||||
socket.triggerConnect()
|
||||
await vi.waitFor(() => expect(axiosHarness.get).toHaveBeenCalled())
|
||||
|
||||
axiosHarness.get.mockImplementation(async (url: string) => {
|
||||
if (String(url).includes('/messages')) {
|
||||
return { data: { messages: [] } }
|
||||
}
|
||||
return {
|
||||
data: {
|
||||
session: {
|
||||
metadataVersion: 5,
|
||||
metadata: {
|
||||
path: '/tmp',
|
||||
host: 'h',
|
||||
flavor: 'claude',
|
||||
lifecycleState: 'archived',
|
||||
archivedBy: 'hub',
|
||||
archiveReason: 'Archived from hub (CLI unreachable)'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
socket.connected = false
|
||||
socket.trigger('disconnect', 'transport close')
|
||||
socket.triggerConnect()
|
||||
|
||||
await vi.waitFor(() => expect(archived).toBe(true))
|
||||
expect(client.getMetadata()?.archivedBy).toBe('hub')
|
||||
client.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('ApiSessionClient agy transcript messages', () => {
|
||||
|
||||
+141
-29
@@ -234,6 +234,15 @@ function hasSameJsonValue(left: unknown, right: unknown): boolean {
|
||||
export class ApiSessionClient extends EventEmitter {
|
||||
private reconnectHandler: (() => void) | null = null
|
||||
onReconnect(handler: (() => void) | null): void { this.reconnectHandler = handler }
|
||||
/** When false, socket.io must not keep the CLI immortal after hub archive (#1910). */
|
||||
private allowReconnect = true
|
||||
/**
|
||||
* Latch for hub-archived EXIT (#1911 criterion 6). Bootstrap may apply
|
||||
* hub-archived via updateMetadata CAS before flavor runners call
|
||||
* registerKillSessionHandler; EventEmitter does not replay past emits, so
|
||||
* the handler must read this synchronously at registration.
|
||||
*/
|
||||
hubArchived = false
|
||||
private readonly token: string
|
||||
readonly sessionId: string
|
||||
private metadata: Metadata | null
|
||||
@@ -332,10 +341,19 @@ export class ApiSessionClient extends EventEmitter {
|
||||
logger.debug('Socket connected successfully')
|
||||
this.awaitingMaterializedConnection = false
|
||||
this.rpcHandlerManager.onSocketConnect(this.socket)
|
||||
if (this.hasConnectedOnce) {
|
||||
const isReconnect = this.hasConnectedOnce
|
||||
if (isReconnect) {
|
||||
this.needsBackfill = true
|
||||
}
|
||||
void this.backfillIfNeeded()
|
||||
// Hub may have archived while we were offline (KillSession miss +
|
||||
// no live update-session). On reconnect, reconcile metadata before
|
||||
// message backfill so hub-archived can stop reconnect immortality (#1910).
|
||||
const afterMeta = isReconnect
|
||||
? this.reconcileSessionMetadata()
|
||||
: Promise.resolve()
|
||||
void afterMeta.finally(() => {
|
||||
void this.backfillIfNeeded()
|
||||
})
|
||||
this.hasConnectedOnce = true
|
||||
this.reconnectHandler?.()
|
||||
this.socket.emit('session-alive', {
|
||||
@@ -353,6 +371,13 @@ export class ApiSessionClient extends EventEmitter {
|
||||
logger.debug('[API] Socket disconnected:', reason)
|
||||
this.rpcHandlerManager.onSocketDisconnect()
|
||||
this.terminalManager.closeAll()
|
||||
if (!this.allowReconnect) {
|
||||
try {
|
||||
this.socket.io.opts.reconnection = false
|
||||
this.socket.disconnect()
|
||||
} catch { /* already tearing down */ }
|
||||
return
|
||||
}
|
||||
if (this.hasConnectedOnce) {
|
||||
this.needsBackfill = true
|
||||
}
|
||||
@@ -477,13 +502,7 @@ export class ApiSessionClient extends EventEmitter {
|
||||
|
||||
if (data.body.t === 'update-session') {
|
||||
if (data.body.metadata && data.body.metadata.version > this.metadataVersion) {
|
||||
const parsed = MetadataSchema.safeParse(data.body.metadata.value)
|
||||
if (parsed.success) {
|
||||
this.metadata = parsed.data
|
||||
} else {
|
||||
logger.debug('[API] Ignoring invalid metadata update', { version: data.body.metadata.version })
|
||||
}
|
||||
this.metadataVersion = data.body.metadata.version
|
||||
this.applyRemoteMetadata(data.body.metadata.version, data.body.metadata.value)
|
||||
}
|
||||
if (data.body.agentState && data.body.agentState.version > this.agentStateVersion) {
|
||||
const next = data.body.agentState.value
|
||||
@@ -819,6 +838,63 @@ export class ApiSessionClient extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a remote metadata snapshot (Socket.IO update-session or reconnect
|
||||
* REST reconcile). Advances metadataVersion and emits hub-archived when
|
||||
* the hub flipped lifecycle while this CLI was unreachable.
|
||||
*/
|
||||
private applyRemoteMetadata(version: number, value: unknown): void {
|
||||
if (version <= this.metadataVersion) return
|
||||
const parsed = MetadataSchema.safeParse(value)
|
||||
if (!parsed.success) {
|
||||
logger.debug('[API] Ignoring invalid metadata update', { version })
|
||||
this.metadataVersion = version
|
||||
return
|
||||
}
|
||||
const wasHubArchived = this.metadata?.lifecycleState === 'archived'
|
||||
&& this.metadata?.archivedBy === 'hub'
|
||||
this.metadata = parsed.data
|
||||
this.metadataVersion = version
|
||||
// #1910: hub may archive via metadata when KillSession cannot reach
|
||||
// this CLI. Stop reconnect immortality and let runners exit instead
|
||||
// of sitting as PPID=1 orphans.
|
||||
if (!wasHubArchived
|
||||
&& parsed.data.lifecycleState === 'archived'
|
||||
&& parsed.data.archivedBy === 'hub') {
|
||||
this.noteHubArchived()
|
||||
}
|
||||
}
|
||||
|
||||
private noteHubArchived(): void {
|
||||
this.hubArchived = true
|
||||
this.allowReconnect = false
|
||||
try {
|
||||
this.socket.io.opts.reconnection = false
|
||||
} catch { /* socket may be mid-teardown */ }
|
||||
this.emit('hub-archived')
|
||||
}
|
||||
|
||||
/** Fetch current hub metadata after reconnect; apply hub-archived if set. */
|
||||
private async reconcileSessionMetadata(): Promise<void> {
|
||||
try {
|
||||
const response = await axios.get(
|
||||
`${configuration.apiUrl}/cli/sessions/${encodeURIComponent(this.sessionId)}`,
|
||||
{
|
||||
headers: buildHubRequestHeaders({
|
||||
Authorization: `Bearer ${this.token}`,
|
||||
'Content-Type': 'application/json'
|
||||
}),
|
||||
timeout: 15_000
|
||||
}
|
||||
)
|
||||
const session = response.data?.session
|
||||
if (!session || typeof session.metadataVersion !== 'number') return
|
||||
this.applyRemoteMetadata(session.metadataVersion, session.metadata)
|
||||
} catch (error) {
|
||||
logger.debug('[API] Session metadata reconcile failed', error)
|
||||
}
|
||||
}
|
||||
|
||||
private async backfillMessages(): Promise<void> {
|
||||
if (this.backfillInFlight) {
|
||||
await this.backfillInFlight
|
||||
@@ -1308,6 +1384,13 @@ export class ApiSessionClient extends EventEmitter {
|
||||
}
|
||||
this.metadataLock.inLock(async () => {
|
||||
await backoff(async () => {
|
||||
// #1911 M1 criterion 6: hub-archived → EXIT (same as applyRemoteMetadata).
|
||||
if (this.metadata?.lifecycleState === 'archived' && this.metadata.archivedBy === 'hub') {
|
||||
this.noteHubArchived()
|
||||
logger.debug('[API] Skipping metadata update; session hub-archived')
|
||||
return
|
||||
}
|
||||
|
||||
const current = this.metadata ?? ({} as Metadata)
|
||||
const updated = handler(current)
|
||||
|
||||
@@ -1317,26 +1400,55 @@ export class ApiSessionClient extends EventEmitter {
|
||||
metadata: updated
|
||||
}) as unknown
|
||||
|
||||
applyVersionedAck(answer, {
|
||||
valueKey: 'metadata',
|
||||
parseValue: (value) => {
|
||||
const parsed = MetadataSchema.safeParse(value)
|
||||
return parsed.success ? parsed.data : null
|
||||
},
|
||||
applyValue: (value) => {
|
||||
this.metadata = value
|
||||
},
|
||||
applyVersion: (version) => {
|
||||
this.metadataVersion = version
|
||||
},
|
||||
logInvalidValue: (context, version) => {
|
||||
const suffix = context === 'success' ? 'ack' : 'version-mismatch ack'
|
||||
logger.debug(`[API] Ignoring invalid metadata value from ${suffix}`, { version })
|
||||
},
|
||||
invalidResponseMessage: 'Invalid update-metadata response',
|
||||
errorMessage: 'Metadata update failed',
|
||||
versionMismatchMessage: 'Metadata version mismatch'
|
||||
})
|
||||
try {
|
||||
applyVersionedAck(answer, {
|
||||
valueKey: 'metadata',
|
||||
parseValue: (value) => {
|
||||
const parsed = MetadataSchema.safeParse(value)
|
||||
return parsed.success ? parsed.data : null
|
||||
},
|
||||
applyValue: (value) => {
|
||||
// Route success+preserve (and mismatch) through the same
|
||||
// hub-archived detector as applyRemoteMetadata so the CLI
|
||||
// exits rather than booting against an archived row.
|
||||
const wasHubArchived = this.metadata?.lifecycleState === 'archived'
|
||||
&& this.metadata?.archivedBy === 'hub'
|
||||
this.metadata = value
|
||||
if (
|
||||
!wasHubArchived
|
||||
&& value?.lifecycleState === 'archived'
|
||||
&& value?.archivedBy === 'hub'
|
||||
) {
|
||||
this.noteHubArchived()
|
||||
}
|
||||
},
|
||||
applyVersion: (version) => {
|
||||
this.metadataVersion = version
|
||||
},
|
||||
logInvalidValue: (context, version) => {
|
||||
const suffix = context === 'success' ? 'ack' : 'version-mismatch ack'
|
||||
logger.debug(`[API] Ignoring invalid metadata value from ${suffix}`, { version })
|
||||
},
|
||||
invalidResponseMessage: 'Invalid update-metadata response',
|
||||
errorMessage: 'Metadata update failed',
|
||||
versionMismatchMessage: 'Metadata version mismatch'
|
||||
})
|
||||
} catch (error) {
|
||||
// True version races still throw; if hub archived mid-flight,
|
||||
// applied metadata is archived — exit, do not spin.
|
||||
if (this.metadata?.lifecycleState === 'archived' && this.metadata.archivedBy === 'hub') {
|
||||
this.noteHubArchived()
|
||||
return
|
||||
}
|
||||
throw error
|
||||
}
|
||||
|
||||
// Success+preserve terminates backoff without throw; ensure EXIT
|
||||
// if ack applied hub-archived (detector above already fired).
|
||||
if (this.metadata?.lifecycleState === 'archived' && this.metadata.archivedBy === 'hub') {
|
||||
this.noteHubArchived()
|
||||
return
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user