Files
hapi/cli/src/runner/lateRunnerWebhook.integration.test.ts
T
2b63091421 fix(runner): adopt untracked runner webhooks only from the reporting process generation (#1943)
* feat(cli): report the CLI process start marker in session metadata

The session webhook only names hostPid. A PID alone does not tell the
runner which process generation reported it: once the CLI exits, the OS
can hand the same PID to any other process, including another
runner-spawned CLI.

Add metadata.hostStartMarker, the reporting process's own start marker
(getProcessStartMarker(process.pid), the same probe the runner uses for
its resume records and the stop-session generation check).
buildSessionMetadata is the only place that sets hostPid, so every
bootstrap path (fresh, existing, lazy, shared Codex) reports it. The
runner uses it to decide whether an untracked runner webhook may be
adopted (decideUntrackedRunnerWebhook).

* fix(runner): adopt untracked runner webhooks only from the reporting process generation

Since #1911 the runner adopts an untracked `startedBy: 'runner'` webhook
(runner restart before the webhook; Claude has no HAPI id on argv) and
persists getProcessStartMarker(pid) of whatever currently owns hostPid,
so that a later stopSession can tree-kill that generation.

If the webhook is late - the reporting CLI has exited and the OS has
already reused its PID - the persisted marker belongs to a foreign
process, possibly another runner-spawned CLI. The generation check in
stopSession then passes and the foreign process tree is killed.

The webhook now has to prove the generation itself: the CLI reports its
own marker as metadata.hostStartMarker and decideUntrackedRunnerWebhook
adopts only when it matches the live PID. A marker that names another
generation, or a failed probe, is ignored without killing anything,
also on the webhook-timeout path: a proven mismatch means the PID is not
the ghost. A webhook without a marker (older CLI) is left untracked
instead of adopted blindly; the timeout kill for those CLIs is unchanged.

The integration test drives a real runner over its control server: a
webhook carrying the live marker is adopted, listed and stoppable; a
webhook with a stale or missing marker leaves the PID untracked and
alive, and stop-session reports unknown.

Follow-up to #1911 (#1910).

* fix(cli): probe the process start marker only for runner-started sessions

buildSessionMetadata probed getProcessStartMarker(process.pid) on every
bootstrap, terminal starts included. The runner reads
metadata.hostStartMarker only in the untracked `startedBy: 'runner'`
path (decideUntrackedRunnerWebhook); a terminal webhook never consumes
it, and neither do hub or web.

The probe is a synchronous spawn: `ps` on POSIX, `tasklist` plus a
PowerShell Get-CimInstance call on Windows, each with a 10 s timeout.
A terminal `hapi` start on Windows therefore blocked on a PowerShell
cold start before the session row existed, for a value nobody reads.
With shared Codex the probe also ran per /new and /fork on the event
loop shared by all sibling roots.

Set hostStartMarker only when startedBy is 'runner'; that is the one
case the runner evaluates. This narrows the earlier commit: every
bootstrap path reports the marker for runner-started sessions, not for
terminal ones.

* fix(codex): keep hostStartMarker in the shared runtime's runner webhook

The shared Codex runtime bootstraps its roots with reportStarted: false
and reports them to the runner itself, from root.session.getMetadata().
That is the hub's view of the metadata: the hub parses stored metadata
through MetadataSchema, so a hub that predates hostStartMarker strips
the field and the webhook arrives without it. The hub's update-metadata
merge never returns it either, so the loss lasts for the root's
lifetime.

With the generation check, an untracked runner-started webhook without
a marker is ignored instead of adopted. A runner restart (for example
the version handoff of a single-host upgrade, where the hub process
keeps running the old code) followed by /new or /fork in a long-lived
shared root therefore left the root unlisted and untracked, the very
situation the adoption from #1911 addresses. The Claude and lazy
bootstrap paths are unaffected: they report the local metadata.

Take the marker from the local bootstrap metadata instead. The opt-in
runtime integration test models the stripped hub view in its session
mock and asserts the reported payload still carries the marker. The
mock also gains the `on` subscription that SharedCodexRoot requires
since #1911, without which the suite's bind test could not run.

* test(runner): run the late-webhook suite in the serial integration project

lateRunnerWebhook.integration.test.ts starts a real runner and a
detached child process per case, but the default vitest project picked
it up and `bun run test` ran it in a parallel worker. Tests that spawn
runner process trees belong to the serial integration project
(vitest.integration.config.ts, CI job `integration`), as the config
comments and .github/workflows/test.yml require.

Include the file in the integration project, exclude it from the
default one, and remove the temporary HAPI_HOME of each case, which
was left behind in the temp directory.

---------

Co-authored-by: Benedict Groppe <195190800+BGGBTAC@users.noreply.github.com>
2026-10-28 12:10:21 +08:00

148 lines
6.9 KiB
TypeScript

import { describe, expect, it } from 'vitest'
import { spawn, type ChildProcess } from 'node:child_process'
import { randomUUID } from 'node:crypto'
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { getProcessStartMarker } from '@/utils/process'
import { TEST_OWNED_MARKER_KEY } from '@/test/integrationEnv'
/**
* Drives a real runner over its control server with a synthetic child that
* stands in for a CLI whose webhook arrives while the runner has no tracking
* entry for its PID (runner restart, or a webhook that outlived its sender).
* Adoption persists the PID's start marker for the stopSession generation
* check, so the webhook must prove it comes from the process that owns the
* PID right now (decideUntrackedRunnerWebhook). Start markers come from
* `ps`, so the suite is POSIX only, like the shared Codex full stack test.
*/
type RunnerState = { pid: number; httpPort: number }
type ResumeRecord = { requestedSessionId: string; confirmedSessionId?: string; pid: number; processStartMarker: string }
async function until<T>(probe: () => Promise<T | undefined>, label: string, timeoutMs = 20_000): Promise<T> {
const deadline = Date.now() + timeoutMs
while (Date.now() < deadline) {
const value = await probe()
if (value !== undefined) return value
await new Promise(resolve => setTimeout(resolve, 100))
}
throw new Error(`Timed out: ${label}`)
}
function exited(child: ChildProcess): boolean {
return child.exitCode !== null || child.signalCode !== null
}
async function stop(child: ChildProcess | undefined): Promise<void> {
if (!child || exited(child)) return
child.kill('SIGTERM')
try {
await until(async () => exited(child) ? true : undefined, `PID ${child.pid} shutdown`, 5_000)
} catch {
child.kill('SIGKILL')
}
}
/** Every process carries the run marker so the globalSetup sweep reaps leaks. */
function testEnv(overrides: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv {
return { ...process.env, [TEST_OWNED_MARKER_KEY]: process.env.HAPI_HOME, ...overrides }
}
/** Long-lived stand-in for a CLI process; its own group so tree kills stay local. */
function spawnIdleChild(): ChildProcess {
const child = spawn(process.env.HAPI_BUN_EXEC!, ['--eval', 'setInterval(() => {}, 1000)'], {
env: testEnv(), stdio: 'ignore', detached: true,
})
expect(child.pid).toBeDefined()
return child
}
/** Isolated HAPI_HOME: own state file, lock and resume records; same temp hub. */
function startRunner(home: string): ChildProcess {
return spawn(process.env.HAPI_BUN_EXEC!, ['--cwd', resolve('.'), resolve('src/index.ts'), 'runner', 'start-sync'], {
env: testEnv({ HAPI_HOME: home, HAPI_DISABLE_VERSION_HANDOFF: '1' }), stdio: 'ignore',
})
}
async function runnerReady(home: string, runner: ChildProcess): Promise<RunnerState> {
return until(async () => {
try {
const state = JSON.parse(await readFile(join(home, 'runner.state.json'), 'utf8')) as Partial<RunnerState>
return state.pid === runner.pid && state.httpPort ? state as RunnerState : undefined
} catch {
return undefined
}
}, 'isolated runner ready')
}
function control(state: RunnerState) {
return async (path: string, body: unknown = {}): Promise<Record<string, unknown>> => {
const response = await fetch(`http://127.0.0.1:${state.httpPort}${path}`, {
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body),
})
expect(response.status).toBe(200)
return await response.json() as Record<string, unknown>
}
}
async function readResumeRecords(home: string): Promise<ResumeRecord[]> {
return JSON.parse(await readFile(join(home, 'runner.state.json.resume-processes.json'), 'utf8')) as ResumeRecord[]
}
describe.skipIf(process.platform === 'win32')('untracked runner webhook adoption', () => {
it('adopts a webhook that carries the live process start marker, so the child is listed and stoppable', async () => {
const sessionId = randomUUID()
const home = await mkdtemp(join(tmpdir(), 'hapi-late-webhook-adopt-'))
// A child of the previous runner generation: nothing on disk names it.
const child = spawnIdleChild()
const runner = startRunner(home)
try {
const post = control(await runnerReady(home, runner))
const hostStartMarker = getProcessStartMarker(child.pid!)
expect(hostStartMarker).not.toBeNull()
await post('/session-started', { sessionId, metadata: { hostPid: child.pid, hostStartMarker, startedBy: 'runner' } })
expect(exited(child)).toBe(false)
expect(await post('/list')).toMatchObject({ children: [{ happySessionId: sessionId, pid: child.pid, startedBy: 'runner' }] })
expect(await readResumeRecords(home)).toMatchObject([
{ pid: child.pid, requestedSessionId: sessionId, confirmedSessionId: sessionId, processStartMarker: hostStartMarker },
])
expect(await post('/stop-session', { sessionId })).toMatchObject({ status: 'stopped' })
await until(async () => exited(child) ? true : undefined, 'adopted child stopped')
} finally {
await stop(runner)
await stop(child)
await rm(home, { recursive: true, force: true })
}
}, 30_000)
it('leaves a webhook alone that cannot prove the process generation, and never kills the PID', async () => {
const sessionId = randomUUID()
const home = await mkdtemp(join(tmpdir(), 'hapi-late-webhook-reused-pid-'))
// Stands in for the process the OS handed the PID to after the
// reporting CLI exited: the webhook names a generation that is gone,
// or (older CLI) no generation at all.
const bystander = spawnIdleChild()
const runner = startRunner(home)
try {
const post = control(await runnerReady(home, runner))
for (const metadata of [
{ hostPid: bystander.pid, hostStartMarker: 'Thu Jan 1 00:00:00 1970', startedBy: 'runner' },
{ hostPid: bystander.pid, startedBy: 'runner' },
]) {
await post('/session-started', { sessionId, metadata })
expect(await post('/list')).toMatchObject({ children: [] })
expect(await readResumeRecords(home)).toEqual([])
expect(await post('/stop-session', { sessionId })).toMatchObject({ status: 'unknown' })
// Killing is escalated asynchronously; give a wrongful kill time to land.
await new Promise(resolve => setTimeout(resolve, 300))
expect(exited(bystander)).toBe(false)
}
} finally {
await stop(runner)
await stop(bystander)
await rm(home, { recursive: true, force: true })
}
}, 30_000)
})