Files
hapi/cli/src/persistence.ts
T
a6176014fd fix(runner): self-restart resilience under systemd / external process supervision (#814)
* feat(runner): HAPI_DISABLE_VERSION_HANDOFF opt-out for mtime self-restart

The heartbeat in cli/src/runner/run.ts triggers spawnHappyCLI(['runner','start'])
+ process.exit(0) when getInstalledCliMtimeMs() differs from startedWithCliMtimeMs.
The same mtime guard fires in controlClient.isRunnerRunningCurrentlyInstalledHappyVersion
when a fresh CLI invocation inspects the live runner.

For operators who own process supervision (systemd, tmux, custom rebuild
pipelines, etc.), source-file mtimes shift for reasons unrelated to npm
upgrades. The clean exit defeats Restart=on-failure under systemd and
leaves the machine offline.

Setting HAPI_DISABLE_VERSION_HANDOFF=1 in the runner's environment now skips
both checks while keeping the rest of the heartbeat (session pruning, state
file persistence) intact. Default behavior is unchanged for npm consumers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(runner): preserve original argv across self-restart and verify handoff

The mtime-driven self-restart in cli/src/runner/run.ts spawned
`hapi runner start` with no arguments, then process.exit(0)'d
unconditionally after a 10s sleep. Two failure modes:

1. The forwarded `runner start-sync` lost the operator's --workspace-root
   flags (anything passed at the original invocation). Browse + spawn
   silently degraded to "no workspace roots".
2. If the replacement runner failed to come up at all (build was mid-flight,
   binary missing, etc.) the original runner still exited cleanly. Under
   systemd Restart=on-failure that means no runner is brought back, and
   the machine drops off the hub until manual intervention.

Changes:

- persistence.ts: add startedWithArgv?: string[] to RunnerLocallyPersistedState
- run.ts: snapshot process.argv.slice(2) at startup, persist it on initial
  state write and on every heartbeat, replay it as the new runner's argv
  (default to ['runner','start-sync'] when nothing was captured)
- controlClient.ts: new waitForRunnerHandoff(oldPid, {timeoutMs}) polls
  runner.state.json for a different live PID
- run.ts: only clearInterval + process.exit(0) when handoff is confirmed.
  On spawn failure or 30s timeout, refresh the mtime baseline (so we don't
  respawn-loop on the same drift) and stay alive so the machine keeps
  serving.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(runner): address Codex review findings on #814

Two Major correctness fixes flagged by upstream Codex review on PR #814:

(1) Stale-mtime poisoning on failed handoff (run.ts:854,867)

The previous failure paths assigned
  startedWithCliMtimeMs = installedCliMtimeMs
which the next heartbeat persisted to runner.state.json. Downstream
isRunnerRunningCurrentlyInstalledHappyVersion() then reported the
still-stale runner as current, masking the failure until the *next*
genuine mtime change. Symptom: an mtime change that briefly failed
to hand off would be silently forgotten.

Fix: leave startedWithCliMtimeMs immutable. Gate handoff entry on
a new nextHandoffAttemptAt timestamp; failure paths bump it by
HANDOFF_RETRY_BACKOFF_MS (5 min) via deferHandoffRetry(). The
heartbeat continues to write the honest "still on the old code"
mtime, and the runner naturally re-attempts after the cooldown.

(2) HAPI_DISABLE_VERSION_HANDOFF not honored by live runner
    (controlClient.ts:192, persistence.ts)

The env var was only checked in the invoking CLI process. Under the
documented systemd use case the env is set on the service unit but
NOT on the operator's interactive shell - so a shell `hapi runner
start` would still treat mtime drift as stale and kill the supervised
runner during a rebuild. The exact regression this layer was built
to prevent.

Fix: capture HAPI_DISABLE_VERSION_HANDOFF at runner start time into
state.startedWithVersionHandoffDisabled, persisted via the heartbeat.
The controlClient mtime check now OR's the live env var with the
persisted snapshot, so any caller honours the running runner's
opt-out regardless of their own environment.

Tests: cli typecheck clean; 14/14 runner unit tests pass.
Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(runner): address Codex #814 [Major] argv-capture + handoff race

Two additional Major findings on the runner self-restart layer that
were not addressed in a49fc57:

1. run.ts:672 - process.argv.slice(2) returns ['start-sync', ...] in
   compiled binary mode (raw argv is [hapi, runner, start-sync, ...]),
   so the handoff spawned `hapi start-sync ...` which resolveCommand
   treats as an unknown top-level and falls back to Claude. Replaced
   with getCliArgs() (the project's canonical argv normalizer) plus a
   defensive guard that falls back to ['runner', 'start-sync'] if the
   captured argv does not begin with 'runner'.

2. run.ts:892 - waitForRunnerHandoff did not actually keep the old
   runner alive. The child's startRunner() unconditionally called
   stopRunner() before acquiring the lock or writing its own state,
   so the parent's /stop handler resolved shutdown and exited BEFORE
   the child committed. If the child then failed (lock contention,
   auth error, anything between stopRunner and writeRunnerState),
   the machine went offline with no runner at all.

   New handoff protocol:
   - Parent sets HAPI_RUNNER_HANDOFF_FROM_PID=<pid> on the spawned
     child's env, then releases the lock BEFORE entering
     waitForRunnerHandoff (breaks the parent-holds-lock /
     child-needs-lock-to-write-state deadlock).
   - On wait-timeout the parent re-acquires the lock (long-retry, 30s)
     and defers retry; if re-acquire fails (third party took the
     lock) the parent exits cleanly so it does not stay alive without
     the lock invariant.
   - Child detects the env signal; if state.pid matches and that pid
     is alive, this is an authorized handoff: skip stopRunner(),
     skip the version-match early-exit, and acquire the lock with a
     longer retry window (60 attempts x 500ms) so it waits through
     the parent's asynchronous release.

CLI typecheck clean. 14/14 runner unit tests still pass. The wider
46/664 failures in the CLI suite are pre-existing in this branch
(unrelated: AppServerEventConverter, cursorEventConverter, hook
server, Query) - baseline before this commit has 42+; my changes do
not regress them.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-10 13:59:27 +08:00

280 lines
9.0 KiB
TypeScript

/**
* Minimal persistence functions for HAPI CLI
*
* Handles settings, encryption key, and runner state storage in ~/.hapi/ (or HAPI_HOME override)
*/
import { FileHandle } from 'node:fs/promises'
import { readFile, writeFile, mkdir, open, unlink, rename, stat } from 'node:fs/promises'
import { existsSync, writeFileSync, readFileSync, unlinkSync } from 'node:fs'
import { configuration } from '@/configuration'
import { isProcessAlive } from '@/utils/process';
interface Settings {
// This ID is used as the actual database ID on the server
// All machine operations use this ID
machineId?: string
machineIdConfirmedByServer?: boolean
runnerAutoStartWhenRunningHappy?: boolean
cliApiToken?: string
// API URL for server connections (priority: env HAPI_API_URL > this > default)
apiUrl?: string
// Legacy field name (for migration, read-only)
serverUrl?: string
}
const defaultSettings: Settings = {}
/**
* Runner state persisted locally (different from API RunnerState)
* This is written to disk by the runner to track its local process state
*/
export interface RunnerLocallyPersistedState {
pid: number;
httpPort: number;
startTime: string;
startedWithCliVersion: string;
startedWithCliMtimeMs?: number;
startedWithApiUrl?: string;
startedWithMachineId?: string;
startedWithCliApiTokenHash?: string;
/**
* Original process.argv.slice(2) of the runner process at start time, e.g.
* ['runner', 'start-sync', '--workspace-root', '/home/user/code'].
* Used by the self-restart handoff so the replacement runner inherits the
* same workspace-root / flag configuration instead of starting with defaults.
*/
startedWithArgv?: string[];
lastHeartbeat?: string;
runnerLogPath?: string;
/**
* Snapshot of HAPI_DISABLE_VERSION_HANDOFF=1 at the time this runner
* started. Lets a later `hapi runner start` invocation (from a shell where
* the env var is NOT set, e.g. operator's interactive terminal vs a
* systemd service that owns supervision) honour the running runner's
* opt-out instead of treating mtime drift as a reason to kill it.
*
* Codex review #814 [Major]: env-only check in controlClient meant the
* supervised use case (env set on service only) would still trigger a
* mid-rebuild stop. Persisting this fixes that.
*/
startedWithVersionHandoffDisabled?: boolean;
}
export async function readSettings(): Promise<Settings> {
if (!existsSync(configuration.settingsFile)) {
return { ...defaultSettings }
}
try {
const content = await readFile(configuration.settingsFile, 'utf8')
return JSON.parse(content)
} catch {
return { ...defaultSettings }
}
}
export async function writeSettings(settings: Settings): Promise<void> {
if (!existsSync(configuration.happyHomeDir)) {
await mkdir(configuration.happyHomeDir, { recursive: true })
}
await writeFile(configuration.settingsFile, JSON.stringify(settings, null, 2))
}
/**
* Atomically update settings with multi-process safety via file locking
* @param updater Function that takes current settings and returns updated settings
* @returns The updated settings
*/
export async function updateSettings(
updater: (current: Settings) => Settings | Promise<Settings>
): Promise<Settings> {
// Timing constants
const LOCK_RETRY_INTERVAL_MS = 100; // How long to wait between lock attempts
const MAX_LOCK_ATTEMPTS = 50; // Maximum number of attempts (5 seconds total)
const STALE_LOCK_TIMEOUT_MS = 10000; // Consider lock stale after 10 seconds
if (!existsSync(configuration.happyHomeDir)) {
await mkdir(configuration.happyHomeDir, { recursive: true });
}
const lockFile = configuration.settingsFile + '.lock';
const tmpFile = configuration.settingsFile + '.tmp';
let fileHandle;
let attempts = 0;
// Acquire exclusive lock with retries
while (attempts < MAX_LOCK_ATTEMPTS) {
try {
// 'wx' = create exclusively, fail if exists (cross-platform compatible)
fileHandle = await open(lockFile, 'wx');
break;
} catch (err: any) {
if (err.code === 'EEXIST') {
// Lock file exists, wait and retry
attempts++;
await new Promise(resolve => setTimeout(resolve, LOCK_RETRY_INTERVAL_MS));
// Check for stale lock
try {
const stats = await stat(lockFile);
if (Date.now() - stats.mtimeMs > STALE_LOCK_TIMEOUT_MS) {
await unlink(lockFile).catch(() => { });
}
} catch { }
} else {
throw err;
}
}
}
if (!fileHandle) {
throw new Error(`Failed to acquire settings lock after ${MAX_LOCK_ATTEMPTS * LOCK_RETRY_INTERVAL_MS / 1000} seconds`);
}
try {
// Read current settings with defaults
const current = await readSettings() || { ...defaultSettings };
// Apply update
const updated = await updater(current);
// Write atomically using rename
await writeFile(tmpFile, JSON.stringify(updated, null, 2));
await rename(tmpFile, configuration.settingsFile); // Atomic on POSIX
return updated;
} finally {
// Release lock
await fileHandle.close();
await unlink(lockFile).catch(() => { }); // Remove lock file
}
}
//
// Authentication
//
export async function writeCredentialsDataKey(credentials: { publicKey: Uint8Array, machineKey: Uint8Array, token: string }): Promise<void> {
if (!existsSync(configuration.happyHomeDir)) {
await mkdir(configuration.happyHomeDir, { recursive: true })
}
await writeFile(configuration.privateKeyFile, JSON.stringify({
encryption: { publicKey: Buffer.from(credentials.publicKey).toString('base64'), machineKey: Buffer.from(credentials.machineKey).toString('base64') },
token: credentials.token
}, null, 2));
}
export async function clearCredentials(): Promise<void> {
if (existsSync(configuration.privateKeyFile)) {
await unlink(configuration.privateKeyFile);
}
}
export async function clearMachineId(): Promise<void> {
await updateSettings(settings => ({
...settings,
machineId: undefined
}));
}
/**
* Read runner state from local file
*/
export async function readRunnerState(): Promise<RunnerLocallyPersistedState | null> {
try {
if (!existsSync(configuration.runnerStateFile)) {
return null;
}
const content = await readFile(configuration.runnerStateFile, 'utf-8');
return JSON.parse(content) as RunnerLocallyPersistedState;
} catch (error) {
// State corrupted somehow :(
console.error(`[PERSISTENCE] Runner state file corrupted: ${configuration.runnerStateFile}`, error);
return null;
}
}
/**
* Write runner state to local file (synchronously for atomic operation)
*/
export function writeRunnerState(state: RunnerLocallyPersistedState): void {
writeFileSync(configuration.runnerStateFile, JSON.stringify(state, null, 2), 'utf-8');
}
/**
* Clean up runner state file and lock file
*/
export async function clearRunnerState(): Promise<void> {
if (existsSync(configuration.runnerStateFile)) {
await unlink(configuration.runnerStateFile);
}
// Also clean up lock file if it exists (for stale cleanup)
if (existsSync(configuration.runnerLockFile)) {
try {
await unlink(configuration.runnerLockFile);
} catch {
// Lock file might be held by running runner, ignore error
}
}
}
/**
* Acquire an exclusive lock file for the runner.
* The lock file proves the runner is running and prevents multiple instances.
* Returns the file handle to hold for the runner's lifetime, or null if locked.
*/
export async function acquireRunnerLock(
maxAttempts: number = 5,
delayIncrementMs: number = 200
): Promise<FileHandle | null> {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
// 'wx' ensures we only create if it doesn't exist (atomic lock acquisition)
const fileHandle = await open(configuration.runnerLockFile, 'wx');
// Write PID to lock file for debugging
await fileHandle.writeFile(String(process.pid));
return fileHandle;
} catch (error: any) {
if (error.code === 'EEXIST') {
// Lock file exists, check if process is still running
try {
const lockPid = readFileSync(configuration.runnerLockFile, 'utf-8').trim();
if (lockPid && !isNaN(Number(lockPid))) {
if (!isProcessAlive(Number(lockPid))) {
// Process doesn't exist, remove stale lock
unlinkSync(configuration.runnerLockFile);
continue; // Retry acquisition
}
}
} catch {
// Can't read lock file, might be corrupted
}
}
if (attempt === maxAttempts) {
return null;
}
const delayMs = attempt * delayIncrementMs;
await new Promise(resolve => setTimeout(resolve, delayMs));
}
}
return null;
}
/**
* Release runner lock by closing handle and deleting lock file
*/
export async function releaseRunnerLock(lockHandle: FileHandle): Promise<void> {
try {
await lockHandle.close();
} catch { }
try {
if (existsSync(configuration.runnerLockFile)) {
unlinkSync(configuration.runnerLockFile);
}
} catch { }
}