mirror of
https://github.com/wu736139669/hapi.git
synced 2026-08-05 06:24:37 +00:00
* feat(runner): HAPI_DISABLE_VERSION_HANDOFF opt-out for mtime self-restart The heartbeat in cli/src/runner/run.ts triggers spawnHappyCLI(['runner','start']) + process.exit(0) when getInstalledCliMtimeMs() differs from startedWithCliMtimeMs. The same mtime guard fires in controlClient.isRunnerRunningCurrentlyInstalledHappyVersion when a fresh CLI invocation inspects the live runner. For operators who own process supervision (systemd, tmux, custom rebuild pipelines, etc.), source-file mtimes shift for reasons unrelated to npm upgrades. The clean exit defeats Restart=on-failure under systemd and leaves the machine offline. Setting HAPI_DISABLE_VERSION_HANDOFF=1 in the runner's environment now skips both checks while keeping the rest of the heartbeat (session pruning, state file persistence) intact. Default behavior is unchanged for npm consumers. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): preserve original argv across self-restart and verify handoff The mtime-driven self-restart in cli/src/runner/run.ts spawned `hapi runner start` with no arguments, then process.exit(0)'d unconditionally after a 10s sleep. Two failure modes: 1. The forwarded `runner start-sync` lost the operator's --workspace-root flags (anything passed at the original invocation). Browse + spawn silently degraded to "no workspace roots". 2. If the replacement runner failed to come up at all (build was mid-flight, binary missing, etc.) the original runner still exited cleanly. Under systemd Restart=on-failure that means no runner is brought back, and the machine drops off the hub until manual intervention. Changes: - persistence.ts: add startedWithArgv?: string[] to RunnerLocallyPersistedState - run.ts: snapshot process.argv.slice(2) at startup, persist it on initial state write and on every heartbeat, replay it as the new runner's argv (default to ['runner','start-sync'] when nothing was captured) - controlClient.ts: new waitForRunnerHandoff(oldPid, {timeoutMs}) polls runner.state.json for a different live PID - run.ts: only clearInterval + process.exit(0) when handoff is confirmed. On spawn failure or 30s timeout, refresh the mtime baseline (so we don't respawn-loop on the same drift) and stay alive so the machine keeps serving. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): address Codex review findings on #814 Two Major correctness fixes flagged by upstream Codex review on PR #814: (1) Stale-mtime poisoning on failed handoff (run.ts:854,867) The previous failure paths assigned startedWithCliMtimeMs = installedCliMtimeMs which the next heartbeat persisted to runner.state.json. Downstream isRunnerRunningCurrentlyInstalledHappyVersion() then reported the still-stale runner as current, masking the failure until the *next* genuine mtime change. Symptom: an mtime change that briefly failed to hand off would be silently forgotten. Fix: leave startedWithCliMtimeMs immutable. Gate handoff entry on a new nextHandoffAttemptAt timestamp; failure paths bump it by HANDOFF_RETRY_BACKOFF_MS (5 min) via deferHandoffRetry(). The heartbeat continues to write the honest "still on the old code" mtime, and the runner naturally re-attempts after the cooldown. (2) HAPI_DISABLE_VERSION_HANDOFF not honored by live runner (controlClient.ts:192, persistence.ts) The env var was only checked in the invoking CLI process. Under the documented systemd use case the env is set on the service unit but NOT on the operator's interactive shell - so a shell `hapi runner start` would still treat mtime drift as stale and kill the supervised runner during a rebuild. The exact regression this layer was built to prevent. Fix: capture HAPI_DISABLE_VERSION_HANDOFF at runner start time into state.startedWithVersionHandoffDisabled, persisted via the heartbeat. The controlClient mtime check now OR's the live env var with the persisted snapshot, so any caller honours the running runner's opt-out regardless of their own environment. Tests: cli typecheck clean; 14/14 runner unit tests pass. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(runner): address Codex #814 [Major] argv-capture + handoff race Two additional Major findings on the runner self-restart layer that were not addressed in a49fc57: 1. run.ts:672 - process.argv.slice(2) returns ['start-sync', ...] in compiled binary mode (raw argv is [hapi, runner, start-sync, ...]), so the handoff spawned `hapi start-sync ...` which resolveCommand treats as an unknown top-level and falls back to Claude. Replaced with getCliArgs() (the project's canonical argv normalizer) plus a defensive guard that falls back to ['runner', 'start-sync'] if the captured argv does not begin with 'runner'. 2. run.ts:892 - waitForRunnerHandoff did not actually keep the old runner alive. The child's startRunner() unconditionally called stopRunner() before acquiring the lock or writing its own state, so the parent's /stop handler resolved shutdown and exited BEFORE the child committed. If the child then failed (lock contention, auth error, anything between stopRunner and writeRunnerState), the machine went offline with no runner at all. New handoff protocol: - Parent sets HAPI_RUNNER_HANDOFF_FROM_PID=<pid> on the spawned child's env, then releases the lock BEFORE entering waitForRunnerHandoff (breaks the parent-holds-lock / child-needs-lock-to-write-state deadlock). - On wait-timeout the parent re-acquires the lock (long-retry, 30s) and defers retry; if re-acquire fails (third party took the lock) the parent exits cleanly so it does not stay alive without the lock invariant. - Child detects the env signal; if state.pid matches and that pid is alive, this is an authorized handoff: skip stopRunner(), skip the version-match early-exit, and acquire the lock with a longer retry window (60 attempts x 500ms) so it waits through the parent's asynchronous release. CLI typecheck clean. 14/14 runner unit tests still pass. The wider 46/664 failures in the CLI suite are pre-existing in this branch (unrelated: AppServerEventConverter, cursorEventConverter, hook server, Query) - baseline before this commit has 42+; my changes do not regress them. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
280 lines
9.0 KiB
TypeScript
280 lines
9.0 KiB
TypeScript
/**
|
|
* Minimal persistence functions for HAPI CLI
|
|
*
|
|
* Handles settings, encryption key, and runner state storage in ~/.hapi/ (or HAPI_HOME override)
|
|
*/
|
|
|
|
import { FileHandle } from 'node:fs/promises'
|
|
import { readFile, writeFile, mkdir, open, unlink, rename, stat } from 'node:fs/promises'
|
|
import { existsSync, writeFileSync, readFileSync, unlinkSync } from 'node:fs'
|
|
import { configuration } from '@/configuration'
|
|
import { isProcessAlive } from '@/utils/process';
|
|
|
|
interface Settings {
|
|
// This ID is used as the actual database ID on the server
|
|
// All machine operations use this ID
|
|
machineId?: string
|
|
machineIdConfirmedByServer?: boolean
|
|
runnerAutoStartWhenRunningHappy?: boolean
|
|
cliApiToken?: string
|
|
// API URL for server connections (priority: env HAPI_API_URL > this > default)
|
|
apiUrl?: string
|
|
// Legacy field name (for migration, read-only)
|
|
serverUrl?: string
|
|
}
|
|
|
|
const defaultSettings: Settings = {}
|
|
|
|
/**
|
|
* Runner state persisted locally (different from API RunnerState)
|
|
* This is written to disk by the runner to track its local process state
|
|
*/
|
|
export interface RunnerLocallyPersistedState {
|
|
pid: number;
|
|
httpPort: number;
|
|
startTime: string;
|
|
startedWithCliVersion: string;
|
|
startedWithCliMtimeMs?: number;
|
|
startedWithApiUrl?: string;
|
|
startedWithMachineId?: string;
|
|
startedWithCliApiTokenHash?: string;
|
|
/**
|
|
* Original process.argv.slice(2) of the runner process at start time, e.g.
|
|
* ['runner', 'start-sync', '--workspace-root', '/home/user/code'].
|
|
* Used by the self-restart handoff so the replacement runner inherits the
|
|
* same workspace-root / flag configuration instead of starting with defaults.
|
|
*/
|
|
startedWithArgv?: string[];
|
|
lastHeartbeat?: string;
|
|
runnerLogPath?: string;
|
|
/**
|
|
* Snapshot of HAPI_DISABLE_VERSION_HANDOFF=1 at the time this runner
|
|
* started. Lets a later `hapi runner start` invocation (from a shell where
|
|
* the env var is NOT set, e.g. operator's interactive terminal vs a
|
|
* systemd service that owns supervision) honour the running runner's
|
|
* opt-out instead of treating mtime drift as a reason to kill it.
|
|
*
|
|
* Codex review #814 [Major]: env-only check in controlClient meant the
|
|
* supervised use case (env set on service only) would still trigger a
|
|
* mid-rebuild stop. Persisting this fixes that.
|
|
*/
|
|
startedWithVersionHandoffDisabled?: boolean;
|
|
}
|
|
|
|
export async function readSettings(): Promise<Settings> {
|
|
if (!existsSync(configuration.settingsFile)) {
|
|
return { ...defaultSettings }
|
|
}
|
|
|
|
try {
|
|
const content = await readFile(configuration.settingsFile, 'utf8')
|
|
return JSON.parse(content)
|
|
} catch {
|
|
return { ...defaultSettings }
|
|
}
|
|
}
|
|
|
|
export async function writeSettings(settings: Settings): Promise<void> {
|
|
if (!existsSync(configuration.happyHomeDir)) {
|
|
await mkdir(configuration.happyHomeDir, { recursive: true })
|
|
}
|
|
|
|
await writeFile(configuration.settingsFile, JSON.stringify(settings, null, 2))
|
|
}
|
|
|
|
/**
|
|
* Atomically update settings with multi-process safety via file locking
|
|
* @param updater Function that takes current settings and returns updated settings
|
|
* @returns The updated settings
|
|
*/
|
|
export async function updateSettings(
|
|
updater: (current: Settings) => Settings | Promise<Settings>
|
|
): Promise<Settings> {
|
|
// Timing constants
|
|
const LOCK_RETRY_INTERVAL_MS = 100; // How long to wait between lock attempts
|
|
const MAX_LOCK_ATTEMPTS = 50; // Maximum number of attempts (5 seconds total)
|
|
const STALE_LOCK_TIMEOUT_MS = 10000; // Consider lock stale after 10 seconds
|
|
|
|
if (!existsSync(configuration.happyHomeDir)) {
|
|
await mkdir(configuration.happyHomeDir, { recursive: true });
|
|
}
|
|
|
|
const lockFile = configuration.settingsFile + '.lock';
|
|
const tmpFile = configuration.settingsFile + '.tmp';
|
|
let fileHandle;
|
|
let attempts = 0;
|
|
|
|
// Acquire exclusive lock with retries
|
|
while (attempts < MAX_LOCK_ATTEMPTS) {
|
|
try {
|
|
// 'wx' = create exclusively, fail if exists (cross-platform compatible)
|
|
fileHandle = await open(lockFile, 'wx');
|
|
break;
|
|
} catch (err: any) {
|
|
if (err.code === 'EEXIST') {
|
|
// Lock file exists, wait and retry
|
|
attempts++;
|
|
await new Promise(resolve => setTimeout(resolve, LOCK_RETRY_INTERVAL_MS));
|
|
|
|
// Check for stale lock
|
|
try {
|
|
const stats = await stat(lockFile);
|
|
if (Date.now() - stats.mtimeMs > STALE_LOCK_TIMEOUT_MS) {
|
|
await unlink(lockFile).catch(() => { });
|
|
}
|
|
} catch { }
|
|
} else {
|
|
throw err;
|
|
}
|
|
}
|
|
}
|
|
|
|
if (!fileHandle) {
|
|
throw new Error(`Failed to acquire settings lock after ${MAX_LOCK_ATTEMPTS * LOCK_RETRY_INTERVAL_MS / 1000} seconds`);
|
|
}
|
|
|
|
try {
|
|
// Read current settings with defaults
|
|
const current = await readSettings() || { ...defaultSettings };
|
|
|
|
// Apply update
|
|
const updated = await updater(current);
|
|
|
|
// Write atomically using rename
|
|
await writeFile(tmpFile, JSON.stringify(updated, null, 2));
|
|
await rename(tmpFile, configuration.settingsFile); // Atomic on POSIX
|
|
|
|
return updated;
|
|
} finally {
|
|
// Release lock
|
|
await fileHandle.close();
|
|
await unlink(lockFile).catch(() => { }); // Remove lock file
|
|
}
|
|
}
|
|
|
|
//
|
|
// Authentication
|
|
//
|
|
|
|
export async function writeCredentialsDataKey(credentials: { publicKey: Uint8Array, machineKey: Uint8Array, token: string }): Promise<void> {
|
|
if (!existsSync(configuration.happyHomeDir)) {
|
|
await mkdir(configuration.happyHomeDir, { recursive: true })
|
|
}
|
|
await writeFile(configuration.privateKeyFile, JSON.stringify({
|
|
encryption: { publicKey: Buffer.from(credentials.publicKey).toString('base64'), machineKey: Buffer.from(credentials.machineKey).toString('base64') },
|
|
token: credentials.token
|
|
}, null, 2));
|
|
}
|
|
|
|
export async function clearCredentials(): Promise<void> {
|
|
if (existsSync(configuration.privateKeyFile)) {
|
|
await unlink(configuration.privateKeyFile);
|
|
}
|
|
}
|
|
|
|
export async function clearMachineId(): Promise<void> {
|
|
await updateSettings(settings => ({
|
|
...settings,
|
|
machineId: undefined
|
|
}));
|
|
}
|
|
|
|
/**
|
|
* Read runner state from local file
|
|
*/
|
|
export async function readRunnerState(): Promise<RunnerLocallyPersistedState | null> {
|
|
try {
|
|
if (!existsSync(configuration.runnerStateFile)) {
|
|
return null;
|
|
}
|
|
const content = await readFile(configuration.runnerStateFile, 'utf-8');
|
|
return JSON.parse(content) as RunnerLocallyPersistedState;
|
|
} catch (error) {
|
|
// State corrupted somehow :(
|
|
console.error(`[PERSISTENCE] Runner state file corrupted: ${configuration.runnerStateFile}`, error);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Write runner state to local file (synchronously for atomic operation)
|
|
*/
|
|
export function writeRunnerState(state: RunnerLocallyPersistedState): void {
|
|
writeFileSync(configuration.runnerStateFile, JSON.stringify(state, null, 2), 'utf-8');
|
|
}
|
|
|
|
/**
|
|
* Clean up runner state file and lock file
|
|
*/
|
|
export async function clearRunnerState(): Promise<void> {
|
|
if (existsSync(configuration.runnerStateFile)) {
|
|
await unlink(configuration.runnerStateFile);
|
|
}
|
|
// Also clean up lock file if it exists (for stale cleanup)
|
|
if (existsSync(configuration.runnerLockFile)) {
|
|
try {
|
|
await unlink(configuration.runnerLockFile);
|
|
} catch {
|
|
// Lock file might be held by running runner, ignore error
|
|
}
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Acquire an exclusive lock file for the runner.
|
|
* The lock file proves the runner is running and prevents multiple instances.
|
|
* Returns the file handle to hold for the runner's lifetime, or null if locked.
|
|
*/
|
|
export async function acquireRunnerLock(
|
|
maxAttempts: number = 5,
|
|
delayIncrementMs: number = 200
|
|
): Promise<FileHandle | null> {
|
|
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
|
try {
|
|
// 'wx' ensures we only create if it doesn't exist (atomic lock acquisition)
|
|
const fileHandle = await open(configuration.runnerLockFile, 'wx');
|
|
// Write PID to lock file for debugging
|
|
await fileHandle.writeFile(String(process.pid));
|
|
return fileHandle;
|
|
} catch (error: any) {
|
|
if (error.code === 'EEXIST') {
|
|
// Lock file exists, check if process is still running
|
|
try {
|
|
const lockPid = readFileSync(configuration.runnerLockFile, 'utf-8').trim();
|
|
if (lockPid && !isNaN(Number(lockPid))) {
|
|
if (!isProcessAlive(Number(lockPid))) {
|
|
// Process doesn't exist, remove stale lock
|
|
unlinkSync(configuration.runnerLockFile);
|
|
continue; // Retry acquisition
|
|
}
|
|
}
|
|
} catch {
|
|
// Can't read lock file, might be corrupted
|
|
}
|
|
}
|
|
|
|
if (attempt === maxAttempts) {
|
|
return null;
|
|
}
|
|
const delayMs = attempt * delayIncrementMs;
|
|
await new Promise(resolve => setTimeout(resolve, delayMs));
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* Release runner lock by closing handle and deleting lock file
|
|
*/
|
|
export async function releaseRunnerLock(lockHandle: FileHandle): Promise<void> {
|
|
try {
|
|
await lockHandle.close();
|
|
} catch { }
|
|
|
|
try {
|
|
if (existsSync(configuration.runnerLockFile)) {
|
|
unlinkSync(configuration.runnerLockFile);
|
|
}
|
|
} catch { }
|
|
}
|