hapi/docs/local-deployment.md

Local deployment branch

local/deploy-main is the source of the locally deployed HAPI binary.

Update rule

Never reset this branch to main or recreate it from main. Update it with:

git fetch upstream main
git merge upstream/main

Resolve conflicts by preserving the local features below. Build and deploy only after typecheck and focused regression tests pass.

Local features carried by this branch

• Claude local history import (tiann/hapi#1429)
• Codex and Cursor mid-turn steering (tiann/hapi#1443)
• Separate setting to pin all active sessions (tiann/hapi#1447)
• Claude custom models from settings.json (customClaudeModels); upstream PR tiann/hapi#1318 was closed, so this must remain local
• macOS case-safe Storage Usage module names, required for local typecheck/build

When one of the open PRs is merged upstream, drop only the equivalent local commits after verifying the merged implementation is present. Do not drop the other local features.

Local work preserved on separate branches

• Notification preferences and customizable push copy: feat/notification-preferences (tiann/hapi#1360)
• HTML preview and completed-unseen marker: feat/html-preview
• Earlier recovery work, including Codex quick import: feat/recover-local-features

These branches must not be deleted during cleanup or upstream updates.

macOS executable deployment (important)

Deploy with:

bun run build:single-exe
scripts/deploy-local.sh [tag]

Remote Macs with the same layout:

scripts/deploy-remote.sh <ssh-target> [tag]     # e.g. scripts/deploy-remote.sh k2lab card-dedupe

Fixed install path (why)

The binary always installs to the fixed path ~/.hapi/bin/hapi (a real file). macOS TCC keys permission grants (Documents, Downloads, Apple Music/media library, ...) to the executable path, so one stable path means the user grants access once and macOS remembers it across deploys. The earlier "new versioned filename per build + symlink" scheme re-prompted for every build: each deploy looked like a brand-new app, and while the prompt was unanswered every process touching a protected folder (model probes, session startup) blocked - which showed up as slow/timeout requests in the app. Do not reintroduce versioned filenames.

TCC grants are pinned to a code requirement (the cdhash trap)

A fixed path plus a pinned signing identity is necessary but not sufficient. Every TCC grant stores a code requirement (csreq), and a grant created while the binary was ad-hoc signed (or signed by a different identity) is pinned to that one build's cdhash. Replacing the binary at the fixed path then matches nothing - and because a record exists, macOS never prompts again, it denies silently. Processes already running keep TCC's cached "allow" until macOS re-evaluates them (hours later; a runningboardd cleanup was the trigger we saw), then lose access to ~/Documents mid-life, while every process started after the swap is fine. The symptom is confusing: sessions started hours ago stop answering (OpenCode service failure, getcwd: Operation not permitted for spawned children) while brand-new sessions on the same machine work, with (Sandbox) System Policy: ... deny(1) file-read-data /Users/.../Documents/... in the kernel log and no TCC prompt anywhere.

Inspect what is stored (needs Full Disk Access for the calling app):

sqlite3 "$HOME/Library/Application Support/com.apple.TCC/TCC.db" \
  "select writefile('/tmp/hapi.csreq',csreq) from access \
   where service='kTCCServiceSystemPolicyDocumentsFolder' and client='$HOME/.hapi/bin/hapi'"
csreq -r /tmp/hapi.csreq -t

• identifier "run.hapi.cli" and anchor apple generic and certificate leaf[subject.CN] = "Apple Development: ..." - identity-based, survives every later rebuild.
• cdhash H"..." - pinned to one build; it must be recreated.

Recreate it (the normal way): System Settings -> Privacy & Security -> Files and Folders -> hapi, remove the entry (toggling alone may only flip auth_value and leave the old csreq), then let any session touch a project under ~/Documents and click Allow. Re-check with the commands above - the recreated record is identity-based.

Fallback when the prompt never appears (macOS still honoring its cached allow, and tccutil reset cannot help because it wants a bundle id): back up the database, delete the stale row, restart tccd (killall tccd; launchd brings it back), and - if the prompt still does not come - insert a record copied from a machine that already holds an identity-based grant for the same signing certificate (k2lab had one; mini's three rows were rebuilt this way on 2026-09-17). Keep the row shape identical: client_type=1, auth_value=2, auth_reason=2, auth_version=1, indirect_object_identifier='UNUSED', flags=0, and re-verify with csreq -r. Do the same for kTCCServiceSystemPolicyDownloadsFolder and kTCCServiceSystemPolicyNetworkVolumes - they are granted at the same time and rot together.

Do the re-grant once per machine right after changing the signing identity.

Code-signature cache (why deploys verify, and roll back)

The fixed path has one hazard: the kernel caches the Mach-O signature per path, so overwriting it can kill new processes with:

OS_REASON_CODESIGNING
embedded signature doesn't match attached signature

This is a deployment/install issue, not a HAPI application error. The deploy scripts handle it end to end:

1. back up the installed binary to ~/.hapi/bin/backups/ (newest HAPI_KEEP_BACKUPS kept, default 2; never exec from the backup dir)
2. install the new file with a fresh mtime so the path-keyed signature cache re-reads it
3. prove it execs repeatedly (hapi --version x3) and codesign --verify
4. restart the hub (local) / kickstart the launchd job (remote) and verify (/health locally; runner state + exec path on the remote)
5. on any failure restore the backup onto the fixed path and restart

Never leave an unverified binary installed.

Signing

Sign every build with one pinned identity; never ship an ad-hoc signature (codesign --sign -), which has no stable identity and makes TCC re-ask every permission:

• scripts/sign-build.sh (called by both deploy scripts) stores the chosen Apple Development SHA-1 in ~/.hapi/signing-identity and reuses it; override with HAPI_SIGN_IDENTITY when rotating certificates
• all builds use the signed identifier run.hapi.cli
• with no Apple Development identity the scripts fall back to ad-hoc; that still runs, but expect TCC prompts to reappear after every deploy

codesign reads the private key from the login keychain, and that key is reachable only from a GUI (Aqua) session: an agent or daemon shell runs in launchd's Background domain, where codesign fails with errSecInternalComponent even while Keychain Access shows the keychain unlocked. Measured on macOS 15.6: in a Background session (even a plain ssh localhost) security show-keychain-info already fails with SecKeychainCopySettings ...: User interaction is not allowed (exit 36), while the same command in Aqua reports the keychain unlocked - the failure is session-scoped keychain access, not an app allow-list on the key.

Two ways out:

• run the deploy from a Terminal window on the machine (the normal path), or
• if it must run from a Background session, unlock the keychain inside that session with the password - the interactive form cannot prompt there, so use security unlock-keychain -p '<login password>' ~/Library/Keychains/login.keychain-db (the CI-style recipe; verify it with the sign probe before relying on it).

Granting the key "allow all applications" in Keychain Access does not address this: the key's ACL is only consulted once the session can reach the keychain at all.

Agent sessions should also avoid recursive $HOME sweeps (find ~, du -sh ~, ...) unless they prune TCC-protected folders (~/Music, ~/Pictures, ~/Movies, ~/Library): a walk into ~/Music/Music/Media.localized raises a spurious Apple Music prompt attributed to the hapi binary.

Refresh the runner

The runner is a long-lived detached process. New sessions it spawns resolve ~/.hapi/bin/hapi and therefore pick up the new binary automatically, but the runner's own code stays old until the process restarts:

• compiled binaries never self-update: the heartbeat compares the mtime of the runner's own resolved exec path, which is fixed for the life of the process
• a stale runner keeps old machine RPC handlers and capability flags, so hub features fail with "restart the runner" errors

scripts/deploy-local.sh runs hapi runner start when a runner is already running; the CLI stops the stale runner and starts a fresh one with HAPI_CLI_EXECUTABLE pinned to the fixed path. Running sessions are detached and survive the restart.

Manual sequence

set -euo pipefail
build=cli/dist-exe/bun-darwin-arm64/hapi
bin_dir="$HOME/.hapi/bin"
stable="$bin_dir/hapi"
stamp=$(date +%Y%m%d-%H%M%S)

# Sign with the pinned identity (see scripts/sign-build.sh).
bash scripts/sign-build.sh "$build"

# Back up the installed binary for rollback.
mkdir -p "$bin_dir/backups"
backup="$bin_dir/backups/hapi.$stamp"
[ -f "$stable" ] && cp -p "$stable" "$backup"

# Install to the fixed path with a fresh mtime, then prove it execs.
rm -f "$stable"
cp "$build" "$stable"
chmod 755 "$stable"
"$stable" --version && "$stable" --version && "$stable" --version
codesign --verify --deep --strict "$stable"

launchctl kickstart -k "gui/$(id -u)/com.hapi.hub"
sleep 2
curl -fsS http://127.0.0.1:3006/health >/dev/null
"$stable" --version

# Refresh a running runner so its machine RPCs/capabilities match.
HAPI_CLI_EXECUTABLE="$stable" "$stable" runner start

If the health check or a cold start fails, restore the backup and restart:

rm -f "$stable"
cp "$backup" "$stable"
chmod 755 "$stable"
"$stable" --version
launchctl kickstart -k "gui/$(id -u)/com.hapi.hub"

Remote machines

scripts/deploy-remote.sh <ssh-target> [tag] performs the same flow over SSH: signs locally, checks the remote arch matches, uploads next to the fixed path, installs by move (fresh inode + mtime), proves the binary execs, then kickstarts the launchd job (com.hapi.runner by default; override with HAPI_REMOTE_LAUNCHD_LABEL) and verifies the runner executes the fixed path. On any failure it restores the backup and kicks the job again. Running sessions survive; the remote host needs no build toolchain.