hapi/docs/local-deployment.md

# Local deployment branch

`local/deploy-main` is the source of the locally deployed HAPI binary.

## Update rule

Never reset this branch to `main` or recreate it from `main`. Update it with:

```bash
git fetch upstream main
git merge upstream/main
```

Resolve conflicts by preserving the local features below. Build and deploy only
after typecheck and focused regression tests pass.

## Local features carried by this branch

- Claude local history import (`tiann/hapi#1429`)
- Codex and Cursor mid-turn steering (`tiann/hapi#1443`)
- Separate setting to pin all active sessions (`tiann/hapi#1447`)
- Claude custom models from `settings.json` (`customClaudeModels`); upstream PR
  `tiann/hapi#1318` was closed, so this must remain local
- macOS case-safe Storage Usage module names, required for local typecheck/build

When one of the open PRs is merged upstream, drop only the equivalent local
commits after verifying the merged implementation is present. Do not drop the
other local features.

## Local work preserved on separate branches

- Notification preferences and customizable push copy:
  `feat/notification-preferences` (`tiann/hapi#1360`)
- HTML preview and completed-unseen marker: `feat/html-preview`
- Earlier recovery work, including Codex quick import:
  `feat/recover-local-features`

These branches must not be deleted during cleanup or upstream updates.

## macOS executable deployment (important)

Deploy with:

```bash
bun run build:single-exe
scripts/deploy-local.sh [tag]
```

Remote Macs with the same layout:

```bash
scripts/deploy-remote.sh <ssh-target> [tag]     # e.g. scripts/deploy-remote.sh k2lab card-dedupe
```

Both deploy scripts verify that the compiled artifact contains durable Codex
history synchronization with a separate invalidation epoch, confirmed transcript delivery, and persistent usage
storage before signing or installation. A matching version number is not
enough: local builds can all report the same version while containing different
fixes. Build from the complete deployment workspace. If using an isolated
checkout, include all local changes and untracked source files; a checkout of
tracked files plus one task's patch can silently remove another task's fixes.
Use the current deployment scripts, not an older copy from that checkout.
Remote transfer uses rsync with a staged copy of the installed binary when
both Macs have rsync; unchanged blocks are reused. Otherwise it uses scp.
This avoids a global `PATH` override for a temporary transfer wrapper.

### Fixed install path (why)

The binary always installs to the **fixed path** `~/.hapi/bin/hapi` (a real
file). macOS TCC keys permission grants (Documents, Downloads, Apple
Music/media library, ...) to the executable path, so one stable path means the
user grants access **once** and macOS remembers it across deploys. The earlier
"new versioned filename per build + symlink" scheme re-prompted for every
build: each deploy looked like a brand-new app, and while the prompt was
unanswered every process touching a protected folder (model probes, session
startup) blocked - which showed up as slow/timeout requests in the app. Do not
reintroduce versioned filenames.

### TCC grants are pinned to a code requirement (the cdhash trap)

A fixed path plus a pinned signing identity is necessary but not sufficient.
Every TCC grant stores a *code requirement* (`csreq`), and a grant created while
the binary was ad-hoc signed (or signed by a different identity) is pinned to
that one build's **cdhash**. Replacing the binary at the fixed path then matches
nothing - and because a record exists, macOS never prompts again, it denies
silently. Processes already running keep TCC's cached "allow" until macOS
re-evaluates them (hours later; a `runningboardd` cleanup was the trigger we
saw), then lose access to `~/Documents` mid-life, while every process started
after the swap is fine. The symptom is confusing: sessions started hours ago
stop answering (`OpenCode service failure`, `getcwd: Operation not permitted`
for spawned children) while brand-new sessions on the same machine work, with
`(Sandbox) System Policy: ... deny(1) file-read-data /Users/.../Documents/...`
in the kernel log and no TCC prompt anywhere.

Inspect what is stored (needs Full Disk Access for the calling app):

```bash
sqlite3 "$HOME/Library/Application Support/com.apple.TCC/TCC.db" \
  "select writefile('/tmp/hapi.csreq',csreq) from access \
   where service='kTCCServiceSystemPolicyDocumentsFolder' and client='$HOME/.hapi/bin/hapi'"
csreq -r /tmp/hapi.csreq -t
```

- `identifier "run.hapi.cli" and anchor apple generic and certificate leaf[subject.CN] = "Apple Development: ..."` - identity-based, survives every later rebuild.
- `cdhash H"..."` - pinned to one build; it must be recreated.

Recreate it (the normal way): System Settings -> Privacy & Security -> Files and
Folders -> hapi, remove the entry (toggling alone may only flip `auth_value` and
leave the old `csreq`), then let any session touch a project under `~/Documents`
and click Allow. Re-check with the commands above - the recreated record is
identity-based.

Fallback when the prompt never appears (macOS still honoring its cached allow,
and `tccutil reset` cannot help because it wants a bundle id): back up the
database, delete the stale row, restart `tccd` (`killall tccd`; launchd brings it
back), and - if the prompt still does not come - insert a record copied from a
machine that already holds an identity-based grant for the *same* signing
certificate (k2lab had one; mini's three rows were rebuilt this way on
2026-09-17). Keep the row shape identical: `client_type=1`, `auth_value=2`,
`auth_reason=2`, `auth_version=1`, `indirect_object_identifier='UNUSED'`,
`flags=0`, and re-verify with `csreq -r`. Do the same for
`kTCCServiceSystemPolicyDownloadsFolder` and
`kTCCServiceSystemPolicyNetworkVolumes` - they are granted at the same time and
rot together.

Do the re-grant once per machine right after changing the signing identity.

### Code-signature cache (why deploys verify, and roll back)

The fixed path has one hazard: the kernel caches the Mach-O signature per
path, so overwriting it can kill new processes with:

```text
OS_REASON_CODESIGNING
embedded signature doesn't match attached signature
```

This is a deployment/install issue, not a HAPI application error. The deploy
scripts handle it end to end:

1. back up the installed binary to `~/.hapi/bin/backups/` (newest
   `HAPI_KEEP_BACKUPS` kept, default 2; never exec from the backup dir)
2. install the new file with a fresh mtime so the path-keyed signature cache
   re-reads it
3. prove it execs repeatedly (`hapi --version` x3) and `codesign --verify`
4. restart the hub (local) / kickstart the launchd job (remote) and verify
   (`/health` locally; runner state + exec path on the remote)
5. on any failure restore the backup onto the fixed path and restart

Never leave an unverified binary installed.

### Signing

Sign every build with one pinned identity; never ship an ad-hoc signature
(`codesign --sign -`), which has no stable identity and makes TCC re-ask every
permission:

- `scripts/sign-build.sh` (called by both deploy scripts) stores the chosen
  Apple Development SHA-1 in `~/.hapi/signing-identity` and reuses it; override
  with `HAPI_SIGN_IDENTITY` when rotating certificates
- all builds use the signed identifier `run.hapi.cli`
- with no Apple Development identity the scripts fall back to ad-hoc; that
  still runs, but expect TCC prompts to reappear after every deploy

codesign reads the private key from the login keychain, and that key is
reachable only from a GUI (Aqua) session: an agent or daemon shell runs in
launchd's Background domain, where codesign fails with `errSecInternalComponent`
even while Keychain Access shows the keychain unlocked. Measured on macOS 15.6:
in a Background session (even a plain `ssh localhost`) `security
show-keychain-info` already fails with `SecKeychainCopySettings ...: User
interaction is not allowed` (exit 36), while the same command in Aqua reports
the keychain unlocked - the failure is session-scoped keychain access, not an
app allow-list on the key.

Two ways out:

- run the deploy from a Terminal window on the machine (the normal path), or
- if it must run from a Background session, unlock the keychain inside that
  session with the password - the interactive form cannot prompt there, so use
  `security unlock-keychain -p '<login password>' ~/Library/Keychains/login.keychain-db`
  (the CI-style recipe; verify it with the sign probe before relying on it).

Granting the key "allow all applications" in Keychain Access does not address
this: the key's ACL is only consulted once the session can reach the keychain at
all.

Agent sessions should also avoid recursive `$HOME` sweeps (`find ~`,
`du -sh ~`, ...) unless they prune TCC-protected folders (`~/Music`,
`~/Pictures`, `~/Movies`, `~/Library`): a walk into
`~/Music/Music/Media.localized` raises a spurious Apple Music prompt
attributed to the hapi binary.

### Refresh the runner

The runner is a long-lived detached process. New sessions it spawns resolve
`~/.hapi/bin/hapi` and therefore pick up the new binary automatically, but the
runner's own code stays old until the process restarts:

- compiled binaries never self-update: the heartbeat compares the mtime of the
  runner's own resolved exec path, which is fixed for the life of the process
- a stale runner keeps old machine RPC handlers and capability flags, so hub
  features fail with "restart the runner" errors

`scripts/deploy-local.sh` runs `hapi runner start` when a runner is already
running; the CLI stops the stale runner and starts a fresh one with
`HAPI_CLI_EXECUTABLE` pinned to the fixed path. Running sessions are detached
and survive the restart.

The refreshed runner inherits the deployment shell's environment, including
`PATH`. Deploy from a normal login Terminal without replacing `PATH` with a
short list of system directories: agents installed under `~/.local/bin`,
`~/.opencode/bin`, `~/.grok/bin`, or nvm otherwise disappear from availability
checks and model discovery fails with `spawn codex ENOENT`. Temporary `scp` or
`rsync` wrappers must prepend to the existing `PATH` and apply only to the
remote transfer command, not the local deployment or runner refresh.

After refreshing, verify the authenticated Hub APIs
`GET /api/machines/<machineId>/agent-availability` and
`GET /api/machines/<machineId>/codex-models`. Compare availability with the
installed agents before deployment, and require a successful Codex model
response where Codex is installed. `/health`, binary version, signature, and
runner PID alone do not validate the runner's launch environment. If it was
replaced, restore it from a normal login Terminal, then repeat the API checks:

```bash
HAPI_CLI_EXECUTABLE="$HOME/.hapi/bin/hapi" "$HOME/.hapi/bin/hapi" runner start
```

### Manual sequence

```bash
set -euo pipefail
build=cli/dist-exe/bun-darwin-arm64/hapi
bin_dir="$HOME/.hapi/bin"
stable="$bin_dir/hapi"
stamp=$(date +%Y%m%d-%H%M%S)

# Sign with the pinned identity (see scripts/sign-build.sh).
bash scripts/sign-build.sh "$build"

# Back up the installed binary for rollback.
mkdir -p "$bin_dir/backups"
backup="$bin_dir/backups/hapi.$stamp"
[ -f "$stable" ] && cp -p "$stable" "$backup"

# Install to the fixed path with a fresh mtime, then prove it execs.
rm -f "$stable"
cp "$build" "$stable"
chmod 755 "$stable"
"$stable" --version && "$stable" --version && "$stable" --version
codesign --verify --deep --strict "$stable"

launchctl kickstart -k "gui/$(id -u)/com.hapi.hub"
sleep 2
curl -fsS http://127.0.0.1:3006/health >/dev/null
"$stable" --version

# Refresh a running runner so its machine RPCs/capabilities match.
HAPI_CLI_EXECUTABLE="$stable" "$stable" runner start
```

If the health check or a cold start fails, restore the backup and restart:

```bash
rm -f "$stable"
cp "$backup" "$stable"
chmod 755 "$stable"
"$stable" --version
launchctl kickstart -k "gui/$(id -u)/com.hapi.hub"
```

### Remote machines

`scripts/deploy-remote.sh <ssh-target> [tag]` performs the same flow over
SSH: signs locally, checks the remote arch matches, uploads next to the fixed
path, installs by move (fresh inode + mtime), proves the binary execs, then
kickstarts the launchd job (`com.hapi.runner` by default; override with
`HAPI_REMOTE_LAUNCHD_LABEL`) and verifies the runner executes the fixed path.
On any failure it restores the backup and kicks the job again. Running
sessions survive; the remote host needs no build toolchain.