Pasted images were stored inline in the message JSON as full-size base64
data URLs. Every conversation load (open, paginate, mobile reconnect) had
to download and parse those megabytes before anything rendered, with no
lazy loading and no per-image caching. Measured on this hub: 115 messages
holding 138MB of base64, the largest single message 18.3MB.
New messages now carry only a small thumbnail (browser-generated, 768px
webp/jpeg data URL); the original is uploaded as before and the hub keeps
a durable copy under ~/.hapi/attachments (table chat_attachments, schema
v27). The chat renders the thumbnail instantly and lazily swaps in the
original from GET /api/sessions/:id/attachments/:id when the viewer opens;
that endpoint answers sha256 ETag + immutable Cache-Control so repeated
opens are free. Sessions scope the URL, deletes remove the copy, and
hub/scripts/attachments.ts covers stats/verify/gc/export.
Old messages keep their inline previews (migration deliberately deferred).
Verified end-to-end against the local hub: upload returns the attachment
id/url, original 200 with ETag, revalidation 304, cross-session 404, delete
removes file+row (404 after). Typecheck clean; suite status matches the
pre-change baseline; fixtures unchanged.
Implement two-stage content codec: truncate oversized agent messages
(48KB head + 12KB tail + marker, idempotent, never user messages) and
compress message content JSON ≥256 bytes via zstd with fallback.
- contentCodec.ts: new codec with truncation and compression routines
- messages.ts: wire codec into addMessage (truncate+compress),
copyMessageToSession (lossless), toStoredMessage (decode both formats)
- index.ts: schema version 15→16 with no-op migration (enforces
schema match on downgrade, no DDL change)
- codexDesktop.ts: pass content through idempotent truncation for
canonical comparison in transcript-import
- cleanup-sessions.ts: decode compressed rows, replace full-scan with
per-session batched scan (LIMIT 50) stopping at first user message
- compact-db.ts: new offline compactor that retroactively applies
truncate+compress+VACUUM to existing DBs; guards against schema
version mismatch, symlink aliasing, and handles re-runs
- Tests: contentCodec round-trip + truncation + idempotence + legacy
decode; messages integration (compressed/truncated round-trip);
migration tests updated to version 16
- README.md: document codec and both maintenance scripts
Measured on 2.26GB production DB: 2159MB→768MB (-64%) in 12s,
748k messages intact, integrity verified.
wushenghua
weishu