When an Anthropic upstream returned HTTP 200 but then emitted an SSE
`event: error` frame (overloaded_error / rate_limit_error / api_error /
etc.), Forward's stream branch matched on `err.Error() == "have error in
stream"` and returned `UpstreamFailoverError{StatusCode: 403}` with no
ResponseBody. That dropped three pieces of evidence:
- handleFailoverExhausted → ExtractUpstreamErrorMessage(nil) = "" →
ops_error_logs.upstream_error_message was empty.
- errorPassthroughService.MatchRule(_, 403, nil) could only match rules
without keywords, so keyword-based passthrough rules silently never
fired.
- upstream_errors carried no stream_error record, leaving ops looking at
a generic 403 with no clue whether the upstream was throttled,
overloaded, or rejecting the request. ping-during-slot-wait amplified
this by skipping failover (writerSizeBeforeForward guard), so 403s
ballooned in the ops view well past the upstream's actual rate.
Fix:
- Introduce *sseStreamErrorEventError that carries the SSE data line.
Error() still returns "have error in stream" so existing log searches
keep working.
- Forward extracts via errors.As, appends an OpsUpstreamErrorEvent
(kind="stream_error", with the sanitized message and a truncated raw
body honoring LogUpstreamErrorBody*), and returns
UpstreamFailoverError{StatusCode: 403, ResponseBody: rawJSON}.
StatusCode 403 is preserved verbatim: mapUpstreamError, failover
decisions (shouldFailoverUpstreamError(403)=true), client-visible message,
RetryableOnSameAccount, and rateLimitService side-effects (this path
already didn't invoke them) all match prior behavior. OAuth and API Key
accounts share this path; the API-Key passthrough branch is independent
and already forwards SSE error frames untouched, so it's unaffected.
Adds four unit tests: typed-error contract + RawData, empty data line,
event:error after partial stream output (streamStarted=true), and
non-JSON data line.
Three independent CI blockers landed on main from concurrent PR merges:
- openai_quota_service.go (introduced by b8169492): const block spacing
not gofmt-compliant + trailing blank line. golangci-lint v2.9 flagged it
on every push after the merge.
- openai_images_failover_test.go (introduced by PR #3155, da30c599):
NewOpenAIGatewayHandler call missing the opsService argument added by
PR #3230 (b62b573f). Test was authored before #3230 and merged without
rebase, causing "not enough arguments" compile error.
- account_quota_reset_test.go: TestIsFixedDailyPeriodExpired_NotExpired
and TestIsFixedWeeklyPeriodExpired_NotExpired used time.Now()-1min as
periodStart, which crosses the 09:00 UTC reset boundary when CI runs in
the 09:00:00-09:00:59 window. Anchoring periodStart to today's 12:00
UTC removes the race.
Adds an admin-side action that mirrors the Codex Desktop "rate-limit reset"
flow against chatgpt.com upstream for OpenAI OAuth accounts.
Backend
- OpenAIQuotaService.QueryUsage / ResetCredit hit /wham/usage and
/wham/rate-limit-reset-credits/consume with the Codex Desktop header set,
reusing OpenAITokenProvider for refreshed tokens and PrivacyClientFactory
for the impersonated Chrome TLS fingerprint.
- Honors the account's configured proxy by reading the eager-loaded
account.Proxy directly (falls back to proxyRepo only when missing).
- GET /api/v1/admin/openai/accounts/:id/quota
POST /api/v1/admin/openai/accounts/:id/reset-quota
- Wire DI for the new service + handler dependency.
Frontend
- OpenAIQuotaResetCell renders a single action row in AccountUsageCell's
OpenAI section: the existing local "查询" (active sampling) is injected
via #pre-actions, alongside a "次数 N" button that doubles as the
upstream query trigger and the available-credit indicator, and a "重置"
button that consumes one credit.
- No duplicate 5h/7d window display; the local UsageProgressBar owns those
bars to avoid confusion.
Resolves GHSA-hmw2-7cc7-3qxx (CRLF injection) flagged by
frontend-security CI. axios pulls form-data ^4.0.5 which locked to the
vulnerable 4.0.5; override forces all transitive consumers to 4.0.6+
without needing an audit exception.
shaw