Orders API 5xx spikeOrders API 5xx spike
You
Why are we seeing 5xx errors on the orders API this afternoon?
Assistant
The 5xx spike on /v2/orders started at 14:02 with deploy api@8f3c1d2.
- Cause: the new rate limiter reads a Redis key that is only set in staging.
- Impact: 2.1% of order reads failed for 18 minutes.
- Fix: PR #742 defaults the limit when the key is missing.
You
When exactly did the first failures start?
Assistant
The first failing request was at 14:02:11, 40 seconds after the deploy finished. All failures were reads on /v2/orders; writes were unaffected.
You
Why did staging not catch this?
Assistant
Why staging did not catch it: the key is seeded there by a setup script that production does not run.
I added the key to the production config checklist and a startup check that fails fast if it is missing.
You
Draft a short status update for the support channel.
Assistant
Draft status update:
Orders API read errors between 14:02 and 14:20 UTC (2.1% of requests). Cause identified and fix PR #742 is in review. No data loss.