You
Which queue should the new export pipeline use? Compare SQS, Kafka, and our Postgres queue.
Assistant
I compared the three queue options for the export pipeline:
| Option | Throughput | Ops burden | Fit |
|---|---|---|---|
| SQS | 3,000 msg/s | low | good |
| Kafka | 50,000 msg/s | high | overkill |
| Postgres queue | 800 msg/s | lowest | enough today |
Peak export load is 120 msg/s, so I recommend the Postgres queue now and revisiting at 10x growth.
You
At what load should we switch away from the Postgres queue?
Assistant
At 10x (1,200 msg/s) the Postgres queue is past its 800 msg/s ceiling, so the switch point is about 6x current load.
- Below 6x: Postgres queue, no new infrastructure.
- Above 6x: SQS, since the job interface stays the same.
You
What does each option cost per month?
Assistant
Costs per month at current volume:
| Option | Estimate |
|---|---|
| Postgres queue | $0 extra |
| SQS | about $15 |
| Kafka (managed) | about $480 |
You
Sketch the interface so a later switch is cheap.
Assistant
Proposal: put the queue behind a JobQueue interface with enqueue and claim. The Postgres implementation uses FOR UPDATE SKIP LOCKED, and a later SQS adapter is a drop-in.