Realtime gateway memory leakRealtime gateway memory leak
You
The realtime gateway pods get OOM killed every 8 hours. Find the leak.
Assistant
The memory growth is a listener leak in the websocket gateway.
Each reconnect registers a new onPresence handler without removing the old one, so a client that reconnects 50 times holds 50 handlers. Heap snapshots show 38,000 retained closures after 6 hours.
- Fix: unregister the handler in the
closeevent. - After the fix the heap stays flat at about 210MB over a 2 hour soak test.
You
Could we just raise the pod memory limit instead?
Assistant
No. The pods were restarting every 8 hours because the heap grew steadily, so a bigger memory limit would only stretch the interval to roughly 16 hours.
You
Does the fix handle abrupt disconnects, not just clean closes?
Assistant
The fix covers both close paths:
- Clean close: the
closeevent handler callssocket.off('presence', onPresence). - Abrupt drop: the heartbeat timeout calls the same cleanup.
I added a test that reconnects 100 times and asserts the listener count stays at 1.
You
How should we roll it out?
Assistant
Rollout plan:
- Merge and deploy to one pod.
- Watch heap for 2 hours; expect a flat line near 210MB.
- Roll out to the rest of the fleet.
You
Add an alert so we catch this earlier next time.
Assistant
Added a memory alert at 400MB sustained for 15 minutes, routed to the realtime channel.