Demo ModeExit demoStart free
Ready
You
The realtime gateway pods get OOM killed every 8 hours. Find the leak.
Assistant

The memory growth is a listener leak in the websocket gateway.

Each reconnect registers a new onPresence handler without removing the old one, so a client that reconnects 50 times holds 50 handlers. Heap snapshots show 38,000 retained closures after 6 hours.

  • Fix: unregister the handler in the close event.
  • After the fix the heap stays flat at about 210MB over a 2 hour soak test.
You
Could we just raise the pod memory limit instead?
Assistant

No. The pods were restarting every 8 hours because the heap grew steadily, so a bigger memory limit would only stretch the interval to roughly 16 hours.

You
Does the fix handle abrupt disconnects, not just clean closes?
Assistant

The fix covers both close paths:

  • Clean close: the close event handler calls socket.off('presence', onPresence).
  • Abrupt drop: the heartbeat timeout calls the same cleanup.

I added a test that reconnects 100 times and asserts the listener count stays at 1.

You
How should we roll it out?
Assistant

Rollout plan:

  1. Merge and deploy to one pod.
  2. Watch heap for 2 hours; expect a flat line near 210MB.
  3. Roll out to the rest of the fleet.
You
Add an alert so we catch this earlier next time.
Assistant

Added a memory alert at 400MB sustained for 15 minutes, routed to the realtime channel.