prd-us-db memory exhaustion

Why the shared v3 Postgres instance crossed its 95% memory alarm on August 13, and why it will keep crossing it every busy afternoon until it gets more memory.

What fired

At 21:20 BST the prd-us-db-memory P2 fired: RDS freeable memory below 5% of the instance's 2 GiB, which is about 102 MB. The alarm is generated by the shared rds/pg/v1 Terraform module from FreeableMemory, so "memory utilization above 95%" means the operating system on the DB host has less than 102 MB it could hand out.

Memory on this instance follows a daily cycle: it recovers overnight to roughly 250 MB free, then falls through the US business afternoon as queries warm the working set. The alert is simply the afternoon trough crossing the line. Here is the day it fired:

Three months of ratchet

The trough has been sinking all quarter. Daily minimum freeable memory was around 400 MB in early May; this week it bottomed between 77 and 130 MB. The drops are not gradual: each one lines up with services being added to the instance.

The mechanism is connections, but not the services' own pools. A live check of pg_stat_activity (August 13, 22:30 UTC) shows who actually holds them: the CDC user has 106 of the ~156 connections, and all sixteen service pools together hold 42, each service sitting at 2 or 3. pgbouncer's per-service cap of 20 never comes close. The CDC connections scale with databases: every database on the instance carries a prd_us_v3_* replication slot costing ~3 connections, and the five databases Sequin actively syncs (messaging, lawrence_api, identity, matter, work) hold 14 CDC connections each. So a service launch raises the floor mainly through CDC: a new database means a new slot, and Sequin onboarding means ~11 more connections on top. A Postgres connection is a process with several MB of resident memory, so 106 CDC connections cost hundreds of MB that never come back.

The overnight connection floor still tells the growth story cleanly, it just needs the right reading: each step is a launch bringing its database, its replication slots, and its CDC sync connections, not a service-side pool.

The host has been compensating by swapping. Average swap usage grew from 24 MB in mid-July to around 100 MB now, which is the operating system telling us the working set no longer fits in RAM:

What shares this instance

The instance is defined in lawhive.infra.compute/databases/tf/modules/service/service.tf and sits behind a pgbouncer + haproxy proxy layer. One logical database per service:

The two CDC readers hold 22 active replication slots between them (a prd_us_v3_* slot per database, plus a sequin_slot_* on each actively synced one) and 106 of the instance's ~156 connections. Each walsender doing logical decoding can also claim up to logical_decoding_work_mem (64 MB by default) while it works, so CDC is the biggest memory consumer on the box as well as the biggest connection holder. CPU is not part of this story: utilization averages 7% and the burst-credit balance has sat pegged at its 576 maximum for the entire month.

Root cause

The fix, and what applying it costs

lawhive.infra.compute#122 bumps prd-us from db.t4g.small to db.t4g.large: same 2 vCPU, memory quadruples to 8 GiB, cost goes from about $24 to about $96 a month. The backend guild (2026-08-14) chose large over medium deliberately: two firm data migrations land in the coming months, bulk loads amplify through logical decoding across all 22 CDC slots (each walsender can claim up to 64 MB while decoding), and going straight to large is one resize outage instead of a possible two. The guild also agreed to migrate this instance to Aurora, which resets the sizing question later; large is the interim ceiling, not a trajectory.

Applying it needs care on two points:

• Merging is safe; applying is the outage. The Opentofu (db) workflow only runs on manual dispatch, so nothing happens on merge. But the shared RDS module sets apply_immediately = true, so the resize starts the moment someone dispatches the apply, not at the next maintenance window.
• The resize is a hard stop. The instance is single-AZ with no standby, so it restarts on the new instance class: typically 5 to 15 minutes with the database unreachable for all 16 services and both CDC readers. Dispatch the apply in a quiet window; the instance's RDS maintenance window (Monday 06:00 to 07:00 UTC) is a reasonable default.

Before the data-migration service lands

A new service for data migrations is planned, and it will bulk-load two firms' worth of data into this instance. The resize to large covers the memory side: another database means another replication slot per CDC reader plus a small service pool, and 8 GiB absorbs the decode-buffer spikes that bulk loads push through the CDC slots. Two other limits will bite first:

• Storage has a hard 100 GB cap. The stack sets initial_gb = 100, max_gb = 100, which disables storage autoscaling entirely. Only ~7 GB is used today, but "dump a ton of data" plus 14 days of backup churn can close that gap fast, and running out of storage takes Postgres down harder than running out of memory. Raising max_gb is an online change with no downtime (RDS allows one storage modification per 6 hours), so raise it before the migration service ships, not during its first big load.
• Bulk loads amplify through CDC. Every row written flows into the logical replication stream that Sequin and Datastream consume. If a consumer can't keep up, Postgres retains WAL on disk until it does, which eats into the same 100 GB. Either pause or filter CDC for migration tables, or watch replication-slot lag during the first load.

Follow-ups