Diagnosing Postgres /dev/shm Exhaustion in Containers: A Field Runbook

The Symptom Root Cause Diagnostic Method 1. Confirm the ceiling 2. Confirm the query actually goes parallel 3. Build a timeline from the Postgres log The Fix, in Layers Measure the tourniquet's cost rather than guessing Platform Gotchas: Coolify Pre-Flight Validation Monitoring Results Second-Order Finding: Indexes and Pattern Matching Runbook Generalizable Lessons

🎯

The one-line version: If containerized Postgres throws could not resize shared memory segment ... No space left on device while the host has plenty of RAM and disk, the cause is almost certainly Docker's default 64 MB /dev/shm. Parallel query workers allocate dynamic shared memory there, and a few concurrent queries exhaust it.

The Symptom

A production Django application began returning intermittent 500s under load, alongside general slowness. The tracked error:


OperationalError: could not resize shared memory segment "/PostgreSQL.4099765926"
to 16777216 bytes: No space left on device

The stack trace ended somewhere unremarkable: Django admin's paginator calling COUNT(*), and DRF pagination calling queryset.count(). Meanwhile the host was entirely healthy: 30 GiB RAM with 25 GiB available, disk at 38 percent, load average 0.41 on 8 vCPU.

That contradiction is the whole trap. The phrase No space left on device sends people to df, to disk alarms, to RAM graphs. All of them look fine, so the investigation stalls and the issue gets labelled a hardware mystery.

Root Cause

Postgres runs parallel queries by launching worker processes that share state through a dynamic shared memory (DSM) segment. With the default dynamic_shared_memory_type = posix, those segments are files in /dev/shm.

Docker gives every container a 64 MB /dev/shm by default. It is a tmpfs cap, unrelated to host memory. When several parallel queries run at once, each growing a segment toward 8 or 16 MB, they collectively hit the 64 MB ceiling and ftruncate fails. Postgres surfaces the ENOSPC verbatim.

So the chain is:


Admin changelist / DRF pagination
  -> SELECT COUNT(*)
  -> planner chooses a parallel plan (Gather node)
  -> workers allocate a DSM segment in /dev/shm
  -> 64 MB cap hit
  -> ENOSPC -> OperationalError -> HTTP 500

⚠️

Why it hides from inspection. DSM segments are transient. They are created per query and released when it finishes. Checking /dev/shm at idle shows roughly 1 MB used out of 64 MB, which looks like enormous headroom. The exhaustion only exists during concurrent load, so a spot check actively misleads you.

Diagnostic Method

1. Confirm the ceiling


docker inspect $PG_CONTAINER -f 'ShmSize={{.HostConfig.ShmSize}}'
# 67108864  = 64 MB = the Docker default, never overridden

PID=$(docker inspect $PG_CONTAINER -f '{{.State.Pid}}')
grep ' /dev/shm ' /proc/$PID/mountinfo | grep -o 'size=[0-9]*k'
# size=65536k

Read the mount from /proc/PID/mountinfo on the host. BusyBox df inside a minimal image often cannot resolve /dev/shm through nsenter, and the host-side mounts/shm path does not exist on all Docker versions.

2. Confirm the query actually goes parallel


EXPLAIN SELECT COUNT(*) FROM app_order_item_versions;

If the plan contains a Gather node with Workers Planned: 2, that query allocates DSM. A plan with a plain Aggregate does not. This is the step that converts a plausible theory into a closed causal chain.

3. Build a timeline from the Postgres log

The container log is independent of your application's logging, so it survives application-side blind spots.


# errors per day
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | awk '{print $1}' | sort | uniq -c

# errors per hour on a spike day
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | grep '^YYYY-MM-DD' | cut -c12-13 | sort | uniq -c

# requested segment size distribution by month
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | sed -E 's/^([0-9]{4}-[0-9]{2}).* to ([0-9]+) bytes.*/\1 \2/' | sort | uniq -c

Two things this surfaced that nothing else did:

Errors clustered into single hours (504 in one hour on one day). That density rules out human browsing and points at automated traffic or a scheduled job.

The requested segment size shifted over time. Early failures were all at 16 MB; later ones were mostly at 8 MB. DSM segments grow by doubling, so failing at a smaller size means the pool is more contended than before. That is a pressure signal you get for free from the log text.

🔍

On attributing a start date. It is tempting to treat the first-seen date in an error tracker as the day the offending workload began. Check what else changed around then. In this case a code change three days earlier had added an unindexed icontains search field plus a join to the same endpoint, which made an already expensive query heavier. A code change explained the onset far better than the "new client appeared" theory, and the sources that looked like corroboration were only reflecting when a different log started. Verify that each of your evidence sources is genuinely independent before treating agreement between them as confirmation.

The Fix, in Layers

Layer	Change	Downtime	Role
A. Tourniquet	`max_parallel_workers_per_gather = 0` plus `pg_reload_conf()`	None (SIGHUP)	Stops the bleeding immediately by removing Gather nodes entirely
B. Real fix	`--shm-size=1g` on the container	Container recreate	Removes the actual ceiling
C. Tuning	`shared_buffers`, `work_mem`, `effective_cache_size`, `maintenance_work_mem`, `random_page_cost`	Only `shared_buffers` needs a restart	Addresses the separate slowness problem

🚨

Ordering is load-bearing. Raise /dev/shm before raising work_mem and before restoring parallelism. A larger work_mem makes parallel hash joins allocate bigger DSM segments. Doing tuning first actively makes the 500s worse. This is the single easiest way to turn a fix into an outage.

Measure the tourniquet's cost rather than guessing

Disabling parallelism is not free, but the cost is measurable in about a minute. Use a session-level SET, which affects only your own connection:


EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM big_table;
SET max_parallel_workers_per_gather = 0;
EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM big_table;

🧪

Control for cache warming. The first run pulls pages into cache, so whichever query runs second looks artificially fast. Our first measurement showed serial beating parallel, which was pure cache effect. Re-running with the order reversed gave the true picture: parallel 330 ms vs serial 601 ms. Always run the comparison in both orders before believing either number.

Platform Gotchas: Coolify

These generalize to any PaaS that generates compose files from its own database.

💣

Never hand-edit the generated docker-compose.yml. Coolify regenerates it from its internal Postgres on every deploy. A manual shm_size: works until the next deploy silently reverts it, and the incident returns weeks later with no apparent cause. Set it through the UI field that persists in the platform's own database.

--shm-size is supported for databases. It is whitelisted, mapped to compose shm_size, and merged into the service definition. The UI field lives under Runtime and network → Custom Docker options.

The "Custom PostgreSQL configuration" field is a landmine. It does not append to your config. It repoints config_file to a new path, and hba_file and ident_file default to the directory containing config_file. Move the config and those defaults move with it, to a directory that does not exist. listen_addresses also reverts to localhost, so the application cannot connect. Any custom config must re-anchor all four explicitly:


data_directory = '/var/lib/postgresql/data'
hba_file = '/var/lib/postgresql/data/pg_hba.conf'
ident_file = '/var/lib/postgresql/data/pg_ident.conf'
listen_addresses = '*'

postgresql.auto.conf wins. Anything set via ALTER SYSTEM overrides the platform's custom config file. If you used ALTER SYSTEM for a tourniquet, you must ALTER SYSTEM RESET it, or the platform UI will appear to be ignored.

Verify the restart button recreates rather than restarts. A plain docker restart will not pick up a new shm_size; only recreation will. Coolify's restart does docker rm -f then regenerate then up, which is correct. Confirm this for your platform before relying on it.

Pre-Flight Validation

This technique de-risked the entire change and is the most reusable thing here. You can validate a Postgres config without starting a server. postgres -C reads a config and prints a resolved value, then exits.


docker cp candidate.conf $PG_CONTAINER:/tmp/pgtest.conf
docker exec $PG_CONTAINER sh -c 'chown postgres /tmp/pgtest.conf'

for s in hba_file ident_file listen_addresses data_directory \
         shared_buffers work_mem effective_cache_size; do
  printf '%-24s ' $s
  docker exec $PG_CONTAINER su postgres -c \
    "postgres -D /var/lib/postgresql/data -c config_file=/tmp/pgtest.conf -C $s"
done

This confirmed ahead of time that hba_file resolved back into the data directory rather than the non-existent one, which is precisely the failure that would have prevented startup. Also check shared_memory_type (should be mmap, so a large shared_buffers uses anonymous mmap rather than SysV limits or /dev/shm) and that pg_hba.conf actually exists at the resolved path.

Monitoring

📊

Cumulative counters lie. pg_stat_database survives restarts. If stats_reset is NULL, the cache hit ratio you are reading covers the entire life of the database, dominated by however long it ran misconfigured. Ours read 85.5 percent lifetime with 385 TB of disk reads, which said nothing about the present.

Sample a delta over a live window instead:


Q="SELECT sum(blks_hit), sum(blks_read) FROM pg_stat_database WHERE datname='mydb';"
A=$(psql -t -A -F'|' -c "$Q"); sleep 90; B=$(psql -t -A -F'|' -c "$Q")
echo $A $B | awk -F'[ |]' '{h=$3-$1; r=$4-$2; \
  printf "cache_hit=%.2f%% (hits=%d reads=%d)\n", 100.0*h/(h+r+0.0001), h, r}'

That returned 100 percent, with five disk reads in 90 seconds of production traffic, versus 85.5 percent lifetime. Completely different stories from the same table.

For the change window itself, poll the container's StartedAt until it moves, then capture verification automatically. A restart triggered by someone else in a UI is easy to miss, and you want the post-restart state captured at the moment it happens rather than reconstructed later.

Results

Metric	Before	After
`/dev/shm`	64 MB	1 GB
`shared_buffers`	128 MB	8 GB
`work_mem`	4 MB	32 MB
shm errors	72 per 72 hours	0 in 34 days
`COUNT(*)` on 4.5M rows	330 ms	164 ms
`COUNT(*)` on 330k rows	118 ms	52 ms
Cache hit (live window)	85.5% lifetime	99.8% measured
Host load average	1.39	0.39

Cost of the change: one restart producing 8 seconds of connection errors affecting 25 users. Worth stating plainly in your own write-ups. The estimate beforehand was 30 to 60 seconds, so the real number was better, but it was not zero and real people saw errors.

A month later: 34 days uninterrupted uptime, zero shm errors, every setting persisted through the platform's own config management.

Second-Order Finding: Indexes and Pattern Matching

Raising the ceiling stopped the outages but did not reduce the load. Investigating the expensive queries produced a lesson worth separating out, because it contradicts a common instinct.

"Add indexes to the columns used in search" is frequently wrong. Verified against a column that already carried both a unique B-tree and a varchar_pattern_ops index:

Predicate	Django / DRF equivalent	Plan
`col = 'ABC'`	`exact`	Index Scan
`col LIKE 'ABC%'`	`startswith`	Index Scan
`col ILIKE 'ABC%'`	`istartswith`, DRF `^` prefix	Seq Scan
`col ILIKE '%ABC%'`	`icontains`	Seq Scan
`lower(col) LIKE 'abc%'`	manual lowering	Seq Scan without a functional index

A B-tree cannot serve a leading-wildcard match, and case-insensitivity defeats it even for prefixes. So indexing the columns behind an icontains search accomplishes nothing, and DRF's ^ prefix marker does not get you an index either, which surprises most people.

The correct order of work:

Add exact filters first. Convert the lookup from ILIKE '%x%' to = 'x'. That is the only shape a B-tree can serve.

Then index them. The index is worthless until step 1 exists.

Short-circuit expensive optional branches. One filter here OR-ed four unindexable JSONB icontains conditions against a pk__in subquery, which destroys index usage on both sides. Running that branch only when the cheap search returns nothing is a small change with a large payoff.

Reach for pg_trgm GIN indexes last, only for free-text paths you have decided to keep.

For scale: one single-record lookup through the unindexable path read 1.2 GB of buffers to return zero rows. The equivalent indexed lookup ran in 58 ms.

📌

Also worth knowing: django-simple-history accounted for 59 percent of this database. History tables held roughly 15 revision rows per live record and grew 17 percent in one month. If you use it, plan retention from day one. It is the one category of growth that compounds while you are not looking.

Runbook

Substitute your own container name. Verify after every step.


# 0. BASELINE - record everything before touching anything
psql -c "SELECT name,setting,source FROM pg_settings WHERE name IN
  ('max_parallel_workers_per_gather','shared_buffers','work_mem',
   'effective_cache_size','maintenance_work_mem','random_page_cost');"
cat /var/lib/postgresql/data/postgresql.auto.conf
docker inspect $PG_CONTAINER -f 'ShmSize={{.HostConfig.ShmSize}}'

# 1. TOURNIQUET - zero downtime, stops the 500s now
psql -c "ALTER SYSTEM SET max_parallel_workers_per_gather = 0;"
psql -c "SELECT pg_reload_conf();"
# verify on a NEW connection - SHOW in the same session reports the old value
psql -c "SHOW max_parallel_workers_per_gather;"   # expect 0
psql -c "EXPLAIN SELECT COUNT(*) FROM big_table;" # expect NO Gather node

# 2. RAISE SHM LIVE - still zero downtime, does not survive restart
PID=$(docker inspect $PG_CONTAINER -f '{{.State.Pid}}')
nsenter -t $PID -m -- /bin/mount -o remount,size=1G /dev/shm
grep ' /dev/shm ' /proc/$PID/mountinfo | grep -o 'size=[0-9]*k'  # 1048576k

# 3. PERSIST IT - platform UI field, never the generated compose file
#    Coolify: Runtime and network -> Custom Docker options -> --shm-size=1g

# 4. TUNING - reload-safe settings only. SHM MUST ALREADY BE RAISED.
psql <<'SQL'
ALTER SYSTEM SET work_mem = '32MB';
ALTER SYSTEM SET effective_cache_size = '22GB';   -- ~75% of RAM
ALTER SYSTEM SET maintenance_work_mem = '1GB';
ALTER SYSTEM SET random_page_cost = 1.1;          -- SSD
SELECT pg_reload_conf();
SQL

# 5. RESTORE PARALLELISM - only after shm is confirmed raised
psql -c "ALTER SYSTEM RESET max_parallel_workers_per_gather;"
psql -c "SELECT pg_reload_conf();"

# 6. DEFERRED - shared_buffers needs a restart. Back up first.
#    ALTER SYSTEM SET shared_buffers = '8GB';   -- 25% of RAM

# 7. VERIFY
docker logs --since 1h $PG_CONTAINER 2>&1 | grep -c 'could not resize shared memory'

✅

Verification traps worth internalizing. SHOW in the same session that ran ALTER SYSTEM reports the pre-reload value; always check on a fresh connection. ALTER SYSTEM without pg_reload_conf() writes the setting but does not activate it, which is genuinely useful when you want a change to take effect only at the next restart. And confirm your data lives in a named volume before any recreation: docker rm -f removes only the container, while docker compose down -v destroys volumes.

Generalizable Lessons

Read the error message literally, then find the right device. "No space left on device" was accurate. The device was a 64 MB tmpfs inside a container, not the 619 GB disk everyone was staring at.

A resource that looks idle may still be exhausted. Transient allocations are invisible to spot checks. Sample under load or reason from the failure itself.

Separate the ceiling from the load. Raising /dev/shm stopped the outages. It did not make the queries cheaper. Both are real work and conflating them hides one of them.

Validate config before you apply it. postgres -C turns a risky restart into a verified one for about 30 seconds of effort.

Distrust cumulative counters. Delta-sample over a live window instead.

Sequence changes by their interactions. work_mem up before /dev/shm up makes things worse. Write the ordering constraint down before you start.

Check whether your evidence sources are actually independent. Two sources agreeing means nothing if they share an upstream cause or a common blind spot.

Report the cost honestly. Eight seconds of errors for 25 users is a real cost. Publishing it alongside the win is what makes the next change easy to approve.

Investigation and remediation performed with Claude Code. All hostnames, IP addresses, container identifiers, credentials, schema names, and administrative paths have been genericized or removed.