# Diagnosing Postgres /dev/shm Exhaustion in Containers: A Field Runbook

> 🎯
> **The one-line version:** If containerized Postgres throws `could not resize shared memory segment ... No space left on device` while the host has plenty of RAM and disk, the cause is almost certainly Docker's default 64 MB `/dev/shm`. Parallel query workers allocate dynamic shared memory there, and a few concurrent queries exhaust it.
> 
## The Symptom

A production Django application began returning intermittent 500s under load, alongside general slowness. The tracked error:

```
OperationalError: could not resize shared memory segment "/PostgreSQL.4099765926"
to 16777216 bytes: No space left on device
```
The stack trace ended somewhere unremarkable: Django admin's paginator calling `COUNT(*)`, and DRF pagination calling `queryset.count()`. Meanwhile the host was entirely healthy: 30 GiB RAM with 25 GiB available, disk at 38 percent, load average 0.41 on 8 vCPU.

That contradiction is the whole trap. The phrase **No space left on device** sends people to `df`, to disk alarms, to RAM graphs. All of them look fine, so the investigation stalls and the issue gets labelled a hardware mystery.

## Root Cause

Postgres runs parallel queries by launching worker processes that share state through a **dynamic shared memory (DSM) segment**. With the default `dynamic_shared_memory_type = posix`, those segments are files in `/dev/shm`.

**Docker gives every container a 64 MB **`**/dev/shm**`** by default.** It is a `tmpfs` cap, unrelated to host memory. When several parallel queries run at once, each growing a segment toward 8 or 16 MB, they collectively hit the 64 MB ceiling and `ftruncate` fails. Postgres surfaces the `ENOSPC` verbatim.

So the chain is:

```
Admin changelist / DRF pagination
  -> SELECT COUNT(*)
  -> planner chooses a parallel plan (Gather node)
  -> workers allocate a DSM segment in /dev/shm
  -> 64 MB cap hit
  -> ENOSPC -> OperationalError -> HTTP 500
```
> ⚠️
> **Why it hides from inspection.** DSM segments are transient. They are created per query and released when it finishes. Checking `/dev/shm` at idle shows roughly 1 MB used out of 64 MB, which looks like enormous headroom. The exhaustion only exists during concurrent load, so a spot check actively misleads you.
> 
## Diagnostic Method

### 1. Confirm the ceiling

```Bash
docker inspect $PG_CONTAINER -f 'ShmSize={{.HostConfig.ShmSize}}'
# 67108864  = 64 MB = the Docker default, never overridden

PID=$(docker inspect $PG_CONTAINER -f '{{.State.Pid}}')
grep ' /dev/shm ' /proc/$PID/mountinfo | grep -o 'size=[0-9]*k'
# size=65536k
```
Read the mount from `/proc/PID/mountinfo` on the host. BusyBox `df` inside a minimal image often cannot resolve `/dev/shm` through `nsenter`, and the host-side `mounts/shm` path does not exist on all Docker versions.

### 2. Confirm the query actually goes parallel

```SQL
EXPLAIN SELECT COUNT(*) FROM app_order_item_versions;
```
If the plan contains a `Gather` node with `Workers Planned: 2`, that query allocates DSM. A plan with a plain `Aggregate` does not. This is the step that converts a plausible theory into a closed causal chain.

### 3. Build a timeline from the Postgres log

The container log is independent of your application's logging, so it survives application-side blind spots.

```Bash
# errors per day
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | awk '{print $1}' | sort | uniq -c

# errors per hour on a spike day
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | grep '^YYYY-MM-DD' | cut -c12-13 | sort | uniq -c

# requested segment size distribution by month
docker logs $PG_CONTAINER 2>&1 | grep 'could not resize shared memory' \
  | sed -E 's/^([0-9]{4}-[0-9]{2}).* to ([0-9]+) bytes.*/\1 \2/' | sort | uniq -c
```
Two things this surfaced that nothing else did:

- Errors clustered into **single hours** (504 in one hour on one day). That density rules out human browsing and points at automated traffic or a scheduled job.
- The **requested segment size shifted over time**. Early failures were all at 16 MB; later ones were mostly at 8 MB. DSM segments grow by doubling, so failing at a _smaller_ size means the pool is more contended than before. That is a pressure signal you get for free from the log text.
> 🔍
> **On attributing a start date.** It is tempting to treat the first-seen date in an error tracker as the day the offending workload began. Check what else changed around then. In this case a code change three days earlier had added an unindexed `icontains` search field plus a join to the same endpoint, which made an already expensive query heavier. A code change explained the onset far better than the "new client appeared" theory, and the sources that looked like corroboration were only reflecting when a _different_ log started. Verify that each of your evidence sources is genuinely independent before treating agreement between them as confirmation.
> 
## The Fix, in Layers

| Layer | Change | Downtime | Role |
| --- | --- | --- | --- |
| A. Tourniquet | `max_parallel_workers_per_gather = 0` plus `pg_reload_conf()` | None (SIGHUP) | Stops the bleeding immediately by removing Gather nodes entirely |
| B. Real fix | `--shm-size=1g` on the container | Container recreate | Removes the actual ceiling |
| C. Tuning | `shared_buffers`, `work_mem`, `effective_cache_size`, `maintenance_work_mem`, `random_page_cost` | Only `shared_buffers` needs a restart | Addresses the separate slowness problem |
> 🚨
> **Ordering is load-bearing.** Raise `/dev/shm` _before_ raising `work_mem` and _before_ restoring parallelism. A larger `work_mem` makes parallel hash joins allocate **bigger** DSM segments. Doing tuning first actively makes the 500s worse. This is the single easiest way to turn a fix into an outage.
> 
### Measure the tourniquet's cost rather than guessing

Disabling parallelism is not free, but the cost is measurable in about a minute. Use a session-level `SET`, which affects only your own connection:

```SQL
EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM big_table;
SET max_parallel_workers_per_gather = 0;
EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM big_table;
```
> 🧪
> **Control for cache warming.** The first run pulls pages into cache, so whichever query runs second looks artificially fast. Our first measurement showed serial _beating_ parallel, which was pure cache effect. Re-running with the order reversed gave the true picture: parallel 330 ms vs serial 601 ms. Always run the comparison in both orders before believing either number.
> 
## Platform Gotchas: Coolify

These generalize to any PaaS that generates compose files from its own database.

> 💣
> **Never hand-edit the generated **`**docker-compose.yml**`**.** Coolify regenerates it from its internal Postgres on every deploy. A manual `shm_size:` works until the next deploy silently reverts it, and the incident returns weeks later with no apparent cause. Set it through the UI field that persists in the platform's own database.
> 
- `**--shm-size**`** is supported** for databases. It is whitelisted, mapped to compose `shm_size`, and merged into the service definition. The UI field lives under **Runtime and network → Custom Docker options**.
- **The "Custom PostgreSQL configuration" field is a landmine.** It does not append to your config. It repoints `config_file` to a new path, and `hba_file` and `ident_file` default to _the directory containing config_file_. Move the config and those defaults move with it, to a directory that does not exist. `listen_addresses` also reverts to `localhost`, so the application cannot connect. Any custom config must re-anchor all four explicitly:
```Plain Text
data_directory = '/var/lib/postgresql/data'
hba_file = '/var/lib/postgresql/data/pg_hba.conf'
ident_file = '/var/lib/postgresql/data/pg_ident.conf'
listen_addresses = '*'
```
- `**postgresql.auto.conf**`** wins.** Anything set via `ALTER SYSTEM` overrides the platform's custom config file. If you used `ALTER SYSTEM` for a tourniquet, you must `ALTER SYSTEM RESET` it, or the platform UI will appear to be ignored.
- **Verify the restart button recreates rather than restarts.** A plain `docker restart` will not pick up a new `shm_size`; only recreation will. Coolify's restart does `docker rm -f` then regenerate then up, which is correct. Confirm this for your platform before relying on it.
## Pre-Flight Validation

This technique de-risked the entire change and is the most reusable thing here. **You can validate a Postgres config without starting a server.** `postgres -C` reads a config and prints a resolved value, then exits.

```Bash
docker cp candidate.conf $PG_CONTAINER:/tmp/pgtest.conf
docker exec $PG_CONTAINER sh -c 'chown postgres /tmp/pgtest.conf'

for s in hba_file ident_file listen_addresses data_directory \
         shared_buffers work_mem effective_cache_size; do
  printf '%-24s ' $s
  docker exec $PG_CONTAINER su postgres -c \
    "postgres -D /var/lib/postgresql/data -c config_file=/tmp/pgtest.conf -C $s"
done
```
This confirmed ahead of time that `hba_file` resolved back into the data directory rather than the non-existent one, which is precisely the failure that would have prevented startup. Also check `shared_memory_type` (should be `mmap`, so a large `shared_buffers` uses anonymous mmap rather than SysV limits or `/dev/shm`) and that `pg_hba.conf` actually exists at the resolved path.

## Monitoring

> 📊
> **Cumulative counters lie.** `pg_stat_database` survives restarts. If `stats_reset` is NULL, the cache hit ratio you are reading covers the entire life of the database, dominated by however long it ran misconfigured. Ours read 85.5 percent lifetime with 385 TB of disk reads, which said nothing about the present.
> 
Sample a delta over a live window instead:

```Bash
Q="SELECT sum(blks_hit), sum(blks_read) FROM pg_stat_database WHERE datname='mydb';"
A=$(psql -t -A -F'|' -c "$Q"); sleep 90; B=$(psql -t -A -F'|' -c "$Q")
echo $A $B | awk -F'[ |]' '{h=$3-$1; r=$4-$2; \
  printf "cache_hit=%.2f%% (hits=%d reads=%d)\n", 100.0*h/(h+r+0.0001), h, r}'
```
That returned **100 percent, with five disk reads in 90 seconds** of production traffic, versus 85.5 percent lifetime. Completely different stories from the same table.

For the change window itself, poll the container's `StartedAt` until it moves, then capture verification automatically. A restart triggered by someone else in a UI is easy to miss, and you want the post-restart state captured at the moment it happens rather than reconstructed later.

## Results

| Metric | Before | After |
| --- | --- | --- |
| `/dev/shm` | 64 MB | 1 GB |
| `shared_buffers` | 128 MB | 8 GB |
| `work_mem` | 4 MB | 32 MB |
| shm errors | 72 per 72 hours | 0 in 34 days |
| `COUNT(*)` on 4.5M rows | 330 ms | 164 ms |
| `COUNT(*)` on 330k rows | 118 ms | 52 ms |
| Cache hit (live window) | 85.5% lifetime | 99.8% measured |
| Host load average | 1.39 | 0.39 |
**Cost of the change:** one restart producing 8 seconds of connection errors affecting 25 users. Worth stating plainly in your own write-ups. The estimate beforehand was 30 to 60 seconds, so the real number was better, but it was not zero and real people saw errors.

A month later: 34 days uninterrupted uptime, zero shm errors, every setting persisted through the platform's own config management.

## Second-Order Finding: Indexes and Pattern Matching

Raising the ceiling stopped the outages but did not reduce the load. Investigating the expensive queries produced a lesson worth separating out, because it contradicts a common instinct.

**"Add indexes to the columns used in search" is frequently wrong.** Verified against a column that already carried both a unique B-tree and a `varchar_pattern_ops` index:

| Predicate | Django / DRF equivalent | Plan |
| --- | --- | --- |
| `col = 'ABC'` | `exact` | Index Scan |
| `col LIKE 'ABC%'` | `startswith` | Index Scan |
| `col ILIKE 'ABC%'` | `istartswith`, DRF `^` prefix | **Seq Scan** |
| `col ILIKE '%ABC%'` | `icontains` | **Seq Scan** |
| `lower(col) LIKE 'abc%'` | manual lowering | **Seq Scan** without a functional index |
A B-tree cannot serve a leading-wildcard match, and case-insensitivity defeats it even for prefixes. So indexing the columns behind an `icontains` search accomplishes nothing, and **DRF's **`**^**`** prefix marker does not get you an index either**, which surprises most people.

The correct order of work:

1. **Add exact filters first.** Convert the lookup from `ILIKE '%x%'` to `= 'x'`. That is the only shape a B-tree can serve.
2. **Then index them.** The index is worthless until step 1 exists.
3. **Short-circuit expensive optional branches.** One filter here OR-ed four unindexable JSONB `icontains` conditions against a `pk__in` subquery, which destroys index usage on _both_ sides. Running that branch only when the cheap search returns nothing is a small change with a large payoff.
4. **Reach for **`**pg_trgm**`** GIN indexes last**, only for free-text paths you have decided to keep.
For scale: one single-record lookup through the unindexable path read **1.2 GB of buffers to return zero rows**. The equivalent indexed lookup ran in 58 ms.

> 📌
> **Also worth knowing:** `django-simple-history` accounted for 59 percent of this database. History tables held roughly 15 revision rows per live record and grew 17 percent in one month. If you use it, plan retention from day one. It is the one category of growth that compounds while you are not looking.
> 
## Runbook

Substitute your own container name. Verify after every step.

```Bash
# 0. BASELINE - record everything before touching anything
psql -c "SELECT name,setting,source FROM pg_settings WHERE name IN
  ('max_parallel_workers_per_gather','shared_buffers','work_mem',
   'effective_cache_size','maintenance_work_mem','random_page_cost');"
cat /var/lib/postgresql/data/postgresql.auto.conf
docker inspect $PG_CONTAINER -f 'ShmSize={{.HostConfig.ShmSize}}'

# 1. TOURNIQUET - zero downtime, stops the 500s now
psql -c "ALTER SYSTEM SET max_parallel_workers_per_gather = 0;"
psql -c "SELECT pg_reload_conf();"
# verify on a NEW connection - SHOW in the same session reports the old value
psql -c "SHOW max_parallel_workers_per_gather;"   # expect 0
psql -c "EXPLAIN SELECT COUNT(*) FROM big_table;" # expect NO Gather node

# 2. RAISE SHM LIVE - still zero downtime, does not survive restart
PID=$(docker inspect $PG_CONTAINER -f '{{.State.Pid}}')
nsenter -t $PID -m -- /bin/mount -o remount,size=1G /dev/shm
grep ' /dev/shm ' /proc/$PID/mountinfo | grep -o 'size=[0-9]*k'  # 1048576k

# 3. PERSIST IT - platform UI field, never the generated compose file
#    Coolify: Runtime and network -> Custom Docker options -> --shm-size=1g

# 4. TUNING - reload-safe settings only. SHM MUST ALREADY BE RAISED.
psql <<'SQL'
ALTER SYSTEM SET work_mem = '32MB';
ALTER SYSTEM SET effective_cache_size = '22GB';   -- ~75% of RAM
ALTER SYSTEM SET maintenance_work_mem = '1GB';
ALTER SYSTEM SET random_page_cost = 1.1;          -- SSD
SELECT pg_reload_conf();
SQL

# 5. RESTORE PARALLELISM - only after shm is confirmed raised
psql -c "ALTER SYSTEM RESET max_parallel_workers_per_gather;"
psql -c "SELECT pg_reload_conf();"

# 6. DEFERRED - shared_buffers needs a restart. Back up first.
#    ALTER SYSTEM SET shared_buffers = '8GB';   -- 25% of RAM

# 7. VERIFY
docker logs --since 1h $PG_CONTAINER 2>&1 | grep -c 'could not resize shared memory'
```
> ✅
> **Verification traps worth internalizing.** `SHOW` in the same session that ran `ALTER SYSTEM` reports the pre-reload value; always check on a fresh connection. `ALTER SYSTEM` without `pg_reload_conf()` writes the setting but does not activate it, which is genuinely useful when you want a change to take effect only at the next restart. And confirm your data lives in a **named volume** before any recreation: `docker rm -f` removes only the container, while `docker compose down -v` destroys volumes.
> 
## Generalizable Lessons

1. **Read the error message literally, then find the right device.** "No space left on device" was accurate. The device was a 64 MB tmpfs inside a container, not the 619 GB disk everyone was staring at.
2. **A resource that looks idle may still be exhausted.** Transient allocations are invisible to spot checks. Sample under load or reason from the failure itself.
3. **Separate the ceiling from the load.** Raising `/dev/shm` stopped the outages. It did not make the queries cheaper. Both are real work and conflating them hides one of them.
4. **Validate config before you apply it.** `postgres -C` turns a risky restart into a verified one for about 30 seconds of effort.
5. **Distrust cumulative counters.** Delta-sample over a live window instead.
6. **Sequence changes by their interactions.** `work_mem` up before `/dev/shm` up makes things worse. Write the ordering constraint down before you start.
7. **Check whether your evidence sources are actually independent.** Two sources agreeing means nothing if they share an upstream cause or a common blind spot.
8. **Report the cost honestly.** Eight seconds of errors for 25 users is a real cost. Publishing it alongside the win is what makes the next change easy to approve.

---

_Investigation and remediation performed with Claude Code. All hostnames, IP addresses, container identifiers, credentials, schema names, and administrative paths have been genericized or removed._