Cutting Kubernetes Cluster RAM by 50%: What Actually Caused the Bloat

We were paying for RAM we never used. This is exactly why the bloat existed - and the reason is almost never where people look first.

The cluster ran a queue-driven backend: campaign delivery over email, WhatsApp and SMS, report generation, webhook fan-out, and an AI/RAG path for analytics Q&A. Every worker pod requested around 1.2 GB. Every worker pod used around 1.2 GB. Autoscaling looked healthy. Nothing was crashing. And yet more than half of that memory was doing nothing at all.

The fix required no new infrastructure, no rewrite, and no vendor. It was an architectural change to how responsibilities were grouped inside a single worker image. Cluster RAM dropped by roughly 50%, p95 latency improved, and OOMKills stopped.

1. The mental model that causes the problem

Most teams reason about background workers like this:

That statement is false for every long-running process, in every language, on every orchestrator. It is the single most expensive misconception in queue-based architecture.

A worker process is not a function invocation. It is a process that boots once, imports your entire module graph, and then stays alive for hours or days pulling messages off a broker. What it imports at boot is what it holds until it dies - regardless of what work actually arrives.

Import cost is paid at startup, not at call time

Consider a task module that looks completely reasonable:

import torch                 # ~500 MB resident
import pandas as pd          # ~120 MB resident
from reportlab import ...    # PDF engine

@task(queue="ai")
def classify_document(doc_id):
    ...

@task(queue="emails")
def send_email(user_id):
    ...

@task(queue="reports")
def build_pdf(report_id):
    ...

When the worker boots, the interpreter executes the module top to bottom. It imports torch. It imports pandas. It imports the PDF engine. Only then does it register the tasks and begin listening.

If that worker listens to the emails queue and processes nothing but emails for the next 24 hours, it still holds every byte of torch in resident memory. classify_document is never called. The memory is spent anyway.

Now multiply by replicas

This is where it becomes an infrastructure bill rather than a curiosity. One deployment, one image, one entrypoint listening to all queues, scaled to N replicas:

Six pods. One of them actually needs the AI dependency tree. All six paid for it. The task distribution is irrelevant - the cost was locked in at boot, before a single message was consumed.

Kubernetes cannot save you here. The scheduler sees a process holding 1,180 MB of RSS. It does not know that 600 MB of it is an inference library that will never be touched by this replica. It schedules honestly against a number that is architecturally wrong.

2. Where the bloat actually came from

When I profiled the workers, the memory fell into four buckets. Only one of them was real work.

Read that last row again. Sixteen percent of provisioned memory was doing the job. The other eighty-four percent was the cost of a shared process boundary.

Bucket 1 - Boot imports (the big one)

One image, one requirements file, one entrypoint listening to every queue. Convenient to deploy, catastrophic for memory. The heaviest consumer in the dependency tree sets the floor for every pod in the fleet.

Bucket 2 - Warm caches that only grow

Short-lived request handlers get away with sloppy caching because the process dies. Workers do not die. A module-level dict that caches tenant configuration is harmless in a web request and becomes a slow leak in a process that lives for a week.

_config_cache = {}   # unbounded, process-lifetime, never evicted

def get_config(tenant_id):
    if tenant_id not in _config_cache:
        _config_cache[tenant_id] = db.fetch_config(tenant_id)
    return _config_cache[tenant_id]

At 50 tenants this is invisible. At 5,000 it is a memory profile.

Bucket 3 - Fragmentation and the high-water mark

A worker that handles one 400 MB report at 2 a.m. does not return to its baseline afterward. The allocator holds the arenas. RSS stays at the peak. Your resource limits end up sized for the worst task any queue ever produced - applied uniformly to every pod, forever.

Bucket 4 - The actual work

Small. It was almost always small.

3. The fix: separate responsibilities, not just code

The change was conceptually trivial and worth roughly half the cluster: stop letting one process own unrelated responsibilities. Grouping work by queue is not enough if all queues live behind one entrypoint. The boundary has to be the process, and therefore the deployment.

Step 1 - Group by dependency weight, not by domain

The instinct is to split by business domain - billing, notifications, analytics. That is the wrong axis. Split by what the code has to import:

• Light tier - stdlib plus an HTTP client. Email, SMS, WhatsApp, webhooks, cache warmers.
• Medium tier - dataframes, templating, PDF engines. Reports and exports.
• Heavy tier - model runtimes, embedding libraries, vector clients. Inference and RAG.

Three deployments, three images (or one image with three entrypoints and lazy top-level modules), three independent scaling curves.

# deployment: workers-light      replicas: 8   limit: 256Mi
worker --queues emails,sms,whatsapp,webhooks

# deployment: workers-reports    replicas: 2   limit: 900Mi
worker --queues reports,exports

# deployment: workers-ai         replicas: 1   limit: 1800Mi
worker --queues ai,embeddings

The light tier no longer imports the model runtime because the light tier no longer contains the code that imports it. That is the whole trick.

Step 2 - Make imports honest

If you cannot split the image immediately, defer the expensive import to the call site. It does not fix everything, but it stops replicas that never run the task from paying for it:

@task(queue="ai")
def classify_document(doc_id):
    import torch                # paid only by pods that run this
    ...

Step 3 - Bound every long-lived structure

from cachetools import TTLCache

_config_cache = TTLCache(maxsize=500, ttl=300)   # bounded and evicting

Anything at module scope in a worker is a permanent allocation until proven otherwise. Give it a ceiling and an expiry.

Step 4 - Recycle processes deliberately

Fragmentation is not a bug you fix; it is a property you manage. Bound the number of tasks a child process handles before it is replaced:

worker --max-tasks-per-child 200 --max-memory-per-child 400000

The process restarts, the arenas return to the OS, and RSS resets to baseline. Cheap, boring, effective - and it converts a slow upward drift into a flat line.

Step 5 - Right-size limits per tier

This step only becomes possible after step 1. When every pod is identical, every limit must accommodate the worst case. Once tiers are separated, each tier gets a limit that reflects its real profile - and the light tier, which is the bulk of the fleet, becomes tiny.

4. Before and after

Before - one deployment, all queues

Python runtime            80 MB
Framework + app          140 MB
Inference libraries      500 MB   <- unused on 5 of 6 pods
Dataframe/PDF stack      180 MB   <- unused on 5 of 6 pods
Unbounded caches         160 MB   <- grows until restart
Fragmentation headroom   120 MB
---------------------------------
Total                  1,180 MB

After - tiered deployments

That last clause is the part worth internalizing. We did not shrink the fleet. We grew it. More concurrency on the queues that actually had backlog, less memory overall, because the pods we added were the cheap ones.

Second-order effects

• p95 latency dropped ~45%. Email tasks stopped queuing behind memory-heavy report tasks on a shared pod. Isolation gave scheduling fairness for free.
• OOMKills went to zero. The heavy tier got a limit that matched reality instead of a compromise.
• Deploys got faster and safer. Shipping an email template fix no longer redeployed the inference tier.
• Autoscaling started working. Email traffic spikes now add 190 MB pods, not 1,180 MB pods - which is the difference between scaling and not scaling.

5. How to find this in your own cluster

The diagnostic sequence is short, and you can run it this afternoon.

Measure the floor, not the average

Start a worker. Send it nothing. Whatever RSS it reports at idle is what every replica pays, permanently, forever. If idle RSS is close to your limit, imports are your problem and no amount of tuning will help.

kubectl top pods -l app=workers --containers
# then compare against a pod that has been idle since boot

Attribute the imports

python -X importtime -c "import tasks" 2>&1 | sort -k2 -n -r | head -20

Import time correlates well with import weight. The top of that list is your bill.

Ask the only question that matters

For each heavy dependency: what fraction of my replicas actually execute code that needs this? If the answer is “one in six,” you have found your 50%.

Watch the slope

Plot RSS per pod over 24 hours. Flat is healthy. A staircase that never descends is an unbounded cache. A sawtooth is fragmentation plus recycling, which is fine. A steady climb into the limit is a leak.

6. The principle underneath

That sentence applies far beyond Celery and far beyond Kubernetes. It is true of any worker pool, any daemon, any serverless container with a warm start, any monolith that boots every module to serve one endpoint. The moment you co-locate a light responsibility with a heavy one inside a single process boundary, the light one inherits the heavy one’s cost - permanently, and multiplied by every replica.

Responsibility separation is usually argued on grounds of maintainability. In a queue-driven system it is a resource-efficiency argument, and it is the cheapest one available: no rewrite, no new tooling, no migration. Split the entrypoint, split the deployment, bound the caches, recycle the processes.

Half our RAM was an import statement executed by pods that had no use for it. Yours might be too.

Appendix: the checklist

• One deployment listening to every queue - split it by dependency weight.
• Idle RSS close to your memory limit - your imports are the problem, not your tasks.
• Unbounded module-level caches - give every one a maxsize and a TTL.
• No process recycling - set max-tasks-per-child and max-memory-per-child.
• Uniform limits across dissimilar workloads - size per tier, not per fleet.
• Heavy imports at module scope in a light path - move them to the call site, then move the code out entirely.
• Scaling the fleet to handle a spike on one queue - scale the tier that has the backlog instead.