Ephemeral Container #

In the world of containerization, Docker is often associated with containers running main applications — web servers, APIs, workers — that live for days, weeks, or even months. But there’s another equally important category of containers: ephemeral containers, created for one specific task and then immediately gone.

Ephemeral containers are disposable workers. They’re created, run a single command or workflow, then deleted. There’s no state to maintain, no process to restart, and no data that must survive their death — because there shouldn’t be any.

Understanding the ephemeral concept is a prerequisite for this entire storage section. Why? Because almost every Docker storage decision — why volumes are needed, why data must be separated, why backups focus on data — starts from one simple principle: containers are ephemeral, data is persistent.

This article covers what ephemeral containers are, their characteristics, when to use them, and how to distinguish them from long-running containers. By the end, you’ll have a clear thinking framework for deciding whether a workload should run as an ephemeral or long-running container.

What Is an Ephemeral Container? #

By definition, an ephemeral container is one that:

• Is created for a temporary purpose — usually one specific task, not an application that stays on.
• Stores no important state — any data it produces must be written to a volume, or is intentionally lost.
• Needs no restart — when it dies, let it die. The task is done or it had to be force-stopped.
• Is automatically deleted when finished — the --rm flag ensures containers don’t pile up.

docker run --rm alpine echo "Hello, ephemeral world"

The command above is the simplest example. A container is created, runs echo, finishes, and is immediately deleted by the Docker daemon because of the --rm flag. Its total lifetime may be less than a second.

flowchart LR
    A[Create container] --> B[Run the task]
    B --> C{Task done?}
    C -- Yes --> D[Container auto-remove]
    C -- No --> B
    D --> E[Cleanup<br/>nothing left behind]

Distinguishing Characteristics #

Ephemeral containers have traits you can recognize from their design:

• Short lifespan — seconds, minutes, or until the task finishes. Never days.
• Stateless — no data to maintain between invocations.
• Purpose-specific — one container = one purpose. Not multi-purpose.
• Disposable — allowed to die, allowed to be deleted, no recovery.
• Idempotent — re-running the same task must produce the same result (or may differ without consequence).

The most striking difference from long-running containers is the expectation around death. For long-running containers, death is an anomaly — you must restart, debug, alert. For ephemeral containers, death is the goal — when the container dies, it means the task is done.

Ephemeral vs Long-Running: the Fundamental Difference #

Knowing when to use each is an architectural decision that’s often overlooked. Many engineers run every workload as long-running by default, even though many would actually be better suited as ephemeral.

flowchart TB
    subgraph LONG["Long-Running Container"]
        L1[Always on] --> L2[Serving traffic]
        L2 --> L3[Restart policy: always]
        L3 --> L4[Stateful OK]
    end
    
    subgraph EPH["Ephemeral Container"]
        E1[Created on-demand] --> E2[Run one task]
        E2 --> E3[Task done → exit]
        E3 --> E4[Auto-remove with --rm]
    end

A way to remember: a long-running container is a “24-hour restaurant” — customers arrive at any time, it must be ready. An ephemeral container is a “one-time courier” — deliver one task, done.

Ephemeral Container Anatomy #

To understand how ephemeral works, dissect what happens when you run a disposable container.

Typical Lifecycle #

stateDiagram-v2
    [*] --> Created: docker run --rm image cmd
    Created --> Running: container start
    Running --> Exited: command finished
    Exited --> Removed: auto-remove (--rm)
    Removed --> [*]

There’s no Restarting state, no transition to Paused, no Restart after exit. The lifecycle is linear: born, runs, finishes, gone.

What Happens Behind the Scenes #

When docker run --rm image command executes:

1. Docker pulls the image from cache or the registry (if not already present).
2. Docker creates a new container with an empty writable layer.
3. The container starts and runs command.
4. command finishes, the container exits.
5. Because of the --rm flag, Docker immediately deletes the container and its writable layer.

In total: the upperdir is created, briefly used, then deleted. No side effects on the host, no accumulating containers, no leaked ports.

Resources After the Container Dies #

Because the upperdir is deleted along with it, all runtime changes disappear:

• Files written to /tmp — gone.
• Caches generated in /var/cache — gone.
• Logs written to stdout/stderr — still capturable (see best practices).
• Files in volumes or bind mounts — still there (because they’re stored on the host, not in the upperdir).

This explains why important data must be written to volumes, not to the container filesystem. An ephemeral container has no concept of “data must survive” — any data that needs to persist must be externalized.

Ephemeral Container Use Cases #

Ephemeral containers are a great fit for workloads that are batch-style, run-once, or stateless. Here are the most common use cases.

1. Running One-Off Commands #

The simplest and most common:

docker run --rm node:20 node -e "console.log('Hello from Node')"

You don’t need to install Node.js on the host. No environment setup. No pollution of the system. The container lives for a fraction of a second, then disappears.

# Check a tool's version without installing it
docker run --rm python:3.12 python --version
docker run --rm golang:1.22 go version
docker run --rm postgres:16 psql --version

This pattern is very useful in CI/CD pipelines and on developer workstations that want to stay clean.

2. Database Migrations #

When deploying a new version of an application, you often need to run migrations. A migration is a run-once task, not a process that must stay alive:

docker run --rm \
  --network app-network \
  -e DATABASE_URL=postgres://... \
  myapp-image:latest \
  ./migrate up

The container is created, the migration runs, it finishes, the container is deleted. The database gets upgraded without logging into the database container or running migrations manually.

flowchart LR
    A[CI/CD triggers deploy] --> B[Pull new image]
    B --> C[Run migration container]
    C --> D{Migration successful?}
    D -- Yes --> E[Deploy new app container]
    D -- No --> F[Rollback]

3. Debugging a Running Container #

When a production application has issues and you don’t want to install debug tools in the production container, use an ephemeral container sharing the same network or volumes:

# Enter the production container's network
docker run --rm -it \
  --network container:myapp-container \
  alpine sh

# Access the production database volume for inspection
docker run --rm -it \
  --volumes-from mysql-container \
  alpine sh

This debug container:

• Shares the network namespace with the target container (you can curl localhost to the same service).
• Shares specific volumes (you can read data files directly).
• Doesn’t modify the target container.
• When debugging is done, the debug container is deleted, leaving no trace.

Advanced pattern: since Kubernetes 1.16, there’s an official ephemeral debug container feature doing something similar in clusters. The Docker CLI follows the same pattern with docker run --rm --network container:....

4. CI/CD Pipelines #

Almost every modern CI pipeline runs in ephemeral containers. GitHub Actions, GitLab CI, CircleCI — all run each job in a fresh container deleted when the job finishes.

# GitHub Actions example
jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: node:20
    steps:
      - uses: actions/checkout@v4
      - run: npm install
      - run: npm test
    # the container is automatically deleted after the job finishes

The advantages of this pattern:

• Clean environment per job — no state leaking between runs.
• Reproducible — pipelines always start from the same image state.
• Parallel-ready — many jobs run simultaneously without resource conflicts.
• No cleanup code — the CI provider handles disposal.

5. Batch Processing and ETL #

Jobs that read data, process it, and write output — without state between runs — are perfect ephemeral candidates:

# Daily backup
docker run --rm \
  -v postgres-data:/source:ro \
  -v /backups:/backup \
  postgres:16 \
  pg_dump -U postgres mydb > /backup/mydb-$(date +%F).sql

# Image processing
docker run --rm \
  -v /photos:/input:ro \
  -v /output:/output \
  imagemagick \
  mogrify -resize 50% /input/*.jpg

# Data transform
docker run --rm \
  -v /raw:/data:ro \
  python:3.12 \
  python /scripts/transform.py

The container finishes, the output is stored on the host (via volumes), the container disappears. Clean and efficient.

6. Isolated Testing #

Running integration tests that need a database, message broker, or other services without polluting the host:

# Test with a temporary database
docker run --rm -d --name test-db -e POSTGRES_PASSWORD=test postgres:16
docker run --rm --network container:test-db myapp:test ./run-integration-tests
docker rm -f test-db

Or more elegantly with a Docker Compose test profile:

services:
  db:
    image: postgres:16
    profiles: ["test"]
  app-test:
    build: .
    profiles: ["test"]
    depends_on: [db]
    command: npm test

docker compose --profile test run --rm app-test

Ephemeral Containers in Docker Compose #

Docker Compose has a run sub-command specifically for running one-off containers within a stack:

docker compose run --rm app npm test

docker compose run characteristics:

• Doesn’t join the up lifecycle — it doesn’t add or recreate services.
• Isn’t restarted — when it exits, it’s done.
• Done → gone — with the --rm flag, the container is deleted automatically.
• Can run a different command from the command in the compose file.

This is useful for running migrations, seeding data, or administrative commands in an already-running stack:

# Run a migration in the production stack
docker compose run --rm app ./migrate up

# Run a seed
docker compose run --rm app ./seed

# Back up the database directly
docker compose run --rm db pg_dump -U postgres mydb > backup.sql

flowchart LR
    UP[docker compose up<br/>long-running services] --> APP[App container]
    UP --> DB[DB container]
    UP --> REDIS[Redis container]
    
    RUN[docker compose run --rm<br/>ephemeral task] --> MIG[Migration]
    RUN --> SEED[Seed data]
    RUN --> BACKUP[Backup]

The important difference between docker compose up and docker compose run:

Important Flags for Ephemeral Containers #

Several CLI flags you must know when working with ephemeral containers.

--rm — Auto Remove #

docker run --rm alpine echo "bye"

The container is deleted automatically after exit. Mandatory for ephemeral — without --rm, the container stays around (in exited state) and piles up in docker ps -a.

--name — Identifier (Optional) #

docker run --rm --name temp-tool alpine sh -c "echo 'task'"

Useful for debugging, but should be avoided in parallel workflows (name collisions).

-e / --env — Environment Variables #

docker run --rm -e DATABASE_URL=postgres://... myapp ./task

For passing configuration to a one-off container.

--network container:NAME — Share a Network #

docker run --rm --network container:app myapp curl http://localhost:8080

Useful for debugging — the new container shares the target container’s network namespace.

--volumes-from NAME — Share Volumes #

docker run --rm --volumes-from mysql-data alpine ls /var/lib/mysql

Reads another container’s volume contents without creating any of its own.

-i -t — Interactive Terminal #

docker run --rm -it alpine sh

For entering an interactive shell. On ephemeral debug containers, this is a very common combination.

--restart=no — Default for Ephemeral #

The default --restart is no — the container won’t be restarted after exit. This fits ephemeral, but don’t use it for long-running services.

Ephemeral Containers and --rm — What Actually Happens #

To truly understand ephemeral, it’s important to know what --rm does and doesn’t do.

What --rm Does #

• Removes the container from the docker ps -a list after exit.
• Cleans up the container’s writable layer.
• Releases the ports and resources used by the container.

What --rm Does NOT Do #

• Doesn’t delete the image — the image stays, ready for reuse.
• Doesn’t delete volumes — mounted volumes remain (even after the container is gone).
• Doesn’t delete networks — created networks (via --network) remain unless you delete them manually.
• Doesn’t delete bind mounts — host paths remain; the container just stops accessing them.

This explains why data in volumes stays safe even when an ephemeral container dies. Volumes are stored on the host, not in the container’s upperdir.

flowchart TB
    IMG[Image<br/>stays] -.-> C
    VOL[Volume<br/>stays] -.-> C
    NET[Network<br/>stays] -.-> C
    
    subgraph LIFECYCLE["Container Lifecycle"]
        C[Container<br/>upperdir]
    end
    
    C -- exit --> DEL[Deleted by --rm]
    
    style DEL fill:#ff6b6b
    style IMG fill:#51cf66
    style VOL fill:#51cf66
    style NET fill:#51cf66

Be careful: if you forget --rm and run many containers without deleting them, they’ll pile up as “exited containers”. They don’t consume CPU/RAM, but they fill the docker ps -a output and can confuse audits. Clean them up with docker container prune.

The “Cattle, Not Pets” Philosophy #

Ephemeral containers are the concrete manifestation of a famous infrastructure principle:

Pets are given names like “puss” and “boff”. Cattle are given numbers like “3458” and “8877”. — Bill Baker, Microsoft

Pets — cared for, patched, lovingly named. When sick, treated. When dead, buried with honors. Example: traditional servers with hostnames like web-server-01 that must be maintained for years.

Cattle — given numbers, not names. When sick or unproductive, slaughtered and replaced. Example: containers with random names that get restarted or recreated without ceremony.

Ephemeral containers are cattle. You must not get attached to them. Don’t name them, don’t store state inside them, don’t try to “repair” a broken container — just delete it and run a new one.

flowchart TB
    subgraph PETS["PETS — Traditional"]
        P1[Server web-01] --> P2[Patch OS]
        P2 --> P3[Update app]
        P3 --> P4[Monitor]
        P4 --> P5{Broken?}
        P5 -- Yes --> P6[Debug & repair]
    end
    
    subgraph CATTLE["CATTLE — Container"]
        C1[Random container] --> C2{Running?}
        C2 -- No --> C3[Delete]
        C2 -- Yes --> C4[Work until done]
        C4 --> C5[Auto-remove]
        C3 --> C6[Create new container]
        C6 --> C1
    end

This principle isn’t just philosophy — it has real technical consequences:

• Containers can be replaced without data loss (because data lives in volumes).
• Containers can be scaled by running new instances (orchestrators like Kubernetes or Swarm do this automatically).
• Containers can be updated with new images without ceremony (rolling updates).
• Containers can be rolled back by going back to an old image (if the new one has issues).

Without the cattle pattern, containers just become “lighter VMs” — you’re still stuck in a pet mentality. With the cattle pattern, containers become truly disposable and scalable compute units.

When NOT to Use Ephemeral Containers #

Ephemeral isn’t a silver bullet. There are situations where long-running is more appropriate:

• Web servers, APIs, or services that must accept requests at any time — if the container dies, the next request fails. These need restart policies and high uptime.
• Workers processing jobs continuously — queue consumers must stay alive, pulling jobs from the broker.
• Database servers — even though they don’t accept user requests, DBs must stay alive to serve queries.
• Stateful services with persistent connections — game servers, SSH bastions, or WebSocket applications whose connections must be maintained.
• Services with slow startup — if a container takes 5 minutes to become ready, running it as ephemeral for every incoming request isn’t practical.

For workloads like these, use long-running containers with:

• A restart policy of always or unless-stopped.
• Persistent volumes for data.
• Health checks for auto-restart when hung.
• An orchestrator (Swarm, Kubernetes) for automatic scaling and healing.

Ephemeral Container Best Practices #

1. Always Use --rm for One-Off Containers #

# CORRECT
docker run --rm alpine echo "hello"

# ANTI-PATTERN: containers pile up as exited
docker run alpine echo "hello"

# Clean up exited containers you forgot to remove
docker container prune

2. Make Sure the Task Is Idempotent #

Ephemeral containers can be re-run. The task must produce the same output (or at least not conflict) no matter how many times it runs.

# Idempotent: create a table, re-running is still safe
docker run --rm myapp ./migrate up

# Not idempotent: append without checking
docker run --rm myapp ./append-data  # ← can duplicate when re-run

3. Store State Outside the Container #

All state must be written to volumes or external systems:

# CORRECT: write to a volume
docker run --rm -v /backup:/output postgres:16 pg_dump ... > /output/db.sql

# ANTI-PATTERN: write to the container filesystem
docker run --rm postgres:16 pg_dump ... > /tmp/db.sql  # lost when the container dies

4. Capture Output via stdout/stderr #

For logging, write to stdout/stderr and let Docker or a log driver handle it:

# Output can be captured to a host file
docker run --rm myapp ./task > /logs/task.log 2>&1

5. Use Small Images for Fast Startup #

Ephemeral containers that run frequently (CI/CD, batch) must start as fast as possible. Use small base images and multi-stage builds.

# Slow: 800 MB image with Ubuntu + toolchain
docker run --rm myapp:dev ./task  # takes 5 seconds to start

# Fast: 15 MB alpine or distroless image
docker run --rm myapp:prod ./task  # takes 0.3 seconds to start

6. Set Resource Limits if Needed #

Ephemeral containers running on shared hosts must have resource limits so they don’t disturb other workloads:

docker run --rm \
  --memory=256m \
  --cpus=0.5 \
  myapp ./heavy-task

7. Handle Exit Codes Properly #

An ephemeral container’s exit code can be captured by the caller (CI/CD, scheduler):

docker run --rm myapp ./task
EXIT_CODE=$?

if [ $EXIT_CODE -ne 0 ]; then
  echo "Task failed"
  exit $EXIT_CODE
fi

Exit code 0 = success, 1-125 = application error, 126 = command couldn’t execute, 127 = command not found, 137 = SIGKILL (OOM or timeout).

Ephemeral Containers in Modern Orchestrators #

The ephemeral pattern isn’t just for manual docker run. Modern orchestrators like Kubernetes and Docker Swarm make ephemeral the default pattern.

Kubernetes — Jobs and CronJobs #

Kubernetes has a Job resource for ephemeral workloads:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: myapp:1.2
        command: ["./migrate", "up"]
      restartPolicy: Never
  backoffLimit: 3

A Job runs until completion. If it fails, Kubernetes retries up to backoffLimit. Once finished, the Job is considered done.

For periodic workloads, there’s CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: backup-tool:1.0
            command: ["./backup.sh"]
          restartPolicy: Never

This runs an ephemeral container every night at 2 AM. The container lives only as long as the backup runs, then dies.

Docker Swarm — Global/Replication Mode #

Swarm can run ephemeral containers globally on all nodes for maintenance tasks:

docker service create \
  --mode global \
  --restart-condition none \
  alpine sh -c "do-maintenance"

The container runs on all nodes, dies when done, and isn’t restarted.

Anti-Patterns to Avoid #

1. Relying on the Container Filesystem for Important Data #

# ANTI-PATTERN: data is lost when the container dies
docker run --rm myapp ./process  # writes output to /tmp or /app/output

Always write to volumes or external storage.

2. Assuming the Container Will Always Be There #

Don’t write logic that depends on a specific container. Always assume a container can die at any moment.

3. Storing Manual Configuration Inside the Container #

Configuration must be passed via env vars, files in volumes, or baked into the image reproducibly.

4. Not Using --rm #

Piled-up exited containers fill docker ps -a, confuse audits, and sometimes make engineers mistakenly think a container is still running.

# Audit exited containers
docker ps -a --filter "status=exited"

# Clean up
docker container prune

5. Forcing Long-Running onto Batch Tasks #

Don’t use the always restart policy for tasks that must finish. Use long-running for services, ephemeral for batch.

Relationship with Other Concepts #

Ephemeral containers are part of a larger ephemeral infrastructure:

The core of all these concepts is the same: don’t repair what’s broken, rebuild from scratch. This produces systems that are:

• More consistent — no drift because everything always starts from a definition.
• Easier to debug — problems are reproducible and can be recreated in a controlled environment.
• Safer — no sensitive state left behind.

Summary #

• Ephemeral containers are disposable containers: created, run one task, finished, deleted (--rm). The opposite of long-running containers that stay on continuously.
• Main characteristics: short lifespan, stateless, purpose-specific, disposable, and idempotent. Death isn’t an incident — death is the goal.
• Main use cases: one-off commands, database migrations, debugging, CI/CD pipelines, batch processing, ETL, and isolated tests. Almost any batch-style, run-once workflow fits ephemeral.
• Difference from long-running: long-running has restart policies, low death tolerance, and is usually stateful (with volumes). Ephemeral has no restart policy, death is the goal, and statelessness is mandatory.
• Docker Compose supports ephemeral via docker compose run --rm <service> <command>. This differs from docker compose up, which runs long-running services.
• Mandatory flags: --rm for auto-remove, -it for interactive, --network container: for debugging. Without --rm, exited containers pile up.
• The “cattle, not pets” philosophy is the soul of ephemeral containers. Containers aren’t named, cared for, or loved — when broken, delete and create a new one.
• Contraindications: web servers, APIs, queue workers, database servers, and any service that must accept requests at any time. These fit long-running better.
• Ephemeral data still must be written to volumes or external storage. The container’s writable layer disappears, and data in /tmp goes with it. This topic is covered in depth in the next article.

← Previous: File System & Layer   Next: Persistent Data →