Data Loss #

Docker is hugely popular for its ability to run applications consistently and in isolation. However, one of the most common — and most expensive — mistakes in using Docker is losing data. Many engineers treat containers as permanent “mini servers”, when conceptually Docker isn’t designed to store data persistently inside containers.

Data loss problems usually only surface when:

• A container is restarted or recreated.
• An image is updated to a new version.
• An automated deployment runs via CI/CD.
• The node hosting the container crashes or goes into maintenance.

At that point, months of accumulated data — database transactions, user file uploads, logs, runtime configuration — vanish in an instant. Recovery is often impossible, and the impact can be fatal: downtime, lost customers, financial losses, or even regulatory sanctions.

This article thoroughly covers the causes of data loss in Docker, why each one happens, and how to prevent them. The goal isn’t to scare you away from Docker — Docker itself is safe and powerful. What’s dangerous is the misunderstanding of how Docker handles data.

A Lost Container Means Lost Data #

To understand data loss, you must first understand how data is stored in a container — and what happens when that container dies.

flowchart TB
    subgraph NORMAL["Container Running"]
        A1[App process] --> A2[Writes data to /var/lib/mysql]
        A2 --> A3[Data enters the writable layer]
    end
    
    subgraph DEATH["Container Removed"]
        B1[docker rm container] --> B2[Writable layer deleted]
        B2 --> B3[Data destroyed]
    end
    
    NORMAL -.->|docker rm| DEATH
    
    style DEATH fill:#ff6b6b,color:#fff

When you run docker run mysql, MySQL stores data at /var/lib/mysql inside the container. These files are written to the container’s writable layer — a per-container layer living on the host, but tied to the container’s lifecycle.

When the container is deleted (docker rm), Docker removes this writable layer. All the data written there — databases, logs, runtime configuration, temporary files — is destroyed with it. No recycle bin, no undo. Gone permanently.

The Five Main Causes of Data Loss in Docker #

Data loss in Docker almost always boils down to these five patterns. Understand each one, and you’re already a step ahead of most engineers.

1. Storing Data Directly in the Container Filesystem #

This is cause #1 of data loss in Docker, and the most common. The pattern is simple: a developer runs a database or application in a container without mounting a volume, data is written to the internal filesystem, and it’s lost when the container restarts.

A very common example of the mistake:

# ANTI-PATTERN: database without a volume
docker run -d --name mysql \
  -e MYSQL_ROOT_PASSWORD=secret \
  mysql:8

The database runs, the application connects, transactions are stored. But /var/lib/mysql lives in the writable layer. When the container is restarted with a new image, or recreated, all the databases are gone.

# Container restart due to an update
docker rm -f mysql
docker run -d --name mysql \
  -e MYSQL_ROOT_PASSWORD=secret \
  mysql:8
# A new database, EMPTY.

Other equally dangerous examples:

• An application uploading files to /uploads inside the container, without a volume.
• Application logs written to /var/log/app.log inside the container, without a bind mount.
• Caches stored in /tmp inside the container — lost on container restart.
• Configuration manually edited inside the container (docker exec ... vim config.yaml) — lost when the container is recreated.

A non-negotiable principle: Important data MUST NOT be stored in the container filesystem. Always. Without exception. This isn’t a negotiable best practice — it’s how Docker’s immutability works.

2. Rebuilding Images and Recreating Containers #

In modern workflows, containers are recreated routinely — on deploy, on scale, on image update. If data lives inside the container, recreate = data loss.

flowchart LR
    A[Code change] --> B[CI/CD builds a new image]
    B --> C[Push image to registry]
    C --> D[Deploy: remove old container]
    D --> E[Run new container]
    
    D -.->|data in container| X[DATA LOST]
    E -.->|data in volume| OK[Data safe]
    
    style X fill:#ff6b6b,color:#fff
    style OK fill:#51cf66,color:#fff

Docker never moves data automatically between containers. There’s no mechanism to “extract data from the old container into the new one”. If the data is in the container, well — it’s gone.

3. Assuming docker restart Is Always Safe #

docker restart differs from docker rm + docker run. On restart:

• The container stays (the writable layer isn’t deleted).
• The process inside is stopped, then started again.
• The filesystem stays intact — written files still exist.

So docker restart looks “safe”. But it’s not a persistence strategy, and there are situations where even docker restart doesn’t save the data:

• Host crash — the Docker daemon dies suddenly, and in some cases the writable layer can corrupt.
• Disk errors — bad sectors on host storage make files unreadable.
• OOMKilled — the kernel kills the process due to memory exhaustion. Data being written at that moment can be lost.
• Force removal — docker rm -f or automatic cleanup removes the container without a graceful shutdown.
• Image updates — docker pull image:new && docker compose up -d recreates the container with a new image; old container data is destroyed.

Bottom line: docker restart is safe by accident, not by design. For important data, always use volumes.

4. Misconfigured Volumes #

Volumes are the solution, but misconfigured, they become a new problem.

Mistake #1: Forgetting to mount a volume

# ANTI-PATTERN: container without a volume
docker run -d --name app -p 3000:3000 my-app

# Should be:
docker run -d --name app -p 3000:3000 \
  -v app-data:/app/data \
  my-app

The container runs normally, no errors. But /app/data lives in the writable layer, and the data is lost when the container is recreated. The volume was never attached.

Mistake #2: Typo in the mount path

# ANTI-PATTERN: wrong path
docker run -d -v /data:/app/data my-app
# ^^^^^ mounts the /data folder on the host, NOT /app/data

The application still runs, no errors. But data is written to the wrong path. When you inspect, you’re confused because the data “disappeared” when it’s actually somewhere you didn’t expect.

Mistake #3: Mounting to the wrong path in the container

# ANTI-PATTERN: wrong target path
docker run -d -v app-data:/var/lib/mysqls mysql
#                              ^^^^^ typo

MySQL writes data to /var/lib/mysql (the default), but the volume is mounted at /var/lib/mysqls (a typo). Data stays in the writable layer; the volume stays empty.

Mistake #4: Volume deleted

docker volume rm app-data   # data lost
docker volume prune          # deletes all unused volumes

docker volume prune cleans volumes not currently used by containers. Useful for housekeeping, but dangerous if you forget that a volume contains important data.

5. Data in Bind Mounts Without Backups #

Bind mounts store data on the host filesystem. This is more “transparent” since you can browse it directly, but it also means the data is subject to every risk of the host filesystem:

• Disk failure — if the host disk dies, data in the bind mount dies too.
• Accidental deletion — rm -rf in the wrong folder can wipe production data.
• No backups — a bind mount isn’t a backup. It only moves the data location from container to host.
• Host migration — when migrating to a new host, data must be copied manually.

# ANTI-PATTERN: bind mount without backups
docker run -d \
  -v /home/app/data:/app/data \
  my-app
# Data in /home/app/data on the host, without backups

Docker isn’t a backup solution. Docker only stores data in a more controlled place. Backup remains your responsibility.

The Impact of Data Loss: Why It Matters #

Data loss isn’t a small problem. Its impact can spread across many aspects.

Downtime and Lost Revenue #

For e-commerce or fintech applications, a few minutes of downtime can mean hundreds of thousands of dollars in losses. Data loss corrupting a database can cause hours or days of downtime.

Lost Customer Trust #

Customers who lose personal data, transactions, or file uploads won’t come back. Trust built over years can be destroyed in a single incident.

Recovery Costs #

Recovering from data loss without backups almost always requires:

• Storage forensics (expensive and not always successful).
• Manual data reconstruction from logs or other sources.
• Rebuilding the database from scratch.
• Rolling back lost transactions (if possible).

These costs are often hundreds of times larger than the cost of setting up proper backups from the start.

Compliance Issues #

For regulated industries (finance, healthcare, government), data loss can result in:

• Financial sanctions.
• Mandatory reporting to regulators.
• Deep audits.
• Loss of operating licenses.

Risk Map: Storage Types and Data Loss Levels #

Not all storage mechanisms carry the same risk. The table below summarizes:

flowchart TD
    A[Data Loss Risk] --> B[High]
    A --> C[Medium]
    A --> D[Low]
    
    B --> B1[Container FS]
    B --> B2[tmpfs]
    
    C --> C1[Bind mount without backup]
    C --> C2[Volume without backup]
    
    D --> D1[External Storage]
    D --> D2[Managed Database]
    D --> D3[Object Storage]
    
    style B1 fill:#ff6b6b,color:#fff
    style B2 fill:#ff6b6b,color:#fff
    style C1 fill:#ffd43b
    style C2 fill:#ffd43b
    style D1 fill:#51cf66,color:#fff
    style D2 fill:#51cf66,color:#fff
    style D3 fill:#51cf66,color:#fff

Myths vs Facts About Data Loss #

Many misconceptions circulate. Here are the most common:

Myth 1: “Docker Volumes = Backups” #

Fact: Volumes move data from the container lifecycle to the host. But if the host crashes or the disk fails, volume data is also lost. A volume isn’t a backup. A backup is a copy of data to another location (S3, NFS, tape) independent of the host.

Myth 2: “docker restart Is Safe for Data” #

Fact: docker restart is safe by default (the writable layer isn’t deleted). But it’s not a persistence strategy. For important data, still use volumes. And remember, docker compose down removes containers (though not volumes), and redeploying with a new image also removes containers.

Myth 3: “docker commit Can Save Data” #

Fact: docker commit does freeze the writable layer into a new image. But this is an anti-pattern because:

• The image becomes bloated with runtime data.
• Every commit creates a new layer; the image balloons.
• Sensitive data (passwords, secrets) gets committed into the image.
• Image distribution becomes insecure.

Use backup tools (pg_dump, mysqldump, tar) to save data, not docker commit.

Myth 4: “If the Container Is in Kubernetes, It’s Safe” #

Fact: Kubernetes doesn’t automatically back up data. Pods can be evicted, deployments rolled back, and PVCs (Persistent Volume Claims) can be deleted. Kubernetes makes deployment easier, but doesn’t replace a backup strategy.

Myth 5: “Files in the Writable Layer Are Safe While the Container Lives” #

Fact: The writable layer can corrupt because of:

• A full disk.
• OOMKill while a process is writing.
• A sudden host crash.
• Docker daemon bugs (rare, but they exist).

Files “in the middle of being written” are highly vulnerable to corruption. For important data, the database transaction log must be flushed to disk periodically (wal, fsync, etc.).

Best Practices for Preventing Data Loss #

1. Use Docker Volumes for Important Data #

services:
  db:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

This is the first and most important step. Always mount volumes for databases, uploads, and other stateful data.

2. Separate State and Stateless #

Containers should be stateless. Data is stored in:

• Volumes for data acceptable to host locally.
• Database services (RDS, Cloud SQL) for structured data.
• Object storage (S3, GCS) for large files.
• Cache services (Redis, Memcached) for caches.

flowchart TB
    APP[App Container<br/>stateless] --> DB[(Database Service)]
    APP --> OBJ[Object Storage]
    APP --> CACHE[Cache Service]
    APP --> VOL[Docker Volume<br/>for local data]
    
    style APP fill:#a8d8a8
    style DB fill:#ffd8a8
    style OBJ fill:#ffd8a8
    style CACHE fill:#ffd8a8
    style VOL fill:#ffd8a8

3. Use External Storage for Critical Data #

For data that must never be lost, use managed services:

• Database: AWS RDS, Google Cloud SQL, Azure Database.
• File uploads: AWS S3, Google Cloud Storage, MinIO.
• Cache: AWS ElastiCache, Redis Labs, Upstash.
• Search: AWS OpenSearch, Elastic Cloud.

Managed services provide:

• Automatic backups.
• Point-in-time recovery.
• Replication across multiple zones.
• Monitoring and alerting.
• Patch management.

4. Don’t Store Data in Images #

# ANTI-PATTERN
FROM ubuntu
COPY ./database.dump /tmp/db.dump
COPY ./user-uploads/ /uploads/

An image is an immutable artifact. Runtime data must not live in images. Data must:

• Be read from a volume at runtime.
• Be restored from a backup on first run.
• Be seeded from migrations at container start.

5. Always Have a Backup Strategy #

Backups must be:

• Automatic — not dependent on a human remembering.
• Regular — daily for active data, weekly for archives.
• Off-site — copies in a different geographic location.
• Tested — restores must be tested regularly, not assumed to work.

# Example daily cron backup
0 2 * * * docker run --rm \
  -v pgdata:/source:ro \
  -v /backup:/backup \
  postgres:16 \
  pg_dump -U postgres mydb > /backup/db-$(date +%F).sql

6. Implement Health Checks #

Health checks help the orchestrator detect hung containers and restart them. This reduces the risk of data corruption from zombie processes.

services:
  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 3

7. Monitor Storage Usage #

Volumes can fill up unnoticed. Monitor with:

# Check storage usage
docker system df

# Detail per volume
docker system df -v

# Automatic cleanup (careful, can delete data!)
docker volume prune

Alert in your monitoring system when volume usage exceeds 80%.

8. Use the Right Images #

Choose mature, battle-tested database images. Unofficial or modified images may not handle persistence correctly.

# Use official images
image: postgres:16        # ✅ official
image: mysql:8.0          # ✅ official
image: bitnami/postgresql # ✅ trusted third party
image: some-random-user/custom-pg  # ❌ not guaranteed

9. Document the Storage Layout #

Newly onboarded team members must immediately understand:

• Which volume holds what.
• The host path for each volume.
• The backup procedure.
• The restore procedure.
• The backup SLA (how far back data can be recovered).

Case Studies: Real-World Data Loss #

Case 1: E-commerce Startup Loses Its Database on Deploy #

Situation: A startup just launched; MySQL runs in a container without a volume. A developer pushes new code, CI/CD redeploys, the container is recreated.

Impact: Empty database; 3 months of customer transaction data gone. No backups.

Cause: No volume mount. CI/CD had no backup step.

Solution: Mount the mysql-data volume. Add daily backups to S3. Set up monitoring alerts for volume usage.

Lesson: Backups must be part of the pipeline, not an afterthought.

Case 2: File Upload Application Loses Files on Update #

Situation: A web application accepts PDF uploads from users. Files are stored in /app/uploads inside the container.

Impact: On deploying a new image, all uploaded files are lost. Customers complain; the support team is overwhelmed.

Cause: Data in the container, no volume.

Solution: Mount the uploads-data volume to /app/uploads. For larger scale, migrate to S3.

Lesson: User file uploads almost always need persistent storage. The default should be a volume, not the container filesystem.

Case 3: Bind Mount Wrong Path, Developer Unaware #

Situation: A developer mounts -v ./data:/app/data for development. The ./data path resolves incorrectly, and the container stores data in an unexpected path.

Impact: When the developer checks “why is the data empty”, it turns out the data was written elsewhere. The bug is discovered too late.

Cause: Paths are relative to the working directory, not the docker-compose.yml file (in Compose, paths are relative to the compose file; in docker run, relative to the shell).

Solution: Use absolute paths or explicit relative paths. Add health checks and log volume paths in the startup script.

Lesson: Mount paths must be verified. Better over-explicit than under-explicit.

Case 4: Full Disk, Database Crash #

Situation: A database volume fills up because application logs aren’t rotated.

Impact: Database crashes, the application errors. Recovery requires a container restart + cleanup.

Cause: No log rotation, no disk usage monitoring.

Solution: Set up log rotation (logrotate, or the Docker json-file log driver with max-size). Monitor disk usage and alert at 80%.

Lesson: Storage without monitoring is a time bomb.

Anti-Patterns to Avoid #

1. Backing Up Images Instead of Data #

# ANTI-PATTERN
docker commit running-container my-backup:latest
docker save my-backup:latest > backup.tar

This isn’t a backup. An image containing runtime data:

• Contains sensitive data.
• Is inconsistent (the database may be mid-transaction).
• Is large and slow.
• Can’t be selectively restored.

Use pg_dump, mysqldump, or tar to back up data, not docker commit.

2. Never Testing Restores #

A backup that’s never restored = a backup that doesn’t exist. Many teams back up routinely but never actually perform a restore. When disaster strikes, they’re shocked to find the backup files corrupt or the restore process failing.

Best practice: Restore backups to a test environment regularly. Verify data integrity. Train the team on the restore procedure.

3. Manual Backups Without Automation #

# ANTI-PATTERN
# "Oh right, forgot to back up this month..."

Manual backups will definitely be missed. Use cron, scheduled CI jobs, or a backup service for automation.

4. Single Point of Failure #

A backup on the same host as the data = no backup. If the host crashes, backup and data are lost together. Always copy backups to another location.

flowchart LR
    A[Host] -->|backup| B[Same Disk]
    A -->|backup| C[Different Host]
    A -->|backup| D[Cloud Storage]
    
    B -.->|disk fail| X[Backup LOST]
    C -.->|host fail| OK[Backup SAFE]
    D -.->|region fail| OK2[Backup SAFE]
    
    style B fill:#ff6b6b,color:#fff
    style X fill:#ff6b6b,color:#fff

5. Not Documenting the Restore Procedure #

During a disaster, you don’t want to be opening notes to find the restore steps. The procedure must:

• Be recorded in RUNBOOK.md.
• Be runnable with a single script.
• Have been tested.
• Be known to every on-call engineer.

Concrete Steps: Auditing Your Docker Storage #

Before this article ends, run a simple audit of your current Docker setup:

DATA LOSS AUDIT CHECKLIST:

CONTAINERS:
  □ Are there containers storing data in their filesystem?
  □ Are there bind mounts containing important data?
  □ Are there volumes whose contents are unclear?

BACKUPS:
  □ Are volumes backed up regularly?
  □ Are backups stored in a different location?
  □ Has a restore ever been tested?

MONITORING:
  □ Is disk usage monitored?
  □ Is there an alert for full volumes?
  □ Is there an alert for frequently restarting containers?

DOCUMENTATION:
  □ Is the storage layout documented?
  □ Is the restore procedure in the runbook?
  □ Does the team know how to restore during a disaster?

If any item isn’t checked, you have a PR to do this week.

Summary #

• Data loss in Docker isn’t a bug, but a conceptual misunderstanding. Containers are ephemeral, and storing data in the container filesystem is the most common anti-pattern.
• The five main causes: storing data in containers, recreating containers on deploy, assuming docker restart is always safe, misconfigured volumes (forgotten mounts, path typos), and bind mounts without backups.
• Volume ≠ backup. Volumes move data to the host but don’t replace backups. For important data, always back up to another location (S3, NFS).
• Risk levels: Container FS and tmpfs = very high risk. Volumes and bind mounts without backups = medium risk. External storage and managed services = low risk.
• The core principle: Containers = stateless, data = stateful outside the container. Databases must use volumes. File uploads must use volumes or object storage. Images are immutable, not a place for runtime data.
• Best practices: use volumes for important data, separate state from stateless, use external storage for critical data, don’t store data in images, always have a backup strategy (automatic, regular, off-site, tested).
• Myths to correct: volumes aren’t backups, docker restart isn’t a persistence strategy, docker commit isn’t a backup method, Kubernetes doesn’t auto-backup, the writable layer isn’t always safe.
• Audit storage regularly: check containers storing data without volumes, verify backups, test restores, monitor disk usage.

← Previous: Persistent Data   Next: Volume →