Development Workflows with Compose

A team's new-hire onboarding README used to say: "install Postgres 16, Redis 7, Node 20, Python 3.11, run migrations, seed the database, start five services in five terminals." It took a full day. Now it says: "git clone, docker compose up, done." The actual README content is 3 lines. The compose file does all the work — every service, every version, every config — and stays in sync with production because the same Dockerfiles build both.

The best reason to use Compose is not production orchestration. It is development ergonomics. When a compose file is well-built, local dev feels like a scripted demo: one command spins up the world, code changes hot-reload, dependencies upgrade in one YAML change, and the whole team is on the same setup on day one. This lesson is the patterns that get you there: hot reload, source mounts, dev overrides, seeding, and avoiding the Docker Desktop traps on macOS/Windows.

---

The Compose-for-Dev Pattern

The core idea: your compose.yaml is the shared, committed, production-leaning definition. A compose.override.yaml (auto-merged on docker compose up) adds the dev-specific bits.

# compose.yaml  (committed, used in CI and prod-like runs)
services:
  api:
    image: ghcr.io/myorg/api:${VERSION:-latest}
    environment:
      NODE_ENV: production
      DATABASE_URL: postgres://user:pass@db:5432/app

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

# compose.override.yaml  (often gitignored, or committed for dev-only files)
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev
    command: npm run dev
    environment:
      NODE_ENV: development
      LOG_LEVEL: debug
    volumes:
      - ./src:/app/src
      - ./package.json:/app/package.json
      - ./package-lock.json:/app/package-lock.json
      - node_modules:/app/node_modules
    ports:
      - "127.0.0.1:8080:8080"

  db:
    ports:
      - "127.0.0.1:5432:5432"       # expose db for local psql

volumes:
  node_modules:

docker compose up picks up both files automatically. compose.override.yaml only affects dev; the committed compose.yaml stays clean for CI and prod-like runs.

---

Hot Reload: Source Mount Patterns

The basic pattern — bind-mount source into the container so edits are reflected immediately:

services:
  api:
    command: npm run dev           # dev server that watches files
    volumes:
      - ./src:/app/src             # source tree — edits hot-reload
      - ./package.json:/app/package.json:ro
      - /app/node_modules          # anonymous volume "protects" the image's deps

The last line — - /app/node_modules — is the critical trick. Without it, docker compose up bind-mounts your project dir, including a host node_modules that may not exist (or may be for a different platform), over the image's /app/node_modules. Suddenly your container has no deps.

The anonymous volume lives inside Docker's volume area and keeps the image's /app/node_modules intact even though /app/src is a host mount. Same pattern works for Python's .venv, Ruby's vendor/bundle, Go's cached modules:

# Python
volumes:
  - ./src:/app/src
  - /app/.venv

# Go
volumes:
  - .:/src
  - gocache:/root/.cache/go-build

Hot reload needs a watching process

The mount only gets you a fresh file in the container. Something inside the container has to notice and restart. Options:

• Language-specific dev servers — Next.js dev, Vite, Flask debug mode, Django dev server, nodemon, air for Go, watchexec for anything.
• Filesystem watchers — works on Linux natively; on Docker Desktop (macOS/Windows), inotify does not always fire through the bind mount. You may need polling mode (CHOKIDAR_USEPOLLING=1 for webpack, --poll for Jest, etc.).

services:
  api:
    environment:
      CHOKIDAR_USEPOLLING: "true"    # webpack/jest/next polling for Docker Desktop
      WATCHPACK_POLLING: "true"      # webpack-specific

Polling costs more CPU but is reliable. On native Linux Docker, inotify works fine through bind mounts.

---

Compose Watch (v2.22+): Better Than Bind Mounts on Slow FSes

Recent Compose has a native watch feature that sidesteps bind-mount performance problems:

services:
  api:
    build: .
    command: npm start
    develop:
      watch:
        - action: sync             # file changes → copied into container
          path: ./src
          target: /app/src
        - action: rebuild           # requires rebuild
          path: ./package.json
        - action: sync+restart      # sync file, then restart service
          path: ./config/*.yaml
          target: /app/config

Run:

docker compose up --watch
# or
docker compose watch

This avoids bind-mount slowness on Docker Desktop (file sync is host → container, not a live mount). The trade-off: it is Compose-specific; the running container is not looking at your real filesystem.

For projects with big node_modules / .venv trees on macOS, watch often outperforms bind mounts by 5-10×.

---

Seeding the Database

Two common patterns for initial data:

Init scripts in the image

Many database images (Postgres, MySQL, Mongo) run /docker-entrypoint-initdb.d/*.sql and *.sh on first container start:

services:
  db:
    image: postgres:16-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./db/init:/docker-entrypoint-initdb.d:ro

-- db/init/01-schema.sql
CREATE TABLE users (id SERIAL PRIMARY KEY, email TEXT UNIQUE);
-- db/init/02-seed.sql
INSERT INTO users (email) VALUES ('test@example.com');

These run only on an empty data dir — subsequent starts see data and skip init. Perfect for dev / CI.

Dedicated seeder service

For complex seed data, a one-shot service:

services:
  db:
    # ... postgres ...
    healthcheck: { ... }

  seeder:
    image: myorg/api:dev
    depends_on:
      db: { condition: service_healthy }
    command: npm run seed
    restart: "no"
    profiles: [seed]        # only runs when explicitly activated

  api:
    depends_on:
      db: { condition: service_healthy }
    # ...

docker compose --profile seed up -d
# starts db, waits for healthy, runs seeder, then api

Profiles keep the seeder out of the default startup once the DB is already seeded.

---

Running One-Off Commands

docker compose exec — in a running container

docker compose exec api sh           # shell in the running api
docker compose exec api npm test     # run tests in the running api
docker compose exec db psql -U user app  # psql into the db

docker compose run — new container, same image

docker compose run --rm api npm test         # new container, runs tests, removed
docker compose run --rm api bash             # new container, shell
docker compose run --rm api npm run migrate  # migrations

run is for tasks you want to do before/without the main service running. exec is for tasks in the already-running container. run --rm is the equivalent of "bundle exec" / "npx" for your stack.

---

Secrets in Dev vs Prod

Dev compose files routinely contain plaintext passwords ("pass", "dev"). That is fine — the file never hits production. The anti-pattern is committing real secrets:

The right pattern

# compose.yaml — committed, no secrets
services:
  api:
    env_file:
      - .env                    # auto-loaded, committed as .env.example
      - .env.local              # gitignored, each developer has their own
    environment:
      DATABASE_URL: postgres://user:${DB_PASSWORD}@db:5432/app

# .env.example (committed)
DB_PASSWORD=changeme
API_KEY=dev-api-key

# .env.local (gitignored)
DB_PASSWORD=real-dev-password
API_KEY=real-dev-api-key

.env.example is the manifest of what devs need to set. .env.local has their actual values. Production uses a secret manager, not files.

Compose's native secrets

services:
  api:
    secrets:
      - db_password
    environment:
      DATABASE_URL_FILE: /run/secrets/db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

Secrets are mounted as files under /run/secrets/<name>. Your app reads the file, not an env var — fewer accidental leaks. Most supported in Swarm mode; compose up supports file-based secrets as shown.

---

Debugging Inside the Stack

# Which services are up?
docker compose ps

# What are they logging?
docker compose logs -f api

# Why did this one fail?
docker compose logs --tail=50 seeder

# Attach a debugger
# In compose.override.yaml:
#   api:
#     ports:
#       - "127.0.0.1:9229:9229"          # Node.js inspector
#     command: node --inspect=0.0.0.0:9229 server.js
# Then connect your IDE's debugger to localhost:9229

# Top / stats
docker compose top
docker compose stats

Blow it all away and start fresh

docker compose down -v --rmi local
# Stops containers, removes networks, removes NAMED VOLUMES (-v), removes locally built images (--rmi local)

docker compose build --no-cache
docker compose up -d

This is the "nuke it and start over" sequence when something is stuck. Be careful: -v deletes volumes unconditionally.

---

Docker Desktop Gotchas (macOS / Windows)

Most dev-compose problems that "don't happen on the CI runner" come from Docker Desktop's VM-hopping filesystem.

Bind mounts are slow

As covered in Lesson 3.3: every bind-mount read/write crosses from macOS/Windows into the Linux VM. Can be 10-50× slower than native. Solutions:

• Named volumes for generated content (node_modules, .venv, vendor/, target/).
• Compose watch (sync) — pushes changes into the container rather than cross-mounting.
• VirtioFS (Docker Desktop setting, default in recent versions) — significantly faster than the old gRPC-FUSE driver.

host.docker.internal

Inside a container on Docker Desktop, host.docker.internal resolves to the Mac/Windows host (not the Linux VM). Useful when a container needs to reach a service running on the Mac.

On Linux, host.docker.internal does not exist by default — you need --add-host=host.docker.internal:host-gateway or the Linux-equivalent trick. Most compose files for cross-platform teams include it:

services:
  app:
    extra_hosts:
      - "host.docker.internal:host-gateway"

With host-gateway, Docker replaces it with the host's IP at container start — works on Linux and Desktop alike.

File permissions

On Linux, bind-mounted host files appear with their host UID. On Docker Desktop (VM-hopping), files often appear owned by the user that the VM is running as — sometimes root, sometimes the dev user. If your image's user is different, writes may fail. Fix by running the container as the host UID:

services:
  app:
    user: "1000:1000"
    volumes:
      - ./:/app

Or by running as root with USER root in your dev Dockerfile (less secure but often simpler for dev).

---

A Full, Realistic Dev Setup

Combining everything — here is a proper dev-mode stack:

# compose.yaml (committed)
name: myapp
services:
  api:
    image: ghcr.io/myorg/api:${VERSION:-latest}
    restart: unless-stopped
    environment:
      NODE_ENV: production
      DATABASE_URL: postgres://user:${POSTGRES_PASSWORD}@db:5432/app
    depends_on:
      db: { condition: service_healthy }
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/health"]
      interval: 15s

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user"]
      interval: 5s
      retries: 20

volumes:
  pgdata:

# compose.override.yaml (committed for dev convenience, or gitignored)
services:
  api:
    build:
      context: .
      target: dev                      # multi-stage Dockerfile dev target
    command: npm run dev
    environment:
      NODE_ENV: development
      LOG_LEVEL: debug
    ports:
      - "127.0.0.1:8080:8080"
      - "127.0.0.1:9229:9229"          # debugger
    volumes:
      - ./src:/app/src
      - ./tests:/app/tests
      - ./package.json:/app/package.json
      - ./package-lock.json:/app/package-lock.json
      - node_modules:/app/node_modules

  db:
    ports:
      - "127.0.0.1:5432:5432"          # expose for local psql
    volumes:
      - ./db/init:/docker-entrypoint-initdb.d:ro    # auto-seed

  mailhog:
    image: mailhog/mailhog:latest
    ports:
      - "127.0.0.1:8025:8025"          # web UI
      - "127.0.0.1:1025:1025"          # SMTP
    profiles: [local]

volumes:
  node_modules:

# .env.example (committed)
VERSION=dev
POSTGRES_PASSWORD=devpass

# .env (gitignored, per developer)
POSTGRES_PASSWORD=mySecretDev123

Daily dev flow:

docker compose up -d              # start everything
docker compose logs -f api        # watch logs
docker compose exec api npm test  # run tests

# Edit source — hot-reloads automatically via nodemon

docker compose run --rm api npm run migrate
# apply migrations

docker compose --profile local up -d
# with mailhog

docker compose down               # stop, keep data
docker compose down -v            # full reset (destroys data)

---

Key Concepts Summary

• Use compose.override.yaml for dev-only additions. Keep compose.yaml prod-faithful.
• Source mounts + anonymous volume for node_modules/.venv. Protect the image's deps from being shadowed.
• Hot reload needs a watcher inside. Dev server, nodemon, air, watchdog, etc. Set polling env vars for Docker Desktop.
• Compose watch (v2.22+) beats bind mounts for Docker Desktop file sync.
• Seed with docker-entrypoint-initdb.d for simple cases, a dedicated seeder service for complex ones.
• docker compose exec for running commands; docker compose run --rm for one-off new containers.
• Commit .env.example, gitignore .env. Compose substitutes ${VAR} from .env.
• host.docker.internal + host-gateway work across Linux and Docker Desktop.
• Docker Desktop file I/O is slower than native — named volumes or Compose watch hide it.
• docker compose down -v destroys volumes — do not run it casually.

Common Mistakes

• Bind-mounting the whole project dir without protecting node_modules / .venv / vendor. The image's deps get shadowed.
• Committing .env with real credentials. Even "dev" creds belong in an untracked file.
• Running docker compose down -v to "clean up" and losing the dev database. Use down alone unless you truly want data gone.
• Expecting hot reload on Docker Desktop without polling env vars. inotify does not fire reliably through VirtioFS / gRPC-FUSE.
• Making the override file too big. Small overrides are maintainable; a 200-line override that duplicates half of compose.yaml is a smell.
• Forgetting to add the override file (or its related files) to the README. If devs need to copy compose.override.example.yaml → compose.override.yaml, document it.
• Putting secrets in environment: rather than secret managers or file-based secrets. Someone will commit them.
• Ignoring docker compose config output when debugging. It shows you what Compose actually thinks your stack is.
• Hardcoding ports in the override. Multiple projects all wanting port 5432 conflict; parameterize via .env (HOST_PG_PORT=5433, - "${HOST_PG_PORT}:5432").
• Forgetting to include extra_hosts: host.docker.internal:host-gateway for Linux devs. Containers that "talk to my Mac" silently break on Linux.

---