How We Eliminated Deployment Downtime with Docker DNS and Sequential Container Swaps

Every push to production took the site down for 2–3 minutes. The Drone CI pipeline ran docker compose down, killed all four containers simultaneously, then brought them back one at a time with health checks. Backend, frontend, SSR renderer, and nginx — all dead at once. Clients saw 502 errors. Every. Single. Deploy.

This was “normal” for months. But with active users hitting the site and Stripe transactions in flight, a 3-minute outage window was unacceptable.

The root cause: nginx used static upstream blocks that resolved container IPs once at startup. If a container was recreated, nginx held a stale IP and requests failed until nginx itself restarted.

The Solution

Two changes that together eliminated the outage:

1. DNS-based upstream resolution in nginx — Replace static upstream blocks with Docker’s internal DNS resolver (127.0.0.11). Nginx re-resolves container names on every request. Containers can be recreated freely; nginx picks up the new IP automatically.
2. Sequential rolling deploys in Drone — Instead of docker compose down followed by docker compose up, build all new images first (zero impact on running containers), then recreate each service one at a time with --no-deps --force-recreate.

Result: ~5 seconds of downtime (nginx swap only) instead of 2–3 minutes. A 97% reduction.

Architecture

Before: The old way

docker compose down → ALL containers die → 2-3 min outage → docker compose up
         │
         ▼
   ┌──────────┐
   │  nginx   │  ✗ dead
   └──────────┘
   ┌──────────┐
   │ frontend │  ✗ dead
   └──────────┘
   ┌──────────┐
   │   SSR    │  ✗ dead
   └──────────┘
   ┌──────────┐
   │ backend  │  ✗ dead
   └──────────┘

After: Rolling deploy

docker compose build  →  images ready, all containers still serving
   │
   ├── up -d --force-recreate backend      → ~2s backend blip
   │     sleep 15
   ├── up -d --force-recreate frontend-ssr → ~2s SSR blip
   │     sleep 15
   ├── up -d --force-recreate frontend     → ~2s frontend blip
   │     sleep 10
   └── up -d --force-recreate nginx        → ~5s nginx swap
         sleep 5
            │
            ▼
   All healthy. Deploy complete.

At any moment during the deploy, at least 3 of 4 services are live. Nginx buffers and retries through the brief gaps. The health check endpoint never dropped a single probe during our orchestration smoke test.

The Key Change: Nginx DNS Resolver

The entire strategy hinges on one nginx directive:

# Old: static upstream blocks — resolved once at startup
upstream backend {
    server wehireit-backend-prod:80;
}
proxy_pass http://backend/api/;

# New: Docker DNS resolver — resolved on every request
resolver 127.0.0.11 valid=30s ipv6=off;

location /api/ {
    set $backend_upstream wehireit-backend-prod:80;
    proxy_pass http://$backend_upstream;
}

Docker’s embedded DNS server at 127.0.0.11 tracks container IPs in real time. When docker compose up --force-recreate swaps a container, the DNS record updates within seconds. Nginx picks up the new IP on the next request.

The valid=30s parameter caches DNS responses for 30 seconds — a balance between performance and freshness. If a container swap happens mid-cache, nginx may get a connection refused on the first retry, but proxy_connect_timeout handles that gracefully.

Critical Gotcha: Variable-Based proxy_pass and URIs

When you use a variable in proxy_pass, nginx treats URI paths differently:

# WITHOUT variable — nginx replaces matched location prefix
location /api/ {
    proxy_pass http://backend/api/;  # /api/products → /api/products ✓
}

# WITH variable — URI path REPLACES entire request URI
location /api/ {
    proxy_pass http://$backend_upstream/api/;  # /api/products → /api/ ✗
}

The fix: for pass-through locations where the proxy_pass path matches the location prefix, drop the URI path entirely:

location /api/ {
    proxy_pass http://$backend_upstream;  # /api/products → /api/products ✓
}

For rewrite locations (like /sitemap.xml → /api/Product/sitemap.xml), keep the URI path — the full-URI replacement is the desired behaviour.

The Drone Pipeline

steps:
  - name: deploy
    commands:
      # Build everything first — zero impact on running site
      - docker compose build --no-cache backend frontend frontend-ssr nginx

      # Sequential rolling deploy — one service at a time
      - docker compose up -d --no-deps --force-recreate backend
      - sleep 15
      - docker compose up -d --no-deps --force-recreate frontend-ssr
      - sleep 15
      - docker compose up -d --no-deps --force-recreate frontend
      - sleep 10
      - docker compose up -d --no-deps --force-recreate nginx
      - sleep 5

No docker compose down. No --remove-orphans. No mass service killing. The --no-deps flag ensures only the named service is touched — other containers keep running.

Testing Before Production

Before pushing to prod, we built an orchestration smoke test — a 4-service mirror stack (nginx proxy + 3 upstreams) with a Python test runner that:

1. Starts the stack with docker compose up
2. Launches continuous health checks at 500ms intervals
3. Simulates a build phase
4. Runs sequential recreations of all 3 upstream services
5. Analyses the health probe log for gaps

Result: 137 probes, 137× 200 OK. Zero gaps. The pattern proved itself in isolation before touching production.

The test harness and configs are runnable on any machine with Docker — no production access needed to validate the approach.

Why This Approach Wins

For a single-server deployment with 4 services, the DNS resolver pattern hits the sweet spot: near-zero downtime with no new infrastructure. No load balancers. No orchestration platforms. Just Docker’s built-in DNS and one nginx config change.

Files Changed

Two files. One commit. 97% downtime reduction.

Reproduce It Yourself

1. Switch nginx from static upstreams to DNS resolver (see nginx config above)
2. Change your CI pipeline from down; up to sequential up -d --no-deps --force-recreate
3. Build images first, then recreate services from least-dependent to most-dependent (backend → SSR → frontend → nginx)
4. That’s it. No new tools. No new servers. No new costs.

Tested on: Docker 28 + nginx:alpine + Drone CI on Oracle ARM64. Works the same on any Docker engine.