When Docker Starts Before Your Network Storage: Building a Safe Retry Watchdog

Network storage is excellent for a home server. It lets several services share a large media or data pool, while the Docker host keeps its operating system and application configuration on a local disk. The catch is startup order: after a reboot, the network, the storage server, the file-sharing service, and Docker may all become ready at different times.

That timing difference recently caused a media application to remain offline. The application itself was healthy before the reboot. Docker could not create its process because the host bind mount pointed at an SMB share that was not ready. Once the share mounted, starting the existing container restored the service immediately.

Why Docker restart policies are not enough

A Compose setting such as restart: unless-stopped is still useful, but it mainly handles failures after the container has started and its process has exited. A host bind mount is prepared earlier, while Docker is creating the container process. If that mount cannot be established, the application entrypoint never gets a chance to run.

This is also why putting a retry loop in the image is the wrong layer. The container cannot check the storage mount until Docker has successfully mounted it into the container. The retry controller belongs on the Docker host.

The dangerous failure mode

Linux mount points can exist as ordinary directories or systemd automount placeholders before the remote filesystem is actually available. Docker may then see a path but not the intended SMB or NFS filesystem. Depending on the configuration, an application can either fail to start or start while seeing an empty local directory.

The second case is particularly confusing: the container reports “running”, but the application appears to have lost its media or data. A safe controller should require a real filesystem at the path, not merely the existence of a directory.

A reusable host-side design

The solution we deployed is a small systemd timer and watchdog script. A configuration file maps each storage-dependent container to one or more required host mount paths:

# container|comma-separated required mount paths
media-service|/mnt/shared-media
photo-service|/mnt/photos,/mnt/archive

Every 30 seconds, the watchdog uses Docker’s local API through the Docker CLI to inspect each configured container. Running containers are left alone. Stopped containers are only started after every required path passes the mount check.

status=$(docker inspect -f '{{.State.Status}}' "$container")

if [ "$status" != "running" ] && findmnt -rn -T "$required_mount" | grep -qv autofs; then
    docker start "$container"
fi

The production version also triggers the corresponding systemd mount unit, waits through a bounded retry period, logs why a container is waiting, and retries the Docker start on the next timer run. It does not restart healthy applications and it does not stop unrelated containers.

Why systemd is part of the solution

For network filesystems, the /etc/fstab entry should use network-aware options such as _netdev, a deliberate mount timeout, and—when appropriate—systemd automounting. The mount unit should not hold the entire operating system hostage when the storage server is offline, but dependent applications should wait rather than start against an empty directory.

A systemd timer is a good fit because it survives Docker daemon restarts, host reboots, and temporary storage outages. It also keeps the recovery logic outside the application images, so the same mechanism works for Jellyfin, Immich, backup tools, download managers, databases, and custom services.

Health checks still matter

A successful docker start only proves that the process launched. Add an application health check as a second layer: an HTTP health endpoint, a TCP check, or a service-specific command. Storage readiness answers “can the container start safely?”; a health check answers “is the application actually usable?”

Practical checklist

  • Keep application configuration and databases on storage that is available during boot, or give them the same dependency treatment.
  • Use findmnt to confirm the expected filesystem type, rather than checking only that a directory exists.
  • Keep restart: unless-stopped for ordinary process crashes.
  • Use an external host-side retry controller for mount and Docker-start failures.
  • Log each waiting and recovery decision, then monitor the application health endpoint.
  • Test recovery deliberately by stopping a test container and temporarily making its required mount unavailable.
  • Never put passwords, access tokens, private IP addresses, or real mount credentials in a public article or repository.

The central principle is simple: a container should not be allowed to start until the host resources it depends on are genuinely ready. Once that dependency is made explicit and retriable, a transient reboot race becomes a recoverable event instead of a mysterious outage.

Leave a Reply

Your email address will not be published. Required fields are marked *