Server Load and Docker Resource Monitoring Guide

Monitoring your server’s resource usage is crucial for maintaining optimal performance and identifying potential bottlenecks. This guide covers various methods to monitor CPU, RAM, disk usage, and specifically how to monitor Docker containers.

Quick Docker Monitoring

Real-time Container Stats

The fastest way to check Docker resource usage is with the built-in docker stats command:

# Show real-time stats for all running containers
docker stats

# Show stats for specific containers
docker stats container1 container2

# Show stats without streaming (one-time snapshot)
docker stats --no-stream

# Format output for easier parsing
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"

Container Resource Limits

Check what resource limits are set on your containers:

# Inspect container resource constraints
docker inspect <container_name> | grep -A 10 "Resources"

# Show container resource usage with limits
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.BlockIO}}"

System-Wide Resource Monitoring

CPU Usage

# Real-time CPU usage
top
htop  # More user-friendly (install with: sudo apt install htop)

# CPU usage summary
cat /proc/loadavg

# Detailed CPU info
lscpu

Memory Usage

# Memory usage summary
free -h

# Detailed memory information
cat /proc/meminfo

# Memory usage by process
ps aux --sort=-%mem | head -10

Disk Usage

# Disk space usage
df -h

# Directory size usage
du -sh /path/to/directory

# Find largest files/directories
du -h / | sort -rh | head -20

# Docker-specific disk usage
docker system df
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

Network Usage

# Network interface statistics
ifstat
iotop  # I/O usage by process

# Network connections
netstat -tulpn
ss -tulpn  # Modern replacement for netstat

Advanced Docker Monitoring

Docker System Information

# Overall Docker system info
docker system df

# Detailed space usage
docker system df -v

# Clean up unused resources
docker system prune

# Container logs with resource context
docker logs --details <container_name>

Container Process Monitoring

# Show processes running in a container
docker top <container_name>

# Execute commands inside running container
docker exec -it <container_name> top
docker exec -it <container_name> ps aux

Docker Compose Monitoring

# Monitor all services in a compose stack
docker-compose top

# View logs for all services
docker-compose logs -f

# Check service status
docker-compose ps

Setting Up Continuous Monitoring

Using Built-in Tools

1. Create a monitoring script:

#!/bin/bash
# save as monitor.sh

echo "=== System Overview $(date) ==="
echo "Load Average: $(cat /proc/loadavg)"
echo "Memory Usage:"
free -h
echo ""
echo "=== Docker Stats ==="
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"
echo ""
echo "=== Disk Usage ==="
df -h
echo ""
echo "=== Docker Disk Usage ==="
docker system df

2. Run monitoring script periodically:

# Make executable
chmod +x monitor.sh

# Run every 5 minutes
watch -n 300 ./monitor.sh

# Or add to crontab for logging
crontab -e
# Add: */5 * * * * /path/to/monitor.sh >> /var/log/system-monitor.log

Using Monitoring Tools

Prometheus + Grafana (Recommended)

Create a monitoring/docker-compose.yml:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  grafana:
    image: grafana/grafana
    ports:
      - "3001:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  node-exporter:
    image: prom/node-exporter
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'

  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

volumes:
  prometheus_data:
  grafana_data:

Simple Resource Alerts

#!/bin/bash
# save as alert.sh

CPU_THRESHOLD=80
MEM_THRESHOLD=80
DISK_THRESHOLD=85

# Check CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then
    echo "HIGH CPU USAGE: $CPU_USAGE%"
fi

# Check memory usage
MEM_USAGE=$(free | grep Mem | awk '{printf("%.0f", $3/$2 * 100.0)}')
if [ $MEM_USAGE -gt $MEM_THRESHOLD ]; then
    echo "HIGH MEMORY USAGE: $MEM_USAGE%"
fi

# Check disk usage
DISK_USAGE=$(df / | tail -1 | awk '{print $5}' | cut -d'%' -f1)
if [ $DISK_USAGE -gt $DISK_THRESHOLD ]; then
    echo "HIGH DISK USAGE: $DISK_USAGE%"
fi

Troubleshooting Common Issues

High CPU Usage

# Find CPU-intensive processes
ps aux --sort=-%cpu | head -10

# Find CPU-intensive Docker containers
docker stats --format "table {{.Container}}\t{{.CPUPerc}}" | sort -k2 -nr

# Check if containers have CPU limits
docker inspect <container> | grep -i cpu

High Memory Usage

# Find memory-intensive processes
ps aux --sort=-%mem | head -10

# Check for memory leaks in containers
docker stats --format "table {{.Container}}\t{{.MemUsage}}"

# Check container memory limits
docker inspect <container> | grep -i memory

High Disk Usage

# Find large Docker images
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | sort -k3 -hr

# Clean up Docker resources
docker system prune -a
docker volume prune

# Find large files
find / -type f -size +100M 2>/dev/null | head -20

Network Issues

# Check network connections per container
docker exec <container> netstat -tulpn

# Monitor network I/O
iotop -n

# Check Docker network usage
docker network ls
docker network inspect <network_name>

Performance Optimization Tips

Set Resource Limits: Always set CPU and memory limits for containers
Use Multi-stage Builds: Reduce image sizes
Regular Cleanup: Schedule regular docker system prune
Monitor Logs: Keep log sizes under control with log rotation
Health Checks: Implement container health checks

Useful Commands Reference

# Quick system overview
htop
docker stats --no-stream
df -h
free -h

# Docker-specific monitoring
docker system df -v
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
docker logs --tail 50 <container>

# Performance analysis
iostat -x 1
vmstat 1
sar -u 1

# Clean up commands
docker system prune -a
docker volume prune
docker image prune -a

This guide provides a comprehensive approach to monitoring your server and Docker resources. Start with the basic commands and gradually implement more sophisticated monitoring solutions as needed.