Docker Logging — From docker logs to ELK Stack

July 12, 2025 · 8 min read

DevOps & Cloud Learning Hub

Your application logs are the single most important debugging tool you have. In a containerized world, those logs disappear when the container is removed — unless you have a logging strategy. Most teams start with docker logs and stop there. That works for a single container. It falls apart completely at 50 containers across 10 services.

How Docker Logging Works

By default, Docker captures everything your application writes to stdout and stderr. It stores this output as JSON files on the host at /var/lib/docker/containers/<container-id>/<container-id>-json.log. That is the json-file logging driver, and it is the default on every Docker installation.

# See the raw log file on the host
sudo ls -la /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' myapp)/
# ...-json.log    (this is where docker logs reads from)

# Each line is a JSON object
sudo head -3 /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' myapp)/*-json.log
# {"log":"Server started on port 3000\n","stream":"stdout","time":"2025-07-12T10:00:00.123Z"}

This default works. But it has a critical flaw: there is no log rotation by default. Your container runs for weeks, the log file grows to 50 GB, and your disk fills up. Production crashes because of logs.

The docker logs Command

docker logs reads from whatever logging driver is configured. With the default json-file driver, it gives you powerful filtering options.

# View all logs
docker logs myapp

# Follow logs in real-time (like tail -f)
docker logs -f myapp

# Last 100 lines
docker logs --tail 100 myapp

# Logs with timestamps
docker logs -t myapp

# Logs since a specific time
docker logs --since "2025-07-12T10:00:00" myapp

# Logs from the last 30 minutes
docker logs --since 30m myapp

# Logs between two timestamps
docker logs --since "2025-07-12T10:00:00" --until "2025-07-12T11:00:00" myapp

# Combine: follow, with timestamps, last 50 lines
docker logs -f -t --tail 50 myapp

# Pipe to grep for filtering
docker logs myapp 2>&1 | grep "ERROR"

# Count errors in the last hour
docker logs --since 1h myapp 2>&1 | grep -c "ERROR"

# Save logs to a file
docker logs myapp > app.log 2>&1

Important: docker logs only works with the json-file and journald logging drivers. If you switch to syslog, fluentd, or awslogs, docker logs returns nothing — you must use the external system to view logs.

Docker Logging Drivers

Docker supports multiple logging drivers that send logs to different destinations.

Driver	Destination	docker logs works?	Use Case
`json-file`	Local JSON files	Yes	Default, single host
`local`	Optimized local files	Yes	Better performance than json-file
`journald`	systemd journal	Yes	Linux systems using systemd
`syslog`	Syslog server	No	Legacy infrastructure
`fluentd`	Fluentd collector	No	Centralized logging
`awslogs`	CloudWatch Logs	No	AWS deployments
`gcplogs`	Google Cloud Logging	No	GCP deployments
`gelf`	Graylog (GELF format)	No	Graylog stack
`splunk`	Splunk HTTP Event Collector	No	Enterprise Splunk

# Set logging driver per container
docker run -d \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=5 \
  myapp:latest

# Set logging driver for AWS CloudWatch
docker run -d \
  --log-driver awslogs \
  --log-opt awslogs-region=us-east-1 \
  --log-opt awslogs-group=myapp-production \
  --log-opt awslogs-stream=api \
  myapp:latest

Set the default logging driver for the entire Docker daemon:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5",
    "compress": "true"
  }
}

# Restart Docker daemon to apply
sudo systemctl restart docker

Log Rotation Configuration

Without log rotation, a busy container can fill your disk in days. Always configure rotation.

# Per-container rotation
docker run -d \
  --log-opt max-size=10m \
  --log-opt max-file=5 \
  myapp:latest
# Each log file maxes at 10MB, keeps 5 rotated files = 50MB max per container

# docker-compose.yml
services:
  api:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
        compress: "true"

  worker:
    image: myworker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"

For production, use the local driver — it is optimized for performance and has built-in rotation:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Structured Logging (JSON Format)

Plain text logs like [2025-07-12 10:00:00] ERROR: Connection refused are human-readable but machine-hostile. Structured JSON logs can be parsed, filtered, and indexed by log aggregation tools.

// Node.js structured logging with pino
const pino = require('pino');
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// Usage
logger.info({ userId: 123, action: 'login' }, 'User logged in');
logger.error({ err, requestId: req.id }, 'Database query failed');

# Python structured logging
import json
import logging
import sys

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "logger": record.name,
            "module": record.module,
        }
        if record.exc_info:
            log_record["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_record)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
logging.root.addHandler(handler)
logging.root.setLevel(logging.INFO)

Output:

{"timestamp":"2025-07-12T10:00:00","level":"INFO","message":"User logged in","userId":123,"action":"login"}
{"timestamp":"2025-07-12T10:00:01","level":"ERROR","message":"Database query failed","requestId":"abc-123","error":"connection refused"}

Centralized Logging with ELK Stack

The ELK stack (Elasticsearch + Logstash + Kibana) is the most popular centralized logging solution. Here is a complete setup with Docker Compose.

# docker-compose.elk.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      retries: 5

  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
      - "12201:12201/udp"
    depends_on:
      elasticsearch:
        condition: service_healthy

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy

  # Your application — logs sent via GELF driver
  api:
    image: myapp:latest
    logging:
      driver: gelf
      options:
        gelf-address: "udp://localhost:12201"
        tag: "api"

volumes:
  es-data:

# logstash.conf
input {
  gelf {
    port => 12201
    type => "docker"
  }
}

filter {
  if [message] =~ /^\{/ {
    json {
      source => "message"
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "docker-logs-%{+YYYY.MM.dd}"
  }
}

Loki + Grafana Alternative

Loki is a lighter-weight alternative to ELK. It does not index log content — only labels. This makes it significantly cheaper to operate at scale.

# docker-compose.loki.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - grafana-data:/var/lib/grafana

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

volumes:
  loki-data:
  grafana-data:

# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    static_configs:
      - targets: [localhost]
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*-json.log
    pipeline_stages:
      - json:
          expressions:
            log: log
            stream: stream
            time: time
      - output:
          source: log

Fluent Bit as a Log Forwarder

Fluent Bit is a lightweight log processor and forwarder — much lighter than Fluentd. It can ship Docker logs to Elasticsearch, Loki, CloudWatch, or any other destination.

# docker-compose.yml
services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    ports:
      - "24224:24224"

  api:
    image: myapp:latest
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: myapp.api

# fluent-bit.conf
[SERVICE]
    Flush        5
    Log_Level    info

[INPUT]
    Name         forward
    Listen       0.0.0.0
    Port         24224

[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       json

[OUTPUT]
    Name         es
    Match        *
    Host         elasticsearch
    Port         9200
    Index        docker-logs
    Type         _doc

Production Logging Best Practices

Practice	Why
Always configure log rotation	Prevent disk full crashes
Use structured JSON logs	Machine-parseable, filterable
Log to stdout/stderr, not files	Let Docker handle log collection
Include request IDs	Trace requests across services
Set appropriate log levels	DEBUG in dev, INFO/WARN in production
Never log secrets	Passwords, tokens, PII
Add context to errors	Stack traces, request data, user ID
Use a centralized logging system	Search across all services

# Check disk usage from Docker logs
docker system df -v | head -20

# Find containers with the largest logs
for container in $(docker ps -q); do
  name=$(docker inspect -f '{{.Name}}' $container)
  size=$(docker inspect -f '{{.LogPath}}' $container | xargs ls -lh 2>/dev/null | awk '{print $5}')
  echo "$name: $size"
done

Wrapping Up

Start with the basics: configure log rotation on every container, switch to structured JSON logging, and use docker logs for quick debugging. When you outgrow single-host logging, deploy Loki + Grafana for a lightweight centralized solution, or the full ELK stack for advanced search and analytics. The key principle is simple — your application writes to stdout, and the infrastructure handles collection, shipping, and storage.

In the next post, we will cover Docker Resource Limits — how to set CPU and memory limits, understand OOM kills, and right-size your containers so they do not starve each other or crash the host.

How Docker Logging Works​

The docker logs Command​

Docker Logging Drivers​

Log Rotation Configuration​

Structured Logging (JSON Format)​

Centralized Logging with ELK Stack​

Loki + Grafana Alternative​

Fluent Bit as a Log Forwarder​

Production Logging Best Practices​

Wrapping Up​

Stay Updated