Docker container logs can consume disk space rapidly and impact system performance when unmanaged. A single long-running container can generate gigabytes of log data, leading to "No Space Left on Device" errors that bring entire systems to a halt. This guide covers proven methods to safely clear Docker logs, implement automated rotation, and prevent future disk space issues.
Whether you're dealing with emergency disk space situations or implementing proactive log management strategies, these solutions address the needs of developers, SREs, platform engineers, and DevOps professionals managing containerized environments.
Understanding Docker Container Logs
Docker captures all output from containerized applications through stdout and stderr streams. By default, Docker uses the json-file logging driver, storing logs as JSON-formatted files in /var/lib/docker/containers/<container_id>/<container_id>-json.log on the host system.
The Problem: Uncontrolled Log Growth
Docker logs accumulate indefinitely by default, creating critical issues:
- Disk exhaustion: Logs grow from megabytes to gigabytes within days
- System instability: 100% disk usage causes container crashes or startup failures
- Performance degradation: Large log files slow
docker logsoperations - Backup complications: Massive logs complicate backup and recovery processes
Docker Log Architecture
Understanding Docker's log handling enables effective management:
- Container Output: Applications write to
stdoutandstderr - Logging Driver: Captures and processes output (default:
json-file) - Storage Location: Logs stored in
/var/lib/docker/containers/directory - Access Method: Retrieved via
docker logscommand or direct file access
Method 1: Configure Preventive Log Rotation (Recommended)
The most effective approach prevents excessive log accumulation through proper configuration. Docker's built-in log rotation automatically manages log file sizes before they become problematic.
Global Docker Daemon Configuration
Configure rotation for all containers by modifying the Docker daemon:
sudo nano /etc/docker/daemon.json
Add or update the logging configuration:
{
"log-driver": "local",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
This configuration limits log files to 10MB each, keeping a maximum of 3 rotated files per container. The local driver provides better performance and automatic compression compared to the traditional json-file driver.
Restart Docker to apply changes:
sudo systemctl restart docker
Per-Container Configuration
For granular control, configure logging when starting containers:
docker run -d \
--log-driver=local \
--log-opt max-size=10m \
--log-opt max-file=3 \
nginx:latest
Docker Compose Configuration
Implement log rotation in your docker-compose.yml files:
version: "3.8"
services:
web:
image: nginx:latest
logging:
driver: "local"
options:
max-size: "10m"
max-file: "3"
database:
image: postgres:13
logging:
driver: "local"
options:
max-size: "20m"
max-file: "5"
This approach ensures consistent log management across your entire application stack.
Method 2: Safe Log File Truncation
When you need to immediately clear logs from running containers, truncation provides a safe method that preserves log file structure while freeing disk space.
Single Container Truncation
Clear logs for a specific container:
# Get the container ID
docker ps
# Truncate the log file
truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log
For easier identification, use the container name:
docker inspect --format='{{.LogPath}}' container_name | xargs truncate -s 0
Bulk Log Truncation
Clear logs for all containers simultaneously:
# Truncate all container logs
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log
# Alternative method using find
sudo find /var/lib/docker/containers -name "*-json.log" -exec truncate -s 0 {} \;
Verification
Confirm successful truncation:
# Check log file sizes
du -sh /var/lib/docker/containers/*/*-json.log
# Verify logs are cleared
docker logs container_name
Note: Truncation may interrupt docker logs -f (follow) commands. Active log streaming requires restarting.
Method 3: Complete Log File Deletion
For scenarios requiring complete log removal, deletion provides thorough cleanup but requires careful execution.
Safe Deletion Process
- Stop the container (recommended for safety):
docker stop container_name
- Delete the log file:
rm /var/lib/docker/containers/<container_id>/<container_id>-json.log
- Restart Docker daemon (to recreate log file structure):
sudo systemctl restart docker
- Start the container:
docker start container_name
Bulk Deletion Script
For multiple containers, create an automated script:
#!/bin/bash
# bulk_log_cleanup.sh
# Stop all containers
docker stop $(docker ps -q)
# Remove all log files
sudo rm /var/lib/docker/containers/*/*-json.log
# Restart Docker
sudo systemctl restart docker
# Start containers
docker start $(docker ps -aq)
echo "Log cleanup completed successfully"
Make the script executable and run:
chmod +x bulk_log_cleanup.sh
./bulk_log_cleanup.sh
Method 4: System-Level Log Rotation with Logrotate
For advanced log management, integrate Docker logs with the system's logrotate utility for scheduled automatic rotation.
Create Logrotate Configuration
sudo nano /etc/logrotate.d/docker-container-logs
Add the following configuration:
/var/lib/docker/containers/*/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
copytruncate
create 0640 root root
maxsize 100M
postrotate
/usr/bin/docker kill -s USR1 $(docker ps -q) 2>/dev/null || true
endscript
}
Configuration Options Explained
daily: Rotate logs dailyrotate 7: Keep 7 days of rotated logscompress: Compress rotated logs to save spacedelaycompress: Wait one cycle before compressingmissingok: Don't error if log files are missingnotifempty: Don't rotate empty filescopytruncate: Copy and truncate the original filemaxsize 100M: Rotate if file exceeds 100MB regardless of schedulepostrotate: Send USR1 signal to containers for log reopening
Test Logrotate Configuration
# Test the configuration
sudo logrotate -d /etc/logrotate.d/docker-container-logs
# Force rotation for testing
sudo logrotate -f /etc/logrotate.d/docker-container-logs
Method 5: Alternative Logging Drivers
Reduce local storage consumption by configuring Docker to send logs directly to external systems.
Syslog Driver Configuration
Send logs to the system log service:
docker run -d \
--log-driver=syslog \
--log-opt syslog-address=tcp://localhost:514 \
--log-opt tag="docker/{{.Name}}" \
nginx:latest
Journald Integration
Integrate with systemd's journal:
docker run -d \
--log-driver=journald \
--log-opt labels=service \
nginx:latest
# View logs through journalctl
journalctl -u docker.service -f
Remote Logging Solutions
Configure containers to send logs directly to remote systems:
# Fluentd driver
docker run -d \
--log-driver=fluentd \
--log-opt fluentd-address=localhost:24224 \
--log-opt tag="docker.{{.Name}}" \
application:latest
# AWS CloudWatch
docker run -d \
--log-driver=awslogs \
--log-opt awslogs-group=my-application \
--log-opt awslogs-stream=container-logs \
application:latest
Emergency Disk Space Recovery
When your system is critically low on disk space and containers are failing, follow these emergency steps:
1. Immediate Space Recovery
# Quick truncation of all logs
sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'
# Verify space recovery
df -h
2. Identify Largest Log Files
# Find the largest log files
sudo find /var/lib/docker/containers -name "*-json.log" -exec du -h {} \; | sort -hr | head -10
# Detailed breakdown per container
sudo du -sh /var/lib/docker/containers/*/
3. Clean Docker System
# Remove unused containers, networks, images
docker system prune -a -f
# Remove dangling volumes
docker volume prune -f
Automation and Monitoring
Automated Cleanup Script
Create a comprehensive cleanup script for regular execution:
#!/bin/bash
# docker-log-cleanup.sh
LOG_DIR="/var/lib/docker/containers"
MAX_SIZE="100M"
BACKUP_DIR="/var/backups/docker-logs"
echo "Starting Docker log cleanup - $(date)"
# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"
# Function to cleanup logs larger than MAX_SIZE
cleanup_large_logs() {
find "$LOG_DIR" -name "*-json.log" -size +$MAX_SIZE -exec bash -c '
container_id=$(basename $(dirname "$1"))
container_name=$(docker inspect --format="{{.Name}}" "$container_id" 2>/dev/null | sed "s/^///")
if [ -n "$container_name" ]; then
echo "Cleaning logs for container: $container_name ($container_id)"
# Create backup before cleanup
cp "$1" "'"$BACKUP_DIR"'/$(date +%Y%m%d-%H%M%S)-$container_name.log"
# Truncate the log file
truncate -s 0 "$1"
echo "Cleaned: $1 ($(du -h "$1" | cut -f1))"
fi
' bash {} \;
}
# Execute cleanup
cleanup_large_logs
# Clean old backups (older than 7 days)
find "$BACKUP_DIR" -name "*.log" -mtime +7 -exec rm {} \;
echo "Docker log cleanup completed - $(date)"
Cron Job Setup
Schedule automated cleanup:
# Edit crontab
crontab -e
# Add daily cleanup at 2 AM
0 2 * * * /path/to/docker-log-cleanup.sh >> /var/log/docker-cleanup.log 2>&1
# Add hourly checking for critical disk space
0 * * * * df -h | awk '$5 > 90 {print $0}' | grep -q . && /path/to/emergency-cleanup.sh
Monitoring Script
Create a monitoring script to alert on log growth:
#!/bin/bash
# docker-log-monitor.sh
ALERT_THRESHOLD="1G"
EMAIL="admin@example.com"
check_log_sizes() {
find /var/lib/docker/containers -name "*-json.log" -size +$ALERT_THRESHOLD -exec bash -c '
container_id=$(basename $(dirname "$1"))
container_name=$(docker inspect --format="{{.Name}}" "$container_id" 2>/dev/null | sed "s/^///")
size=$(du -h "$1" | cut -f1)
echo "ALERT: Container $container_name has log file size: $size"
# Optional: Send email alert
# echo "Container $container_name log size exceeded threshold: $size" | mail -s "Docker Log Alert" $EMAIL
' bash {} \;
}
check_log_sizes
Best Practices for Docker Log Management
1. Implement Structured Logging
Configure applications to output JSON-formatted logs for better parsing and analysis:
{
"timestamp": "2025-07-24T10:30:00Z",
"level": "ERROR",
"service": "auth-service",
"message": "Authentication failed",
"user_id": "12345",
"error_code": "AUTH001"
}
2. Establish Log Retention Policies
Define clear policies based on your requirements:
- Development: 1-3 days retention
- Staging: 7-14 days retention
- Production: 30-90 days retention (based on compliance needs)
3. Monitor Disk Usage
Implement automated monitoring:
# Disk usage alert script
#!/bin/bash
THRESHOLD=80
USAGE=$(df /var/lib/docker | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $USAGE -gt $THRESHOLD ]; then
echo "WARNING: Docker directory usage at ${USAGE}%"
# Trigger cleanup or alert
fi
4. Security Considerations
- Sensitive Data: Configure log filters to exclude passwords, tokens, and PII
- Access Control: Restrict access to log directories using appropriate file permissions
- Encryption: Use TLS for remote logging to protect data in transit
5. Performance Optimization
- Non-blocking Mode: Use
mode=non-blockingfor high-throughput applications - Buffer Configuration: Adjust
max-buffer-sizefor remote logging drivers - Resource Limits: Set appropriate CPU and memory limits for logging infrastructure
Troubleshooting Common Issues
"No Space Left on Device" Errors
Immediate Solution:
# Quick space recovery
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log
# Clean Docker system
docker system prune -a -f
Root Cause Prevention:
- Implement log rotation configuration
- Set up disk usage monitoring
- Create automated cleanup scripts
Container Restart Doesn't Clear Logs
Problem: Many expect docker restart to clear logs, but logs persist across container restarts.
Solution: Logs are only cleared when containers are removed and recreated:
# This DOES NOT clear logs
docker restart container_name
# This DOES clear logs
docker rm container_name
docker run [original_options] image_name
Log Rotation Not Working
Verification Steps:
# Check Docker daemon configuration
cat /etc/docker/daemon.json
# Verify container logging configuration
docker inspect container_name | grep -A 10 "LogConfig"
# Test log rotation manually
docker run -d --log-opt max-size=1m --log-opt max-file=2 busybox sh -c 'while true; do echo "Test log entry $(date)"; sleep 1; done'
Performance Impact Analysis
Log Clearing Methods Comparison
| Method | Speed | Safety | Downtime | Use Case |
|---|---|---|---|---|
| Truncation | Fast | High | None | Running containers |
| Deletion | Fast | Medium | Required | Maintenance windows |
| Rotation | Automated | High | None | Preventive measure |
| Driver Config | Automated | High | None | New deployments |
Disk Space Recovery Results
Based on real-world implementations:
- Log Rotation: 60-80% storage reduction through compression
- Truncation: 100% immediate space recovery
- Centralized Logging: 95% local storage reduction
- Automated Cleanup: 70% average storage optimization
Get Started with SigNoz
SigNoz provides comprehensive observability for Docker environments through advanced log management capabilities integrated with OpenTelemetry. The platform features Docker-specific log collection with automatic service detection, enhanced log processing through logspout-signoz, and real-time log analysis with correlation across metrics, logs, and traces.
SigNoz offers an OpenTelemetry Collector-based approach for Docker log collection, featuring pre-configured setups for seamless integration. The platform's logspout-signoz router automatically detects service names and metadata from Docker containers, enriching logs with context for better filtering and querying capabilities.
You can choose between various deployment options in SigNoz. The easiest way to get started with SigNoz is SigNoz cloud. We offer a 30-day free trial account with access to all features.
Those who have data privacy concerns and can't send their data outside their infrastructure can sign up for either enterprise self-hosted or BYOC offering.
Those who have the expertise to manage SigNoz themselves or just want to start with a free self-hosted option can use our community edition.
Hope we answered all your questions regarding Docker log management. If you have more questions, feel free to use the SigNoz AI chatbot, or join our slack community.
Conclusion
Effective Docker log management requires a multi-layered approach combining preventive configuration, reactive cleanup methods, and ongoing monitoring. Success lies in implementing proactive log rotation before problems occur, while maintaining emergency response capabilities.
Key Takeaways:
- Prevention First: Configure log rotation using Docker's built-in drivers or docker-compose settings
- Safe Cleaning: Use truncation for running containers, deletion only during maintenance windows
- Automation: Implement scheduled cleanup scripts and monitoring alerts
- Integration: Leverage centralized logging solutions like SigNoz for comprehensive observability
- Best Practices: Establish clear retention policies, implement structured logging, and maintain security protocols
By following these practices and implementing the methods outlined in this guide, you'll maintain optimal Docker performance while ensuring log data remains available for troubleshooting and analysis when needed.
