Log Management

When something goes wrong on a Linux system, logs are where you find out why. Knowing where logs live, how to read them efficiently, and how to stop them from eating all your disk space is core sysadmin knowledge.

/var/log

Traditional log files live in /var/log. Each service typically writes its own file here.

ls /var/log

Important files and directories:

Reading log files

cat /var/log/auth.log          # print entire file
tail /var/log/auth.log         # last 10 lines
tail -n 50 /var/log/auth.log   # last 50 lines
tail -f /var/log/auth.log      # follow in real time

tail -f is one of the most useful commands for watching a service in real time. Open it in one terminal while you reproduce a problem in another.

Searching logs with grep

grep "Failed password" /var/log/auth.log
grep "ERROR" /var/log/nginx/error.log
grep -i "error" /var/log/syslog          # case insensitive
grep -r "segfault" /var/log/             # recursive across all logs

Combine with tail for recent errors only:

tail -n 200 /var/log/syslog | grep "error"

grep with context

When you find an error, you usually want the lines around it:

grep -A 5 "Failed password" /var/log/auth.log   # 5 lines after
grep -B 5 "Failed password" /var/log/auth.log   # 5 lines before
grep -C 5 "Failed password" /var/log/auth.log   # 5 lines either side

journalctl in depth

systemd's journal is the other major log source. We covered the basics in the systemd post — here is more depth.

Filtering by time

journalctl --since today
journalctl --since "1 hour ago"
journalctl --since "2026-05-01" --until "2026-05-02"
journalctl --since "2026-05-01 09:00:00" --until "2026-05-01 09:30:00"

Filtering by priority

journalctl -p err          # errors and above
journalctl -p warning      # warnings and above
journalctl -p 0..3         # range: emerg through err

Filtering by unit and combining filters

journalctl -u nginx -p err --since today

Filtering by executable or PID

journalctl _EXE=/usr/bin/sshd
journalctl _PID=1234

Kernel messages

journalctl -k               # kernel messages only
journalctl -k --since today # today's kernel messages

dmesg also shows kernel messages, with a focus on the current boot:

dmesg
dmesg | tail -20
dmesg -T    # human-readable timestamps
dmesg -l err,warn    # filter by level

Journal disk usage

journalctl --disk-usage

Vacuum old entries to free space:

sudo journalctl --vacuum-time=30d    # delete entries older than 30 days
sudo journalctl --vacuum-size=500M   # reduce journal to 500MB

Persistent journal

By default on some distributions the journal does not persist across reboots. To enable persistence:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

logrotate

Log files grow indefinitely if nothing manages them. logrotate handles this by compressing and rotating log files on a schedule.

The main config is /etc/logrotate.conf. Per-application configs live in /etc/logrotate.d/.

cat /etc/logrotate.d/nginx

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        nginx -s reopen
    endscript
}

Breaking this down:

• daily — rotate every day
• missingok — do not error if the log file is missing
• rotate 14 — keep 14 rotated copies before deleting
• compress — gzip old log files
• delaycompress — wait one rotation before compressing (so the previous log is still readable)
• notifempty — do not rotate empty files
• create 0640 www-data adm — create a new empty log with these permissions and ownership
• postrotate / endscript — commands to run after rotation (here, telling nginx to reopen its log files)

Testing logrotate

sudo logrotate --debug /etc/logrotate.d/nginx

--debug shows what would happen without actually doing it. Useful for validating a config.

To force rotation immediately:

sudo logrotate --force /etc/logrotate.d/nginx

A practical troubleshooting workflow

When something is broken:

# 1. Check if the service is running
systemctl status myservice
 
# 2. Check its recent logs
journalctl -u myservice -n 50
 
# 3. Check for errors specifically
journalctl -u myservice -p err --since "1 hour ago"
 
# 4. Check auth logs if it is a permission or login issue
sudo tail -50 /var/log/auth.log
 
# 5. Check kernel logs if it is hardware or driver related
dmesg | tail -30
 
# 6. Search broadly across all logs
grep -r "myservice" /var/log/ 2>/dev/null

Logs tell the full story of what a system has been doing. The ability to read them quickly — knowing where to look, how to filter, and how to follow output in real time — is what separates someone who guesses at problems from someone who diagnoses them.

That is the end of the technical deep dive. The series now covers everything from the basics of the filesystem through to process management, networking, SSH, systemd, disk management, user administration, scheduled tasks, firewalls, environment variables, and log management. That is a real foundation.