Filesystem full alerts are common and usually simple. The tricky cases happen when df disagrees with du: a deleted file that is still held open by a process.

Start with the basics

df -h
df -i

Inodes matter too. If df -i shows 100 percent usage you have too many small files, not too much data.

Find the space hogs

du -x -h --max-depth=1 / | sort -h | tail

The -x flag keeps the scan on one filesystem, so mounts do not distort the totals.

The classic hidden culprit

When du reports far less usage than df, a process still holds a deleted file open. The space is only released when the process closes the file or restarts.

lsof +L1 | grep deleted
  1. Identify the process from the output.
  2. Restart the service gracefully during a maintenance window.
  3. Confirm the space is released with df -h.

Log files and rotated logs

Journal logs can grow quickly. Cap the journal and enable rotation:

journalctl --vacuum-size=200M
sudo nano /etc/systemd/journald.conf   # set SystemMaxUse=200M

Never delete a log file that a running process still has open. Truncate it instead: > /var/log/nginx/access.log

Prevention

  • Alert at 80 percent, not 100.
  • Use logrotate with compression.
  • Watch tmp directories used by uploads and caches.