"No space left on device" but df shows free space
Tested on Ubuntu 24.04 with ext4, and on Ubuntu 26.04, whose du, df and truncate are the new Rust rewrites: the commands work the same.
Check four things, in this order: inodes (df -i), deleted files that a process still holds open (sudo lsof +L1), ext4's 5% reserve for root, and whether the path that failed is really on the filesystem you checked (df -h /that/path).
1. Out of inodes
Every file uses an inode as well as space, and a filesystem has a fixed number of them. Millions of tiny files (session files, mail queues, cache entries) can use them all up while gigabytes stay free.
df -i /var
If IUse% is at or near 100%, find where the files are:
sudo du --inodes -x -d 3 / 2>/dev/null | sort -n | tail -15
The fix is deleting the files, usually a cache or queue directory that nothing cleans, and then making something clean it.
2. Deleted, but still open
Deleting a file only removes its name. If a process still has it open, typically a log deleted by hand while the service kept writing, the space isn't freed until that process closes it. df counts the space as used; du can't see the file, so the two disagree.
sudo lsof +L1
Restart or reload the process to release it. If you can't restart it yet, empty the file through the process's open file handle, using the PID and FD number from lsof:
sudo truncate -s 0 /proc/1234/fd/5
Next time, empty a live log with truncate -s 0 instead of deleting it, or let logrotate's copytruncate do it.
3. The root-only reserve
ext4 keeps 5% of the disk for root by default, so the system can still log and recover when it's full. Ordinary users and services get "No space left on device" while root-owned processes carry on writing, and df shows Avail at 0 even though Size minus Used leaves a few gigabytes. Check the reserve:
sudo tune2fs -l /dev/sdb1 | grep -iE 'reserved block count|block size'
On a big data disk 5% can be a lot of space. You can lower it there (here to 1%); leave the root filesystem's reserve alone:
sudo tune2fs -m 1 /dev/sdb1
4. A different filesystem
The full filesystem isn't always the one you looked at. /tmp may be a small tmpfs in memory, /var/lib/docker or a database directory may be its own partition, and a container writes into its own layer. Ask about the exact path that failed:
df -h /var/lib/postgresql
Less common
- Quotas: the user has hit their limit (
quota -s). - btrfs: metadata space runs out before data space (
sudo btrfs filesystem usage /). - Docker: logs, images and build cache; see Docker is filling my disk.
Alert before it happens
Inodes deserve their own alert, because a space alert never sees them coming. With node_exporter and Prometheus:
- alert: DiskInodesLow
expr: |
node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_files < 0.05
and node_filesystem_readonly == 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} {{ $labels.mountpoint }} has under 5% of its inodes left"
The space alerts that go with it are in A disk space alert that doesn't cry wolf.
Something started writing
A full disk is the symptom; the cause is whatever changed: a job creating files it never removes, a service that switched to debug logging. Whyfire compares each server's signals, log volume included, with the hour before, and names the first one that changed when an alert fires.