df says disk is full, but it is not

— ny_wk

When df -h reports a partition at 100% but du -sh only accounts for a fraction of that space, the missing space is almost always held by deleted-but-still-open files, hidden behind a mountpoint, reserved for root, or your real problem is inode exhaustion rather than blocks. This guide walks through each cause in the order a seasoned Linux admin checks them, with the exact commands to find and reclaim the space safely.

Why df and du report different numbers

The two tools measure fundamentally different things, so a mismatch is expected, not a bug. Understanding what each one actually counts is the key to solving a Linux disk full mystery.

• df reads the filesystem superblock and reports how many blocks the filesystem itself considers allocated. It sees the whole picture, including space the kernel still owns.
• du walks the directory tree and sums the sizes of files it can reach by name. If a file has no name in the tree, du never sees it.

That single difference explains the classic symptom: df -h shows a root partition 100% used (say 29 GB), while du -shx / finds only 9 GB. The other 20 GB is real and allocated, but it has no path for du to traverse. The sections below isolate exactly where it went.

Step 1: Confirm the discrepancy precisely

Before chasing ghosts, measure both numbers correctly. Run these from a shell with enough privileges to read every directory.

1. Check what the filesystem reports as used and available:
   df -hP /
2. Sum what is reachable on that same filesystem, staying on one device:
   sudo du -shx /

The -x flag is critical: it tells du to stay on a single filesystem and not wander into /proc, /sys, or other mounted partitions, which would inflate or distort the total. The -P flag forces POSIX output so the columns line up predictably in scripts. If the two totals differ by more than a rounding margin, you have a genuine hidden-space problem and should continue.

Step 2: Reclaim space from deleted-but-open files (the usual culprit)

This is the single most common cause of missing disk space in Linux. On any POSIX system, deleting a file with rm only unlinks it from the directory tree. If a running process still holds the file open, the kernel keeps the data blocks allocated until the last file descriptor is closed. The file is gone by name, so du cannot count it, but df still sees the blocks in use.

Typical offenders are long-running daemons writing to log files that someone deleted instead of rotating, for example Apache, Asterisk, a JBoss/JVM process, MySQL, or Oracle.

Find the open, deleted files

List every deleted file that a process is still holding open, along with its size:

1. Show all deleted-but-open files:
   sudo lsof +L1
   The +L1 filter matches files whose on-disk link count has dropped below 1, which is exactly the deleted-and-open case. The older idiom still works too:
   sudo lsof | grep deleted
2. Read the columns. A line such as
   java 49097 awdmw 77w REG 253,6 33955068440 1283397 /opt/jboss/.../server.log (deleted)
   tells you the command is java, the PID is 49097, the file descriptor is 77, and the held size is roughly 34 GB.

To total the wasted space across all such files at once:

sudo lsof -Fn -Fs | grep -B1 -i deleted | grep '^s' | cut -c 2- | awk '{s+=$1} END {print s" bytes"}'

Free the space the safe way

The cleanest fix is to make the holding process release its descriptor. You have two options, in order of preference.

1. Restart or reload the service. For a log file held by a daemon, a graceful reload or restart closes the old descriptor and the kernel frees the blocks immediately. For example, reloading Asterisk's logger module (asterisk -rx "logger reload") or restarting Apache (sudo systemctl restart httpd) instantly returns the space.
2. Truncate the descriptor in place when you cannot restart the process. Because the deleted file is still reachable through /proc/<pid>/fd/<fd>, you can empty it without touching the running program:
   sudo truncate -s 0 /proc/49097/fd/77
   or the older redirection form:
   sudo sh -c ': > /proc/49097/fd/77'

Substitute the PID and fd from your own lsof output. Confirm you are truncating the right handle first:

sudo file /proc/49097/fd/77

It should print something like broken symbolic link to '/opt/jboss/.../server.log (deleted)', confirming this is the deleted file and not a live one.

Step 3: Look for files hidden behind a mountpoint

If lsof turns up nothing, the space may be buried under a directory that was later used as a mountpoint. Suppose files were written to /home while it lived on the root filesystem, and /home was afterward mounted onto its own partition. Those original files still consume root's blocks, but they are now shadowed by the mount, so du on the running system cannot reach them.

Expose the shadowed files without unmounting anything in use by bind-mounting the root filesystem somewhere fresh:

1. Create a temporary mount target and bind-mount root:
   sudo mkdir -p /mnt/root
sudo mount --bind / /mnt/root
2. Measure the bind view, staying on one filesystem:
   sudo du -shx /mnt/root
3. Compare to your earlier du -shx /. A larger total here means real files are hidden under a mountpoint. Browse /mnt/root/home (or wherever the gap is) to find and remove them.
4. Clean up when finished:
   sudo umount /mnt/root

Step 4: Check root's reserved space

Ext2/3/4 filesystems set aside a percentage of capacity, 5% by default, that only root may write into. This reserve keeps critical daemons running when ordinary users fill the disk, and it helps the allocator avoid fragmentation. The side effect: a non-root user can see 0 bytes available while df still shows a few percent technically unused.

On a large data volume, 5% can be wasteful. To inspect and tune the reserve:

1. View the current reserved block count:
   sudo tune2fs -l /dev/sda3 | grep -i 'reserved block count'
2. Lower the reserve to 1% (replace the device with your own):
   sudo tune2fs -m 1 /dev/sda3

Important: keep a reserve on the partition that holds / (1% is a reasonable floor). Setting it to 0 on the root filesystem can let a runaway process starve system services. On a pure data partition with no system processes, 0 is acceptable.

Step 5: Rule out inode exhaustion (No space left on device)

A different but related failure shows the error "No space left on device" even though df -h reports plenty of free space. This happens when you run out of inodes rather than blocks. Every file, directory, and symlink consumes exactly one inode, and the count is fixed when the filesystem is created. Millions of tiny or zero-byte files (mail spools, session caches, PHP sessions, broken cron output) can exhaust inodes long before they fill the disk.

Diagnose and fix exhausted inodes

1. Confirm blocks are not the problem:
   df -h
2. Check inode usage; look for IUse% at or near 100:
   df -i
3. Find which top-level directory holds the most entries:
   for i in /*; do echo -n "$i "; find "$i" -xdev 2>/dev/null | wc -l; done
4. Drill down into the suspect directory (here, per user home):
   for i in /home/*; do echo -n "$i "; find "$i" -xdev 2>/dev/null | wc -l; done
5. Once you have located the dumping ground of tiny files, remove them. For very large counts, a streamed delete avoids the rm: argument list too long error:
   sudo find /home/bad_user/spamdir -type f -delete
   or for an entire throwaway tree:
   sudo rm -rf /home/bad_user/spamdir

Re-run df -i afterward; IUse% should drop sharply. Note that on XFS, inodes are allocated dynamically and rarely exhaust, so this scenario is mainly an ext-family concern.

Common pitfalls when chasing a full disk

• Deleting a log file instead of truncating it. If a daemon has it open, rm frees nothing. Use truncate -s 0 file.log or set up logrotate with copytruncate instead.
• Forgetting -x on du. Without it, du crosses into /proc, bind mounts, and other partitions, producing numbers that never match df.
• Setting the root reserve to 0. Tempting on a small /, but it removes the safety buffer that keeps the system bootable and serviceable under pressure.
• Truncating the wrong descriptor. Always run file /proc/<pid>/fd/<fd> first and verify it points to a (deleted) path before emptying it.
• Assuming blocks when it is inodes. If writes fail but df -h looks fine, run df -i before anything else.

Verification: confirm the space is actually back

After applying a fix, prove it worked rather than assuming it did.

1. Re-check block usage; the gap between df and du should close:
   df -hP / && sudo du -shx /
2. Re-check inodes if that was the issue:
   df -i
3. Confirm no large deleted files remain open:
   sudo lsof +L1
4. Do a real write test to be certain applications can use the disk again:
   sudo dd if=/dev/zero of=/var/tmp/spacetest bs=1M count=100 && sudo rm /var/tmp/spacetest

When df and du agree (within rounding), inode usage is healthy, and a test write succeeds, the disk is genuinely clear.

Key Takeaways

• df counts allocated blocks; du counts reachable files — a mismatch points to space the directory tree cannot see.
• The most common cause is a deleted-but-open file; find it with lsof +L1 and free it by restarting the process or truncating /proc/<pid>/fd/<fd>.
• Also check files hidden behind a mountpoint using mount --bind / /mnt/root and re-measuring with du -shx.
• "No space left on device" with free blocks means inode exhaustion — diagnose with df -i and clean out the directory full of tiny files.
• Always verify with df, du, df -i, and a real write test before declaring the disk fixed.

Frequently Asked Questions

Why does df show the disk full but du shows free space?

Because they measure different things. df reports blocks the filesystem has allocated, including data held by files that were deleted while still open by a process. du only sums files it can reach by name, so it cannot see those orphaned-but-open files. Run sudo lsof +L1 to find them.

How do I free space from a deleted file that is still open?

Restart or gracefully reload the process holding it open, which closes the descriptor and releases the blocks. If you cannot restart it, truncate the live descriptor in place with sudo truncate -s 0 /proc/<pid>/fd/<fd>, using the PID and fd from lsof.

Why do I get "No space left on device" when df shows free space?

You have likely run out of inodes, not disk blocks. Each file consumes one inode, and a directory full of tiny or zero-byte files can exhaust them. Check with df -i; if IUse% is near 100%, locate and delete the excess small files.

Is it safe to reduce the reserved space with tune2fs?

On a dedicated data partition, lowering the reserve with sudo tune2fs -m 1 /dev/sdXn is fine and reclaims usable space. On the root filesystem, keep at least 1% so critical system processes still have a write buffer if the disk fills.

For more hands-on Linux and sysadmin walkthroughs, subscribe on YouTube @explorenystream.