Disks getting full on Linux is a common scenario any administrator have probably experienced. There are usually some large files in /var/log directory that can be compressed or deleted to save disk space. Have you ever experienced deleting a large log file and noticed that the disk usage remained the same ? It’s as if the file size usage is not released even though you’ve deleted it?
This scenario can happen if you deleted/renamed/moved an existing file that is currently in use by a running program. This usually happens on log files that are not rotated regularly and then when the file size becomes large, you deleted (or compressed) the file without even checking if it is in-use. Don’t worry, you can still recover your disk space as shown from the example below.
Check first before deleting
First, it’s better off if you can avoid this situation entirely. If you are going to delete a large log file, first check if the file currently in use using the lsof or fuser command:
# lsof secure
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 518 root 7w REG 202,2 1104387 17708 secure
We can see that the secure file name is in-use by the rsyslogd process with PID 518. You have two choices to delete this file,
a. Stop the process that is using the file ( which is rsyslog in this case), delete the file, and then start that process again.
Since the file has been deleted, it will be recreated again when that process starts again.
b. Truncate the file. You will simply empty the contents of the file without stopping/restarting the process using it. This is safe to use if the process is critical and can’t be restarted just to delete a log file. You can truncate the a file using the following command.
echo > /path/to/filename
What if I already deleted it?
OK, what if the file is already deleted and I can’t restart the critical process. How can I release the used disk space ?
– You can’t safely release the disk usage of the deleted file without restarting the application process using it. However, if you REALLY need to clear it’s disk usage, you can do that by truncating the file descriptor pointing to the deleted file (which should still be in memory since you did not restart the process using it).
Find the file that was deleted using the lsof command:
# lsof -nP +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
rsyslogd 518 root 7w REG 202,2 1104623 0 17708 /var/log/secure (deleted)
Seen on this example is that secure file was deleted. However, the rsyslog process is still writing to this file so the disk usage will continue to increase.
So find the file descriptor for this file, note the PID and the FD field of the lsof command. In this case, PID: 518 and FD: 7w.
File descriptor are located on /proc/PID#/fd/FD# on Linux. You can do an check the FD to see if it points to the deleted file:
# ls -lh /proc/518/fd/7
l-wx------ 1 root root 64 Nov 6 07:51 /proc/518/fd/7 -> /var/log/secure (deleted)
Now, this is the file that we can truncate.
echo > /proc/518/fd/7
You can verify now that the disk usage should be recovered since the file size is back to zero.
# lsof -nP +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
rsyslogd 518 root 7w REG 202,2 0 0 17708 /var/log/secure (deleted)
Now this is just a temporary solution to recover the disk usage from a deleted file. Eventually, you really need to stop/start or restart the process using the file to completely release it’s disk usage.
– masterkenneth