Forum Discussion

drbyrd_191910's avatar
drbyrd_191910
Icon for Nimbostratus rankNimbostratus
Jan 25, 2016

mystery files clogging up disk partition var_log

Version 11.5.2. Here's the message I'm getting every few minutes: 011d0004: Disk partition var_log has only 9% free emerg alertd[6319]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion

According to "df -h", 6.0G of 6.9 G is used in partition var_log. However, "du -h" shows a total of only 62M in the var/log directory. The output of "ls -lRH --block-size=M" matches the output of "du -h", only 62 M of files and directories.

How do I figure out what the offending files are? Thanks in advance...

[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log  df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg--db--vda-set.1.root
                      380M  211M  150M  59% /
/dev/mapper/vg--db--vda-set.1._config
                      3.0G   76M  2.8G   3% /config
/dev/mapper/vg--db--vda-set.1._usr
                      2.5G  1.8G  536M  78% /usr
/dev/mapper/vg--db--vda-set.1._var
                      3.0G  438M  2.4G  16% /var
/dev/mapper/vg--db--vda-dat.share.1
                       20G  224M   19G   2% /shared
/dev/mapper/vg--db--vda-dat.log.1
                    **6.9G  6.0G  638M  91% /var/log**
none                  5.6G  916K  5.6G   1% /dev/shm
none                  5.6G   45M  5.6G   1% /shared/rrd.1.2
none                  5.6G  7.8M  5.6G   1% /var/tmstat
none                  5.6G  1.6M  5.6G   1% /var/run
prompt                4.0M   28K  4.0M   1% /var/prompt
none                  5.6G     0  5.6G   0% /var/loipc

[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log  pwd
/var/log
[imanadmin@LTM-1:/S1-green-P:Active:In Sync] log  du -h /
 
8.0K    /var/log/ts/temp
8.0K    /var/log/ts/units
8.0K    /var/log/ts/archive/tmp
16K     /var/log/ts/archive
8.0K    /var/log/ts/unsent
144K    /var/log/ts
8.0K    /var/log/dosl7
524K    /var/log/avr
18M     /var/log/auditd
16K     /var/log/cbr
8.0K    /var/log/wa/push
8.0K    /var/log/wa/stats/system
8.0K    /var/log/wa/stats/perfmonitor
8.0K    /var/log/wa/stats/tmp
32K     /var/log/wa/stats
8.0K    /var/log/wa/access
56K     /var/log/wa
8.0K    /var/log/dwbl
17M     /var/log/iprepd
20K     /var/log/tomcat
26M     /var/log/monitors
396K    /var/log/sa
8.0K    /var/log/pem
20K     /var/log/lost+found
16K     /var/log/pam
20K     /var/log/httpd
**62M     /var/log**
application delivery
LTM

5 Replies

Sort By
  • drbyrd's avatar
    drbyrd
    Icon for Nimbostratus rankNimbostratus
    Jan 27, 2016

    Thanks Mike for the info. In this case, df and du are nowhere close to each other (6G versus 62M). Something is being measured by df that du can't see. I'm thinking the diskmonitor utility uses df or something similar- the % free in the syslog message always agrees with what df says. Still can't find the offending files.

     

  • drbyrd's avatar
    drbyrd
    Icon for Nimbostratus rankNimbostratus
    Jan 27, 2016

    I found the problem after finding this DevCentral page. I had deleted monitor log files before stopping the "monitor logging" on a few nodes. Basically there were files I couldn't see that were still being written to.

     

    I ran "lsof | grep deleted" to find the files, then I just started/stopped monitor logging for each of the affected nodes. That seemed to clear the unlinked files and I now have 97% free in var/log.