Saturday, September 4, 2010

du vs df - The Open File Descriptor Problem

You are more likely to encounter a situation in which df on a filesystem reports 100% whereas the actual usage would be vey less(reported by du)

eg)
(server1:/)# df -k /usrlocaltrpt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dg1/usrlocaltrpt
                     2031711 1970765       0   100%    /usrlocaltrpt

(server1:/)# du -sh /usrlocaltrpt
 254M   /usrlocaltrpt

Open file descriptor is main causes of such wrong information.
For example if file called /usrlocaltrpt/temp.log is open by an application OR by a user and same file is deleted or somehow not accessible, both df and du reports different output

This situation could be solved in 2 ways,
1. Unmount the fs and remount
2. Find the process/application responsible and restart or kill them.

First way is simple but not feasible for some critical filesystems.

Second way needs some work to be done to identify the open files and the process associated with them.
 ->There is a 3rd party tool called as 'lsof' which lists the open files
 ->Scan the proc diretory for the files open and search manually
 ->user fuser -cu and find the process

eg) to find process from proc directory

(server1:/proc)# for i in *; do pfiles $i | grep -i trpt; done
      /usr/local/trpt/Encore.log
      /usr/local/trpt/Encore.log
      /usr/local/trpt/Encore.log
      /usr/local/trpt/modem.log
      /usr/local/trpt/params.xml

or

(server1:/usrlocaltrpt)# fuser -cu /var/
/usrlocaltrpt/:     1519c(root)   26639o(root)   29155c(root)   21540o(root)    6184o(ipde)    1293o(noaccess)    1212o(root)    1091c(root)    1039o(root)     986co(root)     943o(root)     939o(root)     926o(root)     788co(smmsp)     783o(root)     745o(root)     743o(root)     742o(root)     731o(root)     683c(daemon)     657co(root)     185o(root)       7o(root)
(server1:/usrlocaltrpt)#

use a loop with the above pid's and find the associated process. Check for defunct process or a process that is opening a file.

No comments:

Post a Comment