rootninja

Stale NFS file handle

If you’ve used NFSv2/3 for any length of time, you’ve probably run into the old “Stale NFS file handle” problem. Network hiccup, file system or disk crashes, whatever the circumstance the end result is the same. You can’t access the mounted directories and files.

The usual fix for this is to re-export or restart nfs services from the server side. You could try unmount -f to force the mount point to unmount or reboot the client.

It gets harder when multiple users are online. If someone has a home directory or a remote shell open using a directory under the nfs mount root, good luck fixing it on the fly.

List open files

A quick check with lsof will tell you what files are standing in your way.

lsof | grep home | grep google
chrome 1234 user 60u REG 8,5 1621567 72117 /home/user/.cache/google-chrome/Cache/index
chrome 1234 user 61u REG 8,5 1621568 72118 /home/user/.cache/google-chrome/Cache/data_0
chrome 1234 user 62u REG 8,5 1621569 72119 /home/user/.cache/google-chrome/Cache/data_1

It gets worse when two NFS file systems are mounted parent-child.

Example /etc/fstab:
server:/exports/home /home nfs _netdev,sync,rw 0 0
server:/exports/chrome /home/chrome nfs _netdev,sync,rw 0 0

chrome is mounted under home!

It seems like a catch-22. You can’t force /home to unmount because /home/chrome is still mounted and spits out nfs errors, but I can’t access chrome because it’s lost somewhere under /home!

Once I encounter the stale nfs problem, I have yet to find a solution other than to raise the white flag and reboot.

[root@server /]# wall get off teh system so i can reboot
Broadcast message from root (pts/0) (Mon Dec 32 00:00:01 2012):
get off teh system so i can reboot
[root@server /]# uptime
 11:39:36 up 398 days,  2:54,  2 users,  load average: 0.00, 0.00, 0.00
[root@server /]# echo sigh
sigh
[root@server /]# shutdown -t 60 -k 5