Table of Contents
In this article, I will take you through the usage of nfsstat and nfsiostat to troubleshoot NFS Performance in Linux. If you are using NFS Server in your Organization then you might be aware of the issue NFS Server could have due to the usage of existing Network Infrastructure. Any slight disruption in the network could affect the NFS Performance adversely.
We have two very important tools which can provide you the useful information about NFS mount points and its performance from both Server and Client end. nfsstat and nfsiostat are the two famous NFS dedicated tools which can be used for this purpose. I will explain more about the usage of nfsstat and nfsiostat command to troubleshoot the NFS Performance issues in below sections.
nfsstat and nfsiostat examples to troubleshoot NFS Performance in Linux
Also Read: 10 Popular nfsstat command examples in Linux for Professionals (Cheatsheet)
Troubleshooting Example 1: How to Check NFS Client Statistics using nfsstat command
If you want to check NFS Client statistics then you need to use nfsstat -c command as shown below. If you want to troubleshoot the network overload then you need to probably look at the retransmission rate(retrans). RPC Calls are synchronous and hence will block the client until it completes the request. If the client does not get a response from NFS Server, it will keep retransmitting the same request to a specific number of times until it finally quits.
Increase in retransmission can happen due to various reasons. It could be due to increase Server overloading which might delay the Server response, due to network congestion which further slow down the transmission, due to network card performance which might result in packet loss, due to disturbance in the network medium or may be due to faulty network devices between Server and Client. All these factors needs to be checked and verified.
[root@localhost ~]# nfsstat -c Client rpc stats: calls retrans authrefrsh 23 0 23 Client nfs v4: null read write commit open open_conf 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% open_noat open_dgrd close setattr fsinfo renew 0 0% 0 0% 0 0% 0 0% 3 13% 0 0% setclntid confirm lock lockt locku access 0 0% 0 0% 0 0% 0 0% 0 0% 2 9% getattr lookup lookup_root remove rename link 2 9% 3 13% 1 4% 0 0% 0 0% 0 0% symlink create pathconf statfs readlink readdir 0 0% 0 0% 2 9% 0 0% 0 0% 0 0% server_caps delegreturn getacl setacl fs_locations rel_lkowner 5 22% 0 0% 0 0% 0 0% 0 0% 0 0% secinfo exchange_id create_ses destroy_ses sequence get_lease_t 0 0% 0 0% 1 4% 1 4% 0 0% 0 0% reclaim_comp layoutget getdevinfo layoutcommit layoutreturn getdevlist 0 0% 1 4% 0 0% 0 0% 0 0% 0 0% (null) 1 4%
-c : Print only client-side statistics. Check nfsstat command Man Page for more info.
NOTE:
root user to run all the below commands.You can use any user with sudo access to run all these commands. For more information Please check Step by Step: How to Add User to Sudoers to provide sudo access to the User.Troubleshooting Example 2: How to Check NFS Server Statistics using nfsstat command
If you want to check NFS Server statistics then you need to use nfsstat -s command as shown below. You can use this command to troubleshoot the NFS Server latency issue. Here you need to look at the badcalls value if it goes beyond 0. badcalls value provides the information about number of rejected calls by the RPC Layer. More the value is more will be the latency.
root@localhost:~# nfsstat -s Server rpc stats: calls badcalls badfmt badauth badclnt 23 0 0 0 0 Server nfs v4: null compound 1 4% 22 95% Server nfs v4 operations: op0-unused op1-unused op2-future access close 0 0% 0 0% 0 0% 2 2% 0 0% commit create delegpurge delegreturn getattr 0 0% 0 0% 0 0% 0 0% 18 25% getfh link lock lockt locku 4 5% 0 0% 0 0% 0 0% 0 0% lookup lookup_root nverify open openattr 3 4% 0 0% 0 0% 0 0% 0 0% open_conf open_dgrd putfh putpubfh putrootfh 0 0% 0 0% 17 24% 0 0% 2 2% read readdir readlink remove rename 0 0% 0 0% 0 0% 0 0% 0 0% renew restorefh savefh secinfo setattr 0 0% 0 0% 0 0% 0 0% 0 0% setcltid setcltidconf verify write rellockowner 0 0% 0 0% 0 0% 0 0% 0 0% bc_ctl bind_conn exchange_id create_ses destroy_ses 0 0% 0 0% 1 1% 1 1% 0 0% free_stateid getdirdeleg getdevinfo getdevlist layoutcommit 0 0% 0 0% 0 0% 0 0% 0 0% layoutget layoutreturn secinfononam sequence set_ssv 0 0% 0 0% 1 1% 20 28% 0 0% test_stateid want_deleg destroy_clid reclaim_comp allocate 0 0% 0 0% 0 0% 1 1% 0 0% copy copy_notify deallocate ioadvise layouterror 0 0% 0 0% 0 0% 0 0% 0 0% layoutstats offloadcancel offloadstatus readplus seek 0 0% 0 0% 0 0% 0 0% 0 0% write_same 0 0%
-s : Print only server-side statistics. Check nfsstat command Man Page for more info.
Troubleshooting Example 3: How to check iostat for NFS mount point using nfsiostat command
If you want to check iostat for NFS mount point then you need to use nfsiostat command in Linux as shown below. nfsiostat command will get its input from /proc/self/mountstats and provide the information about the input output performance of NFS mount points. Below are the output parameters from nfsstat command output.
- op/s : This is the number of operations per second.
- rpc bklog : This is the length of the backlog queue.
- kB/s: This is the number of kB written/read per second.
- kB/op : This is the number of kB written/read per each operation.
- retrans : This is the number of retransmissions.
- avg RTT (ms) : This is the duration from the time that client's kernel sends the RPC request until the time it receives the reply.
- avg exe (ms) :This is the duration from the time that NFS client does the RPC request to its kernel until the RPC request is completed, this includes the RTT time above. More can be checked on nfsiostat Man Page.
[root@localhost ~]# nfsiostat 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000
Troubleshooting Example 4: How to Display Attribute Cache Statistics using nfsiostat command
If you want to display attribute cache statistics then you need to use nfsiostat -a command as shown below.
[root@localhost ~]# nfsiostat -a 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 getattr: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.422 0 (0.0%) 5.500 11.000 access: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.355 0 (0.0%) 0.000 0.000 1 inode revalidations, hitting in cache -100.00% of the time 0 open operations (mandatory GETATTR requests)
-a : displays statistics related to the attribute cache. Check nfsiostat Man Page for more info.
Troubleshooting Example 5: How to Sort NFS Mount Point by ops/sec using nfsiostat command
If you want to sort NFS mount point by ops/sec then you need to use nfsiostat -s command as shown below.
[root@localhost ~]# nfsiostat -s 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 1.26 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000
-s : Sort NFS mount points by ops/second. Check nfsiostat Man Page for more info.
Troubleshooting Example 6: How to display page cache related Statistics using nfsiostat command
If you want to display all the page cache related statistics then you need to use nfsiostat -p command as shown below.
[root@localhost ~]# nfsiostat -p 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 0 nfs_readpage() calls read 0 pages 0 nfs_readpages() calls read 0 pages 0 nfs_updatepage() calls 0 nfs_writepage() calls wrote 0 pages 0 nfs_writepages() calls wrote 0 pages
-p : displays statistics related to the page cache. Check nfsiostat Man Page for more info.
Troubleshooting Example 7: How to Display Statistics related to directory operations
If you want to display directory related statistics then you need to use nfsiostat -d command as shown below.
[root@localhost ~]# nfsiostat -d 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 lookup: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.488 0 (0.0%) 0.000 0.000 readdir: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 0 open operations (pathname lookups) 0 dentry revalidates and 1 vfs lookup requests resulted in 1 LOOKUPs on the wire 0 vfs getdents calls resulted in 0 READDIRs on the wire
-d : displays statistics related to directory operations. Check nfsiostat Man Page for more info.
Troubleshooting Example 8: How to display nfsiostat output at a certain Interval
If you want to display nfsiostat command output 2 times in every 3 seconds then you need to use nfsiostat 3 2 command as shown below. You can change the interval and number of output that you require as per your requirements.
[root@localhost ~]# nfsiostat 3 2 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.00 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000
Troubleshooting Example 9: How to Display nfsiostat output for a Specific Partition
Above example will display the nfsiostat output for all the mounted NFS partition but if you want to see the statistics of a specific partition then you need to use below nfsiostat command. In this example, we are checking stats of /nfsclient mount point using nfsiostat 5 2 /nfsclient command. This command will display output 2 times with an interval of 5 seconds.
[root@localhost ~]# nfsiostat 5 2 /nfsclient 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.02 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 192.168.0.108:/mnt/nfs mounted on /nfsclient: op/s rpc bklog 0.00 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.000 0.000 0.000 0 (0.0%) 0.000 0.000
Troubleshooting Example 10: How to check RPC Call related information using nfsstat command
If you want to only check the RPC Call related information and ignore others stats then you can simple use the nfsstat -r command as shown below.
[root@localhost ~]# nfsstat -r Client rpc stats: calls retrans authrefrsh 331 2 331
-r : Print only RPC statistics.
Popular Recommendations:-
How to Install VLC Media Player in RHEL / CentOS 8 Using 6 Easy Steps
52 Useful cut command in Linux/Unix with Examples for Beginners
How to Delete Elasticsearch Unassigned Shards in 4 Easy Steps
How to Delete Elasticsearch Red Status Indices in 3 Easy Steps
Understanding Kafka Console Producer and Consumer in 10 Easy Steps
Popular Apache Kafka Architecture Explained Using 4 Basic Components
17 Useful Apache Kafka Examples on Linux (RedHat/CentOS 7/8)