NFS TIPS FOR LARGE SP2 CONFIGURATIONS
POWERparallel Service Bulletin No. 8
Title: NFS TIPS FOR LARGE SP2 CONFIGURATIONS.
Date: August 7, 1995
Author: Bernie King-Smith
Reviewer: Cliff Carson
Reviewer: Rich Bloomer
Text: Tuning NFS for large SP2 configurations can aggravate
several pitfalls regarding the inherent design of NFS.
These pitfalls are mainly the number of concurrent NFS
requests to a NFS server, and tuning the NFS daemons ( nfsd
and biod ). All of these tips can be found in more detail
in the usual NFS documentation, but these tips are tailored
more specifically to SP2 configurations.
The first problem that we generally see in large SP2
configurations is where a single node is acting as the NFS
server for a large number of nodes. In this scenario, the
aggregate number of NFS requests can overwhelm the nfsd
daemons on the server, or you can not have enough daemons
configured.
When NFS is configured on a node, by default AIX configures
8 nfsd's and 6 biod's. The nfsd's are for handling NFS
requests between a client node and the server. The biod's
are for handling block I/O primarily in NFS writes from the
client node. As far as the server configuration, we are
primarily concerned with nfsd's. If you have a 64 node
configuration, and configured one NFS server for the other
63 nodes, then if all 63 client nodes made a NFS request at
the same time, you will need at least 63 nfsd's on the
server. If you only had 8 nfsd's as in the default
configuration, then 55 NFS requests would be rejected, and
they would time out before NFS retried the request. As you
can see, your average NFS response time will be very slow.
However, there is a limit as to how many nfsd's you can
configure before the amount of processing on behalf of the
NFS traffic overwhelms the server. Generally, when you
configure more than 48 nfsd's you start to see NFS
degradation depending on the characteristics of the NFS
traffic. It all depends on the size of the requests, and
mix of NFS operations. It also depends on the number of
processes on each node that generates the NFS requests. A
good place to start looking for indications that you are
generating too much NFS traffic to a server, is to examine
the nfsstat command output. If you see RPC timeouts or RPC
retransmits when checking on the client nodes, then you
probably are overwhelming the current number of nfsd's on
the server.
As far as the biod's, things are a little more complicated.
The biod's are used when writing to a NFS server from a
client node. This means that you configure enough biod's
on a client node for the NFS write traffic you expect.
However, NFS has a limit of using a maximum of 6 biod's on
a client node per NFS mounted file system. ( For more
explanation on why this is done, see a comprehensive book
on NFS design ) If you have only one process on a client
node writing to a single file system on the server, then
there is no reason to increase the number of biod's. Also,
if you only have one remote mounted NFS file system on each
client, you can only use 6 biod's no matter how many
processes are writing to the file system. Part of the
reason for limiting 6 biod's comes with NFS Version 2.0
which comes with all current versions of AIX.
In NFS Version 2.0, all NFS writes use a fully synchronous
4K write model. This means that when you write to a NFS
mounted file system, you can only send 4K at a time from
the client to the server. If you only had one I/O
outstanding at a time, this would be very slow. A fully
synchronous model says that the data must be written on the
disk at the server before the client can send the next 4K
block. To speed this up a bit, NFS says that you can stage
6 requests at the client per file system, but no more. If
you look a the time it takes to write a block on disk in
AIX, one block at a time is very slow. At 6 blocks at a
time we see an average time per block of 5 msec when
writing to a NFS file system across the High Performance
Switch using SP2 wide nodes. This times the 4K NFS writes
gives us the .8 MBPS performance we see using NFS over the
High Performance Switch.
If you have more than one NFS mounted file system on your
client nodes, you can configure up to 6 biod's per file
system. Once again, if the number of biod's gets too high,
performance will suffer just like when you configure too
many nfsd's on the server. Once again a general rule is
more than 48 biod's is where we start to see a degradation.
As far as nfsd's on the client nodes, the same rules apply
as for the server. However, we don't recommend that you
use the same number of nfsd's on the client nodes as the
servers when a large number of nodes all use the same
server ( i.e. 16 clients to 1 server ). If all nodes had
the same number of nfsd's, then each nfsd on the server can
potentially be sent 16 NFS requests, when they can only
handle one request at a time. Once again, in this
situation you can monitor nfsstat on both the client and
server nodes looking for RPC timeouts and retransmits.
If you already have a lot of nfsd's configured on the
server, the best solution is to split the NFS server duties
up across several nodes, and keep the potential number of
concurrent NFS request below 48. Configuring 48 nfsd's on
a server generally works well on the SP2. This is only a
general guideline, and as usual, your mileage may vary.
For more detailed information regarding tuning NFS, you
should look in the AIX 3.2.5 and 4.1 Performance Monitoring
and Tuning Guide, SC23-2365, as well as the AIX System
Management Guide: Communications and Networks, GC23-2487.