webrev: http://cr.illumos.org/~webrev/marcel/il-3783-rpcmod-flowcontrol/
In case the NFS clients are fast enough and they are able to send more requests than the NFS server is able to handle (for various reasons) the number of queued requests in rpcmod could grow and grow without any limit until the NFS server's memory is exhausted.
This fix implements two conditions to constrain the queued requests. The first condition is based on number of requests queued. The second condition is based on total size occupied by queued requests. For more details about the conditions please read the comment for the svc_flowcontrol() function in the diffs.
Since this fix is in generic rpcmod layer, it applies to other rpcmod pools (like nlm and nfscbd) too.
The flow control is not implemented for RDMA transport, so the RDMA transports will behave unchanged. For TCP, once the flow control turns on, the TCP window will close and the NFS client won't be able to send more requests. For UDP, incoming requests are just discarded once the flow control is activated.
New tunable svc_flowcontrol_disable is implemented too. Once it is set to non-zero it should disable this new flow control functionality.
The fix contains some code and comments cleanup in the rpcmod module.

This fix was thoroughly tested about two years ago in a lab and also by several
customers who were suffered by it.

This fix is included in the NexentaStor since version 3.1.4.2, so the fix is
permanently tested by all NexentaStor NFS users for very long time now.  No
issues/regressions were reported with this fix.