pm24.git/include/linux/sunrpc, branch v3.19-rc7

sunrpc: convert to lockless lookup of queued server threads

2014-12-09T16:22:22Z

Testing has shown that the pool->sp_lock can be a bottleneck on a busy server. Every time data is received on a socket, the server must take that lock in order to dequeue a thread from the sp_threads list. Address this problem by eliminating the sp_threads list (which contains threads that are currently idle) and replacing it with a RQ_BUSY flag in svc_rqst. This allows us to walk the sp_all_threads list under the rcu_read_lock and find a suitable thread for the xprt by doing a test_and_set_bit. Note that we do still have a potential atomicity problem however with this approach. We don't want svc_xprt_do_enqueue to set the rqst->rq_xprt pointer unless a test_and_set_bit of RQ_BUSY returned zero (which indicates that the thread was idle). But, by the time we check that, the bit could be flipped by a waking thread. To address this, we acquire a new per-rqst spinlock (rq_lock) and take that before doing the test_and_set_bit. If that returns false, then we can set rq_xprt and drop the spinlock. Then, when the thread wakes up, it must set the bit under the same spinlock and can trust that if it was already set then the rq_xprt is also properly set. With this scheme, the case where we have an idle thread no longer needs to take the highly contended pool->sp_lock at all, and that removes the bottleneck. That still leaves one issue: What of the case where we walk the whole sp_all_threads list and don't find an idle thread? Because the search is lockess, it's possible for the queueing to race with a thread that is going to sleep. To address that, we queue the xprt and then search again. If we find an idle thread at that point, we can't attach the xprt to it directly since that might race with a different thread waking up and finding it. All we can do is wake the idle thread back up and let it attempt to find the now-queued xprt. Signed-off-by: Jeff Layton Tested-by: Chris Worley Signed-off-by: J. Bruce Fields

sunrpc: fix potential races in pool_stats collection

2014-12-09T16:22:22Z

In a later patch, we'll be removing some spinlocking around the socket and thread queueing code in order to fix some contention problems. At that point, the stats counters will no longer be protected by the sp_lock. Change the counters to atomic_long_t fields, except for the "sockets_queued" counter which will still be manipulated under a spinlock. Signed-off-by: Jeff Layton Tested-by: Chris Worley Signed-off-by: J. Bruce Fields

sunrpc: add a rcu_head to svc_rqst and use kfree_rcu to free it

2014-12-09T16:22:22Z

...also make the manipulation of sp_all_threads list use RCU-friendly functions. Signed-off-by: Jeff Layton Tested-by: Chris Worley Signed-off-by: J. Bruce Fields

sunrpc: convert sp_task_pending flag to use atomic bitops

2014-12-09T16:22:21Z

In a later patch, we'll want to be able to handle this flag without holding the sp_lock. Change this field to an unsigned long flags field, and declare a new flag in it that can be managed with atomic bitops. Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: move rq_cachetype field to better optimize space

2014-12-09T16:22:21Z

There are a couple of holes in the svc_rqst field on x86_64. Move the rq_cachetype to a different location to eliminate both of them. Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: move rq_splice_ok flag into rq_flags

2014-12-09T16:22:21Z

Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: move rq_dropme flag into rq_flags

2014-12-09T16:22:20Z

Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: move rq_usedeferral flag to rq_flags

2014-12-09T16:22:20Z

Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: move rq_local field to rq_flags

2014-12-09T16:21:21Z

Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields

sunrpc: add a generic rq_flags field to svc_rqst and move rq_secure to it

2014-12-09T16:21:20Z

In a later patch, we're going to need some atomic bit flags. Since that field will need to be an unsigned long, we mitigate that space consumption by migrating some other bitflags to the new field. Start with the rq_secure flag. Signed-off-by: Jeff Layton Signed-off-by: J. Bruce Fields