diff options
| author | Frederic Weisbecker <frederic@kernel.org> | 2021-05-27 13:34:41 +0200 |
|---|---|---|
| committer | Peter Zijlstra <peterz@infradead.org> | 2021-05-31 10:14:49 +0200 |
| commit | f268c3737ecaefcfeecfb4cb5e44958a8976f067 (patch) | |
| tree | 895c3ffc8d716b3d00851b569eb183f73a95d9fc /include/linux | |
| parent | 02da26ad5ed6ea8680e5d01f20661439611ed776 (diff) | |
tick/nohz: Only check for RCU deferred wakeup on user/guest entry when needed
Checking for and processing RCU-nocb deferred wakeup upon user/guest
entry is only relevant when nohz_full runs on the local CPU, otherwise
the periodic tick should take care of it.
Make sure we don't needlessly pollute these fast-paths as a -3%
performance regression on a will-it-scale.per_process_ops has been
reported so far.
Fixes: 47b8ff194c1f (entry: Explicitly flush pending rcuog wakeup before last rescheduling point)
Fixes: 4ae7dc97f726 (entry/kvm: Explicitly flush pending rcuog wakeup before last rescheduling point)
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lkml.kernel.org/r/20210527113441.465489-1-frederic@kernel.org
Diffstat (limited to 'include/linux')
| -rw-r--r-- | include/linux/entry-kvm.h | 3 | ||||
| -rw-r--r-- | include/linux/tick.h | 7 |
2 files changed, 9 insertions, 1 deletions
diff --git a/include/linux/entry-kvm.h b/include/linux/entry-kvm.h index 8b2b1d68b954..136b8d97d8c0 100644 --- a/include/linux/entry-kvm.h +++ b/include/linux/entry-kvm.h @@ -3,6 +3,7 @@ #define __LINUX_ENTRYKVM_H #include <linux/entry-common.h> +#include <linux/tick.h> /* Transfer to guest mode work */ #ifdef CONFIG_KVM_XFER_TO_GUEST_WORK @@ -57,7 +58,7 @@ int xfer_to_guest_mode_handle_work(struct kvm_vcpu *vcpu); static inline void xfer_to_guest_mode_prepare(void) { lockdep_assert_irqs_disabled(); - rcu_nocb_flush_deferred_wakeup(); + tick_nohz_user_enter_prepare(); } /** diff --git a/include/linux/tick.h b/include/linux/tick.h index 7340613c7eff..1a0ff88fa107 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -11,6 +11,7 @@ #include <linux/context_tracking_state.h> #include <linux/cpumask.h> #include <linux/sched.h> +#include <linux/rcupdate.h> #ifdef CONFIG_GENERIC_CLOCKEVENTS extern void __init tick_init(void); @@ -300,4 +301,10 @@ static inline void tick_nohz_task_switch(void) __tick_nohz_task_switch(); } +static inline void tick_nohz_user_enter_prepare(void) +{ + if (tick_nohz_full_cpu(smp_processor_id())) + rcu_nocb_flush_deferred_wakeup(); +} + #endif |