summaryrefslogtreecommitdiff
path: root/include
diff options
context:
space:
mode:
authorVlastimil Babka <vbabka@suse.cz>2021-05-08 02:28:02 +0200
committerVlastimil Babka <vbabka@suse.cz>2021-09-04 01:12:21 +0200
commit0b303fb402862dcb7948eeeed2439bd8c99948b5 (patch)
treecf582cf0547f8120a341b4e56cda547e0fac0d1d /include
parente500059ba55268e1c5212632e4f21e45f54dc6d9 (diff)
mm, slub: do initial checks in ___slab_alloc() with irqs enabled
As another step of shortening irq disabled sections in ___slab_alloc(), delay disabling irqs until we pass the initial checks if there is a cached percpu slab and it's suitable for our allocation. Now we have to recheck c->page after actually disabling irqs as an allocation in irq handler might have replaced it. Because we call pfmemalloc_match() as one of the checks, we might hit VM_BUG_ON_PAGE(!PageSlab(page)) in PageSlabPfmemalloc in case we get interrupted and the page is freed. Thus introduce a pfmemalloc_match_unsafe() variant that lacks the PageSlab check. Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Mel Gorman <mgorman@techsingularity.net>
Diffstat (limited to 'include')
-rw-r--r--include/linux/page-flags.h9
1 files changed, 9 insertions, 0 deletions
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 5922031ffab6..7fda4fb85bdc 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -815,6 +815,15 @@ static inline int PageSlabPfmemalloc(struct page *page)
return PageActive(page);
}
+/*
+ * A version of PageSlabPfmemalloc() for opportunistic checks where the page
+ * might have been freed under us and not be a PageSlab anymore.
+ */
+static inline int __PageSlabPfmemalloc(struct page *page)
+{
+ return PageActive(page);
+}
+
static inline void SetPageSlabPfmemalloc(struct page *page)
{
VM_BUG_ON_PAGE(!PageSlab(page), page);