From: Alexei Starovoitov <ast@xxxxxxxxxx> v2->v3: - Adopted Sebastian's local_lock_cpu_slab(), but dropped gfpflags to avoid extra branch for performance reasons, and added local_unlock_cpu_slab() for symmetry. - Dropped local_lock_lockdep_start/end() pair and switched to per kmem_cache lockdep class on PREEMPT_RT to silence false positive when the same cpu/task acquires two local_lock-s. - Refactorred defer_free per Sebastian's suggestion - Fixed slab leak when it needs to be deactivated via irq_work and llist as Vlastimil proposed. Including defer_free_barrier(). - Use kmem_cache->offset for llist_node pointer when linking objects instead of zero offset, since whole object could be used for slabs with ctors and other cases. - Fixed "cnt = 1; goto redo;" issue. - Fixed slab leak in alloc_single_from_new_slab(). - Retested with slab_debug, RT, !RT, lockdep, kasan, slab_tiny - Added acks to patches 1-4 that should be good to go. v2: https://lore.kernel.org/bpf/20250709015303.8107-1-alexei.starovoitov@xxxxxxxxx/ v1->v2: Added more comments for this non-trivial logic and addressed earlier comments. In particular: - Introduce alloc_frozen_pages_nolock() to avoid refcnt race - alloc_pages_nolock() defaults to GFP_COMP - Support SLUB_TINY - Added more variants to stress tester to discover that kfree_nolock() can OOM, because deferred per-slab llist won't be serviced if kfree_nolock() gets unlucky long enough. Scraped previous approach and switched to global per-cpu llist with immediate irq_work_queue() to process all object sizes. - Reentrant kmalloc cannot deactivate_slab(). In v1 the node hint was downgraded to NUMA_NO_NODE before calling slab_alloc(). Realized it's not good enough. There are odd cases that can trigger deactivate. Rewrote this part. - Struggled with SLAB_NO_CMPXCHG. Thankfully Harry had a great suggestion: https://lore.kernel.org/bpf/aFvfr1KiNrLofavW@hyeyoo/ which was adopted. So slab_debug works now. - In v1 I had to s/local_lock_irqsave/local_lock_irqsave_check/ in a bunch of places in mm/slub.c to avoid lockdep false positives. Came up with much cleaner approach to silence invalid lockdep reports without sacrificing lockdep coverage. See local_lock_lockdep_start/end(). v1: https://lore.kernel.org/bpf/20250501032718.65476-1-alexei.starovoitov@xxxxxxxxx/ Alexei Starovoitov (6): locking/local_lock: Expose dep_map in local_trylock_t. locking/local_lock: Introduce local_lock_is_locked(). mm: Allow GFP_ACCOUNT to be used in alloc_pages_nolock(). mm: Introduce alloc_frozen_pages_nolock() slab: Introduce kmalloc_nolock() and kfree_nolock(). slab: Make slub local_trylock_t more precise for LOCKDEP include/linux/gfp.h | 2 +- include/linux/kasan.h | 13 +- include/linux/local_lock.h | 2 + include/linux/local_lock_internal.h | 16 +- include/linux/rtmutex.h | 9 + include/linux/slab.h | 4 + kernel/bpf/syscall.c | 2 +- kernel/locking/rtmutex_common.h | 9 - mm/Kconfig | 1 + mm/internal.h | 4 + mm/kasan/common.c | 5 +- mm/page_alloc.c | 54 ++-- mm/slab.h | 7 + mm/slab_common.c | 3 + mm/slub.c | 471 +++++++++++++++++++++++++--- 15 files changed, 513 insertions(+), 89 deletions(-) -- 2.47.1
