On Fri, Sep 12, 2025 at 7:53 PM Lorenzo Stoakes
<lorenzo.stoakes@xxxxxxxxxx> wrote:
>
> On Fri, Sep 12, 2025 at 04:28:46PM +0800, Yafang Shao wrote:
> > On Thu, Sep 11, 2025 at 10:34 PM Lorenzo Stoakes
> > <lorenzo.stoakes@xxxxxxxxxx> wrote:
> > >
> > > On Wed, Sep 10, 2025 at 10:44:39AM +0800, Yafang Shao wrote:
> > > > This patch introduces a new BPF struct_ops called bpf_thp_ops for dynamic
> > > > THP tuning. It includes a hook bpf_hook_thp_get_order(), allowing BPF
> > > > programs to influence THP order selection based on factors such as:
> > > > - Workload identity
> > > > For example, workloads running in specific containers or cgroups.
> > > > - Allocation context
> > > > Whether the allocation occurs during a page fault, khugepaged, swap or
> > > > other paths.
> > > > - VMA's memory advice settings
> > > > MADV_HUGEPAGE or MADV_NOHUGEPAGE
> > > > - Memory pressure
> > > > PSI system data or associated cgroup PSI metrics
> > > >
> > > > The kernel API of this new BPF hook is as follows,
> > > >
> > > > /**
> > > > * @thp_order_fn_t: Get the suggested THP orders from a BPF program for allocation
> > > > * @vma: vm_area_struct associated with the THP allocation
> > > > * @vma_type: The VMA type, such as BPF_THP_VM_HUGEPAGE if VM_HUGEPAGE is set
> > > > * BPF_THP_VM_NOHUGEPAGE if VM_NOHUGEPAGE is set, or BPF_THP_VM_NONE if
> > > > * neither is set.
> > > > * @tva_type: TVA type for current @vma
> > > > * @orders: Bitmask of requested THP orders for this allocation
> > > > * - PMD-mapped allocation if PMD_ORDER is set
> > > > * - mTHP allocation otherwise
> > > > *
> > > > * Return: The suggested THP order from the BPF program for allocation. It will
> > > > * not exceed the highest requested order in @orders. Return -1 to
> > > > * indicate that the original requested @orders should remain unchanged.
> > > > */
> > > > typedef int thp_order_fn_t(struct vm_area_struct *vma,
> > > > enum bpf_thp_vma_type vma_type,
> > > > enum tva_type tva_type,
> > > > unsigned long orders);
> > > >
> > > > Only a single BPF program can be attached at any given time, though it can
> > > > be dynamically updated to adjust the policy. The implementation supports
> > > > anonymous THP, shmem THP, and mTHP, with future extensions planned for
> > > > file-backed THP.
> > > >
> > > > This functionality is only active when system-wide THP is configured to
> > > > madvise or always mode. It remains disabled in never mode. Additionally,
> > > > if THP is explicitly disabled for a specific task via prctl(), this BPF
> > > > functionality will also be unavailable for that task.
> > > >
> > > > This feature requires CONFIG_BPF_GET_THP_ORDER (marked EXPERIMENTAL) to be
> > > > enabled. Note that this capability is currently unstable and may undergo
> > > > significant changes—including potential removal—in future kernel versions.
> > >
> > > Thanks for highlighting.
> > >
> > > >
> > > > Suggested-by: David Hildenbrand <david@xxxxxxxxxx>
> > > > Suggested-by: Lorenzo Stoakes <lorenzo.stoakes@xxxxxxxxxx>
> > > > Signed-off-by: Yafang Shao <laoar.shao@xxxxxxxxx>
> > > > ---
> > > > MAINTAINERS | 1 +
> > > > include/linux/huge_mm.h | 26 ++++-
> > > > mm/Kconfig | 12 ++
> > > > mm/Makefile | 1 +
> > > > mm/huge_memory_bpf.c | 243 ++++++++++++++++++++++++++++++++++++++++
> > > > 5 files changed, 280 insertions(+), 3 deletions(-)
> > > > create mode 100644 mm/huge_memory_bpf.c
> > > >
> > > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > > index 8fef05bc2224..d055a3c95300 100644
> > > > --- a/MAINTAINERS
> > > > +++ b/MAINTAINERS
> > > > @@ -16252,6 +16252,7 @@ F: include/linux/huge_mm.h
> > > > F: include/linux/khugepaged.h
> > > > F: include/trace/events/huge_memory.h
> > > > F: mm/huge_memory.c
> > > > +F: mm/huge_memory_bpf.c
> > >
> > > THanks!
> > >
> > > > F: mm/khugepaged.c
> > > > F: mm/mm_slot.h
> > > > F: tools/testing/selftests/mm/khugepaged.c
> > > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> > > > index 23f124493c47..f72a5fd04e4f 100644
> > > > --- a/include/linux/huge_mm.h
> > > > +++ b/include/linux/huge_mm.h
> > > > @@ -56,6 +56,7 @@ enum transparent_hugepage_flag {
> > > > TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
> > > > TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG,
> > > > TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG,
> > > > + TRANSPARENT_HUGEPAGE_BPF_ATTACHED, /* BPF prog is attached */
> > > > };
> > > >
> > > > struct kobject;
> > > > @@ -270,6 +271,19 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
> > > > enum tva_type type,
> > > > unsigned long orders);
> > > >
> > > > +#ifdef CONFIG_BPF_GET_THP_ORDER
> > > > +unsigned long
> > > > +bpf_hook_thp_get_orders(struct vm_area_struct *vma, vm_flags_t vma_flags,
> > > > + enum tva_type type, unsigned long orders);
> > >
> > > Thanks for renaming!
> > >
> > > > +#else
> > > > +static inline unsigned long
> > > > +bpf_hook_thp_get_orders(struct vm_area_struct *vma, vm_flags_t vma_flags,
> > > > + enum tva_type tva_flags, unsigned long orders)
> > > > +{
> > > > + return orders;
> > > > +}
> > > > +#endif
> > > > +
> > > > /**
> > > > * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma
> > > > * @vma: the vm area to check
> > > > @@ -291,6 +305,12 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
> > > > enum tva_type type,
> > > > unsigned long orders)
> > > > {
> > > > + unsigned long bpf_orders;
> > > > +
> > > > + bpf_orders = bpf_hook_thp_get_orders(vma, vm_flags, type, orders);
> > > > + if (!bpf_orders)
> > > > + return 0;
> > >
> > > I think it'd be easier to just do:
> > >
> > > /* The BPF-specified order overrides which order is selected. */
> > > orders &= bpf_hook_thp_get_orders(vma, vm_flags, type, orders);
> > > if (!orders)
> > > return 0;
> >
> > good suggestion!
>
> Thanks, though this does come back to 'are we masking on orders' or not.
>
> Obviously this is predicated on that being the case.
>
> > > > struct thpsize {
> > > > diff --git a/mm/Kconfig b/mm/Kconfig
> > > > index d1ed839ca710..4d89d2158f10 100644
> > > > --- a/mm/Kconfig
> > > > +++ b/mm/Kconfig
> > > > @@ -896,6 +896,18 @@ config NO_PAGE_MAPCOUNT
> > > >
> > > > EXPERIMENTAL because the impact of some changes is still unclear.
> > > >
> > > > +config BPF_GET_THP_ORDER
> > >
> > > Yeah, I think we maybe need to sledgehammer this as already Lance was confused
> > > as to the permenancy of this, and I feel that users might be too, even with the
> > > '(EXPERIMENTAL)' bit.
> > >
> > > So maybe
> > >
> > > config BPF_GET_THP_ORDER_EXPERIMENTAL
> > >
> > > Just to hammer it home?
> >
> > ack
>
> Thanks!
>
> >
> > >
> > > > + bool "BPF-based THP order selection (EXPERIMENTAL)"
> > > > + depends on TRANSPARENT_HUGEPAGE && BPF_SYSCALL
> > > > +
> > > > + help
> > > > + Enable dynamic THP order selection using BPF programs. This
> > > > + experimental feature allows custom BPF logic to determine optimal
> > > > + transparent hugepage allocation sizes at runtime.
> > > > +
> > > > + WARNING: This feature is unstable and may change in future kernel
> > > > + versions.
> > > > +
> > > > endif # TRANSPARENT_HUGEPAGE
> > > >
> > > > # simple helper to make the code a bit easier to read
> > > > diff --git a/mm/Makefile b/mm/Makefile
> > > > index 21abb3353550..f180332f2ad0 100644
> > > > --- a/mm/Makefile
> > > > +++ b/mm/Makefile
> > > > @@ -99,6 +99,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o
> > > > obj-$(CONFIG_NUMA) += memory-tiers.o
> > > > obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
> > > > obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
> > > > +obj-$(CONFIG_BPF_GET_THP_ORDER) += huge_memory_bpf.o
> > > > obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
> > > > obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
> > > > obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
> > > > diff --git a/mm/huge_memory_bpf.c b/mm/huge_memory_bpf.c
> > > > new file mode 100644
> > > > index 000000000000..525ee22ab598
> > > > --- /dev/null
> > > > +++ b/mm/huge_memory_bpf.c
> > > > @@ -0,0 +1,243 @@
> > > > +// SPDX-License-Identifier: GPL-2.0
> > > > +/*
> > > > + * BPF-based THP policy management
> > > > + *
> > > > + * Author: Yafang Shao <laoar.shao@xxxxxxxxx>
> > > > + */
> > > > +
> > > > +#include <linux/bpf.h>
> > > > +#include <linux/btf.h>
> > > > +#include <linux/huge_mm.h>
> > > > +#include <linux/khugepaged.h>
> > > > +
> > > > +enum bpf_thp_vma_type {
> > > > + BPF_THP_VM_NONE = 0,
> > > > + BPF_THP_VM_HUGEPAGE, /* VM_HUGEPAGE */
> > > > + BPF_THP_VM_NOHUGEPAGE, /* VM_NOHUGEPAGE */
> > > > +};
> > >
> > > I'm really not so sure how useful this is - can't a user just ascertain this
> > > from the VMA flags themselves?
> >
> > I assume you are referring to checking flags from vma->vm_flags.
> > There is an exception where we cannot use vma->vm_flags: in
> > hugepage_madvise(), which calls khugepaged_enter_vma(vma, *vm_flags).
> >
> > At this point, the VM_HUGEPAGE flag has not been set in vma->vm_flags
> > yet. Therefore, we must pass the separate *vm_flags variable.
> > Perhaps we can simplify the logic with the following change?
>
> Ugh god.
>
> I guess this is the workaround for the vm_flags thing right.
>
> >
> > diff --git a/mm/madvise.c b/mm/madvise.c
> > index 35ed4ab0d7c5..5755de80a4d7 100644
> > --- a/mm/madvise.c
> > +++ b/mm/madvise.c
> > @@ -1425,6 +1425,8 @@ static int madvise_vma_behavior(struct
> > madvise_behavior *madv_behavior)
> > VM_WARN_ON_ONCE(madv_behavior->lock_mode != MADVISE_MMAP_WRITE_LOCK);
> >
> > error = madvise_update_vma(new_flags, madv_behavior);
> > + if (new_flags & VM_HUGEPAGE)
> > + khugepaged_enter_vma(vma);
>
> Hm ok, that's not such a bad idea, though ofc this should be something like:
>
> if (!error && (new_flags & VM_HUGEPAGE))
> khugepaged_enter_vma(vma);
ack
>
> And obviously dropping this khugepaged_enter_vma() from hugepage_madvise().
Thanks for the reminder.
--
Regards
Yafang
