On Fri, Sep 12, 2025 at 04:28:46PM +0800, Yafang Shao wrote:
> On Thu, Sep 11, 2025 at 10:34 PM Lorenzo Stoakes
> <lorenzo.stoakes@xxxxxxxxxx> wrote:
> >
> > On Wed, Sep 10, 2025 at 10:44:39AM +0800, Yafang Shao wrote:
> > > This patch introduces a new BPF struct_ops called bpf_thp_ops for dynamic
> > > THP tuning. It includes a hook bpf_hook_thp_get_order(), allowing BPF
> > > programs to influence THP order selection based on factors such as:
> > > - Workload identity
> > > For example, workloads running in specific containers or cgroups.
> > > - Allocation context
> > > Whether the allocation occurs during a page fault, khugepaged, swap or
> > > other paths.
> > > - VMA's memory advice settings
> > > MADV_HUGEPAGE or MADV_NOHUGEPAGE
> > > - Memory pressure
> > > PSI system data or associated cgroup PSI metrics
> > >
> > > The kernel API of this new BPF hook is as follows,
> > >
> > > /**
> > > * @thp_order_fn_t: Get the suggested THP orders from a BPF program for allocation
> > > * @vma: vm_area_struct associated with the THP allocation
> > > * @vma_type: The VMA type, such as BPF_THP_VM_HUGEPAGE if VM_HUGEPAGE is set
> > > * BPF_THP_VM_NOHUGEPAGE if VM_NOHUGEPAGE is set, or BPF_THP_VM_NONE if
> > > * neither is set.
> > > * @tva_type: TVA type for current @vma
> > > * @orders: Bitmask of requested THP orders for this allocation
> > > * - PMD-mapped allocation if PMD_ORDER is set
> > > * - mTHP allocation otherwise
> > > *
> > > * Return: The suggested THP order from the BPF program for allocation. It will
> > > * not exceed the highest requested order in @orders. Return -1 to
> > > * indicate that the original requested @orders should remain unchanged.
> > > */
> > > typedef int thp_order_fn_t(struct vm_area_struct *vma,
> > > enum bpf_thp_vma_type vma_type,
> > > enum tva_type tva_type,
> > > unsigned long orders);
> > >
> > > Only a single BPF program can be attached at any given time, though it can
> > > be dynamically updated to adjust the policy. The implementation supports
> > > anonymous THP, shmem THP, and mTHP, with future extensions planned for
> > > file-backed THP.
> > >
> > > This functionality is only active when system-wide THP is configured to
> > > madvise or always mode. It remains disabled in never mode. Additionally,
> > > if THP is explicitly disabled for a specific task via prctl(), this BPF
> > > functionality will also be unavailable for that task.
> > >
> > > This feature requires CONFIG_BPF_GET_THP_ORDER (marked EXPERIMENTAL) to be
> > > enabled. Note that this capability is currently unstable and may undergo
> > > significant changes—including potential removal—in future kernel versions.
> >
> > Thanks for highlighting.
> >
> > >
> > > Suggested-by: David Hildenbrand <david@xxxxxxxxxx>
> > > Suggested-by: Lorenzo Stoakes <lorenzo.stoakes@xxxxxxxxxx>
> > > Signed-off-by: Yafang Shao <laoar.shao@xxxxxxxxx>
> > > ---
> > > MAINTAINERS | 1 +
> > > include/linux/huge_mm.h | 26 ++++-
> > > mm/Kconfig | 12 ++
> > > mm/Makefile | 1 +
> > > mm/huge_memory_bpf.c | 243 ++++++++++++++++++++++++++++++++++++++++
> > > 5 files changed, 280 insertions(+), 3 deletions(-)
> > > create mode 100644 mm/huge_memory_bpf.c
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index 8fef05bc2224..d055a3c95300 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -16252,6 +16252,7 @@ F: include/linux/huge_mm.h
> > > F: include/linux/khugepaged.h
> > > F: include/trace/events/huge_memory.h
> > > F: mm/huge_memory.c
> > > +F: mm/huge_memory_bpf.c
> >
> > THanks!
> >
> > > F: mm/khugepaged.c
> > > F: mm/mm_slot.h
> > > F: tools/testing/selftests/mm/khugepaged.c
> > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> > > index 23f124493c47..f72a5fd04e4f 100644
> > > --- a/include/linux/huge_mm.h
> > > +++ b/include/linux/huge_mm.h
> > > @@ -56,6 +56,7 @@ enum transparent_hugepage_flag {
> > > TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
> > > TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG,
> > > TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG,
> > > + TRANSPARENT_HUGEPAGE_BPF_ATTACHED, /* BPF prog is attached */
> > > };
> > >
> > > struct kobject;
> > > @@ -270,6 +271,19 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
> > > enum tva_type type,
> > > unsigned long orders);
> > >
> > > +#ifdef CONFIG_BPF_GET_THP_ORDER
> > > +unsigned long
> > > +bpf_hook_thp_get_orders(struct vm_area_struct *vma, vm_flags_t vma_flags,
> > > + enum tva_type type, unsigned long orders);
> >
> > Thanks for renaming!
> >
> > > +#else
> > > +static inline unsigned long
> > > +bpf_hook_thp_get_orders(struct vm_area_struct *vma, vm_flags_t vma_flags,
> > > + enum tva_type tva_flags, unsigned long orders)
> > > +{
> > > + return orders;
> > > +}
> > > +#endif
> > > +
> > > /**
> > > * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma
> > > * @vma: the vm area to check
> > > @@ -291,6 +305,12 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
> > > enum tva_type type,
> > > unsigned long orders)
> > > {
> > > + unsigned long bpf_orders;
> > > +
> > > + bpf_orders = bpf_hook_thp_get_orders(vma, vm_flags, type, orders);
> > > + if (!bpf_orders)
> > > + return 0;
> >
> > I think it'd be easier to just do:
> >
> > /* The BPF-specified order overrides which order is selected. */
> > orders &= bpf_hook_thp_get_orders(vma, vm_flags, type, orders);
> > if (!orders)
> > return 0;
>
> good suggestion!
Thanks, though this does come back to 'are we masking on orders' or not.
Obviously this is predicated on that being the case.
> > > struct thpsize {
> > > diff --git a/mm/Kconfig b/mm/Kconfig
> > > index d1ed839ca710..4d89d2158f10 100644
> > > --- a/mm/Kconfig
> > > +++ b/mm/Kconfig
> > > @@ -896,6 +896,18 @@ config NO_PAGE_MAPCOUNT
> > >
> > > EXPERIMENTAL because the impact of some changes is still unclear.
> > >
> > > +config BPF_GET_THP_ORDER
> >
> > Yeah, I think we maybe need to sledgehammer this as already Lance was confused
> > as to the permenancy of this, and I feel that users might be too, even with the
> > '(EXPERIMENTAL)' bit.
> >
> > So maybe
> >
> > config BPF_GET_THP_ORDER_EXPERIMENTAL
> >
> > Just to hammer it home?
>
> ack
Thanks!
>
> >
> > > + bool "BPF-based THP order selection (EXPERIMENTAL)"
> > > + depends on TRANSPARENT_HUGEPAGE && BPF_SYSCALL
> > > +
> > > + help
> > > + Enable dynamic THP order selection using BPF programs. This
> > > + experimental feature allows custom BPF logic to determine optimal
> > > + transparent hugepage allocation sizes at runtime.
> > > +
> > > + WARNING: This feature is unstable and may change in future kernel
> > > + versions.
> > > +
> > > endif # TRANSPARENT_HUGEPAGE
> > >
> > > # simple helper to make the code a bit easier to read
> > > diff --git a/mm/Makefile b/mm/Makefile
> > > index 21abb3353550..f180332f2ad0 100644
> > > --- a/mm/Makefile
> > > +++ b/mm/Makefile
> > > @@ -99,6 +99,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o
> > > obj-$(CONFIG_NUMA) += memory-tiers.o
> > > obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
> > > obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
> > > +obj-$(CONFIG_BPF_GET_THP_ORDER) += huge_memory_bpf.o
> > > obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
> > > obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
> > > obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
> > > diff --git a/mm/huge_memory_bpf.c b/mm/huge_memory_bpf.c
> > > new file mode 100644
> > > index 000000000000..525ee22ab598
> > > --- /dev/null
> > > +++ b/mm/huge_memory_bpf.c
> > > @@ -0,0 +1,243 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * BPF-based THP policy management
> > > + *
> > > + * Author: Yafang Shao <laoar.shao@xxxxxxxxx>
> > > + */
> > > +
> > > +#include <linux/bpf.h>
> > > +#include <linux/btf.h>
> > > +#include <linux/huge_mm.h>
> > > +#include <linux/khugepaged.h>
> > > +
> > > +enum bpf_thp_vma_type {
> > > + BPF_THP_VM_NONE = 0,
> > > + BPF_THP_VM_HUGEPAGE, /* VM_HUGEPAGE */
> > > + BPF_THP_VM_NOHUGEPAGE, /* VM_NOHUGEPAGE */
> > > +};
> >
> > I'm really not so sure how useful this is - can't a user just ascertain this
> > from the VMA flags themselves?
>
> I assume you are referring to checking flags from vma->vm_flags.
> There is an exception where we cannot use vma->vm_flags: in
> hugepage_madvise(), which calls khugepaged_enter_vma(vma, *vm_flags).
>
> At this point, the VM_HUGEPAGE flag has not been set in vma->vm_flags
> yet. Therefore, we must pass the separate *vm_flags variable.
> Perhaps we can simplify the logic with the following change?
Ugh god.
I guess this is the workaround for the vm_flags thing right.
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 35ed4ab0d7c5..5755de80a4d7 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -1425,6 +1425,8 @@ static int madvise_vma_behavior(struct
> madvise_behavior *madv_behavior)
> VM_WARN_ON_ONCE(madv_behavior->lock_mode != MADVISE_MMAP_WRITE_LOCK);
>
> error = madvise_update_vma(new_flags, madv_behavior);
> + if (new_flags & VM_HUGEPAGE)
> + khugepaged_enter_vma(vma);
Hm ok, that's not such a bad idea, though ofc this should be something like:
if (!error && (new_flags & VM_HUGEPAGE))
khugepaged_enter_vma(vma);
And obviously dropping this khugepaged_enter_vma() from hugepage_madvise().
> out:
> /*
> * madvise() returns EAGAIN if kernel resources, such as
>
> >
> > Let's keep the interface as minimal as possible.
> >
> > > +
> > > +/**
> > > + * @thp_order_fn_t: Get the suggested THP orders from a BPF program for allocation
> >
> > orders -> order?
>
> ack
Thanks!
>
> >
> > > + * @vma: vm_area_struct associated with the THP allocation
> > > + * @vma_type: The VMA type, such as BPF_THP_VM_HUGEPAGE if VM_HUGEPAGE is set
> > > + * BPF_THP_VM_NOHUGEPAGE if VM_NOHUGEPAGE is set, or BPF_THP_VM_NONE if
> > > + * neither is set.
> >
> > Obv as above let's drop this probably :)
> >
> > > + * @tva_type: TVA type for current @vma
> > > + * @orders: Bitmask of requested THP orders for this allocation
> >
> > Shouldn't requested = available?
>
> ack
Thanks!
>
> >
> > > + * - PMD-mapped allocation if PMD_ORDER is set
> > > + * - mTHP allocation otherwise
> >
> > Not sure these 2 points are super useful.
>
> will remove it.
Thanks!
>
> >
> > > + *
> > > + * Return: The suggested THP order from the BPF program for allocation. It will
> > > + * not exceed the highest requested order in @orders. Return -1 to
> > > + * indicate that the original requested @orders should remain unchanged.
> > > + */
> > > +typedef int thp_order_fn_t(struct vm_area_struct *vma,
> > > + enum bpf_thp_vma_type vma_type,
> > > + enum tva_type tva_type,
> > > + unsigned long orders);
> > > +
> > > +struct bpf_thp_ops {
> > > + thp_order_fn_t __rcu *thp_get_order;
> > > +};
> > > +
> > > +static struct bpf_thp_ops bpf_thp;
> > > +static DEFINE_SPINLOCK(thp_ops_lock);
> > > +
> > > +/*
> > > + * Returns the original @orders if no BPF program is attached or if the
> > > + * suggested order is invalid.
> > > + */
> > > +unsigned long bpf_hook_thp_get_orders(struct vm_area_struct *vma,
> > > + vm_flags_t vma_flags,
> > > + enum tva_type tva_type,
> > > + unsigned long orders)
> > > +{
> > > + thp_order_fn_t *bpf_hook_thp_get_order;
> > > + unsigned long thp_orders = orders;
> > > + enum bpf_thp_vma_type vma_type;
> > > + int thp_order;
> > > +
> > > + /* No BPF program is attached */
> > > + if (!test_bit(TRANSPARENT_HUGEPAGE_BPF_ATTACHED,
> > > + &transparent_hugepage_flags))
> > > + return orders;
> > > +
> > > + if (vma_flags & VM_HUGEPAGE)
> > > + vma_type = BPF_THP_VM_HUGEPAGE;
> > > + else if (vma_flags & VM_NOHUGEPAGE)
> > > + vma_type = BPF_THP_VM_NOHUGEPAGE;
> > > + else
> > > + vma_type = BPF_THP_VM_NONE;
> >
> > As per above, not sure this is all that useful.
> >
> > > +
> > > + rcu_read_lock();
> > > + bpf_hook_thp_get_order = rcu_dereference(bpf_thp.thp_get_order);
> > > + if (!bpf_hook_thp_get_order)
> > > + goto out;
> > > +
> > > + thp_order = bpf_hook_thp_get_order(vma, vma_type, tva_type, orders);
> > > + if (thp_order < 0)
> > > + goto out;
> > > + /*
> > > + * The maximum requested order is determined by the callsite. E.g.:
> > > + * - PMD-mapped THP uses PMD_ORDER
> > > + * - mTHP uses (PMD_ORDER - 1)
> >
> > I don't think this is quite right, highest_order() figures out the highest set
> > bit, so mTHP can be PMD_ORDER - 1 or less (in theory ofc).
> >
> > I think we can just replace this with something simpler like - 'depending on
> > where the BPF hook is invoked, we check for either PMD order or mTHP orders
> > (less than PMD order)' or something.
>
> ack
Thanks!
>
> >
> > > + *
> > > + * We must respect this upper bound to avoid undefined behavior. So the
> > > + * highest suggested order can't exceed the highest requested order.
> > > + */
> >
> > I think this sentence is also unnecessary.
>
> will remove it.
Thanks!
>
> --
> Regards
> Yafang
Cheers, Lorenzo
