Background
==========
Our production servers consistently configure THP to "never" due to
historical incidents caused by its behavior. Key issues include:
- Increased Memory Consumption
THP significantly raises overall memory usage, reducing available memory
for workloads.
- Latency Spikes
Random latency spikes occur due to frequent memory compaction triggered
by THP.
- Lack of Fine-Grained Control
THP tuning is globally configured, making it unsuitable for containerized
environments. When multiple workloads share a host, enabling THP without
per-workload control leads to unpredictable behavior.
Due to these issues, administrators avoid switching to madvise or always
modes—unless per-workload THP control is implemented.
To address this, we propose BPF-based THP policy for flexible adjustment.
Additionally, as David mentioned, this mechanism can also serve as a
policy prototyping tool (test policies via BPF before upstreaming them).
Proposed Solution
=================
This patch introduces a new BPF struct_ops called bpf_thp_ops for dynamic
THP tuning. It includes a hook thp_get_order(), allowing BPF programs to
influence THP order selection based on factors such as:
- Workload identity
For example, workloads running in specific containers or cgroups.
- Allocation context
Whether the allocation occurs during a page fault, khugepaged, swap or
other paths.
- VMA's memory advice settings
MADV_HUGEPAGE or MADV_NOHUGEPAGE
- Memory pressure
PSI system data or associated cgroup PSI metrics
The new interface for the BPF program is as follows:
/**
* @thp_get_order: Get the suggested THP orders from a BPF program for allocation
* @vma: vm_area_struct associated with the THP allocation
* @vma_type: The VMA type, such as BPF_THP_VM_HUGEPAGE if VM_HUGEPAGE is set
* BPF_THP_VM_NOHUGEPAGE if VM_NOHUGEPAGE is set, or BPF_THP_VM_NONE
* if neither is set.
* @tva_type: TVA type for current @vma
* @orders: Bitmask of requested THP orders for this allocation
* - PMD-mapped allocation if PMD_ORDER is set
* - mTHP allocation otherwise
*
* Return: The suggested THP order from the BPF program for allocation. It will
* not exceed the highest requested order in @orders. Return -1 to
* indicate that the original requested @orders should remain unchanged.
*/
int thp_get_order(struct vm_area_struct *vma,
enum bpf_thp_vma_type vma_type,
enum tva_type tva_type,
unsigned long orders);
Only a single BPF program can be attached at any given time, though it can
be dynamically updated to adjust the policy. The implementation supports
anonymous THP, shmem THP, and mTHP, with future extensions planned for
file-backed THP.
This functionality is only active when system-wide THP is configured to
madvise or always mode. It remains disabled in never mode. Additionally,
if THP is explicitly disabled for a specific task via prctl(), this BPF
functionality will also be unavailable for that task
**WARNING**
- This feature requires CONFIG_BPF_GET_THP_ORDER (marked EXPERIMENTAL) to
be enabled.
- The interface may change
- Behavior may differ in future kernel versions
- We might remove it in the future
Selftests
=========
BPF CI
------
Patch #7: Implements a basic BPF THP policy that restricts THP allocation
via khugepaged to tasks within a specified memory cgroup.
Patch #8: Provides tests for dynamic BPF program updates and replacement.
Patch #9: Includes negative tests for invalid BPF helper usage, verifying
proper verification by the BPF verifier.
Currently, several dependency patches reside in mm-new but haven't been
merged into bpf-next. To enable BPF CI testing, these dependencies were
manually applied to bpf-next. All selftests in this series pass
successfully [0].
Performance Evaluation
----------------------
Performance impact was measured given the page fault handler modifications.
The standard `perf bench mem memset` benchmark was employed to assess page
fault performance.
Testing was conducted on an AMD EPYC 7W83 64-Core Processor (single NUMA
node). Due to variance between individual test runs, a script executed
10000 iterations to calculate meaningful averages.
- Baseline (without this patch series)
- With patch series but no BPF program attached
- With patch series and BPF program attached
The results across three configurations show negligible performance impact:
Number of runs: 10,000
Average throughput: 40-41 GB/sec
Production verification
-----------------------
We have successfully deployed a variant of this approach across numerous
Kubernetes production servers. The implementation enables THP for specific
workloads (such as applications utilizing ZGC [1]) while disabling it for
others. This selective deployment has operated flawlessly, with no
regression reports to date.
For ZGC-based applications, our verification demonstrates that shmem THP
delivers significant improvements:
- Reduced CPU utilization
- Lower average latencies
We are continuously extending its support to more workloads, such as
TCMalloc-based services. [2]
Deployment Steps in our production servers are as follows,
1. Initial Setup:
- Set THP mode to "never" (disabling THP by default).
- Attach the BPF program and pin the BPF maps and links.
- Pinning ensures persistence (like a kernel module), preventing
disruption under system pressure.
- A THP whitelist map tracks allowed cgroups (initially empty -> no THP
allocations).
2. Enable THP Control:
- Switch THP mode to "always" or "madvise" (BPF now governs actual allocations).
3. Dynamic Management:
- To permit THP for a cgroup, add its ID to the whitelist map.
- To revoke permission, remove the cgroup ID from the map.
- The BPF program can be updated live (policy adjustments require no
task interruption).
4. To roll back, disable THP and remove this BPF program.
**WARNING**
Be aware that the maintainers do not suggest this use case, as the BPF hook
interface is unstable and might be removed from the upstream kernel—unless
you have your own kernel team to maintain it ;-)
Future work
===========
file-backed THP policy
----------------------
Based on our validation with production workloads, we observed mixed
results with XFS large folios (also known as file-backed THP):
- Performance Benefits
Some workloads demonstrated significant improvements with XFS large
folios enabled
- Performance Regression
Some workloads experienced degradation when using XFS large folios
These results demonstrate that File THP, similar to anonymous THP, requires
a more granular approach instead of a uniform implementation.
We will extend the BPF-based order selection mechanism to support
file-backed THP allocation policies.
Hooking fork() with BPF for Task Configuration
----------------------------------------------
The current method for controlling a newly fork()-ed task involves calling
prctl() (e.g., with PR_SET_THP_DISABLE) to set flags in its mm->flags. This
requires explicit userspace modification.
A more efficient alternative is to implement a new BPF hook within the
fork() path. This hook would allow a BPF program to set the task's
mm->flags directly after mm initialization, leveraging BPF helpers for a
solution that is transparent to userspace. This is particularly valuable in
data center environments for fleet-wide management.
Link: https://github.com/kernel-patches/bpf/pull/9706 [0]
Link: https://wiki.openjdk.org/display/zgc/Main#Main-EnablingTr... [1]
Link: https://google.github.io/tcmalloc/tuning.html#system-level-optimizations [2]
Changes:
=======:
v6->v7:
Key Changes Implemented Based on Feedback:
