On 30/07/2025 09.31, Dragos Tatulea wrote:
On Tue, Jul 29, 2025 at 01:20:07PM -0700, Dipayaan Roy wrote:
On Tue, Jul 29, 2025 at 12:15:23PM +0200, Jesper Dangaard Brouer wrote:
On 23/07/2025 21.07, Dipayaan Roy wrote:
This patch enhances RX buffer handling in the mana driver by allocating
pages from a page pool and slicing them into MTU-sized fragments, rather
than dedicating a full page per packet. This approach is especially
beneficial on systems with large page sizes like 64KB.
Key improvements:
- Proper integration of page pool for RX buffer allocations.
- MTU-sized buffer slicing to improve memory utilization.
- Reduce overall per Rx queue memory footprint.
- Automatic fallback to full-page buffers when:
* Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
* The XDP path is active, to avoid complexities with fragment reuse.
- Removal of redundant pre-allocated RX buffers used in scenarios like MTU
changes, ensuring consistency in RX buffer allocation.
Testing on VMs with 64KB pages shows around 200% throughput improvement.
Memory efficiency is significantly improved due to reduced wastage in page
allocations. Example: We are now able to fit 35 rx buffers in a single 64kb
page for MTU size of 1500, instead of 1 rx buffer per page previously.
Tested:
- iperf3, iperf2, and nttcp benchmarks.
- Jumbo frames with MTU 9000.
- Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
testing the XDP path in driver.
- Page leak detection (kmemleak).
- Driver load/unload, reboot, and stress scenarios.
Chris (Cc) discovered a crash/bug[1] with page pool fragments used
from the mlx5 driver.
He put together a BPF program that reproduces the issue here:
- [2] https://github.com/arges/xdp-redirector
Can I ask you to test that your driver against this reproducer?
[1] https://lore.kernel.org/all/aIEuZy6fUj_4wtQ6@861G6M3/
--Jesper
Hi Jesper,
I was unable to reproduce this issue on mana driver.
Please note that I had to make a few adjustments to get reprodduction on
mlx5:
- Make sure that the veth MACs are recognized by the device. Otherwise
traffic might be dropped by the device.
- Enable GRO on the veth device. Otherwise packets get dropped before
they reach the devmap BPF program.
Try starting the test program with one thread and see if you see packets
coming through veth1-ns1 end of the veth pair.
Hi Dipayaan,
Enabling GRO on the veth device is quite important for the test to be valid.
I've asked Chris to fix this in the reproducer. He can report back when
he have done this, so you can re-run the test. It is also good advice
from Dragos that you should check packets are coming through the veth
pair, to make sure the test is working.
The setup.sh script also need to be modified, as it is loading xdp on a
net_device called "ext0" [0], which is specific to our systems (which
default also have GRO enabled for veth).
[0] https://github.com/arges/xdp-redirector/blob/main/setup.sh#L28
--Jesper
