On Fri, Jul 4, 2025 at 2:13 PM Paolo Abeni <pabeni@xxxxxxxxxx> wrote: > > On 7/3/25 4:50 PM, Jason Xing wrote: > > From: Jason Xing <kernelxing@xxxxxxxxxxx> > > > > This patch provides a setsockopt method to let applications leverage to > > adjust how many descs to be handled at most in one send syscall. It > > mitigates the situation where the default value (32) that is too small > > leads to higher frequency of triggering send syscall. > > > > Considering the prosperity/complexity the applications have, there is no > > absolutely ideal suggestion fitting all cases. So keep 32 as its default > > value like before. > > > > The patch does the following things: > > - Add XDP_MAX_TX_SKB_BUDGET socket option. > > - Set max_tx_budget to 32 by default in the initialization phase as a > > per-socket granular control. > > - Set the range of max_tx_budget as [32, xs->tx->nentries]. > > > > The idea behind this comes out of real workloads in production. We use a > > user-level stack with xsk support to accelerate sending packets and > > minimize triggering syscalls. When the packets are aggregated, it's not > > hard to hit the upper bound (namely, 32). The moment user-space stack > > fetches the -EAGAIN error number passed from sendto(), it will loop to try > > again until all the expected descs from tx ring are sent out to the driver. > > Enlarging the XDP_MAX_TX_SKB_BUDGET value contributes to less frequency of > > sendto() and higher throughput/PPS. > > > > Here is what I did in production, along with some numbers as follows: > > For one application I saw lately, I suggested using 128 as max_tx_budget > > because I saw two limitations without changing any default configuration: > > 1) XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by > > net.core.wmem_default. As to XDP_MAX_TX_SKB_BUDGET, the scenario behind > > this was I counted how many descs are transmitted to the driver at one > > time of sendto() based on [1] patch and then I calculated the > > possibility of hitting the upper bound. Finally I chose 128 as a > > suitable value because 1) it covers most of the cases, 2) a higher > > number would not bring evident results. After twisting the parameters, > > a stable improvement of around 4% for both PPS and throughput and less > > resources consumption were found to be observed by strace -c -p xxx: > > 1) %time was decreased by 7.8% > > 2) error counter was decreased from 18367 to 572 > > > > [1]: https://lore.kernel.org/all/20250619093641.70700-1-kerneljasonxing@xxxxxxxxx/ > > > > Signed-off-by: Jason Xing <kernelxing@xxxxxxxxxxx> > > --- > > v7 > > Link: https://lore.kernel.org/all/20250627110121.73228-1-kerneljasonxing@xxxxxxxxx/ > > 1. use 'copy mode' in Doc > > 2. move init of max_tx_budget to a proper position > > 3. use the max value in the if condition in setsockopt > > 4. change sockopt name to XDP_MAX_TX_SKB_BUDGET > > 5. set MAX_PER_SOCKET_BUDGET to 32 instead of TX_BATCH_SIZE because they > > have no correlation at all. > > > > v6 > > Link: https://lore.kernel.org/all/20250625123527.98209-1-kerneljasonxing@xxxxxxxxx/ > > 1. use [32, xs->tx->nentries] range > > 2. Since setsockopt may generate a different value, add getsockopt to help > > application know what value takes effect finally. > > > > v5 > > Link: https://lore.kernel.org/all/20250623021345.69211-1-kerneljasonxing@xxxxxxxxx/ > > 1. remove changes around zc mode > > > > v4 > > Link: https://lore.kernel.org/all/20250619090440.65509-1-kerneljasonxing@xxxxxxxxx/ > > 1. remove getsockopt as it seems no real use case. > > 2. adjust the position of max_tx_budget to make sure it stays with other > > read-most fields in one cacheline. > > 3. set one as the lower bound of max_tx_budget > > 4. add more descriptions/performance data in Doucmentation and commit message. > > > > V3 > > Link: https://lore.kernel.org/all/20250618065553.96822-1-kerneljasonxing@xxxxxxxxx/ > > 1. use a per-socket control (suggested by Stanislav) > > 2. unify both definitions into one > > 3. support setsockopt and getsockopt > > 4. add more description in commit message > > > > V2 > > Link: https://lore.kernel.org/all/20250617002236.30557-1-kerneljasonxing@xxxxxxxxx/ > > 1. use a per-netns sysctl knob > > 2. use sysctl_xsk_max_tx_budget to unify both definitions. > > --- > > Documentation/networking/af_xdp.rst | 9 +++++++++ > > include/net/xdp_sock.h | 1 + > > include/uapi/linux/if_xdp.h | 1 + > > net/xdp/xsk.c | 21 +++++++++++++++++++-- > > tools/include/uapi/linux/if_xdp.h | 1 + > > 5 files changed, 31 insertions(+), 2 deletions(-) > > > > diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst > > index dceeb0d763aa..95ff1836e5c6 100644 > > --- a/Documentation/networking/af_xdp.rst > > +++ b/Documentation/networking/af_xdp.rst > > @@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. > > Once the option is set, kernel will refuse attempts to bind that socket > > to a different interface. Updating the value requires CAP_NET_RAW. > > > > +XDP_MAX_TX_SKB_BUDGET setsockopt > > +---------------------------- > > I'm sorry, kdoc is not happy about the above: > > /home/doc-build/testing/Documentation/networking/af_xdp.rst:442: > WARNING: Title underline too short.XDP_MAX_TX_SKB_BUDGET setsockopt > ---------------------------- Ah, I see... Sorry, I should have checked this. I will post a v8 patch with this one and RCT issues solved. Thanks, Jason > > /P >
