On 7/3/25 4:50 PM, Jason Xing wrote: > From: Jason Xing <kernelxing@xxxxxxxxxxx> > > This patch provides a setsockopt method to let applications leverage to > adjust how many descs to be handled at most in one send syscall. It > mitigates the situation where the default value (32) that is too small > leads to higher frequency of triggering send syscall. > > Considering the prosperity/complexity the applications have, there is no > absolutely ideal suggestion fitting all cases. So keep 32 as its default > value like before. > > The patch does the following things: > - Add XDP_MAX_TX_SKB_BUDGET socket option. > - Set max_tx_budget to 32 by default in the initialization phase as a > per-socket granular control. > - Set the range of max_tx_budget as [32, xs->tx->nentries]. > > The idea behind this comes out of real workloads in production. We use a > user-level stack with xsk support to accelerate sending packets and > minimize triggering syscalls. When the packets are aggregated, it's not > hard to hit the upper bound (namely, 32). The moment user-space stack > fetches the -EAGAIN error number passed from sendto(), it will loop to try > again until all the expected descs from tx ring are sent out to the driver. > Enlarging the XDP_MAX_TX_SKB_BUDGET value contributes to less frequency of > sendto() and higher throughput/PPS. > > Here is what I did in production, along with some numbers as follows: > For one application I saw lately, I suggested using 128 as max_tx_budget > because I saw two limitations without changing any default configuration: > 1) XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by > net.core.wmem_default. As to XDP_MAX_TX_SKB_BUDGET, the scenario behind > this was I counted how many descs are transmitted to the driver at one > time of sendto() based on [1] patch and then I calculated the > possibility of hitting the upper bound. Finally I chose 128 as a > suitable value because 1) it covers most of the cases, 2) a higher > number would not bring evident results. After twisting the parameters, > a stable improvement of around 4% for both PPS and throughput and less > resources consumption were found to be observed by strace -c -p xxx: > 1) %time was decreased by 7.8% > 2) error counter was decreased from 18367 to 572 > > [1]: https://lore.kernel.org/all/20250619093641.70700-1-kerneljasonxing@xxxxxxxxx/ > > Signed-off-by: Jason Xing <kernelxing@xxxxxxxxxxx> > --- > v7 > Link: https://lore.kernel.org/all/20250627110121.73228-1-kerneljasonxing@xxxxxxxxx/ > 1. use 'copy mode' in Doc > 2. move init of max_tx_budget to a proper position > 3. use the max value in the if condition in setsockopt > 4. change sockopt name to XDP_MAX_TX_SKB_BUDGET > 5. set MAX_PER_SOCKET_BUDGET to 32 instead of TX_BATCH_SIZE because they > have no correlation at all. > > v6 > Link: https://lore.kernel.org/all/20250625123527.98209-1-kerneljasonxing@xxxxxxxxx/ > 1. use [32, xs->tx->nentries] range > 2. Since setsockopt may generate a different value, add getsockopt to help > application know what value takes effect finally. > > v5 > Link: https://lore.kernel.org/all/20250623021345.69211-1-kerneljasonxing@xxxxxxxxx/ > 1. remove changes around zc mode > > v4 > Link: https://lore.kernel.org/all/20250619090440.65509-1-kerneljasonxing@xxxxxxxxx/ > 1. remove getsockopt as it seems no real use case. > 2. adjust the position of max_tx_budget to make sure it stays with other > read-most fields in one cacheline. > 3. set one as the lower bound of max_tx_budget > 4. add more descriptions/performance data in Doucmentation and commit message. > > V3 > Link: https://lore.kernel.org/all/20250618065553.96822-1-kerneljasonxing@xxxxxxxxx/ > 1. use a per-socket control (suggested by Stanislav) > 2. unify both definitions into one > 3. support setsockopt and getsockopt > 4. add more description in commit message > > V2 > Link: https://lore.kernel.org/all/20250617002236.30557-1-kerneljasonxing@xxxxxxxxx/ > 1. use a per-netns sysctl knob > 2. use sysctl_xsk_max_tx_budget to unify both definitions. > --- > Documentation/networking/af_xdp.rst | 9 +++++++++ > include/net/xdp_sock.h | 1 + > include/uapi/linux/if_xdp.h | 1 + > net/xdp/xsk.c | 21 +++++++++++++++++++-- > tools/include/uapi/linux/if_xdp.h | 1 + > 5 files changed, 31 insertions(+), 2 deletions(-) > > diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst > index dceeb0d763aa..95ff1836e5c6 100644 > --- a/Documentation/networking/af_xdp.rst > +++ b/Documentation/networking/af_xdp.rst > @@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. > Once the option is set, kernel will refuse attempts to bind that socket > to a different interface. Updating the value requires CAP_NET_RAW. > > +XDP_MAX_TX_SKB_BUDGET setsockopt > +---------------------------- I'm sorry, kdoc is not happy about the above: /home/doc-build/testing/Documentation/networking/af_xdp.rst:442: WARNING: Title underline too short.XDP_MAX_TX_SKB_BUDGET setsockopt ---------------------------- /P
