On Thu, May 08, 2025 at 11:41:18AM +0300, Edward Srouji wrote: > > On 5/6/2025 5:22 PM, Jason Gunthorpe wrote: > > On Tue, May 06, 2025 at 10:19:06AM -0400, Chuck Lever wrote: > > > > > In this patch I'm trying to include the reg/inv multiplier in the > > > > > calculation, but that doesn't seem to be enough to make "accept" > > > > > reliable, IMO due to this extra calculation in calc_sq_size(). > > > > Did ib_create_qp get called with more than max_qp_wr ? > > > The request was for, like, 9300 SQEs. max_qp_wr is 32K on my systems. > > Sounds like it is broken then.. > > > > props->max_qp_wr = 1 << MLX5_CAP_GEN(mdev, log_max_qp_sz); > > > > So it is ignoring the wqe_size adustment.. It should adjust by the worst > > case result of calc_send_wqe() for the device.. > How do you suggest adjusting to the worst case? > How inline messages could be addressed and taken into account? I think assume 0 size inline for computing max sizes > Even if we ignore the inline size, worst case potentially could be less than > 1/8th of the max HCA CAP, not sure we want to deliver this as a limitation > to users. The math is simply wrong - log_max_qp_sz is not the number of work queue entries in the queue, it is the number of MLX5_SEND_WQE_BB's units which is some internal value. For a verbs API the result should be the max number of work queue entries that can be requested for any of XRC/RC/UC/UD QP types using a 0 inline size, 1 SGL and no other special features. Even for a simple RC QP sq_overhead() will return 132 which already makes props->max_qp_wr uselessly wrong. 132 goes into here: return ALIGN(max_t(int, inl_size, size), MLX5_SEND_WQE_BB); Comes out as 192 - so props->max_qp_wr is off by 3x even for a simple no-feature RC QP. Chuck is getting: calc_sq_size:618:(pid 1514): send queue size (9326 * 256 / 64 -> 65536) exceeds limits(32768) So I suppose that extra 64 bytes is coming from cap.max_send_sge >= 3? Without a new API we can't make it fully discoverable, but the way it is now is clearly wrong. Jason
