On Wed, Jun 18, 2025 at 03:16:15PM -0700, Stanislav Fomichev wrote:
> On 06/16, Mark Bloch wrote:
> > From: Dragos Tatulea <dtatulea@xxxxxxxxxx>
> >
> > Declare netmem TX support in netdev.
> >
> > As required, use the netmem aware dma unmapping APIs
> > for unmapping netmems in tx completion path.
> >
> > Signed-off-by: Dragos Tatulea <dtatulea@xxxxxxxxxx>
> > Reviewed-by: Tariq Toukan <tariqt@xxxxxxxxxx>
> > Reviewed-by: Mina Almasry <almasrymina@xxxxxxxxxx>
> > Signed-off-by: Mark Bloch <mbloch@xxxxxxxxxx>
> > ---
> > drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 3 ++-
> > drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 ++
> > 2 files changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> > index e837c21d3d21..6501252359b0 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
> > @@ -362,7 +362,8 @@ mlx5e_tx_dma_unmap(struct device *pdev, struct mlx5e_sq_dma *dma)
> > dma_unmap_single(pdev, dma->addr, dma->size, DMA_TO_DEVICE);
> > break;
> > case MLX5E_DMA_MAP_PAGE:
> > - dma_unmap_page(pdev, dma->addr, dma->size, DMA_TO_DEVICE);
> > + netmem_dma_unmap_page_attrs(pdev, dma->addr, dma->size,
> > + DMA_TO_DEVICE, 0);
>
> For this to work, the dma->addr needs to be 0, so the callers of the
> dma_map() need to be adjusted as well, or am I missing something?
> There is netmem_dma_unmap_addr_set to handle that, but I don't see
> anybody calling it. Do we need to add the following (untested)?
>
Hmmmm... yes. I figured that skb_frag_dma_map() would do the work
but I was wrong, it is not enough.
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> index 55a8629f0792..fb6465210aed 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> @@ -210,7 +210,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
> if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
> goto dma_unmap_wqe_err;
>
> - dseg->addr = cpu_to_be64(dma_addr);
> + dseg->addr = 0;
> + if (!netmem_is_net_iov(skb_frag_netmem(frag)))
> + dseg->addr = cpu_to_be64(dma_addr);
AFAIU we still want to pass the computed dma_address to the data segment
to the HW. We only need to make sure in mlx5e_dma_push() to set dma_addr
to 0, to avoid calling netmem_dma_unmap_page_attrs() with dma->addr 0.
Like in the snippet below. Do you agree?
We will send a fix patch once the above question is answered. Also, is
there a way to test this with more confidence? The ncdevmem tx test
passed just fine.
Thanks,
Dragos
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 55a8629f0792..ecee2e4f678b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -214,6 +214,9 @@ mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
dseg->lkey = sq->mkey_be;
dseg->byte_count = cpu_to_be32(fsz);
+ if (!netmem_is_net_iov(skb_frag_netmem(frag)))
+ dma_addr = 0;
+
mlx5e_dma_push(sq, dma_addr, fsz, MLX5E_DMA_MAP_PAGE);
num_dma++;
dseg++;
