[dpdk-dev] [PATCH v2 3/5] net/mlx5: use buffer address for LKEY search

[Yongseok Koh]

> On Jul 3, 2017, at 7:06 AM, Nélio Laranjeiro <nelio.laranjeiro at 6wind.com> wrote:
> 
> On Fri, Jun 30, 2017 at 12:23:31PM -0700, Yongseok Koh wrote:
>> When searching LKEY, if search key is mempool pointer, the 2nd cacheline
>> has to be accessed and it even requires to check whether a buffer is
>> indirect per every search. Instead, using address for search key can reduce
>> cycles taken. And caching the last hit entry is beneficial as well.
>> 
>> Signed-off-by: Yongseok Koh <yskoh at mellanox.com>
>> ---
>> drivers/net/mlx5/mlx5_mr.c   | 17 ++++++++++++++---
>> drivers/net/mlx5/mlx5_rxtx.c | 39 +++++++++++++++++++++------------------
>> drivers/net/mlx5/mlx5_rxtx.h |  4 +++-
>> drivers/net/mlx5/mlx5_txq.c  |  3 +--
>> 4 files changed, 39 insertions(+), 24 deletions(-)
>> 
>> diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
>> index 0a3638460..287335179 100644
>> --- a/drivers/net/mlx5/mlx5_mr.c
>> +++ b/drivers/net/mlx5/mlx5_mr.c
>> @@ -265,18 +266,28 @@ txq_mp2mr_iter(struct rte_mempool *mp, void *arg)
>> 	struct txq_mp2mr_mbuf_check_data data = {
>> 		.ret = 0,
>> 	};
>> +	uintptr_t start;
>> +	uintptr_t end;
>> 	unsigned int i;
>> 
>> 	/* Register mempool only if the first element looks like a mbuf. */
>> 	if (rte_mempool_obj_iter(mp, txq_mp2mr_mbuf_check, &data) == 0 ||
>> 			data.ret == -1)
>> 		return;
>> +	if (mlx5_check_mempool(mp, &start, &end) != 0) {
>> +		ERROR("mempool %p: not virtually contiguous",
>> +		      (void *)mp);
>> +		return;
>> +	}
>> 	for (i = 0; (i != RTE_DIM(txq_ctrl->txq.mp2mr)); ++i) {
>> -		if (unlikely(txq_ctrl->txq.mp2mr[i].mp == NULL)) {
>> +		struct ibv_mr *mr = txq_ctrl->txq.mp2mr[i].mr;
>> +
>> +		if (unlikely(mr == NULL)) {
>> 			/* Unknown MP, add a new MR for it. */
>> 			break;
>> 		}
>> -		if (txq_ctrl->txq.mp2mr[i].mp == mp)
>> +		if (start >= (uintptr_t)mr->addr &&
>> +		    end <= (uintptr_t)mr->addr + mr->length)
>> 			return;
>> 	}
>> 	txq_mp2mr_reg(&txq_ctrl->txq, mp, i);
> 
> if (start >= (uintptr_t)mr->addr &&
>     end <= (uintptr_t)mr->addr + mr->length)
> 
> Is this expected to have a memory region bigger than the memory pool
> space?  I mean I was expecting to see strict equality in the addresses.
In mlx5_mp2mr(), start/end of a memory region are rounded up to make it
aligned to its hugepage size.

struct ibv_mr *
mlx5_mp2mr(struct ibv_pd *pd, struct rte_mempool *mp)
{
[...]
        /* Round start and end to page boundary if found in memory segments. */
        for (i = 0; (i < RTE_MAX_MEMSEG) && (ms[i].addr != NULL); ++i) {
                uintptr_t addr = (uintptr_t)ms[i].addr;
                size_t len = ms[i].len;
                unsigned int align = ms[i].hugepage_sz;

                if ((start > addr) && (start < addr + len))
                        start = RTE_ALIGN_FLOOR(start, align);
                if ((end > addr) && (end < addr + len))
                        end = RTE_ALIGN_CEIL(end, align);
        }

Thanks,
Yongseok