[dpdk-dev] mlx5 and secondary processes

[Olivier Gournet]

Hi Nelio,

On 11/14/2017 11:34 AM, Nelio Laranjeiro wrote:
> Hi Olivier,
> 
> On Tue, Nov 14, 2017 at 10:58:02AM +0100, Olivier Gournet wrote:
>> Hi,
>>
>> I can't get TX on secondary process to works with the lastest dpdk, it was running fine with dpdk-16.11.
>> RX/TX is ok on primary processs, and I'm not interested in RX on secondary process. Each process has its
>> owns TX queues.
>>
>> I upgraded everything to:
>>    linux 4.14
>>    dpdk 17.11-rc4
>>    rdma-core from yesterday git-master
>>    mlx-fw 12.21.1000
>>
>> On seconday process, TX queue gets full and is never emptied.
>> It seems like bf_reg is correctly re-mapped from secondary process; here's dmesg from primary process:
>>
>> [...]
>> [85361.895778] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x6, pfn 0x00000000000f8020
>> [85361.895781] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000
>> [85361.895784] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x7, pfn 0x00000000000f8021
>> [85361.895787] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000
>>
>> Then from secondary process:
>>
>> [...]
>> [85408.038295] mlx5_1:mlx5_ib_mmap:1778:(pid 9551): mapped internal timer at 0x7ffff7f7f000, PA 0xf8001000
>> [85408.040229] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x6, pfn 0x00000000000f8020
>> [85408.040233] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7faa000, PA 0x00000000f8020000
>> [85408.040239] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x7, pfn 0x00000000000f8021
>> [85408.040241] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 0x7ffff7fa9000, PA 0x00000000f8021000
>>
>> But then from mlx5_rxtx.h:mlx5_tx_dbrec_cond_wmb() it doesn't seems like writing to txq->bf_reg has any effect.
>> I know it isn't supposed to be read, but when reading from primary process is always give *dst==0xe5ccdabae5ccdaba,
>> whereas from secondary process *dst is zero or the last written value.
>>
>> I don't really have any other clues, and don't know where to search. Can anybody give me some hints ?
> 
> Since Yongseok patch [1] the UAR pages are no more configured though the
> Write Combining of the PCI but in your logs it seems it is still using
> the cache.
> 
> Do you have the "MLX5_SHUT_UP_BF" set to 1 in your environment?  If not
> does adding it solves your issue?

I set it (on both primary and secondary) and it is seen by kernel driver,
but unfortunately it is still not working.

[98321.605791] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x7ffff7faa000, PA 0x00000000f8020000
[98321.605799] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x7ffff7fa9000, PA 0x00000000f8021000

Thanks,