[dpdk-dev] [RFC 00/14] mlx5: support SubFunction

[Xia, Chenbo]

Hi Thomas,

> -----Original Message-----
> From: Thomas Monjalon <thomas at monjalon.net>
> Sent: Friday, June 11, 2021 3:54 PM
> To: Yigit, Ferruh <ferruh.yigit at intel.com>; Xia, Chenbo <chenbo.xia at intel.com>
> Cc: Xueming Li <xuemingl at nvidia.com>; Viacheslav Ovsiienko
> <viacheslavo at nvidia.com>; dev at dpdk.org; parav at nvidia.com; jgg at nvidia.com
> Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction
> 
> 11/06/2021 07:14, Xia, Chenbo:
> > From: Thomas Monjalon <thomas at monjalon.net>
> > > 10/06/2021 12:33, Ferruh Yigit:
> > > > On 5/27/2021 2:37 PM, Xueming Li wrote:
> > > > > SubFunction [1] is a portion of the PCI device, a SF netdev has its
> > > own
> > > > > dedicated queues(txq, rxq). A SF shares PCI level resources with other
> > > > > SFs and/or with its parent PCI function. Auxiliary bus is the
> > > > > fundamental of SF.
> > > > >
> > > > > This patch set introduces SubFunction support for mlx5 PMD driver
> > > > > including class net, regex, vdpa and compress.
> > > > >
> > > >
> > > > There is already an mdev patch, originated from long ago. Aren't
> > > subfunctions
> > > > presented as mdev device? If so can't we use mdev for it?
> > >
> > > No unfortunately that's different.
> > > mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.
> >
> > Just out of curiosity:
> >
> > Does SF use mdev before aux bus is introduced in kernel. I see some history
> > of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField
> > software v2.5 is using mdev for SF. I saw it yesterday and try to figure
> > out the history. Since you are here, guess you know something 😊
> >
> > [1] https://patchwork.ozlabs.org/project/netdev/cover/20191107160448.20962-
> 1-parav at mellanox.com/
> > [2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices
> 
> Kernel maintainers rejected the use of mdev for this purpose
> and suggested to use a real bus.
> You can follow the discussion here:
> https://lore.kernel.org/netdev/20191108205204.GB1277001@kroah.com/

OK. Thanks for the info.

> 
> Does Intel plan to use mdev for SF?

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel
Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to realize there
is difference between SF and ADI: SF considers multi-function devices which may include
net/regex/vdpa/... But ADI only focuses on the virtualization of the devices and splitting
devices to logic parts and providing huge number of interfaces to host APP. I think SF
also considers this but is mainly used for multi-function devices (like DPU in your term?
Correct me if I'm wrong).

And I also noticed that the mdev-based interface can only be used in userspace but aux-based
interface can also be used by other kernel sub-system (like for net, wrap it as netdev).

Thanks,
Chenbo

> 
> For the sake of follow-up discussion, this is the official mdev doc:
> https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt
> 
>