[dpdk-dev] kernel binding of devices + hotplug

[Bruce Richardson]

On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> Hi all
> 
> From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > To: Bruce Richardson <bruce.richardson at intel.com>; Thomas Monjalon
> > <thomas at monjalon.net>
> > Cc: dev at dpdk.org; pmatilai at redhat.com; david.marchand at 6wind.com;
> > jia.guo at intel.com; Matan Azrad <matan at mellanox.com>;
> > konstantin.ananyev at intel.com; stephen at networkplumber.org;
> > fbl at redhat.com
> > Subject: Re: kernel binding of devices + hotplug
> > 
> > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > >> It's time to think (again) how we bind devices with kernel modules.
> > >> We need to decide how we want to manage hotplugged devices with
> > DPDK.
> > >>
> > >> A bit of history first.
> > >> There was some code in DPDK for bind/unbind, but it has been removed
> > >> in DPDK 1.7 -
> > >>
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
> > >>
> > k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > %7C6ea5
> > >>
> > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > C0%7
> > >>
> > C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > x8H
> > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > >> "
> > >> 	The bind/unbind operations should not be handled by the eal.
> > >> 	These operations should be either done outside of dpdk or
> > >> 	inside the PMDs themselves as these are their problems.
> > >> "
> > >>
> > >> The question raised at this time (4 years ago) is still under discussion.
> > >> Should we manage binding inside or outside DPDK?
> > >> Should it be controlled in the application or in the OS base?
> > >>
> > >> As you know, we use dpdk-devbind.py.
> > >> This tool lacks two major features:
> > >> 	- persistent configuration
> > >> 	- hotplug
> > >>
> > >> If we consider that the DPDK applications should be able to apply its
> > >> own policy to choose the devices to bind, then we need to implement
> > >> binding in the PMD (with EAL helpers).
> > >>
> > >> On the other hand, if we consider that it is the system
> > >> responsibility, then we could choose systemd/udev and driverctl.
> > >>
> > >> The debate is launched!
> > >>
> > >
> > > Allow me to nail my colours to the mast early! :-)
> > >
> > > I believe it's system not application responsibility.
> > > I also believe I have previously explained my reasons for that choice
> > > in some of the previous email threads.
> > 
> > For what it's worth, I tend to agree, if only because writing code for what is
> > essentially a bunch of read/write/filesystem enumeration in C is extremely
> > fiddly and error prone :) IMO things like this are better handled either by
> > scripts, or by tools whose sole purpose is doing exactly that (or both).
> > 
> > I like having scripts like devbind in DPDK because we can tailor them to our
> > use cases better, and having them is amenable to automation, but while I
> > wouldn't be opposed to removing them altogether in favor of some external
> > tool (systemd/udev/driverctl/whatever), in my humble opinion moving them
> > back into EAL or even PMD's would be a mistake.
> > 
> 
> Since the application runs in the system by a command of the system user I think the responsibility is for the user.
> The DPDK user forwards the control of some devices to the DPDK application using the EAL whitelist\blacklist mode to specify the devices,
> Any DPDK PMD should know which binding it needs to probe\control the device and can apply it,
> So, if the user asks to control on a device by DPDK application it makes sense that the application will do the correct binding to the device since the user wants to use it(no need to ask more operation of pre binding from the user).

Completely agree that it is ultimately up to the user. However, what I
don't want to see is the case where the user always has to specify a big
long list of device whitelist and blacklist options to each run of an
application. Instead, if device management is done at the system level via
udev (for example) configured via devicectl, then the application
commandline can be vastly simplified. It also allows better usability
across systems, since the same commandline can be used on multiple systems
with different hardware, with the actual device management rules having
been already configured at system install/setup time in udev.

> 
> Regarding the conflict of system rules for a device, it is again the user responsibility, whatever we will decide for the binding procedure of DPDK application the user needs to take it into account and to solve such like conflicts.
> One option is to remove any binding rules of a DPDK device in the DPDK application initialization and adjust the new rules by the PMDs, then any conflict should not disturb the user.

If the device management is only managed in one place, i.e. not in DPDK,
then there is no conflict to manage.

> 
> In current hot-plug case the application will need to do a lot of work to bind\remap devices in plug-in\plug-out events while the PMD could have all the knowledge to do it. 

At the cost of duplicating a lot of code between PMDs.

> 
> One more issue with the script is that the user should do different bind per device, in case of PMD responsibility the user can forget it:
> Think about that, any time the user wants to switch\add new supported nic it should update the script usage and to do per nic operation contrary to the DPDK principles.
> 

The udev rules syntax should provide adequate capabilities here for us to
match the correct binding behaviour. No need to have it in DPDK too.

/Bruce