[dpdk-dev] Did we reduce unnecessary linkage too well?

[Christian Ehrhardt]

Hi,
I was finally getting to more deeply re-validate Openvswitch 2.6 together
with DPDK 16.07. And I think I found a whiplash of our effort to reduce
unnecessary hard linkage.

Trying to avoid cross-posting, picking DPDK list and the main involved
people on TO/CC.

TL;DR:
- pmd drivers are no more "auto"-loaded
- adding -d ...so to all consuming applications feels obnoxious
- do we really have to intentionally overlink some?
- I hope I just overlook something trivial to fix this.


Detail:

Essentially in the past openvswitch was linked to "the one" combined
library being libdpdk.so.
Due to all the great work on separating the libraries for versioning and
reducing overlinking I now have an openvswitch that only depends on the
core dpdk libs.

$ldd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk | grep rte
       librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1
       librte_pmd_ring.so.2 =>
/usr/lib/x86_64-linux-gnu/librte_pmd_ring.so.2
       librte_mbuf.so.2 => /usr/lib/x86_64-linux-gnu/librte_mbuf.so.2
       librte_vhost.so.3 => /usr/lib/x86_64-linux-gnu/librte_vhost.so.3
       librte_mempool.so.2 => /usr/lib/x86_64-linux-gnu/librte_mempool.so.2
       librte_meter.so.1 => /usr/lib/x86_64-linux-gnu/librte_meter.so.1
       librte_ring.so.1 => /usr/lib/x86_64-linux-gnu/librte_ring.so.1
       librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2
       librte_kvargs.so.1 => /usr/lib/x86_64-linux-gnu/librte_kvargs.so.1

As we all know that is due to the build of openvswitch now utilizing the
linker script which is
$cat /usr/lib/x86_64-linux-gnu/libdpdk.so
GROUP ( librte_power.so librte_pmd_null_crypto.so librte_ip_frag.so
librte_pmd_e1000.so librte_acl.so librte_port.so librte_mbuf.so
librte_pmd_virtio.so librte_kvargs.so li
brte_cryptodev.so librte_reorder.so librte_pmd_ring.so librte_eal.so
librte_distributor.so librte_pmd_pcap.so librte_pmd_af_packet.so
librte_pipeline.so librte_pdump.so lib
rte_pmd_vhost.so librte_jobstats.so librte_pmd_enic.so librte_mempool.so
librte_pmd_ixgbe.so librte_cmdline.so librte_meter.so librte_timer.so
librte_kni.so librte_pmd_vmxn
et3_uio.so librte_ring.so librte_pmd_fm10k.so librte_cfgfile.so
librte_pmd_xenvirt.so librte_pmd_i40e.so librte_pmd_null.so librte_lpm.so
librte_pmd_ena.so librte_vhost.so
libethdev.so librte_pmd_bnxt.so librte_sched.so librte_table.so
librte_pmd_bond.so librte_hash.so librte_pmd_cxgbe.so )

You see it lists the PMDs, but since this is more a plugin structure there
are no references by openvswitch to them. So they don't make it into the
eventual binary.
Which is kind of what we wanted - avoid over-linking right?

But this leads to IMHO unexpected behaviour. For example I have a system
properly prepared with two ixgbe cards and start OVS-DPDK in "the right
way" I get the issue of not having a valid PMD driver.
"could not open network device dpdk0"
=> https://launchpadlibrarian.net/287083118/ovs-vswitchd-bad.log

Now this can work if one tells EAL to load the proper driver, so after
$ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=[...] -d
/usr/lib/x86_64-linux-gnu/librte_pmd_ixgbe.so"
Everything works fine now and as-expected.
=> https://launchpadlibrarian.net/287083132/ovs-vswitchd-good.log
One can see ixgbe pmd driver loading and taking over the devices.

I think this puts too much pressure to know the right drivers to the admin
- so we can't really make this the way it should be right?
It feels obnoxious having to add a full path to a .so file to a ovsdb
parameter.

It might be worth to note that I can reproduce the same behavior with
l2fwd, without the -d to the lib on the same systemit quits with:
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye
I considered this a test/developer tool where it is kind of ok.
But now I think it is time to discuss it in the scope of more
customer-centric dpdk consuming applications like openvswitch+dpdk.

The build logs of how the tested openvswitch and dpdk were build are here -
just in case one of you want/can spot an issue causing all this in there:

   - openvswitch (all of it, not just openvswitch-dpdk):
   https://launchpadlibrarian.net/284431032/buildlog_ubuntu-yakkety-amd64.openvswitch_2.6.0~git20160912.dc61b4e-0ubuntu4_BUILDING.txt.gz
   - dpdk-16.07:
   https://launchpadlibrarian.net/285321110/buildlog_ubuntu-yakkety-amd64.dpdk_16.07-0ubuntu3_BUILDING.txt.gz


I haven't seen nor found a discussion on this - nor people testing my
packages complainning so far -, but find it so weird that I start to think
it is just me or my broken setup :-)
So I guess there is hope that there is something obvious I overlook -
better that than needing major reworks.
Like a default search dir for the libs that I mismatch in my setup or
anything like it?
If not, should we do something - I hate to say that - to intentionally
overlink dpdk libraries to get all pre-provided PMDs autoloaded?

Kind Regards,

-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd