[dpdk-dev] [PATCH 4/4] pmd_hw_support.py: Add tool to query binaries for hw support information

[Panu Matilainen]

On 05/23/2016 04:55 PM, Neil Horman wrote:
> On Mon, May 23, 2016 at 02:56:18PM +0300, Panu Matilainen wrote:
>> On 05/20/2016 05:06 PM, Neil Horman wrote:
>> [...]
>>> Look, I think we're simply not going to agree on this issue at all.  What about
>>> this in the way of compromise.  I simply am not comfortable with automatically
>>> trying to guess what hardware support will exist in an application based on the
>>> transient contents of a plugin directory, because of all the reasons we've
>>> already gone over, but I do understand the desire to get information about what
>>> _might_ be automatically loaded for an application.  what if we added a 'plugin
>>> mode' to pmdinfo. In this mode you would dpecify a dpdk installation directory
>>> and an appropriate mode option.  When specified pmdinfo would scan librte_eal in
>>> the specified directory, looking for an exported json string that informs us of
>>> the configured plugin directory.  If found, we iterate through all the libraries
>>> there displaying hw support.  That allows you to query the plugin directory for
>>> available hardware support, while not implying that the application is
>>> guaranteed to get it (because you didn't specifically state on the command line
>>> that you wanted to know about the applications hardware support).
>>
>> That brings it all to one tiny step away from what I've been asking: have
>> the plugin mode automatically locate librte_eal from an executable. So I'm
>> not quite sure where the compromise is supposed to be here :)
> The compromise is that I'm not willing to quietly assume that a given
> application linked to the dpdk library in /usr/lib64/dpdk-<version>, will get
> hardware support for the cxgb4, mlx5 and ixgbe pmds, because those DSO's are in
> the exported RTE_EAL_PMD_PATH.

Why not? I dont get it.

> With this method, you at least have to tell the
> pmdinfo application that I wish to scan that path for pmds and report on
> hardware support for whatever is found there.  Thats a different operation from
> getting a report on what hardware an application supports.  i.e. its the
> difference between asking the questions:
>
> "What hardware does the application /usr/bin/foo support"
> and
> "What hardware support is available via the plugin DSO's pointed to by the dpdk
> version in /usr/lib64/dpdk-2.2"

Well, for the application to be able to load any PMDs, it will have to 
be linked to some version of librte_eal, which will have to be somewhere 
in the library search path (or use rpath).


> I feel its important for users to understand that autoloading doesn't not
> guarantee support for the hardware that is autoloaded to any application.  My
> compromise is to provide what your asking for, but doing so in a way that
> attempts to make that differentiation clear.

I would think requiring a specific option to enable the plugin scan 
should be quite enough to make that point clear.

>>
>> I do appreciate wanting to differentiate between "physically" linked-in and
>> runtime-loaded hardware support, they obviously *are* different from a
>> technical POV. But its also a difference an average user might not even know
>> about or understand, let alone care about, they just want to know "will it
>> work?"
>>
>
> Which average user are we talking about here?  Keep in mind the DPDK is
> anything but mainstream.  Its a toolkit to implement high speed network
> communications for niche or custom purpose applications.  The users of tools
> like this are going to be people who understand the nuance of how
> applications are built and want to tune them to work at their most efficient
> point, not some person just trying to get firefox to connect to digg.  I think
> its reasonable to assume that people who are running the tool have sufficient
> knoweldge to understand that DSO's and static binaries may embody hardware
> support differently, and that things which are loaded at run time may not be
> reported at scan time (by way of corressponding example, though its not perfect,
> people seem to understand that iscsid supports lots of different HBA's, but the
> specific hardware support for those HBA's isn't codified in /usr/sbin/iscsid,
> but rather in the modules under /lib/modules/<kversion/...).  I understand thats
> a lousy comparison, as the notion of static linkage of the kernel doesn't really
> compare to static application linkage, but still, people can figure out whats
> going on there pretty readily, and I think they can here too.
>
> As to your other comment, yes, the end users just want to know "will it work".
> But from a philisophical standpont, I don't agree with your assertion that
> we can answer this question by doing what you're asking me to do.  The answer to
> "Will my application support the hardware on this system with the plugins found
> in RTE_EAL_PMD_PATH?" is "I don't know".  Thats because you don't know what the
> contents of that directory will be when the application is run later.

Come on. RTE_EAL_PMD_PATH is expected to point to a system directory 
owned by root or such, stuff just doesn't randomly come and go. 
Everything is subject to root changing system configuration between now 
and some later time.

> The only things we can tell at the time we run pmdinfo are:
>
> 1) What the hardware support of a static binary is for its linked in libraries
>
> 2) What the hardware support of a dynamic binary is via its DT_NEEDED entries
>
> 3) What the hardware support of a specific PMD DSO is
>
> I am fundamentally opposed to trying to guess what hardware support will be
> loaded dynamically via dlopen methods when an application is run at a later
> time, and firmly believe that it is more consistent to simply not report that
> information in both the static and dynamic case, and educate the user about how
> to determine hardware support for dynamically loaded PMD's (perhaps a man page
> would be worthwhile here)

Well, with the plugin mode and pmd path export in your v3 patches 
(thanks for that!) all the necessary pieces are there so fishing out the 
information is a rather trivial one-liner of a script now:

---
#!/bin/sh

/usr/share/dpdk/tools/pmdinfo.py -p $(ldd $1 |awk '/librte_eal/{print $3}')
---

I fail to understand how having a separate script to handle this is 
better than having pmdinfo do it directly when run in the plugin mode, 
but as you wish.

	- Panu -