[dpdk-dev] Beyond DPDK 2.0

[Matthew Hall]

On Apr 25, 2015, at 5:10 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
> I'm more focused on why that level of participation is not higher

Hi Neal,

This mail is probably way too long, but here is what I saw about participation, in my case I used DPDK on two projects so far:

1) proprietary project for a L4-L7 stateful replay DPI firewall performance tester at a large network test and measurement corporation using the old original 6WIND DPDK 1.X under NDA before it became available to a wider channel

2) open-source self-created threat intelligence sensor project using DPDK 1.7.X, under development in my spare time

I think the things that making more DPDK contributions are ironically more technical and social in nature than legal or bureaucratic in nature as one would normally suspect, and as you theorized in your original mail. Let me go through some things and see what people think.

At the social level, there are not very many people in the world who are familiar and comfortable with the LKML style embedded coding workflow (heavy mailing list usage, sending and reviewing patches in emails, putting specific people in To or Cc to get patches approved and ACKed in subtrees, etc.) I happen to know some if not most of this tribally, because I used Linux since 1997, but very few developers have any clue about this stuff. However I never participated in the actual LKML flow any further than tester / bug reporter, and I actually use DPDK very deliberately, to avoid fighting with the headaches of the linux-net code and flamewars.

This is why I was proposing that we try to find a way to allow contributions via Github or Bitbucket... their fork-and-pull model is much simpler for outsiders to comprehend and make quick patches when they find little bugs or issues as they integrate with the library... given we are BSD licensed, the low barrier to entry is the ultimate way to keep the patch velocity as high as possible, and keep the community going.

At the technical level, I see two or three difficulties:

1) A lot of the various performance enhancements one can use are kind of "magical" or "jigsaw puzzle" and not presented in a unified way, where I can methodically go through a checklist and enable everything my app should use even though I have no clue what they all are. For example, 1) let's talk about hashing... there is RSS hashing (symmetric or asymmetric), JHASH, CRC hash, ... not sure how many different ones. 2) Let's talk about CPU models... SSE, SSE4, SSE4.1, SSE4.2, etc. I don't know what I have myself, much less what my users will have, much less what I actually need or should use, without guidance from some processor people. 3) Let's talk about PCIe bus... there is DCA, some other non-DCA acceleration that's faster if you are on the same NUMA node as the PCIe slot, but fails to work if you aren't... etc.

A lot of the people from Intel, 6WIND, and the kernel people are just thinking "this stuff is obvious... we've used it since 200X and it's 2015!" That's true if you are a processor / kernel hacker... but if you spent your whole career on packet processing or network security like a lot of us app developers might have done, that's very orthogonal to Intel-specific and compiler-specific and hardware-specific performance hacks... so a lot of us have no real clue how to configure and test them all, much less enhance them and make some patches to it. We just blindly trust 6WIND and Intel to get it right, because as far as we can see, all the DPDK code is pretty clean and readable, and we're pretty sure we don't know anything better than what's already coded and put into the repository, if we don't have some checklists to follow to enable and test every combination, and find any more bugfixes to suggest.

2) A lot of the network adapters DPDK uses, especially when you begin using the more crazy accelerations, are either hard to obtain or expensive... for example from what I saw in my jobs, the 10 gigabit boards were a minimum ~$250 USD in manufacturing quantities. The 2-port gigabit latest-gen Intel board I got was $100 USD... I think a lot of higher-ed students and overseas people from less developed nations might have a hard time paying for some of this stuff to start hacking... some kind of program to get these sort of people some sample gear might help.

I also hit difficulties with the virtio-net driver... it doesn't work with the virtio-net adapter in Virtualbox... which makes it harder for me to use cool, convenient environments configured by the tool Vagrant, to make very simple dev VM environments to quickly get new hackers up to speed on my open source running just a command or two, and by extension harder for me to show them why they should use DPDK for all their cool new network ${GADGET}s.

The same difficulty comes into place if I wanted to do some performance patches... I don't have the money to buy the VTune profiler for my spare time project, that you would want if you're going to get in there and start dumping out all the CPU time and cache counters, so I could help you guys to squash the performance bugs more quickly by testing the behavior in a real packet processing app and not just some random microbenchmark that doesn't do any real work on real stuff.

3) The DPDK library itself only covers Layer 2 which makes it pretty esoteric for average Joes. This meant that I had to implement ARP, NDP, IPv4, IPv6, UDP, and TCP myself, which was very educational but probably not super efficient for attracting community attention. All except TCP weren't too bad... TCP was a ton of work... I had to go to the university library downtown in my city and get hold of Tanenbaum and some of the other networking textbooks and write quite a lot of paper notes, just to implement enough of the server half of TCP to process the traffic and pull out the data stream... and as Jay Rolette can attest since he gave me pointers... I cheated like hell to do it, totally ignoring reordering and retransmissions... but I am proud to say it seems to work great for how half-baked of a job I did coding it. I would gladly trade a first-born child for a better TCP implementation available with the DPDK. ;)

But if I look at the world of coders out there... a lot of guys aren't going to have any idea what to do with an L2-only network stack that doesn't support the two most popular Transport protocols out of the box... they only know about what to do with TCP and UDP, not how to redo those themselves from the ground up. In my career I only knew a handful of guys who would be good enough and patient enough to go pound that stuff out... of course if I start my own company someday that's precisely who I'd try to hire, but there aren't so many of these people to go around.

Matthew.