Blog

By Lincoln Lavoie, UNH-IOL, Principal Engineer

A little more than 3 years ago, we set out to improve the qualities and capabilities of the continuous integration (CI) testing of the DPDK source code and new patches.  Now, it’s not uncommon for testing to come last in the process, and sometimes it doesn’t receive it’s fair share of attention.  Which of us hasn’t worked on a project, with the boss pushing off the testing efforts or milestones, in order to allow “just that bit more” focus or expansion on the features or customer wishes for the project?  However, as DPDK has matured and is a key foundation for many applications and deployments, we need to ensure the stability and robustness of the codebase and incoming patches.

Enter the Community CI team, a virtual team of DPDK community members who meet regularly and help maintain all of the tooling that enables the testing activities.  We’re working to better organize and support the testing resources available to the DPDK development teams.  Today, there are 3 primary testing sources, running as TravisCI, Intel’s hosted lab, and the community lab hosted at the UNH-IOL. Each of the “labs” provides results back to the DPDK project’s patchwork tracking system, and tests different aspects of the “code,” ranging from compile and unit tests, to performance and functionality tests on bare-metal with real NICs and drivers. 

We thought it might be worthwhile to spend a few words on a blog to highlight what testing is already running, what testing is under development, and finally, where do we see things going from here.  Starting with the Travis CI systems, which are triggered from a github robot that is tracking commits on the main branch, running compile and unit testing across both x86 and aarch64 architectures, with cross compiles to 32-bit, all on top of the Ubuntu OS.  The robot and CI jobs for this integration have been developed and maintained largely by our Redhat peers, so, shoutouts where they’re due, “great job guys!”  You can check out the Travis CI builds here: https://travis-ci.com/github/DPDK

Next up, Intel’s hosted lab, where they are running compile testing across a number of different operating systems on x86 hardware.  The operating systems range from Linux distros, to FreeBSD, and even Windows.  Adding another dimension, the compilation is also checked using both Makefile and Meson.  The system is tracking both the main branch and as well as providing results for patches, before their merge onto the main, and provides early feedback to the patch submitter if issues pop up from a patch, afterall, “who knew” that approach didn’t work over all of DPDK’s supported OSes (well, now you know).  Intel’s support in providing both the hosted lab environment and keeping things running smoothly is a great help, “thanks Intel team!”  You can see a sample of the Intel hosted lab output in patchworks here: https://patchwork.dpdk.org/patch/75413/

Lastly, for the currently operating testing, is the community lab, hosted at the UNH-IOL.  Testing in the community lab is also tracking both the main branch and submitted patches, providing a combination of compile, unit testing, functional testing, and performance testing.  The compile and unit testing also add one additional dimension, with some cooperation to downstream projects, such as Open vSwitch and SPDK, providing early detection of a patch breaking outside of DPDK directly.  For the functional and performance testing, the lab hosts a number of “bare-metal” systems with NICs provided by DPDK member companies, including Broadcom, Intel, Mellanox, and NXP currently.  The performance testing runs a throughput test on each NIC (in some cases there are multiple speeds or hardware variants included), checking for a drop in forwarding performance from a well known reference.  Gold and Silver DPDK member companies are eligible to submit hardware for hosting in the community lab and are strongly encouraged to do so by the governing and tech boards of DPDK, as this directly expands the testing coverages.  Functional testing is the newest and growing aspect within the community lab output, adding more tests running on “bare-metal”, based on the DTS (DPDK Test Suite) project.  Aiming to provide majority coverage for the NIC features list.  The community lab results also feed directly into the patchwork system, with additional information available on the lab portal as well: https://lab.dpdk.org/results/dashboard/

Looking to the future, there are two efforts currently underway.  First, the robot that is tracking the github main branch is being extended to also drive builds on the OpenSUSE Build Service (OBS), further extending that coverage to more OS / build variants.  Second is the expansion of the feature testing coverage, which is focused on extending and updating the DTS coverage, with that work driving back into the functional testing run in the community lab.  

The Community CI team meets every other week, on Thursdays, at 1pm UTC.  All DPDK community members and maintainers are welcome to join and participate in the discussions, just drop a line to ci@dpdk.org.  We hope to see you there!

A new release is available:
        https://fast.dpdk.org/rel/dpdk-20.08.tar.xz

This version could be named Twins: 20.08 is released on 2020-08-08.
The other numbers are:

• 1175 commits from 159 authors
• 1199 files changed, 137604 insertions(+), 35611 deletions(-)

It is not planned to start a maintenance branch for 20.08. This version as previous ones are ABI-compatible with 19.11.

Below are some new features, grouped by category.

• General
  • external thread registration API
  • bit operations API for drivers
  • VFIO PF with VF token
  • new mempool ring modes (RTS/HTS)
• Networking
  • eCPRI offload with flow API
  • Tx scheduling offload
• Cryptography
  • crypto-CRC chained operation for DOCSIS protocol
• RegEx
  • new device class API for future RegEx drivers
  • Nvidia Mellanox RegEx driver for BlueField-2 DPU
• Tools
  • testpmd 5-tuple swap for L2/L3/L4
  • performance test application for flow rules
  • l2fwd forwarding between asymmetric ports

More details in the release notes:
        http://doc.dpdk.org/guides/rel_notes/release_20_08.html

There are 52 new contributors (including authors, reviewers and testers).
Welcome to Adrián Moreno, Archit Pandey, Bin Huang, Devendra Singh Rawat, Dodji Seketeli, Dongyang Pan, Evgeny Efimov, Francis Kelly, Gaurav Singh, Gregory Etelson, Guy Kaneti, Hongzhi Guo, Hrvoje Habjanic,
Jakub Chylkowski, Jecky Pei, Jeff Kirsher,  Kamil Bednarczyk,
Levend Sayar, Long Li, Lotem Leder, Maciej Hefczyc, Mao Jiang,
Mateusz Kowalski, Netanel Gonen, Nick Nunley, Nir Efrati, Parav Pandit,
Patrick Fu, Pavel Ivashchenko, Piotr Skajewski, Renata Saiakhova,
Roman Fridlyand, Roman Kapl, Sasha Neftin, Sergey Madaminov,
Shibin Koikkara Reeny, Shuanglin Wang, Shy Shyman, Simon Horman,
Stanislaw Grzeszczak, Surabhi Boob, Thierry Martin, Timothy McDaniel,
Todd Fujinaka, Vitaly Lifshits, Vivien Didelot, Wei Xie, Weifeng Li,
Yicai Lu, Yuval Avnery, Yuying Zhang, and Zhiguang He.

Below is the number of commits per employer (with authors count):
        362     Intel (45)
        276     Nvidia Mellanox (24)
        157     Broadcom (15)
         68     Huawei (10)
         64     Red Hat (5)
         51     NXP (8)
         50     ARM (5)
         49     Marvell (12)
         32     Microsoft (3)
         16     BIFIT (1)
          7     Emumba (2)
          6     Solarflare (1)
          6     Chelsio (2)
          5     OKTET Labs (4)

The new features for 20.11 can be submitted during one month.
DPDK 20.11 will be a big and busy release. Please check the schedule:
        http://core.dpdk.org/roadmap#dates

Thanks everyone, stay safe and keep up morale.

The DPDK Project Governing board held a “virtual” meeting on July 13th, welcoming both Doug Stamper from Microsoft and Marvell’s new representative, Prasun Kapoor, to the board. Bruce Richardson (of Intel) rotated into the Technical Board representative seat, previously held by Thomas Monjalon (of Mellanox/NVIDIA), in Q3. Note, the Technical Board seat is rotated among Technical Board members on a quarterly basis.

It was noted that 20.08 RC2 went out this week, and it’s shaping up to be a big release! The DPDK project is looking for more code reviewers. If you are an active contributor to DPDK, we invite you to help review your fellow contributors’ patches; help us collectively improve DPDK releases (team work makes the dream work). We would like to thank the entire DPDK developer community who does the hard work that makes our project what it is.

The board recognized the ongoing efforts by Stephen Hemminger (Microsoft) and project maintainers to adopt more inclusive code language, with the replacement of terms like “Blacklist-Whitelist” and “Master-Slave.” Inclusive language aims to avoid expressions and terms that could be perceived as racist, sexist, ableist, or generally biased, prejudiced or demeaning to any particular group of people. DPDK joins other open source groups, including the Linux Kernel community, in working towards improving the use of inclusive language blocks. We know these changes require significant work. We are appreciative to the commitment of our entire community to support the transition to more inclusive terminology in the upcoming 20.11 release.

Additional updates from the Governing Board:

Updated DPDK Charter: Following up on our earlier summer update, the Governing Board, in conjunction with the Technical Board, concluded its months-long review and revision of the DPDK Charter, presenting it for ratification to the Governing Board at the meeting this week. The Governing Board unanimously approved the charter revisions in their entirety.

The edits bring the Charter up-to-date and serve as a current reflection of the project as it is today. The latest charter is available at https://www.dpdk.org/charter/, with changes noted in the table. Please take a look.

Events: We are moving solely to virtual events this year. The DPDK Summit Userspace event will take place virtually September 22-23. Registration is now live, and we encourage you to register early. Be on the lookout for agenda updates.

Budget: The Board has updated the budget to reflect our real-time changes as a result of the impact of COVID-19.

Lab: A sub-team has been working with the DPDK Community Lab team, including UNH-IOL. We are making good progress to properly resource, fund, and improve our lab’s community benefit.

Member Outreach: As mentioned earlier this summer, the board has also started a sub-committee that is looking to grow the DPDK Project’s membership. If you have any suggestions for companies or individuals that we should reach out to, please let us know.

Thank you for your continued engagement with the DPDK community! We hope everyone is staying healthy and keeping busy during these uncertain times

We hope you are keeping safe and healthy, as we all adjust to the new normal we find ourselves in, with coronavirus and social unrest around us.  We’re proud of the leadership of our technical community to tackle our own issues and moving to more inclusive language, removing divisive terminology (e.g master/slave) that does not reflect our community values. Thank you for your leadership and work. It is greatly appreciated.

The DPDK project Governing Board, Technical Board, Marketing and other working groups have been busy over the past couple months. 

The Governing Board has held two meetings since the lock-downs went into effect, in April and early June, reassessing the project budget, marketing and event plans, while the Technical Board continued to focus on the technical priorities of DPDK, with the recent release of 20.05. Our community continues their stellar reputation of delivering our releases on time. Well done!

As we will not be able to meet in person for some time, and unable to provide the usual Governing Board readout on stage at one of the DPDK Summits, we wanted to provide a short written update, that we will continue on a regular basis moving forward. And we do apologize for not providing more timely governing board activity updates in the past. We will strive to improve and keep the communication open and timely. Do not hesitate to reach out should you have any questions. 

DPDK Welcomes New Members

Our community thrives through the work of the community and the financial support enabled by our members. We are pleased to share recent additions to the DPDK Project membership. The DPDK Project welcomes Microsoft as our newest Gold member, with Mr. Douglas Stamper joining the Governing Board, bringing his expertise with DPDK and the Windows ecosystem. Additionally, AMD joined as a Silver member. Welcome to both companies!

         

 

DPDK Userspace Summit 2020 Moves to a Virtual Format

Due to the prevailing coronavirus situation across the globe and related travel restrictions, we made the difficult decision to transition the DPDK Userspace Summit to a virtual experience, happening September 22-23. The Call for proposals and registration are now open. Note that the CFP closes July 12. Please consider submitting an abstract! We hope you will join us! 

DPDK will continue to monitor the prevailing situation and assess the feasibility of safely holding physical community gatherings. Please bear with us. 

• China and Japan events are on hold. It is currently looking unlikely we can make in-person events happen in 2020. 
• The US DPDK Summit event is pushed out to Q1 of 2021.

Strategic and Financial Update

It is of note that the DPDK project is in a financially sound position to see 2020 through, under the watchful gaze of the Governing Board’s Financial committee, chaired by Rashid Khan from Red Hat. 

The Governing Board reviewed and approved an updated 2020 budget including our anticipated 2020 expenses of $316k, deferring unused event allocations to our reserve fund, in anticipation of 2021 events and the Community Lab expansion. 

The Governing Board has noted the importance of member retention and recruitment, particularly in these uncertain times, with the formation of a Strategic Outreach working group, comprised of a sub-group of Governing Board members, tasked with ensuring the identity, focus, and charter of DPDK is clearly captured, published and communicated, along with a clear and compelling membership value proposition. 

Towards this end, the Governing Board in conjunction with the Technical Board, has spent the last month reviewing and updating the DPDK Charter, which has not seen any significant changes over the past 3 years. The edits will bring the Charter up to date and serve as a current reflection of the project as it is today.    

The DPDK Project has also launched a new project analytics dashboard, providing insights into development activity and contributions. We continue to be an active healthy community. Please try out the new tool. Let us know if you feel the data is accurate and reflects the community breadth of contributors.

Community Lab Expansion 

As Jim and Thomas shared in an earlier message to the community, we have been evaluating the value and role of the DPDK Community Lab, hosted at University of New Hampshire’s InterOperability Lab, (UNH IoL).  Upon the recommendations of the Technical Board, the lab continues to play a key role in regression testing. The Governing Board has approved up to $35,000 in additional investment to expand the test coverage and support provided to the community through the Community Lab. We anticipate increasing the depth and breadth of test cases available to the community.  

The Community Lab meets on a bi-weekly basis and is open to the DPDK community. We would encourage you to participate –  reach out to ci@dpdk.org to be added to the meeting calendar invite.

DPDK Marketing Activities

The marketing working group has a new project whitepaper and companion video series, to be published in the coming month.  Keep an eye out for it!

In 2020, DPDK celebrates 10 years as an active open source project from its formative years at Intel. We have come a long way since then, becoming the world’s leading open source packet processing tool kit.  Stay tuned for a fun look back across the past 10 years. 

Expect to see some exciting things happening over the next couple months. Stay tuned for more! And please let us know if you would like more information on any of these topics. 

 

A new DPDK release, 20.05,  is available here: https://fast.dpdk.org/rel/dpdk-20.05.tar.xz

It was quite a big release cycle, with:

•         1304 commits from 189 authors
•         1983 files changed, 145825 insertions(+), 29147 deletions(-)

There are no plans (yet) to start a maintenance branch for 20.05.

This version, as the previous one (20.02), is ABI-compatible with 19.11.

Below are some new features, grouped by category.

• General
  • packet processing graph
  • ring synchronisation modes for VMs and containers
  • RCU API for deferred resource reclamation
  • telemetry rework
  • low overhead tracing framework
  • GCC 10 support
• Networking
  •  flowaging API
  • driver for Intel Foxville I225
• Cryptography
  • ChaCha20-Poly1305 crypto algorithm
  • event mode in the example application ipsec-secgw
•  Baseband
  • 5G driver for Intel FPGA-based N3000

More details in the release notes: http://doc.dpdk.org/guides/rel_notes/release_20_05.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Andrea Arcangeli, Asim Jamshed, Cheng Peng, Christos Ricudis, Dave Burley, Dong Zhou, Dongsheng Rong, Eugeny Parshutin, Evan Swanson, Fady Bader, Farah Smith, Guy Tzalik, Hailin Xu, Igor Russkikh, JP Lee, Jakub Neruda, James Fox, Jianwei Mei, Jiawei Wang, Jun W Zhou, Juraj Linkeš, Karra Satwik, Kishore Padmanabha, Lihong Ma, Lijian Zhang, Linsi Yuan, Louise Kilheeney, Lukasz Wojciechowski, Mairtin o Loingsigh, Martin Spinler, Matteo Croce, Michael Haeuptle, Mike Baucom, Mit Matelske, Mohsin Shaikh, Muhammad Ahmad, Muhammad Bilal, Nannan Lu, Narcisa Vasile, Niall Power, Peter Spreadborough, Przemyslaw Patynowski, Qi Fu, Real Valiquette, Rohit Raj, Roland Qi, Sarosh Arif, Satheesh Paul, Shahaji Bhosle, Sharon Haroni, Sivaprasad Tummala, Souvik Dey, Steven Webster, Tal Shnaiderman, Tasnim Bashar, Vadim Podovinnikov, Venky Venkatesh, Vijaya Mohan Guvva, Vu Pham, Wentao Cui, Xi Zhang, Xiaoxiao Zeng, Xinfeng Zhao, Yash Sharma, Yu Jiang, Zalfresso-Jundzillo, Zhihong Peng, Zhimin Huang, and Zhiwei He.

Here is a breakout of patches per company contribution:

The new features for 20.08 may be submitted during the next 17 days.  DPDK 20.08 should be released in early August, in a tight schedule:  http://core.dpdk.org/roadmap#dates

Thanks to the entire DPDK community, and happy birthday to our RTE_MAGIC!

Greetings DPDK Project Community:

 We hope you are managing okay and remaining safe and healthy as the coronavirus crisis has thrown the world into a frenzy.  While many of us are used to working remotely and using online collaborative tools, we are having to adapt to new circumstances at home. It’s hard on everyone.  

 The DPDK Governing Board, Technical Board, and Marketing Committee have been hard at work locking down our final 2020 budget, crafting plans around worldwide DPDK Summit events, and exploring hosting a broader data plane projects developer event.  However, upcoming public gatherings and events have been cancelled or postponed and are no longer an option in the near term. DPDK has not yet migrated to virtual events. For now, our plans around DPDK Summit events are in flux:

•  China and Japan events are on hold. As things develop further we will see if we can make something happen, perhaps in Q3 or Q4. 

• The US DPDK Summit event is  pushed out to Q1 of 2021. The team made this decision before the pandemic emerged in order to create some  calendar space between the Summit and Userspace events. We have targeted the second half of September for the Userspace event, continuing to host it in Bordeaux.  As of now this is still our intent. But as we’re sure you can understand, the situation remains fluid until the pandemic subsides. Please bear with us.

 We have also continued to explore options to increase the value and role of the DPDK Community Lab that is currently hosted at the University of New Hampshire, InterOperability Lab, also known as UNH/IoL. We’d like to acknowledge Lincoln Lavoie and team’s continued efforts here, especially as he remains in the lab to triage hardware issues. However, we’ve struggled to find community volunteers to step up and lead the lab effort.  We are now looking at alternatives including a minimum set of activities and regression tests that the Technical Board recommends we maintain. Other efforts to expand the use of the lab will likely require additional budget. We are trying to converge on a solution. Help is still welcome should anyone want to get more involved.

The community continues to be amazing.  This year we’ve delivered our 20.02 release with few time accommodations.  Work is underway for the 20.05 release with several new features and improvements. https://core.dpdk.org/roadmap/ 20.08 and 20.11 will follow.  We are confident the community collaboration and execution capability will serve us well during these times.  We are not expecting too much delay in releases. Of note, while most git maintainers do have a back-up in place, please let us know if you are able to help fill in any gaps. 

Overall we want to ensure you prioritize yourself and your family; your personal health and safety are paramount to us. Rest assured the DPDK community is here for you – whether it be  help on a features, code, or guidance on how to handle a challenging home dilemma, please do reach out. The DPDK family is strong and tremendously supportive. We are glad to be part of it.

Stay safe and healthy.

All the best,

Thomas and Jim

 

The latest DPDK release, 20.02, is now available:  https://fast.dpdk.org/rel/dpdk-20.02.tar.xz

The statistics show  a post-LTS ABI-compatible release:

        820 commits from 156 authors

        876 files changed, 59941 insertions(+), 22565 deletions(-)

There are currently no plans (yet) to start a maintenance branch for 20.02. This release and the next ones (20.05, 20.08) are ABI-compatible with 19.11.

Below are some new features, grouped by category.

General:

•         ABI check tooling
•         FreeBSD 13 support
•         Linux kernel modules disabled in default build
•         API wait until equal
•         use CPU lcore with ID > total max
•         ring with configurable element size
•         pool of mbufs with pinned external buffers

Networking:

•         Pensando ionic driver
•         Marvell driver for OCTEON TX2 end point
•         Mellanox vDPA driver
•         rte_flow API for DSCP
•         rte_flow API for L2TPv3 over IP

Cryptography:

•          Marvell OCTEON TX2 inline IPsec
•          CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
•          synchronous CPU crypto API
•         more algorithms: ECDSA, ECPM

Applications:

•         event mode in l3fwd example

More details are available in the  technical release notes: http://doc.dpdk.org/guides/rel_notes/release_20_02.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Abed Kamaluddin, Abhishek Marathe, Adam Ludkiewicz, Adrian Podlawski, Aleksandr Loktionov, Alexander Kozyrev, Alfredo Cardigliano, Andrew Pinski, Apeksha Gupta, Balakrishna Bhamidipati, Carolyn Wyborny, Chandu Babu N, Cheng Jiang, Chengchang Tang, Damian Milosek, Dariusz Chaberski, Dariusz Jagus, Dexuan Cui, Dmitry Kozlyuk, Donald Lee, Dzmitry Sautsa, Eugenio Pérez, Fang TongHao, Gal Cohen, Gargi Sau, Girish Nandibasappa, Ilja Van Sprundel, Itsuro Oda, Jaroslaw Gawin, Jörg Thalheim, Kiran Patil, Krzysztof Galazka, Kumar Amber, Li Feng, Lijun Ou, Mahipal Challa, Manish Chopra, Marcin Formela, Martyna Szapar, Mateusz Rusinski, Michael Baum, Michal Litwicki, Michal Swiatkowski, Narcisa Vasile, Niclas Storm, Pandi Kumar Maharajan, Piotr Azarewicz, Piotr Kwapulinski, Piotr Pietruszewski, Prateek Agarwal, Praveen Shetty, Rafael Ávila de Espíndola, Ricardo Roldan, Robert Konklewski, Satananda Burla, Savinay Dharmappa, Scott Wasson, Selwin Sebastian, Shannon Nelson, Shiri Kuzin, Sunil Pai G, Sylwia Wnuczko, Thomas Faivre, Tomasz Konieczny, Vitaliy Mysak, Xuan Ding, Xuan Li, Xueming Zhang, and Yisen Zhuang.

Below is the percentage of patches per company:

Based on Reviewed-by and Acked-by tags, the top reviewers are:

     76     Viacheslav Ovsiienko <viacheslavo@mellanox.com>

     56     Qi Zhang <qi.z.zhang@intel.com>

     47     Xiaolong Ye <xiaolong.ye@intel.com>

     46     Akhil Goyal <akhil.goyal@nxp.com>

     44     Ferruh Yigit <ferruh.yigit@intel.com>

     43     Matan Azrad <matan@mellanox.com>

     43     Jerin Jacob <jerinj@marvell.com>

     38     Beilei Xing <beilei.xing@intel.com>

     35     Qiming Yang <qiming.yang@intel.com>

     34     Ajit Khaparde <ajit.khaparde@broadcom.com>

     33     Maxime Coquelin <maxime.coquelin@redhat.com>

     31     Ori Kam <orika@mellanox.com>

     25     Gavin Hu <gavin.hu@arm.com>

     24     David Marchand <david.marchand@redhat.com>

     23     Konstantin Ananyev <konstantin.ananyev@intel.com>

The new features for 20.05 may be submitted during the next 21 days, in order to be reviewed and integrated before mid-April. DPDK 20.05 should be released on 20-05-20.  http://core.dpdk.org/roadmap#dates The schedule is pushed because of the special worldwide context. Some future release cycles will have to be shorter.

Thanks to the entire DPDK community!

 

Our 2020 event planning is taking longer than usual, given the current situation with the Novel CoronaVirus. We are working diligently to solidify plans for events throughout 2020 and will have information soon.

In the interim, please read the below excerpt from the Linux Foundation’s official response regarding the situation and potential impact across all Linux Foundation events:

The Linux Foundation is continuously monitoring the Novel Coronavirus situation and is committed to ensuring the safety of our event participants and staff at our upcoming events. We will be following all recommended guidelines from the Centers for Disease Control (CDC) and the World Health Organization (WHO) as the situation progresses.

The Linux Foundation wants to assure our community and all event participants that their health and well being are of the utmost importance to us. We will continue to regularly check the latest official information and updates from all major health agencies leading up to our events and will make any necessary changes based on that information. We recognize that much is unknown at this time, and will strive to make decisions based on facts rather than fear. We encourage our event participants to do the same. To all those who are being affected, please accept our sincerest sympathies.

Please read the full update, which includes specific recommended action for onsite events, here: https://events.linuxfoundation.org/2020/02/12/novel-coronavirus-update/

A new major release is available:
https://fast.dpdk.org/rel/dpdk-19.11.tar.xz

The statistics are quite impressive:
1611 commits from 199 authors
1914 files changed, 164270 insertions(+), 44609 deletions(-)

The branch 19.11 should be supported for at least two years,
making it recommended for system integration and deployment.
The maintainer of this new LTS is Luca Boccassi.

The new major ABI version is 20.
The next releases 20.02, 20.05 and 20.08 will be ABI compatible with 19.11.

Below are some new features, grouped by category.
General:
– new ABI policy and versioning
– API reworks in preparation of 1 year without ABI breakage
– mempool objects not crossing pages
– dynamic mbuf layout
– eBPF arm64 JIT
– LTO build
Networking:
– Hisilicon hns3 driver
– NXP PFE driver
– virtio packed ring optimizations
– ethdev hairpin API
– ethdev flow tag API
– ethdev packet type range API
– ethdev LRO packet size API
– RIB/FIB libraries
– KNI with VA
Cryptography:
– Marvell Nitrox driver
– Marvell OCTEON TX2 crypto driver
– session-less asymmetric crypto
– IPsec SAD API
Applications:
– IOAT example
– l2fwd-event example
– several examples are removed

More details in the release notes:
http://doc.dpdk.org/guides/rel_notes/release_19_11.html

There are 70 new contributors (including authors, reviewers and testers).
Welcome to Abhishek Sachan, Adrian Moreno, Alvin Zhang,
Amaranath Somalapuram, Anand Sunkad, Andy Gospodarek, Anna Lukin,
Antara Ganesh Kolar, Bing Zhao, Bo Chen, Chengchang Tang,
Chengwen Feng, Chenxu Di, Chunsong Feng, Cristian Bidea, Doug Dziggel,
Feifei Wang, Guinan Sun, Hao Chen, Heinrich Kuhn, Hongbo Zheng,
Huisong Li, Igor Chauskin, Ivan Ilchenko, Jeremy Plsek, Jerry Hao OS,
Jiaqi Min, Jin Yu, Junfeng Guo, Junyu Jiang, Kommula Shiva Shankar,
Lin Li, Lu Qiuwen, Lunyuan Cui, Manish Tomar, Marcin Baran,
Md Fahad Iqbal Polash, Michael Pfeiffer, Michael Shamis,
Min Hu (Connor), Min Wang (Jushui), Mitch Williams, Nagadheeraj Rottela,
Paul Atkins, Pawel Modrak, Phanendra Vukkisala, Priyanka Jain,
Rahul Shah, Rajesh Ravi, Scott W Taylor, Shougang Wang,
Shuki Katzenelson, Subrahmanyam Nilla, Sucharitha Sarananaga,
Sunila Sahu, Sylvain Rodon, Tony Nguyen, Vakul Garg, Venkat Duvvuru,
Venkateshwarlu Nalla, Wangyu (Eric), Wei Hu (Xavier), Xiaobing Zhang,
Xiaofeng Deng, Xiaoyun Wang, Xun Ni, Yahui Cao,  Yilong Lv, Yu Zhang,
and Zengmo Gao.

Below is the number of patches per company (with authors count):
559     Intel (73)
209     Mellanox (16)
173     Marvell (23)
170     NXP (10)
119     Broadcom (7)
77     Red Hat (8)
61     Huawei (9)
49     OKTET Labs (4)
41     ARM (7)
35     Microsoft (3)
21     6WIND (6)
17     Solarflare (1)
13     Chelsio (1)
12     Cisco (3)
9     IBM (1)
5     AMD (1)

Based on Reviewed-by and Acked-by tags, the top reviewers are:
154     Xiaolong Ye, Intel
118     Ferruh Yigit, Intel
113     Akhil Goyal, NXP
85     Matan Azrad, Mellanox
78     Maxime Coquelin, Redhat
63     Ajit Khaparde, Broadcom
62     Viacheslav Ovsiienko, Mellanox
58     Qi Zhang, Intel
58     Jerin Jacob, Marvel
52     Qiming Yang, Intel
48     Somnath Kotur, Broadcom
44     Thomas Monjalon, Mellanox
39     Hemant Agrawal, NXP
36     Nipun Gupta, NXP
33     David Marchand, Redhat
32     Bruce Richardson, Intel
32     Andrew Rybchenko, SolarFlare
31     Gavin Hu, Arm
29     Luca Boccassi, Debian
29     Konstantin Ananyev, Intel
25     Ori Kam Mellanox
24     Anatoly Burakov, Intel
22     Olivier Matz, 6 Wind

The new features for 20.02 may be submitted during the next 12 days,
in order to be reviewed and integrated before mid-January.
DPDK 20.02 should be released on Valentine’s Day.
http://core.dpdk.org/roadmap#dates

Thanks everyone, and happy Thanksgiving to our American friends!

 

A Q&A with Rashid Khan, DPDK Board Member and Director, Networking at Red Hat

Rashid Khan is the Director of Networking development at Red Hat. His team’s responsibilities include upstream and downstream development of Linux kernel networking, DPDK, OVS OVN, NFV, MPTCP, ebpf XDP. Before joining Red Hat, Rashid managed the Linux team at Broadcom consumer products division. There he was in charge of board support drivers team, silicon bring up, and application deployment. Prior to that Rashid has approximately 20 years of networking, telco, voice over IP, signal processing experience at various  startups and huge conglomerates. 

What is Red Hat’s history with DPDK? How long have you been involved in the community?

Red Hat is an open-source Linux company that participates in the development of the Linux operating system, and provides support to its customers via Red Hat’s Linux distribution (RHEL). About four years ago, Red Hat noticed a lot of traction for the Data Plane Development Kit (DPDK) project to get traffic from network interface cards into virtual machines (VMs) and containers, with better packet speeds and development, resulting in faster time-to-market. Allowing new features and new NICs to work with Linux kernel is a bit hard, because Linux maintains the principle of not breaking things that are already working. DPDK offered a new approach; it did not come with the same constraints, and it offered an easier way to move network traffic, add new drivers for new cards, and add more features quickly.

DPDK provided a method for getting packets faster to VMs, and by helping accelerate networking for its virtualization products.  

What was the primary use case that got you started with DPDK?

Red Hat  primarily uses DPDK with Open vSwitch to get packets routed within servers from Network Interface Cards (NIC), to VMs (we leverage DPDK as the foundation for our NFV stack).

Can you talk a little more about some of these (and other) use cases?

Sure, there are a few I can touch on:

• Telcos are used to custom SW/HW builds from vendors,  but because of increasing consumer demand for new features without a willingness to pay more, there is a strong need to go with commodity hardware and open-source software, and to put applications on top through virtual machines and containers. So right now, we’re investing heavily in solving those issues and getting network traffic to VMs, containers and pods as efficiently as possible. 
• We’re also investing time and resources into the hardening of DPDK itself, taking it to the next level if you will. We’ve worked really hard to get our CI tests working at the University of New Hampshire’s Interoperability Lab (UNH-IOL). We can detect regressions in DPDK-OVS and alert the submitters of patches. HW vendors collaborating very closely with us on this include Intel, Mellanox, Netronome, and Broadcom. 

So what are some of the challenges DPDK is experiencing, and how do you think they can be addressed?

• DPDK is being used extensively everywhere, and that is great. Now we need to collaborate to take it to the next level of maturity. For example, the community needs to collaborate to find a way to continue adding advanced features without compromising the stability of existing HW and existing features. Stability of Long Term Releases (LTS) has to be solidified.  DPDK will have to adapt to cater to that — number of releases, duration of releases, integration with other products and projects will have to be evaluated. But the community is up for the challenge! There are enough senior/experienced people who are putting the right steps in place to solve for this. 
• Another challenge is that there are a lot of consumers of DPDK, but not very many active participants. Red Hat, Mellanox, and Intel are really stepping up and putting in resources, but we’d love to see more organizations participating. There are folks working full time on LTSs, and Red Hat has actually hired head count just for DPDK: we have about a half-dozen people focused on DPDK upstream, plus 8-9 working with DPDK for product integration.
• There’s also the challenge of debuggability.  The more we add to DPDK the harder it is to debug systems running DPDK.  And larger customers demand bug fixes plus an absolute root cause for each bug. That requires debuggability of every layer and complete predictability. In some cases adding debug code moves the problem, or may even alleviate the problem. Unfortunately, that is not enough for large telco customers. They need to know the root cause of the problem, and how a patch fixes that problem directly. More work is needed to get DPDK to that level.  As products mature, more of these “Day 2” problems need to be addressed. The same is the case with DPDK; the time has come to get the project to the next level.
  
• The next challenge for DPDK is to integrate with smart(er) NICs to enable HW offloads so main motherboard CPUs inside the servers are not doing the heavy lifting of the packet movement. We are working with multiple HW vendors to address that challenge. 

How can someone get involved in DPDK? 

There are so many ways to get involved! There are no impediments in joining — it’s easy to participate. You can start by making DPDK work for your specific hardware, enhance CI activities, collaborate on UNH lab, patch review, or just start by participating in online meetings. There are no barriers to entry – joining mailing lists and conference calls is a good first step. Does not have to be a large commitment in the beginning, you can start with just a few hours a week. 

And again –  we see a lot more consumers of DPDK than we do participants. Some of the biggest benefits of participation are to (selfishly) safeguard your own hardware and products, but to also have a voice in the overall direction of the project. And the more diverse the community, the better – that’s the spirit of open source.