This year’s DPDK North America Summit highlighted the projects ongoing technical excellence and innovation in high-performance networking. The event gathered experts and enthusiasts from around the globe, including project pioneers and new community contributors. They engaged in discussions and demonstrations focused on recent code developments, the technical board’s plans, and notably, exciting use cases and applications.
Over the past 14 years, DPDK has methodically developed an open stack that meets a broad variety of user requirements. The project’s open development approach and adaptability to community needs have been invaluable. They showcase the project’s commitment to open source principles. However, these practices have also led to a more expansive and less streamlined code base. Nevertheless, the technical board skillfully manages the necessary compromises for both core developers and end users, highlighting some exciting developments at the summit.
The review of new technology integrations and organic implementation alongside the community’s evolution has been impressive. The project’s development, impact, and extensive application across a global infrastructure have been significant. This includes not just data centers, enterprise cloud services, and network equipment, but also transportation networks, telecom systems, financial trading platforms, industrial control systems, and even particle processors, and astronomical data processing!
The project has reached a pivotal stage of maturity, with use cases and applications expanding dynamically. This evolution presents an opportune moment to explore and showcase real-world applications, extending far beyond its conventional roles in routers and firewalls.
One highlight of the summit was the presentation by Robin Jarry and David Marchand from Red Hat, who introduced “Grout,” a graph router based on DPDK. This tool is designed to simulate network functions and physical routers to replicate the behavior of typically closed-source VNFs and CNFs using an open source tool. They provided a detailed explanation of the rte_graph library’s role in data path processing and showcased Grout’s capabilities.
Another notable session was led by Dr. John Romein from the Netherlands Institute for Radio Astronomy (ASTRON). He discussed how advanced GPU technologies, specifically the NVIDIA Grace Hopper Superchip, are being utilized to process vast amounts of data from radio telescopes. This session not only emphasized the integration of DPDK with GPU technology but also demonstrated its real-world applications in astronomical data processing, pushing the boundaries of modern hardware capabilities.
Each session, from discussions on the challenges of implementing DPDK on non-cache coherent platforms by Hemant Agrawal and Gagandeep Singh from NXP, to insights into machine learning inference within network processing by Srikanth Yalavarthi from Marvell, highlighted the versatility and robustness of DPDK. These discussions underscored its ability to meet the increasingly complex demands of network performance solutions.
In summary the latest DPDK summit provided a platform for learning and sharing, reinforcing the community’s commitment to driving innovation in network performance through open development and governance, as highlighted by the ongoing initiatives of the project.
Watch all the presentations on the DPDK youtube here
Have a use case you’d like to share and feature on the website? Email marketing@dpdk.org
DPDK’s latest release, 24.03, marks the culmination of months of dedicated work and collaboration. Accessible now for download here, this release showcases the efforts of 154 authors, culminating in 987 commits. The extensive changes include 1,334 files being modified, with 79,260 insertions and 22,824 deletions, highlighting the substantial growth and refinement in the project.
Notably, DPDK 24.03 will not have a separate maintenance branch, and it maintains ABI compatibility with the preceding 23.11 version, ensuring a smooth transition for developers.
New Features and Enhancements
The 24.03 release introduces several significant features and improvements, including:
A new argument parsing library to streamline application development.
Standardization of dynamic logging for enhanced diagnostics.
Introduction of the HiSilicon UACCE bus, expanding hardware support.
Enhanced query capabilities for transmission queues.
Advanced flow matching capabilities including random and field comparison.
NAT64 flow action, offering new network translation options.
Flow template table resizing for more efficient resource management.
Continued preparations for MSVC build support, indicating a commitment to broader compatibility.
An expansion of DTS tests and general cleanups to enhance stability and performance.
For a detailed overview of all updates and improvements, read the release here.
Community Growth and Acknowledgments
This release also celebrates the inclusion of 31 new contributors, ranging from authors to reviewers and testers, showcasing the DPDK community’s continuous growth. Contributions spanned across a diverse group of companies, with significant commits from Marvell, NVIDIA, Intel, and many others, reflecting the wide-ranging industry support for DPDK.
Looking Ahead
The project is now looking towards version, 24.07, expected in July. The community is encouraged to submit new feature proposals in the upcoming weeks, aligning with the project’s roadmap available here.
Additionally, the upcoming webinar Hyperscaling in the cloud offers an excellent opportunity for learning and engagement. Interested participants can register here.
Shout Out
A big thank to all courageous people who reviewed other’s work. Based on Reviewed-by and Acked-by tags, the top non-PMD reviewers are:
50 Akhil Goyal @Marvel 44 Ferruh Yigit @AMD 40 Chengwen Feng @Huawei 36 Anoob Joseph @Marvel 32 Morten Brørup @Smartsharesystems 26 Tyler Retzlaff @Linux_Microsoft 21 Dariusz Sosnowski @NVIDIA 18 Ori Kam @NVIDIA 18 Bruce Richardson @INTEL
Let me first explain why I decided to submit a patch…
Recently, I was working on a research project that utilized the DPDK framework. During the process, I realized that the framework’s driver did not implement all the functionalities specified by the network card. Since one particular functionality, the “launch time feature”, was essential for my work, I spent some time modifying the driver for my project to use this feature.
After that, I decided to invest some time integrating this patch into the DPDK framework, basically for two main reasons. First, I believe this feature is essential for real-time community to use DPDK, and I have seen several discussions online about it, suggesting that its implementation could benefit many others. Secondly, I had never contributed to a large open-source project before and saw this as an opportunity to learn the basic process of submitting patches.
1. Preparation
Before submitting the patch, it’s crucial to understand the project’s development process. DPDK is an open-source project currently managed by Linux Foundation. The development of DPDK happens on the mailing list (like Linux kernel) instead of on GitHub.
This means we can’t simply use git commit to submit patches, then followed by a pull request. Instead, patches are submitted via email to maintainers. Therefore, some extra preparatory steps are needed before submission, including:
Registering and subscribing to the DPDK mailing list
Configuring email to use git-send-email
Identifying the maintainers and the subtree of the module you are modifying
Mailing List: The process of registering and subscribing to the mailing list is relatively straightforward and can be done by following the steps in the DPDK official documentation. I won’t repeat that here. I also recommend registering for Patchwork to easily track the code review process. From my experience, it’s best to disable receiving all the emails from the mailing list or set the digest mode to only receive one summary per day, otherwise the daily generated patches will mix with the regular email and overflow the mailbox…
Email Configuration: Setting up the email is also quite simple. I used a Google email, and configured it following this article to link git send-email. I didn’t encounter any issues during the process.
Finding Maintainers: DPDK is a large, modular open-source project, designed for collaborative work. Each module has a designated maintainer, and some even have specific maintenance branches. Therefore, it’s necessary to find the maintainer and the branch of the module you want to modify, and submit the patch to them for integration into the mainline repository. An important point to note is to use the script devtools/get-maintainer.sh in the repository to automatically find the maintainers to Cc, just like this:xxxxxxxxxxgit send-email –to dev@dpdk.org –cc-cmd devtools/get-maintainer.sh
The first time, I attempted to find maintainer information in the list provided in the documentation and manually added it to the CC field. However, I later discovered that this information was outdated, which led me to inadvertently send the patch to a previous maintainer…
As for understanding the project’s coding style, I suggest not spending too much time studying the official style guide. This is because you can ensure the consistency of the coding style with the automated review scripts mentioned later.
2. Writing the Patch
The process of writing the patch is essentially modifying the original code to fix bug or add new functionalities, and after formatting it with diff, these modifications become the patch. An important suggestion here is to disable the auto-formatting feature of your IDE. Many projects have unique code formatting styles, and auto-formatting might mess up the format with Tab/Space changes everywhere.
2.1 Testing
Testing is divided into three main parts:
Compilation and functional testing
Code style check
ABI Policy
Compilation and Functional Testing: Since my modifications were minor, I only compiled it on my local machine. The specific process is largely similar to installing DPDK, except the first step involves using meson setup --wipe build to clear the previous build directory. For functional testing, I recommend testing all DPDK built-in apps that might be affected, such as dpdk-testpmd.
Code Style Check: This part can be done following the Section 5.7 Checking the Patches in the DPDK official tutorial. It’s important to note that you need to download checkpatch.pl yourself (You can get the source code by Google searching, just a simple script). After generating codespell-dpdk.txt as instructed, run devtools/checkpatches.sh patch_name.patch in the same directory. The output log shows the results, where both errors and warnings need to be corrected, but some checks may not need changes if they are unreasonable.
ABI Policy: ABI Policy refers to the requirements for variable and function naming. It is recommended to self-check against the official ABI Guideline. But personally, I feel if no new file is created, there shouldn’t be many issues, just compare with the surrounding code to get a sense of the appropriate naming conventions.
2.2 Submitting the Patch
After testing, it is good to generate the patch. Before doing so, it’s recommended to double-check your git username and email, using git config --global user.name & git config --global user.email.
My method of generating patches might not be the best practice. I initially used git add & git commit for a simple commit, then git commit --amend to modify the commit message. Finally, I generated the patch with git format-patch -1 -o ~/patch/. Here -1 indicates the patch-set includes the last few commits, but I only needed to generate one patch.
Before submission, check if the patch contains the Signed-off-by: xxxxx <xxxxxx@xxx> line. Mine wasn’t generated automatically, so I added it manually.
Finally, the patch can be submitted via git send-email, typically --to the mailing list dev@dpdk.org, and --cc to maintainers.xxxxxxxxxxgit send-email \–to dev@dpdk.org \–cc-cmd devtools/get-maintainer.sh \ ./patch_name.patch
Before sending, it is best to first send it to yourself to check if the email format is correct with git send-email --to=[your email] patch_name.patch.
If you have registered for Patchwork, you’ll receive the patch submission notification after a few hours.
3. Code Review
Compared to smaller open-source projects, code review in DPDK is quite stringent, involving both automated reviews from the system and manual reviews from maintainers.
The results of the automated review are received via email within a few hours of submission. Fortunately, despite this being my first patch submission, no errors were detected with beginner luck.
In the subsequent manual review phase, maintainers pose questions about the content of the patch, which feels similar to the rebuttal phase of a research paper submission process. The main issues raised in my case focused on:
Explaining the functionality of different lines of code
Justifying the chosen solution
Providing test results
Responding directly to these questions is usually sufficient. A few minor details to note:
When replying, use the --in-reply-to parameter to specify which email you are responding to.
Pay attention to formatting. Indicate references to previous emails by adding >.
In the subject, add RE: before the title of the patch. Remember, no matter how many times you reply, there should only be one RE:, not a chain like RE: RE: RE:.
4. Lessons
Although the process seemed somewhat complicated to me initially, the friendly atmosphere made it smoother. From submission to the final review, it took me over ten days, which included three replies and one code update. Along the way, I encountered some technical issues and uncertainties. However, these were resolved with the help of the maintainers and other developers, who provided guidance and support throughout the process.
The biggest takeaway for me from this experience is that contributing to a large open-source project is not as daunting as it might seem. It’s often the intricate rules and the stereotypes about open-source communities that dissuade people from getting involved. However, I found that the community was welcoming, and the process, while detailed, was manageable when you are willing to spend some time on it.
Although contributing to the open-source community is not my focus, attempting to submit a patch was definitely a valuable experience.
DPDK now supports the HiSilicon UACCE (Unified/User-space-access-intended Accelerator Framework) bus driver, enabling accelerator devices like compressors, cryptos, DMA, and ethernet devices to be seamlessly integrated and registered within DPDK applications.
1.2. Argument Parsing Library
The addition of the argparse library simplifies the development of user-friendly applications by replacing the usage of getopt(), enhancing usability and flexibility in application design.
1.3. Improved RSS Hash Algorithm Support
Enhancements to the RSS hash algorithm support include the addition of rte_eth_find_rss_algo function, allowing for easier retrieval of RSS hash algorithms by name.
1.4. Flow Matching Items
New flow matching items such as RTE_FLOW_ITEM_TYPE_COMPARE and RTE_FLOW_ITEM_TYPE_RANDOM have been introduced, enabling more granular packet field comparison and random value matching in flow processing.
1.5. Flow Template Table Resizing
The ability to resize flow template tables dynamically has been added, providing greater flexibility in managing flow capacities and configurations.
1.6. Updated Drivers
Several DPDK drivers have been updated to include support for new devices and optimize performance. For example, the Marvell cnxk net driver now supports additional flow items like RTE_FLOW_ITEM_TYPE_PPPOES and RTE_FLOW_ACTION_TYPE_SAMPLE, enhancing its capabilities.
2. Removed Items
Certain items, like statically defined logtypes, have been removed from DPDK, emphasizing the shift towards dynamic logtype registration and improved code organization.
3. API Changes
Several API changes have been implemented to align with intended design and improve code robustness, including the removal of typeof(type) from RTE_DEFINE_PER_LCORE macros and the use of C11 static_assert in RTE_BUILD_BUG_ON.
4. ABI Changes
No ABI changes have been made in this release that would break compatibility with the previous version, ensuring seamless migration and backward compatibility.
In summary, DPDK Release 24.03 delivers a significant boost to network acceleration and development efficiency, making it a crucial update for developers and network engineers alike.
