benthomas, Author at DPDK

May 01

DPDK Long Term Stable (LTS) Release 22.11.05

By benthomas Blog

The latest DPDK Long Term Stable (LTS) Release 22.11.05 includes several updates and enhancements across various components of the DPDK framework. Significant changes in this release involve numerous file modifications, which are indicated by a large number of insertions and deletions across the codebase. The release notes document extensive additions, suggesting improvements and new features in the areas of network interface controllers (NICs), cryptographic devices, event devices, and baseband processing.

Notable adjustments were made to the build system, documentation, and driver support for various hardware. Improvements in error handling, memory management, and device operation stability are also reflected in the release notes. The release also addresses various bug fixes and performance enhancements to ensure better stability and efficiency.

Contributors

The update involved 246 files with a total of 3,235 insertions and 2,053 deletions. A big shout out to all the contributors to this release including:

Ajit Khaparde, Akhil Goyal, Akshay Dorwat, Alan Elder, Alex Vesker, Ali Alnubani, Andrew Boyer, Anoob Joseph, Arkadiusz Kusztal, Bing Zhao, Bruce Richardson, Chaoyong He, Chengwen Feng, Ciara Power, Dariusz Sosnowski, David Marchand, Dengdui Huang, Edwin Brossette, Eli Britstein, Emi Aoki, Erez Shitrit, Ferruh Yigit, Fidel Castro, Flore Norceide, Ganapati Kundapura, Gregory Etelson, Hamdan Igbaria, Hanumanth Pothula, Hao Chen, Harman Kalra, Hernan Vargas, Holly Nichols, Huisong Li, Jie Hai, Jonathan Erb, Joyce Kong, Kaiwen Deng, Kalesh AP, Kevin Traynor, Kiran Kumar K, Kishore Padmanabha, Kommula Shiva Shankar, Konstantin Ananyev, Kumara Parameshwaran, Long Li, Luca Boccassi, Maayan Kashani, Masoumeh Farhadi Nia, Maxime Coquelin, Michael Baum, Mingjin Ye, Morten Brørup, Mário Kuka, Neel Patel, Nithin Dabilpuram, Pavan Nikhilesh, Pengfei Sun, Qi Zhang, Qian Hao, Radu Nicolau, Rahul Bhansali, Rakesh Kudurumalla, Robin Jarry, Rongwei Liu, Satheesh Paul, Shai Brandes, Shaowei Sun, Shihong Wang, Shun Hao, Simei Su, Sivaprasad Tummala, Sivaramakrishnan Venkat, Stephen Hemminger, Suanming Mou, Sunil Kumar Kori, Sunyang Wu, Tom Jones, Viacheslav Ovsiienko, Wathsala Vithanage, Weiguo Li, Yajun Wu, and Yunjian Wang.

These contributors addressed various aspects from software fixes and performance enhancements to security improvements across multiple components of the system.

Download it here: DPDK 22.11.5

The git tree for this version can be accessed here: DPDK Stable 22.11

May 01

Love0

DPDK’s Role in Hyperscaling

By benthomas Blog

In the rapidly evolving digital landscape, hyperscaling in the cloud has emerged as a critical strategy for businesses aiming to scale their operations efficiently. The webinar, “Hyperscaling in the Cloud,” hosted by Honnappa Nagarahalli (Arm), from the DPDK Tech Board, brings together industry experts to discuss how the Data Plane Development Kit (DPDK) is revolutionizing hyperscale cloud environments.

The Webinar Panelists

The webinar featured three distinguished panelists:

1. Brian Denton: A Senior Program Manager at Microsoft Azure, Brian brings a wealth of experience in Azure’s host networking. He shared insights into Azure’s implementation of DPDK, emphasizing its use in enhancing Ethernet, and overall network performance.

2. Rushil Gupta: As a Senior Software Engineer at Google, Rushil highlighted the critical role of DPDK in financial technology (Fintech) applications on Google Cloud Platform (GCP). His discussion focused on achieving consistency, performance, and reliability in high-frequency trading platforms.

3. Jim Thompson: Co-founder of Netgate, Jim delved into the use of DPDK in networking applications outside the traditional cloud domain. His contribution illuminated the versatility of DPDK across different cloud environments and its impact on virtual private networks (VPNs).

Insights from the Webinar

DPDK in Azure’s Cloud Networking

Brian Denton’s presentation offered a glimpse into how Microsoft Azure leverages DPDK to offload packet processing from the CPU to dedicated hardware. This approach significantly reduces latency and improves throughput, enabling Azure to offer enhanced performance for virtual machines (VMs) and networking services.

Brian shared valuable insights into how DPDK has been instrumental in Azure’s network infrastructure, particularly highlighting its impact on Azure’s host networking and the broader ecosystem of partners and customers. He explained that Azure has integrated DPDK to address the need for high-speed packet processing, which is crucial for a wide range of applications, from basic web services to complex, latency-sensitive tasks like real-time analytics and high-frequency trading.

One of the key points Brian made was about the technical architecture that enables Azure to leverage DPDK’s capabilities. He detailed how DPDK is used in conjunction with Azure’s hardware, such as SmartNICs, to offload and accelerate network functions traditionally handled by software. This hardware-software synergy, as Denton explained, not only reduces CPU overhead but also significantly decreases latency, providing Azure customers with improved network performance and efficiency.

Furthermore, Brian highlighted real-world applications of DPDK in Azure, illustrating how partners and customers utilize DPDK for scenarios that require minimal latency and maximum throughput. He also discussed the continuous evolution of Azure’s networking stack, underscored by the introduction of new hardware and the ongoing optimization of DPDK to meet the growing demands of cloud computing.

Some examples:

Clearent by Xplor used Azure SQL Database Hyperscale to revamp its merchant transaction reporting system. Previously operating on in-house systems, Clearent, which handles over 500 million transactions annually, shifted to a cloud-based setup. This move significantly boosted their ability to process and report data.

Protocall Services, a provider of telephonic crisis and behavioral health digital tools, embarked on a cloud migration journey to enhance the reliability, security, and scalability of its IT infrastructure.

DPDK’s Impact on Fintech Applications

Rushil Gupta’s discussion on the use of DPDK in financial technology (Fintech) applications, particularly in the realm of high-frequency trading (HFT) platforms on Google Cloud Platform (GCP), sheds light on how bleeding-edge network processing technologies are evolving financial markets.

In the fast-paced world of HFT, where milliseconds can equate to millions of dollars, the need for ultra-low latency and high throughput is paramount. Traditional cloud networking approaches may falter under such demanding requirements due to the involvement of kernel-based networking stacks that introduce additional latency. Here, DPDK’s bypass of the kernel networking stack, allowing direct access to network hardware, presents a compelling solution. This direct path significantly reduces latency and increases packet processing speed, enabling HFT platforms to operate at the speed required to capitalize on fleeting market opportunities.

Rushil illustrates how Google leverages DPDK to empower fintech customers on GCP, providing them with the infrastructure necessary to achieve the high throughput and low-latency communication essential for HFT platforms. One notable application is in the construction of complex event processing (CEP) systems, which are at the heart of many trading platforms. These systems analyze and act upon market data in real-time, necessitating the rapid processing capabilities that DPDK facilitates.

Rushil discusses the role of DPDK in enhancing data replication and recovery processes within fintech applications. In an industry where data integrity and availability are critical, DPDK’s efficiency in handling large volumes of data packets ensures that financial institutions can maintain robust data replication frameworks. This capability not only supports the high availability demands of trading platforms but also aids in achieving regulatory compliance related to data persistence and recovery.

Rushil explained how DPDK’s application in fintech on GCP demonstrates the technology’s pivotal role in enabling HFT and other financial services to meet their stringent performance and reliability criteria. With DPDK, Google provides a competitive edge to fintech applications, facilitating new levels of speed and efficiency in financial markets.

Some examples:

1. CME Group. As one of the world’s leading derivatives marketplaces, CME Group leverages GCP and DPDK for enhanced market data analytics and to facilitate high-speed trading. Their partnership aims to accelerate CME Group’s move to the cloud, transforming the global markets ecosystem with cloud-based innovation and scaling capacity dynamically to meet market demands.

2. Talos. Specializing in digital asset trading technology, Talos utilizes GCP’s infrastructure to support its trading platform. With DPDK, Talos benefits from reduced latency and increased throughput, essential for executing trades and managing orders across multiple exchanges and liquidity pools efficiently.

3. Clowd9. This cloud-based trading technology provider uses GCP to offer a scalable and secure platform for trading firms and financial institutions. DPDK supports Clowd9’s need for high performance and low latency in executing trades, managing risk, and processing real-time market data.

4. Freetrade. Freetrade, an investment platform, leverages GCP to power its app, offering users commission-free trading. GCP’s global infrastructure and DPDK’s network optimization capabilities ensure that Freetrade can manage high volumes of transactions and data analysis.

5. TD Securities Automated Trading (TDSAT): TDSAT uses GCP for trading fixed-income bonds, benefiting from DPDK’s high-performance packet processing capabilities. This enables TDSAT to execute trades at high speed and with precision, critical for maintaining competitiveness in the fixed income market.

These customers and use cases underscore the importance of DPDK in enhancing network performance on GCP, making it an ideal platform for capital market applications that demand high throughput, low latency, and scalability. By leveraging GCP and DPDK, capital market firms innovate and adapt quickly to market changes, manage risks more effectively, and unlock new opportunities for growth.

Broadening DPDK’s Application Scope in VPNs and Software Routers

Jim Thompson’s insights during the DPDK webinar shed light on how DPDK is leveraged in cloud networking through the lens of Netgate’s product, TNSR (pronounced ‘Tensor’). This serves as a case study of DPDK’s implementation outside its traditional use cases. TNSR, a virtual router developed by Netgate, underscores the adaptability and robustness of DPDK in addressing specific cloud networking challenges.

In cloud environments, networking demands can quickly escalate due to the sheer volume of data transfer and the need for secure connections. Traditional VPN solutions often fall short due to bandwidth limitations and the number of tunnels they can support. Jim highlighted how these constraints could hinder the scalability and performance of cloud-based services. This scenario is particularly relevant for large organizations that require extensive interconnectivity across various cloud environments.

The introduction of TNSR as a DPDK-powered solution exemplifies how DPDK’s high-performance packet processing capabilities can be extended beyond typical use cases to solve complex cloud networking problems. By utilizing DPDK’s efficient polling mode drivers (PMDs) for network and cryptography offload, TNSR significantly enhances throughput and reduces latency in VPN connections.

Jim explained how TNSR facilitates seamless connectivity between on-premise networks and cloud regions, highlighting the importance of VPN connections for secure data transfer. He underscored the limitations of existing cloud VPN solutions, such as bandwidth caps and tunnel number restrictions, which can significantly hamper large organizations’ networking needs. By leveraging DPDK, TNSR bypasses these limitations, providing a more flexible and scalable solution for cloud-based networking.

Take a look at Netgate’s customer stories here.

Conclusion

The webinar underscored DPDK’s pivotal role in enabling hyperscaling in the cloud. By providing a high-performance packet processing framework, DPDK not only enhances network efficiency but also opens new avenues for application development across various industries. As cloud architectures continue to evolve, the collaboration between cloud providers, technology firms, and the open source community will be vital in harnessing the full potential of DPDK.

Join in the Hyperscaling discussion and the community on slack here.

May 01

Love0

Unleashing Network Performance with Microsoft Azure MANA and DPDK

By benthomas User Stories

Introduction

In the modern cloud computing era, network performance and efficiency are paramount. Microsoft Azure has been at the forefront of this revolution, introducing innovative solutions like the Microsoft Azure Network Adapter (MANA) and integrating the Data Plane Development Kit (DPDK) to enhance the network capabilities of Azure virtual machines.

In this user story we interview Brian Denton, and Matt Reat, Senior Program Managers for Azure Core. Brian’s role has been pivotal, focusing on engaging with all network virtual appliance partners to ensure they are prepared and supported for the introduction of a new Network Interface Card (NIC) into Azure.

Matt’s journey at Microsoft began primarily within the networking domain. His career commenced with network monitoring before transitioning, about four years ago, into what is referred to as the host networking space. This area encompasses the SDN software stack and hardware acceleration efforts aimed at enhancing customers’ ability to utilize an open virtual network (OVN) and improve their overall experience on Azure.

A natural progression of his work has involved spearheading innovations in software and the development of hardware, which have recently been introduced to the public as Azure Boost. Additionally, his contributions include the development of the MANA NIC, a product developed in-house at Microsoft.

The Genesis of Azure MANA

Azure MANA represents a leap in network interface technology, designed to provide higher throughput and reliability for Azure virtual machines. As the demand for faster and more reliable cloud services grows, Azure’s response with MANA smartNICs marks a significant milestone, aiming to match and surpass AWS Nitro-like functions in network and storage speed acceleration.

Microsoft’s strategy encompasses a comprehensive approach, with a primary focus on hardware acceleration from top to bottom. This effort involves current work being conducted on the host and in the hypervisor (Hyper-V), aiming to advance hardware capabilities. Such initiatives are also being pursued by competitors, including AWS with its Nitro system and Google with a similar project, marking Microsoft’s contribution to this competitive field.

Behind the scenes, the team implemented several enhancements that remained undisclosed until the announcement of Azure Boost last July. This development compelled them to reveal their progress, especially with the introduction of the MANA NIC, which had been concealed from customer view until then.

The introduction of the new MANA NIC, boasting ratings of up to 200 Gbps in networking throughput, represents a significant enhancement of the current Azure offerings, in-line with Microsoft’s competition. The reliance on off-the-shelf solutions proved to be cost-prohibitive, prompting a shift to a fully proprietary, in-house solution integrated with their Field-Programmable Gate Array (FPGA).

DPDK’s Role in Azure’s Network Evolution

DPDK offers a set of libraries and drivers that accelerate packet processing on a wide array of CPU architectures. Microsoft Azure’s integration of DPDK into its Linux Virtual Machines (VMs) is specifically designed to address the needs of applications that demand high throughput and low latency, making Azure a compelling choice for deploying network functions virtualization (NFV), real-time analytics, and other network-intensive workloads.

The technical essence of DPDK’s acceleration capabilities lies in its bypass of the traditional Linux kernel network stack. By operating in user space, DPDK enables direct access to network interface cards (NICs), allowing for faster data plane operations. This is achieved through techniques such as polling for packets instead of relying on interrupts, batch processing of packets, and extensive use of CPU cache to avoid unnecessary memory access. Additionally, DPDK supports a wide range of cryptographic algorithms and protocols for secure data processing, further enhancing its utility in cloud environments.

Azure enhances DPDK’s capabilities by offering support for a variety of NICs optimized for use within Azure’s infrastructure, including those that support SR-IOV (Single Root I/O Virtualization), providing direct VM access to physical NICs for even lower latency and higher throughput. Azure’s implementation also includes provisions for dynamically managing resources such as CPU cores and memory, ensuring optimal performance based on workload demands.

Microsoft’s commitment to DPDK within Azure Linux VMs underscores a broader strategy to empower developers and organizations with the tools and platforms necessary to build and deploy high-performance applications at scale. By leveraging DPDK’s packet processing acceleration in conjunction with Azure’s global infrastructure and services, users can achieve the highest possible performance on Azure.

Enhancing Cloud Networking with Azure MANA and DPDK

Azure MANA and DPDK work in tandem to push the boundaries of cloud networking. MANA’s introduction into Azure’s ecosystem not only enhances VM throughput but also supports DPDK, enabling network-focused Azure partners and customers to access hardware-level functionalities. When introducing a new Network Interface Card (NIC), it is essential to have support for the Data Plane Development Kit (DPDK). The primary concern is that Azure customers will begin to encounter Mana NICs across various Virtual Machine (VM) sizes, necessitating support for these devices. This situation highlights a notable challenge.

The scenario involves three NICs and two Mellanox drivers requiring support, indicating a significant transition. The introduction of this new NIC and its drivers is intended for long-term use. The goal is for the MANA driver to be forward-compatible, ensuring that the same driver remains functional many yearsfrom now, without the need to introduce new drivers for new NICs with future revisions, as previously experienced with ConnectX and Mellanox.

The objective is a long-term support driver that abstracts hardware changes in Azure and the cloud affecting guest VMs, offering a steadfast solution for network I/O. Although the future specifics remain somewhat to be determined, the overarching aim is to support the features available on Azure, focusing on those needs rather than the broader spectrum of Mellanox’s customer requirements. Some features necessary for Azure may not be provided by Mellanox, and vice versa. Thus, the ultimate goal is to support Azure customers with tailored features, ensuring compatibility and functionality for the long term.

Microsoft offers a wide array of networking appliances that are essential to their customers’ architectures in Azure. Therefore, part of their effort and emphasis on supporting DPDK is to ensure our customers receive the support they need to operate their tools effectively and achieve optimal performance.

Supporting DPDK is essential to accommodate those toolsets. Indeed, maximizing the use of our hardware is also crucial. This is an important point because there’s potential for greater adoption of DPDK.

Matt Reat, Senior Program Manager at Microsoft

Typically, Microsoft’s users, mainly those utilizing network virtual appliances, leverage DPDK, and they are observing increased adoption not only among Microsoft’s Virtual Academy’s but also among customers who express intentions to use DPDK. It’s not limited to virtual appliance products alone. They also have large customers with significant performance requirements who seek to maximize their Azure performance. To achieve this, leveraging DPDK is absolutely essential.

The Technicals of MANA and DPDK

The MANA poll mode driver library (librte_net_mana) is a critical component in enabling high-performance network operations within Microsoft Azure environments. It provides specialized support for the Azure Network Adapter Virtual Function (VF) in a Single Root I/O Virtualization (SR-IOV) context. This integration facilitates direct and efficient access to network hardware, bypassing the traditional networking stack of the host operating system to minimize latency and maximize throughput.

By leveraging the DPDK (Data Plane Development Kit) framework, the MANA poll mode driver enhances packet processing capabilities, allowing applications to process network packets more efficiently. This efficiency is paramount in environments where high data rates and low latency are crucial, such as in cloud computing, high-performance computing, and real-time data processing applications.

The inclusion of SR-IOV support means that virtual functions of the Azure Network Adapter can be directly assigned to virtual machines or containers. This direct assignment provides each VM or container with its dedicated portion of the network adapter’s resources, ensuring isolated, near-native performance. It allows for scalable deployment of network-intensive applications without the overhead typically associated with virtualized networking.

Overall, the technical sophistication of the MANA poll mode driver library underscores Microsoft Azure’s commitment to providing advanced networking features that cater to the demanding requirements of modern applications. Through this library, Azure ensures that its cloud infrastructure can support a wide range of use cases, from web services to complex distributed systems, by optimizing network performance and resource utilization.

“The MANA poll mode driver library, coupled with DPDK’s efficient packet processing, allows us to optimize network traffic at a level we couldn’t before. It’s about enabling our customers to achieve more with their Azure-based applications.”

Matt Reat, Senior Program Manager at Microsoft

The setup procedure for MANA DPDK outlined in Microsoft’s documentation provides a practical foundation for these advancements, ensuring that users can leverage these enhancements with confidence. Furthermore, the support for Microsoft Azure Network Adapter VF in an SR-IOV context, as implemented in the MANA poll mode driver library, is a testament to the technical prowess underlying this integration.

Performance Evaluation and Use Cases

Evaluating the performance impact of MANA and DPDK on Linux VMs highlights significant improvements in networking performance. Azure’s documentation provides insights into setting up DPDK for Linux VMs, emphasizing the practical benefits and scenarios where the combination of MANA and DPDK can dramatically improve application responsiveness and data throughput.

Microsoft effectively utilizes the Data Plane Development Kit (DPDK) on the host side to optimize network performance across its Azure services. This approach not only supports customer applications by enhancing the speed and efficiency of data processing on virtual machines but also strengthens Microsoft’s own infrastructure.

By leveraging DPDK, Azure can handle higher data loads more effectively, which is crucial for performance-intensive applications. For a deeper understanding of how DPDK facilitates these improvements in cloud computing, view the latest webinar, “Hyperscaling in the Cloud,” which discusses the scale and scope of DPDK’s impact on Azure’s network architecture.

“We’re aiming to push the boundaries of network performance within Azure, leveraging MANA alongside DPDK to achieve unprecedented throughput and reliability for our virtual machines.”

Brian Denton, Senior Program Manager, Microsoft Azure Core

Significant emphasis is placed on the first 200 gig NIC, highlighting a substantial focus on achieving high throughput. Additionally, the necessity to support a high packet rate stands as a corollary to this objective. To comprehend and benchmark their throughput across various packet sizes, extensive work is undertaken. DPDK serves as the primary method for testing their hardware in this regard.

Microsoft’s engineering counterparts focus on the overall testing methodology for developing a DPDK driver set, as well as testing the hardware itself and the VM performance on that hardware. This includes client-side involvement in testing. Currently, only Linux is officially supported for DPDK, although there have been attempts to use Windows and FreeBSD. Various host configurations also play a crucial role in qualifying their hardware.

Future Directions and Community Engagement

As Azure continues to evolve, the collaboration between Microsoft’s engineering teams and the open-source community remains vital. The development of MANA and its integration with DPDK reflects a broader commitment to open innovation and community-driven improvements in cloud networking.

Conclusion

As Microsoft Azure continues to evolve, the partnership between Microsoft’s engineering teams and the DPDK open-source community is poised to play a crucial role in shaping the future of cloud networking. The development of the Microsoft Azure Network Adapter (MANA) and its integration with the Data Plane Development Kit (DPDK) underscore a commitment to leveraging open innovation and fostering community-driven advancements.

The future role of Azure MANA, in conjunction with the DPDK community, is expected to focus on breaking new technical limits in cloud networking. This collaboration could lead to significant enhancements in network performance, including higher throughput, reduced latency, and greater efficiency in packet processing. By leveraging DPDK’s efficient packet processing capabilities alongside the hardware acceleration offered by MANA, Azure aims to provide an optimized networking stack that can meet the demanding requirements of modern applications and services.

Moreover, this is likely to drive the development of new features and capabilities that are specifically tailored to the needs of Azure’s diverse user base. This could include advancements in virtual network functions (VNFs), network function virtualization (NFV), and software-defined networking (SDN), which are essential components in a cloud-native networking landscape.

The open-source nature of DPDK also ensures that the broader community can contribute to and benefit from these developments, promoting a cycle of continuous improvement and innovation. This collaborative approach not only enhances the capabilities of Azure’s networking services but also contributes to the evolution of global cloud networking standards and practices.

Ultimately, the future of Microsoft Azure MANA and the DPDK open-source community is likely to be characterized by the breaking of current technical barriers, the introduction of groundbreaking networking solutions, and the establishment of Azure as a leading platform for high-performance, cloud-based networking services.

Check out the summary and additional use cases on Hyperscaling in the Cloud here.

Join the community on slack here.

Mar 29

Love0

First Patch Submission to the DPDK Open-Source Project

By benthomas Community Spotlight

Let me first explain why I decided to submit a patch…

Recently, I was working on a research project that utilized the DPDK framework. During the process, I realized that the framework’s driver did not implement all the functionalities specified by the network card. Since one particular functionality, the “launch time feature”, was essential for my work, I spent some time modifying the driver for my project to use this feature.

After that, I decided to invest some time integrating this patch into the DPDK framework, basically for two main reasons. First, I believe this feature is essential for real-time community to use DPDK, and I have seen several discussions online about it, suggesting that its implementation could benefit many others. Secondly, I had never contributed to a large open-source project before and saw this as an opportunity to learn the basic process of submitting patches.

1. Preparation

Before submitting the patch, it’s crucial to understand the project’s development process. DPDK is an open-source project currently managed by Linux Foundation. The development of DPDK happens on the mailing list (like Linux kernel) instead of on GitHub. This means we can’t simply use git commit to submit patches, then followed by a pull request. Instead, patches are submitted via email to maintainers. Therefore, some extra preparatory steps are needed before submission, including:

Registering and subscribing to the DPDK mailing list
Configuring email to use git-send-email
Identifying the maintainers and the subtree of the module you are modifying

Mailing List: The process of registering and subscribing to the mailing list is relatively straightforward and can be done by following the steps in the DPDK official documentation. I won’t repeat that here. I also recommend registering for Patchwork to easily track the code review process. From my experience, it’s best to disable receiving all the emails from the mailing list or set the digest mode to only receive one summary per day, otherwise the daily generated patches will mix with the regular email and overflow the mailbox…

Email Configuration: Setting up the email is also quite simple. I used a Google email, and configured it following this article to link git send-email. I didn’t encounter any issues during the process.

Finding Maintainers: DPDK is a large, modular open-source project, designed for collaborative work. Each module has a designated maintainer, and some even have specific maintenance branches. Therefore, it’s necessary to find the maintainer and the branch of the module you want to modify, and submit the patch to them for integration into the mainline repository. An important point to note is to use the script devtools/get-maintainer.sh in the repository to automatically find the maintainers to Cc, just like this:

git send-email –to dev@dpdk.org –cc-cmd devtools/get-maintainer.sh

The first time, I attempted to find maintainer information in the list provided in the documentation and manually added it to the CC field. However, I later discovered that this information was outdated, which led me to inadvertently send the patch to a previous maintainer…

As for understanding the project’s coding style, I suggest not spending too much time studying the official style guide. This is because you can ensure the consistency of the coding style with the automated review scripts mentioned later.

2. Writing the Patch

The process of writing the patch is essentially modifying the original code to fix bug or add new functionalities, and after formatting it with diff, these modifications become the patch. An important suggestion here is to disable the auto-formatting feature of your IDE. Many projects have unique code formatting styles, and auto-formatting might mess up the format with Tab/Space changes everywhere.

2.1 Testing

Testing is divided into three main parts:

Compilation and functional testing
Code style check
ABI Policy

Compilation and Functional Testing: Since my modifications were minor, I only compiled it on my local machine. The specific process is largely similar to installing DPDK, except the first step involves using meson setup –wipe build to clear the previous build directory. For functional testing, I recommend testing all DPDK built-in apps that might be affected, such as dpdk-testpmd.

Code Style Check: This part can be done following the Section 5.7 Checking the Patches in the DPDK official tutorial. It’s important to note that you need to download checkpatch.pl yourself (You can get the source code by Google searching, just a simple script). After generating codespell-dpdk.txt as instructed, run devtools/checkpatches.sh patch_name.patch in the same directory. The output log shows the results, where both errors and warnings need to be corrected, but some checks may not need changes if they are unreasonable.

ABI Policy: ABI Policy refers to the requirements for variable and function naming. It is recommended to self-check against the official ABI Guideline. But personally, I feel if no new file is created, there shouldn’t be many issues, just compare with the surrounding code to get a sense of the appropriate naming conventions.

2.2 Submitting the Patch

After testing, it is good to generate the patch. Before doing so, it’s recommended to double-check your git username and email, using git config –global user.name & git config –global user.email.

My method of generating patches might not be the best practice. I initially used git add & git commit for a simple commit, then git commit –amend to modify the commit message. Finally, I generated the patch with git format-patch -1 -o ~/patch/. Here -1 indicates the patch-set includes the last few commits, but I only needed to generate one patch.

Before submission, check if the patch contains the Signed-off-by: xxxxx <xxxxxx@xxx> line. Mine wasn’t generated automatically, so I added it manually.

Finally, the patch can be submitted via git send-email, typically –to the mailing list dev@dpdk.org, and –cc to maintainers.

git send-email \

–to dev@dpdk.org \

–cc-cmd devtools/get-maintainer.sh \

./patch_name.patch

Before sending, it is best to first send it to yourself to check if the email format is correct with git send-email –to=[your email] patch_name.patch.

If you have registered for Patchwork, you’ll receive the patch submission notification after a few hours.

3. Code Review

Compared to smaller open-source projects, code review in DPDK is quite stringent, involving both automated reviews from the system and manual reviews from maintainers.

The results of the automated review are received via email within a few hours of submission. Fortunately, despite this being my first patch submission, no errors were detected with beginner luck.

In the subsequent manual review phase, maintainers pose questions about the content of the patch, which feels similar to the rebuttal phase of a research paper submission process. The main issues raised in my case focused on:

Explaining the functionality of different lines of code
Justifying the chosen solution
Providing test results

Responding directly to these questions is usually sufficient. A few minor details to note:

When replying, use the –in-reply-to parameter to specify which email you are responding to.

git send-email \

–in-reply-to=xxxxxxxx.namprd11.prod.outlook.com \

–to=xxxx@xxx.com \

–cc=xxxx@xxx.com \

–subject=”RE: [PATCH] net/e1000: support launchtime feature” \

./reply.txt

Pay attention to formatting. Indicate references to previous emails by adding >.
In the subject, add RE: before the title of the patch. Remember, no matter how many times you reply, there should only be one RE:, not a chain like RE: RE: RE:.

4. Lessons

Although the process seemed somewhat complicated to me initially, the friendly atmosphere made it smoother. From submission to the final review, it took me over ten days, which included three replies and one code update. Along the way, I encountered some technical issues and uncertainties. However, these were resolved with the help of the maintainers and other developers, who provided guidance and support throughout the process.

The biggest takeaway for me from this experience is that contributing to a large open-source project is not as daunting as it might seem. It’s often the intricate rules and the stereotypes about open-source communities that dissuade people from getting involved. However, I found that the community was welcoming, and the process, while detailed, was manageable when you are willing to spend some time on it.

Although contributing to the open-source community is not my focus, attempting to submit a patch was definitely a valuable experience.

You can follow Chuanyu on Medium here

And get started with DPDK here

Mar 28

Love0

Overview of the Latest DPDK Release 24.03

By benthomas Uncategorized

DPDK’s latest release, 24.03, marks the culmination of months of dedicated work and collaboration. Accessible now for download here, this release showcases the efforts of 154 authors, culminating in 987 commits. The extensive changes include 1,334 files being modified, with 79,260 insertions and 22,824 deletions, highlighting the substantial growth and refinement in the project.

Notably, DPDK 24.03 will not have a separate maintenance branch, and it maintains ABI compatibility with the preceding 23.11 version, ensuring a smooth transition for developers.

New Features and Enhancements

The 24.03 release introduces several significant features and improvements, including:

A new argument parsing library to streamline application development.
Standardization of dynamic logging for enhanced diagnostics.
Introduction of the HiSilicon UACCE bus, expanding hardware support.
Enhanced query capabilities for transmission queues.
Advanced flow matching capabilities including random and field comparison.
NAT64 flow action, offering new network translation options.
Flow template table resizing for more efficient resource management.
Continued preparations for MSVC build support, indicating a commitment to broader compatibility.
An expansion of DTS tests and general cleanups to enhance stability and performance.

For a detailed overview of all updates and improvements, read the release here.

Community Growth and Acknowledgments

This release also celebrates the inclusion of 31 new contributors, ranging from authors to reviewers and testers, showcasing the DPDK community’s continuous growth. Contributions spanned across a diverse group of companies, with significant commits from Marvell, NVIDIA, Intel, and many others, reflecting the wide-ranging industry support for DPDK.

Looking Ahead

The project is now looking towards version, 24.07, expected in July. The community is encouraged to submit new feature proposals in the upcoming weeks, aligning with the project’s roadmap available here.

Additionally, the upcoming webinar Hyperscaling in the cloud offers an excellent opportunity for learning and engagement. Interested participants can register here.

Shout Out

A big thank to all courageous people who reviewed other’s work. Based on Reviewed-by and Acked-by tags, the top non-PMD reviewers are:

50 Akhil Goyal @Marvel
44 Ferruh Yigit @AMD
40 Chengwen Feng @Huawei
36 Anoob Joseph @Marvel
32 Morten Brørup @Smartsharesystems
26 Tyler Retzlaff @Linux_Microsoft
21 Dariusz Sosnowski @NVIDIA
18 Ori Kam @NVIDIA
18 Bruce Richardson @INTEL

Download it here

Mar 28

Love0