Introduction
The open-source community thrives on collective efforts to improve and augment software products. In this post, we take a closer look at the operations of the Data Plane Development Kit (DPDK) community lab and its contributions towards testing and refining the DPDK software project.
DPDK, a set of libraries and drivers for fast packet processing, is embraced by major technology players including Red Hat, NVIDIA, Intel, Arm, and more. However, a crucial element that bolsters the confidence of these companies in DPDK is the dedicated testing Community Lab hosted by the University of New Hampshire Interoperability Lab. Their primary service to the DPDK community involves rigorous testing of code contributions and providing valuable feedback to the developers.
The DPDK Community Lab: An Overview
Initiated around 2017-2018, the DPDK Community Lab was conceived to ensure that incoming code contributions didn’t introduce performance regressions across various vendor channels. Initial testing focused primarily on performance testing and gradually expanded to encompass a broader range of checks.
The lab executes multiple categories of tests on incoming source code contributions, including performance, functional testing, unit and compile testing, and Application Binary Interface (ABI) stability checks. ABI testing provides stability for development across different versions within that cycle. For instance, releases 22.11, 23.03, and 23.07 are/will be ABI compliant with the current major abi version. A new major ABI version will be released with the 23.11 LTS release which may break ABI compliance with releases from the current cycle. But then the subsequent 24.03 and 24.07 releases will be ABI compliant to the ABI major version introduced at 23.11.
The Testing Suite: A Developer’s First Encounter
One of the first steps for developers contributing to DPDK involves navigating the project’s mailing list, where they submit patches for review. Any code change or patch is subsequently passed through the community lab’s testing suite. Should a patch trigger an issue or fail the automated tests, the developers are notified through email as well as the ‘patchwork’ system, an automated web portal that tracks incoming source code patches.
Understanding the testing suite is an essential part of a new developer’s journey. Once developers familiarize themselves with the codebase, the next stage involves learning how to submit patches and navigate the unique mailing list patch archive approach, which can initially be somewhat alien to developers more accustomed to using the major code collaboration and version control tools like GitLab, GitHub, or Bitbucket’s direct patch submission process.
Extending Test Coverage: Evolution Over Time
Over the years, the DPDK community lab has significantly extended its test coverage, adding new types of hardware, architectures, and operating systems to its testing suite. In 2022 and 2023 the Community Lab has worked to update our internal container build system to make it OCI compliant. That has allowed the lab to seamlessly offer the same test coverage for arm64 platforms which was previously offered only for x86 systems. The lab has also made other expansions for ARM, like adding arm32 unit tests to CI runs, and is proud to say there is now parity between x86 and ARM test coverage in the Community Lab.
The lab operates on the principle of relative comparison; it does not publish absolute performance numbers. Instead, it compares the performance of a patch against the mainline version of the codebase. This approach ensures that no individual patch introduces significant performance degradation, which could negatively impact end users and integrators.
Policies and Test Cases
The lab derives its test cases from several sources, primarily the DPDK core developer community and the DPDK Test Suite (DTS). The latter focuses on functional testing and performance testing of the DPDK system as a whole, while the former is focused on individual building blocks within the codebase. There’s an ongoing effort within the community to merge DTS into the DPDK main repository to align its release cycles more closely with those of DPDK itself.
The third category of tests comes from purpose-built sample applications, such as the Federal Information Processing Standards (FIPS) test cases. These cases test the implementation of cryptographic algorithms built into DPDK. With recent advancements from NIST, the lab has transitioned from a manual vector request and testing approach to an automated API-driven process, which greatly facilitates the automation of cryptographic implementation testing.
Testing Methods
At the heart of the lab’s testing approach are three key methods: unit testing, functional testing, and performance testing. Each of these contributes a different perspective, ensuring a comprehensive review of the application. For instance, unit testing focuses on the smallest parts of the application, while functional and performance testing assess the application’s functionality and speed under various conditions.
Compiling is an important step in the process, which they undertake before running ABI (Application Binary Interface) tests. Ensuring that the interface remains consistent is vital, as it confirms the binary formatting of the interfaces isn’t altered. This ensures that applications depending on the DPDK stack don’t face unexpected failures.
The lab’s Continuous Integration (CI) platform, Jenkins, is instrumental in organizing the testing workflow. A graphical representation of our Jenkins Testing Tree can offer a clearer understanding of the testing process, starting from patch application, branching into various forms of testing, and then cycling back for reiteration.
As patches flow into the system, Jenkins starts the process by applying these patches to the DPDK Mainline. If the patches apply correctly, the updated DPDK enters the testing pipeline. The system then breaks down into functional and performance testing using the DPDK Test Suite (DTS) across different servers and network interface cards (NICs). Concurrently, ABI testing ensures consistency at the driver kernel level. Simultaneously, they conduct compile testing on both x86 and ARM architectures. Finally, DPDK’s inbuilt unit tests confirm the robustness of each small part of the software.
Expanding Partner Coverage
In terms of expanding the lab’s test coverage and partnering with member companies, they have historically relied on organic growth within the community. However, recent efforts have focused on DPDK Gold members, targeting the integration of their latest generation hardware into the Lab’s testing suite. This direct engagement ensures their systems are among the first tested by DPDK’s community infrastructure, providing a clear benefit for their participation.
The DPDK Community lab does not eliminate the need for in-house testing, instead it complements it, providing an additional layer of assurance and reliability. Being an open-source project, the lab’s testing capabilities offer wide-ranging coverage, ensuring that the project is robust and dependable. The ongoing expansion of the lab test coverage is testament to the DPDK community’s commitment to delivering consistently robust code.
Learn more about the DPDK Community Lab testing and for how to get involved visit: https://lab.dpdk.org/results/dashboard/about/