benthomas

Introduction:

How does a developer go from a fascination with programming to creating a leading open source project that reshapes network performance testing? Jianzhang Peng’s story begins as a developer working on L4 load balancers, using DPDK to tackle the complex challenges of modern networking. He created DPERF, a high-performance load testing tool, to address gaps in existing solutions and share it with the open source community. His journey into open source and the DPDK ecosystem is one of passion, persistence, and innovation.

When asked about the beginning of his programming journey, Jianzhang reflects: “I started programming during my university days, about 20 years ago. Initially, it was a skill I developed through academic study, but it soon became a career that allowed me to solve real-world problems.” 

His career began at a startup, where he worked on both L4 and L7 load balancers before joining Baidu Cloud as a developer specializing in L4 load balancing. It was during his time at Baidu Cloud that Jianzhang encountered the limitations of commercial testing tools. “We were using a commercial tester, but the performance just wasn’t good enough,” he recalls. “I decided to write a testing tool myself, initially for personal use. Over time, I realized it could benefit others as well, so I decided to open source it.”

“We were using a commercial tester, but the performance just wasn’t good enough,” he recalls. “I decided to write a testing tool myself, initially for personal use. Over time, I realized it could benefit others as well, so I decided to open source it.

From Commercial Tools to Open Source

At Baidu Cloud, Jianzhang’s primary focus was on developing L4 load balancers. These systems manage huge amounts of traffic by distributing client requests across thousands of backend servers. “A lot of clients send requests to the L4 load balancer,” he explains, “which holds the public IP address, processes the incoming connections, and schedules them to the backend servers. The workload is immense, requiring highly optimized solutions.”

“A lot of clients send requests to the L4 load balancer, which holds the public IP address, processes the incoming connections, and schedules them to the backend servers. The workload is immense, requiring highly optimized solutions.”

While working on these load balancers, Jianzhang’s team relied on commercial testing tools to evaluate performance. However, these tools were not portable and struggled to meet the high-performance demands of L4 load balancers. Frustrated by these limitations, Jianzhang decided to create a solution tailored to his needs. The result was DPERF, a tool designed to provide high-performance, portable, and efficient testing specifically for L4 load balancers.

Joining the DPDK Community

Jianzhang’s introduction to DPDK came while working on Baidu Cloud’s load balancers, many of which were powered by DPDK. This open source framework enables developers to build fast, efficient packet processing applications. Recognizing the synergy between DPDK and his project, Jianzhang leveraged its capabilities to enhance DPERF’s performance. “We used DPDK for L4 load balancing,” he notes. “Its performance was a key factor in the success of our systems.”

“We used DPDK for L4 load balancing,” he notes. “Its performance was a key factor in the success of our systems.”

The Beginnings of DPERF

While working at Baidu, Jianzhang encountered significant challenges with the commercial testing tools available at the time. “The performance of these tools was not good enough,” he recalls. “They were designed as physical devices that had to be installed in a data center, which made them difficult to use in distributed or remote environments.” This limitation inspired Jianzhang to create a new solution.

“The performance of these tools was not good enough,” he recalls. “They were designed as physical devices that had to be installed in a data center, which made them difficult to use in distributed or remote environments.”

“I found that most open source testers were focused on Layer 7 (L7) testing, but the performance wasn’t sufficient for L4 load balancers,” he explains. “L4 load balancers process packets at an extremely fast rate, and I wanted to create a tool specifically optimized for this purpose. It had to be simple, efficient, and high-performing.”

The turning point came when Jianzhang participated in a hackathon at Baidu in 2021. “The hackathon provided the perfect opportunity to develop the first version of DPERF,” he shares. “Although I didn’t win, the experience gave me the passion and motivation to continue refining the tool.” Despite the initial lack of recognition, his colleagues and teammates appreciated the significance of his work. “They knew it was a great project, even if others didn’t realize it at the time,” Jianzhang reflects.

“The hackathon provided the perfect opportunity to develop the first version of DPERF,” he shares. “Although I didn’t win, the experience gave me the passion and motivation to continue refining the tool.”

Fueled by his hackathon experience, Jianzhang dedicated himself to creating a lightweight, high-performance tester tailored to L4 load balancers. This marked the beginning of DPERF, a tool that would soon make an impact in the DPDK community.

From Concept to Version One

The first version of DPERF was completed in just two weeks. Jianzhang describes this period as “crazy,” working tirelessly from early morning until midnight every day. “The initial codebase was less than 5,000 lines,” he shares. “I wrote it all myself, focusing on simplicity and performance.”

At the outset, Jianzhang envisioned importing the TCP stack from FreeBSD but quickly realized that the task was too large to complete within the hackathon’s timeframe. “Then, an idea struck me—to simplify the TCP stack,” he recalls. “I focused on identifying which parts could be streamlined while still maintaining the essential functionality.”

Then, an idea struck me—to simplify the TCP stack,” he recalls. “I focused on identifying which parts could be streamlined while still maintaining the essential functionality.”

Jianzhang credits this breakthrough to inspiration and his deep experience with L4 load balancing. “I designed the most important components of the architecture in just a few minutes,” he says. “From there, I refined and simplified every part of the TCP stack to meet the performance goals.”

The intensity of this period took a physical toll, as Jianzhang recalls experiencing repetitive strain injuries from non-stop coding. Despite this, he describes the process as exhilarating. “I was so passionate about the project that I enjoyed every moment of it,” he says. “It was incredible to see the tool take shape so quickly.”

Jianzhang’s approach was unique in that he didn’t follow a conventional development workflow. Instead of iterating through a structured design-test-refine process, he coded the entire tool in one go. “I just started and worked all the way through to the end,” he explains. “It was all about maintaining focus and momentum.”

 “I just started and worked all the way through to the end,” he explains. “It was all about maintaining focus and momentum.”

Optimizing and Refining DPERF

Following the initial release, Jianzhang continued to optimize DPERF. With over a decade of experience working with user-space TCP stacks, he applied his deep knowledge of L4 and L7 load balancers to improve performance further. “I knew how to integrate L2, L3, L4, and L7 into a simplified stack,” he explains. “This allowed me to create a highly efficient and scalable solution.”

The architecture of DPERF consists of a client-server model where each CPU independently handles packet transmission and reception. This design ensures that CPUs do not need to share state, significantly improving performance. “Each CPU sends and receives its own packets,” Jianzhang details. “This way, we eliminate unnecessary inter-CPU communication and maximize efficiency.”

A key innovation in DPERF is the caching mechanism for packet headers. “Instead of regenerating the entire packet for each test, we cache the payload and only modify the necessary header fields,” Jianzhang explains. “This drastically reduces processing overhead and enhances performance.”

“Instead of regenerating the entire packet for each test, we cache the payload and only modify the necessary header fields,”

Another critical optimization is the use of a pre-allocated socket table. “All sockets are created at the beginning,” he says. “This allows us to instantly locate sockets based on IP addresses and ports, ensuring rapid packet processing.”

Through continuous refinement and feedback from users, DPERF evolved into a highly reliable and high-performance tool, widely adopted in production environments.

Reflections on the DPDK Community and Open Source

While deeply involved in DPERF, Jianzhang’s contributions to the broader DPDK project have been limited due to his current role in a crypto trading company. “If I worked at a networking-focused company, I might engage more directly with the DPDK community,” he reflects. “But I still appreciate the support from my company, which allows me to continue maintaining DPERF as an open source project.”

When asked about the growth of DPDK adoption in Asia, Jianzhang acknowledges its increasing use, particularly among network device manufacturers and cloud service providers. “A lot of companies are using DPDK for firewalls, security applications, and cloud networking,” he says.

 “A lot of companies are using DPDK for firewalls, security applications, and cloud networking,” 

He also notes that some major technology firms—including TikTok (ByteDance), Baidu Cloud, and Alibaba Cloud—use DPERF internally, though these details are not publicly shared. “Many companies rely on open source software without publicly acknowledging it,” he says. “But knowing that DPERF is helping real-world deployments is what matters most to me.”

“But knowing that DPERF is helping real-world deployments is what matters most to me.”

Advice for Aspiring Open Source Developers

As a respected developer in the DPDK ecosystem, Jianzhang offers advice for those looking to follow in his footsteps. “Develop something truly useful,” he says. “If it helps you and others solve real-world problems, it has value.”

“Develop something truly useful,” he says. “If it helps you and others solve real-world problems, it has value.”

He also emphasizes the importance of persistence and passion. “Building an open source project isn’t easy,” he says. “But if you believe in what you’re creating, keep refining it, keep listening to feedback, and keep pushing forward.”

“Building an open source project isn’t easy,” he says. “But if you believe in what you’re creating, keep refining it, keep listening to feedback, and keep pushing forward.”

Jianzhang’s journey—from solving a personal technical challenge to developing a widely used open source tool—highlights the impact of individual contributions in the DPDK community. His work on DPERF has not only improved network performance testing but also demonstrated the power of innovation and collaboration.

Find out more about the latest release! 1.8.0 

Introduction: 

Grout is an emerging open source project hosted under the DPDK community that aims to simplify high-performance networking and provide a reference application for building applications using DPDK. Spearheaded by Robin Jarry from Red Hat, the project was born out of the need for a vendor-neutral, practical tool to validate server setups optimized for DPDK performance.

This user story dives into the origins, challenges, and aspirations of the Grout project.

Building Grout to Simplify High-Performance Networking

Robin Jarry, a software engineer at Red Hat and member of the OpenStack NFV Integration team, has been at the forefront of tackling the intricate challenges of deploying high-performance networking solutions. Based in France, Robin works primarily with telco customers, focusing on tasks like power tuning, CPU isolation, and configuring servers to maximize the performance of DPDK.

While these optimizations are critical for achieving the desired results, validating the setups proved to be a recurring issue in his work. The tools available, such as testpmd, were limited in scope. Although testpmd is a valuable application for testing DPDK drivers, it lacks networking functionality and cannot fully verify the correctness of server configurations. 

Recognizing this gap, Robin and his colleagues decided to create a solution that could act as a vendor-neutral application for validating DPDK-based server setups. This effort gave rise to Grout, a project designed to simplify and standardize the use of DPDK in high-performance environments.

Like any new project, the early stages of Grout were challenging, with foundational work requiring significant effort. However, once the initial framework was laid out, the team found it increasingly easy to add new features and protocols. Their hard work culminated in a presentation Grout # a Graph Router Based on DPDK at the DPDK Summit North America 2024 event, where Robin and key contributor David Marchand received overwhelmingly positive feedback.

“I was thrilled by how well Grout was received, the feedback during and after the talk confirmed that we’re solving a real problem, and it’s exciting to see the community’s enthusiasm for this project.” 

Robin credits the project’s success not only to his efforts but also to the collaboration with key contributors, including David Marchand, a long-time DPDK maintainer, and Christophe Fontaine, a colleague at Red Hat.

Christophe focuses on OpenStack engineering for Telco use cases, leading the design of Network Functions Virtualization (NFV) within the OpenStack team to optimize high-performance connectivity for Virtual Network Functions (VNFs) and Cloud-Native Functions (CNFs). Before joining Red Hat, he worked at Qosmos (ENEA), where he contributed to FD.io and OpenDaylight, enhancing dynamic service function chaining for OpenStack on x86 and ARM servers.

Additionally, Christophe has provided detailed documentation, engaged with the community, and shared insights on performance tuning and Grout integration at DPDK summits, making advanced features more accessible to both new and experienced users.

Together, they are pushing Grout forward as a reference application for DPDK, designed to simplify its adoption and showcase best practices for achieving high performance in networking workloads.

A Journey into Open Source: From Sound Waves to Network Packets

Robin’s path to becoming a key contributor in the DPDK community is anything but conventional. Before writing code and configuring high-performance servers, Robin was immersed in a completely different world: the music industry. Working as a sound engineer in Brussels, Robin honed his technical skills in an environment that demanded precision and creativity. However, financial challenges forced him to make a career change—a decision that ultimately led him to software engineering and the open source world.

“I had to start from scratch,” Robin shared. “I went to night school, earned my engineering degree, and began exploring what I could do in the world of technology.” 

The Connection Between Music and Code

Interestingly, Robin sees parallels between his past career as a sound engineer and his work as a software engineer. Both disciplines require problem-solving, creativity, and a focus on building something that works seamlessly. “There’s a certain creative energy in both fields. Coding feels like crafting something, almost like creating music, where every piece fits together to form a complete whole.”

“There’s a certain creative energy in both fields. Coding feels like crafting something, almost like creating music, where every piece fits together to form a complete whole.”

Robin’s unique perspective isn’t uncommon in the tech world. A recent poll by another Linux Foundation project revealed that nearly 38% of developers are also musicians—a convergence that highlights the shared skills of discipline, creativity, and focus. For Robin, this duality reinforces the idea that diverse backgrounds can bring fresh perspectives to technical challenges.

A recent poll by another Linux Foundation project revealed that nearly 38% of developers are also musicians—a convergence that highlights the shared skills of discipline, creativity, and focus.

Discovering Open Source and the DPDK Community

Robin’s first exposure to open source came from an unexpected place: side projects in video game development. While exploring these projects, he became captivated by the ethos of open source. “What struck me was how these developers were building software for free, just to help others and share knowledge. I wanted to understand their mindset, and it resonated deeply with me,” Robin explained.

“What struck me was how these developers were building software for free, just to help others and share knowledge. I wanted to understand their mindset, and it resonated deeply with me,” 

This philosophy of collaboration and giving back to the community would become a cornerstone of Robin’s career. His professional journey into open source began in earnest when he joined 6WIND, the company responsible for establishing the open source DPDK community in 2013. Here, he worked on projects that blended his growing technical expertise with his newfound passion for open source development.

Over the years, Robin’s involvement with DPDK became both a professional responsibility and a personal conviction. 

“I don’t just do open source work because it’s part of my job—I genuinely believe in the values of open source, it fits with my character. I enjoy the process of contributing, maintaining projects, and seeing the positive impact on the community.” 

Giving Back to the Open Source Community

Robin’s commitment to open source extends beyond DPDK. In his spare time, he maintains personal projects and contributes to other open-source initiatives. He describes this as a “virtuous cycle” of giving and receiving energy: “People are often grateful when you contribute something useful, and that gratitude motivates you to keep going. It’s a positive feedback loop.”

This passion for open source is also evident in Grout, where Robin and his team have made contributions not just to their own project, but back to the DPDK ecosystem. By identifying and fixing issues within DPDK’s core libraries, they’re embodying the principles of open source collaboration.

A Welcoming Community

Reflecting on his experience with DPDK, Robin praised the community’s inclusiveness and support for newcomers. “When I started contributing to DPDK about 10 years ago, I found it much more welcoming compared to other projects, which have a steeper entry barrier,” he said. “The DPDK community is patient, helpful, and understanding—even with people who are just getting started.”

 “The DPDK community is patient, helpful, and understanding—even with people who are just getting started.”

This collaborative spirit has been a key factor in Robin’s long-term involvement with DPDK, as well as his enthusiasm for helping others onboard into the ecosystem.

Grout at the Crossroads: A Pivotal Shift for DPDK

As the DPDK project evolves, it finds itself in a transitional phase—moving from years of foundational development into a focus on real-world applications and use cases. Grout, a new addition to the DPDK ecosystem, is emerging as a key project in this transformation. 

The origins of Grout lie in the complexity of getting started with DPDK. While DPDK has been a pioneering project for high-performance networking, its setup process can be daunting. “You have too many things to configure,” Robin explained, “from enabling VFIO and allocating huge pages to binding devices to the right drivers. It’s complex, and people have told us that this is one of the biggest hurdles.”

“You have too many things to configure, from enabling VFIO and allocating huge pages to binding devices to the right drivers. It’s complex, and people have told us that this is one of the biggest hurdles.”

Grout is designed to simplify this experience. It aims to serve as a reference example—a guide to using DPDK libraries in a clean, best-practice-driven way. This includes demonstrating how to configure networking for high performance or even integrate AI applications using DPDK.

Robin highlighted that the DPDK community is keenly aware of this need. “We’ve received a lot of feedback asking for either best practices or an example of how things are intended to be used. Grout could become that example—a model of how DPDK can be leveraged effectively and efficiently.”

Laying the Foundation with the Graph Library

Grout isn’t starting from scratch. One of its core building blocks is the DPDK Graph Library, which provides a structured approach for building applications. According to Robin, the Graph Library already delivers roughly “a third of the structure needed to build a DPDK application.”

This modularity makes Grout not only a great starting point but also a potential framework for future DPDK projects. By leveraging existing tools and adding layers of usability, Grout is poised to streamline development for other contributors and users.

“Part of this foundation is already in DPDK itself,” Robin said, emphasizing that Grout’s architecture is tightly integrated with existing libraries, making it an ideal reference point for those entering the ecosystem.”

Moving Toward Applications and Use Cases

The feedback from the DPDK community underscores the importance of Grout’s focus on usability and practical application. As DPDK transitions from being a toolkit to supporting more application-level use cases, projects like Grout play an essential role in bridging the gap.

This shift is especially relevant as the networking world increasingly embraces cloud-native technologies. Grout’s primary use case targets container-native and cloud-native network functions (CNFs), making it a valuable asset for telcos and enterprises deploying high-performance networking in OpenStack, Kubernetes, and OpenShift environments.

“Grout isn’t a replacement for other tools like OVS, but it’s designed to run in VMs or containers, rather than on a host server. Its focus is entirely different—Grout is a router, not a virtual switch.”

A Catalyst for the Future of DPDK

Grout is still in its early stages, but its potential is already clear. By addressing pain points in the DPDK ecosystem and providing a practical, scalable reference architecture, it paves the way for easier adoption and innovation. Whether it’s serving as a blueprint for building high-performance CNFs or simplifying the onboarding process for new contributors, Grout is an important part of the next chapter of DPDK.

With its foundations rooted in the community and a focus on usability and real-world applications, Grout exemplifies the values of open source development. As Robin aptly put it, 

“Grout is here to simplify, to guide, and to inspire what’s possible with DPDK.”

Target Use Case

Grout’s target use case is clear: cloud-native network functions (CNFs) and container-native network functions. These applications are vital for telcos and high-performance networking environments but represent a niche within the broader world of cloud computing.

Robin contrasted this with the dominant web application model in the cloud. “A web service won’t really be affected by Grout,” he noted. “But for telco applications or high-performance networking, where the requirements are far more specialized, Grout provides the extra care needed.”

While this niche focus may seem limiting at first glance, it’s precisely this specialization that makes Grout so valuable. By addressing the unique challenges of high-performance networking, Grout fills a critical gap in the DPDK ecosystem.

“A web service won’t really be affected by Grout,” he noted. “But for telco applications or high-performance networking, where the requirements are far more specialized, Grout provides the extra care needed.”

The Future of Grout: Usability, Automation, and Niche Applications

As Grout evolves, its focus remains on bridging gaps in the DPDK ecosystem and addressing specific use cases that demand high-performance networking. While not directly tied to AI or machine learning, Grout contributes to the infrastructure that supports automation and efficiency, helping simplify workflows and creating a foundation for innovation.

For instance, Grout could play a role in automating aspects of testing or continuous integration (CI) processes in environments requiring high-performance networking. By providing a standardized reference for DPDK applications, it could ensure seamless interactions between hardware and software, laying the groundwork for advanced automation tasks.

Impact on High-Performance Applications

In the short term, Grout’s primary contribution is its role as a reference for building containerized, high-performance network functions. Long-term, its automation and usability features could influence broader workflows, including testing, CI pipelines, and operational efficiency in telco and enterprise environments.

By simplifying the complexities of DPDK setups and offering clear examples of best practices, Grout is poised to make high-performance networking more accessible—not just for seasoned developers, but for newcomers to the ecosystem as well.

The Road Ahead: Plans for Usability Improvements

Having reached its minimum viable product (MVP) stage, Grout’s next steps focus on usability. Robin outlined several key areas for improvement, emphasizing the need to simplify tasks that currently require manual intervention.

One prominent feature on the roadmap is the automatic assignment of port RX queues to CPUs. “Right now, this is something you need to configure manually, but automating this will make the setup process much easier,” Robin said. This improvement would make Grout more accessible to users deploying DPDK applications in cloud environments, whether in OpenStack, OpenShift, or Kubernetes.

Another goal is to refine Grout’s role as a reference application—a good example of how DPDK libraries and tools should be used. By demonstrating best practices, Grout aims to set a standard for developers, helping them build applications without grappling with unnecessary complexity.

Evolving Organically

As for the future, Robin believes that Grout will evolve based on community needs and feedback. “Maybe it will serve as an example, as a reference architecture,” he said. “I don’t want to overstate its impact—it’s going to evolve organically based on how the community uses it.”

“Maybe it will serve as an example, as a reference architecture,” he said. “I don’t want to overstate its impact—it’s going to evolve organically based on how the community uses it.”

This flexible approach reflects the open source mindset at the heart of Grout’s development. Rather than dictating its direction, Robin and his team are focused on providing a solid foundation and letting the project grow in response to real-world use cases.

A Tool Built for Collaboration

As Grout continues to evolve, it stands as a testament to the power of open source collaboration. Whether serving as a blueprint for DPDK applications, simplifying cloud-native networking setups, or enabling automation in niche use cases, Grout demonstrates how focused, community-driven efforts can lead to impactful tools that solve real problems.

In Robin’s words, “Grout isn’t trying to be everything, but by focusing on what it does best—providing clarity and structure—it has the potential to make a lasting difference in the DPDK ecosystem.”

“Grout isn’t trying to be everything, but by focusing on what it does best—providing clarity and structure—it has the potential to make a lasting difference in the DPDK ecosystem.”

Red Hat’s Strategy: Leveraging DPDK and Grout

At Red Hat, DPDK plays a critical role, particularly in conjunction with Open vSwitch (OVS). However, Grout occupies a distinct niche in this ecosystem. “OVS runs on the host, while Grout is designed to run in VMs or containers, It’s not meant to replace OVS; it’s a router, not a virtual switch. The scopes are entirely different.”

Grout aligns with Red Hat’s strategy to develop examples of virtual network functions (VNFs) and cloud-native network functions (CNFs) that demonstrate DPDK’s capabilities. While Grout is still in its early stages, its development reflects an experimental approach: build something, see if it gains traction, and adapt based on feedback.

This iterative mindset mirrors the ethos of open source—collaborate, experiment, and refine based on community needs.

Cross-Project Collaboration: Building Bridges Across Ecosystems

The DPDK community has long emphasized the importance of integrating its technology with other open-source projects, and Grout exemplifies this philosophy.

Robin highlighted Kubernetes as a natural integration point for DPDK, given the growing adoption of containerized environments in networking. “Kubernetes is an obvious pathway for collaboration, especially for projects like Grout that target container-native applications,” he noted.

“Kubernetes is an obvious pathway for collaboration, especially for projects like Grout that target container-native applications”.

Additionally, Grout has facilitated a reverse flow of contributions—where challenges encountered during Grout’s development led to improvements in the DPDK core itself. This reciprocal relationship underscores the project’s commitment to the principles of open source: contribute back to what you use.

“Instead of just consuming DPDK, we’ve raised issues, fixed them, and sent patches back to the community,” Robin said. “This approach isn’t just rewarding; it’s essential for maintaining the ecosystem’s vitality.”

Raising Awareness and Building Momentum

Looking ahead, Robin sees opportunities to raise awareness about Grout and its role in the DPDK ecosystem. Collaborating with organizations like the Linux Foundation and participating in events can help showcase the project’s potential. As Robin summed it up, “Grout is as much about helping others as it is about building something new. It’s a shared effort, and that’s what makes it special.”

“Grout is as much about helping others as it is about building something new. It’s a shared effort, and that’s what makes it special.”

To get involved, join the mailing list at grout@dpdk.org, explore the project on GitHub at https://github.com/DPDK/grout, and share your ideas for how you’d like to use Grout!

Don’t forget to catch the Grout presentation at FOSDEM 2025 on February 2.

---

Introduction

The demand for high-performance network security solutions is at an all-time high, as organizations constantly seek faster and more efficient ways to handle traffic, detect threats, and ensure real-time response capabilities. 

Suricata as an open-source high-performance network security engine has long been at the forefront of these efforts. Network security professionals appreciate Suricata for its capabilities to act as an IDS (Intrusion Detection System), IPS (Intrusion Prevention System), and as an NSM (Network Security Monitoring) system.

But it’s the integration of the Data Plane Development Kit (DPDK) into Suricata that has allowed it to reach unprecedented performance levels, providing a vital boost for packet processing at high speeds. 

This story explores the journey of Suricata’s DPDK integration, the technical challenges and solutions, and the ongoing impact on Suricata’s functionality and performance.

Origins of Suricata: A Security Solution with Community at Its Core

In 2008, a group of security-focused professionals came together with a vision to improve open-source network security. 

Victor Julien, who was working as a contractor in the network security field, joined forces with Matt Jonkman, who led an early threat intelligence project (known as Emerging Threats), and Will Metcalf, who was involved in developing an inline version of Snort—a popular intrusion detection and prevention system (IDS/IPS). 

Their collaborative work in network security sparked the idea to create something new that would address gaps in existing solutions.

The journey truly began when Victor experimented with code on his own in 2007, without expecting much traction. However, after meeting Matt and Will at a conference in the U.S. and sharing his prototype with them, the project gained momentum. 

By 2008, they secured initial seed funding from the Department of Homeland Security (DHS), allowing them to pursue their vision formally. This funding was instrumental in establishing the Open Information Security Foundation (OISF), a nonprofit entity designed to ensure that the project would remain community-oriented and free from corporate control.

From the start, they were committed to making Suricata an open-source, community-driven project. With the OSF foundation’s setup, they chose the GPLv2 license, reflecting their belief in open collaboration and safeguarding the project from being absorbed by larger corporations. DHS funding, while crucial, was temporary, so they developed a sustainable model that allowed vendors to join OSF as members, offering a more flexible licensing option.

This foundational approach set the stage for what has now been a 15-year journey of innovation and collaboration in the network security field.

“We wanted to establish an organization that would make Suricata safe from acquisition, which we’d seen happen to other open-source projects at the time.” 

– Victor Julien – Suricata IDS/IPS Lead Developer

Since then, Suricata has gained adoption from large enterprises, including AWS, which integrates Suricata in its network firewall services.

A Perfect Fit: The Role of DPDK in Suricata’s Development

With increasing demand for high-performance network security tools, Suricata’s team saw an opportunity to leverage DPDK. DPDK provides a set of libraries and drivers for fast packet processing, bypassing traditional kernel limitations. 

This high-performance potential caught the attention of users and developers alike, many of whom were eager to see DPDK integration in Suricata. Lukas Sismis, a contributor who led Suricata’s DPDK integration, explained that several teams had previously worked on integrating DPDK with Suricata. 

However, most of these efforts were specific to unique use cases and lacked general applicability, which is why they hadn’t been contributed back to the Suricata codebase.

Lukas initially engaged with Suricata’s architecture through a master’s thesis, where his primary goal was to expand Suricata’s packet capture capabilities using DPDK. He explains, “Suricata’s architecture, with its separate capture logic, made it easy to add a new capture method.” 

His work, later incorporated into Suricata’s main codebase, helped create a general-purpose DPDK integration, ensuring Suricata’s compatibility with multiple DPDK-supported network interface cards (NICs) and enabling seamless configuration.

“Suricata’s architecture, with its separate capture logic, made it easy to add a new capture method.”

 – Lukas Sismis, Software Engineer at Suricata & Cesnet

Suricata’s Architectural Evolution and DPDK Integration

Suricata’s multi-threaded, modular design made it an ideal candidate for integration with DPDK. Suricata supports packet-capturing methods through its modular “capture interface,” which allows users to swap out packet capture techniques. 

DPDK, as an input method, fits naturally within this design and supports Suricata’s scalability goal: Suricata aims to run effectively across small, low-power deployments to high-speed data centers.

Lukas’ integration efforts involved setting up DPDK within Suricata as an alternative capture method, making it possible to directly interface with high-speed NICs while bypassing kernel overhead. Some of the major steps in this integration included:

• Creating a New Capture Method: Lukas established DPDK as a new capture method within Suricata’s architecture, mapping configuration options for different NICs.
• Testing Different NICs: Through this process, Lukas tested various NICs supported by DPDK, noting disparities in how each handled DPDK configurations.
• Traffic Distribution Strategies: To manage packet distribution effectively, Lukas leveraged DPDK to configure hash-based packet distribution, efficiently balancing traffic load across CPU cores.

While most initial optimizations focused on enabling basic packet capture, this work laid the foundation for further enhancements. Testing showed a notable 10-15% performance gain, an exciting outcome that validated the decision to integrate DPDK as a core feature of Suricata’s capture options.

Tackling Hardware Offloading for Enhanced Performance

Beyond standard packet capture, the Suricata team recognized a significant opportunity in DPDK’s hardware offloading capabilities. Suricata’s high-speed packet processing can greatly benefit from the offloading of repetitive tasks to hardware, potentially bypassing certain types of network traffic. 

Lukas and his team began exploring offload capabilities that would allow Suricata to selectively filter traffic in hardware.

The primary focus of Suricata’s hardware offloading research has been on:

• Flow Bypass: Allowing Suricata to ignore certain flows after initial inspection, reducing the processing load on uninteresting traffic.
• Packet Filter: Discarding unwanted traffic from the start helps Suricata use more resources on the important part.
• Decapsulation and Encapsulation Offloads: Offloading these operations can reduce overhead in packet analysis, freeing up CPU resources for other tasks.

Although full offload implementation is still underway, initial testing shows promising potential. DPDK’s RegEx accelerator API, supported by NVIDIA BlueField and Marvell NICs, is an example of hardware that could handle pattern-matching offloads. This ongoing work has been presented at Suricon 2024.

Since Suricata’s detection engine performs extensive pattern matching, a hardware-based solution could significantly reduce CPU load.

Challenges and Technical Hurdles in DPDK Integration

Lukas encountered several challenges while working with DPDK, primarily related to hardware compatibility and traffic distribution. While DPDK offers a standardized API, not all NICs perform identically, which led to variations in performance during testing. 

One challenge was to cover and unify the different configurations of the load balancing hash function (RSS) in the NICs. This required NIC-specific experimentation and testing with different configuration mechanisms.

Lukas also had to modify Suricata’s configuration parsing to map settings to DPDK-compatible options, ensuring a more user-friendly experience. 

This testing phase highlighted the need for adaptable configurations to support a wide range of DPDK-enabled hardware.

Despite these challenges, Lukas’ integration work has laid a strong foundation for Suricata’s use of DPDK, making Suricata more adaptable to high-performance environments.

Leveraging Community and Industry Feedback

Suricata’s community engagement plays a vital role in its development. Lukas worked closely with the CESNET team, a network research institution with deep experience in DPDK. 

This collaboration allowed him to troubleshoot issues in real time without relying solely on online forums. In addition, Victor and Lukas sought feedback from DPDK maintainers like Thomas Monjalon and David Marchand, whose insights were invaluable in refining Suricata’s integration.

Suricata’s developers also participate in community channels, including a Discourse forum, Redmine, and a Discord server. While direct communication with the DPDK team has been limited, Suricata’s community-driven model allows users to share feedback directly with developers, accelerating improvements and ensuring the tool meets evolving needs.

Real-World Impact: Enhanced Packet Processing for Modern Network Demands

DPDK’s integration has brought measurable performance gains to Suricata, providing faster packet processing for users. Major security vendors are already leveraging Suricata with the DPDK integration in their products, attesting to its reliability and scalability.

DPDK’s impact is particularly evident in high-speed environments where packet capture bottlenecks could otherwise lead to packet drops or latency. The integration allows Suricata to handle higher packet rates efficiently, extending its utility in demanding, real-time network security use cases.

Looking Forward: New Horizons with AI and Machine Learning

As artificial intelligence and machine learning applications expand across technology sectors, Suricata’s team remains open to exploring AI-driven enhancements. 

Victor explained that AI’s most promising role would likely be in post-processing. Suricata currently exports JSON-formatted data, which can be fed into AI models for insights beyond immediate packet inspection. 

Many current machine learning models operate at a macro level, analyzing data patterns over time rather than in real time, which aligns well with Suricata’s current functionality as a data generator for other analytics tools.

Real-time AI inference for packet processing, however, remains a challenge. Victor elaborated, “Most AI models require milliseconds for inference, which is too slow for packet-level detection in real-time.” Still, the team is ready to adopt AI models once hardware advances make real-time AI feasible.

“Most AI models require milliseconds for inference, which is too slow for packet-level detection in real-time.”

–– Victor Julien – Suricata IDS/IPS Lead Developer

Future Development: Suricata as a Library for Broader Integration

A major long-term goal for Suricata is to establish a core API, effectively transforming Suricata’s detection engine into a library that other tools can leverage. 

This approach could enable seamless integration of Suricata’s capabilities with other applications, such as proxy servers, endpoint security products, and cloud-based services. 

While the foundational work for this API exists, achieving a fully developed API will take time. Victor noted that this goal, motivated by growing encryption in network traffic, could broaden Suricata’s utility in increasingly secure environments.

This library initiative would allow third-party developers to incorporate Suricata’s detection features in novel ways, creating a flexible, modular ecosystem where Suricata is part of larger, more complex security infrastructures.

Expanding Community Engagement Through Events

Suricata’s annual conference, Suricon, exemplifies the project’s community-centric approach. Suricon gathers developers, users, and industry professionals to share insights, discuss roadmap goals, and showcase new features. 

With a mix of training sessions and talks, Suricon provides a valuable opportunity for knowledge exchange and collaboration. DPDK community members have shown interest in attending future events, strengthening cross-community relationships, and fostering a shared development approach.

Suricata’s collaboration model has proven instrumental in its growth. This strong community foundation ensures that Suricata can keep pace with rapidly changing security demands.

Conclusion: Pushing Network Security Boundaries

Suricata’s integration with DPDK marks a significant milestone in its evolution, empowering it to achieve higher performance, greater adaptability, and better hardware compatibility. 

From initial testing to real-world deployments, DPDK’s impact has been transformative, enabling Suricata to meet the demands of today’s high-speed, security-focused networks. 

Through community feedback, industry collaboration, and a forward-looking approach to hardware offloading and AI, Suricata continues to redefine what’s possible in open-source network security.

As Suricata looks ahead, its development team remains committed to innovation and community-driven progress. With a roadmap that includes expanded hardware offloading, AI-driven enhancements, and new API integrations, Suricata is well-positioned to lead the next generation of network security solutions. 

This DPDK integration story exemplifies how open-source collaboration can drive meaningful advancements, pushing technology forward in response to real-world needs.

Learn more about contributing to DPDK here