Introduction
The demand for high-performance network security solutions is at an all-time high, as organizations constantly seek faster and more efficient ways to handle traffic, detect threats, and ensure real-time response capabilities.
Suricata as an open-source high-performance network security engine has long been at the forefront of these efforts. Network security professionals appreciate Suricata for its capabilities to act as an IDS (Intrusion Detection System), IPS (Intrusion Prevention System), and as an NSM (Network Security Monitoring) system.
But it’s the integration of the Data Plane Development Kit (DPDK) into Suricata that has allowed it to reach unprecedented performance levels, providing a vital boost for packet processing at high speeds.
This story explores the journey of Suricata’s DPDK integration, the technical challenges and solutions, and the ongoing impact on Suricata’s functionality and performance.
Origins of Suricata: A Security Solution with Community at Its Core
In 2008, a group of security-focused professionals came together with a vision to improve open-source network security.
Victor Julien, who was working as a contractor in the network security field, joined forces with Matt Jonkman, who led an early threat intelligence project (known as Emerging Threats), and Will Metcalf, who was involved in developing an inline version of Snort—a popular intrusion detection and prevention system (IDS/IPS).
Their collaborative work in network security sparked the idea to create something new that would address gaps in existing solutions.
The journey truly began when Victor experimented with code on his own in 2007, without expecting much traction. However, after meeting Matt and Will at a conference in the U.S. and sharing his prototype with them, the project gained momentum.
By 2008, they secured initial seed funding from the Department of Homeland Security (DHS), allowing them to pursue their vision formally. This funding was instrumental in establishing the Open Information Security Foundation (OISF), a nonprofit entity designed to ensure that the project would remain community-oriented and free from corporate control.
From the start, they were committed to making Suricata an open-source, community-driven project. With the OSF foundation’s setup, they chose the GPLv2 license, reflecting their belief in open collaboration and safeguarding the project from being absorbed by larger corporations. DHS funding, while crucial, was temporary, so they developed a sustainable model that allowed vendors to join OSF as members, offering a more flexible licensing option.
This foundational approach set the stage for what has now been a 15-year journey of innovation and collaboration in the network security field.
“We wanted to establish an organization that would make Suricata safe from acquisition, which we’d seen happen to other open-source projects at the time.”
– Victor Julien – Suricata IDS/IPS Lead Developer
Since then, Suricata has gained adoption from large enterprises, including AWS, which integrates Suricata in its network firewall services.
A Perfect Fit: The Role of DPDK in Suricata’s Development
With increasing demand for high-performance network security tools, Suricata’s team saw an opportunity to leverage DPDK. DPDK provides a set of libraries and drivers for fast packet processing, bypassing traditional kernel limitations.
This high-performance potential caught the attention of users and developers alike, many of whom were eager to see DPDK integration in Suricata. Lukas Sismis, a contributor who led Suricata’s DPDK integration, explained that several teams had previously worked on integrating DPDK with Suricata.
However, most of these efforts were specific to unique use cases and lacked general applicability, which is why they hadn’t been contributed back to the Suricata codebase.
Lukas initially engaged with Suricata’s architecture through a master’s thesis, where his primary goal was to expand Suricata’s packet capture capabilities using DPDK. He explains, “Suricata’s architecture, with its separate capture logic, made it easy to add a new capture method.”
His work, later incorporated into Suricata’s main codebase, helped create a general-purpose DPDK integration, ensuring Suricata’s compatibility with multiple DPDK-supported network interface cards (NICs) and enabling seamless configuration.
“Suricata’s architecture, with its separate capture logic, made it easy to add a new capture method.”
– Lukas Sismis, Software Engineer at Suricata & Cesnet
Suricata’s Architectural Evolution and DPDK Integration
Suricata’s multi-threaded, modular design made it an ideal candidate for integration with DPDK. Suricata supports packet-capturing methods through its modular “capture interface,” which allows users to swap out packet capture techniques.
DPDK, as an input method, fits naturally within this design and supports Suricata’s scalability goal: Suricata aims to run effectively across small, low-power deployments to high-speed data centers.
Lukas’ integration efforts involved setting up DPDK within Suricata as an alternative capture method, making it possible to directly interface with high-speed NICs while bypassing kernel overhead. Some of the major steps in this integration included:
- Creating a New Capture Method: Lukas established DPDK as a new capture method within Suricata’s architecture, mapping configuration options for different NICs.
- Testing Different NICs: Through this process, Lukas tested various NICs supported by DPDK, noting disparities in how each handled DPDK configurations.
- Traffic Distribution Strategies: To manage packet distribution effectively, Lukas leveraged DPDK to configure hash-based packet distribution, efficiently balancing traffic load across CPU cores.
While most initial optimizations focused on enabling basic packet capture, this work laid the foundation for further enhancements. Testing showed a notable 10-15% performance gain, an exciting outcome that validated the decision to integrate DPDK as a core feature of Suricata’s capture options.
Tackling Hardware Offloading for Enhanced Performance
Beyond standard packet capture, the Suricata team recognized a significant opportunity in DPDK’s hardware offloading capabilities. Suricata’s high-speed packet processing can greatly benefit from the offloading of repetitive tasks to hardware, potentially bypassing certain types of network traffic.
Lukas and his team began exploring offload capabilities that would allow Suricata to selectively filter traffic in hardware.
The primary focus of Suricata’s hardware offloading research has been on:
- Flow Bypass: Allowing Suricata to ignore certain flows after initial inspection, reducing the processing load on uninteresting traffic.
- Packet Filter: Discarding unwanted traffic from the start helps Suricata use more resources on the important part.
- Decapsulation and Encapsulation Offloads: Offloading these operations can reduce overhead in packet analysis, freeing up CPU resources for other tasks.
Although full offload implementation is still underway, initial testing shows promising potential. DPDK’s RegEx accelerator API, supported by NVIDIA BlueField and Marvell NICs, is an example of hardware that could handle pattern-matching offloads. This ongoing work has been presented at Suricon 2024.
Since Suricata’s detection engine performs extensive pattern matching, a hardware-based solution could significantly reduce CPU load.
Challenges and Technical Hurdles in DPDK Integration
Lukas encountered several challenges while working with DPDK, primarily related to hardware compatibility and traffic distribution. While DPDK offers a standardized API, not all NICs perform identically, which led to variations in performance during testing.
One challenge was to cover and unify the different configurations of the load balancing hash function (RSS) in the NICs. This required NIC-specific experimentation and testing with different configuration mechanisms.
Lukas also had to modify Suricata’s configuration parsing to map settings to DPDK-compatible options, ensuring a more user-friendly experience.
This testing phase highlighted the need for adaptable configurations to support a wide range of DPDK-enabled hardware.
Despite these challenges, Lukas’ integration work has laid a strong foundation for Suricata’s use of DPDK, making Suricata more adaptable to high-performance environments.
Leveraging Community and Industry Feedback
Suricata’s community engagement plays a vital role in its development. Lukas worked closely with the CESNET team, a network research institution with deep experience in DPDK.
This collaboration allowed him to troubleshoot issues in real time without relying solely on online forums. In addition, Victor and Lukas sought feedback from DPDK maintainers like Thomas Monjalon and David Marchand, whose insights were invaluable in refining Suricata’s integration.
Suricata’s developers also participate in community channels, including a Discourse forum, Redmine, and a Discord server. While direct communication with the DPDK team has been limited, Suricata’s community-driven model allows users to share feedback directly with developers, accelerating improvements and ensuring the tool meets evolving needs.
Real-World Impact: Enhanced Packet Processing for Modern Network Demands
DPDK’s integration has brought measurable performance gains to Suricata, providing faster packet processing for users. Major security vendors are already leveraging Suricata with the DPDK integration in their products, attesting to its reliability and scalability.
DPDK’s impact is particularly evident in high-speed environments where packet capture bottlenecks could otherwise lead to packet drops or latency. The integration allows Suricata to handle higher packet rates efficiently, extending its utility in demanding, real-time network security use cases.
Looking Forward: New Horizons with AI and Machine Learning
As artificial intelligence and machine learning applications expand across technology sectors, Suricata’s team remains open to exploring AI-driven enhancements.
Victor explained that AI’s most promising role would likely be in post-processing. Suricata currently exports JSON-formatted data, which can be fed into AI models for insights beyond immediate packet inspection.
Many current machine learning models operate at a macro level, analyzing data patterns over time rather than in real time, which aligns well with Suricata’s current functionality as a data generator for other analytics tools.
Real-time AI inference for packet processing, however, remains a challenge. Victor elaborated, “Most AI models require milliseconds for inference, which is too slow for packet-level detection in real-time.” Still, the team is ready to adopt AI models once hardware advances make real-time AI feasible.
“Most AI models require milliseconds for inference, which is too slow for packet-level detection in real-time.”
–– Victor Julien – Suricata IDS/IPS Lead Developer
Future Development: Suricata as a Library for Broader Integration
A major long-term goal for Suricata is to establish a core API, effectively transforming Suricata’s detection engine into a library that other tools can leverage.
This approach could enable seamless integration of Suricata’s capabilities with other applications, such as proxy servers, endpoint security products, and cloud-based services.
While the foundational work for this API exists, achieving a fully developed API will take time. Victor noted that this goal, motivated by growing encryption in network traffic, could broaden Suricata’s utility in increasingly secure environments.
This library initiative would allow third-party developers to incorporate Suricata’s detection features in novel ways, creating a flexible, modular ecosystem where Suricata is part of larger, more complex security infrastructures.
Expanding Community Engagement Through Events
Suricata’s annual conference, Suricon, exemplifies the project’s community-centric approach. Suricon gathers developers, users, and industry professionals to share insights, discuss roadmap goals, and showcase new features.
With a mix of training sessions and talks, Suricon provides a valuable opportunity for knowledge exchange and collaboration. DPDK community members have shown interest in attending future events, strengthening cross-community relationships, and fostering a shared development approach.
Suricata’s collaboration model has proven instrumental in its growth. This strong community foundation ensures that Suricata can keep pace with rapidly changing security demands.
Conclusion: Pushing Network Security Boundaries
Suricata’s integration with DPDK marks a significant milestone in its evolution, empowering it to achieve higher performance, greater adaptability, and better hardware compatibility.
From initial testing to real-world deployments, DPDK’s impact has been transformative, enabling Suricata to meet the demands of today’s high-speed, security-focused networks.
Through community feedback, industry collaboration, and a forward-looking approach to hardware offloading and AI, Suricata continues to redefine what’s possible in open-source network security.
As Suricata looks ahead, its development team remains committed to innovation and community-driven progress. With a roadmap that includes expanded hardware offloading, AI-driven enhancements, and new API integrations, Suricata is well-positioned to lead the next generation of network security solutions.
This DPDK integration story exemplifies how open-source collaboration can drive meaningful advancements, pushing technology forward in response to real-world needs.
Learn more about contributing to DPDK here