benthomas

Introduction

In the fast-paced world of telecommunications, companies are constantly seeking solutions to address evolving challenges and meet the demands of their customers. Ericsson, a global leader in the industry, has been at the forefront of incorporating new technologies into its product portfolio. One such technology is the Data Plane Development Kit (DPDK), which has proven instrumental in revolutionizing packet processing for Ericsson’s network infrastructure. This user story delves into Ericsson’s utilization of DPDK, the benefits it has brought, and the challenges associated with transitioning to a cloud-native environment.

Ericsson’s Shifting Landscape 

Ericsson, a prominent vendor in the telecommunications domain, has a rich history of innovation and adaptability. With over 100,000 employees and a diverse range of products, Ericsson has witnessed a significant shift from traditional infrastructure to cloud-native solutions. As the industry embraces cloud-native architectures, Ericsson recognizes the importance of incorporating new technologies that align with this paradigm shift. DPDK, though not entirely new, has emerged as a critical component within Ericsson’s product portfolio, facilitating efficient packet processing and enabling the company to remain competitive in an evolving market.

Exploring DPDK’s Role

Niklas Widell – Standardization Manager at Ericsson AB, and Maria Lingemark – Senior Software Engineer at Ericsson shed light on the company’s adoption of DPDK. Maria, who has been involved in evaluating and utilizing DPDK since 2016, emphasizes the need for high-speed packet processing and the ability to split packet flows into multiple parallel streams. DPDK’s Event Dev implementation has been instrumental in achieving these goals, enabling Ericsson to process a large number of packets per second while maintaining the flexibility to distribute packet processing across multiple steps.

Transitioning from Specialized Hardware 

Before incorporating DPDK, Ericsson relied on proprietary ASIC hardware to handle packet processing. However, the company recognized the need to shift toward more readily deployable commercial off-the-shelf (COTS) hardware solutions. DPDK played a crucial role in enabling Ericsson to transition from specialized hardware to a more versatile and scalable environment, reducing the reliance on custom solutions and increasing the reach of their offerings to a broader customer base.

Flexibility and Cost Efficiency

DPDK offers Ericsson the flexibility to deploy their packet processing solutions across a range of hardware configurations, both on ASIC hardware and on common x86-based platforms. By leveraging DPDK’s capabilities, Ericsson can scale their applications and efficiently utilize the available CPU resources. Moreover, the compatibility of DPDK with multiple drivers allows Ericsson to leverage hardware-specific features where available, enhancing performance and optimizing resource utilization.

Challenges of Observability and Cloud-Native Adoption 

As Ericsson embraces cloud-native architectures, they encounter challenges related to observability, performance monitoring, and troubleshooting. Observing and comprehending the behavior of a complex system that processes packets in parallel across multiple CPUs and threads can be daunting. Balancing observability with performance optimization becomes crucial, requiring continuous refinement and adaptation. Additionally, the shift to cloud-based deployments necessitates rethinking observability strategies and ensuring seamless performance monitoring in these environments.

We needed to shift from doing packet processing on special purpose hardware, to doing it on cloud-based general compute hardware. DPDK enabled this – it created versatility and broadened external access. It significantly helped Ericsson meet our customers’ needs and demands as those changed in scale, and gave our team greater portability as well. And the ability to be able to reuse it across different departments without having to rewrite code was, and is, a significant benefit. – Maria Lingemark, Senior Software Engineer – Ericsson

To tackle the observability challenges, Ericsson leverages the eBPF (extended Berkeley Packet Filter) integration in DPDK. By deploying eBPF programs within the DPDK framework, they have achieved efficient packet processing, improved throughput, and enhanced network visibility. The flexibility offered by eBPF allows Ericsson to tailor their networking solutions to specific use cases, ensuring optimal performance and resource utilization. 

“Ericsson uses the included eBPF support in DPDK to simplify observability in complex cloud environments.” Anders Hansen, Cloud RAN System Developer – Ericsson

DPDK BBDev (Baseband Device)

DPDK BBDev (Baseband Device) plays a critical role in Ericsson’s ability to develop a portable and efficient Radio Access Network (RAN) implementation that seamlessly integrates with hardware acceleration from leading silicon vendors. This integration enables Ericsson to leverage the full potential of specialized hardware acceleration features offered by these vendors, enhancing the performance and efficiency of their RAN solutions.

By utilizing DPDK BBDev, Ericsson gains access to a standardized programming interface that abstracts the complexities of hardware-specific optimizations. This allows them to focus on developing high-performance RAN software while ensuring compatibility with various hardware platforms. The portability provided by DPDK BBDev enables Ericsson to deploy their RAN solutions across a wide range of hardware architectures, offering flexibility to their customers, while cultivating a heath ORAN eco-system in the industry.

“DPDK BBDev enables Ericsson to create a portable and efficient RAN implementation that is well integrated with HW acceleration from major silicon vendors” – Michael Lundkvist,
Principal Developer, RAN Application Architect – Ericsson

The integration of HW acceleration from major silicon vendors further boosts Ericsson’s RAN implementation. These hardware accelerators are specifically designed to offload computationally intensive tasks, such as FEC processing, resulting in improved throughput, lower latency, and reduced power consumption. By effectively utilizing these acceleration capabilities through DPDK BBDev, Ericsson delivers efficient and high-performing RAN solutions to their customers.

For more in-depth information on how DPDK BBDev enables Ericsson’s portable and efficient RAN implementation, you can refer to the white paper provided here. This white paper will delve into the technical details and showcase the advantages of integrating DPDK BBDev with hardware acceleration from major silicon vendors, offering valuable insights into Ericsson’s innovative RAN solutions.

DPDK and the Open Source Linux Foundation Community

————————————————–

Ericsson derives substantial benefits from its active involvement in both the open-source DPDK (Data Plane Development Kit) community and the larger Linux Foundation. By being an integral part of these communities, Ericsson experiences several advantages that contribute to their success and technological advancements.

First and foremost, being part of the DPDK community grants Ericsson access to a thriving ecosystem of contributors and developers focused on advancing high-performance packet processing. This access enables Ericsson to stay at the forefront of technological developments, leverage new features, and benefit from ongoing enhancements to DPDK. The collaborative nature of the open-source community encourages continuous innovation, allowing Ericsson to deliver cutting-edge solutions to their customers.

Engaging in the DPDK community also fosters collaboration and knowledge sharing with industry peers and experts. Ericsson can exchange ideas, best practices, and insights, benefitting from the collective expertise of the community. This collaboration helps Ericsson overcome challenges, improve their solutions, and accelerate their development cycles, all while contributing to the growth and success of the DPDK project.

Furthermore, Ericsson experiences a faster time to market by utilizing DPDK and collaborating within the community. By leveraging the work done by the DPDK community, Ericsson can capitalize on existing libraries, APIs, and optimizations, saving valuable development effort and resources. This efficiency enables Ericsson to bring their solutions to market more rapidly, meeting customer demands, gaining a competitive edge, and seizing market opportunities promptly.

Interoperability and compatibility are additional advantages that Ericsson enjoys through their involvement in the DPDK community and the larger Linux Foundation. DPDK’s emphasis on interoperability and common standards allows Ericsson to seamlessly integrate their solutions with other systems and platforms. This compatibility fosters a broader ecosystem, enabling Ericsson to collaborate effectively with other vendors and organizations, further expanding their market reach.

Participating in these open-source communities also positions Ericsson as an influential player and thought leader in high-performance networking and packet processing. Their contributions to the DPDK project not only enhance the framework’s functionality but also demonstrate their technical expertise and commitment to open-source initiatives. Ericsson’s influence and leadership within the community allow them to shape the direction and evolution of DPDK, driving the adoption of industry standards and best practices.

Lastly, being part of the larger Linux Foundation ecosystem offers Ericsson access to a vast network of organizations, developers, and industry leaders. This ecosystem provides collaboration opportunities, potential partnerships, and access to a network of expertise. By leveraging these connections, Ericsson can foster innovation, explore joint development efforts, and stay at the forefront of technological advancements in networking and telecommunications.

Author: Michel Machado – michel@digirati.com.br

Overview 

Originally developed at Boston University, Gatekeeper is the brainchild of researchers who looked at the state of distributed denial-of-service (DDoS) attacks and realized that the community lacked an affordable, performant, and deployable solution to defending against such attacks. On one hand, cloud companies offer DDoS protection as a service, but this can be costly. On the other hand, many research proposals have been developed to allow Internet operators to protect their own networks, but none of these ideas have yielded production-quality software systems. Gatekeeper puts theory into practice by providing network operators with an instantly deployable and affordable solution for defending against DDoS attacks, and does so without sacrificing performance by leveraging DPDK as a key enabling technology.

The Challenge

Part of the challenge in defending against DDoS attacks is differentiating good traffic from bad traffic in seconds. To do so most effectively requires capturing the qualities of individual flows as they pass through the DDoS mitigation system — this allows the system to rate limit flows, apply policies based on the traffic features, and punish flows that misbehave by blocking them completely. Capturing these qualities for each packet that passes through the system requires an extreme amount of CPU and memory resources, especially during attacks that nowadays stretch beyond 1 Tbps. To withstand attacks of this magnitude, DDoS mitigation systems either need to be widely deployed in parallel (which is expensive), or need to be especially careful in how they process packets. The latter is where Gatekeeper utilizes DPDK to be able to work efficiently and affordably.

The Solution

To be able to process packets at this scale, kernel bypass is absolutely necessary. We chose DPDK as a kernel bypass solution because of its stability and support from industry, as well as the feature set that it supports. In fact, the feature set of DPDK is so rich that it significantly reduced our time to market since we did not have to write everything from scratch.

Gatekeeper heavily relies on three key features in DPDK: (1) NUMA-aware memory management, (2) burst packet I/O, and (3) eBPF. These features allow Gatekeeper to enforce policies as programs instead of firewall rules, and to do so efficiently. This gives operators a lot of flexibility in determining how flows are processed by Gatekeeper without having to sacrifice performance.

On occasion, we found some shortcomings in DPDK libraries. For example, the LPM6/FIB6/RIB6 libraries that perform longest prefix matching on IPv6 were not a good fit, and we had to implement our own. But for each issue we have come across, we’ve found huge success with other libraries as described below. Furthermore, the community is hard at work to address production demands such as dynamically setting memory zones (see rte_memzone_max_set() for details), which previously required patching DPDK to change.

The Results

With DPDK, Gatekeeper achieves the following benefits:

• NUMA-aware memory management allows Gatekeeper to reduce memory access latency by enabling CPU cores to access local memory instead of remote memory.
• Burst packet I/O reduces the per-packet cost of accessing and updating queues, enabling Gatekeeper to keep up with volumetric attacks.
• eBPF (integrated in DPDK) enables deployers to write policies that are impossible to express in other solutions such as requiring TCP friendliness, bandwidth per flow, and quotas for auxiliary packets (e.g. ICMP, TCP SYN, fragments) per flow. Thanks to the guarantee of termination of eBPF programs, Gatekeeper can gracefully continue processing packets even when an eBPF program is buggy.

Many other DPDK features, including prefetching, the kernel-NIC interface, and packet filters play key supporting roles. With DPDK’s help, a modest Gatekeeper server can track 2 billion flows while processing at the very least 10 Mpps through eBPF program policies to decide how to allow, rate limit, or drop traffic.

Gatekeeper puts DDoS defense back on the hands of network operators, administrators and the general Open Source community. What was until recently only available via opaque and expensive third-party services can now be deployed by anyone with the appropriate infrastructure, with levels of flexibility and control that simply did not formerly exist. Andre Nathan – Digirati

The Benefits

DDoS attacks cause great financial, political, and social damage, and are only increasing in magnitude, complexity, and frequency. With Gatekeeper, network operators have a production-quality, open source choice in the market to defend their infrastructure and services. With the aid of technologies like DPDK, Gatekeeper is able to flexibly and efficiently defend against attacks, lowering the cost of deployment and enabling many stakeholders to protect themselves. To date, Gatekeeper has been deployed by Internet service providers, data centers, and gaming companies, and hopes to reach new deployers to eventually eradicate DDoS attacks.

Check out their GitHub here

White paper link 

Have a user story of your own that you would like to share across the DPDK and Linux foundation communities? Submit one here.

The Company / Product

SmartShare Systems is a small privately held company founded in 2006 by Morten Brørup in Denmark. SmartShare Systems develops innovative network appliances and related services with R&D in Denmark and hardware manufacturing in Taiwan. Their solutions have quickly become popular and are currently used by schools, commercial businesses, apartment buildings, hotels, military bases, cruise ships and internet service providers. The products are sold through value-added resellers, with expertise in the field of networks and system integration. SmartShare’s main product line, the StraightShaper products, is focused on WAN Optimization. WAN Optimization is typically used to reduce the data consumption on a costly WAN link. However, the primary purpose of the StraightShaper appliance is to make the WAN link (typically an internet connection) run smoothly for every user. WAN Optimization is relevant when users don’t have access to unlimited WAN bandwidth, or if the network infrastructure doesn’t have infinite bandwidth capacity, e.g:

• A drilling rig crew sharing a VSAT satellite internet connection.
• Soldiers in a military camp in the middle of nowhere, sharing whatever internet connection is available.
• Cruise ship guests using on-board Wi-Fi, sharing the ship’s LTE/5G antenna array or Starlink satellite internet connection while at sea.
• Students taking their final exams online at the school gym, sharing the school’s fiber internet connection.
• (But probably not for a family of four sharing a gigabit fiber internet connection at home.)

The key WAN Optimization technologies in the StraightShaper products are:

• User Load Balancing: Distributing the available bandwidth to the active users ensures that everyone has bandwidth all the time. This can include configuration options to assign various priorities and bandwidths to individual subscribers.
• Bufferbloat Prevention: All network products have buffers, where the packets can queue up and
  cause increased latency. This is known as “network lag” by online gamers and “bufferbloat” by network professionals. By managing the buffers with this in mind, bufferbloat can be prevented, so the users are not exposed to excessive delays when using the internet.
• Dynamic Quality of Service: Automatically detecting and prioritizing voice packets over data packets ensures good sound quality for IP telephony.
• Caching: Caching e.g. DNS replies reduces the total time it takes to load a web page, if someone else has recently visited the same web site.
• Content Filtering: By optionally blocking certain internet services that use a lot of bandwidth, internet link capacity is freed up for other purposes.

“Development velocity is much higher with DPDK than it was with our Linux kernel-based product line. With DPDK, we only have to develop what we need, and it makes our application code cleaner than when trying to fit our code into some other framework.”
MORTEN BRØRUP, CTO, SMARTSHARE SYSTEMS

The Challenge – WAN Optimization and the Linux kernel:

Some years ago, as bandwidth demands increased, SmartShare’s initial StraightShaper product, based on the Linux kernel, started facing challenges as Linux is not designed for highly specialized packet processing, nor does it support it well. This presented two main challenges:

• Performance: The Linux kernel’s “qdisc” shaping system does not scale to multiple cores per network interface, and rewriting the kernel would be a major effort; the product could not scale beyond a few gigabits per second, which customers were starting to look for.
• Complexity: The Linux kernel’s IP stack is extremely advanced and feature-rich, which is great for many purposes. In the Linux kernel, each packet passes through a large number of predefined functions and hooks, and depending on various criteria, packets take different routes through these functions and hooks. SmartShare’s products only use very few of these features, and don’t always fit perfectly into the predefined routes of the functions and hooks. Those other features, however,
  would sometimes get in the way and create unwanted complexity for SmartShare’s developers.

It became clear that with customer demand for bandwidths beyond 1 Gbit/s on the rise, the Linux kernel-based StraightShaper was not scalable.

“We have customers today that we wouldn’t have if we didn’t use DPDK, leading to revenue that would not be generated otherwise.”
MORTEN BRØRUP, CTO, SMARTSHARE SYSTEMS

The Solution

Given the scalability issues of the Linux kernel in specialized packet processing, combined with anticipation of increased customer demands for high bandwidth, SmartShare Systems decided to develop the next generation StraightShaper solutions using DPDK instead of the Linux kernel. DPDK enables developers to decide which functions the packets pass through, and when. This allows
developers to design their own flow (vs. adapting to pre-set routes through the system), and can pick and choose from DPDK libraries and functions.

However, this meant writing a whole new architecture from scratch to support and scale to multiple cores for increased processing, analysis and egress packet scheduling. Most publicly known DPDK projects are based on a “run-to-completion” design. The SmartShare StraightShaper CSP uses a lot of packet buffering, so SmartShare chose a “pipeline” design and developed its framework such that
available CPU cores are assigned to one or more pipeline stages as appropriate.

The Results

When we started using DPDK, it was more or less an ambition to create a version of the existing product, but based on DPDK to generate added performance (e.g. more than 1 Gbit/s). It quickly became clear that working with DPDK makes it much easier to develop these network appliances; and
DPDK’s well-documented library of functions is robust, mature and reliable.

Performance Impact

When the development of the DPDK based StraightShaper CSP firmware began back in 2016 — with the goal of creating a version of the existing product but with added performance — it was internally named “the 10 Gbit/s project”, because that was the problem it was supposed to solve. However, when the new DPDK based product was ready for testing, it was quickly apparent that not only did it push 10 Gbit/s, but easily pushed much more. Referring to it as “the 100 Gbit/s project” would be more appropriate, as the DPDK based firmware easily handles that, and more.

“When we started development of our DPDK based StraightShaper CSP firmware back in 2016, we named it ‘the 10 Gbit/s project’ because that was the problem it was supposed to solve. Now, we know that ‘the 100 Gbit/s project’ would be more appropriate, as our DPDK based firmware can easily handle that, and more!”

MORTEN BRØRUP, CTO, SMARTSHARE SYSTEMS

The Benefits – Impact on Complexity

As mentioned previously, DPDK enables developers to pick and choose which functions packets pass through, and when. This greatly simplifies the entire process and generates results faster and more efficiently.

DPDK enables adding more advanced features to the product, such as specific bandwidth allocation, bufferbloat prevention, and bandwidth shaping within the network core (i.e. inside the SmartShare appliance vs. in low-cost switches at the edge of the network).

Because the SmartShare manages the bandwidth in the core of the network, where the bandwidth capacity is extremely high, bursts and microbursts can be easily absorbed and smoothed out, so they don’t reach the edge of the network and cause packet drops and/or latency issues.

Currently, SmartShare Systems maintains both the Linux kernel-based product line (“StraightShaper”) and the more high-end DPDK-based product line (“StraightShaper CSP”). The new StraightShaper CSP has been deployed in customers’ production networks since 2019, and is a fully mature product,
which continuously evolves with improvements and new features with each firmware release.

Looking ahead, SmartShare Systems has plans to add all features of the initial Linux kernel-based product into the DPDK version. They are also looking at other new projects that leverage DPDK for other use cases, still under development