[dpdk-dev] [PATCH RFC] Memcpy optimization

[Linhaifeng]

On 2014/11/14 17:08, Wang, Zhihong wrote:
> Hi all,
> 
> I'd like to propose an update on DPDK memcpy optimization.
> Please see RFC below for details.
> 
> 
> Thanks
> John
> 
> ---
> 
> DPDK Memcpy Optimization
> 
> 1. Introduction
> 2. Terminology
> 3. Mechanism
>     3.1 Architectural Insights
>     3.2 DPDK memcpy optimization
>     3.3 Code change
> 4. Glibc memcpy analysis
> Acknowledgements
> Author's Address
> 
> 
> 1. Introduction
> 
> This document describes DPDK memcpy optimization, for both SSE and AVX platforms.
> 
> Glibc memcpy is for general uses, it's not so efficient for DPDK where copies are small and from cache to cache mainly.
> Also, glibc is changing over versions, some tradeoffs it made have negative impact on DPDK performance. This in the meantime makes DPDK memcpy performance glibc version dependent.
> For this cause, it's necessary to maintain a standalone memcpy implementation to take full advantage of hardware features, and make special optimization aiming at DPDK scenarios.
> 
> Current DPDK memcpy has the following improvement areas:
> * No support for 256-bit load/store
> * Poor performance for unaligned cases
> * Performance drops at certain odd copy sizes
> * Make slow glibc call for constant copies
> 
> It can be improved significantly by utilizing 256-bit AVX instructions and applying more optimization techniques.
> 
> 2. Terminology
> 
> Aligned copy: Same offset for source & destination starting addresses
> Unaligned copy: Different offsets for source & destination starting addresses
> Constant payload size: Copy length can be decided at compile time
> Variable payload size: Copy length can't be decided at compile time
> 
> 3. Mechanism
> 
> 3.1 Architectural Insights
> 
> New architectures are likely to have better cache performance and stronger ISA implementation.
> Memcpy needs to make full utilization of cache bandwidth, and implement different mechanisms according to hardware features.
> Below is the architecture analysis for memory performance in Haswell and Sandy Bridge.
> 
> Haswell has significant improvements in memory hierarchy over Sandy Bridge:
> * 2x cache bandwidth: From 48 B/cycle to 96 B/cycle
> * Sandy Bridge suffers from L1D bank conflicts, Haswell doesn't
> * Sandy Bridge has 2 split line buffers, Haswell has 4
> * Forwarding latency is 2 cycles for 256-bit AVX loads in Sandy Bridge, 1 in Haswell
> 
> 3.2 DPDK memcpy optimization
> 
> DPDK memcpy calls are mainly cache to cache cases with payload no larger than 8KB, they can be categorized into 4 scenarios:
> * Aligned copy, with constant payload size
> * Aligned copy, with variable payload size
> * Unaligned copy, with constant payload size
> * Unaligned copy, with variable payload size
> 
> Each scenario should be optimized according to its characteristics:
> * For aligned cases, no special optimization techniques are required
> * For unaligned cases:
>     * Make store address aligned is a basic technique to improve performance
>     * Load address alignment is a tradeoff between bit shifting overhead and unaligned memory access penalty, which should be assessed by test
>     * Load/store address should be made available as early as possible to fully utilize the pipeline
> * For constant cases, inlining can bring significant benefits by means of gcc optimization at compile time
> * For variable cases, it's important to reduce branches and make good use of hardware prefetch
> 
> Memcpy optimization is summarized below:
> * Utilize full cache bandwidth
>     * SSE: 128 bit
>     * AVX/AVX2: 128/256 bit, depends on hardware implementation
> * Enforce aligned stores
> * Apply load address alignment based on architecture features
>     * Enforce aligned loads for Sandy Bridge like architectures
>     * No need to enforce aligned loads for Haswell because unaligned loads is improved, also the AVX2 VPALIGNR is not efficient for 256-bit shifting and leads to extra overhead
> * Make load/store address available as early as possible
> 
> Finally, general optimization techniques should be applied, like inlining, branch reducing, prefetch pattern access, etc.
> 

Is this optimization in compile time or run time?

> 3.3 Code change
> 
> DPDK memcpy is implemented in a standalone file "rte_memcpy.h".
> The memcpy function is "rte_memcpy_func", which contains the copy flow, and calls the inline move functions for actual data copy.
> 
> There will be major code change described as follows:
> * Differentiate architectural features based on CPU flags
>     * Implement separated copy flow specifically optimized for target architecture
>     * Implement separated move functions for SSE/AVX/AVX2 to make full utilization of cache bandwidth
> * Rewrite the memcpy function "rte_memcpy_func"
>     * Add store aligning
>     * Add load aligning for Sandy Bridge and older architectures
>     * Put block copy loop into inline move functions for better control of instruction order
>     * Eliminate unnecessary MOVs
> * Rewrite the inline move functions
>     * Add move functions for unaligned load cases for Sandy Bridge and older architectures
>     * Change instruction order in copy loops for better pipeline utilization
>     * Use intrinsics instead of assembly code
> * Remove slow glibc call for constant copies
> 
> Current memcpy performance test is in "test_memcpy_perf.c", which will also be updated with unaligned test cases.
> 
> 4. Glibc memcpy analysis
> 
> Glibc 2.16 (Fedora 20) and 2.20 (Currently the latest, released on Sep 07, 2014) are analyzed.
> 
> Glibc 2.16 issues:
> * No support for 256-bit load/store
> * Significant slowdown for unaligned constant cases due to split loads and 4k aliasing
> 
> Glibc 2.20 issue:
> * Removed load address alignment, which can lead to significant slowdown for unaligned cases in former architectures like Sandy Bridge
> 
> Also, calls to glibc can't be optimized by gcc at compile time.
> 
> Acknowledgements
> 
> Valuable suggestions from: Liang Cunming, Zhu Heqing, Bruce Richardson, and Chen Wenjun.
> 
> Author's Address
> 
> Wang Zhihong (John)
> Email: zhihong.wang at intel.com
> 
> 
> .
>