[dpdk-dev] [RFC v2] Compression API in DPDK

Trahe, Fiona fiona.trahe at intel.com
Fri Nov 24 17:54:40 CET 2017


With the vast amounts of data being transported around networks and stored in storage systems, reducing data size is becoming ever more important. There are both software libraries and hardware devices available that provide compression, but no common API. This RFC proposes a compression API for DPDK to address this need.

Features:
• Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
• LZS algorithm (https://tools.ietf.org/html/rfc2395)
• Static and Dynamic Huffman encoding.
• Compression levels
• Checksum generation
• Asynchronous burst API
• Session-based (for stateless a session may be useable across devices, for stateful it wouldn’t be.) 


Note 1: Split of functionality above/below API
When considering whether features should be supported on the API or not, the decision was based on the following:
The purpose of the API is to decouple the application from the compute-intensive functions needed for compression by abstracting them under a common API. These can then be implemented by either hardware accelerators or optimised software libraries. Where features are not compute-intensive and unlikely to be offloaded or optimised, there’s nothing to be gained by each PMD having to separately implement them, and it makes more sense for them to be done above the API. So the following are not handled on the API and can be done above. 
• Prepending/appending protocol headers (gzip, zlib) 
• File-handling, breaking files up into packets, reassembling.
• Synchronous API


Note 2: Stateful compression.
A better compression ratio can be achieved by using stateful compression, allowing pattern matching across multiple sequential packets. This will be supported on the API, but in the interests of getting an initial RFC out for feedback, there are placeholder parameters on the API. Feedback welcome on what parameters are needed for this feature.


Note 3: The tricky question of where the API belongs
We considered 
1. Extending cryptodev
2. New acceldev APIs for device handling + compressdev APIs for data path 
3. New acceldev for all APIs
4. New compressdev API
We've gone with option 4, a compressdev API.  See original RFC for reasons.
We explored wrapping this around a generic acceldev that would be hidden from the API 
but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
but this added complexity and indirection and didn't add enough value, so we've abandoned it.

Opens:
 - Clarify stateful behaviour
 - Define Capability structures and API
 - Define structures and API for proposed hash functionality
 - simplifications possible in code derived from cryptodev, but maybe over-engineered for compressdev, e.g.
   - session architected to be useable across PMDs
   - xform chains


Changes from RFC v1:
 - rebased off 17.11 and extended to include all device APIs and to compile.
 - Added LZS algorithm
 - Clarified NULL algorithm
 - Use FIXED terminology rather than STATIC for Huffman type
 - removed proposed data verification functionality
 - quashed xform chain into a single xform for each direction
 - added FLUSH_SYNC
 - renamed OVERFLOW to OUT_OF_SPACE and clarified comment
 - added #defines for Level values
 - clarified how src and dst buffer lengths are passed on API

http://dpdk.org/ml/archives/dev/2017-October/078944.html


Trahe, Fiona (1):
  [RFC v2] lib: add compressdev API

 config/common_base                                 |    7 +
 lib/Makefile                                       |    3 +
 lib/librte_compressdev/Makefile                    |   54 +
 lib/librte_compressdev/rte_comp.h                  |  565 ++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 1181 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  863 ++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       |  193 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  535 +++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   58 +
 lib/librte_eal/common/include/rte_log.h            |    1 +
 10 files changed, 3460 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map



More information about the dev mailing list