[dpdk-dev] [RFC v3 0/1] Compression API in DPDK

[Trahe, Fiona]

With the vast amounts of data being transported around networks and stored in
storage systems, reducing data size is becoming ever more important. There
are both software libraries and hardware devices available that provide
compression, but no common API. This RFC proposes a compression API for
DPDK to address this need.

Features:
• Deflate Algorithm (https://tools.ietf.org/html/rfc1951)
• LZS algorithm (https://tools.ietf.org/html/rfc2395)
• Static and Dynamic Huffman encoding.
• Compression levels
• Checksum generation
• Asynchronous burst API
• Session-based (a session contains immutable data only and is useable across devices) 
• stream-based to maintain state and history data for stateful flows.

Note 1: Split of functionality above/below API
When considering whether features should be supported on the API or not, the
decision was based on the following:
The purpose of the API is to decouple the application from the compute-intensive
functions needed for compression by abstracting them under a common API. These
can then be implemented by either hardware accelerators or optimised software
libraries. Where features are not compute-intensive and unlikely to be
offloaded or optimised, there’s nothing to be gained by each PMD having
to separately implement them, and it makes more sense for them to be done
above the API. So the following are not handled on the API and can be done above. 
• Prepending/appending protocol headers (gzip, zlib) 
• File-handling, breaking files up into packets, reassembling.
• Synchronous API
• Serialisation of stateful requests


Note 2: The tricky question of where the API belongs
We considered 
1. Extending cryptodev
2. New acceldev APIs for device handling + compressdev APIs for data path 
3. New acceldev for all APIs
4. New compressdev API
We've gone with option 4, a compressdev API.  See original RFC [1] for reasons.
We explored wrapping this around a generic acceldev that would be hidden from the API 
but could be common to cryptodev, compressdev and other accelerators on the PMD interface,
but this added complexity and indirection and didn't add enough value, so we've abandoned it.

Opens:
 - Define structures and API for proposed hash functionality
 - Agree on stateful behaviour
 - Complete capability APIs 

Changes in RFC v3:
 - Fixed incorrect op pool element size
 - removed unnecessary rte_compressdev_qp_conf
   and renamed nb_descriptors to max_inflight_ops
 - Removed session_pool from qp setup as not needed
 - Removed session_pool from rte_compressdev_data as not needed
 - renamed CDEV_ to COMPDEV_ to avoid build clash with cryptodev log
 - fixed rte_comp_op_reset() so phys_addr and mempool not deleted.
 - added window size to xforms
 - added lib dependency in rte.app.mk
 - returned PMD value directly in rte_compressdev_get_private_session_size()
   as no need to adjust for hdr size as not supporting sessionless
 - renamed rte_compressdev_session_free()->..terminate()
 - added capability structs
 - added stateless/stateful enum, stream APIs and text about flush flag and stateful behaviour
 - removed intermediate_buffer_size from device info and config structs as not generic.
   Can be handled by PMD-specific config in the config file instead.
 - added max_nb_streams_per_qp to info struct.
 - Fix checkpatch errors.


Changes in RFC v2:
 - rebased off 17.11 and extended to include all device APIs and to compile.
 - Added LZS algorithm
 - Clarified NULL algorithm
 - Use FIXED terminology rather than STATIC for Huffman type
 - removed proposed data verification functionality
 - quashed xform chain into a single xform for each direction
 - added FLUSH_SYNC
 - renamed OVERFLOW to OUT_OF_SPACE and clarified comment
 - added #defines for Level values
 - clarified how src and dst buffer lengths are passed on API

[1] http://dpdk.org/ml/archives/dev/2017-October/078944.html


Trahe, Fiona (1):
  lib: Add compressdev API

 config/common_base                                 |    7 +
 lib/Makefile                                       |    3 +
 lib/librte_compressdev/Makefile                    |   54 +
 lib/librte_compressdev/rte_comp.h                  |  608 ++++++++++
 lib/librte_compressdev/rte_compressdev.c           | 1167 ++++++++++++++++++++
 lib/librte_compressdev/rte_compressdev.h           |  892 +++++++++++++++
 lib/librte_compressdev/rte_compressdev_pmd.c       |  194 ++++
 lib/librte_compressdev/rte_compressdev_pmd.h       |  533 +++++++++
 lib/librte_compressdev/rte_compressdev_version.map |   50 +
 lib/librte_eal/common/include/rte_log.h            |    1 +
 mk/rte.app.mk                                      |    1 +
 11 files changed, 3510 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_compressdev/Makefile
 create mode 100644 lib/librte_compressdev/rte_comp.h
 create mode 100644 lib/librte_compressdev/rte_compressdev.c
 create mode 100644 lib/librte_compressdev/rte_compressdev.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.c
 create mode 100644 lib/librte_compressdev/rte_compressdev_pmd.h
 create mode 100644 lib/librte_compressdev/rte_compressdev_version.map