[dpdk-dev] [RFC v2] doc compression API for DPDK

[Trahe, Fiona]

Hi Ahmed, Shally,

> -----Original Message-----
> From: Verma, Shally [mailto:Shally.Verma at cavium.com]
> Sent: Thursday, February 15, 2018 5:53 AM
> To: Ahmed Mansour <ahmed.mansour at nxp.com>; Trahe, Fiona <fiona.trahe at intel.com>;
> dev at dpdk.org
> Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish
> <Ashish.Gupta at cavium.com>; Sahu, Sunila <Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo
> <pablo.de.lara.guarch at intel.com>; Challa, Mahipal <Mahipal.Challa at cavium.com>; Jain, Deepak K
> <deepak.k.jain at intel.com>; Hemant Agrawal <hemant.agrawal at nxp.com>; Roy Pledge
> <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
> Subject: RE: [RFC v2] doc compression API for DPDK
> 
> 
> 
> >-----Original Message-----
> >From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
> >Sent: 14 February 2018 22:25
> >To: Verma, Shally <Shally.Verma at cavium.com>; Trahe, Fiona <fiona.trahe at intel.com>; dev at dpdk.org
> >Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish
> <Ashish.Gupta at cavium.com>; Sahu, Sunila
> ><Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo <pablo.de.lara.guarch at intel.com>; Challa,
> Mahipal
> ><Mahipal.Challa at cavium.com>; Jain, Deepak K <deepak.k.jain at intel.com>; Hemant Agrawal
> <hemant.agrawal at nxp.com>; Roy
> >Pledge <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
> >Subject: Re: [RFC v2] doc compression API for DPDK
> >
> >On 2/14/2018 12:41 AM, Verma, Shally wrote:
> >> Hi Ahmed
> >>
> >>> -----Original Message-----
> >>> From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
> >>> Sent: 02 February 2018 02:20
> >>> To: Trahe, Fiona <fiona.trahe at intel.com>; Verma, Shally <Shally.Verma at cavium.com>;
> dev at dpdk.org
> >>> Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish
> <Ashish.Gupta at cavium.com>; Sahu, Sunila
> >>> <Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo <pablo.de.lara.guarch at intel.com>; Challa,
> Mahipal
> >>> <Mahipal.Challa at cavium.com>; Jain, Deepak K <deepak.k.jain at intel.com>; Hemant Agrawal
> <hemant.agrawal at nxp.com>; Roy
> >>> Pledge <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
> >>> Subject: Re: [RFC v2] doc compression API for DPDK
> >>>
> >>>>>> [Fiona] I propose if BFINAL bit is detected before end of input
> >>>>>> the decompression should stop. In this case consumed will be < src.length.
> >>>>>> produced will be < dst buffer size. Do we need an extra STATUS response?
> >>>>>> STATUS_BFINAL_DETECTED  ?
> >>>>> [Shally] @fiona, I assume you mean here decompressor stop after processing Final block right?
> >>>> [Fiona] Yes.
> >>>>
> >>>>  And if yes,
> >>>>> and if it can process that final block successfully/unsuccessfully, then status could simply be
> >>>>> SUCCESS/FAILED.
> >>>>> I don't see need of specific return code for this use case. Just to share, in past, we have practically
> run into
> >>>>> such cases with boost lib, and decompressor has simply worked this way.
> >>>> [Fiona] I'm ok with this.
> >>>>
> >>>>>> Only thing I don't like this is it can impact on performance, as normally
> >>>>>> we can just look for STATUS == SUCCESS. Anything else should be an exception.
> >>>>>> Now the application would have to check for SUCCESS || BFINAL_DETECTED every time.
> >>>>>> Do you have a suggestion on how we should handle this?
> >>>>>>
> >>> [Ahmed] This makes sense. So in all cases the PMD should assume that it
> >>> should stop as soon as a BFINAL is observed.
> >>>
> >>> A question. What happens ins stateful vs stateless modes when
> >>> decompressing an op that encompasses multiple BFINALs. I assume the
> >>> caller in that case will use the consumed=x bytes to find out how far in
> >>> to the input is the end of the first stream and start from the next
> >>> byte. Is this correct?
> >> [Shally]  As per my understanding, each op can be tied up to only one stream as we have only one
> stream pointer per op and one
> >stream can have only one BFINAL (as stream is one complete compressed data) but looks like you're
> suggesting a case where one op
> >can carry multiple independent streams? and thus multiple BFINAL?! , such as, below here is op
> pointing to more than one streams
> >>
> >>             --------------------------------------------
> >> op --> |stream1|stream2| |stream3|
> >>            --------------------------------------------
> >>
> >> Could you confirm if I understand your question correct?
> >[Ahmed] Correct. We found that in some storage applications the user
> >does not know where exactly the BFINAL is. They rely on zlib software
> >today. zlib.net software halts at the first BFINAL. Users put multiple
> >streams in one op and rely on zlib to  stop and inform them of the end
> >location of the first stream.
> 
> [Shally] Then this is practically case possible on decompressor and decompressor doesn't regard flush
> flag. So in that case, I expect PMD to internally reset themselves (say in case of zlib going through cycle
> of deflateEnd and deflateInit or deflateReset) and return with status = SUCCESS with updated produced
> and consumed. Now in such case, if previous stream also has some footer followed by start of next
> stream, then I am not sure how PMD / lib can support that case. Have you had practically run of such
> use-case on zlib? If yes, how then such application handle it in your experience?
> I can imagine for such input zlib would return with Z_FLUSH_END after 1st BFINAL is processed to the
> user. Then application doing deflateReset() or Init-End() cycle before starting with next. But if it starts
> with input that doesn't have valid zlib header, then likely it will throw an error.
> 
[Fiona] The consumed and produced tell the Application hw much data was processed up to 
the end of the first deflate block encountered with a bfinal set.
If there is data, e.g. footer after the block with bfinal, then I think it must be the responsibility of
the application to know this, the PMD can't have any responsibility for this.
The next op sent to the PMD must start with a valid deflate block.


> >>
> >> Thanks
> >> Shally
> >>