[dpdk-dev] [RFC v2] doc compression API for DPDK

[Verma, Shally]

>-----Original Message-----
>From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
>Sent: 14 February 2018 22:25
>To: Verma, Shally <Shally.Verma at cavium.com>; Trahe, Fiona <fiona.trahe at intel.com>; dev at dpdk.org
>Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish <Ashish.Gupta at cavium.com>; Sahu, Sunila
><Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo <pablo.de.lara.guarch at intel.com>; Challa, Mahipal
><Mahipal.Challa at cavium.com>; Jain, Deepak K <deepak.k.jain at intel.com>; Hemant Agrawal <hemant.agrawal at nxp.com>; Roy
>Pledge <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
>Subject: Re: [RFC v2] doc compression API for DPDK
>
>On 2/14/2018 12:41 AM, Verma, Shally wrote:
>> Hi Ahmed
>>
>>> -----Original Message-----
>>> From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
>>> Sent: 02 February 2018 02:20
>>> To: Trahe, Fiona <fiona.trahe at intel.com>; Verma, Shally <Shally.Verma at cavium.com>; dev at dpdk.org
>>> Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish <Ashish.Gupta at cavium.com>; Sahu, Sunila
>>> <Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo <pablo.de.lara.guarch at intel.com>; Challa, Mahipal
>>> <Mahipal.Challa at cavium.com>; Jain, Deepak K <deepak.k.jain at intel.com>; Hemant Agrawal <hemant.agrawal at nxp.com>; Roy
>>> Pledge <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
>>> Subject: Re: [RFC v2] doc compression API for DPDK
>>>
>>>>>> [Fiona] I propose if BFINAL bit is detected before end of input
>>>>>> the decompression should stop. In this case consumed will be < src.length.
>>>>>> produced will be < dst buffer size. Do we need an extra STATUS response?
>>>>>> STATUS_BFINAL_DETECTED  ?
>>>>> [Shally] @fiona, I assume you mean here decompressor stop after processing Final block right?
>>>> [Fiona] Yes.
>>>>
>>>>  And if yes,
>>>>> and if it can process that final block successfully/unsuccessfully, then status could simply be
>>>>> SUCCESS/FAILED.
>>>>> I don't see need of specific return code for this use case. Just to share, in past, we have practically run into
>>>>> such cases with boost lib, and decompressor has simply worked this way.
>>>> [Fiona] I'm ok with this.
>>>>
>>>>>> Only thing I don't like this is it can impact on performance, as normally
>>>>>> we can just look for STATUS == SUCCESS. Anything else should be an exception.
>>>>>> Now the application would have to check for SUCCESS || BFINAL_DETECTED every time.
>>>>>> Do you have a suggestion on how we should handle this?
>>>>>>
>>> [Ahmed] This makes sense. So in all cases the PMD should assume that it
>>> should stop as soon as a BFINAL is observed.
>>>
>>> A question. What happens ins stateful vs stateless modes when
>>> decompressing an op that encompasses multiple BFINALs. I assume the
>>> caller in that case will use the consumed=x bytes to find out how far in
>>> to the input is the end of the first stream and start from the next
>>> byte. Is this correct?
>> [Shally]  As per my understanding, each op can be tied up to only one stream as we have only one stream pointer per op and one
>stream can have only one BFINAL (as stream is one complete compressed data) but looks like you're suggesting a case where one op
>can carry multiple independent streams? and thus multiple BFINAL?! , such as, below here is op pointing to more than one streams
>>
>>             --------------------------------------------
>> op --> |stream1|stream2| |stream3|
>>            --------------------------------------------
>>
>> Could you confirm if I understand your question correct?
>[Ahmed] Correct. We found that in some storage applications the user
>does not know where exactly the BFINAL is. They rely on zlib software
>today. zlib.net software halts at the first BFINAL. Users put multiple
>streams in one op and rely on zlib to  stop and inform them of the end
>location of the first stream.

[Shally] Then this is practically case possible on decompressor and decompressor doesn't regard flush flag. So in that case, I expect PMD to internally reset themselves (say in case of zlib going through cycle of deflateEnd and deflateInit or deflateReset) and return with status = SUCCESS with updated produced and consumed. Now in such case, if previous stream also has some footer followed by start of next stream, then I am not sure how PMD / lib can support that case. Have you had practically run of such use-case on zlib? If yes, how then such application handle it in your experience? 
I can imagine for such input zlib would return with Z_FLUSH_END after 1st BFINAL is processed to the user. Then application doing deflateReset() or Init-End() cycle before starting with next. But if it starts with input that doesn't have valid zlib header, then likely it will throw an error.

>>
>> Thanks
>> Shally
>>