[dpdk-dev] [RFC v2] doc compression API for DPDK

[Trahe, Fiona]

> -----Original Message-----
> From: Ahmed Mansour [mailto:ahmed.mansour at nxp.com]
> Sent: Thursday, February 15, 2018 7:51 PM
> To: Trahe, Fiona <fiona.trahe at intel.com>; Verma, Shally <Shally.Verma at cavium.com>; dev at dpdk.org
> Cc: Athreya, Narayana Prasad <NarayanaPrasad.Athreya at cavium.com>; Gupta, Ashish
> <Ashish.Gupta at cavium.com>; Sahu, Sunila <Sunila.Sahu at cavium.com>; De Lara Guarch, Pablo
> <pablo.de.lara.guarch at intel.com>; Challa, Mahipal <Mahipal.Challa at cavium.com>; Jain, Deepak K
> <deepak.k.jain at intel.com>; Hemant Agrawal <hemant.agrawal at nxp.com>; Roy Pledge
> <roy.pledge at nxp.com>; Youri Querry <youri.querry_1 at nxp.com>
> Subject: Re: [RFC v2] doc compression API for DPDK
> 
> /// snip ///
> >>>>>
> >>>>>>>> [Fiona] I propose if BFINAL bit is detected before end of input
> >>>>>>>> the decompression should stop. In this case consumed will be < src.length.
> >>>>>>>> produced will be < dst buffer size. Do we need an extra STATUS response?
> >>>>>>>> STATUS_BFINAL_DETECTED  ?
> >>>>>>> [Shally] @fiona, I assume you mean here decompressor stop after processing Final block right?
> >>>>>> [Fiona] Yes.
> >>>>>>
> >>>>>>  And if yes,
> >>>>>>> and if it can process that final block successfully/unsuccessfully, then status could simply be
> >>>>>>> SUCCESS/FAILED.
> >>>>>>> I don't see need of specific return code for this use case. Just to share, in past, we have
> practically
> >> run into
> >>>>>>> such cases with boost lib, and decompressor has simply worked this way.
> >>>>>> [Fiona] I'm ok with this.
> >>>>>>
> >>>>>>>> Only thing I don't like this is it can impact on performance, as normally
> >>>>>>>> we can just look for STATUS == SUCCESS. Anything else should be an exception.
> >>>>>>>> Now the application would have to check for SUCCESS || BFINAL_DETECTED every time.
> >>>>>>>> Do you have a suggestion on how we should handle this?
> >>>>>>>>
> >>>>> [Ahmed] This makes sense. So in all cases the PMD should assume that it
> >>>>> should stop as soon as a BFINAL is observed.
> >>>>>
> >>>>> A question. What happens ins stateful vs stateless modes when
> >>>>> decompressing an op that encompasses multiple BFINALs. I assume the
> >>>>> caller in that case will use the consumed=x bytes to find out how far in
> >>>>> to the input is the end of the first stream and start from the next
> >>>>> byte. Is this correct?
> >>>> [Shally]  As per my understanding, each op can be tied up to only one stream as we have only one
> >> stream pointer per op and one
> >>> stream can have only one BFINAL (as stream is one complete compressed data) but looks like you're
> >> suggesting a case where one op
> >>> can carry multiple independent streams? and thus multiple BFINAL?! , such as, below here is op
> >> pointing to more than one streams
> >>>>             --------------------------------------------
> >>>> op --> |stream1|stream2| |stream3|
> >>>>            --------------------------------------------
> >>>>
> >>>> Could you confirm if I understand your question correct?
> >>> [Ahmed] Correct. We found that in some storage applications the user
> >>> does not know where exactly the BFINAL is. They rely on zlib software
> >>> today. zlib.net software halts at the first BFINAL. Users put multiple
> >>> streams in one op and rely on zlib to  stop and inform them of the end
> >>> location of the first stream.
> >> [Shally] Then this is practically case possible on decompressor and decompressor doesn't regard flush
> >> flag. So in that case, I expect PMD to internally reset themselves (say in case of zlib going through
> cycle
> >> of deflateEnd and deflateInit or deflateReset) and return with status = SUCCESS with updated
> produced
> >> and consumed. Now in such case, if previous stream also has some footer followed by start of next
> >> stream, then I am not sure how PMD / lib can support that case. Have you had practically run of such
> >> use-case on zlib? If yes, how then such application handle it in your experience?
> >> I can imagine for such input zlib would return with Z_FLUSH_END after 1st BFINAL is processed to the
> >> user. Then application doing deflateReset() or Init-End() cycle before starting with next. But if it starts
> >> with input that doesn't have valid zlib header, then likely it will throw an error.
> >>
> > [Fiona] The consumed and produced tell the Application hw much data was processed up to
> > the end of the first deflate block encountered with a bfinal set.
> > If there is data, e.g. footer after the block with bfinal, then I think it must be the responsibility of
> > the application to know this, the PMD can't have any responsibility for this.
> > The next op sent to the PMD must start with a valid deflate block.
> [Ahmed] Agreed. This is exactly what I expected. In our case we support
> gzip and zlib header/footer processing, but that does not fundamentally
> change the setup. The user may have other meta data after the footer
> which the PMD is not responsible for. The PMD should stop processing
> depending on the mode. In raw DEFLATE, it should stop immediately. In
> other modes it should stop after the footer. We also have a mode in our
> PMD to simply continue decompression. In that case there cannot be
> header/footer between streams in raw DEFLATE. That mode can be enabled
> perhaps at the session level in the future with a session parameter at
> setup time. We call it "member continue". In this mode the PMD plows
> through as much of the op as possible. If it hits incorrectly setup data
> then it returns what it did decompress successfully and the error code
> in decompressing the data afterwards.
[Fiona] Yes, these would be interesting capabilities which could be 
added to the API in future releases.

> >
> >
> >>>> Thanks
> >>>> Shally
> >>>>
> >