[dpdk-users] Beginners question: rte_eth_tx_burst, rte_mbuf access synchronization

[Philipp Beyer]

Hi Matt,

Thanks for your answers. This helps as I am still stabbing in the dark 
quite a lot.

I actually use 16, not one, distinct buffers to be sent in one burst. 
But still, your conclusion is correct: I mess with the refcount, adjust 
payload after calling rte_eth_tx_burst, and therefore get undefined 
behaviour.

Your answer pretty much sound like you understood my point, so it seems 
the solution I am looking for does not exist. Unfortunately, it is not 
really only one byte i am changing. This was just a simplification, its 
a few byte actually, but still a small portion of the payload. So your 
idea won't really work.

But I might have found another idea: What about preparing all buffers of 
a memory pool with the same payload? I should than get a pre-filled 
buffer from rte_pktmbuf_alloc, right? Let's say, I initialize a buffer 
for transmittion, the transmitting code free's this buffer, and I get 
the same buffer back from rte_pktmbuf_alloc. What do I have to 
re-initialize to have the same buffer again? Only the payload length? Is 
this approach feasible, based on documented/specified behaviour?


Philipp


Am 11.11.2016 um 14:45 schrieb Matt Laswell:
> Hi Philipp,
>
> I'm a little unclear what you mean with your comments about adjusting 
> the refcnt in your mbufs.  You are absolutely correct that 
> rte_eth_tx_burst doesn't synchronously transmit the packets.  Instead, 
> it puts them in a ring that is serviced by the poll mode driver.  
> Eventually, they are handed off to the NIC, which copies them into its 
> buffer and ultimately sends them on the wire.
>
> The architecture you've described won't work for the reasons you've 
> surmised - when you hand a pointer to the pack to the device driver, 
> you are giving it control of the memory pointed to.  If you continue 
> to modify its contents at that point, the results will be 
> unpredictable.  Also, it sounds as though you might really just have 
> 16 pointers to a single packet, with a reference count of 16.  Since 
> you don't actually have 16 buffers, if you modify the contents of any 
> one packet, you're modifying them all.
>
> Let me suggest that you might want to rethink your scheme. Rather than 
> trying to reverse engineer a way to either make the PMD behave 
> synchronously or to give you a callback, I would consider prebuilding 
> packet contents at init time, then allocating mbufs and copying the 
> contents in.  I suspect you've avoided an approach like this because 
> you'd like to not copy mostly the same data over and over when you 
> only want to modify one byte.
>
> An alternative approach would be to use indirect mbufs.  In essence, 
> each packet you want to send might be made up of three mbufs.  The 
> first is an indirect mbuf that points to one that contains the common 
> data at the start of your packets. The second contains the one byte 
> that you wish to change.  The third is an indirect mbuf that points to 
> the common data at the end of your packets.  I haven't used this 
> approach myself, but I suspect it would let you avoid copying so much 
> data.
>
> - Matt
>
> On Fri, Nov 11, 2016 at 3:49 AM, Philipp Beyer <pbeyer at voipfuture.com 
> <mailto:pbeyer at voipfuture.com>> wrote:
>
>     Hi!
>
>     I am just writing my first code using dpdk, a traffic generator,
>     for which I started with the l2fwd example.
>
>     Basically, I need to send the same packet over a single interface,
>     over an over again, with single bytes changed each time.
>     I use rte_eth_tx_burst to send 16 packets at once. As I want to
>     re-use the same buffers in a very simple way, I just increment the
>     refcnt
>     accordingly.
>
>     My current code prepares all 16 buffers, calls rte_eth_tx_burst
>     until all 16 packets are stored in the transmit ring, and starts
>     over again, adjusting the buffers to send the next 16 packets.
>
>     Currently I observe duplicate packets, although every packet
>     should be individual due to single byte adjustments.
>
>     My current problem is, as I guess, that rte_eth_tx_burst does not
>     synchnolously transmit the count of packets, which is returned to
>     the caller, but just stores them in transmit queue. So, I am not
>     allowed to instantly re-use these buffers again.
>
>     My question is: How do I know when to re-use buffers passed to
>     rte_eth_tx_burst. Of course, I can check their refcnt member, and
>     this would be perfectly fine. Apparently, I should have at least
>     BURST_SIZE*2 buffers, passing BURST_SIZE buffers at once, so I can
>     manipulate one set of buffers while the other is transmitted. But
>     I am missing the idea of the best synchronization scheme here: How
>     should I wait on this refcnt to drop?
>
>     Some blind guessing:
>     If I take the documentation of rte_eth_tx_burst literally, I could
>     get the idea that refcounts of buffers are only decreased (buffers
>     are 'freed'), while rte_eth_tx_burst is executed, but one function
>     call might free buffers used by previous function calls. If this
>     is correct, I still do not see a complete synchronization scheme.
>     There is still a chance that I end up without any buffers left,
>     which means I do not have a chance to call rte_eth_tx_burst again
>     to free buffers.
>
>     Thanks for any help,
>     Philipp
>
>

-- 

Philipp Beyer

Software Developer

**

** **

**

Voipfuture GmbH   Wendenstr. 4   20097 Hamburg   Germany

Phone +49 40 688 9001 69   Fax +49 40 688 9001 99 www.voipfuture.com 
<http://www.voipfuture.com/>

Managing Directors   Jan Bastian   Eyal Ullert

Commercial Court AG Hamburg   HRB 109896  VAT ID DE263738086