[dpdk-dev] rte_mbuf.next in 2nd cacheline

[Bruce Richardson]

On Mon, Jun 15, 2015 at 03:20:22PM +0200, Olivier MATZ wrote:
> Hi Damjan,
> 
> On 06/10/2015 11:47 PM, Damjan Marion (damarion) wrote:
> > 
> > Hi,
> > 
> > We noticed 7% performance improvement by simply moving rte_mbuf.next field to the 1st cache line.
> > 
> > Currently, it falls under /* second cache line - fields only used in slow path or on TX */
> > but it is actually used at several places in rx fast path. (e.g.: i40e_rx_alloc_bufs() is setting that field to NULL).
> > 
> > Is there anything we can do here (stop using next field, or move it to 1st cache line)?
> 
> Agree, this is also something I noticed, see:
> http://dpdk.org/ml/archives/dev/2015-February/014400.html
> 
> I did not have the time to do performance testing, but it's something
> I'd like to do as soon as I can. I don't see any obvious reason not to
> do it.
> 
> It seems we currently just have enough room to do it (8 bytes are
> remaining in the first cache line when compiled in 64 bits).

This, to me, is the obvious reason not to do it! It prevents us from taking in
any other offload fields in the RX fast-path into the mbuf.

That being said, I can see why we might want to look to move it - but from the
work done in the ixgbe driver, I'd be hopeful we can get as much performance with
it on the second cache line for most cases, through judicious use of prefetching,
or otherwise.

It took a lot of work and investigation to get free space in the mbuf - especially
in cache line 0, and I'd like to avoid just filling the cache line up again as
long as we possibly can!

/Bruce

> 
> 
> Regards,
> Olivier
> 
> 
> > 
> > Thanks,
> > 
> > Damjan
> > 
> > 
> > 
> >