[dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

most NICs don't support timestamping TCP packets.
It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.

For RX:
I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen.
The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here.

For TX (which seems to be your use case):
It might work depending on your HW, you can test it:

1. call buf:enableTimestamps() on the buf you are interested in
2. send the packet
3. get the timestamp with queue:getTimestamp(maxWaitMicros)

Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc.
You have to read this register via queue:getTimestamp() before another packet can be timestamped.

Our default measureLatency() function might be helpful:
https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 <https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64>

  Paul

> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
> 
> Hi Paul, 
> 
> If I am not wrong this [1] script enables only timestamps for PTP or UDP packets. Is this similar functionality available for TCP packets ? 
> 
> I am generating multiple TCP flows and I just want to time-stamp first packet of each flow. Is this possible using the NICs hardware time-stamping capability ?
> 
> 
> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
> 
> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de>> wrote:
> Hi,
> 
> the examples in "timestamping-tests.lua" are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).
> 
> In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:
> 
>         stats.startStatsTask({rxDev, txDev})
> 
> I'll also add the call to the example script in the repository later today as having this is probably a good idea :)
> 
> Paul
> 
> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang at usi.ch <mailto:huynh.tu.dang at usi.ch>>:
> >
> > Hello Emmerich,
> >
> > MoonGen is really helpful in measuring performance of network devices.
> > I wonder if we could get some information about packet loss
> > while running timestamps-software.lua?
> >
> > Thanks,
> > Tu
> >
> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de><mailto:emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de>>> wrote:
> >
> > Hi,
> >
> > Ajinkya D Kadam:
> > I was reading through your paper and I think this tool will be much more
> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
> > Is this [1] the right code to look at if i want to see how you have
> > achieved hardware based time stamping ?
> >
> > Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
> >
> > In addition, I want to confirm my understanding of why MoonGen is better
> > than PktGen in time stamping context.
> > PktGen reads the value of rdtsc which it then appends to packet, this
> > adds more delay and hence the precision is bad.
> >
> > Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
> >
> > See examples/timestamping-tests/timestamps-software.lua for an example.
> >
> > The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
> > This leads to the 200-300ns random error that I've previously mentioned.
> >
> >
> > In case of MoonGen how does this work ? I am not sure. Could you please
> > elaborate ?
> >
> > MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
> >
> >
> > Paul
> >
> >
> > Thanks,
> > Ajinkya
> >
> >
> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
> >
> > ᐧ
> >
> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de><mailto:emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de>>
> > <mailto:emmericp at net.in.tum.de <mailto:emmericp at net.in.tum.de>>> wrote:
> >
> >   Hi,
> >
> >
> >   Ajinkya D Kadam:
> >
> >       If yes I would like to modify the pktgen code so that each
> >       transmitting and
> >       received packet is timestamped.  Right now I am exploring the
> >       example
> >       applications  like rxtx_callbacks which timestamps packets in
> >       DPDK, Is this
> >       the right direction to go ?
> >
> >
> >   Check out my packet generator MoonGen
> >   https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>
> >   <https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>>
> >
> >   It uses the hardware timestamping features (PTP) to do latency
> >   measurements in the nanosecond-range.
> >
> >   However, if you will run into hardware limitations if you want to
> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
> >   i310, X550) but I don't know a NIC that supports this on TX.
> >
> >   As for the precision that is achievable: ~10ns (depending on the
> >   NIC) with hardware support. Software timestamping will typically
> >   result in a standard deviation of 200-300ns under load and there
> >   will be huge outliers.
> >
> >
> >    Paul
> 
> 

Chair of Network Architectures and Services
Department of Informatics
TU München
Boltzmannstr. 3
85748 Garching bei München, Germany