[dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi Paul,

I was able to read the timestamp value of the packets which are transmitted
using MoonGen.

I tried looking how I can synchronize the timestamp(read from the NIC) of
the transmitted packet  with the packets received on the controller side ?

I read about packet time referencing, epoch time however it doesn't seem to
do what I want.

 Is there a way at all which will synchronize the timestamp of those
packets ??

On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam <ajinkya.kadam at nyu.edu>
wrote:

> Thanks a lot for your time and inputs.
>
> I have added my comments inline.
>
> On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <emmericp at net.in.tum.de>
> wrote:
>
>> Hi,
>>
>> Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
>> I want packets to be generated using different IP address and each packet
>> should be timestamped at two places. (Please refer the topology attached)
>> Once when it is sent from the server (to switch port 1) *i.e t1* and
>> next time when it reaches the controller *i.e t2*. I am trying to
>> measure the time taken by switch to generate packet in message. Also the
>> reason why I want to work with different IP address is the switch doesn't
>> currently understand mac addresses. So I can program the flows in the
>> switch only using IP address information.
>>
>>
>> I believe that I can generate packets from different IP addresses (the
>> above script doesn't have that functionality yet)
>>
>>
>> example for varying IP addresses: https://github.com/
>> emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96
>>
>
> This is what I was looking for. Thanks so much.
>
>> but what my concern is, right now the timestamped logic I have written in
>> line 48-50
>> <https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>
>> returns a "nil" as timestamped value when I try to print it. The NIC that
>> is being used here is Intel X710. Can you please suggest what might be
>> possibly wrong here ?
>>
>>
>> You have to set the timestamp flag on the buffer that you want to
>> timestamp via  buf:enableTimestamps(). See my previous email for example
>> code.
>> Also, you only need to call queue:enableTimestamps() once before the main
>> loop.
>>
>>
> Ahh..Got the mistake. Thanks for the correction.
>
>> Next, another solution is to use ptp packets which are timestamped and
>> then encapsulate them with packets differing in source IP. However the PTP
>> packets timestamped at the server side
>> have nanosecond resolution like "26155" (which is nice, also check
>> wireshark output attached) however the packets i am capturing at controller
>> using libpcap are timestamped according to the unix timestamp which counts
>> the number of seconds passed from january 1st 1970. Is there a way in which
>> I can receive these packets with the same resolution as the one PTP packets
>> are timestamped ? If not is there any other way which I should look for?
>> (The controller NIC is X520 Intel NIC)
>>
>>
>> I'd advise against measuring latencies by taking timestamps on different
>> devices, that's very complicated to get right (and typically involves a GPS
>> receiver for time synchronization).
>> (However, some fancy stuff with PTP might work fine, depending on your
>> requirements on accuracy and precision).
>> Also, using libpcap will lower your accuracy and precision by several
>> microseconds if not backed by hardware timestamping (libpcap actually
>> supports this for NICs that expose this in the driver, the challenge will
>> just be to sync the clocks of the NICs, but I'm not an expert on this).
>> For the granularity in the pcap format: You can try using the pcapng
>> format which supports storing timestamps with nanosecond granularity.
>>
>> If I were running such a test, I would use a different test setup:
>> * only one server attached to the switch
>>
>
> Agreed. I am sorry the topology that I have shown actually has only one
> server. So the controller, Moongen all are running on the same machine. I
> drew it that way to make things more clearer. Sorry about that. The purpose
> of keeping the same machine is to keep the synchronization intact as you
> have said. As the machine is the same can I now just use libpcap feature to
> read timestamps ? I read the X710 datasheet
> <http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf>.
> It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and
> "PRTTSYN_RXTIME" for received timestamps.  Do I need to read these
> registers using libpcap ?
>
>
>> * receive the controller traffic via an interface bound to DPDK, do
>> whatever you need to do there, then bridge it to the host (e.g., KNI or a
>> separate cable)
>>
>
> I believe I can bind the DPDK port to OVS bridge and then run the
> controller there. I will try doing this tonight.
>
>
>>
>>
>
>>
> This is more complicated, but avoids a lot of pitfalls and provides a nice
>> centralized control over the whole setup from your test application.
>> You can use a NIC that is capable of timestamping arbitrary RX packets
>> for the "bridge" port (e.g., 82580).
>>
>>
> I have a 82571 NIC. I will look into it if this can support RX packet
> timestamping.
>
>
>> Paul
>>
>>
>>
>> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp at net.in.tum.de>
>> wrote:
>>
>>> Hi,
>>>
>>> most NICs don't support timestamping TCP packets.
>>> It works for TX on some NICs, but RX is more difficult: of the Intel
>>> NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
>>>
>>> For RX:
>>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still
>>> works since the driver refactoring in MoonGen.
>>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's
>>> datasheet is inconsistent about the registers here.
>>>
>>> For TX (which seems to be your use case):
>>> It might work depending on your HW, you can test it:
>>>
>>
>>
>>> 1. call buf:enableTimestamps() on the buf you are interested in
>>> 2. send the packet
>>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>>>
>>> Note that the timestamp is kept in a register on the NIC. It stores only
>>> one TX timestamp at a time, irregardless of the number of queues etc.
>>> You have to read this register via queue:getTimestamp() before another
>>> packet can be timestamped.
>>>
>>> Our default measureLatency() function might be helpful:
>>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>>>
>>>
>>> Paul
>>>
>>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
>>>
>>> Hi Paul,
>>>
>>> If I am not wrong this [1] script enables only timestamps for PTP or UDP
>>> packets. Is this similar functionality available for TCP packets ?
>>>
>>> I am generating multiple TCP flows and I just want to time-stamp first
>>> packet of each flow. Is this possible using the NICs hardware time-stamping
>>> capability ?
>>>
>>>
>>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64
>>> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>>
>>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp at net.in.tum.de>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> the examples in "timestamping-tests.lua" are only meant as a
>>>> demonstration of the different timestamping capabilities (and/or as a
>>>> starting point for a custom script).
>>>>
>>>> In your case, you could use a device counter to print the whole
>>>> throughput of the device. You can use the default stats task to do that by
>>>> adding the following call in the master task:
>>>>
>>>>         stats.startStatsTask({rxDev, txDev})
>>>>
>>>> I'll also add the call to the example script in the repository later
>>>> today as having this is probably a good idea :)
>>>>
>>>> Paul
>>>>
>>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang at usi.ch>:
>>>> >
>>>> > Hello Emmerich,
>>>> >
>>>> > MoonGen is really helpful in measuring performance of network devices.
>>>> > I wonder if we could get some information about packet loss
>>>> > while running timestamps-software.lua?
>>>> >
>>>> > Thanks,
>>>> > Tu
>>>> >
>>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp at net.in.tum.de
>>>> <mailto:emmericp at net.in.tum.de>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > Ajinkya D Kadam:
>>>> > I was reading through your paper and I think this tool will be much
>>>> more
>>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>>>> > Is this [1] the right code to look at if i want to see how you have
>>>> > achieved hardware based time stamping ?
>>>> >
>>>> > Yes, run this example script with two directly connected ports for a
>>>> simple demo and test of your hardware's capabilities. It will work with
>>>> both of your NICs.
>>>> >
>>>> > In addition, I want to confirm my understanding of why MoonGen is
>>>> better
>>>> > than PktGen in time stamping context.
>>>> > PktGen reads the value of rdtsc which it then appends to packet, this
>>>> > adds more delay and hence the precision is bad.
>>>> >
>>>> > Software timestamping by writing the TSC to the packet is also
>>>> supported (but the API is less nice, see issue #153):
>>>> >
>>>> > See examples/timestamping-tests/timestamps-software.lua for an
>>>> example.
>>>> >
>>>> > The main problem is that there is unpredictable jitter from the NIC
>>>> and PCIe transfer and other random errors. Especially if you are running
>>>> this at higher packet rates.
>>>> > This leads to the 200-300ns random error that I've previously
>>>> mentioned.
>>>> >
>>>> >
>>>> > In case of MoonGen how does this work ? I am not sure. Could you
>>>> please
>>>> > elaborate ?
>>>> >
>>>> > MoonGen enables the hardware timestamping feature of the NIC and uses
>>>> it. The NIC will store the timestamp in a register which needs to be read
>>>> before another packet can be timestamped, this limits the throughput of
>>>> timestamped packets. However, I've found that you rarely need to timestamp
>>>> *all* packets in a packet generator. You'll have to use software
>>>> timestamping if you really need that.
>>>> >
>>>> >
>>>> > Paul
>>>> >
>>>> >
>>>> > Thanks,
>>>> > Ajinkya
>>>> >
>>>> >
>>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407
>>>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>>> >
>>>> > ᐧ
>>>> >
>>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <
>>>> emmericp at net.in.tum.de<mailto:emmericp at net.in.tum.de>
>>>> > <mailto:emmericp at net.in.tum.de>> wrote:
>>>> >
>>>> >   Hi,
>>>> >
>>>> >
>>>> >   Ajinkya D Kadam:
>>>> >
>>>> >       If yes I would like to modify the pktgen code so that each
>>>> >       transmitting and
>>>> >       received packet is timestamped.  Right now I am exploring the
>>>> >       example
>>>> >       applications  like rxtx_callbacks which timestamps packets in
>>>> >       DPDK, Is this
>>>> >       the right direction to go ?
>>>> >
>>>> >
>>>> >   Check out my packet generator MoonGen
>>>> >   https://github.com/emmericp/MoonGen
>>>> >   <https://github.com/emmericp/MoonGen>
>>>> >
>>>> >   It uses the hardware timestamping features (PTP) to do latency
>>>> >   measurements in the nanosecond-range.
>>>> >
>>>> >   However, if you will run into hardware limitations if you want to
>>>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>>>> >   i310, X550) but I don't know a NIC that supports this on TX.
>>>> >
>>>> >   As for the precision that is achievable: ~10ns (depending on the
>>>> >   NIC) with hardware support. Software timestamping will typically
>>>> >   result in a standard deviation of 200-300ns under load and there
>>>> >   will be huge outliers.
>>>> >
>>>> >
>>>> >    Paul
>>>>
>>>>
>>>
>>> Chair of Network Architectures and Services
>>> Department of Informatics
>>> TU München
>>> Boltzmannstr. 3
>>> 85748 Garching bei München, Germany
>>>
>>>
>>>
>>>
>> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>
>>
>>
>> Chair of Network Architectures and Services
>> Department of Informatics
>> TU München
>> Boltzmannstr. 3
>> 85748 Garching bei München, Germany
>>
>>
>>
>>
>