[dpdk-dev] [PATCH v2 3/4] examples: example showing use of callbacks.

[Declan Doherty]

On 16/02/15 17:34, Thomas Monjalon wrote:
> 2015-02-16 15:16, Bruce Richardson:
>> On Mon, Feb 16, 2015 at 03:33:40PM +0100, Olivier MATZ wrote:
>>> Hi John,
>>>
>>> On 02/13/2015 04:39 PM, John McNamara wrote:
>>>> From: Richardson, Bruce <bruce.richardson at intel.com>
>>>>
>>>> Example showing how callbacks can be used to insert a timestamp
>>>> into each packet on RX. On TX the timestamp is used to calculate
>>>> the packet latency through the app, in cycles.
>>>>
>>>> Signed-off-by: Bruce Richardson <bruce.richardson at intel.com>
>>>
>>>
>>> I'm looking at the example and I don't understand what is the advantage
>>> of having callbacks in ethdev layer, knowing that the application can
>>> do the same job by a standard function call.
>>>
>>> What is the advantage of having callbacks compared to:
>>>
>>>
>>> for (port = 0; port < nb_ports; port++) {
>>> 	struct rte_mbuf *bufs[BURST_SIZE];
>>> 	const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
>>> 			bufs, BURST_SIZE);
>>> 	if (unlikely(nb_rx == 0))
>>> 		continue;
>>> 	add_timestamp(bufs, nb_rx);
>>>
>>> 	const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
>>> 			bufs, nb_rx);
>>> 	calc_latency(bufs, nb_tx);
>>>
>>> 	if (unlikely(nb_tx < nb_rx)) {
>>> 		uint16_t buf;
>>> 		for (buf = nb_tx; buf < nb_rx; buf++)
>>> 			rte_pktmbuf_free(bufs[buf]);
>>> 	}
>>> }
>>>
>>>
>>> To me, doing like the code above has several advantages:
>>>
>>> - code is more readable: the callback is explicitly invoked, so there is
>>>    no risk to forget it
>>> - code is faster: the functions calls can be inlined by the compiler
>>> - easier to handle error cases in the callback function as the error
>>>    code is accessible to the application
>>> - there is no need to add code in ethdev api to do this
>>> - if the application does not want to use callbacks (I suppose most
>>>    applications), it won't have any performance impact
>>>
>>> Regards,
>>> Olivier
>>
>> In this specific instance, given that the application does little else, there
>> is no real advantage to using the callbacks - it's just to have a simple example
>> of how they can be used.
>>
>> Where callbacks are really designed to be useful, is for extending or augmenting
>> hardware capabilities. Taking the example of sequence numbers - to use the most
>> trivial example - an application could be written to take advantage of sequence
>> numbers written to packets by the hardware which received them. However, if such
>> an application was to be used with a NIC which does not provide sequence numbering
>> capability, for example, anything using ixgbe driver, the application writer has
>> two choices - either modify his application code to check each packet for
>> a sequence number in the data path, and add it there post-rx, or alternatively,
>> to check the NIC capabilities at initialization time, and add a callback there
>> at initialization, if the hardware does not support it. In the latter case,
>> the main packet processing body of the application can be written as though
>> hardware always has sequence numbering capability, safe in the knowledge that
>> any hardware not supporting it will be back-filled by a software fallback at
>> initialization-time.
>>
>> By the same token, we could also look to extend hardware capabilities. For
>> different filtering or hashing capabilities, there can be limits in hardware
>> which are far less than what we need to use in software. Again, callbacks will
>> allow the data path to be written in a way that is oblivious to the underlying
>> hardware limits, because software will transparently fill in the gaps.
>>
>> Hope this makes the use case clear.
>
> After thinking more about these callbacks, I realize these callbacks won't
> help, as Olivier said.
>
> With callback,
> 1/ application checks device capability
> 2/ application provides hardware emulation as DPDK callback
> 3/ application forgets previous steps
> 4/ application calls DPDK Rx
> 5/ DPDK calls callback (without calling optimization)
>
> Without callback,
> 1/ application checks device capability
> 2/ application provides hardware emulation as internal function
> 3/ application set an internal device-flag to enable this function
> 4/ application calls DPDK Rx
> 5/ application calls the hardware emulation if flag is set
>
> So the only difference is to keep persistent the device information in
> the application instead of storing it as a function pointer in the
> DPDK struct.
> You can also be faster with this approach: at initialization time,
> you can check that your NIC supports the feature and use a specific
> mainloop that adds or not the sequence number without any runtime
> test.
>
> A callback could be justified for asynchronous events, or when
> doing specific processing in the middle of the driver, for instance
> when freeing a mbuf. But in this case it's exactly similar to do
> the processing in the application after Rx (or before Tx).
>

I believe that the introduction of callbacks to the ethdev layer will be 
required for live migration.

For example, in the scenario were we have two ports bonded together in 
active backup mode, the primary slave being a hw port and the other 
slave a virtio port, in normal operation it would be desirable to 
leverage the available hw offload capabilities for maximum performance, 
but for these two devices to be bonded together then it is required that 
the both slave devices support the same set of offload features. In the 
occurrence of a planned or unplanned fail over the backup slave must 
provided the same offloads as the primary device, currently the offloads 
supported are the lowest common denominator of offload features of all 
slave devices but obviously this isn't desirable.


I think that we could extend the bonding library API to take a set of 
desired offloads as input parameters, then during the addition of slaves 
we would interrogate the supported hw offloads available, enable the 
desired ones and then register callbacks to implement the offloads which 
the slave device does not support in hw. This would negate the user 
application needing to have any knowledge of the under lying slave 
offload configuration, and it would be guaranteed than the offloads 
requested are happening irrespective of which slave is in use and allow 
migration of vm transparently to what is happening in the ethdev layer

Declan