[dpdk-dev] Why packet replication is more efficient when done using memcpy( ) as compared to rte_mbuf_refcnt_update() function?

[Shailja Pandey]

Hello,

I am doing packet replication and I need to change the ethernet and IP 
header field for each replicated packet. I did it in two different ways:

 1. Share payload from the original packet using rte_mbuf_refcnt_update
    and allocate new mbuf for L2-L4 headers.
 2. memcpy() payload from the original packet to newly created mbuf and
    prepend L2-L4 headers to the mbuf.

I performed experiments with varying replication factor as well as 
varying packet size and found that memcpy() is performing way better 
than using rte_mbuf_refcnt_update(). But I am not sure why it is 
happening and what is making rte_mbuf_refcnt_update() even worse than 
memcpy().

Here is the sample code for both implementations:

*1. Using rte_mbuf_refcnt_update:*
**struct rte_mbuf *pkt = original packet;**
**
******rte_pktmbuf_adj(pkt, (uint16_t)sizeof(struct 
ether_hdr)+sizeof(struct ipv4_hdr));
         rte_pktmbuf_refcnt_update(pkt, replication_factor);
         for(int i = 0; i < replication_factor; i++) {
               struct rte_mbuf *hdr;
               if (unlikely ((hdr = rte_pktmbuf_alloc(header_pool)) == 
NULL)) {
                 printf("Failed while cloning $$$\n");
                 return NULL;
            }
            hdr->next = pkt;
            hdr->pkt_len = (uint16_t)(hdr->data_len + pkt->pkt_len);
            hdr->nb_segs = (uint8_t)(pkt->nb_segs + 1);
            //*Update more metadate fields*
*
*
**rte_pktmbuf_prepend(hdr, (uint16_t)sizeof(struct ether_hdr));
             //*modify L2 fields*

             rte_pktmbuf_prepend(hdr, (uint16_t)sizeof(struct ipv4_hdr));
             //Modify L3 fields
             .
             .
             .
         }
*
*
*
*
*2. Using memcpy():*
**struct rte_mbuf *pkt = original packet
**struct rte_mbuf *hdr;**
         if (unlikely ((hdr = rte_pktmbuf_alloc(header_pool)) == NULL)) {
                 printf("Failed while cloning $$$\n");
                 return NULL;
         }

         /* prepend new header */
         char *eth_hdr = (char *)rte_pktmbuf_prepend(hdr, pkt->pkt_len);
         if(eth_hdr == NULL) {
                 printf("panic\n");
         }
         char *b = rte_pktmbuf_mtod((struct rte_mbuf*)pkt, char *);
         memcpy(eth_hdr, b, pkt->pkt_len);
         Change L2-L4 header fields in new packet

The throughput becomes roughly half when the packet size is increased 
from 64 bytes to 128 bytes and replication is done using 
*rte_mbuf_refcnt_update(). *The throughput remains more or less same 
when packet size increases and replication is done using *memcpy()*.

Any help would be appreciated.
**

--

Thanks,
Shailja