[dpdk-dev] [RFC PATCH 0/2] using scapy to generate packet templates

[Xueming(Steven) Li]

Beside generating packets, there are more things can be done here with scapy/python, might be useful for developing:

1. generate limited packet and send: 
	tx {pattern} {port} {count}
2. parse and dump packet from binary to Scapy format.
	rx {pattern} {port}
3. loopback send and receive, compare with same packet:
	expect {pattern} {port} {queue}
4. send and expect no packet:
	unexpect {pattern} {port} 

When developing tunnel protocol, I found it better to design a bundle of test cases and run it easily anytime, especially after any PMD bug fix, there are tons of case combination to test, a few examples:
1. setup different inner/outer RSS mode, testing different packet type, expecting different queue and rss hash result from mbuf header.
2. verify csum of out and inner packet.
3. verify FDIR of mbuf header

Embedded Python could help here:
1. Using unit test framework to manage, run and report cases
2. export mbuf and limited dpdk api to python, benefit from python syntax sugar:
	dpdk.tx("mbuf(nsegs=3, pktlen=8000)/Ether()/IP(chksum=0xbad)", count=1000, port=1, verbose=1)
	dpdk.expect(pkt1, pkt2, sport=1, dport=2)
	

A big disadvantage is that compiler can't find out disconnection between C api and exposed python api(scripts), have to run unit test to know it.

> -----Original Message-----
> From: Van Haaren, Harry [mailto:harry.van.haaren at intel.com]
> Sent: Thursday, October 19, 2017 11:22 PM
> To: Xueming(Steven) Li <xuemingl at mellanox.com>; Wu, Jingjing
> <jingjing.wu at intel.com>
> Cc: dev at dpdk.org
> Subject: RE: [dpdk-dev] [RFC PATCH 0/2] using scapy to generate packet
> templates
> 
> > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Xueming Li
> > Sent: Thursday, October 19, 2017 3:07 PM
> > To: Wu, Jingjing <jingjing.wu at intel.com>
> > Cc: Xueming Li <xuemingl at mellanox.com>; dev at dpdk.org
> > Subject: [dpdk-dev] [RFC PATCH 0/2] using scapy to generate packet
> > templates
> 
> Hi,
> 
> First off - cool to make DPDK more dynamic with Scapy, and I see value in
> "somehow" enabling DPDK with Scapy.
> 
> > The target is to make testpmd flexible to generate all kinds of packet
> > in just one line. Examples:
> >
> > # send UDP packet to port 0 queue 0
> > testpmd> scapy 0 Ether()/IP()/UDP()/"hello_world"
> >
> > # send 4 L3 VXLAN packets with inner UDP dport 1-4 to port 1 queue
> > 1-4, one packet per queue
> > testpmd> scapy 1
> Ether()/IP()/UDP()/VXLAN()/IPv6()/UDP(dport=(1,4))/"cool"
> >
> > # Random
> > testpmd> scapy 2
> > Ether(src=RandMAC())/IP(dst=RandIP())/TCP(sport=RandShort(),dport=0x12
> > 34)
> >
> > This patch introduce scapy - a python tool to generate static packet
> > templates.
> > Txonly forwarding mode is modified to support template if available.
> 
> 
> Testpmd is a good use-case, but perhaps we're limiting the scope of
> DPDK/Scapy if we only integrate with Testpmd. What if I wanted to easily
> generate some traffic from a sample app?
> 
> As a suggestion, what about building a library that provides Scapy
> functionality, allowing any application to easily generate mbufs using
> the rte_scapy* functions:
> 
> /* demo usage */
> rte_scapy *scapy = rte_scapy_init();
> rte_mbuf *m = rte_scapy_mbuf_from_str(scapy, "Ether()/IP()/UDP()");
> 
> 
> Testpmd could then use the rte_scapy library, passing in any parameters
> after   testpmd> scapy   command as a string?
> 
> Had you already considered this library type design?
> 

Agree, good suggestion.

> 
> > Two new configuration:
> > CONFIG_RTE_TEST_PMD_SCAPY=y
> > CONFIG_RTE_PYTHON=python2.7
> >
> > Scapy quick demo:
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> > secdev.org%2Fprojects%2Fscapy%2Fdemo.html&data=02%7C01%7Cxuemingl%40me
> > llanox.com%7Cd6c3f991ab8247d35fca08d5170530ad%7Ca652971c7d2e4d9ba6a4d1
> > 49256f461b%7C0%7C0%7C636440233429123890&sdata=QLZinBnpnaJDlalLypdj8KW1
> > ZWi0yja3EZvD7iT%2F9AA%3D&reserved=0
> >
> > There are 2 other potential solution to support more fancy sending in
> > the
> > future:
> > 1. Invoke scapy and expose mbuf hugepage pointers to python
> >    This should allow more flexible and dynamic processing 2. Scapy
> > uses wrapped DPDK port as regular port
> >    Make scapy capable to send high speed streams.
> 
> Not a strong opinion, but a gut feeling; I see more value in generating
> packets with Python and using them from C than giving Python full access
> to DPDK internals and hugepages.

Both direction attractive enough, considering unit test, have to exposing warping api, mbuf and ethdev apis more or less...

> 
> Regards, -Harry
> 
> 
> > Xueming Li (2):
> >   app/testpmd: add packet template
> >   app/testpmd: add scapy command as pkt template
> >
> >  app/test-pmd/Makefile  |  5 +++
> >  app/test-pmd/cmdline.c | 99
> > +++++++++++++++++++++++++++++++++++++++++++++++++-
> >  app/test-pmd/testpmd.c |  3 ++
> >  app/test-pmd/testpmd.h |  1 +
> >  app/test-pmd/txonly.c  | 42 +++++++++++++--------
> >  config/common_base     |  2 +
> >  6 files changed, 135 insertions(+), 17 deletions(-)
> >
> > --
> > 2.13.3