Prerequisites for floating VEB testing
 =============================

1. Get the pci device id of DUT, for example::

./dpdk_nic_bind.py --st

0000:82:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=enp130s0f0 drv=i40e unused=

2. Host PF in DPDK driver. Create 2VFs from 1 PF with dpdk driver. 

./dpdk_nic_bind.py -b igb_uio 82:00.0 
 echo 2 >/sys/bus/pci/devices/0000\:82\:00.0/max_vfs
 ./dpdk_nic_bind.py --st
 0000:82:02.0 'XL710/X710 Virtual Function' unused=i40evf,igb_uio
 0000:82:02.1 'XL710/X710 Virtual Function' unused=i40evf,igb_uio

3. Detach VFs from the host, bind them to pci-stub driver::

modprobe pci-stub

using `lspci -nn|grep -i ethernet` got VF device id, for example "8086 154c",

echo "8086 154c" > /sys/bus/pci/drivers/pci-stub/new_id
 echo "0000:82:02.0" > /sys/bus/pci/drivers/i40evf/unbind
 echo "0000:82:02.0" > /sys/bus/pci/drivers/pci-stub/bind

echo "8086 154c" > /sys/bus/pci/drivers/pci-stub/new_id
 echo "0000:82:02.1" > /sys/bus/pci/drivers/i40evf/unbind
 echo "0000:82:02.1" > /sys/bus/pci/drivers/pci-stub/bind

4. Lauch the VM with the VF PCI passthrough. 

taskset -c 18-19 qemu-system-x86_64 \
 -enable-kvm -m 2048 -smp cores=2,sockets=1 -cpu host -name dpdk1-vm0 \
 -device pci-assign,host=0000:82:02.0 \
 -drive file=/home/image/vm0.img \
 -netdev tap,id=ipvm0,ifname=tap1,script=/etc/qemu-ifup -device rtl8139,netdev=ipvm0,id=net1,mac=00:00:00:00:00:01 \
 -localtime -vnc :1 -daemonize

taskset -c 18-19 qemu-system-x86_64 \
 -enable-kvm -m 2048 -smp cores=2,sockets=1 -cpu host -name dpdk1-vm1 \
 -device pci-assign,host=0000:82:02.1 \
 -drive file=/home/image/vm1.img \
 -netdev tap,id=ipvm1,ifname=tap2,script=/etc/qemu-ifup -device rtl8139,netdev=ipvm1,id=net2,mac=00:00:00:00:00:02 \
 -localtime -vnc :2 -daemonize


Test Case1: Floating VEB Inter-VM VF-VF  
=======================================================

Summary: DPDK PF, then create 2VFs and 2VMs, assign one VF to one VM, say VF1 in VM1, VF2 in VM2, and make PF link down(the cable can be pluged out). VFs in VMs are running dpdk testpmd, VF1 send traffic, and set the packet's DEST MAC to VF2, check if VF2 can receive the packets. Check Inter-VM VF-VF MAC switch when PF is link down as well as up.

Launch PF testpmd: ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1 -- -i

1. Start VM1 with VF1, VM2 with VF2, see the prerequisite part. put the patch in the VM's dpdk

1.1  Reserve huge pages memory on host and VM(before using DPDK):

     echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 
     mkdir /mnt/huge   
     mount -t hugetlbfs nodev /mnt/huge  

1.2  VF bind to dpdk driver

     cd /root/dpdk/tools
     ./dpdk_nic_bind --st
     ./dpdk_nic_bind -b igb_uio ethernet   
   
2. In the host, run testpmd with floating parameters and make the link down::

     ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1 -- -i
    testpmd> port stop all
    testpmd> show port info all

3. In VM1, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>set fwd rxonly
    testpmd>show port info all     //get the vf1_mac_address
    testpmd>start
    testpmd>show port stats all
   
   In VM2, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,vf1_mac_address
    testpmd>set fwd txonly
    testpmd>start
    testpmd>show port stats all

4. check if VF1 can get all the packets. Check the packet content is no corrupted. 

5. Set "testpmd> port start all" and "testpmd> start" in step2, then run the step3-4 again. same result.


Test Case2: Floating VEB PF can't get traffic from VF    
================================================================

DPDK PF, then create 1VF, assign VF1 to VM1, send traffic from VF1 to PF, then check PF will not see any traffic. 

1. Start VM1 with VF1, see the prerequisite part.

2. In host, launch testpmd::

   ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1 -- -i
   testpmd> set fwd rxonly
   testpmd> port start all
   testpmd> start
   testpmd> show port stats all

3. In VM1, run testpmd::

   ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,pf_mac_addr
   testpmd>set fwd txonly
   testpmd>start
   testpmd>show port stats all

4. Check if PF can not get any packets, so VF1->PF is not working. 

5. Set "testpmd> port stop all" in step2, then run the test case again. Same result.



Test Case3: Floating VEB VF can't receive traffic from outside world   
======================================================

DPDK PF, then create 1VF, assign VF1 to VM1, send traffic from tester to VF1, in floating mode, check VF1 can't receive traffic from tester.

1. Start VM1 with VF1, see the prerequisite part.

2. In host, launch testpmd::

   ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1 -- -i 
   testpmd> set fwd mac
   testpmd> port start all
   testpmd> start
   testpmd> show port stats all
   

   In VM1, run testpmd:

   ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>show port info all    //get VF_mac_address
    testpmd>set fwd rxonly
    testpmd>start
    testpmd>show port stats all

   In tester, run scapy

   packet=Ether(dst="VF_mac_address")/IP()/UDP()/Raw('x'*20)
   sendp(packet,iface="enp132s0f0")
   

3. Check if VF1 can not get any packets, so tester->VF1 is not working. 
4. Set "testpmd> port stop all" in step2 in Host, then run the test case again. same result.


Test Case4: Floating VEB VF can not communicate with legacy VEB VF inter-VM
======================================================

Summary: DPDK PF, then create 3VFs and 3VMs, assign one VF to one VM, say VF1 in VM1, VF3 in VM3, floating VEB; VF2 in VM2, lagecy VEB. Make PF link down(port stop all), VFs in VMs are running dpdk testpmd.
1. VF1 send traffic, and set the packet's DEST MAC to VF2, check VF2 can not receive the packets. 
2. VF1 send traffic, and set the packet's DEST MAC to VF3, check VF3 can receive the packets. 
3. VF2 send traffic, and set the packet's DEST MAC to VF1, check VF1 can not receive the packets. 
Check Inter-VM VF-VF MAC switch when PF is link down as well as up.

Launch PF testpmd: ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1,floating_bitmap=0x5 -- -i

1. Start VM1 with VF1, VM2 with VF2, VM3 with VF3, see the prerequisite part.

2. In the host, run testpmd with floating parameters and make the link down::

    ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1,floating_bitmap=0x5 -- -i     //VF1 and VF3 in floating VEB, VF2 in legacy VEB
    testpmd> port stop all
    testpmd> show port info all

3. VF1 send traffic, and set the packet's DEST MAC to VF2, check VF2 can not receive the packets.

    In VM2, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>set fwd rxonly
    testpmd>show port info all     //get the vf2_mac_address
    testpmd>start
    testpmd>show port stats all
   
    In VM1, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,vf2_mac_address
    testpmd>set fwd txonly
    testpmd>start
    testpmd>show port stats all

    Check VF2 can not get any packets, so VF1->VF2 is not working.

4. VF1 send traffic, and set the packet's DEST MAC to VF3, check VF3 can receive the packets.

    In VM3, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>set fwd rxonly
    testpmd>show port info all     //get the vf3_mac_address
    testpmd>start
    testpmd>show port stats all

    In VM1, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,vf3_mac_address
    testpmd>set fwd txonly
    testpmd>start
    testpmd>show port stats all

    Check VF3 can get all the packets. Check the packet content is no corrupted., so VF1->VF2 is working.

5. VF2 send traffic, and set the packet's DEST MAC to VF1, check VF1 can not receive the packets. 

    In VM1, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>set fwd rxonly
    testpmd>show port info all     //get the vf1_mac_address
    testpmd>start
    testpmd>show port stats all

    In VM2, run testpmd::

    ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,vf1_mac_address
    testpmd>set fwd txonly
    testpmd>start
    testpmd>show port stats all

    Check VF1 can not get any packets, so VF2->VF1 is not working.

6. Set "testpmd> port start all" and "testpmd> start" in step2, then run the step3-5 again. same result.


Test Case5: PF can't get traffic from Floating VEB VF, but PF can get traffic from legacy VEB VF.
================================================================

DPDK PF, then create 2VF, assign VF1 to VM1, VF2 to VM2, set VF1 in floating VEB, VF2 in legacy VEB. 
1. Send traffic from VF1 to PF, then check PF will not see any traffic; 
2. Send traffic from VF2 to PF, then check PF will receive all the packets.

Note:The direction can't be reversed. because you must run testpmd in PF first, then you can run testpmd in vf. so when you run testpmd in pf, you don't know the vf's mac_address, you can't set the --eth-peer. if there is not "-eth-peer", vf can't receive the the packet sent by "txonly". even if there is not floatingVEB, vf still can't receive the packet sent by "txonly" without the "-eth-peer".

1. Start VM1 with VF1, VM2 with VF2, see the prerequisite part.

2. In host, launch testpmd::

   ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1,floating_bitmap=0x1 -- -i   //VF1 in floating VEB, VF2 in legacy VEB
   testpmd> set fwd rxonly
   testpmd> port start all
   testpmd> start
   testpmd> show port stats all

3. In VM1, run testpmd:

   ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,pf_mac_addr
   testpmd>set fwd txonly
   testpmd>start
   testpmd>show port stats all

   Check PF can not get any packets, so VF1->PF is not working. (I have verified that if it's not floating VEB, PF can get packets, there is some RX-missed)

4. In VM2, run testpmd:

   ./testpmd -c 0x3 -n 4 -- -i --eth-peer=0,pf_mac_addr
   testpmd>set fwd txonly
   testpmd>start
   testpmd>show port stats all

   Check PF can get all the packets, so VF2->PF is working.

5. Set "testpmd> port stop all" in step2 in Host, then run the test case again. same result.


Test Case6: Floating VEB VF can't receive traffic from outside world, while legacy VEB VF can receive traffic from outside world.
======================================================

DPDK PF, then create 2VF, assign VF1 to VM1, assign VF2 to VM2, set VF1 in floating VEB, set VF2 in legacy VEB
1. send traffic from tester to VF1, check VF1 can't receive traffic from tester.
2. send traffic from tester to VF2, check VF2 can receive all the traffic from tester.

1. Start VM1 with VF1, VM2 with VF2, see the prerequisite part.

2. In host, launch testpmd::

   ./testpmd -c 0x3 -n 4 -w 82:00.0,enable_floating=1,floating_bitmap=0x1 -- -i   //VF1 in floating VEB, VF2 in legacy VEB 
   testpmd> set fwd mac
   testpmd> port start all
   testpmd> start
   testpmd> show port stats all
   

3. In VM1, run testpmd:

   ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>show port info all    //get VF1_mac_address
    testpmd>set fwd rxonly
    testpmd>start
    testpmd>show port stats all

   In tester, run scapy

   packet=Ether(dst="VF1_mac_address")/IP()/UDP()/Raw('x'*20)
   sendp(packet,iface="enp132s0f0")
   
   Check VF1 can not get any packets, so tester->VF1 is not working. PF can receive packets and forwording it.

4. In VM2, run testpmd:

   ./testpmd -c 0x3 -n 4 -- -i 
    testpmd>show port info all    //get VF2_mac_address
    testpmd>set fwd rxonly
    testpmd>start
    testpmd>show port stats all

   In tester, run scapy

   packet=Ether(dst="VF2_mac_address")/IP()/UDP()/Raw('x'*20)
   sendp(packet,iface="enp132s0f0")
   
   Check VF1 can get all the packets, so tester->VF2 is working.

5. Set "testpmd> port stop all" in step2 in Host, then run the test case again. VF1 and VF2 cannot receive any packets. (because PF link down, and PF can't receive any packets. so even if VF2 can't receive any packets.)