[dts] [PATCH] test_plans: add test plan of Run-time Queue number determination

[Peng, Yuan]

Hi Marvin,

I have modified the plan according to your comments.
And submitted it again.

Thanks.
Yuan.
-----Original Message-----
From: Liu, Yong 
Sent: Friday, March 9, 2018 5:21 PM
To: Peng, Yuan <yuan.peng at intel.com>; dts at dpdk.org
Cc: Peng, Yuan <yuan.peng at intel.com>
Subject: RE: [dts] [PATCH] test_plans: add test plan of Run-time Queue number determination

Yuan,
Sorry for ignored your patch. Some comments are inline.

Thanks,
Marvin

> -----Original Message-----
> From: dts [mailto:dts-bounces at dpdk.org] On Behalf Of Peng Yuan
> Sent: Friday, January 12, 2018 5:24 PM
> To: dts at dpdk.org
> Cc: Peng, Yuan <yuan.peng at intel.com>
> Subject: [dts] [PATCH] test_plans: add test plan of Run-time Queue 
> number determination
> 
> Signed-off-by: Peng Yuan <yuan.peng at intel.com>
> 
> diff --git a/test_plans/runtime_queue_number_test_plan.rst
> b/test_plans/runtime_queue_number_test_plan.rst
> new file mode 100644
> index 0000000..4ca1f89
> --- /dev/null
> +++ b/test_plans/runtime_queue_number_test_plan.rst
> @@ -0,0 +1,465 @@
> +.. Copyright (c) <2017>, Intel Corporation
> +      All rights reserved.
> +
> +   Redistribution and use in source and binary forms, with or without
> +   modification, are permitted provided that the following conditions
> +   are met:
> +
> +   - Redistributions of source code must retain the above copyright
> +     notice, this list of conditions and the following disclaimer.
> +
> +   - Redistributions in binary form must reproduce the above copyright
> +     notice, this list of conditions and the following disclaimer in
> +     the documentation and/or other materials provided with the
> +     distribution.
> +
> +   - Neither the name of Intel Corporation nor the names of its
> +     contributors may be used to endorse or promote products derived
> +     from this software without specific prior written permission.
> +
> +   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
> +   FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
> +   COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
> +   INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> +   (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
> +   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> +   HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> +   STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
> +   ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
> +   OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +=====================================================================
> +====
> ==
> +Set the VF max queue number when running the DPDK APP instead of
> compiling.
> +=====================================================================
> +====
> ==

This title is too complicated, I think "Dynamically Configure VF Queue Number" is more sensible. 

> +Description
> +===========
> +
> +   Now RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF is used to determine the max
> queue
> +   number per VF. It's not friendly to the users because it means the
> users
> +   must decide the max queue number when compiling. There's no chance to
> +   change it when deploying their APP. It's good to make the queue number
> +   to be configurable so the users can change it when launching the APP.
> +   This requirement is meaningless to ixgbe since the queue is fixed on
> +   ixgbe.
> +   Without this patch, the number of queues per i40e  VF is defined as 4
> +   by CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF=4 in config/common_base.
> +   It is fixed value determined in building time and can't be changed
> +   during run time.
> +   With this patch, the number of queues per i40e VF can be determinated
> +   during run time. For example, if the PCI address of an i40e PF is
> +   aaaa:bb.cc, with the EAL parameter -w aaaa:bb.cc,queue-num-per-vf=8,
> +   the number of queues per VF created from this PF is 8.
> +   Set the VF max queue number with the PF EAL parameter 
> + "queue-num-per-
> vf".

Developer's patch has been merged, not need to describe the change.
Just describe feature's overview is enough.

> +   the valid values includes 1,2,4,8,16; if the value after the
> +   "queue-num-per-vf" is invalid, it is set as 4 forcibly;
> +   if there is no "queue-num-per-vf" setting in EAL parameters,
> +   it is 4 by default as before.
> +
> +Prerequisites
> +=============
> +
> +1. Hardware:
> +   Fortville
> +
> +2. software:
> +   dpdk: http://dpdk.org/git/dpdk
> +   scapy: http://www.secdev.org/projects/scapy/
> +
> +3. bind the pf port to dpdk driver::

Please capitalized the first character, so it will look like one complete sentence.

> +
> +    ./usertools/dpdk-devbind.py -b igb_uio 05:00.0
> +
> +4. set up two vfs from the pf with DPDK driver::
> +
> +    echo 2 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
> +
> +   bind the two vfs to DPDK driver::
> +
> +    ./usertools/dpdk-devbind.py -b vfio-pci 05:02.0 05:02.1
> +
> +Test case 1: set valid VF max queue number 
> +==========================================
You can remove the case index in the title, as it will be automatically generated after build doc.

> +
> +1. try the valid values 1::
> +
Please capitalized the first character, so it will look like one complete sentence.

> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=1 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i

You can remove target here as it can be changed. 

> +
> +   testpmd can be started normally without any wrong or error.
> +
> +2. start VF testpmd with "--rxq=1 --txq=1", the number of rxq and txq is
> +   consistent with the configured VF max queue number::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i --rxq=1 --txq=1
> +
> +   check the Max possible RX queues and TX queues is 1::
> +
> +    testpmd> show port info all
> +    Max possible RX queues: 1
> +    Max possible number of RXDs per queue: 4096
> +    Min possible number of RXDs per queue: 64
> +    RXDs number alignment: 32
> +    Max possible TX queues: 1
> +    Max possible number of TXDs per queue: 4096
> +    Min possible number of TXDs per queue: 64
> +    TXDs number alignment: 32
> +
> +   start forwarding, you can see the actual queue number is 1::
> +
> +    testpmd> start
> +    RX queues=1 - RX desc=128 - RX free threshold=32
> +    TX queues=1 - TX desc=512 - TX free threshold=32
> +
> +3. repeat step1-2 with "queue-num-per-vf=2/4/8/16", and start VF testpmd
> +   with consistent rxq and txq number. check the max queue num and actual
> +   queue number is 2/4/8/16.

Why not check 32 queues here?

> +
> +Test case 2: set invalid VF max queue number 
> +============================================
> +
> +1. try the invalid value 0::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=0 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +   testpmd started with "i40e_pf_parse_vf_queue_number_handler(): Wrong
> +   VF queue number = 0, it must be power of 2 and equal or less than 16 !,
> +   Now it is kept the value = 4"
> +
> +2. start VF testpmd with "--rxq=4 --txq=4", the number of rxq and txq is
> +   consistent with the default VF max queue number::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i --rxq=4 --txq=4
> +
> +   check the Max possible RX queues and TX queues is 4::
> +
> +    testpmd> show port info all
> +    Max possible RX queues: 4
> +    Max possible TX queues: 4
> +
> +   start forwarding, you can see the actual queue number is 4::
> +
> +    testpmd> start
> +    RX queues=4 - RX desc=128 - RX free threshold=32
> +    TX queues=4 - TX desc=512 - TX free threshold=32
> +
> +3. repeat step1-2 with "queue-num-per-vf=6/17/32", and start VF testpmd
> +   with default max rxq and txq number. check the max queue num and
> actual
> +   queue number is 4.
> +
> +Test case 3: set VF queue number in testpmd command-line options 
> +================================================================
> +
> +1. set VF max queue number::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=8 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +2. start VF testpmd with "--rxq=3 --txq=3"::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i --rxq=3 --txq=3
> +
> +   check the Max possible RX queues and TX queues is 8::
> +
> +    testpmd> show port info all
> +    Max possible RX queues: 8
> +    Max possible TX queues: 8
> +
> +   start forwarding, you can see the actual queue number is 3::
> +
> +    testpmd> start
> +    RX queues=3 - RX desc=128 - RX free threshold=32
> +    TX queues=3 - TX desc=512 - TX free threshold=32
> +
> +3. quit the VF testpmd, then restart VF testpmd with "--rxq=9 --txq=9"::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i --rxq=9 --txq=9
> +
> +   VF testpmd failed to start with the print::
> +
> +    Fail: nb_rxq(9) is greater than max_rx_queues(8)
> +
> +Test case 4: set VF queue number with testpmd function command 
> +==============================================================
> +
> +1. set VF max queue number::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=8 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +2. start VF testpmd without setting "rxq" and "txq"::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i
> +
> +   check the Max possible RX queues and TX queues is 8,
> +   and actual RX queue number and TX queue number is 1::
> +
> +    testpmd> show port info all
> +    Current number of RX queues: 1
> +    Max possible RX queues: 8
> +    Current number of TX queues: 1
> +    Max possible TX queues: 8
> +
> +3. set rx queue number and tx queue number with testpmd function command::
> +
> +    testpmd> port stop all
> +    testpmd> port config all rxq 8
> +    testpmd> port config all txq 8
> +    testpmd> port start all
> +
> +4. start forwarding, you can see the actual queue number is 8::
> +
> +    testpmd> show port info all
> +    Current number of RX queues: 8
> +    Max possible RX queues: 8
> +    Current number of TX queues: 8
> +    Max possible TX queues: 8
> +
> +5. reset rx queue number and tx queue number to 7::
> +
> +    testpmd> port stop all
> +    testpmd> port config all rxq 7
> +    testpmd> port config all txq 7
> +    testpmd> port start all
> +
> +   start forwarding, you can see the actual queue number is 7::
> +
> +    testpmd> show port info all
> +    Current number of RX queues: 7
> +    Max possible RX queues: 8
> +    Current number of TX queues: 7
> +    Max possible TX queues: 8
> +
> +6. reset rx queue number and tx queue number to 9::
> +
> +    testpmd> port stop all
> +    testpmd> port config all txq 9
> +    Fail: nb_txq(9) is greater than max_tx_queues(8)
> +    testpmd> port config all rxq 9
> +    Fail: nb_rxq(9) is greater than max_rx_queues(8)
> +    testpmd> port start all
> +
> +   start forwarding, you can see the actual queue number is still 7::
> +
> +    testpmd> show port info all
> +    Current number of RX queues: 7
> +    Max possible RX queues: 8
> +    Current number of TX queues: 7
> +    Max possible TX queues: 8
> +
> +Test case 5: VF max queue number when VF bound to kernel driver 
> +===============================================================
> +
> +1. set VF max queue number by PF::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=2 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +2. check the VF0 rxq and txq number is 2::
> +
> +    # ethtool -S enp5s2
> +    NIC statistics:
> +         rx_bytes: 0
> +         rx_unicast: 0
> +         rx_multicast: 0
> +         rx_broadcast: 0
> +         rx_discards: 0
> +         rx_unknown_protocol: 0
> +         tx_bytes: 0
> +         tx_unicast: 0
> +         tx_multicast: 0
> +         tx_broadcast: 0
> +         tx_discards: 0
> +         tx_errors: 0
> +         tx-0.packets: 0
> +         tx-0.bytes: 0
> +         tx-1.packets: 0
> +         tx-1.bytes: 0
> +         rx-0.packets: 0
> +         rx-0.bytes: 0
> +         rx-1.packets: 0
> +         rx-1.bytes: 0
> +
> +   check the VF1 rxq and txq number is 2 too.
> +
> +3. repeat step1-2 with "queue-num-per-vf=1/4/8/16", check the rxq and txq
> +   number is 1/4/8/16.
> +
> +Test case 6: set VF max queue number with 32 VFs on one PF port 
> +===============================================================
> +
> +1. set up 32 VFs from one PF with DPDK driver::
> +
> +    echo 32 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
> +
> +   bind the two of the VFs to DPDK driver::
> +
> +    ./usertools/dpdk-devbind.py -b vfio-pci 05:02.0 05:05.7
> +
> +2. set VF max queue number to 16::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=16 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +   PF port failed to started with "i40e_pf_parameter_init():
> +   Failed to allocate 577 queues, which exceeds the hardware maximum 384"
> +
> +3. set VF max queue number to 8::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=8 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +4. start the two VFs testpmd with "--rxq=8 --txq=8" and "--rxq=6 --
> txq=6"::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf0 -n 4 -w 05:02.0 \
> +    --file-prefix=test2 --socket-mem 1024,1024 -- -i --rxq=8 --txq=8
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf00 -n 4 -w 05:05.7 \
> +    --file-prefix=test3 --socket-mem 1024,1024 -- -i --rxq=6 --txq=6
> +
> +   check the Max possible RX queues and TX queues of the two VFs are 
> + both
> 8::
> +
> +    testpmd> show port info all
> +    Max possible RX queues: 8
> +    Max possible TX queues: 8
> +
> +   start forwarding, you can see the actual queue number
> +   VF0::
> +
> +    testpmd> start
> +    RX queues=8 - RX desc=128 - RX free threshold=32
> +    TX queues=8 - TX desc=512 - TX free threshold=32
> +
> +   VF1::
> +
> +    testpmd> start
> +    RX queues=6 - RX desc=128 - RX free threshold=32
> +    TX queues=6 - TX desc=512 - TX free threshold=32
> +
> +   modify the queue number of VF1::
> +
> +    testpmd> stop
> +    testpmd> port stop all
> +    testpmd> port config all rxq 8
> +    testpmd> port config all txq 7
> +    testpmd> port start all
> +
> +   start forwarding, you can see the VF1 actual queue number is 8 and 7::
> +
> +    testpmd> start
> +    RX queues=8 - RX desc=128 - RX free threshold=32
> +    TX queues=7 - TX desc=512 - TX free threshold=32
> +
> +Test case 7: pass through VF to VM
> +==================================
> +
> +1. bind the pf to dpdk driver::
> +
> +    ./usertools/dpdk-devbind.py -b igb_uio 05:00.0
> +
> +   create 1 vf from pf::
> +
> +    echo 1 >/sys/bus/pci/devices/0000:05:00.0/max_vfs
> +
> +   Detach VF from the host, bind them to pci-stub driver::
> +
> +    modprobe pci-stub
> +    echo "8086 154c" > /sys/bus/pci/drivers/pci-stub/new_id
> +    echo "0000:05:02.0" > /sys/bus/pci/drivers/i40evf/unbind
> +    echo "0000:05:02.0" > /sys/bus/pci/drivers/pci-stub/bind
> +
> +   Lauch the VM with the VF PCI passthrough::
> +
> +    taskset -c 5-20 qemu-system-x86_64 \
> +    -enable-kvm -m 8192 -smp cores=16,sockets=1 -cpu host -name 
> + dpdk1-vm1
> \
> +    -drive file=/home/VM/ubuntu-14.04.img \
> +    -device pci-assign,host=0000:05:02.0 \
> +    -netdev tap,id=ipvm1,ifname=tap3,script=/etc/qemu-ifup -device
> rtl8139,netdev=ipvm1,id=net0,mac=00:00:00:00:00:01 \
> +    -localtime -vnc :2 -daemonize
> +
> +2. set VF Max possible RX queues and TX queues to 8 by PF::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 \
> +    -w 05:00.0,queue-num-per-vf=8 --file-prefix=test1 \
> +    --socket-mem 1024,1024 -- -i
> +
> +   testpmd can be started normally without any wrong or error.
> +
> +3. start VF testpmd with "--rxq=6 --txq=6", the number of rxq and txq is
> +   consistent with the configured VF max queue number::
> +
> +    ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i 
> + --rxq=6 --
> txq=6
> +
> +   check the Max possible RX queues and TX queues is 8::
> +
> +    testpmd> show port info all
> +    Max possible RX queues: 8
> +    Max possible TX queues: 8
> +
> +   start forwarding, you can see the actual queue number is 6::
> +
> +    testpmd> start
> +    RX queues=6 - RX desc=128 - RX free threshold=32
> +    TX queues=6 - TX desc=512 - TX free threshold=32
> +
> +   modify the queue number of VF::
> +
> +    testpmd> stop
> +    testpmd> port stop all
> +    testpmd> port config all rxq 8
> +    testpmd> port config all txq 8
> +    testpmd> port start all
> +
> +   start forwarding, you can see the VF1 actual queue number is 8::
> +
> +    testpmd> start
> +    RX queues=8 - RX desc=128 - RX free threshold=32
> +    TX queues=8 - TX desc=512 - TX free threshold=32
> +
> +4. repeat step2-3 with "queue-num-per-vf=1/2/4/16", and start VF testpmd
> +   with consistent rxq and txq number. check the max queue num and actual
> +   queue number is 1/2/4/16.
> +
> +5. bind VF to kernel driver i40evf, check the rxq and txq number.
> +   if set VF Max possible RX queues and TX queues to 2 by PF,
> +   the VF rxq and txq number is 2::
> +
> +    #ethtool -S eth0
> +    NIC statistics:
> +         rx_bytes: 0
> +         rx_unicast: 0
> +         rx_multicast: 0
> +         rx_broadcast: 0
> +         rx_discards: 0
> +         rx_unknown_protocol: 0
> +         tx_bytes: 70
> +         tx_unicast: 0
> +         tx_multicast: 1
> +         tx_broadcast: 0
> +         tx_discards: 0
> +         tx_errors: 0
> +         tx-0.packets: 2
> +         tx-0.bytes: 140
> +         tx-1.packets: 6
> +         tx-1.bytes: 1044
> +         rx-0.packets: 0
> +         rx-0.bytes: 0
> +         rx-1.packets: 0
> +         rx-1.bytes: 0
> +
> +   try to set VF Max possible RX queues and TX queues to 1/4/8/16 by PF,
> +   the VF rxq and txq number is 1/4/8/16::
> --
> 2.5.0