[dts] [PATCH] tests: add short live test suite

[Liu, Yong]

Applied. Thanks.

On 01/19/2016 10:29 AM, Michael Qiu wrote:
> DPDK has a two features in dpdk 16.04.
> 1) reduce start up time
> 2) signal handler when will be killed and do some clean up
>
> This patch is to enable automation for the features.
>
> Signed-off-by: Michael Qiu <michael.qiu at intel.com>
> ---
>   test_plans/short_live_test_plan.rst | 114 ++++++++++++++++++++
>   tests/TestSuite_short_live.py       | 210 ++++++++++++++++++++++++++++++++++++
>   2 files changed, 324 insertions(+)
>   create mode 100755 test_plans/short_live_test_plan.rst
>   create mode 100644 tests/TestSuite_short_live.py
>
> diff --git a/test_plans/short_live_test_plan.rst b/test_plans/short_live_test_plan.rst
> new file mode 100755
> index 0000000..4448cd1
> --- /dev/null
> +++ b/test_plans/short_live_test_plan.rst
> @@ -0,0 +1,114 @@
> +.. Copyright (c) <2016>, Intel Corporation
> +   All rights reserved.
> +
> +   Redistribution and use in source and binary forms, with or without
> +   modification, are permitted provided that the following conditions
> +   are met:
> +
> +   - Redistributions of source code must retain the above copyright
> +     notice, this list of conditions and the following disclaimer.
> +
> +   - Redistributions in binary form must reproduce the above copyright
> +     notice, this list of conditions and the following disclaimer in
> +     the documentation and/or other materials provided with the
> +     distribution.
> +
> +   - Neither the name of Intel Corporation nor the names of its
> +     contributors may be used to endorse or promote products derived
> +     from this software without specific prior written permission.
> +
> +   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
> +   FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
> +   COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
> +   INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> +   (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
> +   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> +   HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> +   STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
> +   ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
> +   OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +======================================
> +DPDK enable short-lived application
> +======================================
> +This feature is to reduce application start up time, and when exit, do more
> +clean up so that it could be re-run many times.
> +
> +Prerequisites
> +-------------
> +To test this feature, need to using linux time
> +and start testpmd by:
> +create and mount hugepage, must create more hugepages so that could measure time more obviously.
> +        # echo 8192 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> +        # mount -t hugetlbfs hugetlbfs /mnt/huge
> +Bind nic to DPDK
> +        ./tools/dpdk_nic_bind.py -b igb_uio xxxx:xx:xx.x
> +start testpmd using time
> +        # echo quit | time ./testpmd -c 0x3 -n 4 -- -i
> +
> +-------------------------------------------------------------------------------
> +Test Case 1: basic fwd testing
> +-------------------------------------------------------------------------------
> +1. start testpmd
> +    ./testpmd -c 0x3 -n 4 -- -i
> +2. set fwd mac
> +3. send packet from pkg
> +4. check all packets could be fwd back
> +
> +-------------------------------------------------------------------------------
> +Test Case 2: Get start up time
> +-------------------------------------------------------------------------------
> +1. start testpmd
> +    echo quit | time ./testpmd -c 0x3 -n 4 --huge-dir /mnt/huge -- -i
> +2. get the time stats of the startup
> +3. repeat step 1~2 for at least 5 times to get the average
> +
> +-------------------------------------------------------------------------------
> +Test Case 3: Clean up with Signal -- testpmd
> +-------------------------------------------------------------------------------
> +1. create 4G hugepages, so that could save times when repeat
> +    echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> +    mount -t hugetlbfs hugetlbfs /mnt/huge1
> +1. start testpmd
> +    ./testpmd -c 0x3 -n 4 --huge-dir /mnt/huge1 -- -i
> +2. set fwd mac
> +3. send packets from pkg
> +4. check all packets could be fwd back
> +5. kill the testpmd in shell using below commands alternately:
> +   SIGINT:  pkill -2  testpmd
> +   SIGTERM: pkill -15 testpmd
> +6. repeat step 1~5 for 20 times, and packet must be fwd back with no error for each time.
> +
> +-------------------------------------------------------------------------------
> +Test Case 4: Clean up with Signal -- l2fwd
> +-------------------------------------------------------------------------------
> +1. create 4G hugepages, so that could save times when repeat
> +    echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> +    mount -t hugetlbfs hugetlbfs /mnt/huge1
> +1. start testpmd
> +    ./l2fwd -c 0x3 -n 4 --huge-dir /mnt/huge1 -- -p 0x01
> +2. set fwd mac
> +3. send packets from pkg
> +4. check all packets could be fwd back
> +5. kill the testpmd in shell using below commands alternately:
> +   SIGINT:  pkill -2  l2fwd
> +   SIGTERM: pkill -15 l2fwd
> +6. repeat step 1~5 for 20 times, and packet must be fwd back with no error for each time.
> +
> +-------------------------------------------------------------------------------
> +Test Case 5: Clean up with Signal -- l3fwd
> +-------------------------------------------------------------------------------
> +1. create 4G hugepages, so that could save times when repeat
> +    echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> +    mount -t hugetlbfs hugetlbfs /mnt/huge1
> +1. start testpmd
> +    ./l3fwd -c 0x3 -n 4 --huge-dir /mnt/huge1 -- -p 0x01 --config="(0,0,1)"
> +2. set fwd mac
> +3. send packets from pkg
> +4. check all packets could be fwd back
> +5. kill the testpmd in shell using below commands alternately:
> +   SIGINT:  pkill -2  l3fwd
> +   SIGTERM: pkill -15 l3fwd
> +6. repeat step 1~5 for 20 times, and packet must be fwd back with no error for each time.
> diff --git a/tests/TestSuite_short_live.py b/tests/TestSuite_short_live.py
> new file mode 100644
> index 0000000..a72fcb9
> --- /dev/null
> +++ b/tests/TestSuite_short_live.py
> @@ -0,0 +1,210 @@
> +# BSD LICENSE
> +#
> +# Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions
> +# are met:
> +#
> +#   * Redistributions of source code must retain the above copyright
> +#     notice, this list of conditions and the following disclaimer.
> +#   * Redistributions in binary form must reproduce the above copyright
> +#     notice, this list of conditions and the following disclaimer in
> +#     the documentation and/or other materials provided with the
> +#     distribution.
> +#   * Neither the name of Intel Corporation nor the names of its
> +#     contributors may be used to endorse or promote products derived
> +#     from this software without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +
> +
> +"""
> +DPDK Test suite.
> +
> +Test short live dpdk app Feature
> +
> +"""
> +
> +import dts
> +import time
> +import re
> +import os
> +from test_case import TestCase
> +from pmd_output import PmdOutput
> +from settings import FOLDERS
> +
> +#
> +#
> +# Test class.
> +#
> +
> +
> +class TestShortLiveApp(TestCase):
> +    #
> +    #
> +    #
> +    # Test cases.
> +    #
> +
> +    def set_up_all(self):
> +        """
> +        Run at the start of each test suite.
> +        """
> +        self.ports = self.dut.get_ports(self.nic)
> +        self.verify(len(self.ports) >= 2, "Insufficient number of ports.")
> +
> +    def set_up(self):
> +        """
> +        Run before each test case.
> +        """
> +        pass
> +
> +    def compile_examples(self, example):
> +        # compile
> +        out = self.dut.build_dpdk_apps("./examples/%s"%example)
> +        self.verify("Error" not in out, "Compilation error")
> +        self.verify("No such" not in out, "Compilation error")
> +
> +
> +    def check_forwarding(self, ports, nic, testerports=[None, None], pktSize=64, received=True):
> +        self.send_packet(ports[0], ports[1], self.nic, testerports[1], pktSize, received)
> +
> +    def send_packet(self, txPort, rxPort, nic, testerports=None, pktSize=64, received=True):
> +        """
> +        Send packages according to parameters.
> +        """
> +
> +        if testerports is None:
> +            rxitf = self.tester.get_interface(self.tester.get_local_port(rxPort))
> +            txitf = self.tester.get_interface(self.tester.get_local_port(txPort))
> +        else:
> +            itf = testerports
> +        smac = self.tester.get_mac(self.tester.get_local_port(txPort))
> +        dmac = self.dut.get_mac_address(txPort)
> +        Dut_tx_mac = self.dut.get_mac_address(rxPort)
> +
> +        self.tester.scapy_background()
> +        count = 1
> +        # if only one port rx/tx, we should check count 2 so that both
> +        # rx and tx packet are list
> +        if (txPort == rxPort):
> +            count = 2
> +
> +        self.tester.scapy_append('p=sniff(iface="%s",count=%d,timeout=5)' % (rxitf, count))
> +        self.tester.scapy_append('RESULT=str(p)')
> +
> +        self.tester.scapy_foreground()
> +
> +        pktlen = pktSize - 14
> +        padding = pktlen - 20
> +        self.tester.scapy_append('sendp([Ether(src="%s", dst="%s")/IP()/Raw(load="P"*%s)], iface="%s")' % (smac, dmac, padding, txitf))
> +
> +        self.tester.scapy_execute()
> +        time.sleep(3)
> +
> +        out = self.tester.scapy_get_result()
> +        if received:
> +            self.verify(('PPP' in out) and 'src=%s'% Dut_tx_mac in out, "Receive test failed")
> +        else:
> +            self.verify('PPP' not in out, "Receive test failed")
> +
> +    def test_basic_forwarding(self):
> +        """
> +        Basic rx/tx forwarding test
> +        """
> +        #dpdk start
> +        self.dut.send_expect("./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i --portmask=0x3", "testpmd>", 120)
> +        self.dut.send_expect("set fwd mac", "testpmd>")
> +        self.dut.send_expect("set promisc all off", "testpmd>")
> +        self.dut.send_expect("start", "testpmd>")
> +        self.check_forwarding([0, 1], self.nic)
> +
> +    def test_start_up_time(self):
> +        """
> +        Using linux time to get start up time
> +        """
> +        time = []
> +        regex = re.compile(".* (\d+:\d{2}\.\d{2}).*")
> +        out = self.dut.send_expect("echo quit | time ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 --no-pci -- -i", "#", 120)
> +        time = regex.findall(out)
> +
> +        if time != []:
> +            print "start time: %s s"%time[0]
> +        else:
> +            self.verify(0, "start_up_time failed")
> +
> +    def test_clean_up_with_signal_testpmd(self):
> +        repeat_time = 5
> +        for i in range(repeat_time):
> +            #dpdk start
> +            print "clean_up_with_signal_testpmd round %d" % (i + 1)
> +            self.dut.send_expect("./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -- -i --portmask=0x3", "testpmd>", 120)
> +            self.dut.send_expect("set fwd mac", "testpmd>")
> +            self.dut.send_expect("set promisc all off", "testpmd>")
> +            self.dut.send_expect("start", "testpmd>")
> +            self.check_forwarding([0, 1], self.nic)
> +
> +            # kill with differen Signal
> +            if i%2 == 0:
> +                self.dut.send_expect("pkill -2 testpmd", "#", 60, True)
> +            else:
> +                self.dut.send_expect("pkill -15 testpmd", "#", 60, True)
> +
> +    def test_clean_up_with_signal_l2fwd(self):
> +        repeat_time = 5
> +        self.compile_examples("l2fwd")
> +        for i in range(repeat_time):
> +            #dpdk start
> +            print "clean_up_with_signal_l2fwd round %d" % (i + 1)
> +            self.dut.send_expect("./examples/l2fwd/build/app/l2fwd -n 4 -c 0xf -- -p 0x3 &", "L2FWD: entering main loop", 60)
> +            self.check_forwarding([0, 1], self.nic)
> +
> +            # kill with differen Signal
> +            if i%2 == 0:
> +                self.dut.send_expect("pkill -2 l2fwd", "#", 60, True)
> +            else:
> +                self.dut.send_expect("pkill -15 l2fwd", "#", 60, True)
> +
> +    def test_clean_up_with_signal_l3fwd(self):
> +        repeat_time = 5
> +        self.compile_examples("l3fwd")
> +        for i in range(repeat_time):
> +            #dpdk start
> +            print "clean_up_with_signal_l3fwd round %d" % (i + 1)
> +            self.dut.send_expect("./examples/l3fwd/build/app/l3fwd -n 4 -c 0xf -- -p 0x3 --config='(0,0,1),(1,0,2)' &", "L3FWD:", 120)
> +            self.check_forwarding([0, 0], self.nic)
> +
> +            # kill with differen Signal
> +            if i%2 == 0:
> +                self.dut.send_expect("pkill -2 l3fwd", "#", 60, True)
> +            else:
> +                self.dut.send_expect("pkill -15 l3fwd", "#", 60, True)
> +
> +    def tear_down(self):
> +        """
> +        Run after each test case.
> +        """
> +
> +        self.dut.kill_all()
> +
> +    def tear_down_all(self):
> +        """
> +        Run after each test suite.
> +        """
> +        self.dut.kill_all()
> +        self.dut.send_expect("rm -rf ./app/test-pmd/testpmd", "#")
> +        self.dut.send_expect("rm -rf ./app/test-pmd/*.o", "#")
> +        self.dut.send_expect("rm -rf ./app/test-pmd/build", "#")