[dpdk-users] Problems running TestPMD on a 32-bit VM running under a 64-bit hypervisor with the Intel XL710 NIC and DPDK 2.2.0

[Michael Arnold]

Hello, I was attempting to get SR-IOV working with the Intel XL710 NIC
running in SR-IOV mode when I encountered the below error message.  The
interesting thing is that the error only seems to occur when the VM is a 32
bit debian install (instead of 64 bit).  This is with DPDK 2.2.0.

Normally I would just do this with the 64 bit VM and call it done, but the
reason I was running TestPMD was to see if this NIC worked in a 32 bit VM,
and it would not be trivial to upgrade said VM to 64 bit at this time for
unrelated reasons.

root at debian-32:/usr/src/dpdk-2.2.0/i686-native-linuxapp-gcc/build# uname -a
Linux debian-32 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
i686 GNU/Linux

root at debian-32:/usr/src/dpdk-2.2.0/i686-native-linuxapp-gcc/build#
lsb_release -da
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.5 (jessie)
Release: 8.5
Codename: jessie
root at debian-32:/usr/src/dpdk-2.2.0/i686-native-linuxapp-gcc/build#

Any assistance will be greatly appreciated.

root at debian-32:/usr/src/dpdk-2.2.0/i686-native-linuxapp-gcc/build#
app/test-pmd/testpmd -c7 -n3 -- -i --nb-cores=2 --nb-ports=2 --rxq=1
--txq=1 --txqflags=0x0f00 --total-num-mbufs=2048
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0
EAL: Ask a virtual area of 0x4c00000 bytes
EAL: Virtual area found at 0xb1e00000 (size = 0x4c00000)
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0xb1400000 (size = 0x800000)
EAL: Ask a virtual area of 0xac00000 bytes
EAL: Virtual area found at 0xa6600000 (size = 0xac00000)
EAL: Requesting 128 pages of size 2MB from socket 0
EAL: TSC frequency is ~2399997 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
clock cycles !
EAL: Master lcore 0 is ready (tid=b7519880;cpuset=[0])
EAL: lcore 2 is ready (tid=a5dfeb40;cpuset=[2])
EAL: lcore 1 is ready (tid=a65ffb40;cpuset=[1])
EAL: PCI device 0000:00:05.0 on NUMA socket 0
EAL:   probe driver: 8086:154c rte_i40evf_pmd
EAL:   PCI memory mapped at 0xb6a00000
EAL:   PCI memory mapped at 0xb6a10000
EAL: PCI device 0000:00:06.0 on NUMA socket 0
EAL:   probe driver: 8086:154c rte_i40evf_pmd
EAL:   PCI memory mapped at 0xb6a14000
EAL:   PCI memory mapped at 0xb6a24000
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_set_rx_function(): Port[0] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=0.
PMD: i40e_set_tx_function(): Xmit tx finally be used.
Fail to start port 0
Configuring Port 1 (socket 0)
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
PMD: i40e_set_rx_function(): Port[1] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=1.
PMD: i40e_set_tx_function(): Xmit tx finally be used.
Fail to start port 1
Please stop the ports first

Back on the Hypervisor, we see error messages.

Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.130874] i40e 0000:81:00.0: VF
0 failed opcode 6, error: -5
Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.133934] i40e 0000:81:00.0:
Invalid message from VF 0, opcode 6, len 116
Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.231520] i40e 0000:81:00.1: VF
0 failed opcode 6, error: -5
Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.234573] i40e 0000:81:00.1:
Invalid message from VF 0, opcode 6, len 116
Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.331606] i40e 0000:81:00.0: VF
0 failed opcode 14, error: -5
Jun 20 17:07:35 PhiPhyPhoPhum kernel: [ 4432.431605] i40e 0000:81:00.1: VF
0 failed opcode 14, error: -5

With the 64-bit debian VM it works.

root at marnold-debian:/usr/src/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build#
app/test-pmd/testpmd -c7 -n3 -- -i --nb-cores=2 --nb-ports=2 --rxq=1
--txq=1 --txqflags=0x0f00 --total-num-mbufs=2048
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
PANIC in rte_eal_init():
Cannot get hugepage information
6: [app/test-pmd/testpmd() [0x43679b]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
[0x7f1feebe1b45]]
4: [app/test-pmd/testpmd(main+0x14) [0x434a24]]
3: [app/test-pmd/testpmd(rte_eal_init+0xba8) [0x47bb68]]
2: [app/test-pmd/testpmd(__rte_panic+0xc9) [0x434a0b]]
1: [app/test-pmd/testpmd(rte_dump_stack+0x18) [0x5a6e88]]
Aborted
root at marnold-debian:/usr/src/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build#
echo "128" /proc/m
meminfo  misc     modules  mounts   mtrr
root at marnold-debian:/usr/src/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build#
echo "128" /proc/sys/vm/nr_hugepages
128 /proc/sys/vm/nr_hugepages
root at marnold-debian:/usr/src/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build#
echo "128" >> /proc/sys/vm/nr_hugepages
root at marnold-debian:/usr/src/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build#
app/test-pmd/testpmd -c7 -n3 -- -i --nb-cores=2 --nb-ports=2 --rxq=1
--txq=1 --txqflags=0x0f00 --total-num-mbufs=2048
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd5e4e00000 (size = 0x200000)
EAL: Ask a virtual area of 0xb000000 bytes
EAL: Virtual area found at 0x7fd5d9c00000 (size = 0xb000000)
EAL: Ask a virtual area of 0x4c00000 bytes
EAL: Virtual area found at 0x7fd5d4e00000 (size = 0x4c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd5d4a00000 (size = 0x200000)
EAL: Requesting 128 pages of size 2MB from socket 0
EAL: TSC frequency is ~2399997 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
clock cycles !
EAL: Master lcore 0 is ready (tid=e6c238c0;cpuset=[0])
EAL: lcore 2 is ready (tid=d41fe700;cpuset=[2])
EAL: lcore 1 is ready (tid=d49ff700;cpuset=[1])
EAL: PCI device 0000:00:05.0 on NUMA socket -1
EAL:   probe driver: 8086:154c rte_i40evf_pmd
EAL:   PCI memory mapped at 0x7fd5e5000000
EAL:   PCI memory mapped at 0x7fd5e5010000
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   probe driver: 8086:154c rte_i40evf_pmd
EAL:   PCI memory mapped at 0x7fd5e5014000
EAL:   PCI memory mapped at 0x7fd5e5024000
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_set_rx_function(): Port[0] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=0.
PMD: i40e_set_tx_function(): Xmit tx finally be used.
Port 0: 52:54:00:DC:52:01
Configuring Port 1 (socket 0)
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
PMD: i40e_set_rx_function(): Port[1] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=1.
PMD: i40e_set_tx_function(): Xmit tx finally be used.
Port 1: 52:54:00:DC:53:01
Checking link statuses...
Port 0 Link Up - speed 10000 Mbps - half-duplex

Port 1 Link Up - speed 10000 Mbps - half-duplex

Done

-- 
Michael Arnold