[dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

Prashant Upadhyaya prashant.upadhyaya at aricent.com
Mon Mar 10 09:00:48 CET 2014


Hi Srini,

Thanks, I could also make it work, thanks to your cue !

Now then, this multi-segment not being supported in vmxnet3 driver is a big party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use of multisegment buffers for sending out the data, so my usecase has failed and I will have to fix that.

Also, can you please adivse how much is the max data rates you have been able to achieve with one vmxnet3 10G port.

Thanks a lot for the advice once again.

Regards
-Prashant


-----Original Message-----
From: Srinivasan J [mailto:srinidpdk at gmail.com]
Sent: Sunday, March 09, 2014 12:38 AM
To: Prashant Upadhyaya
Cc: David Marchand; dev at dpdk.org
Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

Prashant,
             I was also able to hit the issue your hitting using Esxi
5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in defconfig_x86_64-default-linuxapp-gcc configuration file.

Issue seen
========

EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Device is blacklisted, not initializing
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd

Program received signal SIGSEGV, Segmentation fault.
eth_vmxnet3_dev_init (eth_drv=<optimized out>, eth_dev=0x754480
<rte_eth_devices>) at
/root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218
218         ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS);
Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64
(gdb) p hw
$1 = (struct vmxnet3_hw *) 0x7fffd8fc1040
(gdb) p *hw
$2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0,
  adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0,
  tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
(gdb)

#define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \
        ((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define VMXNET3_READ_BAR1_REG(hw, reg) \
        vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg))) lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h

Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y ==================================================

[root at localhost build]# ./l2fwd -c 0xf -b 0000:03:00.0 -n 1 -- -p 0x6
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Skip lcore 4 (not detected)
EAL: Skip lcore 5 (not detected)
EAL: Skip lcore 6 (not detected)
EAL: Skip lcore 7 (not detected)
EAL: Skip lcore 8 (not detected)
EAL: Skip lcore 9 (not detected)
EAL: Skip lcore 10 (not detected)
EAL: Skip lcore 11 (not detected)
EAL: Skip lcore 12 (not detected)
EAL: Skip lcore 13 (not detected)
EAL: Skip lcore 14 (not detected)
EAL: Skip lcore 15 (not detected)
EAL: Skip lcore 16 (not detected)
EAL: Skip lcore 17 (not detected)
EAL: Skip lcore 18 (not detected)
EAL: Skip lcore 19 (not detected)
EAL: Skip lcore 20 (not detected)
EAL: Skip lcore 21 (not detected)
EAL: Skip lcore 22 (not detected)
EAL: Skip lcore 23 (not detected)
EAL: Skip lcore 24 (not detected)
EAL: Skip lcore 25 (not detected)
EAL: Skip lcore 26 (not detected)
EAL: Skip lcore 27 (not detected)
EAL: Skip lcore 28 (not detected)
EAL: Skip lcore 29 (not detected)
EAL: Skip lcore 30 (not detected)
EAL: Skip lcore 31 (not detected)
EAL: Skip lcore 32 (not detected)
EAL: Skip lcore 33 (not detected)
EAL: Skip lcore 34 (not detected)
EAL: Skip lcore 35 (not detected)
EAL: Skip lcore 36 (not detected)
EAL: Skip lcore 37 (not detected)
EAL: Skip lcore 38 (not detected)
EAL: Skip lcore 39 (not detected)
EAL: Skip lcore 40 (not detected)
EAL: Skip lcore 41 (not detected)
EAL: Skip lcore 42 (not detected)
EAL: Skip lcore 43 (not detected)
EAL: Skip lcore 44 (not detected)
EAL: Skip lcore 45 (not detected)
EAL: Skip lcore 46 (not detected)
EAL: Skip lcore 47 (not detected)
EAL: Skip lcore 48 (not detected)
EAL: Skip lcore 49 (not detected)
EAL: Skip lcore 50 (not detected)
EAL: Skip lcore 51 (not detected)
EAL: Skip lcore 52 (not detected)
EAL: Skip lcore 53 (not detected)
EAL: Skip lcore 54 (not detected)
EAL: Skip lcore 55 (not detected)
EAL: Skip lcore 56 (not detected)
EAL: Skip lcore 57 (not detected)
EAL: Skip lcore 58 (not detected)
EAL: Skip lcore 59 (not detected)
EAL: Skip lcore 60 (not detected)
EAL: Skip lcore 61 (not detected)
EAL: Skip lcore 62 (not detected)
EAL: Skip lcore 63 (not detected)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f3a76a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x7c00000 bytes
EAL: Virtual area found at 0x7f3a6ec00000 (size = 0x7c00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f3a6e800000 (size = 0x200000)
EAL: Requesting 64 pages of size 2MB from socket 0
EAL: TSC frequency is ~2200000 KHz
EAL: Master core 0 is ready (tid=76cfe880)
EAL: Core 3 is ready (tid=6cffc700)
EAL: Core 1 is ready (tid=6dffe700)
EAL: Core 2 is ready (tid=6d7fd700)
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   Device is blacklisted, not initializing
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   PCI memory mapped at 0x7f3a76d10000
EAL:   PCI memory mapped at 0x7f3a76d0f000
EAL:   PCI memory mapped at 0x7f3a76d0d000
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
EAL:   PCI memory mapped at 0x7f3a76d0c000
EAL:   PCI memory mapped at 0x7f3a76d0b000
EAL:   PCI memory mapped at 0x7f3a76d09000
Notice: odd number of ports in portmask.
Lcore 0: RX port 1
Skipping disabled port 0
Initializing port 1... done:
Port 1, MAC address: 00:0C:29:F1:9D:A4


Checking link statusdone
Port 1 Link Up - speed 10000 Mbps - full-duplex
L2FWD: lcore 1 has nothing to do
L2FWD: lcore 2 has nothing to do
L2FWD: lcore 3 has nothing to do
L2FWD: entering main loop on lcore 0
L2FWD:  -- lcoreid=0 portid=1

Port statistics ====================================
Statistics for port 1 ------------------------------
Packets sent:                        0
Packets received:                    0
Packets dropped:                     0
Statistics for port 2 ------------------------------
Packets sent:                        0
Packets received:                    0
Packets dropped:                     0
Aggregate statistics ===============================
Total packets sent:                  0
Total packets received:              0
Total packets dropped:               0
====================================================




On Fri, Mar 7, 2014 at 10:43 AM, Prashant Upadhyaya <prashant.upadhyaya at aricent.com> wrote:
> Hi Srini,
>
> Yes, I am still stuck, I am using ESXi5.1 though and Fedora18 in Guest.
> Please let me know what is the guest OS distro you are using in the VM. I'll try with that once.
>
> Regards
> -Prashant
>
>
> -----Original Message-----
> From: Srinivasan J [mailto:srinidpdk at gmail.com]
> Sent: Friday, March 07, 2014 9:08 AM
> To: Prashant Upadhyaya
> Cc: David Marchand; dev at dpdk.org
> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>
> Prashant,
>              Are you still hitting a crash? Do you have hugetlbfs
> enabled? if not try adding enabling hugetlbfs. I was able to get
> vxnet3 up using the vmxnet3 pmd alone in DPDK 1.60r1. I did not use
> the vmxnet3-usermap kernel driver. I am using Esxi 5.5
>
> Eg: echo 1024 > /proc/sys/vm/nr_hugepages; mkdir /huge; mount -t
> hugetlbfs nodev /huge;
>
> On Thu, Mar 6, 2014 at 12:48 PM, Prashant Upadhyaya <prashant.upadhyaya at aricent.com> wrote:
>> And if it is any help, here is the startup log --
>>
>> EAL: Detected lcore 0 as core 0 on socket 0
>> EAL: Detected lcore 1 as core 1 on socket 0
>> EAL: Detected lcore 2 as core 2 on socket 0
>> EAL: Detected lcore 3 as core 3 on socket 0
>> EAL: Detected lcore 4 as core 4 on socket 0
>> EAL: Detected lcore 5 as core 5 on socket 0
>> EAL: Detected lcore 6 as core 6 on socket 0
>> EAL: Detected lcore 7 as core 7 on socket 0
>> EAL: Skip lcore 8 (not detected)
>> EAL: Skip lcore 9 (not detected)
>> EAL: Skip lcore 10 (not detected)
>> EAL: Skip lcore 11 (not detected)
>> EAL: Skip lcore 12 (not detected)
>> EAL: Skip lcore 13 (not detected)
>> EAL: Skip lcore 14 (not detected)
>> EAL: Skip lcore 15 (not detected)
>> EAL: Skip lcore 16 (not detected)
>> EAL: Skip lcore 17 (not detected)
>> EAL: Skip lcore 18 (not detected)
>> EAL: Skip lcore 19 (not detected)
>> EAL: Skip lcore 20 (not detected)
>> EAL: Skip lcore 21 (not detected)
>> EAL: Skip lcore 22 (not detected)
>> EAL: Skip lcore 23 (not detected)
>> EAL: Skip lcore 24 (not detected)
>> EAL: Skip lcore 25 (not detected)
>> EAL: Skip lcore 26 (not detected)
>> EAL: Skip lcore 27 (not detected)
>> EAL: Skip lcore 28 (not detected)
>> EAL: Skip lcore 29 (not detected)
>> EAL: Skip lcore 30 (not detected)
>> EAL: Skip lcore 31 (not detected)
>> EAL: Skip lcore 32 (not detected)
>> EAL: Skip lcore 33 (not detected)
>> EAL: Skip lcore 34 (not detected)
>> EAL: Skip lcore 35 (not detected)
>> EAL: Skip lcore 36 (not detected)
>> EAL: Skip lcore 37 (not detected)
>> EAL: Skip lcore 38 (not detected)
>> EAL: Skip lcore 39 (not detected)
>> EAL: Skip lcore 40 (not detected)
>> EAL: Skip lcore 41 (not detected)
>> EAL: Skip lcore 42 (not detected)
>> EAL: Skip lcore 43 (not detected)
>> EAL: Skip lcore 44 (not detected)
>> EAL: Skip lcore 45 (not detected)
>> EAL: Skip lcore 46 (not detected)
>> EAL: Skip lcore 47 (not detected)
>> EAL: Skip lcore 48 (not detected)
>> EAL: Skip lcore 49 (not detected)
>> EAL: Skip lcore 50 (not detected)
>> EAL: Skip lcore 51 (not detected)
>> EAL: Skip lcore 52 (not detected)
>> EAL: Skip lcore 53 (not detected)
>> EAL: Skip lcore 54 (not detected)
>> EAL: Skip lcore 55 (not detected)
>> EAL: Skip lcore 56 (not detected)
>> EAL: Skip lcore 57 (not detected)
>> EAL: Skip lcore 58 (not detected)
>> EAL: Skip lcore 59 (not detected)
>> EAL: Skip lcore 60 (not detected)
>> EAL: Skip lcore 61 (not detected)
>> EAL: Skip lcore 62 (not detected)
>> EAL: Skip lcore 63 (not detected)
>> EAL: Setting up memory...
>> EAL: Ask a virtual area of 0x80000000 bytes
>> EAL: Virtual area found at 0x7f848ae00000 (size = 0x80000000)
>> EAL: Requesting 1024 pages of size 2MB from socket 0
>> EAL: TSC frequency is ~2000000 KHz
>> EAL: Master core 0 is ready (tid=b3f3f00)
>> EAL: Core 1 is ready (tid=8a1f2700)
>> EAL: Core 2 is ready (tid=899f1700)
>> EAL: Core 3 is ready (tid=891f0700)
>> EAL: Core 4 is ready (tid=889ef700)
>> EAL: Core 5 is ready (tid=7bfff700)
>> EAL: Core 6 is ready (tid=7b7fe700)
>> EAL: Core 7 is ready (tid=7affd700)
>> Pool initialized
>> Global Variables initialized
>> PMD: rte_vmxnet3_pmd_init():  >>
>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>> PMD: eth_vmxnet3_dev_init():  >>
>> Segmentation fault (core dumped)
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Prashant
>> Upadhyaya
>> Sent: Thursday, March 06, 2014 12:20 PM
>> To: David Marchand
>> Cc: dev at dpdk.org
>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>>
>> Hi,
>>
>> Some further update on the crash I am facing.
>> I am using DPDK1.6.0r1 and take over the vmxnet3 with igb_uio and
>> then start the application. (so no external ko or vmxnet usermap
>> etc.) During the port initializations, the crash is happening in the
>> following function -
>>
>> eth_vmxnet3_dev_init
>>
>> and the crash is happening at the following line -
>> /* Check h/w version compatibility with driver. */
>>     ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS);
>>
>> Any hints regarding what could be wrong ?
>>
>> Regards
>> -Prashant
>>
>>
>> From: Prashant Upadhyaya
>> Sent: Wednesday, March 05, 2014 9:01 PM
>> To: 'David Marchand'
>> Cc: Srinivasan J; dev at dpdk.org
>> Subject: RE: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>>
>> Hi David,
>>
>> The compilation error with debug flags on was that the functions -- vmxnet3_txq_dump, vmxnet3_rxq_dump - are defined but not used.
>> Not a serious error, I will try to get rid of the compiler flag which generates this.
>>
>> However, I must reiterate, I _did_ bind my vmxnet3 device with igb_uio (and I did not use any .so, because I was intending to use the builtin vmxnet3 driver of dpdk 1.6.0r1), the bind succeeded, but then when I started the application, the dev init for vmxnet3 gave a core dump.
>> Your patch and solution seems to be suggesting the reverse, i.e. when I don't bind with igb_uio but try to use the native driver. So please do try the above combination as well.
>>
>> Regards
>> -Prashant
>>
>>
>> From: David Marchand [mailto:david.marchand at 6wind.com]
>> Sent: Wednesday, March 05, 2014 8:41 PM
>> To: Prashant Upadhyaya
>> Cc: Srinivasan J; dev at dpdk.org<mailto:dev at dpdk.org>
>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>>
>> Hello Prashant,
>>
>> On Wed, Mar 5, 2014 at 3:28 PM, Prashant Upadhyaya <prashant.upadhyaya at aricent.com<mailto:prashant.upadhyaya at aricent.com>> wrote:
>> Hi,
>>
>> I am also keen to know the answer to the question posted by Srini.
>> The real question is -- is dpdk1.6.0r1 self-sufficient so that I don't need any extensions etc., or do I still need something from outside like the usermap kernel driver etc.
>>
>> Secondly, if I turn on all the debug options for the vmxnet3 pmd in the config file, 1.6.0r1 compilation runs into a problem and reports a function which is defined but not used.
>>
>> Can you send your build error ? (maybe in a separate thread ?)
>>
>>
>> I am trying to bring up DPDK inside Fedora18 Guest on ESXi -- when I used DPDK1.6.0r1 (without debug options turned on for vmxnet3 pmd) the igb_uio could take over the vmxnet3 NIC but I encountered a core dump in the dev init function for the vmxnet3 driver -- anybody encountered a similar issue ?
>>
>> I encountered these problems as well.
>>
>> - igb_uio module does not check if you disable vmxnet3-uio pmds, it will always try to take over vmxnet3 devices.
>> I have a patch waiting in my working dir to cleanly disable vmxnet3-uio pmd.
>>
>> - If you don't bind vmxnet3 devices to uio, but forget to enable vmxnet3-usermap pmd (by specifiying -d librte_pmd_vmxnet3.so), then internal vmxnet3-uio pmd will try to initialise and crash.
>> I did not look any deeper into this, the easiest way is to disable vmxnet3-uio pmd + apply the patch I will send in a few minutes, as a first workaround.
>>
>>
>> Regards,
>> --
>> David Marchand
>>
>>
>>
>>
>> =====================================================================
>> =
>> ========= Please refer to
>> http://www.aricent.com/legal/email_disclaimer.html
>> for important disclosures regarding this electronic communication.
>> =====================================================================
>> =
>> =========
>>
>>
>>
>>
>> =====================================================================
>> =
>> ========= Please refer to
>> http://www.aricent.com/legal/email_disclaimer.html
>> for important disclosures regarding this electronic communication.
>> =====================================================================
>> =
>> =========
>
>
>
>
> ======================================================================
> ========= Please refer to
> http://www.aricent.com/legal/email_disclaimer.html
> for important disclosures regarding this electronic communication.
> ======================================================================
> =========




===============================================================================
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===============================================================================


More information about the dev mailing list