[dpdk-dev] Errors Rx count increasing while pktgen doing nothing on Intel 82598EB 10G

Moon-Sang Lee sang0627 at gmail.com
Fri Jan 29 11:02:45 CET 2016


Let me share one more information about this issue.
I tried to run dpdk-2.2.0/examples/ethtool, which showed the same problem.
(i.e. I guess other examples will also fail.)


[root at centos7 app]# ./ethtool
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 1 on socket 0
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 2 on socket 0
EAL: Detected lcore 5 as core 2 on socket 1
EAL: Detected lcore 6 as core 3 on socket 0
EAL: Detected lcore 7 as core 3 on socket 1
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 0 on socket 1
EAL: Detected lcore 10 as core 1 on socket 0
EAL: Detected lcore 11 as core 1 on socket 1
EAL: Detected lcore 12 as core 2 on socket 0
EAL: Detected lcore 13 as core 2 on socket 1
EAL: Detected lcore 14 as core 3 on socket 0
EAL: Detected lcore 15 as core 3 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1400000 bytes
EAL: Virtual area found at 0x7f79fa800000 (size = 0x1400000)
EAL: Ask a virtual area of 0x7ec00000 bytes
EAL: Virtual area found at 0x7f797ba00000 (size = 0x7ec00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f797b600000 (size = 0x200000)
EAL: Ask a virtual area of 0x7fc00000 bytes
EAL: Virtual area found at 0x7f78fb800000 (size = 0x7fc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f78fb400000 (size = 0x200000)
EAL: Ask a virtual area of 0x100000000 bytes
EAL: Virtual area found at 0x7f77fb200000 (size = 0x100000000)
EAL: Requesting 2048 pages of size 2MB from socket 0
EAL: Requesting 2048 pages of size 2MB from socket 1
EAL: TSC frequency is ~2260984 KHz
EAL: Master lcore 0 is ready (tid=fdc578c0;cpuset=[0])
EAL: lcore 6 is ready (tid=f81cb700;cpuset=[6])
EAL: lcore 4 is ready (tid=f91cd700;cpuset=[4])
EAL: lcore 12 is ready (tid=f51c5700;cpuset=[12])
EAL: lcore 9 is ready (tid=f69c8700;cpuset=[9])
EAL: lcore 8 is ready (tid=f71c9700;cpuset=[8])
EAL: lcore 14 is ready (tid=f41c3700;cpuset=[14])
EAL: lcore 3 is ready (tid=f99ce700;cpuset=[3])
EAL: lcore 13 is ready (tid=f49c4700;cpuset=[13])
EAL: lcore 7 is ready (tid=f79ca700;cpuset=[7])
EAL: lcore 10 is ready (tid=f61c7700;cpuset=[10])
EAL: lcore 11 is ready (tid=f59c6700;cpuset=[11])
EAL: lcore 15 is ready (tid=f39c2700;cpuset=[15])
EAL: lcore 5 is ready (tid=f89cc700;cpuset=[5])
EAL: lcore 2 is ready (tid=fa1cf700;cpuset=[2])
EAL: lcore 1 is ready (tid=fc562700;cpuset=[1])
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:06:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10e8 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:07:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10e8 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:07:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10e8 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:09:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10c7 rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f79fbc00000
EAL:   PCI memory mapped at 0x7f79fbc20000
EAL:   PCI memory mapped at 0x7f79fbc60000
PMD: eth_ixgbe_dev_init(): MAC: 1, PHY: 8
PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x10c7
EAL: PCI device 0000:0a:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10c7 rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f79fbc64000
EAL:   PCI memory mapped at 0x7f79fbc84000
EAL:   PCI memory mapped at 0x7f79fbcc4000
PMD: eth_ixgbe_dev_init(): MAC: 1, PHY: 8
PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10c7
Number of NICs: 2
Init port 0..
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f78f99d7140
sw_sc_ring=0x7f78f99d6c00 hw_ring=0x7f78f99d7680 dma_addr=0x4207d7680
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f78f99c5a40
hw_ring=0x7f78f99c6a80 dma_addr=0x4207c6a80
PMD: ixgbe_set_tx_function(): Using simple tx code path
PMD: ixgbe_set_tx_function(): Vector tx enabled.
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=0).
PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
Init port 1..
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f78f846ad40
sw_sc_ring=0x7f78f846a800 hw_ring=0x7f78f846b280 dma_addr=0x41f26b280
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f78f8459640
hw_ring=0x7f78f845a680 dma_addr=0x41f25a680
PMD: ixgbe_set_tx_function(): Using simple tx code path
PMD: ixgbe_set_tx_function(): Vector tx enabled.
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=1).
PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
EthApp> pause
 [UINT16]: pause <port_id>
     Print port pause state
 [UINT16]: pause <port_id> <all|tx|rx|none>
     Pause/unpause port
EthApp> pause 0
Port 0: Rx Paused
EthApp> pause 0 none
PMD: ixgbe_flow_ctrl_set(): Rx packet buffer size = 0x80000
Port 0: Tx & Rx not paused
EthApp> pause 0
Port 0: Rx Paused
EthApp> quit
[root at centos7 app]# pwd



On Thu, Jan 28, 2016 at 9:57 AM, Moon-Sang Lee <sang0627 at gmail.com> wrote:

>
> Helin, I implemented my own sample application that is a kind of carrier
> grade NAT server.
> It works fine on 1G NIC (i.e. Intel Corporation 82576 Gigabit Network
> Connection (rev 01))
> But, it does not receive packets on 10G NIC (i.e. Intel Corporation
> 82598EB 10-Gigabit AF Network Connection (rev 01)) as described in the
> previous email.
> According to my log messages, it seems that control register for RX DMA is
> not enabled.
>
> Here is some information about my environment.
>
> 1. HW & OS
> [mslee at centos7 ~]$ uname -a
> Linux centos7 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
> [mslee at centos7 ~]$ more /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 26
> model name : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
> stepping : 5
> microcode : 0x19
> cpu MHz : 2262.000
> cache size : 8192 KB
> physical id : 1
> siblings : 8
> core id : 0
> cpu cores : 4
> apicid : 16
> initial apicid : 16
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> rdtscp lm c
> onstant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
> aperfmperf
>  pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1
> sse4_2 po
> pcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4521.93
> clflush size : 64
> cache_alignment : 64
> address sizes : 40 bits physical, 48 bits virtual
> power management:
> ...
>
>
>
> 2. port ocnfigure parameter for rte_eth_dev_configure():
> ret = rte_eth_dev_configure(port, NB_RXQ, NB_TXQ, &port_conf);
> where NB_RXQ=1, NB_TXQ=2, and
> struct rte_eth_conf port_conf = {
>     .rxmode = {
>         //.mq_mode = ETH_MQ_RX_RSS,
>         .mq_mode = ETH_MQ_RX_NONE,  // for 10G NIC
>         .max_rx_pkt_len = ETHER_MAX_LEN,
>         .split_hdr_size = 0,
>         .header_split = 0,      // Header Split disabled
>         .hw_ip_checksum = 0,    // IP checksum offload enabled
>         .hw_vlan_filter = 0,    // VLAN filtering disabled
>         .jumbo_frame = 0,       // Jumbo Frame Support disabled
>         .hw_strip_crc = 0,      // CRC stripped by hardware
>     },
>     .rx_adv_conf = {
>         .rss_conf = {
>             .rss_key = NULL,
>             .rss_hf = ETH_RSS_IP,
>         },
>     },
>     .txmode = {
>         .mq_mode = ETH_MQ_TX_NONE,
>     },
> };
>
>
>
> 3. rx queue setup parameter
> ret = rte_eth_rx_queue_setup(port, RXQ_ID, NB_RXD, socket_id,  NULL,
> pktmbuf_pool[socket_id])
> where RXQ_ID = 0, NB_RXD = 128
>
>
>
> 4. config parameters in config/common_linuxapp
> #
> # Compile burst-oriented IXGBE PMD driver
> #
> CONFIG_RTE_LIBRTE_IXGBE_PMD=y
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_INIT=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX_FREE=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
> CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
> CONFIG_RTE_IXGBE_INC_VECTOR=y
> CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
>
>
>
> 5. where log message is printed
>
> dpdk-2.2.0/drivers/net/ixgbe/ixgbe_rxtx.c:
>
>         /* Allocate buffers for descriptor rings */
>         if (ixgbe_alloc_rx_queue_mbufs(rxq) != 0) {
>             PMD_INIT_LOG(ERR, "Could not alloc mbuf for queue:%d",
>                      rx_queue_id);
>             return -1;
>         }
>         rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx));
>         rxdctl |= IXGBE_RXDCTL_ENABLE;
>         IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(rxq->reg_idx), rxdctl);
>
>         /* Wait until RX Enable ready */
>         poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS;
>         do {
>             rte_delay_ms(1);
>             rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx));
>         } while (--poll_ms && !(rxdctl & IXGBE_RXDCTL_ENABLE));
>         if (!poll_ms)
>             PMD_INIT_LOG(ERR, "*Could not enable Rx Queue %d*",
>  rx_queue_id);
>
>
> I'm going to update firmware of my NIC, but I'm not sure it helps.
> I appreciate any comment.
>
>
>
> On Wed, Jan 27, 2016 at 4:23 PM, Zhang, Helin <helin.zhang at intel.com>
> wrote:
>
>> Moon-Sang
>>
>> Were you using pktgen or else application?
>> Could you help to share with me the detailed steps of your reproducing
>> that issue?
>> We will find time on that soon later. Thanks!
>>
>> Regards,
>> Helin
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Laurent GUERBY
>> Sent: Wednesday, January 27, 2016 3:16 PM
>> To: Moon-Sang Lee <sang0627 at gmail.com>
>> Cc: dev at dpdk.org
>> Subject: Re: [dpdk-dev] Errors Rx count increasing while pktgen doing
>> nothing on Intel 82598EB 10G
>>
>> On Wed, 2016-01-27 at 15:50 +0900, Moon-Sang Lee wrote:
>> >
>> >
>> > Laurent, have you resolved this problem?
>> > I'm using the same NIC as yours (i.e. Intel 82598EB 10G NIC) and faced
>> > the same problem as you.
>> > Here is parts of my log and it says that PMD cannot enable RX queue
>> > for my NIC.
>> > I'm using DPDK 2.2.0 and used 'null' for the 4th parameter in calling
>> > rte_eth_rx_queue_setup().
>> > (i.e. 'null' parameter provides the default rx_conf value.)
>>
>> Hi,
>>
>> I had to reuse my DPDK machines for another task, I will go back to it
>> after FOSDEM.
>>
>> The error you get is the same as mine.
>>
>> Sincerely,
>>
>> Laurent
>>
>> >
>> > Thanks.
>> >
>> >
>> >
>> >
>> >
>> > APP: initialising port 0 ...
>> > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5f27258040
>> > sw_sc_ring=0x7f5f27257b00 hw_ring=0x7f5f27258580 dma_addr=0x41f458580
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f27245940
>> > hw_ring=0x7f5f27247980 dma_addr=0x41f447980
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f272337c0
>> > hw_ring=0x7f5f27235800 dma_addr=0x41f435800
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX
>> > burst size no less than 4 (port=0).
>> > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
>> > APP: port 0 has started
>> > APP: port 0 has entered in promiscuous mode
>> > APP: port 0 initialization is done.
>> > KNI: pci: 09:00:00 8086:10c7
>> > APP: kni allocation is done for port 0.
>> > APP: initialising port 1 ...
>> > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5f27222dc0
>> > sw_sc_ring=0x7f5f27222880 hw_ring=0x7f5f27223300 dma_addr=0x41f423300
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f272106c0
>> > hw_ring=0x7f5f27212700 dma_addr=0x41f412700
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f271fe540
>> > hw_ring=0x7f5f27200580 dma_addr=0x41f400580
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX
>> > burst size no less than 4 (port=1).
>> > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
>> > APP: port 1 has started
>> > APP: port 1 has entered in promiscuous mode
>> > APP: port 1 initialization is done.
>> > KNI: pci: 0a:00:00 8086:10c7
>> > APP: kni allocation is done for port 1.
>> >
>> >
>> > checking link status
>> > .done
>> > Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed
>> > 10000 Mbps - full-duplex
>> >
>> >
>> >
>> > On Mon, Dec 28, 2015 at 5:28 AM, Wiles, Keith <keith.wiles at intel.com>
>> > wrote:
>> >         On 12/27/15, 2:09 PM, "Laurent GUERBY" <laurent at guerby.net>
>> >         wrote:
>> >
>> >         >On Sun, 2015-12-27 at 19:43 +0000, Wiles, Keith wrote:
>> >         >> On 12/27/15, 12:31 PM, "dev on behalf of Laurent GUERBY"
>> >         <dev-bounces at dpdk.org on behalf of laurent at guerby.net> wrote:
>> >         >>
>> >         >> >Hi,
>> >         >> >
>> >         >> >I reported today an issue when using Pktgen-DPDK:
>> >         >> >https://github.com/pktgen/Pktgen-DPDK/issues/52
>> >         >> >
>> >         >> >But I think it's more in DPDK than pktgen
>> >         >> >
>> >         >> >two identical machines with SFP+ DA cable between them
>> >         >> >DPDK 2.2.0 from tarball
>> >         >> >Pktgen-DPDK from git
>> >         >> >two identical machines:
>> >         >> >core i7 2600 (sandy bridge 4C/8T), HT disabled in the BIOS
>> >         >> >ASUS P8H67-M PRO BIOS 3904 (latest available)
>> >         >> >Ethernet controller: Intel Corporation 82598EB 10-Gigabit
>> >         AF Dual Port
>> >         >> >Network Connection (rev 01)
>> >         >> >01:00.0 0200: 8086:10f1 (rev 01)
>> >         >> >    Subsystem: 8086:a21f
>> >         >> >boot kernel 3.16 unbutu 14.04 with isolcpus=2,3,4
>> >         >> >
>> >         >> >When launching pktgen even with no TX asked the Errors RX
>> >         counters keeps
>> >         >> >going up by about 7.4 millions per second:
>> >         >> >
>> >         >> >Errors Rx/Tx : 7471857054/0
>> >         >> >
>> >         >> >In the log I get "Could not enable Rx Queue", might be the
>> >         >> >source of the issue?
>> >         >> >
>> >         >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue
>> >         0
>> >         >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue
>> >         1
>> >         >> >
>> >         >> >When sending traffic  single UDP src/dst/IP/MAC the setup
>> >         >> >reaches 14204188 pps 64 bytes, the error counter is also
>> >         >> >increasing.
>> >         >> >
>> >         >> >Any idea what to look for?
>> >         >>
>> >         >> One more suggestion is to run test_pmd on one machine and
>> >         something
>> >         >> like iperf on the other to verify the DPDK is working
>> >         correct, which I
>> >         >> assume will be true. Not sure the RX errors are reported in
>> >         the
>> >         >> test_pmd or you could use the l3fwd application too.
>> >         >
>> >         >Ok, I will check the test_pmd documentation and try to do
>> >         this test: I'm
>> >         >just starting on DPDK :).
>> >         >
>> >         >> Please also send me the 'lspci | grep Ethernet’ output.
>> >         >
>> >         >I included one line in my original email above (plus extract
>> >         of lspci
>> >         >-vn), here is the full output of the command:
>> >         >
>> >         >01:00.0 Ethernet controller: Intel Corporation 82598EB
>> >         10-Gigabit AF
>> >         >Dual Port Network Connection (rev 01)
>> >         >01:00.1 Ethernet controller: Intel Corporation 82598EB
>> >         10-Gigabit AF
>> >         >Dual Port Network Connection (rev 01)
>> >         >05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>> >         >RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>> >         (rev 06)
>> >         >
>> >         >(The realtek is used only for internet connectivity).
>> >         >
>> >         >> Also send me the command line.
>> >         >
>> >         >On the first machine t1 :
>> >         >root at t1:~/pktgen-dpdk#
>> ./app/app/x86_64-native-linuxapp-gcc/pktgen \
>> >         >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t1-t3.pkt -N
>> >         >
>> >         >And on the other machine t3:
>> >         >root at t3:~/pktgen-dpdk#
>> ./app/app/x86_64-native-linuxapp-gcc/pktgen \
>> >         >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t3-t1.pkt -N
>> >
>> >
>> >         You need to always start with port numbering for Pktgen with
>> >         zero.
>> >
>> >         Change the [2:3].1 to [2:3].0 the reason is you removed one of
>> >         the two ports and Pktgen starts port counting from zero for
>> >         the first available port :-)
>> >
>> >         Sorry, I did not spot that sooner.
>> >         >
>> >         >The two "-f" pkt files are attached to this email, I do
>> >         "start 1"
>> >         >manually at the pktgen prompt.
>> >         >
>> >         >Thanks for your time,
>> >         >
>> >         >Sincerely,
>> >         >
>> >         >Laurent
>> >         >
>> >         >
>> >         >> >
>> >         >> >Laurent
>> >         >> >
>> >         >> >
>> >         >>
>> >         >>
>> >         >> Regards,
>> >         >> Keith
>> >         >>
>> >         >>
>> >         >>
>> >         >>
>> >         >
>> >         >
>> >
>> >
>> >         Regards,
>> >         Keith
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Moon-Sang Lee, SW Engineer
>> > Email: sang0627 at gmail.com
>> > Wisdom begins in wonder. *Socrates*
>>
>>
>>
>
>
> --
> Moon-Sang Lee, SW Engineer
> Email: sang0627 at gmail.com
> Wisdom begins in wonder. *Socrates*
>



-- 
Moon-Sang Lee, SW Engineer
Email: sang0627 at gmail.com
Wisdom begins in wonder. *Socrates*


More information about the dev mailing list