[dpdk-dev] Errors Rx count increasing while pktgen doing nothing on Intel 82598EB 10G
Moon-Sang Lee
sang0627 at gmail.com
Fri Jan 29 11:02:45 CET 2016
Let me share one more information about this issue.
I tried to run dpdk-2.2.0/examples/ethtool, which showed the same problem.
(i.e. I guess other examples will also fail.)
[root at centos7 app]# ./ethtool
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 1 on socket 0
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 2 on socket 0
EAL: Detected lcore 5 as core 2 on socket 1
EAL: Detected lcore 6 as core 3 on socket 0
EAL: Detected lcore 7 as core 3 on socket 1
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 0 on socket 1
EAL: Detected lcore 10 as core 1 on socket 0
EAL: Detected lcore 11 as core 1 on socket 1
EAL: Detected lcore 12 as core 2 on socket 0
EAL: Detected lcore 13 as core 2 on socket 1
EAL: Detected lcore 14 as core 3 on socket 0
EAL: Detected lcore 15 as core 3 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1400000 bytes
EAL: Virtual area found at 0x7f79fa800000 (size = 0x1400000)
EAL: Ask a virtual area of 0x7ec00000 bytes
EAL: Virtual area found at 0x7f797ba00000 (size = 0x7ec00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f797b600000 (size = 0x200000)
EAL: Ask a virtual area of 0x7fc00000 bytes
EAL: Virtual area found at 0x7f78fb800000 (size = 0x7fc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f78fb400000 (size = 0x200000)
EAL: Ask a virtual area of 0x100000000 bytes
EAL: Virtual area found at 0x7f77fb200000 (size = 0x100000000)
EAL: Requesting 2048 pages of size 2MB from socket 0
EAL: Requesting 2048 pages of size 2MB from socket 1
EAL: TSC frequency is ~2260984 KHz
EAL: Master lcore 0 is ready (tid=fdc578c0;cpuset=[0])
EAL: lcore 6 is ready (tid=f81cb700;cpuset=[6])
EAL: lcore 4 is ready (tid=f91cd700;cpuset=[4])
EAL: lcore 12 is ready (tid=f51c5700;cpuset=[12])
EAL: lcore 9 is ready (tid=f69c8700;cpuset=[9])
EAL: lcore 8 is ready (tid=f71c9700;cpuset=[8])
EAL: lcore 14 is ready (tid=f41c3700;cpuset=[14])
EAL: lcore 3 is ready (tid=f99ce700;cpuset=[3])
EAL: lcore 13 is ready (tid=f49c4700;cpuset=[13])
EAL: lcore 7 is ready (tid=f79ca700;cpuset=[7])
EAL: lcore 10 is ready (tid=f61c7700;cpuset=[10])
EAL: lcore 11 is ready (tid=f59c6700;cpuset=[11])
EAL: lcore 15 is ready (tid=f39c2700;cpuset=[15])
EAL: lcore 5 is ready (tid=f89cc700;cpuset=[5])
EAL: lcore 2 is ready (tid=fa1cf700;cpuset=[2])
EAL: lcore 1 is ready (tid=fc562700;cpuset=[1])
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL: probe driver: 8086:10e8 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:06:00.1 on NUMA socket -1
EAL: probe driver: 8086:10e8 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:07:00.0 on NUMA socket -1
EAL: probe driver: 8086:10e8 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:07:00.1 on NUMA socket -1
EAL: probe driver: 8086:10e8 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:09:00.0 on NUMA socket -1
EAL: probe driver: 8086:10c7 rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f79fbc00000
EAL: PCI memory mapped at 0x7f79fbc20000
EAL: PCI memory mapped at 0x7f79fbc60000
PMD: eth_ixgbe_dev_init(): MAC: 1, PHY: 8
PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x10c7
EAL: PCI device 0000:0a:00.0 on NUMA socket -1
EAL: probe driver: 8086:10c7 rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f79fbc64000
EAL: PCI memory mapped at 0x7f79fbc84000
EAL: PCI memory mapped at 0x7f79fbcc4000
PMD: eth_ixgbe_dev_init(): MAC: 1, PHY: 8
PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10c7
Number of NICs: 2
Init port 0..
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f78f99d7140
sw_sc_ring=0x7f78f99d6c00 hw_ring=0x7f78f99d7680 dma_addr=0x4207d7680
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f78f99c5a40
hw_ring=0x7f78f99c6a80 dma_addr=0x4207c6a80
PMD: ixgbe_set_tx_function(): Using simple tx code path
PMD: ixgbe_set_tx_function(): Vector tx enabled.
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=0).
PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
Init port 1..
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f78f846ad40
sw_sc_ring=0x7f78f846a800 hw_ring=0x7f78f846b280 dma_addr=0x41f26b280
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f78f8459640
hw_ring=0x7f78f845a680 dma_addr=0x41f25a680
PMD: ixgbe_set_tx_function(): Using simple tx code path
PMD: ixgbe_set_tx_function(): Vector tx enabled.
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=1).
PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
EthApp> pause
[UINT16]: pause <port_id>
Print port pause state
[UINT16]: pause <port_id> <all|tx|rx|none>
Pause/unpause port
EthApp> pause 0
Port 0: Rx Paused
EthApp> pause 0 none
PMD: ixgbe_flow_ctrl_set(): Rx packet buffer size = 0x80000
Port 0: Tx & Rx not paused
EthApp> pause 0
Port 0: Rx Paused
EthApp> quit
[root at centos7 app]# pwd
On Thu, Jan 28, 2016 at 9:57 AM, Moon-Sang Lee <sang0627 at gmail.com> wrote:
>
> Helin, I implemented my own sample application that is a kind of carrier
> grade NAT server.
> It works fine on 1G NIC (i.e. Intel Corporation 82576 Gigabit Network
> Connection (rev 01))
> But, it does not receive packets on 10G NIC (i.e. Intel Corporation
> 82598EB 10-Gigabit AF Network Connection (rev 01)) as described in the
> previous email.
> According to my log messages, it seems that control register for RX DMA is
> not enabled.
>
> Here is some information about my environment.
>
> 1. HW & OS
> [mslee at centos7 ~]$ uname -a
> Linux centos7 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
> [mslee at centos7 ~]$ more /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 26
> model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
> stepping : 5
> microcode : 0x19
> cpu MHz : 2262.000
> cache size : 8192 KB
> physical id : 1
> siblings : 8
> core id : 0
> cpu cores : 4
> apicid : 16
> initial apicid : 16
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> rdtscp lm c
> onstant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
> aperfmperf
> pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1
> sse4_2 po
> pcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
> bogomips : 4521.93
> clflush size : 64
> cache_alignment : 64
> address sizes : 40 bits physical, 48 bits virtual
> power management:
> ...
>
>
>
> 2. port ocnfigure parameter for rte_eth_dev_configure():
> ret = rte_eth_dev_configure(port, NB_RXQ, NB_TXQ, &port_conf);
> where NB_RXQ=1, NB_TXQ=2, and
> struct rte_eth_conf port_conf = {
> .rxmode = {
> //.mq_mode = ETH_MQ_RX_RSS,
> .mq_mode = ETH_MQ_RX_NONE, // for 10G NIC
> .max_rx_pkt_len = ETHER_MAX_LEN,
> .split_hdr_size = 0,
> .header_split = 0, // Header Split disabled
> .hw_ip_checksum = 0, // IP checksum offload enabled
> .hw_vlan_filter = 0, // VLAN filtering disabled
> .jumbo_frame = 0, // Jumbo Frame Support disabled
> .hw_strip_crc = 0, // CRC stripped by hardware
> },
> .rx_adv_conf = {
> .rss_conf = {
> .rss_key = NULL,
> .rss_hf = ETH_RSS_IP,
> },
> },
> .txmode = {
> .mq_mode = ETH_MQ_TX_NONE,
> },
> };
>
>
>
> 3. rx queue setup parameter
> ret = rte_eth_rx_queue_setup(port, RXQ_ID, NB_RXD, socket_id, NULL,
> pktmbuf_pool[socket_id])
> where RXQ_ID = 0, NB_RXD = 128
>
>
>
> 4. config parameters in config/common_linuxapp
> #
> # Compile burst-oriented IXGBE PMD driver
> #
> CONFIG_RTE_LIBRTE_IXGBE_PMD=y
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_INIT=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX_FREE=n
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
> CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
> CONFIG_RTE_IXGBE_INC_VECTOR=y
> CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
>
>
>
> 5. where log message is printed
>
> dpdk-2.2.0/drivers/net/ixgbe/ixgbe_rxtx.c:
>
> /* Allocate buffers for descriptor rings */
> if (ixgbe_alloc_rx_queue_mbufs(rxq) != 0) {
> PMD_INIT_LOG(ERR, "Could not alloc mbuf for queue:%d",
> rx_queue_id);
> return -1;
> }
> rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx));
> rxdctl |= IXGBE_RXDCTL_ENABLE;
> IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(rxq->reg_idx), rxdctl);
>
> /* Wait until RX Enable ready */
> poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS;
> do {
> rte_delay_ms(1);
> rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx));
> } while (--poll_ms && !(rxdctl & IXGBE_RXDCTL_ENABLE));
> if (!poll_ms)
> PMD_INIT_LOG(ERR, "*Could not enable Rx Queue %d*",
> rx_queue_id);
>
>
> I'm going to update firmware of my NIC, but I'm not sure it helps.
> I appreciate any comment.
>
>
>
> On Wed, Jan 27, 2016 at 4:23 PM, Zhang, Helin <helin.zhang at intel.com>
> wrote:
>
>> Moon-Sang
>>
>> Were you using pktgen or else application?
>> Could you help to share with me the detailed steps of your reproducing
>> that issue?
>> We will find time on that soon later. Thanks!
>>
>> Regards,
>> Helin
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Laurent GUERBY
>> Sent: Wednesday, January 27, 2016 3:16 PM
>> To: Moon-Sang Lee <sang0627 at gmail.com>
>> Cc: dev at dpdk.org
>> Subject: Re: [dpdk-dev] Errors Rx count increasing while pktgen doing
>> nothing on Intel 82598EB 10G
>>
>> On Wed, 2016-01-27 at 15:50 +0900, Moon-Sang Lee wrote:
>> >
>> >
>> > Laurent, have you resolved this problem?
>> > I'm using the same NIC as yours (i.e. Intel 82598EB 10G NIC) and faced
>> > the same problem as you.
>> > Here is parts of my log and it says that PMD cannot enable RX queue
>> > for my NIC.
>> > I'm using DPDK 2.2.0 and used 'null' for the 4th parameter in calling
>> > rte_eth_rx_queue_setup().
>> > (i.e. 'null' parameter provides the default rx_conf value.)
>>
>> Hi,
>>
>> I had to reuse my DPDK machines for another task, I will go back to it
>> after FOSDEM.
>>
>> The error you get is the same as mine.
>>
>> Sincerely,
>>
>> Laurent
>>
>> >
>> > Thanks.
>> >
>> >
>> >
>> >
>> >
>> > APP: initialising port 0 ...
>> > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5f27258040
>> > sw_sc_ring=0x7f5f27257b00 hw_ring=0x7f5f27258580 dma_addr=0x41f458580
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f27245940
>> > hw_ring=0x7f5f27247980 dma_addr=0x41f447980
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f272337c0
>> > hw_ring=0x7f5f27235800 dma_addr=0x41f435800
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX
>> > burst size no less than 4 (port=0).
>> > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
>> > APP: port 0 has started
>> > APP: port 0 has entered in promiscuous mode
>> > APP: port 0 initialization is done.
>> > KNI: pci: 09:00:00 8086:10c7
>> > APP: kni allocation is done for port 0.
>> > APP: initialising port 1 ...
>> > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5f27222dc0
>> > sw_sc_ring=0x7f5f27222880 hw_ring=0x7f5f27223300 dma_addr=0x41f423300
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f272106c0
>> > hw_ring=0x7f5f27212700 dma_addr=0x41f412700
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5f271fe540
>> > hw_ring=0x7f5f27200580 dma_addr=0x41f400580
>> > PMD: ixgbe_set_tx_function(): Using simple tx code path
>> > PMD: ixgbe_set_tx_function(): Vector tx enabled.
>> > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX
>> > burst size no less than 4 (port=1).
>> > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0
>> > APP: port 1 has started
>> > APP: port 1 has entered in promiscuous mode
>> > APP: port 1 initialization is done.
>> > KNI: pci: 0a:00:00 8086:10c7
>> > APP: kni allocation is done for port 1.
>> >
>> >
>> > checking link status
>> > .done
>> > Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed
>> > 10000 Mbps - full-duplex
>> >
>> >
>> >
>> > On Mon, Dec 28, 2015 at 5:28 AM, Wiles, Keith <keith.wiles at intel.com>
>> > wrote:
>> > On 12/27/15, 2:09 PM, "Laurent GUERBY" <laurent at guerby.net>
>> > wrote:
>> >
>> > >On Sun, 2015-12-27 at 19:43 +0000, Wiles, Keith wrote:
>> > >> On 12/27/15, 12:31 PM, "dev on behalf of Laurent GUERBY"
>> > <dev-bounces at dpdk.org on behalf of laurent at guerby.net> wrote:
>> > >>
>> > >> >Hi,
>> > >> >
>> > >> >I reported today an issue when using Pktgen-DPDK:
>> > >> >https://github.com/pktgen/Pktgen-DPDK/issues/52
>> > >> >
>> > >> >But I think it's more in DPDK than pktgen
>> > >> >
>> > >> >two identical machines with SFP+ DA cable between them
>> > >> >DPDK 2.2.0 from tarball
>> > >> >Pktgen-DPDK from git
>> > >> >two identical machines:
>> > >> >core i7 2600 (sandy bridge 4C/8T), HT disabled in the BIOS
>> > >> >ASUS P8H67-M PRO BIOS 3904 (latest available)
>> > >> >Ethernet controller: Intel Corporation 82598EB 10-Gigabit
>> > AF Dual Port
>> > >> >Network Connection (rev 01)
>> > >> >01:00.0 0200: 8086:10f1 (rev 01)
>> > >> > Subsystem: 8086:a21f
>> > >> >boot kernel 3.16 unbutu 14.04 with isolcpus=2,3,4
>> > >> >
>> > >> >When launching pktgen even with no TX asked the Errors RX
>> > counters keeps
>> > >> >going up by about 7.4 millions per second:
>> > >> >
>> > >> >Errors Rx/Tx : 7471857054/0
>> > >> >
>> > >> >In the log I get "Could not enable Rx Queue", might be the
>> > >> >source of the issue?
>> > >> >
>> > >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue
>> > 0
>> > >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue
>> > 1
>> > >> >
>> > >> >When sending traffic single UDP src/dst/IP/MAC the setup
>> > >> >reaches 14204188 pps 64 bytes, the error counter is also
>> > >> >increasing.
>> > >> >
>> > >> >Any idea what to look for?
>> > >>
>> > >> One more suggestion is to run test_pmd on one machine and
>> > something
>> > >> like iperf on the other to verify the DPDK is working
>> > correct, which I
>> > >> assume will be true. Not sure the RX errors are reported in
>> > the
>> > >> test_pmd or you could use the l3fwd application too.
>> > >
>> > >Ok, I will check the test_pmd documentation and try to do
>> > this test: I'm
>> > >just starting on DPDK :).
>> > >
>> > >> Please also send me the 'lspci | grep Ethernet’ output.
>> > >
>> > >I included one line in my original email above (plus extract
>> > of lspci
>> > >-vn), here is the full output of the command:
>> > >
>> > >01:00.0 Ethernet controller: Intel Corporation 82598EB
>> > 10-Gigabit AF
>> > >Dual Port Network Connection (rev 01)
>> > >01:00.1 Ethernet controller: Intel Corporation 82598EB
>> > 10-Gigabit AF
>> > >Dual Port Network Connection (rev 01)
>> > >05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>> > >RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>> > (rev 06)
>> > >
>> > >(The realtek is used only for internet connectivity).
>> > >
>> > >> Also send me the command line.
>> > >
>> > >On the first machine t1 :
>> > >root at t1:~/pktgen-dpdk#
>> ./app/app/x86_64-native-linuxapp-gcc/pktgen \
>> > >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t1-t3.pkt -N
>> > >
>> > >And on the other machine t3:
>> > >root at t3:~/pktgen-dpdk#
>> ./app/app/x86_64-native-linuxapp-gcc/pktgen \
>> > >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t3-t1.pkt -N
>> >
>> >
>> > You need to always start with port numbering for Pktgen with
>> > zero.
>> >
>> > Change the [2:3].1 to [2:3].0 the reason is you removed one of
>> > the two ports and Pktgen starts port counting from zero for
>> > the first available port :-)
>> >
>> > Sorry, I did not spot that sooner.
>> > >
>> > >The two "-f" pkt files are attached to this email, I do
>> > "start 1"
>> > >manually at the pktgen prompt.
>> > >
>> > >Thanks for your time,
>> > >
>> > >Sincerely,
>> > >
>> > >Laurent
>> > >
>> > >
>> > >> >
>> > >> >Laurent
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >> Regards,
>> > >> Keith
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> > >
>> >
>> >
>> > Regards,
>> > Keith
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Moon-Sang Lee, SW Engineer
>> > Email: sang0627 at gmail.com
>> > Wisdom begins in wonder. *Socrates*
>>
>>
>>
>
>
> --
> Moon-Sang Lee, SW Engineer
> Email: sang0627 at gmail.com
> Wisdom begins in wonder. *Socrates*
>
--
Moon-Sang Lee, SW Engineer
Email: sang0627 at gmail.com
Wisdom begins in wonder. *Socrates*
More information about the dev
mailing list