[dpdk-dev] about poor KNI performance

Moon-Sang Lee sang0627 at gmail.com
Tue Sep 22 07:36:21 CEST 2015


I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3,
are directly connected to M2 as below. All interfaces are 1G NICs over 1G
links.

[ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ]

Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and
545Mbps for TCP.
With KNI, it drops almost every packet for UDP (i.e. 87% of received
packets are dropped),
and shows 42Mbps for TCP.

If I use KNI with DPDK, it runs in poll-mode thus there should not be
interrupts.
In my experiment,  something strange is ksoftirqd consumes 50% CPU loads
with KNI.
Why ksoftirqd works such hard?

I'm using the KNI application in the example directory of DPDK 2.1.0 source
tree with options
-c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)".
(i.e. rte_kni.ko is loaded with kthread_mode=multiple option)

Any comments is appreciated.



##### without KNI (i.e. with native iptables) #####

[mslee at localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48099
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.00  sec   376 MBytes   315 Mbits/sec  0.002 ms
 471622/3548257 (13%)
[  5]  10.00-10.04  sec  1.53 MBytes   326 Mbits/sec  0.003 ms  1520/14071
(11%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.04  sec   435 MBytes   363 Mbits/sec  0.003 ms
 473142/3562328 (13%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48100
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec   650 MBytes   545 Mbits/sec
[  5]  10.00-10.04  sec  2.53 MBytes   563 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec   653 MBytes   546 Mbits/sec   97             sender
[  5]   0.00-10.04  sec   652 MBytes   545 Mbits/sec
 receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------





##### with KNI (i.e. with KNI + native iptables) #####

[mslee at localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48102
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.00  sec  57.9 MBytes  48.6 Mbits/sec  0.021 ms
 3239121/3713340 (87%)
[  5]  10.00-10.27  sec   278 KBytes  8.37 Mbits/sec  0.018 ms  16319/18541
(88%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.27  sec   456 MBytes   372 Mbits/sec  0.018 ms
 3255440/3731881 (87%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48103
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  50.6 MBytes  42.4 Mbits/sec
[  5]  10.00-10.04  sec   204 KBytes  44.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  52.0 MBytes  43.4 Mbits/sec  151             sender
[  5]   0.00-10.04  sec  50.8 MBytes  42.4 Mbits/sec
 receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------




##### snapshot of top command #####
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 9519 root      20   0 8659492  20404  19948 R 400.0  0.1  10:41.09
unatd-kni
  114 root       20   0            0           0      0 S       45.7  0.0
1:31.46 ksoftirqd/8
 9525 root      20   0            0           0      0 S       17.2  0.0
0:07.89 kni_p1p3
 9590 root      20   0            0           0      0 S         1.2  0.0
0:01.57 kni_p1p4



-- 
Moon-Sang Lee, SW Engineer
Email: sang0627 at gmail.com
Wisdom begins in wonder. *Socrates*


More information about the dev mailing list