[dpdk-dev] about poor KNI performance
Moon-Sang Lee
sang0627 at gmail.com
Tue Sep 22 07:36:21 CEST 2015
I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3,
are directly connected to M2 as below. All interfaces are 1G NICs over 1G
links.
[ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ]
Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and
545Mbps for TCP.
With KNI, it drops almost every packet for UDP (i.e. 87% of received
packets are dropped),
and shows 42Mbps for TCP.
If I use KNI with DPDK, it runs in poll-mode thus there should not be
interrupts.
In my experiment, something strange is ksoftirqd consumes 50% CPU loads
with KNI.
Why ksoftirqd works such hard?
I'm using the KNI application in the example directory of DPDK 2.1.0 source
tree with options
-c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)".
(i.e. rte_kni.ko is loaded with kthread_mode=multiple option)
Any comments is appreciated.
##### without KNI (i.e. with native iptables) #####
[mslee at localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48099
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 376 MBytes 315 Mbits/sec 0.002 ms
471622/3548257 (13%)
[ 5] 10.00-10.04 sec 1.53 MBytes 326 Mbits/sec 0.003 ms 1520/14071
(11%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.04 sec 435 MBytes 363 Mbits/sec 0.003 ms
473142/3562328 (13%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48100
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 650 MBytes 545 Mbits/sec
[ 5] 10.00-10.04 sec 2.53 MBytes 563 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 653 MBytes 546 Mbits/sec 97 sender
[ 5] 0.00-10.04 sec 652 MBytes 545 Mbits/sec
receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
##### with KNI (i.e. with KNI + native iptables) #####
[mslee at localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48102
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 57.9 MBytes 48.6 Mbits/sec 0.021 ms
3239121/3713340 (87%)
[ 5] 10.00-10.27 sec 278 KBytes 8.37 Mbits/sec 0.018 ms 16319/18541
(88%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.27 sec 456 MBytes 372 Mbits/sec 0.018 ms
3255440/3731881 (87%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48103
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 50.6 MBytes 42.4 Mbits/sec
[ 5] 10.00-10.04 sec 204 KBytes 44.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 52.0 MBytes 43.4 Mbits/sec 151 sender
[ 5] 0.00-10.04 sec 50.8 MBytes 42.4 Mbits/sec
receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
##### snapshot of top command #####
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9519 root 20 0 8659492 20404 19948 R 400.0 0.1 10:41.09
unatd-kni
114 root 20 0 0 0 0 S 45.7 0.0
1:31.46 ksoftirqd/8
9525 root 20 0 0 0 0 S 17.2 0.0
0:07.89 kni_p1p3
9590 root 20 0 0 0 0 S 1.2 0.0
0:01.57 kni_p1p4
--
Moon-Sang Lee, SW Engineer
Email: sang0627 at gmail.com
Wisdom begins in wonder. *Socrates*
More information about the dev
mailing list