[dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[junlee]

Hi,

We fixed like below: use cond_schedule() to check whether there are other
tasks (watchdog, rcu, etc) to run, if no, continue to handle packets.
This can make CPU busy and performance is much better.
Do you think this can be submitted?

kni_thread_single(void *data)
{
struct kni_net *knet = data;
int j;
struct kni_dev *dev;

while (!kthread_should_stop()) {
down_read(&knet->kni_list_lock);
for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
list_for_each_entry(dev, &knet->kni_list_head, list) {
kni_net_rx(dev);
kni_net_poll_resp(dev);
}
}
up_read(&knet->kni_list_lock);
#ifdef RTE_KNI_PREEMPT_DEFAULT
/* reschedule out for a while */
schedule_timeout_interruptible(
usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
#else
                /* schedule only if needed*/
cond_schedule();
#endif
}

return 0;
}

Thanks.
BR//Wendy

2017-11-28 10:24 GMT+08:00 junlee <lijun422 at gmail.com>:

> We also tried to turn off the watchdog, then we got RCU stall.
> INFO: rcu_sched self-detected stall on CPU
>
> Is there any system configuration that should be made before setting
> RTE_KNI_PREEMPT_DEFAULT to n?
>
>
> 2017-11-27 20:49 GMT+08:00 junlee <lijun422 at gmail.com>:
>
>> Hi Jay,
>>
>> Thanks.
>> We have read the thread before, increasing HZ to 1000 helps, but we are
>> looking for latency even less than 1ms.
>> The DPDK version is 17.05.
>>
>> The kernel preempt setting is:
>> CONFIG_PREEMPT_NONE=y
>> # CONFIG_PREEMPT_VOLUNTARY is not set
>> # CONFIG_PREEMPT is not set
>>
>> Seems the watchdog/x process could not get chance to run
>> when schedule_timeout_interruptible() is not called.
>>
>> Thanks.
>> BR//Wendy
>>
>> 2017-11-27 20:36 GMT+08:00 Jay Rolette <rolette at infinite.io>:
>>
>>> That shouldn't cause soft-lockups, but IIRC, there have been KNI bugs in
>>> that past that would. Probably worth checking whatever version of DPDK you
>>> are running vs. the tree for relevant fixes.
>>>
>>> Given what you are looking into, this thread may be of interest:
>>>
>>> http://dpdk.org/ml/archives/dev/2015-June/018858.html
>>>
>>> Jay
>>>
>>> On Sun, Nov 26, 2017 at 7:27 PM, junlee <lijun422 at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are testing KNI performance and finds the
>>>> schedule_timeout_interruptible
>>>> causes extra latency for packets.
>>>>
>>>> We tried to set RTE_KNI_PREEMPT_DEFAULT=n to reduce the latency, but
>>>> ends
>>>> up with CPU soft-lockup.
>>>>
>>>> Anybody knows how to make this work?
>>>>
>>>> static int
>>>> kni_thread_single(void *data)
>>>> {
>>>> struct kni_net *knet = data;
>>>> int j;
>>>> struct kni_dev *dev;
>>>>
>>>> while (!kthread_should_stop()) {
>>>> down_read(&knet->kni_list_lock);
>>>> for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
>>>> list_for_each_entry(dev, &knet->kni_list_head, list) {
>>>> kni_net_rx(dev);
>>>> kni_net_poll_resp(dev);
>>>> }
>>>> }
>>>> up_read(&knet->kni_list_lock);
>>>> #ifdef RTE_KNI_PREEMPT_DEFAULT
>>>> /* reschedule out for a while */
>>>> schedule_timeout_interruptible(
>>>> usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
>>>> #endif
>>>> }
>>>>
>>>> return 0;
>>>> }
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>>
>