[dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

[Shijith Thotton]

On Thu, Sep 14, 2017 at 12:30:55PM +0530, Shijith Thotton wrote:
> On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote:
> > > -----Original Message-----
> > > From: Yigit, Ferruh
> > > Sent: Thursday, September 14, 2017 1:06 AM
> > > To: Hu, Xuekun <xuekun.hu at intel.com>; Shijith Thotton
> > > <shijith.thotton at caviumnetworks.com>; Yang, Qiming
> > > <qiming.yang at intel.com>; Gregory Etelson <gregory at weka.io>
> > > Cc: dev at dpdk.org; Tan, Jianfeng <jianfeng.tan at intel.com>
> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > > and release of device file
> > > 
> > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
> > > > I met the same issue too, only with i40e 2.1.26 PF kernel driver.
> > > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Ferruh Yigit
> > > > Sent: Wednesday, September 13, 2017 7:04 PM
> > > > To: Shijith Thotton <shijith.thotton at caviumnetworks.com>; Yang, Qiming
> > > > <qiming.yang at intel.com>; Gregory Etelson <gregory at weka.io>
> > > > Cc: dev at dpdk.org; Tan, Jianfeng <jianfeng.tan at intel.com>
> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
> > > > during open and release of device file
> > > >
> > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote:
> > > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
> > > >>>    Hi, Shijith
> > > >>>
> > > >>>
> > > >>>
> > > >>>    VF init error will happen after apply your patch, error log as below. If
> > > >>>    revert your commit, all things work well. And this issue is not only occur
> > > >>>    in i40 VF but also ixgbe.
> > > 
> > > Hi Qiming,
> > > 
> > > I can reproduce the issue for the case:
> > > 
> > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
> > > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and
> > > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest,
> > > this works fine)
> > > 
> > > And I confirm the function igbuio_pci_open() added with below patch is causing
> > > it.
> > > 
> > Hi, Shijith
> > Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs.
> 
> Reset was added in open to make the device ready for init. I'm not sure about
> the exact cause of this issue and was unable to reproduce the same with LiquidIO
> VFs.
> 
> Reset happens during open and if VFs are not ready by the time init happens in
> PMD, it can fail. Issue could be in firmware as we don't see it for all versions.
> If there are no other way to fix this, I think, it is best to remove reset
> during open as it breaks PMD from multiple vendors.
> 
> Please try removing igbuio_pci_open.

Sorry. Please try removing pci_reset_function from igbuio_pci_open.

Thanks,
Shijith

> > > igbuio_pci_open() does:
> > >     pci_reset_function(dev);
> > >     pci_set_master(dev);
> > > 
> > > Don't know yet root cause of the error, will dig more.
> > > 
> > > >>>
> > > >>>    Could you help to check it soon?
> > > >>>
> > > >>>
> > > >>>
> > > >>>    [root at localhost app]# ./testpmd -c 7 -n 4 -- -i
> > > >>>
> > > >>>    EAL: Detected 10 lcore(s)
> > > >>>
> > > >>>    EAL: No free hugepages reported in hugepages-1048576kB
> > > >>>
> > > >>>    EAL: Probing VFIO support...
> > > >>>
> > > >>>    EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > > >>>    unreliable clock cycles !
> > > >>>
> > > >>>    EAL: PCI device 0000:00:03.0 on NUMA socket -1
> > > >>>
> > > >>>    EAL:   Invalid NUMA socket, default to 0
> > > >>>
> > > >>>    EAL:   probe driver: 8086:154c net_i40e_vf
> > > >>>
> > > >>>    i40evf_init_vf(): init_adminq failed: -53
> > > >>>
> > > >>>    i40evf_dev_init(): Init vf failed
> > > >>>
> > > >>>    EAL: Requested device 0000:00:03.0 cannot be used
> > > >>>
> > > >>>
> > > >>>
> > > >>>    commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
> > > >>>    Author: Shijith Thotton <shijith.thotton at caviumnetworks.com>
> > > >>>    Date: Fri Jul 7 16:43:51 2017 +0530
> > > >>>
> > > >>>    igb_uio: issue FLR during open and release of device file
> > > >>>
> > > >>>    Set UIO info device file operations open and release. Call pci reset
> > > >>>    function inside open and release to clear device state at start and end.
> > > >>>    Copied this behaviour from vfio_pci kernel module code. With this patch,
> > > >>>    it is not mandatory to issue FLR by PMD's during init and close.
> > > >>>
> > > >>>    Bus master enable and disable are added in open and release respectively
> > > >>>    to take care of device DMA.
> > > >>>
> > > >>>    Signed-off-by: Shijith Thotton <shijith.thotton at caviumnetworks.com>
> > > >>>    Reviewed-by: Jianfeng Tan <jianfeng.tan at intel.com>
> > > >>>    Acked-by: Ferruh Yigit <ferruh.yigit at intel.com>
> > > >>>    Acked-by: Gregory Etelson <gregory at weka.io>
> > > >>>
> > > >>>
> > > >>>
> > > >>>    Best Regard,
> > > >>>
> > > >>>    Yang Qiming
> > > >>>
> > > >>>
> > > >>
> > > >> Hi Yang Qiming,
> > > >>
> > > >> We didn't face this issue while preparing patch. I think, Gregory
> > > >> tested it for Intel NICs. Adding more people to the loop for help.
> > > >>
> > > >
> > > > I will try to reproduce the issue.
> > > >
> > > >> Shijith
> > > >>
> > > >
> >