[dpdk-users] Using PF bound to IGB_UIO, together with sr-iov VFs passed to guests

[ran lifshitz]

Yeah I assume something is going wrong with libvirt logic, direct qemu
command is probably working.
>From the logs it might be related to udev interface which does not detect
properly the vf when libvirt tries to discover the interface .
Can you repeat your scenario, but with libvirt ?


No specific kernel logs from kernel.


here is libvirt debug info:

2000-01-18 00:10:51.573+0000: 2055: info : virEventPollRunOnce:641 :
EVENT_POLL_RUN: nhandles=8 timeout=4979
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollRunOnce:651 : Poll
got 1 event(s)
2000-01-18 00:10:51.574+0000: 2055: debug :
virEventPollDispatchTimeouts:433 : Dispatch 2
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:479
: Dispatch 8
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=0 w=1
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=1 w=2
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=2 w=3
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3 w=4
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=4 w=5
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=5 w=6
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=6 w=7
2000-01-18 00:10:51.574+0000: 2055: info : virEventPollDispatchHandles:507
: EVENT_POLL_DISPATCH_HANDLE: watch=7 events=1
2000-01-18 00:10:51.574+0000: 2055: debug : udevEventHandleCallback:1543 :
udev action: 'add'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:124 :
udev reports device '35' does not have property 'DRIVER'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:139 :
Found property key 'SUBSYSTEM' value 'vfio' for device with sysname '35'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceType:1245 : Could
not determine device type for device with sysfs name '35'
2000-01-18 00:10:51.574+0000: 2055: debug : udevAddOneDevice:1413 :
Discarding device -1 0x5641bbfd2570 /sys/devices/virtual/vfio/35

and later i see:


2000-01-18 00:10:51.575+0000: 2055: info : virEventPollDispatchHandles:507
: EVENT_POLL_DISPATCH_HANDLE: watch=7 events=1
2000-01-18 00:10:51.575+0000: 2055: debug : udevEventHandleCallback:1543 :
udev action: 'remove'
2000-01-18 00:10:51.575+0000: 2055: debug : udevRemoveOneDevice:1314 :
Failed to find device to remove that has udev name
'/sys/devices/virtual/vfio/35'
2000-01-18 00:10:51.575+0000: 2055: debug : virEventPollDispatchHandles:493
: i=7 w=31


which causes the vm bootup to fail:
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerProgramSendError:153 : prog=536903814 ver=1 proc=10 type=1
serial=5 msg=0x5641bbfd3b90 rerr=0x7f49bc0b7db0
2000-01-18 00:10:51.576+0000: 2057: debug : virNetMessageEncodePayload:376
: Encode length as 120
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientSendMessageLocked:1397 : msg=0x5641bbfd3b90 proc=10
len=120 offset=0
2000-01-18 00:10:51.576+0000: 2057: info :
virNetServerClientSendMessageLocked:1405 : RPC_SERVER_CLIENT_MSG_TX_QUEUE:
client=0x5641bbfd59b0 len=120 prog=536903814 vers=1 proc=10 type=1 status=1
serial=5
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientCalculateHandleMode:157 : tls=(nil) hs=-1,
rx=0x5641bbfddc30 tx=0x5641bbfd3b90
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientCalculateHandleMode:192 : mode=3



the xml file:
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
virsh edit vm64efi
or other application using the libvirt API.
-->
<!--  domain type='qemu'  -->
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
<name>VoidVNF-SRIOV</name>
<uuid>05ed74d9-e31f-80c2-fb69-6cb052abdb6b</uuid>
<memory unit="M">3072</memory>
<currentMemory unit="M">3072</currentMemory>
<qemu:commandline>
<qemu:arg value="-object"/>
<qemu:arg
value="memory-backend-file,id=mem,size=3072M,mem-path=/mnt/hugepages,share=on"/>
<qemu:arg value="-numa"/>
<qemu:arg value="node,memdev=mem"/>
<qemu:arg value="-mem-prealloc"/>
</qemu:commandline>
<vcpu placement="static">2</vcpu>
<os>
<type>hvm</type>
<loader>/mnt/vm_snapshot/OVMF.fd</loader>
<boot dev="hd"/>
</os>

<qemu:commandline>
<qemu:arg value='-cpu'/>
<qemu:arg value='host'/>
</qemu:commandline>

<vcpu placement='static' cpuset='0'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='12'/>
<vcpupin vcpu='1' cpuset='13'/>
</cputune>

<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset="utc"/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<!-- on_crash>restart</on_crash -->
<devices>
<!--  emulator>/usr/bin/kvm-spice</emulator  -->
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/mnt/vm_snapshot/MiniVoidVNF.qcow2"/>
<!--  target dev='sda' bus='sata'/  -->
<target dev="hda" bus="ide"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
</disk>
<controller type="usb" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
</controller>
<controller type="pci" index="0" model="pci-root"/>


<interface type="hostdev" managed="yes">
<driver name='vfio'/>
<mac address="52:54:12:34:56:78"/>
<source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x10" function="0x0"/>
</source>
<parameters interfaceid="0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/>
<vlan>
<tag id="30"/>
</vlan>
</interface>
<interface type="hostdev" managed="yes">
<driver name='vfio'/>
<mac address="52:54:12:34:56:79"/>
<source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x10" function="0x2"/>
</source>
<parameters interfaceid="0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/>
<vlan>
<tag id="40"/>
</vlan>
</interface>

<!--
interface type='bridge'>
<mac address='52:54:00:85:0e:d0'/>
<source bridge='lxcbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface
-->
<!--
interface type='bridge'>
<mac address='52:54:00:00:a7:e0'/>
<source bridge='lxcbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</interface
-->
<!--interface type="vhostuser">
<mac address="fa:16:3e:07:88:13"/>
<source type="unix" path="/usr/local/var/run/openvswitch/vhost-0ca8d512-ae"
mode="client"/>
<target dev="vhost-0ca8d512-ae"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
<parameters interfaceid="0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/>
</interface>
<interface type="vhostuser">
<mac address="fa:16:3e:07:88:14"/>
<source type="unix" path="/usr/local/var/run/openvswitch/vhost-0ca8d512-cc"
mode="client"/>
<target dev="vhost-0ca8d512-cc"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
<parameters interfaceid="0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/>
</interface-->
<serial type="pty">
<target port="0"/>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<input type="tablet" bus="usb"/>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="vnc" port="-1" autoport="yes" keymap="en-us"/>
<video>
<model type="cirrus" vram="9216" heads="1"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
</video>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</memballoon>
</devices>
</domain>



On Wed, Jul 12, 2017 at 2:43 PM, Rosen, Rami <rami.rosen at intel.com> wrote:

> Hi Ran,
>
>
>
> >echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
>
> >I'm getting 63 pfs in lspci,
>
>
>
> You mean VFs and not pfs, I assume.
>
>
>
> I tried something similar but with qemu cli, not virsh, using PCU
> passtrhough
>
> (-device pci-assign…) and i40e, but using igb_uio. I tried it with one VF
> and it worked OK.
>
>
>
> Do you see any errors in the kernel log (dmesg)?
>
>
>
>
>
> Regards,
>
> Rami Rosen
>
>
>
> *From:* ran lifshitz [mailto:ranlfg at gmail.com]
> *Sent:* Wednesday, July 12, 2017 14:34
> *To:* Rosen, Rami <rami.rosen at intel.com>
> *Cc:* users at dpdk.org
> *Subject:* Re: [dpdk-users] Using PF bound to IGB_UIO, together with
> sr-iov VFs passed to guests
>
>
>
> Hi Rami,
>
>
>
> I'm creating the vfs, while the pf is bound to igb_uio:
>
> echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
>
>
>
> I'm getting 63 pfs in lspci, but when i try to load the vm:
>
>
>
> virsh create mini1.xml
>
> error: Failed to create domain from mini1.xml
>
> error: An error occurred, but the cause is unknown
>
>
>
>
>
> in the dpdk manual, it's refering to qemu, so I probably assume that the
> libvirt try to initialize the VF mac and vlan, but this is not supported by
> the igb_uio ?
>
> it works well when ixgbe.ko is bound.
>
>
>
> On Wed, Jul 12, 2017 at 1:11 PM, Rosen, Rami <rami.rosen at intel.com> wrote:
>
> Hi, Ran,
>
> What is the internal error you get when bringing up the
> guest ?
>
> Regards,
> Rami Rosen
>
> -----Original Message-----
> From: users [mailto:users-bounces at dpdk.org] On Behalf Of ran lifshitz
> Sent: Tuesday, July 11, 2017 12:24
> To: users at dpdk.org
> Subject: [dpdk-users] Using PF bound to IGB_UIO, together with sr-iov VFs
> passed to guests
>
> Hi,
>
> I'm having trouble trying to build a scenario were the host has the PF
> bound to igb_uio, and working with dpdk application, together with sr-iov
> VFs passthrough to guest machines.
>
> Libvirt is not able to bring up the guest, and exit with internal error.
> I assume that the vlan/mac initialization which is working well, when the
> PF is assigned to ixgbe.k does not work when bound to igb_uio.
>
> Is this an igb_uio issue or libvirt ?
> is there a workaround for this ?
>
> Regards
> Ran
>
>
>
>
>
> --
>
>
>
> [image: Google+]
> <https://plus.google.com/111964218283461713253/posts/p/pub>[image:
> LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]*website
> <http://www.ran-lifshitz.com/> **|**  vCard
> <http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> **|*
> *  email*
>
> [image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/>
>
>
> mobile 972-54-4308444
>
>
>
>
>
>
>



-- 

[image: Google+]
<https://plus.google.com/111964218283461713253/posts/p/pub>[image:
LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]website
<http://www.ran-lifshitz.com/> |  vCard
<http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> |  email
[image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/>

mobile 972-54-4308444