Fully release VFIO resources in DPDK

Question

I am writing a C program that periodically loads/unloads a dynamic library. The dynamic library performs network operations using DPDK, and in my setup, it relies on the VFIO framework.

This is a minimal example of what my program does:

rte_eal_init(argc, argv);
// ... configure the device on port_id with <pci_addr> ...
rte_eth_dev_start(port_id);
// ... do network operations ...
rte_eth_dev_stop(port_id);
rte_eth_dev_close(port_id);
rte_eal_cleanup();

As EAL parameters, I pass the string --file-prefix <random number> to ensure that EAL restarts with a clean context every time.

However, after unloading the library, the VFIO container and group file descriptors remain open in the process, as you can see from the output of lsof:

user@node-0:~$ sudo lsof -p 23322 | grep vfio
myprog 23322 root mem REG 0,14 14038 anon_inode:[vfio-device] (stat: No such file or directory)
myprog 23322 root 15u CHR 10,196 0t0 139 /dev/vfio/vfio
myprog 23322 root 276u CHR 241,0 0t0 734 /dev/vfio/39
myprog 23322 root 278u a_inode 0,14 0 14038 [vfio-device]

This happens when IOMMU is enabled (Type-1) but also if it is used in unsafe mode (noiommu). Hence, I believe that the rte_eal_cleanup(), and the underlying driver cleanup in particular, does not really free the VFIO-related resources. However, I need to fully release them so that I can reinitialize the EAL in the same process and use the same device. Currently, when I try to do that, it fails as expected with an EBUSY error:

EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/nsn_857422/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Cannot open /dev/vfio/39: Device or resource busy
EAL: Failed to open VFIO group 39
EAL: Requested device .&checktime(0000,18,00,':').1 cannot be used
EAL: Bus (pci) probe failed.

Some additional information about my setup: DPDK v22.11 NIC Dual-port Intel X710 10Gbe NIC (i40e driver) OS Ubuntu 22.11

While I was expecting that DPDK, and in particular the i40e PMD I am using in this case, would take care of the resource cleanup, I tried to dig into the DPDK library to find a workaround.

1. Calling rte_vfio_release_device() before the EAL cleanup fails with the EBUSY error;
2. Manually closing the file descriptors obtained with lsof with close() does not close the VFIO container, which remains open;
3. Functions that are available internally in DPDK, such as rte_pci_unmap_device(), are meant to be called by drivers and not by the end user.

When using a different device, i.e., a Mellanox ConnectX-6 DX NIC, which has a different driver (mlx5), the issue is not present. But I need my code to be portable also on VFIO-backed drivers/NICs.

I’m looking for a supported way to fully release VFIO kernel resources in-process; any patterns or techniques to safely stop, clean up, and reinitialize a VFIO-backed DPDK device in the same process.

• stark: Are you checking the return values of rte_eth_dev_stop and rte_eth_dev_close to verify that they complete?
• ellerre: Yes, I am checking them and they both return 0
• ellerre: I temporarily solved the issue by patching the driver, adding a call to the rte_pci_unmap_device() in the i40e_dev_close() function. However, this is not ideal as it forces my users to recompile and reinstall DPDK.