Introduction to VFIO

• located in /dev/vfio
• Each divided into regions
  • Each region maps to a device resource (MMIO BAR, IO BAR, PCI configuration space)
  • Region count and information discovered through ioctl.
    Properties that can be discovered via ioctl are:
  • VFIO_DEVICE_GET_INFO ( /linux/include/uapi/linux/vfio.h:186~204 )

  #define VFIO_DEVICE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 7) 

  #define VFIO_DEVICE_GET_REGION_INFO _IO(VFIO_TYPE, VFIO_BASE + 8) 

  #define VFIO_DEVICE_GET_IRQ_INFO _IO(VFIO_TYPE, VFIO_BASE + 9) 

  

  • vfio_pci_ioctl() function is implemented in /linux/drivers/vfio/pci/vfio_pci.c:545 .

  VFIO-PCI Device Driver Structure #

  • Mainly focus on vfio-pci
  • Implementation is in /linux/drviers/vfio/pci . Main driver code is vfio_pci.c . This kernel module is compiled as vfio_pci , and we load the module as modprobe vfio-pci for VFIO based PCI-passthrough.

  $ sudo modprobe vfio_pci $ sudo vfio-bind 0000:01:00.0 

  • Main kernel module file is vfio_pci.c . When Linux kernel initializes this module, vfio_pci_init() is called.

  static int __init vfio_pci_init(void)  // Allocate shared config space permision data used by all devices vfio_pci_init_perm_bits(); // Register and scan for devices pci_register_device(&vfio_pci_driver); vfio_pci_fill_ids(); return 0; > 

  static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)  struct vfio_pci_device* vdev; vdev = kzalloc(sizeof(*vdev), GFP_KERNEL); vdev->pdev = pdev; vdev->irq_type = VFIO_PCI_NUM_IRQS; vfio_add_group_dev(&pdev->dev, &vfio_pci_ops, vdev); . > 

  static int vfio_pci_open(void *device_data)  struct vfio_pci_device* vdev = device_data; try_module_get(THIS_MODULE); vfio_pci_enable(vdev); vfio_spapr_pci_eeh_open(vdev->pdev); vdev->refcnt++; . > 

  static const struct vfio_device_ops vfio_pci_ops =  .name = "vfio-pci", .open = vfio_pci_open, .release = vfio_pci_release, .ioctl = vfio_pci_ioctl, .read = vfio_pci_read, .write = vfio_pci_write, .mmap = vfio_pci_mmap, .request = vfio_pci_request, >; 

  1. vfio_pci_config_rw (defined in /linux/drivers/vfio/pci/vfio_pci_config.c:1678 )
  2. vfio_pci_bar_rw (defined in /linux/drivers/vfio/pci/vfio_pci_rdwr.c:116 )
  3. vfio_pci_vga_rw (defined in /linux/drivers/vfio/pci/vfio_pci_rdwr.c:183 )

  in terms of the index passed to the function.

  For GPU, vfio_pci_rw() is mostly called with the index 7 (= VFIO_PCI_CONFIG_REGION_INDEX ) and 8 (= VFIO_PCI_VGA_REGION_INDEX ). Small number of calls are done with index 0 (= VFIO_PCI_BAR0_REGION_INDEX ). Note that each index can be found in /linux/include/uapi/linux/vfio.h:418 .

  References #

  • Alex Williamson. An Introduction to PCI Device Assignment with VFIO. [Online]. Aug 2016.
  • Alex Williamson. VFIO: PCI device assignment breaks free of KVM. [Online]. 2011.