From aa6a11a42d56aa882ff8eaa99c3c4a299fdd5122 Mon Sep 17 00:00:00 2001 From: Babis Chalios Date: 2025年8月25日 17:29:29 +0200 Subject: [PATCH 1/2] doc: add documentation on usage of PCI support Add information in our Documentation regarding how users can enable PCI support for Firecracker microVMs and mention requirements for building the guest kernel, as well as the requirements for kernel command line parameters. Also, add an entry in the CHANGELOG mentioning the addition of PCI support. Signed-off-by: Babis Chalios --- CHANGELOG.md | 6 ++++ FAQ.md | 2 +- docs/getting-started.md | 11 +++++-- docs/kernel-policy.md | 66 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 82 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7eb43cf986..5892614b4a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,12 @@ and this project adheres to guest's serial console. Not configuring it means Firecracker will continue to print serial output to stdout. Similarly to the logger, this configuration is not persisted across snapshots. +- [#5364](https://github.com/firecracker-microvm/firecracker/pull/5364): Added + PCI support in Firecracker. PCI support is optional. Users can enable it + passing the `--enable-pci` flag when launching the Firecracker process. When + Firecracker process is launched with PCI support, it will create all VirtIO + devices using a PCI VirtIO transport. If not enabled, Firecracker will use the + MMIO transport instead. ### Changed diff --git a/FAQ.md b/FAQ.md index 9456471df8c..8c47529661d 100644 --- a/FAQ.md +++ b/FAQ.md @@ -129,7 +129,7 @@ Example of a kernel valid command line that enables the serial console (which goes in the `boot_args` field of the `/boot-source` Firecracker API resource): ```console -console=ttyS0 reboot=k panic=1 pci=off nomodule +console=ttyS0 reboot=k panic=1 nomodule ``` ### How can I configure multiple Ethernet devices through the kernel command line? diff --git a/docs/getting-started.md b/docs/getting-started.md index f0b02c6e1b5..17c681da34c 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -194,9 +194,16 @@ API_SOCKET="/tmp/firecracker.socket" sudo rm -f $API_SOCKET # Run firecracker -sudo ./firecracker --api-sock "${API_SOCKET}" +sudo ./firecracker --api-sock "${API_SOCKET} --enable-pci" ``` +The `--enable-pci` flag instructs Firecracker to create all VirtIO devices using +a PCI VirtIO transport. This flag is optional. If not passed, Firecracker will +create devices using the legacy MMIO transport. We suggest that users enable the +PCI transport, as it yields higher throughput and lower latency for VirtIO +devices. For more information regarding guest kernel requirements for using PCI +look at our [kernel policy documentation](./kernel-policy.md). + In a new terminal (do not close the 1st one): ```bash @@ -240,7 +247,7 @@ sudo curl -X PUT --unix-socket "${API_SOCKET}" \ "http://localhost/logger" KERNEL="./$(ls vmlinux* | tail -1)" -KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off" +KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1" ARCH=$(uname -m) diff --git a/docs/kernel-policy.md b/docs/kernel-policy.md index 86823459d44..9387244118b 100644 --- a/docs/kernel-policy.md +++ b/docs/kernel-policy.md @@ -69,6 +69,15 @@ The configuration items that may be relevant for Firecracker are: - use CPU RNG instructions (if present) to initialize RNG. Available for>= 5.10 - ACPI support - `CONFIG_ACPI` and `CONFIG_PCI` +- PCI support: + - `CONFIG_BLK_MQ_PCI` + - `CONFIG_PCI` + - `CONFIG_PCI_MMCONFIG` + - `CONFIG_PCI_MSI` + - `CONFIG_PCIEPORTBUS` + - `CONFIG_VIRTIO_PCI` + - `CONFIG_PCI_HOST_COMMON` + - `CONFIG_PCI_HOST_GENERIC` There are also guest config options which are dependant on the platform on which Firecracker is run: @@ -142,6 +151,63 @@ following configurations: - Only legacy mechanisms - Both ACPI and legacy mechanisms +##### Booting with PCI: + +Firecracker supports booting guest microVMs with PCI support. This option is +enabled using the `--enable-pci` flag when launching the Firecracker process. +With PCI enabled, Firecracker will create all VirtIO devices using a PCI VirtIO +transport. The PCI transport typically achieves higher throughput and lower +latency for VirtIO devices. No further, per device, configuration is needed to +enable the PCI transport. + +PCI support is optional; if it is not enabled Firecracker will create VirtIO +devices using the MMIO transport. + +For Firecracker microVMs to boot properly with PCI support, use a guest kernel +built with PCI support. See the relevant Kconfig flags in our list of +[relevant Kconfig options](#guest-kernel-configuration-items): + +> [!IMPORTANT] +> +> Make sure that the kernel command line **does NOT** include the `pci=off` +> slug, which disables PCI support during boot time within the guest. When PCI +> is disabled, Firecracker will add this slug in the command line to instruct +> the guest kernel to skip useless PCI checks. For more info, look into the +> section for [Kernel command line parameters](#kernel-command-line-parameters). + +> [!NOTE] +> +> On x86_64 systems, `CONFIG_PCI` Kconfig option is needed even when booting +> microVMs without PCI support in case users want to use ACPI to boot. See +> [here](#booting-with-acpi-x86_64-only) for more info. + +## Kernel command line parameters + +By default, Firecracker will boot a guest microVM passing the following command +line parameters to the kernel: + +`reboot=k panic=1 nomodule 8250.nr_uarts=0 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce`. + +- `reboot=k` shut down the guest on reboot, instead of rebooting +- `panic=1` on panic, reboot after 1 second +- `nomodule` disable loadable kernel module support +- `8250.nr_uarts=0` disable 8250 serial interface +- `i8042.noaux` do not probe the i8042 controller for an attached mouse (save + boot time) +- `i8042.nomux` do not probe i8042 for a multiplexing controller (save boot + time) +- `i8042.dumbkbd` do not attempt to control kbd state via the i8042 (save boot + time) +- `swiotlb=noforce` disable software bounce buffers (SWIOTLB) + +When running without [PCI support](#booting-with-pci), Firecracker will also +append `pci=off` to the above list. This option instructs the guest kernel to +avoid PCI probing. + +Users can provide their own command line parameters through the `boot_args` +field of the `/boot-source` +[Firecracker API](../src/firecracker/swagger/firecracker.yaml). + ## Caveats - [Snapshot compatibility across kernel versions](snapshotting/snapshot-support.md#snapshot-compatibility-across-kernel-versions) From 45d5b6cff1b00ee19a0eaa7960b46b4e4d9cc393 Mon Sep 17 00:00:00 2001 From: Babis Chalios Date: 2025年8月25日 17:39:29 +0200 Subject: [PATCH 2/2] pci: document potential contention on PCI bus lock We have a single lock for all devices on the PCI bus that serializes reads and writes on the devices' PCI configuration space and BARs. This should not be a problem at the moment. It should be out of any hot path and only up until we are setting up devices. However, add a comment that mentions the existence of the contention so that we keep it in mind for the futuer (and maybe perform some profiling). Signed-off-by: Babis Chalios --- src/pci/src/bus.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/pci/src/bus.rs b/src/pci/src/bus.rs index 01c9b1f1933..12188ee7ba5 100644 --- a/src/pci/src/bus.rs +++ b/src/pci/src/bus.rs @@ -161,6 +161,9 @@ impl PciConfigIo { return 0xffff_ffff; } + // NOTE: Potential contention among vCPU threads on this lock. This should not + // be a problem currently, since we mainly access this when we are setting up devices. + // We might want to do some profiling to ensure this does not become a bottleneck. self.pci_bus .as_ref() .lock() @@ -195,6 +198,9 @@ impl PciConfigIo { return None; } + // NOTE: Potential contention among vCPU threads on this lock. This should not + // be a problem currently, since we mainly access this when we are setting up devices. + // We might want to do some profiling to ensure this does not become a bottleneck. let pci_bus = self.pci_bus.as_ref().lock().unwrap(); if let Some(d) = pci_bus.devices.get(&(device as u32)) { let mut device = d.lock().unwrap();

AltStyle によって変換されたページ (->オリジナル) /