Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

doc: add documentation on usage of PCI support #5404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bchalios merged 2 commits into firecracker-microvm:main from bchalios:pci_docs
Aug 26, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ and this project adheres to
guest's serial console. Not configuring it means Firecracker will continue to
print serial output to stdout. Similarly to the logger, this configuration is
not persisted across snapshots.
- [#5364](https://github.com/firecracker-microvm/firecracker/pull/5364): Added
PCI support in Firecracker. PCI support is optional. Users can enable it
passing the `--enable-pci` flag when launching the Firecracker process. When
Firecracker process is launched with PCI support, it will create all VirtIO
devices using a PCI VirtIO transport. If not enabled, Firecracker will use the
MMIO transport instead.

### Changed

Expand Down
2 changes: 1 addition & 1 deletion FAQ.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Example of a kernel valid command line that enables the serial console (which
goes in the `boot_args` field of the `/boot-source` Firecracker API resource):

```console
console=ttyS0 reboot=k panic=1 pci=off nomodule
console=ttyS0 reboot=k panic=1 nomodule
```

### How can I configure multiple Ethernet devices through the kernel command line?
Expand Down
11 changes: 9 additions & 2 deletions docs/getting-started.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,9 +194,16 @@ API_SOCKET="/tmp/firecracker.socket"
sudo rm -f $API_SOCKET

# Run firecracker
sudo ./firecracker --api-sock "${API_SOCKET}"
sudo ./firecracker --api-sock "${API_SOCKET} --enable-pci"
```

The `--enable-pci` flag instructs Firecracker to create all VirtIO devices using
a PCI VirtIO transport. This flag is optional. If not passed, Firecracker will
create devices using the legacy MMIO transport. We suggest that users enable the
PCI transport, as it yields higher throughput and lower latency for VirtIO
devices. For more information regarding guest kernel requirements for using PCI
look at our [kernel policy documentation](./kernel-policy.md).

In a new terminal (do not close the 1st one):

```bash
Expand Down Expand Up @@ -240,7 +247,7 @@ sudo curl -X PUT --unix-socket "${API_SOCKET}" \
"http://localhost/logger"

KERNEL="./$(ls vmlinux* | tail -1)"
KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off"
KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1"

ARCH=$(uname -m)

Expand Down
66 changes: 66 additions & 0 deletions docs/kernel-policy.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,15 @@ The configuration items that may be relevant for Firecracker are:
- use CPU RNG instructions (if present) to initialize RNG. Available for >=
5.10
- ACPI support - `CONFIG_ACPI` and `CONFIG_PCI`
- PCI support:
- `CONFIG_BLK_MQ_PCI`
- `CONFIG_PCI`
- `CONFIG_PCI_MMCONFIG`
- `CONFIG_PCI_MSI`
- `CONFIG_PCIEPORTBUS`
- `CONFIG_VIRTIO_PCI`
- `CONFIG_PCI_HOST_COMMON`
- `CONFIG_PCI_HOST_GENERIC`

There are also guest config options which are dependant on the platform on which
Firecracker is run:
Expand Down Expand Up @@ -142,6 +151,63 @@ following configurations:
- Only legacy mechanisms
- Both ACPI and legacy mechanisms

##### Booting with PCI:

Firecracker supports booting guest microVMs with PCI support. This option is
enabled using the `--enable-pci` flag when launching the Firecracker process.
With PCI enabled, Firecracker will create all VirtIO devices using a PCI VirtIO
transport. The PCI transport typically achieves higher throughput and lower
latency for VirtIO devices. No further, per device, configuration is needed to
enable the PCI transport.

PCI support is optional; if it is not enabled Firecracker will create VirtIO
devices using the MMIO transport.

For Firecracker microVMs to boot properly with PCI support, use a guest kernel
built with PCI support. See the relevant Kconfig flags in our list of
[relevant Kconfig options](#guest-kernel-configuration-items):

> [!IMPORTANT]
>
> Make sure that the kernel command line **does NOT** include the `pci=off`
> slug, which disables PCI support during boot time within the guest. When PCI
> is disabled, Firecracker will add this slug in the command line to instruct
> the guest kernel to skip useless PCI checks. For more info, look into the
> section for [Kernel command line parameters](#kernel-command-line-parameters).

> [!NOTE]
>
> On x86_64 systems, `CONFIG_PCI` Kconfig option is needed even when booting
> microVMs without PCI support in case users want to use ACPI to boot. See
> [here](#booting-with-acpi-x86_64-only) for more info.

## Kernel command line parameters

By default, Firecracker will boot a guest microVM passing the following command
line parameters to the kernel:

`reboot=k panic=1 nomodule 8250.nr_uarts=0 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce`.

- `reboot=k` shut down the guest on reboot, instead of rebooting
- `panic=1` on panic, reboot after 1 second
- `nomodule` disable loadable kernel module support
- `8250.nr_uarts=0` disable 8250 serial interface
- `i8042.noaux` do not probe the i8042 controller for an attached mouse (save
boot time)
- `i8042.nomux` do not probe i8042 for a multiplexing controller (save boot
time)
- `i8042.dumbkbd` do not attempt to control kbd state via the i8042 (save boot
time)
- `swiotlb=noforce` disable software bounce buffers (SWIOTLB)

When running without [PCI support](#booting-with-pci), Firecracker will also
append `pci=off` to the above list. This option instructs the guest kernel to
avoid PCI probing.

Users can provide their own command line parameters through the `boot_args`
field of the `/boot-source`
[Firecracker API](../src/firecracker/swagger/firecracker.yaml).

## Caveats

- [Snapshot compatibility across kernel versions](snapshotting/snapshot-support.md#snapshot-compatibility-across-kernel-versions)
Expand Down
6 changes: 6 additions & 0 deletions src/pci/src/bus.rs
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,9 @@ impl PciConfigIo {
return 0xffff_ffff;
}

// NOTE: Potential contention among vCPU threads on this lock. This should not
// be a problem currently, since we mainly access this when we are setting up devices.
// We might want to do some profiling to ensure this does not become a bottleneck.
self.pci_bus
.as_ref()
.lock()
Expand Down Expand Up @@ -195,6 +198,9 @@ impl PciConfigIo {
return None;
}

// NOTE: Potential contention among vCPU threads on this lock. This should not
// be a problem currently, since we mainly access this when we are setting up devices.
// We might want to do some profiling to ensure this does not become a bottleneck.
let pci_bus = self.pci_bus.as_ref().lock().unwrap();
if let Some(d) = pci_bus.devices.get(&(device as u32)) {
let mut device = d.lock().unwrap();
Expand Down

AltStyle によって変換されたページ (->オリジナル) /