PCIe Emulation

OpenVMM emulates a PCIe topology that presents root complexes, root ports, and optional switches to the guest. Endpoint devices (NVMe, virtio, etc.) attach to root ports via a generic bus interface.

Topology

Root Complex (GenericPcieRootComplex)
├── Root Port 0  (PcieDownstreamPort)  → endpoint device
├── Root Port 1  (PcieDownstreamPort)  → endpoint device or switch
└── Root Port N  (PcieDownstreamPort)  → ...

The root complex owns the ECAM MMIO region. On x86_64, a segment-zero root complex that spans the complete bus range also supports legacy configuration access through I/O ports CF8/CFC, unless a PIIX4 PCI bus owns those ports. When segment zero is split across multiple root complexes, those root complexes use ECAM so that one legacy handler does not hide buses owned by another root complex. Root complexes in additional segments are always ECAM-only because CF8/CFC can address only segment zero. Each port has a Type 1 (bridge) configuration space with PCIe Express and MSI capabilities.

Ports may optionally be hotplug-capable. Devices behind non-hotplug ports are attached at VM construction time. Hotplug-capable ports start empty and devices can be added or removed at runtime.

PCIe Hotplug

Native PCIe hotplug follows the same interrupt-driven model as real hardware (PCIe Base Spec §6.7). No ACPI GPE, SCI, or custom protocol is needed — the guest's pciehp driver handles everything via config space registers and MSI.

How it works

1. VMM sets port state: When a device is hot-added, the VMM atomically updates the port's Slot Status (presence_detect_state, presence_detect_changed, data_link_layer_state_changed) and Link Status (data_link_layer_link_active).

2. MSI fires: If the guest has enabled hot_plug_interrupt_enable in the port's Slot Control register, the VMM fires the port's MSI.

3. Guest handles the event: The guest's pciehp driver receives the MSI, reads Slot Status to see what changed, programs the bridge's bus numbers, scans the secondary bus for new devices, and clears the RW1C status bits.

4. Device removal follows the same flow in reverse — presence and link active are cleared, changed bits are set, and MSI fires.

Runtime API

Hot-add and hot-remove are triggered via VmRpc::AddPcieDevice and VmRpc::RemovePcieDevice messages. These resolve a device resource, create the device with MMIO registration, attach it to the named port, and fire the hotplug notification.

Removal waits for the device's teardown hook and payload destruction. Before teardown, guest access is permanently revoked so late accesses cannot reach the removed device. Background resources are released before removal completes only when the device implements the necessary cleanup; the default hook does not join background tasks or drain issued I/O.

stop() is a save-consistent pause, not a requirement to drain disk I/O or a guarantee that DMA has ceased. Devices retaining issued requests across pause must account for them in saved state where save/restore is supported. Device-specific lifecycle changes are separate from this infrastructure.

ChangeDeviceState::teardown(self) is a terminal operation with a default implementation that drops the stopped device. Removal permanently closes the shared wrapper, takes ownership of its payload, and stops it if running; paused and never-started units are already stopped. Async teardown owns the device until cleanup completes and its remaining fields are dropped, even if stale references still hold the inaccessible wrapper. Synchronous devices can use the default hook to release their resources through ordinary destruction. Devices with asynchronous cleanup must override it, retaining and finishing cancellation-unsafe I/O before releasing its resources.

Chipset MMIO and PIO mapping controls unregister their active ranges when dropped. Devices own remappable controls; the device state unit retains fixed-region controls and releases them before acknowledging removal.

Assigned hardware that cannot pause without losing state may quiesce during teardown instead. Ordinary stop must not destructively reset it.

A slow backend can keep the removal RPC pending. Inspection, memory accesses, and NMI requests remain available; state transitions and further hotplug operations wait for removal to finish. Worker shutdown also waits for pending cleanup before destroying the partition.

See the PetriVmRuntime trait for the test API (add_pcie_device / remove_pcie_device) and the pcie_hotplug test in vmm_tests/vmm_tests/tests/tests/multiarch/pcie.rs for a working example.

ACPI _OSC

The SSDT includes an _OSC method on each PCIe root complex that grants native PCIe control to the OS (ACPI spec §6.2.11, PCI Firmware Spec §4.5.1). This tells the OS it can use native hotplug, PME, AER, and other PCIe features rather than ACPI-based fallbacks. Linux assumes native control regardless, but Windows requires _OSC to enable native hotplug.

MSI Interrupt Routing (aarch64)

On aarch64, PCIe MSI/MSI-X interrupts are routed through either a GICv3 ITS or a GICv2m MSI frame, depending on the platform:

• GICv3 ITS (default on KVM with GICv3): The VMM creates a KVM in-kernel ITS device. Each MSI is delivered with a 32-bit ITS device ID of the form (segment << 16) | BDF. The BDF is resolved lazily at interrupt-delivery time by the device's MsiTarget, which combines the parent port's AssignedBusRange (updated by the guest as it programs secondary/subordinate bus numbers) with the device's devfn. Multi-function devices and root/switch ports that need a specific RID pass it explicitly. A single ItsSignalMsi/ItsIrqFd wrapper per segment then prepends the segment to produce the final ITS device ID — no per-device wrappers and no push-based RID synchronization are needed. ACPI boots emit an IORT with an ITS Group node and per-root-complex ID mappings; the device tree includes an its child node under the GIC with msi-controller.

• GICv2m: MSI writes map to a fixed pool of SPIs (64 by default) via a v2m doorbell register. The MADT includes a GICv2m MSI frame entry.

The v2m pool is shared by endpoints and PCIe root ports. Windows drivers can reserve every advertised MSI-X entry: a 64-entry endpoint plus one root-port MSI needs at least 65 SPIs, or Windows reports Code 12 (insufficient resources). When constructing a VM programmatically, size GicMsiConfig::V2m.spi_count for the complete topology rather than only the endpoint's active queues.

The MSI controller can be overridden with the --gic-msi CLI option (auto, its, or v2m).

Implementation notes