Arm SMMUv3

OpenVMM emulates an Arm SMMUv3 IOMMU for aarch64 guests, attached to a PCIe root complex. Devices behind a covered root complex present IOVAs rather than GPAs, and the SMMU translates them using stream table entries (STEs) and context descriptors (CDs) that the guest programs in its own memory.

One SMMU instance covers one root complex. A device's StreamID is its guest-assigned RequesterID (bus/device/function), so the guest's stream table is indexed directly by BDF.

See --smmu for the command-line syntax.

Advertised capabilities

The emulated SMMU advertises a deliberately small feature set. Anything not listed here reads as unsupported in the IDR registers:

Because attribute override is not advertised, the STE fields that would control incoming memory attributes and permissions (MTCFG, MemAttr, SHCFG, ALLOCCFG, NSCFG, PRIVCFG, INSTCFG) are RES0 and ignored.

Translation modes

Two independent things determine how a device's DMA is handled: whether the SMMU is enabled, and what the device's STE says.

While CR0.SMMUEN is 0, GBPA.ABORT selects the global policy: transactions either bypass (IOVA = GPA) or abort. GBPA.ABORT resets to 0, so DMA bypasses before the guest brings the SMMU up — this preserves device access during firmware and early boot.

Once enabled, each transaction's STE.Config decides:

An invalid STE (V = 0) or an out-of-range StreamID also aborts and records an event, per the SMMUv3 architecture.

Software translation

By default the SMMU walks the guest's structures in software: it reads the STE, then the CD, then walks the stage-1 page tables to produce a GPA for each DMA access and MSI write. Faults are recorded to the guest's event queue and signalled with the EVTQ interrupt.

This works for emulated devices, whose DMA already flows through the VMM. It cannot work for a VFIO-assigned device, whose DMA never reaches the VMM — so assigning a device behind a non-accelerated SMMU is rejected at startup rather than silently ignoring the guest's translations.

Accelerated translation

With accel, the SMMU delegates translation to the host IOMMU using iommufd nesting. The guest's stage-1 tables are installed into the physical SMMU as a nested domain whose parent is the stage-2 domain that maps guest GPAs to host physical addresses. The physical SMMU then performs the full IOVA → GPA → HPA walk in hardware.

The emulator still owns the SMMU programming interface. It decodes the guest's STE and dispatches one of three dispositions per device:

Every device is always attached to some page table under the vIOMMU — the abort and bypass page tables are preallocated so a transition never has to leave a device detached.

Invalidation

TLB and configuration invalidation commands the guest writes to the command queue are forwarded to the host. Consecutive forwardable commands accumulate into a batch that is flushed as a single host invalidation at each synchronization or configuration boundary, so a shootdown burst costs one syscall instead of one per command.

Invalidation is scoped to the vIOMMU rather than to individual devices, so a guest command is forwarded once regardless of how many devices sit behind the SMMU.

StreamID binding

The host needs the guest's StreamID to route invalidations and faults, but the guest assigns bus numbers during PCIe enumeration, well after the device is created. The StreamID is therefore captured from the first routed config space write to the device, and rebound if the guest later renumbers the bus. Until it is known, the device is left aborting.

A function level reset — through either the PCIe or the conventional PCI Advanced Features capability — clears the binding across the reset and re-establishes it afterwards.

Requirements and constraints

• The host IOMMU must be an Arm SMMUv3 with iommufd nesting support, and must supply AArch64 page tables, little-endian walks, and the 4 KiB granule. Host capabilities are checked when the first device attaches.
• Assigned devices must use the VFIO cdev + iommufd path. The legacy group/container interface cannot express nested translation.
• All devices behind one accelerated SMMU must share a single iommufd context, because the emulated SMMU maps to exactly one host vIOMMU.
• The guest must boot with ACPI. One MSI IOVA range is shared between the physical-SMMU implementation and the guest: the backend uses it for assigned device MSI writes, and the IORT describes the same range as Reserved Memory so the guest identity-maps it in stage 1. Linux's Arm SMMU driver currently requires its fixed 128 MiB--129 MiB range. A hypervisor-backed SMMU can let OpenVMM select the range and pass its base during partition creation. Without the matching RMR, assigned-device MSIs can fault in guest stage 1.

Output address size

The advertised output address size (IDR5.OAS) cannot exceed the physical SMMU's. With oas=auto, an accelerated device attached before the VM starts causes the emulated SMMU to adopt the host's OAS. Starting the VM freezes the advertised value. A device hotplugged after that point must support the frozen OAS; attachment fails rather than changing a capability the guest may already have observed. With a fixed oas=N, a value larger than the host's is rejected.

Fault reporting

Faults raised by a stream the physical SMMU is actively translating originate in hardware, not in the emulator, so the VMM never sees the transaction that caused them. The host reports them on a virtual event queue attached to the vIOMMU; the VMM drains that queue and writes each record into the guest's event queue unchanged, since the host has already expressed the StreamID in the guest's namespace.

If the host virtual event queue itself overflows, the VMM logs the loss and continues forwarding the records that remain available. It cannot represent that loss as a guest event queue overflow, because EVENTQ_PROD.OVFLG specifically means the guest's event queue was full.