Incubator

Incubator runs commands and interactive shells in disposable Linux VMs using Petri, keeping builds, caches, and logs on the host.

Launcher and boot selection

[incubator] selects the launcher and VM resources; [boot] selects the guest environment and defaults to linux-direct. These profiles run on x64 Linux hosts; OpenVMM requires KVM.

Registered VMM tests use Petri nested tests, not an Incubator Flowey target runner. Each Incubator invocation cold-boots a VM.

QEMU commands

cargo run -p incubator -- \
  --profile petri/incubator/profiles/aarch64-tcg-pcie.toml \
  --kernel path/to/Image --initrd path/to/initrd \
  --share path/to/shared-root \
  --guest-pipette /share/pipette \
  --map-command-path path/to/shared-root/test-binary

The kernel and static Pipette binary must match the guest architecture. Place Pipette and the command binary beneath the shared directory. Profile devices are provisioned for VFIO before the command runs.

OpenVMM commands

Provision an mshv environment

On an x64 Linux host with KVM and nested virtualization enabled:

cargo xflowey incubator \
  --profile petri/incubator/profiles/x86_64-mshv.toml -- \
  --timeout 600 --no-pty -- /bin/sh -c 'test -c /dev/mshv && uname -a'

--profile is required. Provisioning currently supports only x64 OpenVMM profiles with mshv boot; other launcher, architecture, or boot combinations are rejected. Flowey leaves the source file untouched and prepares a copy, preserving its VM settings and boot options while replacing binary, host-boot, firmware, and boot-artifacts with resolved paths. These source paths may be placeholders; use CLI artifact overrides to supply local inputs.

This builds the native OpenVMM/Incubator pair, static x64 Pipette, and EFI host boot application; downloads the pinned mshv root kernel, firmware, shared test initrd, and verified hypervisor/loader RPMs; then boots and executes the command. Everything after Flowey's -- is passed directly to Incubator: execution options such as --timeout, --guest-env, and --no-pty, followed by the guest command (optionally separated by a second --). Flowey does not force --no-pty; Incubator uses its normal terminal detection. Flowey supplies --profile, --kernel, --initrd, --share, --guest-pipette, and --output-dir; do not repeat those in the forwarded arguments. Use the Flowey-side overrides for provisioning inputs instead.

Incubator uses the light build profile by default. --release selects release builds for all components; the normal VMM/agent build profile remains independent of the optimized launcher. It never checks out, configures, patches, or builds a kernel. --kernel path/to/mshv/bzImage overrides the published mshv-test kernel. It selects only the root kernel, not a child VM's kernel.

The writable share defaults to this checkout. --share selects a different workspace; --dir defaults to target/incubator within that share and must remain beneath it so Incubator can read the staged Pipette. Each preparation overwrites profile.toml, including any manual edits. The directory represents the latest preparation, not a saved environment: discard previously printed commands after preparing it again, since they may reference old binaries with the new profile. Use a separate --dir to keep multiple environments. Unchanged binaries are reused across kernel edits. Per-run logs remain in results/incubator-*; do not use the same directory concurrently.

Use --prepare-only without a guest command to prepare everything and print a direct Incubator command for /bin/sh. Run that printed command for an interactive shell without going through Flowey again. Copy the generated profile before customizing it. --kernel-cmdline and --hypervisor-cmdline override the source profile's boot options without rebuilding EFI code.

--initrd, --firmware, and --boot-artifacts accept explicit local inputs. RPMs use Flowey's normal persistent cache; missing RPMs are downloaded as needed. Cached RPMs and extracted payloads are checked on reuse; corrupt caches fail rather than being trusted. Packaged licenses and source URLs/hashes are retained with the extracted payloads. No RPM is installed and no package script runs.

Supply artifacts manually

Build the native launcher and Incubator from the same checkout for protocol compatibility, and build a static guest agent:

cargo build -p openvmm
cargo build -p incubator --profile light
cargo build -p pipette --target x86_64-unknown-linux-musl
cargo build -p mshv_host_boot --target x86_64-unknown-uefi

Copy petri/incubator/profiles/x86_64-mshv.toml outside the checkout and set absolute paths for binary, host-boot, firmware, and boot-artifacts. Set host-boot to the built target/x86_64-unknown-uefi/debug/mshv_host_boot.efi. This environment requires nested-virt = true, a compatible mshv root kernel, and the boot artifacts below.

target/light/incubator \
  --profile path/to/mshv-profile.toml \
  --kernel path/to/mshv/bzImage \
  --initrd path/to/test/initrd \
  --share "$PWD" \
  --guest-pipette /share/target/x86_64-unknown-linux-musl/debug/pipette \
  --output-dir target/mshv-logs \
  --no-pty /bin/sh -c 'test -c /dev/mshv && uname -a'

Omit --no-pty and run /bin/sh for an interactive session. Pipette must be inside the shared directory. Keep the mshv root kernel separate from any child VM's kernel, and the native outer OpenVMM separate from guest binaries.

To expose virtual SMT explicitly, set enable-smt = true alongside processors in [incubator]. Omission preserves the launcher's default. This controls the outer topology, not the child VM's vCPU count. Verify the effective hypervisor scheduler in its boot log when comparing ROOT and CORE; requesting CORE without SMT can select a classic scheduler instead.

Incubator supplies Petri with a UefiGuest::Mshv boot recipe: resolved kernel and payload paths plus an owned, prepared initrd. Petri composes and owns the temporary boot disk during VM preparation; the separate mshv platform preset configures nested virtualization and the required devices.

OpenVMM guests use virtio-console (console=hvc0) for Linux output and guest-initiated virtio-vsock for Pipette. The host listener exists before boot; agent connection does not poll TCP readiness. The network remains available for guest workloads. UEFI/host-boot/HvLoader output stays on the 16550 UART. The mshv default disables Linux UART initialization with 8250.nr_uarts=0; this does not affect firmware output. For early Linux UART diagnostics, select console=ttyS0 and replace that option with 8250.nr_uarts=2 in the mshv kernel options. Selecting both consoles duplicates Linux output. QEMU keeps its existing TCP agent transport.

For OpenVMM without mshv, use petri/incubator/profiles/x86_64-openvmm.toml with a native executable path, an ordinary x64 Linux kernel, and the test initrd. No firmware or hypervisor payloads are needed. Set nested-virt = true to create child KVM VMs, and replace the example's /dev/mshv check with your command.

mshv boot artifacts and kernel

Incubator does not download payloads. Keep authorized binaries, their packaged licenses, and generated images outside Git; public availability does not change licensing terms.

The artifact directory must contain:

boot/
  HvLoader.efi
  lxhvloader.dll
  Windows/System32/...   complete matching runtime

Both hvix64.exe and hvax64.exe must be present. Shim and GRUB are not used. The validated Azure Linux 3 payload recipe uses edk2-hvloader 20240524git3e722403cd16-20 and matching mshv / mshv-bootloader-lx 26100.9444.2609032029.1-1 payloads. The composer does not verify provenance, signatures, architecture, or compatibility.

The root kernel needs the mshv EFI-launch patches, CONFIG_EFI_STUB=y, CONFIG_MSHV_ROOT=y, and built-in PCIe, virtio-blk, virtio-net, virtio-9p, devtmpfs, and initramfs support. Disable modules and reuse the OpenVMM test initrd. A stock upstream test kernel cannot launch this environment. Build virtio-console and virtio-vsock into the root kernel as well. Petri selects the matching vsock initcall blacklist for both Linux-direct and mshv UEFI boot, preventing the competing Hyper-V socket driver from claiming it.

The mshv-test dependency recipe targets released rolling-lts/kata/6.18.34.mshv4 plus the root IO-APIC IRQ-lifecycle fix. That release already restores the two-bank partition-feature ABI; the older mshv2 development kernel needed a separate compatibility patch. Kernel build and source selection remain independent of OpenVMM provisioning.

To compose an image without launching it:

hypervisor_cmdline='MSHV_ROOT=\Windows MSHV_ENABLE=TRUE'
hypervisor_cmdline+=' MSHV_SCHEDULER_TYPE=ROOT MSHV_X2APIC_POLICY=ENABLE'
hypervisor_cmdline+=' MSHV_FORCE_NESTED=TRUE'

cargo run -p incubator --bin incubator-mshv-image -- \
  --host-boot target/x86_64-unknown-uefi/debug/mshv_host_boot.efi \
  --payload path/to/artifacts/boot \
  --kernel path/to/bzImage --initrd path/to/initrd \
  --kernel-cmdline 'console=ttyS0 rdinit=/init' \
  --hypervisor-cmdline "$hypervisor_cmdline" \
  --output path/to/new-mshv.img

The output parent must exist; existing output is never overwritten. Payload files must be regular files with ASCII FAT-compatible names.

Boot command lines

The host boot application uses the EFI Simple File System protocol to read two UTF-8 files at the ESP root. They contain literal load options, not shell or GRUB code:

Incubator requires both kernel-cmdline and hypervisor-cmdline in [boot]; Petri appends the prepared initrd's required rdinit and vsock driver selection. The standalone image composer likewise requires both --kernel-cmdline and --hypervisor-cmdline. Neither path supplies default command lines. Both write these files when composing media. You can also edit the files on an existing ESP without rebuilding the host boot application, kernel, or hypervisor.

Each file is limited to 16 KiB and a single UCS-2-compatible line, with an optional final LF or CRLF. Quotes, spaces, backslashes, and shell metacharacters are passed literally. Missing or malformed configuration fails the boot rather than selecting hidden defaults. The host boot application initializes HvLoader, then enters Linux's EFI stub; the modified kernel retains responsibility for launching the hypervisor.

Sharing and guest trust

The mshv host boot is development-only and refuses Secure Boot mode. Its SHIM_LOCK verification callback deliberately accepts every image; it does not authenticate signatures or implement shim's measurements.

All environments copy the host CA bundle into the temporary initrd and export SSL_CERT_FILE=/incubator-ca-certificates.crt for guest TLS clients. A nonempty host SSL_CERT_FILE selects the bundle; otherwise a conventional Linux system bundle is required. An invalid or unreadable bundle fails the run. The guest inherits that trust, including enterprise CAs, without copying private keys or disabling verification. Its clock is initialized from the host.

Output and failures

Each invocation creates an incubator-* directory beneath --output-dir, with console logs (hvc0.log for OpenVMM Linux, uefi.log for firmware/early UART, or guest.log for QEMU), VMM logs (openvmm.log or qemu.log), and petri.jsonl. Direct Incubator keeps guest stdout/stderr separate from runner diagnostics; xflowey also prints provisioning logs. Use the prepared direct command when stdout must contain only guest output. Use --guest-env or INCUBATOR_ENV for guest environment overrides.

--timeout / INCUBATOR_TIMEOUT bounds boot for Linux-direct and boot plus command execution for mshv. Graceful shutdown and teardown each have a ten-second deadline. OpenVMM cleanup is best-effort: startup cancellation or an unresponsive VMM can leave a host process requiring manual termination. Direct Incubator invocations preserve the guest exit code. Through xflowey, a nonzero guest exit fails the pipeline with Flowey's failure status instead. Unexpected VM exit/reset and timeouts fail. SIGINT/SIGTERM attempt cleanup; SIGKILL cannot. Temporary boot files are removed on normal teardown and handled errors; logs remain on the host.