Save State

OpenHCL supports the mechanism of saving & restoring state. This primitive can be used for various VM operations, but a key use case is for updating OpenHCL at runtime (a.k.a. "Servicing"). This save state can be stored in memory or on durable media, read back at a later time.

Save State First Principles

Here are the principles you must maintain when adding new save & restore code:

1. Save & Restore is Forward & Backwards Compatible: A newer version of OpenHCL must understand save state from a prior version, and an older version must not crash when reading save state from a newer version.
2. Do not break save state after that state is in use: Save state must be compatible from any commit to any other commit, once the product has shipped and started using that save state.¹
3. All Save State is Protocol Buffers: All save state is encoded as ProtoBuf, using mesh.

Best practices

1. Put save state in it's own module: This makes PR reviews easier, to catch any mistakes updating save state.
2. Create a unique package per crate: A logical grouping of saved state should have the same package
3. Avoid unsupported types, when possible: These types don't support the default values needed for safely extending save state:
   • Arrays: if you need to add an array, consider a vec or Option<[T; N]> instead.
   • Enum: if you need to add an enum, add it as Option<MyEnum> instead.
4. Be particularly careful when updating saved state: See below.

Destination lifecycle

Restore into a stopped VM whose components have just been constructed or successfully reset. Stopping a previously running VM is not sufficient: reset it before restoring. Once restoration of unit state begins, reset before attempting another restore, even if it fails. Initial snapshot unit-name validation errors do not require a reset, since no unit state has been restored.

Saving restored state before starting the VM is supported; restoring that snapshot still requires a fresh or reset destination.

Synchronous and asynchronous implementations

New code should implement vmcore::save_restore::AsyncSaveRestore, even if it doesn't need to await anything yet. It will eventually be the only typed save/restore interface. It uses an associated SavedState type; protobuf encoding is handled for you.

Chipset code uses AsyncProtobufSaveRestore to call either kind of device. Existing SaveRestore implementations are adapted automatically, so devices and helpers can migrate over time. See the trait definitions for the required Send and protobuf bounds.

When converting a device, replace its SaveRestore implementation with AsyncSaveRestore; don't implement both. Keep the saved state type and protobuf fields unchanged. Finish the work before save or restore returns, so errors reach the caller rather than surfacing later when the device starts.

Updating (Extending) Saved State

Since saved state is just Protocol Buffers, use the guide to updating Protocol Buffers messages as a starting point, with the following caveats:

1. OpenVMM uses sint32 to represent the i32 native type. Therefore, changing i32 to u32 is a breaking change, for example.
2. The Protocol Buffers docs mention what happens for newly added fields, but it bears adding some nuance here:
   1. Plain arrays and enums are not supported. Reading newer save state with an array or enum will fail on an older build. Instead, wrap these in an Option.
   2. Old -> New Save State: Save state from a prior revision will not contain some newly added fields. Those fields will get the default values. This is how that breaks down for the rust types:
      • Option<T> => None
      • Structs => each field gets that field's default value
      • Vecs => empty vec
      • Numbers => 0
      • Strings => ""
   3. New -> Old Save State: Unknown fields are ignored.
3. Ensure that default values for new saved state fields make semantic sense. Here is the heuristic: if you need a conditional to check if the restored value is the default because it was not included in save state, you should use an Option. If you wouldn't otherwise need a conditional, you should not use an Option.
4. Define your saved state so that the natural default of bool types is false.

Defining Saved State

Saved state is defined as a struct that has #[derive(Protobuf)] and #[mesh(package = "package_name")] attributes. Here is an example, taken from the nvme_driver:

pub mod save_restore {
    use super::*;

    /// Save/restore state for IoQueue.
    #[derive(Protobuf, Clone, Debug)]
    #[mesh(package = "nvme_driver")]
    pub struct IoQueueSavedState {
        #[mesh(1)]
        /// Which CPU handles requests.
        pub cpu: u32,
        #[mesh(2)]
        /// Interrupt vector (MSI-X)
        pub iv: u32,
        #[mesh(3)]
        pub queue_data: QueuePairSavedState,
    }
}