Save State

OpenHCL supports the mechanism of saving & restoring state. This primitive can be used for various VM operations, but a key use case is for updating OpenHCL at runtime (a.k.a. "Servicing"). This save state can be stored in memory or on durable media, read back at a later time.

Save State First Principles

Here are the principles you must maintain when adding new save & restore code:

1. Save & Restore is Forward & Backwards Compatible: A newer version of OpenHCL must understand save state from a prior version, and an older version must not crash when reading save state from a newer version.
2. Do not break save state after that state is in use: Save state must be compatible from any commit to any other commit, once the product has shipped and started using that save state.¹
3. All Save State is Protocol Buffers: All save state is encoded as ProtoBuf, using mesh.

Best practices

1. Put save state in it's own module: This makes PR reviews easier, to catch any mistakes updating save state.
2. Create a unique package per crate: A logical grouping of saved state should have the same package
3. Avoid unsupported types, when possible: These types don't support the default values needed for safely extending save state:
   • Arrays: if you need to add an array, consider a vec or Option<[T; N]> instead.
   • Enum: if you need to add an enum, add it as `Option' instead.
4. Be particularly careful when updating saved state: See below.

Updating (Extending) Saved State

Since saved state is just Protocol Buffers, use the guide to updating Protocol Buffers messages as a starting point, with the following caveats:

1. OpenVMM uses sint32 to represent the i32 native type. Therefore, changing i32 to u32 is a breaking change, for example.
2. The Protocol Buffers docs mention what happens for newly added fields, but it bears adding some nuance here:
   1. Plain arrays and enums are not supported. Reading newer save state with an array or enum will fail on an older build. Instead, wrap these in an Option.
   2. Old -> New Save State: Save state from a prior revision will not contain some newly added fields. Those fields will get the default values. This is how that breaks down for the rust types:
      • Option<T> => None
      • Structs => each field gets that field's default value
      • Vecs => empty vec
      • Numbers => 0
      • Strings => ""
   3. New -> Old Save State: Unknown fields are ignored.
3. Ensure that default values for new saved state fields make semantic sense. Here is the heuristic: if you need a conditional to check if the restored value is the default because it was not included in save state, you should use an Option. If you wouldn't otherwise need a conditional, you should not use an Option.
4. Define your saved state so that the natural default of bool types is false.

Defining Saved State

Saved state is defined as a struct that has #[derive(Protobuf)] and #[mesh(package = "package_name")] attributes. Here is an example, taken from the nvme_driver:

#![allow(unused)]
fn main() {
pub mod save_restore {
    use super::*;

    /// Save/restore state for IoQueue.
    #[derive(Protobuf, Clone, Debug)]
    #[mesh(package = "nvme_driver")]
    pub struct IoQueueSavedState {
        #[mesh(1)]
        /// Which CPU handles requests.
        pub cpu: u32,
        #[mesh(2)]
        /// Interrupt vector (MSI-X)
        pub iv: u32,
        #[mesh(3)]
        pub queue_data: QueuePairSavedState,
    }
}
}