Performance analysis

Besides performance analysis for VTL0 VM and host OS, we'd like to capture trace inside VTL2 as well. This allows us to analyze the Linux kernel and openvmm_hcl worker process inside VTL2.

Prerequisite for capturing perf data

The release Linux image doesn't have user mode perf program by default. To capture perf data inside VTL2, it is necessary to build OpenHCL with the below instructions, which will include the necessary programs.

cargo xflowey build-igvm [RECIPE] --release --with-perf-tools

Increase VTL2 memory page count

To capture and save perf data file inside VTL2, we need to increase VTL2's memory page count by increasing the memory_page_count value in the IGVM configuration file (openhcl-x64-dev.json). E.g. increase it to 524288 like:

{
    "vtl2": {
        "underhill": {
            ....................
            "memory_page_count": 524288
        }
    }
}

Capture perf data inside VTL2

While workloads run inside VTL0 VM, use ohcldiag-dev.exe to run perf to capture perf_events profiling data inside VTL2. E.g. the following command captures 15s of perf data and saves it in a file. You can find more info about perf on external page perf Examples

.\ohcldiag-dev.exe <VM Name> run -- perf record -F 999 -a --call-graph=dwarf -o ./openhcl.fio.perf -- sleep 15

Use perf to convert perf data to plain text and dump it to a file on host

.\ohcldiag-dev.exe <VM Name> run -- perf script -i ./openhcl.fio.perf > .\traces\openhcl.fio.perf.script

Visualize perf profiling data

Please follow up instructions on the external page Flame Graphs to create flame graph SVG file. It requires scripts from FlameGraph GitHub repo, so it is better to do it on WSL2.

Here is an example command on WSL2. It converts the perf script file to SVG file. Both of files are located under d:\tmp\vtl2 folder on Windows.

perf_file=/mnt/d/tmp/vtl2/openhcl.fio.perf.script; cat $perf_file | ./stackcollapse-perf.pl > $perf_file.folded; cat $perf_file.folded | ./flamegraph.pl > $perf_file.svg

If the perf script file doesn't have rust functions demangled correctly, please add rustfilt in the pipe to assist demangling.

perf_file=/mnt/d/tmp/vtl2/openhcl.fio.perf.script; cat $perf_file | rustfilt | ./stackcollapse-perf.pl > $perf_file.folded; cat $perf_file.folded | ./flamegraph.pl > $perf_file.svg