diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/1-preparation.md b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/1-preparation.md new file mode 100644 index 0000000000..9d663aade5 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/1-preparation.md @@ -0,0 +1,52 @@ +--- +title: Preparing the Host for Nested Virtualization +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Why use Nested Virtualization? + +Virtualization allows organizations to mutualize large multi-core servers into smaller environments with a strong level of resource isolation. It is a foundational technology for cloud computing, and Cloud Service Providers typically provide compute resources to their customers using virtualization. + +Nested virtualization enables guest virtual machines to serve as hypervisors, and run guest virtual machines inside a VM or cloud instance. There are a few use-cases where this facility is very useful: + +* **Resource isolation:** When running container applications in cloud instances, containers share a common operating system. By running containers inside microVMs like Firecracker VM, the container workloads share nothing with the host operating system +* **Mobile application development and testing:** It is very desirable to build and test Android applications on cloud infrastructure like Kubernetes. Cloud hosted Kubernetes compute nodes are typically virtual machine instances. Running Android environments in Kubernetes in this situation requires nested virtualization. +* **Testing and development of Embedded applications:** Many embedded applications are running specialized real-time or embedded operating systems, rather than Linux. Developing and testing these environments requires virtualization, which means developing on bare metal or VMs with nested virtualization enabled. + +## Terminology and server set-up + +Throughout this Learning Path, we will refer to the bare metal server as L0, guest VMs running on the bare metal server as L1 VMs, and nested VMs as L2 VMs. We will be starting two VMs on the host, which we will call `fedora-l1-guest` and `fedora-l1-hyper`, and inside `fedora-l1-hyper` we will create another VM called `fedora-l2-guest`. Both "guest" VMs will be 8-core virtual machines, and the `fedora-l1-hyper` VM will be assigned 16 cores. + +This will allow us to allocate similar amounts of resources, and pin those resources to specific cores, to minimize any potential resource conflicts across the bare metal host and the VMs when we are comparing the performance of a reference benchmark at the end. + +Nested virtualization: +![Server layout with Nested Virtualization#center](nv2_stack.png "Figure 1. Nested virtualization") + + +## Installing the virtualization software stack + +To run a nested VM on Arm64, you need both a recent Linux kernel (6.13 or later, preferably 7.2 or later) and a recent version of qemu-kvm (version 10.1.0 or later), and all of the other virtualization tooling that we will need to create and run virtual machines using virsh and NV2_stack + +1. Connect to a host running Fedora 44 or later, and install prerequisite packages: + ``` + sudo dnf -y install libvirt libvirt-daemon-qemu libvirt-client qemu-kvm virt-install dnsmasq genisoimage + ``` +2. Enable nested virtualization on the host by passing the kernel argument `kvm-arm.mode=nested` to the Linux kernel at boot time, then reboot the server: + ``` + sudo grubby --args="kvm-arm.mode=nested" --update-kernel=ALL + sudo reboot + ``` +3. Verify after reboot that capability is available: + ``` + sudo dmesg | grep -i "nested\|vhe\|nv" + # Expected: "CPU features: detected: Nested Virtualization Support" + # Expected: "kvm [1]: VHE+NV2 mode initialized successfully" + ``` + +At the end of this step, you should have installed all of the virtualization tools that we will use to prepare and start virtual machines on the bare metal host, and confirmed that your host is now running with nested virtualization enabled. + +Next, we will create and start an L1 guest VM running Fedora 44. + diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/2-create-l1-guest.md b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/2-create-l1-guest.md new file mode 100644 index 0000000000..c3da7537b8 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/2-create-l1-guest.md @@ -0,0 +1,91 @@ +--- +title: Preparing and Starting an L1 Guest Virtual Machine +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Preparing and Starting an L1 Guest VM + +Before we create a nested VM, we will create an L1 Guest VM which will be running on our host, so that we have an unnested VM to compare the performance overhead of virtualization with the additional overnead of nested virtualization, and to compare the additional work required to create an L1 hypervisor VM at the next step. + +To create a VM, we will first download a cloud image, make some configuration changes to the base image using cloud-config, and install and start the VM using `virt-install`. Once we have started the VM, we will connect to it to confirm that it has started correctly. + +1. Download an Arm64 cloud image to use for the VMs. To be consistent with the host, we use Fedora 44, but any recent Linux distribution will do: + ``` + curl -LO https://dl.fedoraproject.org/pub/fedora/linux/releases/42/Cloud/aarch64/images/Fedora-Cloud-Base-Generic-42-1.1.aarch64.qcow2 + ``` + +2. We will make copies of our base disk image and resize it to give the VM more disk space. For convenience, we also set some environment variables: + ``` + export VMDIR="/var/lib/libvirt/images” + export IMG="${HOME}/Fedora-Cloud-Base-Generic-42-1.1.aarch64.qcow2" + export DISK_L1_GUEST="${VMDIR}/fedora-l1-guest.qcow2" + + # Prepare the L1 guest image + sudo cp "$IMG" "$DISK_L1_GUEST" + sudo qemu-img resize "$DISK_L1_GUEST" 40G + ``` + +3. Create a seed ISO image for `cloud-init`. `cloud-init` will run automatically the first time that we instantiate the virtual machine, and can perform some initialization actions for us. `cloud-init` looks for special files called `meta-data` and `user-data` for configuration information. In this case, we want to create a default unprivileged user (fedora), ensure that the user has permission to run commands as root using `sudo` without a password, and that our public SSH key is included in the `authorized_keys` of that user so that we can SSH into the virtual machine from the host. If you do not have a public/private key-pair, this is a good time to generate the pair, and use the public key file here. + + After creating these files, we will generate a special CD disk image (format ISO) which will be attached to the VM when we instantiate it. Once we have the ISO image, we copy it to the same directory as the qcow2 VM image we created in the previous step. + + ``` + cat > /tmp/user-data < + chpasswd: + expire: False + ssh_pwauth: False + EOF + + cat > /tmp/meta-data < /tmp/user-data < + chpasswd: + expire: False + ssh_pwauth: False + EOF + + cat > /tmp/meta-data < block that enables virtualization functionality in the guest. + + ``` + sudo virsh edit fedora-l1-hyper + + # In the XML file defining the virtual machine, change the opening tag to: + + + # At the end of the file, just before the tag, add the following: + + + + + + # Save the XML file, and verify that it passes validation against the libvirt schema + ``` + +6. At this point, we should be able to boot our L1 hypervisor VM and verify that virtualization is supported: + + ``` + sudo virsh start fedora-l1-hyper + # After a few seconds... + sudo virsh net-dhcp-leases default + ssh –i /path/to/private_key fedora@{ip address} + + #Inside the fedora-l1-hyper VM, verify that virtualization functionality has been passed through + sudo dmesg | grep -i "el2\|vhe\|kvm" + # Expected: "CPU: All CPU(s) started at EL2" + # Expected: "kvm [1]: VHE mode initialized successfully" + ls -la /dev/kvm + # /dev/kvm should exist inside the VM + ``` + +If you see the line “CPU: All CPU(s) started at EL2”, congratulations! You now have a virtual machine that can function as a hypervisor. We will now install virtualization tooling and instantiate our first L2 guest VM. + + diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/4-create-l2-guest.md b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/4-create-l2-guest.md new file mode 100644 index 0000000000..a1c9928c00 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/4-create-l2-guest.md @@ -0,0 +1,101 @@ +--- +title: Booting a nested L2 guest VM in the Hypervisor VM +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Prepare and boot a nested L2 guest VM inside the hypervisor VM + +This process will essentially be identical to the process we followed to create the L1 guest VM on the host. We install all the virtualziation tools we will need in our Hypervisor VM, copy the base image from the host to the fedora-l1-hyper VM, resize its disk, create a new seed ISO, and re-run virt-install for fedora-l2-guest. + +1. Copy the Fedora cloud image and private SSH key from the host into the hypervisor using scp. + + ``` + # In the host: + scp –i /path/to/private/key $IMG /path/to/private/key fedora@{fedora-l1-hyper IP address}:~/ + ``` + +2. Log in to the `fedora-l1-hyper` VM, and install the same virtualization software stack as we installed on the host: + ``` + sudo dnf -y install libvirt libvirt-daemon-qemu libvirt-client qemu-kvm virt-install dnsmasq genisoimage + ``` + +3. After the installation has completed, restart the hypervisor VM from the host to start virtualization services. You will need to wait a minute to be able to connect again with SSH: + ``` + # In the host: + sudo virsh reboot fedora-l1-hyper + # After boot is complete + ssh -i /path/to/private/key fedora@{fedora-l1-hyper IP address} + ``` + +4. After the VM restarts, connect to it again and create the L2 guest VM image: + ``` + export VMDIR="/var/lib/libvirt/images" + export IMG="/home/fedora/Fedora-Cloud-Base-Generic-44-1.7.aarch64.qcow2" + export DISK_L2_GUEST="/var/lib/libvirt/images/fedora-l2-guest.qcow2" + + # Prepare the L2 guest image + sudo cp "$IMG" "$DISK_L2_GUEST" + sudo qemu-img resize "$DISK_L2_GUEST" 40G + ``` + +5. Create a seed ISO image for `cloud-config` similar to the other VMs. + + ``` + cat > /tmp/user-data < + chpasswd: + expire: False + ssh_pwauth: False + EOF + + cat > /tmp/meta-data <` with the domain name for each of the VMs (`fedora-l1-guest`, etc): + ``` + sudo virsh vcpuinfo + sudo virsh emulatorpin + ``` +4. Install the sysbench benchmarking tool in all sustems under test (the host, the L1 guest, and the L2 guest): + ``` + sudo dnf install -y sysbench + ``` +5. On each of the systems under test, run the same benchmark test. You can run these in 3 different terminals, at the same time, and since they are using different cores, they should not significantly interfere with each other: + ``` + # On the host: + taskset -c 8-15 sysbench cpu --cpu-max-prime=20000 --threads=8 --time=60 run + + # On the L1 and L2 guests + sysbench cpu --cpu-max-prime=20000 --threads=8 --time=60 run + ``` + +In our tests, the results were as follows: + +| Metric | Bare Metal | L1 KVM Guest | L2 Nested Guest | +| ------ | ---------- | ------------ | --------------- | +| Events/sec | 2533.02 | 2525.84 | 2306.22 | +| Overhead vs bare metal | — | -0.3% | -8.9% | +| Min latency | 3.14ms | 3.15ms | 3.38ms | +| Avg latency | 3.16ms | 3.17ms | 3.47ms | +| Max latency | 9.16ms | 13.02ms | 19.72ms | +| 95th percentile | 3.13ms | 3.19ms | 3.55ms | +| Thread stddev | 52.18 | 41.46 | 73.47 | + +We see some performance drop-off for the nested VM in events/sec (about 10% between bare metal and nested virtual machine), and there is a significant difference in max latency across the three levels, but the average and P95 latencies are very consistent across all three systems under test. + +What this shows is that while latency-sensitive workloads may not be a good match for nested VMs in performance critical situations, the performance of applications is good enough for the test & development environments where nested virt is the best fit. + diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_index.md b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_index.md new file mode 100644 index 0000000000..0a5a2746d8 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_index.md @@ -0,0 +1,66 @@ +--- +title: Nested Virtualization on an Arm64 Server + +draft: true +cascade: + draft: true + +description: Configure an Arm server to allow guest virtual machines to act as hypervisors, enabling users to run virtual machines inside a virtual machine. + +minutes_to_complete: 20 + +who_is_this_for: This is an advanced topic for System Administrators who are interested in enabling users to run VMs inside virtual machines. + +learning_objectives: + - Starting a Virtual Machine on Linux using virsh + - Configuring a host and guest VM to allow the guest VM to function as a hypervisor + - Using a standard benchmark suite, measure the performance impact of running software inside a nested virtual machine against a regular guest VM and on bare metal + +prerequisites: + - An Arm64 server supporting FEAT_NV2 (available in Arm v8.4-A and later) + - Fedora 44 (or any distribution with a version 7.2 or later Linux kernel) + +author: David Neary + +# New Learning Paths are opted in for the next manual generated summary/FAQ run. +# The generator resets this to false after a successful write. +generate_summary_faq: true + +# Optional one-shot controls: set either field to true to regenerate just that +# generated section the next time the summary/FAQ tool runs. The tool resets +# them to false after a successful write. +rerun_summary: false +rerun_faqs: false + +### Tags +skilllevels: Advanced +subjects: Containers and Virtualization +armips: + - Neoverse +tools_software_languages: + - KVM + - libvirt + - virsh / virt-manager +operatingsystems: + - Linux + +further_reading: + - resource: + title: PLACEHOLDER MANUAL + link: PLACEHOLDER MANUAL LINK + type: documentation + - resource: + title: PLACEHOLDER BLOG + link: PLACEHOLDER BLOG LINK + type: blog + - resource: + title: PLACEHOLDER GENERAL WEBSITE + link: PLACEHOLDER GENERAL WEBSITE LINK + type: website + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/nested-virtualization/nv2_stack.png b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/nv2_stack.png new file mode 100644 index 0000000000..f1be7a3af0 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/nested-virtualization/nv2_stack.png differ