Initial version of Nested Virtualization learning path

content/learning-paths/servers-and-cloud-computing/nested-virtualization/1-preparation.md

@@ -0,0 +1,52 @@
+---
+title: Preparing the Host for Nested Virtualization
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Why use Nested Virtualization?
+
+Virtualization allows organizations to mutualize large multi-core servers into smaller environments with a strong level of resource isolation. It is a foundational technology for cloud computing, and Cloud Service Providers typically provide compute resources to their customers using virtualization.
+
+Nested virtualization enables guest virtual machines to serve as hypervisors, and run guest virtual machines inside a VM or cloud instance. There are a few use-cases where this facility is very useful:
+
+* **Resource isolation:** When running container applications in cloud instances, containers share a common operating system. By running containers inside microVMs like Firecracker VM, the container workloads share nothing with the host operating system 
+* **Mobile application development and testing:** It is very desirable to build and test Android applications on cloud infrastructure like Kubernetes. Cloud hosted Kubernetes compute nodes are typically virtual machine instances. Running Android environments in Kubernetes in this situation requires nested virtualization. 
+* **Testing and development of Embedded applications:** Many embedded applications are running specialized real-time or embedded operating systems, rather than Linux. Developing and testing these environments requires virtualization, which means developing on bare metal or VMs with nested virtualization enabled.
+
+## Terminology and server set-up
+
+Throughout this Learning Path, we will refer to the bare metal server as L0, guest VMs running on the bare metal server as L1 VMs, and nested VMs as L2 VMs. We will be starting two VMs on the host, which we will call `fedora-l1-guest` and `fedora-l1-hyper`, and inside `fedora-l1-hyper` we will create another VM called `fedora-l2-guest`. Both "guest" VMs will be 8-core virtual machines, and the `fedora-l1-hyper` VM will be assigned 16 cores.
+
+This will allow us to allocate similar amounts of resources, and pin those resources to specific cores, to minimize any potential resource conflicts across the bare metal host and the VMs when we are comparing the performance of a reference benchmark at the end.
+
+Nested virtualization:
+![Server layout with Nested Virtualization#center](nv2_stack.png "Figure 1. Nested virtualization")
+
+
+## Installing the virtualization software stack 
+
+To run a nested VM on Arm64, you need both a recent Linux kernel (6.13 or later, preferably 7.2 or later) and a recent version of qemu-kvm (version 10.1.0 or later), and all of the other virtualization tooling that we will need to create and run virtual machines using virsh and NV2_stack
+
+1. Connect to a host running Fedora 44 or later, and install prerequisite packages:
+   ```
+    sudo dnf -y install libvirt libvirt-daemon-qemu libvirt-client qemu-kvm virt-install dnsmasq genisoimage
+   ```
+2. Enable nested virtualization on the host by passing the kernel argument `kvm-arm.mode=nested` to the Linux kernel at boot time, then reboot the server:
+   ```
+    sudo grubby --args="kvm-arm.mode=nested" --update-kernel=ALL 
+    sudo reboot
+   ```
+3. Verify after reboot that capability is available:
+   ```
+    sudo dmesg | grep -i "nested\|vhe\|nv" 
+    # Expected: "CPU features: detected: Nested Virtualization Support" 
+    # Expected: "kvm [1]: VHE+NV2 mode initialized successfully"
+   ```
+
+At the end of this step, you should have installed all of the virtualization tools that we will use to prepare and start virtual machines on the bare metal host, and confirmed that your host is now running with nested virtualization enabled.
+
+Next, we will create and start an L1 guest VM running Fedora 44.
+

content/learning-paths/servers-and-cloud-computing/nested-virtualization/2-create-l1-guest.md

@@ -0,0 +1,91 @@
+---
+title: Preparing and Starting an L1 Guest Virtual Machine
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Preparing and Starting an L1 Guest VM
+
+Before we create a nested VM, we will create an L1 Guest VM which will be running on our host, so that we have an unnested VM to compare the performance overhead of virtualization with the additional overnead of nested virtualization, and to compare the additional work required to create an L1 hypervisor VM at the next step.
+
+To create a VM, we will first download a cloud image, make some configuration changes to the base image using cloud-config, and install and start the VM using `virt-install`. Once we have started the VM, we will connect to it to confirm that it has started correctly.
+
+1. Download an Arm64 cloud image to use for the VMs. To be consistent with the host, we use Fedora 44, but any recent Linux distribution will do:
+   ```
+      curl -LO https://dl.fedoraproject.org/pub/fedora/linux/releases/42/Cloud/aarch64/images/Fedora-Cloud-Base-Generic-42-1.1.aarch64.qcow2 
+   ```
+
+2. We will make copies of our base disk image and resize it to give the VM more disk space. For convenience, we also set some environment variables: 
+   ``` 
+    export VMDIR="/var/lib/libvirt/images” 
+    export IMG="${HOME}/Fedora-Cloud-Base-Generic-42-1.1.aarch64.qcow2" 
+    export DISK_L1_GUEST="${VMDIR}/fedora-l1-guest.qcow2" 
+
+    # Prepare the L1 guest image 
+    sudo cp "$IMG" "$DISK_L1_GUEST" 
+    sudo qemu-img resize "$DISK_L1_GUEST" 40G 
+   ```
+
+3. Create a seed ISO image for `cloud-init`. `cloud-init` will run automatically the first time that we instantiate the virtual machine, and can perform some initialization actions for us. `cloud-init` looks for special files called `meta-data` and `user-data` for configuration information. In this case, we want to create a default unprivileged user (fedora), ensure that the user has permission to run commands as root using `sudo` without a password, and that our public SSH key is included in the `authorized_keys` of that user so that we can SSH into the virtual machine from the host. If you do not have a public/private key-pair, this is a good time to generate the pair, and use the public key file here.
+
+   After creating these files, we will generate a special CD disk image (format ISO) which will be attached to the VM when we instantiate it. Once we have the ISO image, we copy it to the same directory as the qcow2 VM image we created in the previous step.
+
+   ```
+    cat > /tmp/user-data <<EOF 
+    #cloud-config 
+    users: 
+     - name: fedora 
+       sudo: ALL=(ALL) NOPASSWD:ALL 
+       shell: /bin/bash 
+       ssh_authorized_keys: 
+         - <your-public-key> 
+    chpasswd: 
+      expire: False 
+    ssh_pwauth: False 
+    EOF 
+
+    cat > /tmp/meta-data <<EOF 
+    instance-id: my-guest 
+    local-hostname: my-guest 
+    EOF 
+
+    SEEDTMP=$(mktemp -d) 
+    cp /tmp/user-data "$SEEDTMP/user-data" 
+    cp /tmp/meta-data "$SEEDTMP/meta-data" 
+    genisoimage -output ${HOME}/seed_guest.iso \ 
+         -volid cidata -joliet -rock -input-charset utf-8 \ 
+         "$SEEDTMP/user-data" "$SEEDTMP/meta-data" 
+    sudo cp ${HOME}/seed_guest.iso ${VMDIR} 
+    rm -rf "$SEEDTMP" 
+   ```
+
+4. Start the L1 guest VM using `virt-install`. The `virt-install` command includes parameters for our VM image, the ISO CD image we created for `cloud-config`, a "name" parameter to identify the image with `virsh` and `virt-manager`, and parameters for the number of virtual CPUs and amount of memory that will be allocated to the VM. The `os-variant` argument passes hints about what operating system will be running - at the time of writing, `fedora44` is not an accepted option, but any recent `fedoraXX` value should work.
+
+   ```
+    sudo virt-install \
+      --name fedora-l1-guest \
+      --memory 8192 \
+      --vcpus 8 \
+      --arch aarch64 \
+      --machine virt \
+      --disk path=${DISK_L1_GUEST},format=qcow2,bus=virtio \
+      --disk path=${VMDIR}/seed_guest.iso,device=cdrom,readonly=on \
+      --network network=default,model=virtio \
+      --os-variant fedora42 \
+      --import \
+      --noautoconsole 
+   ```
+
+5. After the VM is started, which takes a few seconds, we will get its IP address and connect to it using SSH.
+   ```  
+    sudo virsh net-dhcp-leases default 
+    # The IP address of the VM will be in the output of this command once it has finished booting
+    ssh –i /path/to/private_key fedora@{ip address}
+   ```
+
+At this point, you should be connected to the L1 guest VM over SSH.
+
+In the next step, we will instantiate a hypervisor VM, which will require some additional configuration steps. Once you have verified that we can connect to the L1 Guest VM, you can exit the shell to return to the host environment.
+

content/learning-paths/servers-and-cloud-computing/nested-virtualization/3-create-l1-hyper.md

@@ -0,0 +1,122 @@
+---
+title: Creating a Hypervisor virtual machine
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Creating a Hypervisor virtual machine on the host
+
+At this point, you have already used command line tools to create, boot, and connect to a guest virtual machine running on your Arm64 host. By the end of this step, you will have another virtual machine running on our host which can serve as a hypervisor for nested virtual machines. To complete this step, you will connect to it over SSH, and verify that it is capable of running a virtual machine.
+
+This process is essentially the same as the previous step up to the point that we get a VM running.
+
+1. Set similar convenience environment variables to the previous step. Our L1 hypervisor will need a bigger disk than the L1 guest, since it will contain an identical 40GB L2 guest image inside it: 
+
+   ``` 
+    # If these environment variables are already set, there is no need to re-set them
+    export VMDIR="/var/lib/libvirt/images”
+    export IMG="${HOME}/Fedora-Cloud-Base-Generic-42-1.1.aarch64.qcow2"
+
+    # Use a different QCOW2 filename for the hypervisor VM
+    export DISK_L1_HYPER="${VMDIR}/fedora-l1-hyper.qcow2"
+
+    # Prepare the L1 hypervisor image
+    sudo cp "$IMG" "$DISK_L1_HYPER"
+    sudo qemu-img resize "$DISK_L1_HYPER" 80G
+   ```
+2. Create a seed ISO image for the guest hypervisor for cloud-init. We will use the same SSH key pair that we generated or used in the previous step - the main change here is the hostname of the guest. 
+
+   ```
+    cat > /tmp/user-data <<EOF 
+    #cloud-config 
+    users: 
+     - name: fedora 
+       sudo: ALL=(ALL) NOPASSWD:ALL 
+       shell: /bin/bash 
+       ssh_authorized_keys: 
+         - <your-public-key> 
+    chpasswd: 
+      expire: False 
+    ssh_pwauth: False 
+    EOF 
+
+    cat > /tmp/meta-data <<EOF 
+    instance-id: my-hyper 
+    local-hostname: my-hyper 
+    EOF 
+
+    SEEDTMP=$(mktemp -d) 
+    cp /tmp/user-data "$SEEDTMP/user-data" 
+    cp /tmp/meta-data "$SEEDTMP/meta-data" 
+    genisoimage -output ${HOME}/seed_hyper.iso \
+         -volid cidata -joliet -rock -input-charset utf-8 \
+         "$SEEDTMP/user-data" "$SEEDTMP/meta-data"
+    sudo cp ${HOME}/seed_hyper.iso ${VMDIR} 
+    rm -rf "$SEEDTMP" 
+   ```
+
+3. Start the L1 hypervisor VM. 
+
+   ```
+    sudo virt-install \
+      --name fedora-l1-hyper \
+      --memory 8192 \
+      --vcpus 8 \
+      --arch aarch64 \
+      --machine virt \
+      --disk path=${DISK_L1_HYPER},format=qcow2,bus=virtio \
+      --disk path=/path/to/seed_hyper.iso,device=cdrom,readonly=on \
+      --network network=default,model=virtio \
+      --os-variant fedora42 \
+      --import \
+      --noautoconsole
+   ```
+
+4. After the VM has started and has obtained an IP address, we need to shut it down to make some changes to the XML that defined the VM. This step passes a kernel parameter to the VM to enable passthrough of virtualization capability which is what turns our guest into a hypervisor.
+   ```
+    # Check that VM is fully initialized first
+    sudo virsh net-dhcp-leases default 
+
+    # Once this command shows that the VM has an IP address, proceed
+    sudo virsh shutdown fedora-l1-hyper 
+    sudo virsh list –-all # Verify that VM status is “shut off” 
+   ```
+
+5. Next we edit the XML configuration describing the VM and make two changes. First, we import the XML schema for the qemu namespace which allows us to specify additional parameters to the qemu command that actually instantiates the VM. Second, we define a <qemu:commandline> block that enables virtualization functionality in the guest. 
+
+   ```
+    sudo virsh edit fedora-l1-hyper 
+
+    # In the XML file defining the virtual machine, change the opening <domain> tag to: 
+    <domain type=’kvm’ xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' >
+
+    # At the end of the file, just before the </domain> tag, add the following:
+    <qemu:commandline>
+      <qemu:arg value='-machine'/>
+      <qemu:arg value='virt,virtualization=on'/>
+    </qemu:commandline>
+
+    # Save the XML file, and verify that it passes validation against the libvirt schema
+   ```
+
+6. At this point, we should be able to boot our L1 hypervisor VM and verify that virtualization is supported:
+
+   ```
+    sudo virsh start fedora-l1-hyper
+    # After a few seconds...
+    sudo virsh net-dhcp-leases default
+    ssh –i /path/to/private_key fedora@{ip address}
+
+    #Inside the fedora-l1-hyper VM, verify that virtualization functionality has been passed through
+    sudo dmesg | grep -i "el2\|vhe\|kvm"
+    # Expected: "CPU: All CPU(s) started at EL2"
+    # Expected: "kvm [1]: VHE mode initialized successfully"
+    ls -la /dev/kvm
+    # /dev/kvm should exist inside the VM
+   ```
+
+If you see the line “CPU: All CPU(s) started at EL2”, congratulations! You now have a virtual machine that can function as a hypervisor. We will now install virtualization tooling and instantiate our first L2 guest VM.
+
+

content/learning-paths/servers-and-cloud-computing/nested-virtualization/4-create-l2-guest.md

@@ -0,0 +1,101 @@
+---
+title: Booting a nested L2 guest VM in the Hypervisor VM
+weight: 6
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Prepare and boot a nested L2 guest VM inside the hypervisor VM 
+
+This process will essentially be identical to the process we followed to create the L1 guest VM on the host. We install all the virtualziation tools we will need in our Hypervisor VM, copy the base image from the host to the fedora-l1-hyper VM, resize its disk, create a new seed ISO, and re-run virt-install for fedora-l2-guest. 
+
+1. Copy the Fedora cloud image and private SSH key from the host into the hypervisor using scp.
+
+   ```
+    # In the host: 
+    scp –i /path/to/private/key $IMG /path/to/private/key fedora@{fedora-l1-hyper IP address}:~/ 
+   ```
+
+2. Log in to the `fedora-l1-hyper` VM, and install the same virtualization software stack as we installed on the host: 
+   ```
+    sudo dnf -y install libvirt libvirt-daemon-qemu libvirt-client qemu-kvm virt-install dnsmasq genisoimage 
+   ```
+
+3. After the installation has completed, restart the hypervisor VM from the host to start virtualization services. You will need to wait a minute to be able to connect again with SSH: 
+   ```
+    # In the host:
+    sudo virsh reboot fedora-l1-hyper 
+    # After boot is complete
+    ssh -i /path/to/private/key fedora@{fedora-l1-hyper IP address}
+   ```
+
+4. After the VM restarts, connect to it again and create the L2 guest VM image:
+   ```
+    export VMDIR="/var/lib/libvirt/images" 
+    export IMG="/home/fedora/Fedora-Cloud-Base-Generic-44-1.7.aarch64.qcow2" 
+    export DISK_L2_GUEST="/var/lib/libvirt/images/fedora-l2-guest.qcow2" 
+
+    # Prepare the L2 guest image 
+    sudo cp "$IMG" "$DISK_L2_GUEST" 
+    sudo qemu-img resize "$DISK_L2_GUEST" 40G 
+   ```
+
+5. Create a seed ISO image for `cloud-config` similar to the other VMs. 
+
+   ```
+    cat > /tmp/user-data <<EOF 
+    # cloud-config 
+    users:
+     - name: fedora
+       sudo: ALL=(ALL) NOPASSWD:ALL
+       shell: /bin/bash
+       ssh_authorized_keys:
+         - <your-public-key>
+    chpasswd:
+      expire: False
+    ssh_pwauth: False
+    EOF
+
+    cat > /tmp/meta-data <<EOF
+    instance-id: my-l2-guest
+    local-hostname: my-l2-guest
+    EOF
+
+    SEEDTMP=$(mktemp -d)
+    cp /tmp/user-data "$SEEDTMP/user-data"
+    cp /tmp/meta-data "$SEEDTMP/meta-data"
+    genisoimage -output ${HOME}/seed_guest.iso \
+         -volid cidata -joliet -rock -input-charset utf-8 \
+         "$SEEDTMP/user-data" "$SEEDTMP/meta-data"
+    sudo cp ${HOME}/seed_guest.iso ${VMDIR}
+
+    rm -rf "$SEEDTMP"
+   ```
+
+6. Now start the L2 guest VM and verify that you can connect: 
+   ```
+    sudo virt-install \
+      --name fedora-l2-guest \
+      --memory 8192 \
+      --vcpus 8 \
+      --arch aarch64 \
+      --machine virt \
+      --disk path=${DISK_L2_GUEST},format=qcow2,bus=virtio \
+      --disk path=/path/to/seed_guest.iso,device=cdrom,readonly=on \
+      --network network=default,model=virtio \
+      --os-variant fedora43 \
+      --import \
+      --noautoconsole
+   ``` 
+
+7. After the VM is started, find its IP address and connect to it with SSH.
+   ``` 
+    sudo virsh net-dhcp-leases default
+    ssh –i /path/to/private_key fedora@{fedora-l2-guest ip address}
+   ```
+
+Congratulations! You are now connected to an L2 guest VM  running inside an L1 hypervisor VM over SSH.  
+
+In the next step, we will use a common synthetic benchmark, sysbench, to measure the overhead of running an application inside a nested VM. To minimize any resource conflicts or CPU scheduling issues, we will pin all VMs to a fixed set of cores.
+