Roobyz Ramblings


A responsive site for sharing thoughts and lessons learned on data science, programming, and technology.


PVE: Virtualization for Work and Play (Part 1)

The Plan…​

Say we want a powerful "bang for the buck" home server for games and other system-intensive pursuits. We may want to run powerful analytics applications which would undoubtedly require Linux, but we may also want to run Windows applications. We may want near native 2D and 3D graphics performance inside the guest operating system (OS) while making dual-booting obsolete. Finally, we may want to do all of that from the comfort of our couch using a Windows, Linux or Mac laptop. Lets do it!

Introduction

Hardware virtualization allows multiple operating systems to simultaneously share processor resources. With the open source server management solution, Proxmox Virtual Environment (PVE), we can leverage hardware virtualization to achieve our goals. PVE enables the creation of multiple virtual OS "servers" via a Web GUI; as many as our hardware setup will allow. This guide will document the setup of PVE on the following hardware:

  • AMD Ryzen 7 1600 (8 cores, 16 threads @ 3.7GHz)

  • 64GB 2400MHz DDR4

  • Boot Drive: 1x 512GB NVMe SSD

  • Storage (Striped-mirrored ZFS):

    • 2x 1TB SATA SSD (striped)

    • 1x 2TB 7200rpm mechanical drives (mirrored)

So what’s the difference between a VM and a container anyway, and how do we choose between them? A VM is computer software that emulates a particular computer hardware system and requires an OS to function. In other words, VMs "pretend" to be an actual computer of the type that we specify and will need to have a Guest OS like Windows or Linux running. Containers are software that emulates the Host OS, to enable software to run predictably.

Diagram 1: Comparison of a VM & Container on One Machine vms-cnt

If we want to run multiple applications on one server, to have increased security, or to run an operating system that is different from our host system, then a VM is our choice. To run different versions of an application (i.e RStudio) and validate reproducibility and reliability, then we want to use containers. Compared to VMs, containers are quicker, "lighter weight" and more transient so they can be readily packaged, shared, and moved to other hardware.

Hardware Considerations

Our CPU and motherboard must support "virtualization” (SVM) and IOMMU, which needs to be enabled in firmware for resource sharing. Also, we should have 32GB of RAM or more, so that we can reserve at least 16GB for a single virtual machine (VM) and still have enough memory left over for PVE and potentially other VMs running simultaneously.

While most of our computer hardware can be shared between multiple VMs, the graphics card (GPU) may not readily be shared, so we’ll need at least two GPUs:

  1. One GPU for PVE (the host);

  2. One powerful GPU for our VMs (the guests: Windows, Linux, etc.).

Software Considerations

PVE 5.0 is based in Debian Linux (Stretch). Since our Ryzen hardware is rather new, our host system needs to have a Linux kernel version 4.10 or later. Although in beta at the time of this writing, PVE 5.0 has better support for Ryzen than PVE 4.4.

PVE natively supports both KVM for hardware virtualization and LXC containers for Linux system virtualization. Since the guest systems can run under hardware virtualization, we get some added bonuses. For example, we can benefit from Ryzen hardware and still get Windows 7 updates. We would need to identify our Windows Universally Unique Identifier (UUID) so that it may be identical on our VM. Otherwise, Microsoft may think that we have a new version of Windows that needs to be registered.

We will use ZFS, a storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, and performance. Our ZFS installation will be compressed and striped: our two SSD drives will run in parallel and require less storage space, which improves read/write performance. In addition, our ZFS will be mirrored: our SSD drives will be cloned so that we have a backup in case of drive failure.

lightbulb.png

About That: KVM supports multiple disk formats; raw images, the native QEMU format (qcow2), VMware format, and many more. When working with ZFS on PVE, we need to use raw images. It may not seem obvious at first, but we can easily convert an existing KVM file from one format to a raw image. Near the end of this guide, we’ll cover the process to convert a qcow2 format to the required PVE raw image.

Next Steps…​

This is Part 1 of a multipart tutorial. The next two part will cover installation of PVE and server tweaks we can make to improve performance of our VMs and containers:

comments powered by Disqus