pr1ntf.xyz
Whinings on BSD, virtualization, and other stuff.
pr1ntf.xyz
Installing Windows 10 Under the bhyve Hypervisor.

Install Windows 10 using VNC on FreeBSD 11 and above.

This October 2016 will be a special month for FreeBSD virtualization. Not only will the most recent release of FreeBSD be ready, but it will have been a year since UEFI booting in bhyve was announced via the FreeBSD-Virtualization Mailing List. At the time, bhyve did not have the ability to allow for any type of graphical console, outside of something run on the guest OS like RDP, VNC, or SPICE. Instead, bhyve used a serial console as a means to communicate with the guest operating system. Most UNIX operating systems these days have the ability to have a VT100-like console. If you have ever had to consle into a Cisco Switch or UPS Battery using puTTY, you've probably encountered something similar. Windows has a simliar console called Emergency Management Services that allows you to do various administration tasks like change networking and the ablility to run CMD.EXE over the serial console. This EMS feature comes standard on official Windows Server Edition installation discs. It is not available by default, and must be enabled by using Windows Unattended XML file baked into the installation disc. At first glance this is a tedious process, but with the help of scripting, it can be easy to accomplish over and over again. If you want to enable the EMS on a regular Windows Desktop OS, it even gets tricker as you need to copy the EMS files from a Server Edition disc over to a Desktop Edition.

This changed in the Spring of 2016 when official support for UEFI-GOP which allows bhyve to attach a graphical console to the guest OS. This means you can now install Windows the old fashioned way by clicking "Next" a bunch of times. Since these changes made it to the FreeBSD HEAD branch before the FreeBSD 11 release process began, these features are apart of FreeBSD 11 RELEASE. For this tutorial, we are going to assume that you are running at least FreeBSD 11 RELEASE or newer. If you are using a FreeBSD derivative like TrueOS you should be fine as long as it is based off of FreeBSD 11 or newer. I will be doing this on a machine tracking the 12 CURRENT HEAD branch, but running FreeBSD 11 should work just fine. This tutorial is split into 3 sections:

  • What bhyve is and how to set up your host operating system.
  • Obtaining your guest operating system and preparing for installation
  • Installing and using Windows 10

If you want to follow along at home, you should already know a little bit about UNIX like operating systems like Solaris or ones that use the Linux Kernel. In theory, if you have used a Linux Distribution in the past, you should be able to install FreeBSD and follow this tutorial.

Preparing the Host Operating System

The FreeBSD bhyve Hypervisor first appeared in FreeBSD 10 and has grown quite extensively since then. It is a relativly new hypervisor, being younger than Xen, Linux KVM, esxi, VirtualBox, and others. The bhvye hypervisor was also ported to Macintosh OS X as xhyve which is now used by Docker on Mac. The bhyve hypervisor consists of two main components: the VMM kernel module and the bhyve userland utility. There are other userland utilities like grub2-bhyve and bhyveload that use libvmmapi, but we won't go over these in this tutorial.

Before prerparing your host, I suggest you read the FreeBSD handbook entry regarding bhyve here. Pay special attention to the sections regarding CPU compatability and Section 21.7.1 Preparing the Host. The bhyve hypervisor only works on certain models of CPUs, so be sure to check /var/run/dmesg.boot to make sure your CPU will run bhyve virtual machines. Once you have determined that your hardware can handle bhyve, we can start to prepare the host operating system's kernel and network configuration. As outlined in the handbook, we need to load the VMM kernel module first. You can do this by simply running kldload vmm with super user credientials, or by editing your /boot/loader.conf as to avoid having to load the VMM kernel module manually everytime you reboot your host. Next we need to set up the networking on the host so the virtual machine can reach the internet. Since we are not going to set up any firewalling or NAT, the virtual machine will appear to be on the same network as the host in this situation.

To do this, we are going to create a network tap device, create a network bridge, then attach the our network interface and tap to the bridge. This sounds complicated, but if you take it a step at a time, it's easier to understatnd. Before we begin, we need to see what our primary network interface is by running ifconfig. From my output, I can see that igb0 is my primary network interface, as it has an IP address. Your network interface may be something like em0 or something similar. If you are using WiFi, that is a bit more complicated, and involves creating a NAT and forwarding traffic around. For the purposes of this tutorial, we won't be going into that. Instead we will deal with physical ethernet connections. Now that we know our interface, we can start by creating the network tap. I have multiple taps created already, so I am going to choose a higher number than 0 and number my tap tap42. You can choose any number you like, as long as it is not already in use. Only one virtual machine can utilitze a network tap at a time. To create it, I run ifconfig tap42 create. You should now see tap42 in your ifconfig output.

Next, we need to tell the kernel to bring a tap device up when it is opened. This is so we don't have to run ifconfig tap42 up everytime we start the virtual machine. We do this by running sysctl net.link.tap.up_on_open=1 with super user credientials. You can also set this in /etc/sysctl.conf so the change stays after reboot. Now we need to create our bridge. I am going to create a new bridge0. You can create any bridge number you like. We create the bridge with ifconfig bridge0 create, and add our devices with ifconfig bridge0 addm igb0 addm tap42. Note the use of igb0, this may be different depending on what your interface is. To finish things up, we bring the bridge interface up with ifconfig bridge0 up. You can refer to the handbook page on how to edit your /etc/rc.conf to make these network changes persist after rebooting.

I have simplified this process by writing simple scripts to do the heavy lifting for me. If you take a look my GitHub repo YetAnotherBhyveScript or yabs here you can find the hostprep.sh script to create the bridge and enable the VMM kernel module. Be sure to edit the file before running so it matches your desired network cofiguration. Once hostprep.sh is run, you can setup the network for you guest by attaching it to the bridge you set up using prepyabs.sh. The outputs of hostprep.sh and prepyabs.sh are below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/sh

# Prepare host for running bhyve (v0.2)

iface=igb0
bridge=bridge0

# Load kernel module
kldload vmm
kldload if_tap
kldload if_bridge

# Set sysctl
sysctl net.link.tap.up_on_open=1

# Create bridge
ifconfig ${bridge} create

# Attach interfaces
ifconfig ${bridge} addm ${iface}

# Bring up bridge
ifconfig ${bridge} up
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/sh

# Prepare host for running bhyve guest (v0.2)

tap=tap42
bridge=bridge0

# Create the tap interface
ifconfig ${tap} create

# Attach interfaces
ifconfig ${bridge} addm ${tap}

Obtaining Windows and Preparing for Installation

Before diving into the installation, we still need to obtain a Windows Installation Disc and the correct bhyve UEFI firmware. You can now simply obtain a copy of the UEFI binary by installing a package with a command like pkg install bhyve-firmware. Once the package is installed, a copy for the UEFI binary is dropped into /usr/local/share/uefi-firmware. To make things a bit easier, I tend to copy the BHYVE_UEFI.fd firmware to my working directory. Since I am working in the directory ~/yabs I must run cp /usr/local/share/uefi-firmware/BHYVE_UEFI.fd ~/yabs. Next we will need to download a Windows 10 ISO image from Microsoft. After following the directions on the page, select the 64-bit version. Here's where it can get tricky. If you are not using a GUI on your FreeBSD host, you will have to download the ISO from a browser on another computer, then use something like scp to copy it over to the FreeBSD host. It is not as simple as using fetch to grab the ISO from Microsoft. Since my other host is a Macintosh I can download via Chrome, then use scp to copy over to my FreeBSD host. Once you have copied over your Windows 10 ISO, put it into your working directory, in my case, ~/yabs. If you are on Windows, there are some scp clients out There but I would use the FileZilla client instead.

Next, we need to create the virtual hard drive that Windows will install on. We accomplish this by creating an empty file with truncate. Since the Windows System Requirements calls for at least 20GB for the 64bit version, we need to create a 20GB or larger virtual hard drive. I'm going to give myself some room and create a 24GB drive with truncate -s 24G win10.img. Now we will go into the actual bhyve command we will run to start the installation. To simplify things, I created new yabs.sh shell script that will do this for us. Let's dig in and see how things work under the hood before we run it first, as we should anytime we download a shell script from the internet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/bin/sh

# Yet Another bhyve Script v0.2

name=win10
ram=2048M
cpu=2
disk=win10.img
media=Win10_1607_English_x64.iso
mediatype=cd
tap=tap42
fw=BHYVE_UEFI.fd
ip=127.0.0.1
port=5901
w=wait

bhyve \
  -c ${cpu} -m ${ram} \
  -H -w \
  -s 0,hostbridge \
  -s 1,ahci-${mediatype},${media} \
  -s 2,ahci-hd,${disk} \
  -s 4,lpc \
  -l bootrom,${fw} \
  -s 8,virtio-net,${tap} \
  -s 16,fbuf,tcp=${ip}:${port},${w} \
  -s 17,xhci,tablet \
  ${name} &

If you have used the yabs.sh script from one of my previous posts, this version will look very familiar. As you can see, we start by defining some variables that make up part of the bhyve command we will run. We will name our virtual machine win10 and give it 2048M of RAM with 2 virtual CPUs. The disk name is from earlier when we used truncate to create it, win10.img. The installation media will be our Windows 10 ISO from Microsoft (Win10_1607_English_x64.iso), and then we tell bhyve that it should read it as a cd not an hd. Next we tell it to attach the tap device we set up earlier in preyabs.sh, tap42, and then we tell it where to look for the bhyve bootrom firmware. If you didn't move the firmware to your working directory, you can specify it's full path here instead. After that, we finally get to setting up the VNC portion of the command. We tell the VNC server to attach to IP 127.0.0.1 at port 5901, and then we tell it to wait. If we specify the wait that means we don't want bhyve to start the virtual machine until we have made a connection to the VNC server. This comes in very handy at times. One of the variables you won't see listed in the variables is the xhci,tablet portion of the command. This tells the fbuf (VNC) function of bhyve to use a different driver for the mouse communication. Using the tablet driver, we will have much better control over the mouse via VNC. Please note that not all OS's will have support for this out of the box, including FreeBSD. For our purposes, we will include it since Windows 10 has support for it built in.

Installing Windows 10

Before we start the installation, we need to make sure we have a VNC Client ready to go. If you are on FreeBSD, you can look into the net/TightVNC port or package. If you are using a Mac, you cannot use the built-in VNC viewer, instead you must use another client, like RealVNC. Since there is currently no support for authentication to the VNC server, we don't want open it to the outside world. This is why we chose 127.0.0.1 (localhost) to bind to. Of course, if you want to connect to it from another machine, we will need to set up port forwarding over ssh. Instead of manually running something like ssh -L 5901:127.0.0.1:5901 -p4444 -N -f -l pr1ntf 192.164.42.24 we can just edit and run the script below, also included in yabs script collection. Remember, sshhost is the IP address of the host running bhyve.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/sh

# Prepare ssh tunnel (v0.2)

vncport=5901
sshport=4444
sshuser=pr1ntf
sshhost=192.168.42.24

ssh -L ${vncport}:127.0.0.1:${vncport} \
  -p ${sshport} -N -f -l ${sshuser} ${sshhost}

Once you run the tunnel.sh script or run the ssh manually, we can finally start bhvye to begin the Windows 10 install process. We can start it by either running ./yabs.sh as root, or sudo ./yabs.sh if you have sudo installed. Remember, bhyve won't automatically start with the wait options enabled, so you must connect using your VNC client. The IP address to connect to is your localhost (127.0.0.1) and port 5901 in our case. Once connected, you should see the UEFI loader followed by the words Press any key to boot from CD... so press the any key and begin your installation of Windows 10. If you have the right hardware, you should have no problems installing. I chose to install the Windows 10 Professional Edition with a Custom Install, since we are installing from scratch, not upgrade from an earlier version of Windows. While the installation is running, we can use this time to download the Virtio from The Fedora Project so we can install the network drivers when the installation is finished. Without it, your Windows 10 virtual machine won't be able to use the internet. You can fetch this ISO file from your command line using something like fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso while in your working directory.

Once your installation has moved over all the files, your virtual machine will attempt to reboot. Since there is no built in function in bhyve to reboot, it will just appear that your VM has shut down. We must start our virtual machine the same way we did before and the second part of the install will begin. You should see some loading screens and the virtual machine will reboot again. Start your virtual machine again to begin the final part of the Windows installation process. Once the loading screen is finished, you should be prompted to change some installation settings and create a new user. Once you confirm your settings you will be taken to another loading screen. Once it's done, you should see your brand new Windows Desktop! We are almost done at this point. Remember, you must install the Virtio Net Drivers in order to have an internet connection. Shut down your virtual machine from over VNC and edit your yabs.sh script to attach the virtio-win.iso disc with something like media=virtio-win.iso if your Virtio disc is in your working directory. Once you have started the virtual machine again, you can right click on the Windows Logo in the lower left hand part of the screen, and click "Device Manager." You should now see a device with a yellow triangle called "Ethernet Controller." Right click on that and select "Update Driver" then "Browse my computer for driver software." Here you can tell Windows to search your CD drive and all of it's sub directories for the driver.

That's it, you should now have a Windows 10 virtual machine with access to the internet! Because VNC can be a bit sluggish at times, I like to enable Microsoft Remote Desktop (RDP) instead. Microsoft puts out Microsoft RDP apps for a variety of platforms including Windows of course, but also Mac OS X, iOS, and even Android tablets. You can even send and receive files through RDP, as well as listen to and send audio to the virtual machine over RDP. If you are on FreeBSD, you can also check out the net/freerdp port to use RDP. If the yabs.sh script isn't your style, you can always write your own version, or you can try out one of the many bhyve mangers/wrappers out there. There is Michael Dexter's vmrc, one of my favorites, Allan Jude's bhyveucl, Matt Churchyard's vm-bhyve, my side project iohyve, and it's recent fork from Justin Holcomb chyves. Each uses different methods and techniques to store and manage bhyve virtual machines. All are really great projects that should be able to get you off the ground. Those are just the tip of the iceberg, if you look on GitHub, you can find some more bhyve projects. As always, if you think you found a bug, or if you are having problems don't hesitate to ask questions!

I always like to end these posts thanking some people who have helped me or who are just doing awesome work that is inspiring. For this post, I'd like to thank Kris Moore of the TrueOS project, and Michael Dexter of iXSystems for all of the hard work they put into the BSD community every day.


X11 Forwarding with Kali Linux and bhyve

Use Kali GUI tools under bhyve

There's a common saying in any technical field, "Use the right tool for the job." For common information security tasks, many use the toolbox known as Kali Linux. Kali has many different tools for many different tasks, many of which aren't just for information security. Personally, since I use FreeBSD on my laptop, I use Kali as my "goto" Linux OS for when FreeBSD just doesn't cut it, like when there is a tool...

Read More
Parting Words From Your Server Colocation Monkey

Sit down and shut up, I have something to say.

Today marks the end of an exciting chapter in my life; today is my last day being a colocation monkey for some of you. What's a colocation monkey you ask? It's a name for the the datacenter technician who puts their greasy hands all over your gear you put in our datacenter. For the past year and a half I've rebooted your servers, moved your KVM's, troubleshot your network issues, and swapped your precious hard...

Read More
Running Windows under FreeBSD's bhyve.

How I Learned to Stop Worrying and Love The Windows Operating System Family.

As I wrote about in the past, Windows support was coming to the FreeBSD hypervisor bhyve. Support officially came with FreeBSD 11.0-CURRENT r288524 when it was announced via the FreeBSD Virtualization mailing list. Shortly after, Windows Diplomat Michael Dexter wrote a great how-to on the FreeBSD Wiki to get Windows up and going under bhyve. The secret sauce to getting Windows running under bhyve is the new UEFI...

Read More
Bhyve and ZFS on HardenedBSD.

Running bhyve virtualization on HardenedBSD

Early in the spring of 2015, I started playing with two things: HardenedBSD and bhyve. HardenedBSD is a cool "fork" of the FreeBSD operating system utilizing various operating system hardening methods including address space layout randomization, procfs work, and more including a neat little tool to wrap it all up, called secadm. Bhyve is a FreeBSD native hypervisor, which is a big fancy way of saying FreeBSD wants...

Read More
Leibniz.

Approximate the value of pi! (real slowly)

Even though an exact value of pi may never be found, there are many ways you can try and approximate it. One way is to simply draw a circle as perfectly as you can, then divide the circumference by the diameter, pi = C/d, while using the best precision available to you. There is a neat visualization of this on the pi wikipedia entry. The better the precision you use in your...

Read More

Receive Updates

ATOM

Contacts