Early in the spring of 2015, I started playing with two things: HardenedBSD and bhyve. HardenedBSD is a cool "fork" of the FreeBSD operating system utilizing various operating system hardening methods including address space layout randomization, procfs work, and more including a neat little tool to wrap it all up, called secadm. Bhyve is a FreeBSD native hypervisor, which is a big fancy way of saying FreeBSD wants it's own KVM type thing. It's a bit in it's early stages, only officially virtualizing FreeBSD 8, 9, and 10 RELEASE/STABLE versions, FreeBSD 11-CURRENT, and OpenBSD 5.2+, NetBSD 6.1+, Linux (2.6ish+) using a different bootloader, grub2-bhyve. I haven't been running into many issues at the moment with linux flavours that use grub2 and dont use EFI to boot, although I have found success manually compiling and installing grub2 on a guest operating system. There are rumours of a UEFI bootloader and even VGA emulation support down the road, which could then run Windows. Apparently, the tech they used is full of magical voodoo or pixie dust, because not very many people talk about it.
This post is going to walk through setting up a virtualization server utilizing these two things. In my spare time, I work on this crude shell script, iohyve that makes working with bhyve a bit easier out of the box, and also stores everything in ZFS. Because I want to make sure iohyve will run on both RELEASE and CURRENT snapshots of FreeBSD, I do all of my 11.0-CURRENT testing under HardenedBSD. The code monkeys behind HardenedBSD do a pretty good job at keeping up with FreeBSD snapshots. I happen to be due for another reimage of a development server, so strap in and feel free to follow along at home.
The actual installation is pretty simple. I just grabbed the latest ISO from the HardenedBSD website and popped it in. Because of the name bhyve, and my ~~obession~~ facination with Star Trek, I will be renaming this box to "borgcube."
I tend to grab not only the
lib32stuff, I also grab the
srcstuff because I compile my own kernel, but I won't be going into that today. I suggest you look into doing this if you don't: RTFHB.
Next you'll find the partitioning screen. Nothing fancy here either, except when I run production stuff, I usually choose to run a "root on zfs" setup, so I can do something like a software RAID, but instead using ZFS mirroring. Because the amount of disks I have at my ready are limited (I use just 4x 1.0TB HDDS), I just auto-partition the first disk with UFS and save the other three for the zpool later on. You can also manually partition and make your operating system partition smaller, and add the remaining free space to the zpool later on as well. Yes, ZFS is pretty awesome, I know. Just make sure you have enough RAM to make it happy. Since we are making virtualization system, I assume you have some RAM to spare.
The rest of the installation is pretty self-explanatory, setting a static IP, adding a user, etc... Because this server will never (theoretically) be exposed to the internet, I add the user to the wheel group so I can just switch to the root user after login to administer guest operating systems. (
su -) Bhyve can only be used by root as it now stands, but the bhyve team is looking to change that in the future. After installation, I exit to a shell to install the tools I will need to begin post installation tasks. I do this via
pkg install nano tmux git, although most of the actual programs used by iohyve are built into FreeBSD, or can easily be compiled from source via the ports. In my fresh install tool box I have:
nanoI'm offically not leet enough for
tmuxI use a terminal multiplexer for two reasons, one is to feel like a hacker, the other is to manage multiple terminal sessions when dealing with multiple guest operating systems and their consoles. More on that later.
gitBecause iohyve is not in FreeBSD ports yet, you have to grab the latest version from github.
Post-installation zpool setup.
There are a few things I like to do before to fresh servers, and I'm
sure you do that too. I like to change my motd and the sshd port. Some
sudo and edit the sudoers file. After you do what you need to
do, get the zpool set up so iohyve has a place to store things. For me,
I can find out what disks I have available to me by running
ls /dev |
grep ada but you may have different disk names if you have a different
root@borgcube:~ # ls /dev | grep ada ada0 ada0s1 ada0s1a ada0s1b ada1 ada2 ada3
From my output, I can tell that the operating system is installed on
/dev/ada0 which leaves
use for my zpool. For redundancy, I will be utilizing raidz which acts
much like RAID-5.
root@borgcube:~ # zpool create borgpool raidz1 /dev/ada1 /dev/ada2 /dev/ada3
iohyve installation and prep.
- Before installing and setting up iohyve, we must prepare the host
first. Begin by adding the following to your
- Then add the following to your
- Next we set up the networking in
/etc/rc.conf. Be sure to find out the correct name for your primary ethernet adapter. I can tell mine is
em0because when I run
ifconfigI can see it has a connection. While I am in
/etc/rc.confI will also make sure
zfs_enableis set so
iohyvewill be able to use ZFS. My file looks something like this:
hostname="borgcube" ifconfig_em0="inet XXX.XXX.XXX.XXX netmask XXX.XXX.XXX.0" defaultrouter="XXX.XXX.XXX.XXX" cloned_interfaces="bridge0 tap0" ifconfig_bridge0="addm em0 addm tap0" zfs_enable="YES" sshd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO"
- We can now begin to install
make installto install the script into the
root@borgcube:~ # git clone https://github.com/pr1ntf/iohyve.git Cloning into 'iohyve'... remote: Counting objects: 383, done. remote: Total 383 (delta 0), reused 0 (delta 0), pack-reused 383 Receiving objects: 100% (383/383), 109.21 KiB | 0 bytes/s, done. Resolving deltas: 100% (165/165), done. Checking connectivity... done. root@borgcube:~ # cd iohyve/ root@borgcube:~/iohyve # make install mkdir -p /usr/local/sbin install -m 0555 iohyve /usr/local/sbin/ install rc.d/* /usr/local/etc/rc.d/
At this point, I would
rebootthe system before going any further and check to make sure everything worked.
Check to be sure the kernel modules were loaded:
root@borgcube:~ # kldstat | grep vmm && kldstat | grep nmdm 2 1 0xffffffff81c96000 3634e0 vmm.ko 3 1 0xffffffff81ffa000 50b0 nmdm.ko
- Check the if the tap sysctl is set:
root@borgcube:~ # sysctl -a | grep net.link.tap.up_on_open net.link.tap.up_on_open: 1
- Check the networking for the tap and bridge interfaces:
root@borgcube:~ # ifconfig -a | grep bridge0 && ifconfig -a | grep tap0 bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
- If you would like to run guest operating systems other than
FreeBSD, you will need
grub2-bhyve. I install from ports after running
portsnap fetch && portsnap extract:
root@borgcube:~ # find /usr/ports -name "grub2-bhyve" /usr/ports/sysutils/grub2-bhyve root@borgcube:~ # cd /usr/ports/sysutils/grub2-bhyve && make install clean ... ... ...
- Next finish setting up iohyve by telling it which zpool to use. If
you want to list the zpools available, you can run
root@borgcube:~ # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT borgpool 2.72T 138K 2.72T - 0% 0% 1.00x ONLINE - root@borgcube:~ # iohyve setup borgpool Setting up iohyve...
- You can see that it installed by running
root@borgcube:~ # zfs list NAME USED AVAIL REFER MOUNTPOINT borgpool 318K 1.75T 24.0K /borgpool borgpool/iohyve 48.0K 1.75T 24.0K /iohyve borgpool/iohyve/ISO 24.0K 1.75T 24.0K /iohyve/ISO
Install a FreeBSD guest.
- Fetch the installation media:
root@borgcube:~ # iohyve fetch ftp://ftp.freebsd.org/.../FreeBSD-10.2-RELEASE-amd64-bootonly.iso Fetching ftp://ftp.freebsd.org/.../FreeBSD-10.2-RELEASE-amd64-bootonly.iso... /iohyve/ISO/FreeBSD-10.2-RELEASE-amd64-bootonl100% of 230 MB 6883 kBps 00m34s root@borgcube:~ #
Now here is where
tmuxcomes in. Because bhyve doesn't emulate video, the console output is connected via serial console, in the case with iohyve, it uses
cuto set up
nmdmconsoles for each guest. There are other ways to set this up, like directing output directly to
tmuxpanes or xterm itself. If you opted to install a GUI before all of this, such as i3 you don't even need a multiplexer like
tmux, you can just open a new terminal window. You can theoretically install a guest via one SSH session as well, using
<RETURN> ``.to stop the nmdm connection and bring you back to your SSH prompt. This can get tricky and cause headaches, though.
Create a guest named
freebsdguestthat has a disk size of 8 gigabytes, then run the installation:
root@borgcube:~ # iohyve create freebsdguest 8G Creating freebsdguest... root@borgcube:~ # iohyve isolist Listing ISO's... FreeBSD-10.2-RELEASE-amd64-bootonly.iso root@borgcube:~ # iohyve install freebsdguest FreeBSD-10.2-RELEASE-amd64-bootonly.iso Installing freebsdguest...
If you haven't already done so, open a new terminal (
Ctrl+b + con tmux) and run
iohyve console freebsdguest. You may have to press enter once in the console.
As long as you choose to do a UFS install, you should be able to install just as usual. You can do a "root on zfs" install, but you have to up the amount of RAM the guest uses, as mentioned before, ZFS likes to nomnomnom on some RAM. The default amount of RAM iohyve set is 256M, but you can change that after guest creation, more on that later.
Once the installation is over, you can choose to "reboot" via the menu, but that actually won't reboot the guest, more on that another time. If you choose the "reboot" option on the menu, you must run
iohyve destroy freebsdguestbefore going further. Otherwise, you can just go back to your orginal terminal and run 'iohyve stop freebsdguest' when it asks you to reboot and iohyve will gracefully stop the guest for clean startup on it's next start.
Now you can run
iohyve start freebsdguestand switch back to the
iohyve console freebsdguestterminal watch in glory as the guest operating system boots.
Other cool stuff you can do with iohyve
- Check all of the properties for a given guest:
root@borgcube:~ # iohyve getall freebsdguest Getting freebsdguest props... ram 256M cpu 1 size 8G loader bhyveload install no boot 0 tap tap0 name freebsdguest con nmdm0 persist 1
- Increase RAM amount:
root@borgcube:~ # iohyve set freebsdguest ram=1024M Setting freebsdguest prop ram=1024M... root@borgcube:~ # iohyve get freebsdguest ram Getting freebsdguest prop ram... 1024M
- Delete a guest:
root@borgcube:~ # iohyve list Listing guests... freebsdguest naughtyguest root@borgcube:~ # iohyve delete naughtyguest Are you sure you want to delete naughtyguest [Y/N]? y
- List all running guests:
root@borgcube:~ # iohyve running Listing running guests... freebsdguest
- Start a guest at host boot time. Make sure
iohyve_enable="YES"is set in your
root@borgcube:~ # iohyve set freebsdguest boot=1 Setting freebsdguest prop boot=1...
Run a Debian Linux guest.
- After fetching the ISO, create the guest as usual. I also like to give the guest a bit more RAM, too.
root@borgcube:~ # iohyve create debianguest 8G Creating debianguest... root@borgcube:~ # iohyve set debianguest ram=512MB Setting debianguest prop ram=512MB...
- Change the guest's bootloader by setting it in iohyve. Make sure you have installed grub2-bhyve as mentioned above.
root@borgcube:~ # iohyve set debianguest loader=grub-bhyve Setting debianguest prop loader=grub-bhyve...
- Install the guest as usual:
root@borgcube:~ # iohyve install debianguest debian-8.2.0-amd64-i386-netinst.iso Installing debianguest...