Back to Article Listing

Xen Garden

aka: Running multiple NetBSD virtual machines under Xen

I found installing NetBSD and Xen to be a bit of a struggle. Despite all the documentation out there, I had gaps in my knowledge that were omitted or glossed over in many documents. Hopefully, these notes will help others "Xen" more easily.


My Goal is to run multiple NetBSD virtual machine instances under Xen


This is not to be considered a HOW-TO, definitive guide, or an advocacy for running a particular virtual machine on a particular OS. I just wanted to find out what this "Xen" thing was all about. These are just my notes on the steps I took to get going. I am sure there are other ways of doing things.

(excerpt from Xen Users Guide)

Xen is a paravirtualising virtual machine monitor (VMM), or 'hypervisor', for the x86 processor architecture. Xen can securely execute multiple virtual machines on a single physical system with close-to-native performance. The virtual machine technology facilitates enterprise-grade functionality, including:

  • Virtual machines with performance close to native hardware.
  • Live migration of running virtual machines between physical hosts.
  • Excellent hardware support (supports most Linux device drivers).
  • Sandboxed, restartable device drivers.

Paravirtualisation permits very high performance virtualisation, even on architectures like x86 that are traditionally very hard to virtualise. The drawback of this approach is that it requires operating systems to be ported to run on Xen. Porting an OS to run on Xen is similar to supporting a new hardware platform, however the process is simplified because the paravirtual machine architecture is very similar to the underlying native hardware. Even though operating system kernels must explicitly support Xen, a key feature is that user space applications and libraries do not require modification.

Xen support is available for increasingly many operating systems: right now, Linux 2.4, Linux 2.6 and NetBSD are available for Xen 2.0. A FreeBSD port is undergoing testing and will be incorporated into the release soon. Other OS ports, including Plan 9, are in progress. We hope that that arch-xen patches will be incorporated into the mainstream releases of these operating systems in due course (as has already happened for NetBSD).

Quick Primer

dom0 is the name to describe the master, or "privileged" domain and domU is the generic name of all the "unprivileged" virtual machines. Despite the fact that NetBSD runs 'on top' of Xen, you install it the other way around.. kinda.

First you do a normal installation of NetBSD. This will install a kernel and a base userland to be used as dom0. You are welcome to boot to that kernel anytime, but, if you want to run virtual machines you have to install a bootloader (grub), and Xen.

During a normal Xen boot process, the machine will show grub first. One of the menu choices (explained below) is to start Xen with NetBSD as the domain0 kernel. Only you are not booting the NetBSD kernel you just installed. You are going to boot a special netbsd-xen kernel downloaded separately. The rest of the NetBSD installed software is used normally.

Xen acts as a thin layer, or broker, of system calls from the all the domU's to dom0. Despite having a special kernel, dom0 looks, feels, and acts just like a normal install of NetBSD when you are in it. The dom0 is really just there as a slave for Xen to talk to the computer hardware and as a layer to hold the various tools to manage the domU's. When you are in a domU, devices all have unfamiliar names, like xennet0 is a network card device and xbd1d is a cdrom. Xen has created those devices and maps system calls back to the real devices on dom0.

That is over simplified but I hope it makes the point.

My Steps

I replaced my home desktop a while ago and kept my old one around as a "lab machine". It is a Pentium 4 1800 MHz with 512 MB RAM and a 20G harddrive. So, I was able to start from scratch. I outline a few steps on NetBSD installation in case any readers have not done so before. I installed the latest formal release of NetBSD - 3.0 , from CD. So, before I started, I downloaded an ISO for the i386 platform from FTP and burned it. Please check their mirrors for the closest download server.


Because you are reading this, my assumption is, you have some rudimentary (my level) understanding of computer hardware and BSD.

Design Decisions

How to organize the hard drive.. I wasn't dual booting this machine so I could use the whole hard drive, but, I still have a few options.

  • give each VM its own slice as a physical device
  • put several file images in the same physical device, apart from the main OS

For my first install, I chose the former. I then changed my mind (it is a lab machine, after all), wiped the drive, and opted for the latter. It is your choice, according to your needs. I ended up with slice (a) as a dom0 and have several images sitting on a large slice (e) mounted on /virtual. More on that below.

While I would have liked to keep maintenance to a minimum, I wanted to keep the exercise relatively simple. In the future I will look into more advanced features as outlined by Johnny Lam in his talk to NYCBUG.

For the impatient

  • Install NetBSD as dom0
  • Install grub, Xen, and tools
  • Create file image
  • Install NetBSD as domU in that image
  • Configure the domU, install the OS, and run it
  • Rinse and Repeat

For the patient

Install NetBSD as dom0

  • Boot from CD, follow instructions. Have your hard drive design in mind. The install process holds your hand, so I do not have to.
  • Installing from CD does not require networking, so the installer never asks. If you use FTP it will ask for and, later, ask to permanently save) your networking specifics. My nic was recognized as "tlp0" and I assign a permanent IP [], yours may differ. I use [] as the router and nameserver for simplicity here. Again yours may differ. If networking wasn't done, bring it up. There are a few ways to do it:
First: on the fly
ifconfig tlp0 netmask route add default
Then, make permanent in etc/rc.conf
defaultroute= ifconfig_tlp0="inet netmask" hostname=machine.domain.home
-or- created in separate, single line files
/etc/mygate, /etc/ifconfig.tlp0, inet netmask up /etc/myname, machine.domain.home

Additionally create /etc/resolv.conf by adding these two lines: domain yourdomainname.home (second line) nameserver You may also want to add sshd="yes" to the /etc/rc.conf if you want to ssh into the box and work from another workstation on next boot. You should create a user account for that (or add "PermitRootLogin yes" without quotes to /etc/ssh/sshd_config) Also edit /etc/hosts, for any local resolutions you may need. Add xend="YES" to /etc/rc.conf. It will be needed later. It is also advisable to firewall the special ports that Xen uses. I'm no firewall guru but I found some suggestions in the xen-port mailing list archives.

You are now ready to reboot the machine into NetBSD.. (eject cd) reboot and log in.

Install grub, Xen, and tools

Install notice:
NetBSD: MESSAGE,v 1.5 2005/02/19 10:23:13 wiz Exp

If your root filesystem is of a type not supported directly by grub
(e.g., lfs), you may have difficulties.  See the info docs for more

GRUB is not actually installed on your disk until you run a command
such as
        grub-install /dev/wd0d
        grub-install '(hd0)'

To boot NetBSD, put something similar to these lines into /grub/menu.lst:

        title NetBSD
        root (hd0,0,a)       # NetBSD on 1st MBR partition of 1st IDE disk
        chainloader +1


As instructed I ran without quotes "grub-install --no-floppy /dev/wd0d". On NetBSD, grub installs several files in /grub. The one needing attention is menu.lst. This is the menu put on the screen right after bios and before the OS boots. If you are familiar with grub, then just skip this part.. it will not substitute for the man page but, here's a primer... The menu.lst file has some settings at the top, but the rest of the file is a series of menu 'items' with a Title, kernel, and possibly a module. When you read the file you will get it. If the kernel is Xen, then you need a module like NetBSD. If the kernel is just plain NetBSD, then you do not need a module. Watch out for the 'default' setting. If you do not specifically make a choice from the menu at boot time, the default kicks off automatically after the the number of seconds in the timeout setting. Each menu item is numbered starting with 0, in the order they appear top to bottom. I changed the default setting to 1 (the second menu item, from the top) because I was booting to a monitor not the serial port. The menu items are well commented, but a little confusing if you are not familiar. The second menu item in my file (my default) looks like this:

# Same as above, but using VGA console
# We can use console=tty0 (Linux syntax) or console=pc (NetBSD syntax)
title Xen 2.0 / NetBSD (DEFAULT: hda0, vga console)
kernel (hd0,a)/xen.gz dom0_mem=65536
module (hd0,a)/netbsd-XEN0 root=/dev/hda1 ro console=tty0

In the first line, everything after the word "title" is arbitrary and just to help you choose the right one. I added the word DEFAULT so it would stand out in the menu. The root phrase is for grub to find the right drive. The kernel phrase says load the kernel named xen.gz allocate 64M memory for the dom0 module. Because we are loading Xen in this case, it needs a module to load an OS. That is the special kernel downloaded from the NetBSD FTP (explained below). The regular kernel would not work under Xen, but keep it around in case you want to boot the box in "single OS mode". That is what the last 2 (of 8) menu items are for. #7 boots the NetBSD kernel from the original install. Because grub can not pass all the needed parameters you will need to type in the rest by hand (e.g. root device in my case wd0a, etc.) #8, the chainloader is supposed to handle that, but I could not get it to work. If you do, drop me a note.

  • pkg_add xentools20
Installing files needed by xentools20-2.0.7nb5:






The following files should be created for xentools20-2.0.7nb5:

        /etc/rc.d/xendomains (m=0755)

        /etc/rc.d/xend (m=0755)


Install notice:
NetBSD: MESSAGE.NetBSD,v 1.1 2005/11/08 00:47:35 jlam Exp

Please ensure that the Xen-specific devices needed by xend(8) exist:

        cd /dev && sh MAKEDEV xen

There are example configuration files for setting up a guest domain in:


Please also refer to the the "NetBSD/xen How-To" for more information on
creating a Xen setup:


As instructed I ran

cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/xendomains
cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/xend
cd /dev && sh MAKEDEV xen

Check out the examples for ideas.

  • pkg_add xenkernel20
Install notice:
NetBSD: MESSAGE,v 2005/10/13 20:58:20 agc Exp

You should now copy the file:


to your root file system.

As instructed I ran without quotes "cp /usr/pkg/xen-kernel/xen.gz /". This is the actual Xen kernel. This is the one referenced in the grub menu example above, in the kernel phrase.

  • Additional downloads (seems this could be another pkg_add, but it is not)

    1. You will need the special NetBSD Xen kernel for dom0. This is the one you boot with Xen as a module. It is, literally, a drop-in replacement for the kernel in the root directory acquired during normal installation. I found 2, there may be more.

    Choose one, download it, and untar it in a safe place. The extracted file will be a "NetBSD Xen dom0" kernel. It should be placed in the root directory next to the normal kernel.. with a different name. That name should be typed into the /grub/menu.lst module phrase. In my example above I used "netbsd-XEN0"

    1. While we are downloading kernels.. you will need 2 more! Remember, the normal kernel will not work under Xen. When you install a new domU (a virtual machine) you will need to do another NetBSD installation. That will require a special kernel for the installation process and a special kernel to run the virtual machine when you boot it. This is just like any other BSD install. One kernel boots off the CD (or floppy) and another is for normal use. In the case of Xen, you will need one modified kernel to do an install and another modified kernel to actually run the VM. You will need BOTH of these. Guess which one is which..

      I unzipped them and put them in the root directory to keep all kernels together. I'll explain how they are used below.

      So, now you have 5 kernels! (netbsd, xen, netbsd-XEN0, netbsd-XENU, and netbsd-INSTALL_XENU)

You are now ready to reboot the machine into Xen and the NetBSD dom0 so, and log in.

Create a file image to install the domU

One of the objectives of creating a VM, for me, was to create an image that could be copied several times. I only wanted to install the OS once and be able to re-use it. This is why I chose file images. Remember, I have a large slice on the hard drive created during the original installation (it is dom0, now) that is mounted on /virtual. Up until now it was empty. I will create an empty 1G file to will look and feel like a hard drive. NetBSD does not require a 1G drive to install, but I was not sure what was going to run in the VM and wanted to have plenty of room. In the future this may be done differently as I get more complex like Johnny Lam showed us.

  • Create an image, without quotes "dd if=/dev/zero of=/virtual/netbsd1.img bs=1024k count=1024". (took about 2 minutes on my hardware)
  • Mount it like a device, without quotes "vnconfig vnd0 /virtual/netbsd1.img"
  • Give it an MBR, without quotes "fdisk -i /dev/vnd0d"
  • Unconfigure, without quotes "vnconfig -u vnd0"

You can also compress these images; read vnconfig(8).

Configure the domU, install the OS, and run it

In order to do a NetBSD install into /virtual/netbsd1.img under Xen, it will require some edits to the installed Xen environment, and you need to create a configuration file for Xen to run the domU. After installation, some minor additional edits will be made to that config before we are finished. There are examples of config files for different requirements, as seen in the Install Notice for xentools20. Each VM will need its own config file. I used my own naming convention: config.domX. You can use whatever you want. Xen environment and config files are located in /usr/pkg/etc/xen.

  • Prepare 'shared' networking environment. I have a single nic in this box and have chosen to create a bridge so all the VMs will share that nic. You have other choices, but, not listed here. Create a bridge, bring it up, make it permanent. Again, my device is recognized as tlp0. Your device may differ.

    On the fly

    ifconfig bridge0 create brconfig add tlp0 brconfig bridge0 up

    Make permanent

    create /etc/ifconfig.bridge0 and add these 2 lines: create !brconfig $int add tlp0 up

  • Edit xend-config.sxp to reflect bridge name (vif-bridge bridge0)

The brconfig -a command is a great tool to watching devices that are added to the bridge and mac addresses that are recognized. Here is mine before starting any domU, then after starting 2 of them. Note; the mac addresses listed are obfuscated, but are my workstation ssh'd into dom0 and the network router.

$ brconfig -a
bridge0: flags=41<UP,RUNNING>
                priority 32768 hellotime 2 fwddelay 15 maxage 20
                ipfilter disabled flags 0x0
                tlp0 flags=7<LEARNING,DISCOVER,STP>
                        port 1 priority 128 path cost 55 forwarding
        Address cache (max cache: 100, timeout: 1200):
                00:00:00:00:00:x1 tlp0 4294967205 flags=0<>
                00:00:00:00:00:x2 tlp0 4294967181 flags=0<>

$ brconfig -a
bridge0: flags=41<UP,RUNNING>
                priority 32768 hellotime 2 fwddelay 15 maxage 20
                ipfilter disabled flags 0x0
                xvif2.0 flags=3<LEARNING,DISCOVER>            <== note Xen created device
                        port 7 priority 128
                xvif1.0 flags=3<LEARNING,DISCOVER>            <== note Xen created device
                        port 6 priority 128
                tlp0 flags=7<LEARNING,DISCOVER,STP>
                        port 1 priority 128 path cost 55 listening
        Address cache (max cache: 100, timeout: 1200):
                00:00:00:00:00:x1 tlp0 4294967099 flags=0<>
                00:00:00:00:00:x2 tlp0 4294967075 flags=0<>

Create config.dom1

Here is mine with some comments

kernel = "/netbsd-INSTALL_XENU"         # used for initial installation, then commented
#kernel = "/netbsd-XENU.gz"             # added after install, prior to first boot
memory = 128                            # your choice, just don't max out your box
name = "dom1"                           # arbitrary, but, unique.. used to identify running domains
cpu = -1                                # leave to Xen to pick, if you have more than one
nics = 1                                # your choice, will need a vif for each one
vif = [ 'mac=aa:00:00:50:02:f1, bridge=bridge0' ]
                                        # bridge created above
#disk = [ 'phy:/dev/wd0f,wd0d,w' ]      # left over from my first attempt, left as an example
disk = [ 'phy:/dev/cd0a,cd0a,r', 'file:/virtual/netbsd1.img,wd0d,w' ]
                                        # use cd during install, comment later
#disk = [ 'file:/virtual/netbsd1.img,wd0d,w' ]
                                        # using a file instead of a physical device
root = "/dev/wd0d"

Note in the above kernel line has the INSTALL_XENU version of the kernel. This is just to do the initial install. Before first boot you will need to edit that file by commenting that line and changing the kernel call by uncommenting the one below it. Also note that the netbsd-XENU will be used by all NetBSD VM's.

  • Run config.dom1 the first time to install the NetBSD OS. With out quotes "xm create -c /usr/pkg/etc/xen/config.dom1" You really should read the documentation, but here's some quick commands to get you going.

    • xm list = list running domain names, Id's, and some stats
    • xm create /path/to/config.file = startup a domain that is not running yet
    • xm console $domUname = console into the running domain with the name $domUname, or you could use the $domUId from xm list
    • xm create -c /path/to/config.file = startup a domain that is not running yet and connect immediately
    • CTRL ] = while in a running VM, exit the console by holding the control key and right bracket
    • xm shutdown $domUname = shutdown the running domain with the name $domUname
    • halt -p = recommended way to halt the domain.. but always gives me python errors

Doing the installation of a domU will look and feel just like a normal install, but, with Xen generated device names. If you plan on installing from CD, the device is called xbd1d and you will need to note that. Also, the nic is recognized as xennet0 when you want to bring up the network. The install program correctly sees the hard drive as xbd0. When install finishes choose ">x: Exit Install System" and get dropped to a prompt. Type halt -p and after you see it has unmounted the file system, hit return a few times exit the console and get through the errors back to your dom0 prompt. Eject the cd and edit the config.dom1 file. Comment the INSTALL kernel and uncomment the running kernel. I do not need a CD in the VM, so I commented the disk phrase and just included the one with the img.

  • Run config.dom1 as first true boot xm create -c /path/to/config.file will boot your new VM!
  • Using DHCP I have a dhcp server on my network. After doing the initial installs of domU, I added this section to dhcpd.conf:
host xen.dom1 {
  option host-name "xen-dom1";
  hardware ethernet aa:00:00:50:02:f1;
host xen.dom2 {
  option host-name "xen-dom2";
  hardware ethernet aa:00:00:50:02:f2;
host xen.dom3 {
  option host-name "xen-dom3";
  hardware ethernet aa:00:00:50:02:f3;

Thankfully, the NetBSD dhcp client pays attention to the host-name option. Now, I can make each domU even more generic. I can remove the hostname and IP specifics and just leave the default route. On boot each one gets an IP and a hostname, from the dhcp server, according the mac set in the conf.domX files.

Rinse and Repeat

I can copy the /virtual/netbsdX.img, create a new conf.domX, and boot another VM. While this satisfied my original design, in the future I could create only one root image and package image, and reuse them. Then I will only need to create read-write images for each domU /var and /etc partitions.

Support the Project!

Supporting open source software is simple, easy, and the right thing to do. You can donate money or hardware, purchase CD's, t-shirts, etc.

"Thank you" to the NetBSD and Xen developers for letting us use, and benefit from, your hard work!

Copyright © 20060305 genoverly
(db datestamp: 20070821)

Copyright © 2003-2015 genoverly