Xen Debian Quick Start

WARNING (2005-12-13): We have made changes to our unofficial Debian repositories and their respective wiki pages. If you have already been using them, you may need to take special note of the information in Upgrading_from_Xen_2.0.6_to_Xen_2.0.7.

2005-12-13: We've released the new "Quick Start" for Xen-2.0-stable, which refers to the 2.0.7 packages. The original version, which refers to the 2.0.6 packages, is still available here.

Three paths to your first boot into Xen

Most of our development and productions systems use Debian Sarge + Xen Stable (currently 2.0.6) for the dom0. As the dom0 has a variety of things to worry about, we keep that system as clean as possible (it also makes it easier to catch issues on upgrade). We've used several approaches to get these systems running, choose your flavor. Also, try not to mix and match - the first two options use a different set of debian packages and kernel images than the third, so the grub menu items are different.

The "unofficial" Xen Debian Packages (for Xen Stable version 2.0.7)

Currently (as of 2005-12-09) the packages available from Debian are for Xen version 2.0.6. As we have updated and tested our system for 2.0.7, we've created our own version of the packages. These have all been created with and tested against Sarge/Stable. (They have been altered from the original Debian/Xen packages in a few ways.)

1) If you are starting from scratch you'll need to get and install Debian Sarge. I usually use the floppies or the netinst CD image. (Partition how you want, but if you are going to experiment with LVM leave at least one partition for boot and one for LVM experiments.) If you are not familiar with the Sarge installer, please take a look before you think that these instructions are too brief, or that Debian is "hard to install."

2) Once you have a running Debian system, with grub as a bootloader, add the following to /etc/apt/sources.list:

deb http://www.option-c.com/debian/ experimental main

Then run:

apt-get update
apt-get install xen-system

xen-system is the meta-package that will install the most common components to get the system running with Xen.

3) If you are using a version of xen-system installed after 2005-07-19, then update-grub-xen tried to make this entry for you. However, it's always good to check.

Add the following entry to /boot/grub/menu.lst (adjust for your partitions - also, if you have a dedicated boot partition then you will need to remove "/boot" from the kernel and module lines). 64MB should be enough memory so that you don't have to worry about the system failing to boot because dom0 does not have enough memory. However the "right" number to use in entirely dependent on the needs of the system (number of domUs, type of file system, etc).

title		Debian Xen 2.0.7/2.6.11 
root		(hd0,0)
kernel	 	/boot/xen-2.0.7.gz dom0_mem=65536 noreboot
module		/boot/xen-linux-2.6.11-ocxen0 root=/dev/hda1 ro console=tty0

Or, if you are using a version of the kernel with more module support (and, specifically, if some of those modules enable you to access your root device)

title		Debian Xen 2.0.7/2.6.11 
root		(hd0,0)
kernel	 	/boot/xen-2.0.7.gz dom0_mem=65536 noreboot
module		/boot/xen-linux-2.6.11-ksxen0 root=/dev/hda1 ro ramdisk_size=24576 console=tty0
module		/boot/initrd-2.6.11-ksxen0

The "noreboot" option is to keep your system from rebooting if you have a kernel panic, so that you have a chance to see what might be missing. You'll probably want to remove it once you know the system boots.

4) Reboot, choose the appropriate boot selection in grub, and keep your fingers crossed (see below for further instructions).

The installer CD

Why a dedicated installer CD?

CDs are pretty and shiny
We do this particular task (installing Debian+Xen) fairly often, and under a lot of circumstances (such as sneaking around during parties and changing the hosts operating system), so it makes sense for us to have a dedicated installer that automates the process. If it doesn't make sense for you, feel free to ignore this section

Using the installer CD

Get the current version of the installer iso from sf.net. WARNING: Please note that this installer is currently under heavy development, does not contain the most recent version of the packages (it is for 2.0.6), and probably has other issues, too.
Burn the image to a CD
Boot the CD and follow the prompts (There is a hidden boot option called "nukeandpave" - do not use this! It is for doing automated installs on dedicated hardware, and will destroy all existing data)
The default installation will ask for various information (passwords, partitioning, etc)

What are the known issues?

Primarily grub, and updating grub. Situations that work:

Clean, dedicated install
Installing onto a multi-boot system, but letting the installer detect your other systems and set them to boot via grub

For all other cases, look in /usr/share/doc/xen/grub for a sample of the grub entry you will need to make and tell the installer to put grub on the partitioning you are installing to and do NOT install to the MBR. Otherwise, you may need to to a little work to get back you original grub entries.

As a further note, when the install is "completed" the system you will first log into is the default Debian kernel, not the Xen version. This will be remedied in future versions, however the default kernel will always be installed because it is so much easier to troubleshoot from a running system...

I can't boot from the CD-ROM!

Mount the CD on another system. The file in /install/sbm.bin is a floppy image of the Smart Boot Manager. It should allow you to boot from the floppy, then to the CD. To make a floppy from the image under linux, you can do something like:

dd if=sbm.bin of=/dev/fd0

For Windows, there are files on the CD in the /tools directory to help create the image.

I don't even have a CD-ROM!!

If that's the case, you are probably better off doing a clean install of Debian using another method, then adding the Xen packages (see one of the other two sections)

Debian Sarge + the "official" Debian Packages (for Xen Stable version 2.0.6)

This is a modification of our "original" quick start - it is maintained for those who prefer to use only packages from debian.org, but is not regularly tested.

1) If you are starting from scratch you'll need to get and install Debian Sarge. I usuall use the floppies or the netinst CD image. ( Partition how you want, but if you are going to experiment with LVM leave at least one partition for boot and one for LVM experiments.) If you are not familiar with the Sarge installer, please take a look before you think that these instructions are too brief, or that Debian is "hard to install."

2) Once you have a running Debian Sarge system (with grub as a bootloader), add the following line to /etc/apt/sources.list (feel free to pick a different mirror):

deb ftp://debian.uchicago.edu/debian/ unstable main

You may also want (or need) to add:

deb ftp://debian.uchicago.edu/debian/ testing main

WARNING: The modifications to /etc/apt/sources.list will probably make your system unusable very soon. I won't get into apt-pinning now (which is how we avoid issues); you will probably want to remove the lines that refer to unstable and testing after you've gotten through the install.

3) Run the following (as root):

apt-get update
apt-get upgrade
apt-get install module-init-utils
apt-get install xen

If you get errors such as "xen has the following unmet dependencies, but the packages do not exist" and you didn't add "testing" to your /etc/apt/sources.list, now is probably the time to to that and run "apt-get update" again.

4) The Xen networking scripts that are installed with the package use the commands ip and brctl. The packages that supply them are not technically dependencies for Xen, so are not installed with the above steps. Since this is a "quick start" the easiest option is to add them:

apt-get install iproute
apt-get install bridge-utils

5) The debian package installs the Xen hypervisor image, but not the domO or domU kernels. I just used the ones from the prebuilt install. Extract the package:

tar -xf xen-2.0.6-install.tgz

Copy the entire contents of xen-2.0-install/install/boot/* to /boot.

cp PATH/xen-2.0-install/install/boot/* /boot

Copy xen-2.0-install/install/lib/modules to /lib/modules

cp -dpR PATH/xen-2.0-install/install/lib/modules/* /lib/modules

6) Move /lib/tls to /lib/tls.disabled (If you don't do this, Xen will still work but it will spew a warning on boot and will be very slow). As time passes you will probably need to do this again, as various updates will add /lib/tls back

mv /lib/tls /lib/tls.disabled

7) Add something like the following to /boot/grub/menu.lst

Title   Xen 2.0.6/2.6.10
root	(hd0,0)
kernel	/boot/xen.gz dom0_mem=131072
module	/boot/vmlinuz-2.6.10-xen0 root=/dev/hda1 console=tty0

(You'll need to adjust for what your boot/root partitions are and, if grub is already installed, edit the appropriate version of menu.lst)

8) Reboot. Pick the appropriate selection from the boot menu (unless you've set it up as the default, then just wait.)

Yay - it worked

You should see a relatively normal boot sequence, ending with something like "xen-br0: port 2(vif1.0) entering forwarding state" and then a login. If it errors out and complains that it couldn't mount root from the NFS server, then either there was something wrong with your grub configuration, or the default kernel can't detect your hard drive. Hopefully the former.

If everything has gone well, try this command (as root):

xm list

You should see:

Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0       64    0  r----    192.4

Now for the fun part. For your first domU...

Create_a_Debian_VM_with_debootstrap - there are instructions on the site for using debootstrap

Boo - it didn't work

This is by no means a detailed troubleshooting section. At this point, if you have followed the steps and do not have a bootable system, the most likely problems either involve the grub configuration or the hardware support from the kernel you used (I'm not saying either of those are _small_ problems, by the way, just likely...)

The system booted, but...

If you can get the system to boot, but simple commands such as:

xm list

Result in:

(111, 'Connection refused')
Error: Error connecting to xend, is xend running?

You'll want to check a few things.

Did you actually boot into a Xen kernel?

cat /proc/version

Should result in a response that is one of the xen kernels

Linux version 2.6.11-ocxen0

Xend is running, but...

If it looks like xend is running, but networking or other things aren't, you can do a few other basic tests to make sure things were installed correctly. To check to make sure the basic libraries are there, try

ldd /usr/sbin/xfrd

The output on one of my systems looks something like this:

 libxc.so.2.0 => /usr/lib/libxc.so.2.0 (0xb7fd7000)
 libxutil.so.2.0 => /usr/lib/libxutil.so.2.0 (0xb7fc6000)
 libz.so.1 => /usr/lib/libz.so.1 (0xb7fb4000)
 libcurl.so.3 => /usr/lib/libcurl.so.3 (0xb7f82000)
 libidn.so.11 => /usr/lib/libidn.so.11 (0xb7f52000)
 libssl.so.0.9.7 => /usr/lib/i686/cmov/libssl.so.0.9.7 (0xb7f21000)
 libcrypto.so.0.9.7 => /usr/lib/i686/cmov/libcrypto.so.0.9.7 (0xb7e22000)
 libdl.so.2 => /lib/libdl.so.2 (0xb7e1e000)
 libc.so.6 => /lib/libc.so.6 (0xb7ceb000)
 /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7fea000)

If any line contains information about the library not being, then you'll need to add them (for example, if you are using Woody, you might have libcurl2 instead of libcurl3). Note that xfrd is for migrating domains; however since (in version 2.0.6, at least) it tends to fail rather silently, I added the information here to check if everything is okay.

Check that iproute and bridge-utils are installed. If either of the following commands give you a "command not found" error, then you need to add either iproute or bridge-utils

ip
brctl

The system didn't boot

If you do not have a bootable system at all you should still be able to boot into normal Debian Sarge (any of the instructions in the quickstart will result in an installation of both the default kernel and the Xen kernel for the exact reason that it is easier to troubleshoot from a running system). Also, if you are stuck in a reboot loop, and everything is going to quickly, check your grub configuration. I leave the"noreboot" option for xen (the kernel /boot/xen.gz line) so that you have a chance to see what happened in case of a panic.

Confirm that the line for the hypervisor looks like this (replace "X" with the version number, also remove "/boot" if you have a dedicated boot partition):

kernel      /boot/xen-2.0.X.gz dom0_mem=65536 noreboot

These links might help you troubleshoot:

If really is unsupported hardware, you are (as of now) probably in for a custom compilation of the kernel. There is a kernel available that I built for testing these things, so if you want to try that before custom compiling, make sure the following is in /etc/apt/sources.list

deb http://www.option-c.com/debian/ experimental main

Then:

apt-get install kernel-image-2.6.11-ksxen0

This should also install kernel-modules-2.6.11-ksxen0. If kernel-image-2.6.11-ocxen0 didn't boot, then you'll want to try this one with an initrd image, so:

mkinitrd -o /boot/initrd-2.6.11-ksxen0 2.6.11-ksxen0

Append the following to /boot/grub/menu.lst (the entry that either you created, or was created for you, for Xen):

 module		/boot/initrd-2.6.11-ksxen0

And try to boot again...