Using an XFS root on domU with Xen and pyGRUB
At 46elks, we own and run our own hardware located in a few places in Sweden. We don't want to go to a data centre and install new hardware every time we need a new server though, so we use virtualisation with Xen. Setup of new virtual machines, or domUs, is done with xen-create-image to download a distro and set it up automatically, as well as pyGRUB for booting domUs with their own kernels, making updates of the guest a lot easier.
Now, we would also like to use XFS for the root partition of the domUs to increase I/O performance, but pyGRUB does not support anything other than ext4. Well, does not officially support anything else. pyGRUB needs to access /boot to initialise the initrd, but after doing that, root can be whatever filesystem Linux has support for. So if we put /boot on a separate partition with ext4, we can use XFS for /!
As pyGRUB assumes that / and /boot are on the same partition, we'll need to make some adjustments to the defaults that xen-create-image and pyGRUB sets.
First we need to specify our partition scheme though. We create this file:
# cat /etc/xen-tools/partitions.d/xfsroot
[boot]
size=256M
type=ext4
mountpoint=/boot
[root]
size=10G
type=xfs
mountpoint=/
options=defaults
[swap]
size=2G
type=swap
and run xen-create-image with --partitions=xfsroot and other flags for RAM size, amount of CPUs, and the like. The order of the partitions in the partition configuration file doesn't seem to matter.
This results in an unbootable system, but one we can fix. Trying to start the domU, we get this:
# xl create /etc/xen/mydomu.cfg
Parsing config from /etc/xen/mydomu.cfg
libxl: error: libxl_bootloader.c:635:bootloader_finished: bootloader
failed - consult logfile /var/log/xen/bootloader.42.log
[...]
# cat /var/log/xen/bootloader.42.log
Traceback (most recent call last):
File "/usr/lib/xen-4.8/bin/pygrub", line 882, in <module>
raise RuntimeError, "Unable to find partition containing kernel"
RuntimeError: Unable to find partition containing kernel
pyGRUB can't find the kernel, which is in the boot partition. pyGRUB assumes the first partition in the domU configuration file is the one containing /boot. We can fix this by moving the boot disk entry in the domU config up to be the first one. This is what xen-create-image generates in /etc/xen/mydomu.cfg:
disk = [
'phy:/dev/vg0/mydomu-root,xvda2,w',
'phy:/dev/vg0/mydomu-boot,xvda3,w', # /boot is here
'phy:/dev/vg0/mydomu-swap,xvda1,w',
]
We'll just change that to
disk = [
'phy:/dev/vg0/mydomu-boot,xvda3,w', # move it here!
'phy:/dev/vg0/mydomu-root,xvda2,w',
'phy:/dev/vg0/mydomu-swap,xvda1,w',
]
Trying to boot again:
# xl create /etc/xen/mydomu.cfg
Parsing config from /etc/xen/mydomu.cfg
libxl: error: libxl_bootloader.c:635:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.46.log
[...]
# cat /var/log/xen/bootloader.46.log
pyGRUB version 0.6
┌────────────────────────────────────────────────────────────────────────┐
│ Debian GNU/Linux 10 │
│ Debian GNU/Linux 10 (Single-User) │
│ Debian GNU/Linux 10 (Default Kernel) │
│ Debian GNU/Linux 10 (Default Kernel, Single-User) │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Use the ^ and ┴ keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.
Will boot selected entry in 1 seconds
[Errno 2] No such file or directory
Error opening /boot/vmlinuz-4.19.0-6-amd64 in guest
pyGRUB is able to find the boot partition and the GRUB config! It still can't find the kernel because it is looking for it in /boot, but as we are using a separate partition and are mounting the partition by itself, it will actually be in / during boot.
/boot/grub/menu.cfg looks like this:
[...]
title Debian GNU/Linux 10
root (hd0,0)
kernel /boot/vmlinuz-4.19.0-6-amd64 root=/dev/xvda2 ro elevator=noop
initrd /boot/initrd.img-4.19.0-6-amd64
title Debian GNU/Linux 10 (Single-User)
root (hd0,0)
kernel /boot/vmlinuz-4.19.0-6-amd64 root=/dev/xvda2 ro single elevator=noop
initrd /boot/initrd.img-4.19.0-6-amd64
title Debian GNU/Linux 10 (Default Kernel)
root (hd0,0)
kernel /vmlinuz root=/dev/xvda2 ro elevator=noop
initrd /initrd.img
title Debian GNU/Linux 10 (Default Kernel, Single-User)
root (hd0,0)
kernel /vmlinuz root=/dev/xvda2 ro single elevator=noop
initrd /initrd.img
Some entries are prefixed with /boot, while others aren't. I'm not sure why this is, but there's a simple workaround that makes both work. We create a symlink in /boot called boot that points to the directory itself is in, that is, /boot (boot boot boot).
# mount /dev/vg0/mydomu-boot /mnt
# cd /mnt
# ln -s . boot
# ls -l
lrwxrwxrwx 1 root root 1 Sep 26 13:46 boot -> .
Now you should be able to unmount the partition and finally boot the domU.
Automating
Having to do this every time we create a domU would be annoying, fortunatly, xen-create-image is able to run hooks (called "roles") after the setup is complete where we can script the solution.
The script we use looks like this:
#!/bin/sh
#
# This role fixes using a separate boot partition with pygrub
#
# We receive some arguments from xen-create-image:
prefix=$1 # Path to the mounted domU disk
config=$2 # Name of domU config, given from --role-args <hostname>
# Create a symlink named "boot" in /boot that points back to the same
# directory. Entries in grub's menu.lst on the domU might be prefixed
# with /boot, which is not correct when it is a separate partition.
# It will be mounted by itself by pyGRUB, making the partition
# available on /. This symlink makes both unprefixed and prefixed
# entries work. Note that the boot partition is mounted to /boot.
ln -s . $prefix/boot/boot
# The boot partition must be the first entry of the list of disks in
# the domU config. We use ex to move the entry.
#
# Explanation:
# > regex search for line containing "dev" and "boot" and move there
# > delete line and save it to yank buffer
# > regex search for line starting with "disk" and ending with "[", move there
# > put yanked line after current line
# > write and quit
ex /etc/xen/$config << EOF
/dev.*boot
d
/^disk.*[$
put
wq
EOF
Put the script in /etc/xen-tools/role.d/bootpart
Now you can run
xen-create-image --partitions=xfsroot --finalrole=bootpart --role-args=mydomu.cfg
(+ whatever options you want) to automatically create a working domU with an XFS / and separate ext4 /boot.
Written 2019-10-04 by Rupus Reinefjord