Public
Authored by Jacques Landru

Rancher k3os hybrid multiboot usbkey image howto

personal notes about Rancher K3OS hybrid multiboot (BIOS/GPT/MBR -UEFI/GPT) USB Key image build published under CC-BY-SA

created : 20190828 by J. Landru (IMT Lille Douai / CERI Systemes Numeriques)

modified : 20191008 by J. Landru (Step 4 : complete installation on the master reference image)

These notes are personal notes derived and adapted from previous notes : viminal-hybrid-multi-boot-usb-howto used to build VIMINAL 2.0 hybrid multiboot (BIOS/GPT/MBR - UEFI/GPT) USB key (see https://sourceforge.net/projects/viminal/)

-----oOo-----

Context

For lab usage I plan to deploy a primary k8s cluster used as storage backend cluster. This storage cluster will be based on K3os/k8s/rook/ceph stack. Several secondary dedicated lab clusters will in the future use that primary storage cluster backend. This cluster will be composed of several nodes (reuse of old servers of different generations) with all physical disks dedicated for ceph OSD. For this kind of primary cluster, nodes booting on PXE is probably not the right choice, that's why I put the primary Operating System on USB keys and not on the 1st disk which will be the 1st ceph OSD. As I reuse several generations of old computers, I try to build a unique USB key image able to baremetal boot k3os on BIOS/GPT/MBR or UEFI/GPT computers.

Disclaimer

Warning : all the data on the USB key, will be erased. Make sure this is what you want to do.

Warning (again) : When partitioning or "block copy" (using dd command) on /dev/sdX USB key device or /dev/loopX image file, verify that /dev/sdX or /dev/loopX is your target device. An error on the X device designation, by specifying a wrong device, will irretrievably erase your X device. Do it at you own risk !!!

k3os USB key image build

Nota : The USB key image file will be build on a KVM/libvirt VM as this is my usual VM hypervisor. All steps describe in this howto could probably be adapted in other virtualization environment such Virtualbox or VMware vm players

I) Step 1 : prepare image file used as an hybrid multiboot virtual device

I-a) create virtual device image file

Create a 4GiB image file which will used as disk on our VM using qemu-img tool. For k3os small 4GB usb key is sufficient.

# qemu-img create -f raw cirrus-rancher-k3os-usbkey.img 4G
Formatting 'cirrus-rancher-k3os-usbkey.img', fmt=raw size=4294967296

You can verify your virtual device file characteristics with gdisk -l command.

# gdisk -l cirrus-rancher-k3os-usbkey.img
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.
Disk cirrus-rancher-k3os-usbkey.img: 8388608 sectors, 4.0 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 08027C63-A190-49E4-821B-6D9352A2D00D
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 8388574
Partitions will be aligned on 2048-sector boundaries
Total free space is 8388541 sectors (4.0 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name

I-b) prepare our virtual device as an UEFI conformant device

Using gdisk utility, we structure the virtual device as GPT device

I-b-1) Create 3 GPT partitions on the virtual device using gdisk utilitiy

following the "Hybrid UEFI GPT + BIOS GPT/ MBR boot paragraph of Archlinux "Multiboot USB drive" [2] howto

Note : I read somewhere that for EFI (EF00) partition has to be the 1st one

So using gdisk utility gdisk cirrus-rancher-k3os-usbkey.img

create 3 partitions

  1. An EFI System partition (gdisk type code EF00 with a FAT32 filesystem). This partition can be as small as 50 MiB. I created à 64 MiB,
  2. A BIOS boot partition (gdisk type code EF02 ). This partition can be 1 MiB in size. I created a 2MiB (no need to be formatted).
  3. A data partition (use a filesystem supported by GRUB). This partition can take up the rest of the space of your drive. I create a (gdisk type code 8300 ) (Linux) of 3.5 GiB (don't use the rest of the key as suggested by archlinux howto, as some utility copy a backup of the GPT table a the end of the disk. I had some "partition table overlaps "errors displayed by gdik utility. So I use only 3.5 GiB and not the rest. This data partition will be later ext4 formatted, as k3os rootfs specification[3]. Note : _In gdisk utility when creating this third partition dont size it using +3.5G syntax, this doesn't work size it +3584M instead !! _
# gdisk  cirrus-rancher-k3os-usbkey.img
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.

Command (? for help): n
Partition number (1-128, default 1): 
First sector (34-8388574, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-8388574, default = 8388574) or {+-}size{KMGTP}: +64M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): EF00
Changed type of partition to 'EFI System'

Command (? for help): n
Partition number (2-128, default 2): 
First sector (34-8388574, default = 133120) or {+-}size{KMGTP}: 
Last sector (133120-8388574, default = 8388574) or {+-}size{KMGTP}: +2M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): EF02
Changed type of partition to 'BIOS boot partition'

Command (? for help): n
Partition number (3-128, default 3): 
First sector (34-8388574, default = 137216) or {+-}size{KMGTP}: 
Last sector (137216-8388574, default = 8388574) or {+-}size{KMGTP}: +3584M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'

Command (? for help):

Continue, creating Hybrid MBR partition table as discribe in the "Hybrid UEFI GPT + BIOS GPT/ MBR boot paragraph of Archlinux "Multiboot USB drive" [2] howto

Command (? for help): 
Command (? for help): r

Recovery/transformation command (? for help): h

WARNING! Hybrid MBRs are flaky and dangerous! If you decide not to use one,
just hit the Enter key at the below prompt and your MBR partition table will
be untouched.

Type from one to three GPT partition numbers, separated by spaces, to be
added to the hybrid MBR, in sequence: 1 2 3
Place EFI GPT (0xEE) partition first in MBR (good for GRUB)? (Y/N): Y

Creating entry for GPT partition #1 (MBR partition #2)
Enter an MBR hex code (default EF): 
Set the bootable flag? (Y/N): N

Creating entry for GPT partition #2 (MBR partition #3)
Enter an MBR hex code (default EF): 
Set the bootable flag? (Y/N): N

Creating entry for GPT partition #3 (MBR partition #4)
Enter an MBR hex code (default 83): 
Set the bootable flag? (Y/N): Y

Recovery/transformation command (? for help): x

Expert command (? for help): h

Expert command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to cirrus-rancher-k3os-usbkey.img.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

Verify your hybrid BIOS / UEFI virtual device

# gdisk -l cirrus-rancher-k3os-usbkey.img
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: hybrid
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with hybrid MBR; using GPT.
Disk cirrus-rancher-k3os-usbkey.img: 8388608 sectors, 4.0 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 71DCAF2B-7741-4B97-AA18-7B21A754B1F5
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 8388574
Partitions will be aligned on 2048-sector boundaries
Total free space is 913341 sectors (446.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          133119   64.0 MiB    EF00  EFI System
   2          133120          137215   2.0 MiB     EF02  BIOS boot partition
   3          137216         7477247   3.5 GiB     8300  Linux filesystem

I-b-2) Prepare filesystems for grub and k3os custom installation

  • EFI partition (part 1) needs to be fat32,

Important note : Interactive installation of k3os using the live ISO os-confg tool will format the entire disk. If you have a single hard disk attached to the system it will not ask which disk but just pick the first and only one and create a single partiton on the entire disk. In our case, this will overwrite our EFI/GPT structure. The only way to install k3os on a preformated device is k3os "automated installation" following "Custom partition layout" as indicated at [3].

  • k3os rootfs (part 3) has to be "K3OS_STATE" labeled ext4 partition as mentioned on k3os "Custom partition layout" specifications [3].

Before formating partions on our GPT virtual device file, we have to set it up as loop block device using losetup -P command.

# losetup -P -f --show cirrus-rancher-k3os-usbkey.img 
/dev/loop0

Verify your loop device structure with lsblk command

# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0       7:0    0     4G  0 loop 
├─loop0p1 259:0    0    64M  0 part 
├─loop0p2 259:1    0     2M  0 part 
└─loop0p3 259:2    0   3.5G  0 part 
sda         8:0    0   477G  0 disk 
├─sda1      8:1    0   650M  0 part 
├─sda2      
.
.
.

format EFI partition (part 1) of our virtual device in fat32

# mkfs.fat -F32 /dev/loop0p1
mkfs.fat 4.1 (2017-01-24)

format in ext4 the k3os rootfs partition (part 3) with "K3OS_STATE" label

# mkfs.ext4 -L K3OS_STATE /dev/loop0p3 
mke2fs 1.45.2 (27-May-2019)
Discarding device blocks: done                            
Creating filesystem with 917504 4k blocks and 229376 inodes
Filesystem UUID: 280d29c4-1eef-41c7-9730-5f7a20211db0
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

You can verify your device structure using lsblk options

# lsblk -o FSTYPE,LABEL,SIZE /dev/loop0
FSTYPE LABEL       SIZE
                     4G
vfat                64M
                     2M
ext4   K3OS_STATE  3.5G

Before going further, detach your loop block device using losetup -D

# losetup -D

II) Step 2 : install k3os on you virual EFI/GPT device

In this step we'll install k3os (server or agent) on a dedicated libvirt/kvm virtual machine with our virtual EFI/GPT block device attach as its first disk.

II-a) create a virtual k3os install machine

Create a dedicated install VM, we'll use to install k3os in k3os "automated installation" mode.

My dedicated VM characteristics (lower may works, I don't optimized...)

  • 2 vcpu,
  • Memory : 4GiB,
  • 1 attached virtIO NIC attach on infrastructure lan,
  • IDE CD-ROM attached to k3os install iso file [4],
  • VirtIO Disk 1 attached to our virtual device file in raw mode,

k3os-uefi-device-install-vm-characteristics-935x799

As we'll install k3os following the k3os "automated installation" mode, we need to pass some k3os.install options as kernel paameters at bootup, so we need to activate "direct kernel boot" in "boot options" and specify "kernel path", "initrd path" and "kernel parameters" as ilustrated below

k3os-uefi-device-install-vm-boot-options-935x799

default k3os boot kernel parameters found on k3os ISO file kernel cmdline

  • printk.devkmsg=on
  • console=ttyS0
  • console=tty1

k3os.install kernel cmdline automated installation parameters to install on our custom partition layout device

  • k3os.mode=install
  • k3os.install.force_efi=true
  • k3os.install.no_format=true

Nota : all k3os.install kernel cmdline parameters can be found at k3os "automated installation"

Before lauching the dedicated virtual machine, we have to retrieve initrd and kernel binaries of the k3os iso file and adjust kernel and initrd paths in the VM launch parameters.

Loop mount k3os install is file :

# mount -o loop k3os-amd64-v0.x.y.iso /mnt/tempo/
mount: /mnt/tempo: WARNING: device write-protected, mounted read-only.

initrd is under /mnt/tempo/k3os/system/kernel/current/

#  ls -al /mnt/tempo/k3os/system/kernel/current/
total 292862
dr-xr-xr-x 1 root root      2048 May  8 08:27 .
dr-xr-xr-x 1 root root      2048 May  8 08:27 ..
-r--r--r-- 1 root root  90940892 May  8 08:27 initrd
-r--r--r-- 1 root root 208945152 May  8 08:27 kernel.squashfs

the kernel binary is inside the kernel.squashfs filesystem, so loop mount the kernel.squashfs file and extract vmlinuz kernel binary,

# mkdir /tmp/squashfs-loop
# mount -o loop /mnt/tempo/k3os/system/kernel/current/kernel.squashfs  /tmp/squashfs-loop
# cp /tmp/squashfs-loop/vmlinuz /tmp

then adjust kernel path, initrd path and kernel parametersls in VM parameters. In this example, according to previous commands :

  • kernel path : /tmp/vmlinuz
  • initrd path : /mnt/tempo/k3os/system/kernel/current/initrd
  • kernel parmeters : printk.devkmsg=on console=ttyS0 console=tty1 k3os.mode=install k3os.install.force_efi=true k3os.install.no_format=true

II-b) start the VM to install k3os on the disk image

From the virtual machine console porceed to k3os install (server or agent), giving appropriate interactive responses to adjust config parameters with your own k8s cluster context. Then shutdown the VM, umount loop mounts and free loop devices.

# umount /tmp/squahfs-loop
# umount /mnt/tempo
# losetup -D

III) Step 3 : install Grub in hybrid mode

Before installing Grub on our GPT virtual device file, we have to set it up again as loop block device using losetup -P command.

# losetup -P -f --show cirrus-rancher-k3os-usbkey.img 
/dev/loop0

Verify your loop device structure with lsblk command

# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0       7:0    0     4G  0 loop 
├─loop0p1 259:0    0    64M  0 part 
├─loop0p2 259:1    0     2M  0 part 
└─loop0p3 259:2    0   3.5G  0 part 
sda         8:0    0   477G  0 disk 
├─sda1      8:1    0   650M  0 part 
├─sda2      
.
.
.

First, you need to mount the EFI system partition and the data partition of our virtual USB drive image. Don't forget to create the mountpoint before mounting

# mkdir /tmp/usbk
# mount /dev/loop0p3 /tmp/usbk

Verify there's the boot directory (legacy BIOS boot, needs /boot dir), the EFI mount point for the EFI vfat partition, is /boot/efi.

#  mkdir -p /tmp/usbk/boot/efi

Then mount the EFI partition on /boot/efi mount point.

# mount -t vfat /dev/loop0p1 /tmp/usbk/boot/efi

To install Grub to support both EFI + GPT and BIOS + GPT/MBR, the GRUB configuration (--boot-directory) can be kept in the same place.

A) Install Grub for UEFI using grub-install --target=x86_64-efi --recheck --removable --efi-directory=/EFI_MOUNTPOINT --boot-directory=/DATA_MOUNTPOINT/boot command

# grub-install --target=x86_64-efi --recheck --removable --efi-directory=/tmp/usbk/boot/efi --boot-directory=/tmp/usbk/boot
Installing for x86_64-efi platform.
Installation finished. No error reported.

B) Then install Grub for BIOS with grub-install --target=i386-pc --recheck --boot-directory=/DATA_MOUNTPOINT/boot /dev/loop0 command ( note : for bios legacy boot command you specify the block device /dev/loop0 and not the partition /dev/loop0pX !!! )

# grub-install --target=i386-pc --recheck --boot-directory=/tmp/usbk/boot /dev/loop0
Installing for i386-pc platform.
Installation finished. No error reported.

Then umount the data and EFI partition, and detach your loop block device

# umount -R /tmp/usbk/
# losetup -D

IV) Step 4 : complete installation on the master reference image

We need to run the VM once on that image in read-write mode, some k3os directories and files have to be created to complete installation. Restart the install virtual machine without its CDROM (iso install file) booting on its virtual disk (usb key image). Login with rancher user, on a server image verfify that /var/lib/rancher/k3s/data tree was created, on an agent image verify that /k3os/data tree was created. A this stage, we can also create or copy our configuration files (/var/lib/rancher/k3os/config.yaml or /var/lib/rancher/config.d/* files). Then shutdown the virtual machine.

We have now the master image reference, which can be cloned for our cluster nodes.

  • baremetal nodes : image will be block copied on usb key, to boot node on usb key
  • vm nodes : imgage can be cloned on copy on write file (qcow2 derived from the reference image on our libvirt/kvm environent)

Before block copying this image on USB keys, test it on a virtual machine. To avoid alteration, just derive a qcow2 image from that reference image and start a new VM with its 1rst virtio disk attached to that image.

qemu-img create -f qcow2 -o backing_file=cirrus-rancher-k3os-usbkey.img cirrus-rancher-k3os-test.qcow2

Start a new virtual machine on that cirrus-rancher-k3os-test.qcow2 virtual disk.

It works, but strangly on tty1 login hangs while passwd typing. No hanging on other ttyX, on tty1 after login timeout, second login works. You can now stop that test VM and delete its virtual disk file.

V) Step 5 : Cloning a set of USB keys from the reference image

Transposition of that image reference on real USB keys will done by block copying the image on USB key device using dd command.

Reminder warning: all the data on the target USB key, will be erased. Make sure this is what you want to do.

Reminder warning: When partitioning or "block copy" (using dd command) on /dev/sdX device, verify /dev/sdX is your target usb key device. An error on the X device designation, by specifying a wrong device, will irretrievably erase your X device. Do it at you own risk !!! Plug your target key. Find the USB key device (/dev/sdX) using dmesg or lsblk commande

# dmesg

or

# lsblk

Block copy the image on the target key

#  dd if=cirrus-rancher-k3os-usbkey.img of=/dev/sdc bs=8M status=progress

Be patient ...!! copy can be long.

Strangly, the key boots with no success. Maybe dd modifiy the last sector on the key. I don't find wich dd parameter to adjust to avoid that error !!

That last sector is used by GPT as header backup of the header GPT table. gdisk -l /dev/sdX display that error.

# gdisk -l /dev/sdX
GPT fdisk (gdisk) version 1.0.3

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: hybrid
  BSD: not present
  APM: not present
  GPT: damaged

Found valid MBR and corrupt GPT. Which do you want to use? (Using the
GPT MAY permit recovery of GPT data.)
 1 - MBR
 2 - GPT
 3 - Create blank GPT

Your answer:

We nedd to repair the GPT backup structure on the keys by selcting choice 2, and rewriting the GPT table.

# gdisk  /dev/sdb
GPT fdisk (gdisk) version 1.0.3

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: hybrid
  BSD: not present
  APM: not present
  GPT: damaged

Found valid MBR and corrupt GPT. Which do you want to use? (Using the
GPT MAY permit recovery of GPT data.)
 1 - MBR
 2 - GPT
 3 - Create blank GPT

Your answer: 2

Command (? for help): w
Caution! Secondary header was placed beyond the disk s limits! Moving the
header, but other problems may occur!

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.

lsblk now show the right key GPT structure.

You can now unplug the key and use it to bare metal boot one cluster node !

Nota : During test some images don't boot and display "pfatal Failed to determine boot mode" before giving a bash prompt in rescue mode. It seems to be a known problem see [5] : manually editing grub.cfg on /boot/grub on the third partition to replace the "current" destination to "4.15.0-47-generic" inside the grub 'sqfile' and 'initrd' entries, allow to boot to succeed.

-----oOo-----

URLography :

-----oOo-----

Edited
Rancher k3os hybrid multiboot usbkey image howto 11 KB
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment