It will be a very long story, which is why it is called Saga. It will try to investigate all aspects of the bootstrap process. However, in these examples we will only be using GRUB2, which is why it appears in the title. The OS of choice for this research is CentOS 8.2, the changes specific to this OS will be reflected. This article involves a virtual machine called "centos8", a host server called "kvmhost" and standalone "dhcphost".
We will use KVM for our LAB of course. It can use SeaBIOS (for BIOS boot), TianoCore (for UEFI boot) with and without SecureBoot. During this LAB, we will heavily reformat our boot drive. The OS payload should be left intact. Obviously, it should be on a separate second hard drive. To prepare my testing environment, I've installed minimal CentOS 8.2 on single disk accepting the installer defaults. Then I've added a second disc.
NOTE: I would like to take this opportunity to repeat my opinion on the partitioning of disks intended for LVM. If you have a single bootable disk, then obviously the boot procedure will require at least one additional boot partition or reserved space before of the OS partition. Then you cannot avoid the disk partitioning, and LVM will use the partition as a physical volume. The situation changes when a second or more disks are added to the system. If you intend to use LVM (which is really useful and highly recommended), then there is no need to create any partition on additional disks. I can't figure out the single reason why all Linux storage guides recommend creating a full disk partition first and just then creating LVM on it. There are no advantages to this, the only thing you get is problems with dynamically resizing the disk.As I've already said, I've added the second disk and migrated my OS to it as following:
root@centos8:~ # cat /proc/partitions major minor #blocks name 252 0 20971520 vda 252 1 1048576 vda1 252 2 19921920 vda2 11 0 1048575 sr0 253 0 17821696 dm-0 253 1 2097152 dm-1 252 16 20971520 vdb root@centos8:~ # pvs PV VG Fmt Attr PSize PFree /dev/vda2 cl lvm2 a-- <19.00g 0 root@centos8:~ # pvcreate /dev/vdb # <- the whole disk used as PV, no partitions needed. Physical volume "/dev/vdb" successfully created. root@centos8:~ # vgextend cl /dev/vdb Volume group "cl" successfully extended root@centos8:~ # pvmove /dev/vda2 .. /dev/vda2: Moved: 100.00% root@centos8:~ # pvs PV VG Fmt Attr PSize PFree /dev/vda2 cl lvm2 a-- <19.00g <19.00g /dev/vdb cl lvm2 a-- <20.00g 1.00g root@centos8:~ # vgreduce cl /dev/vda2 Removed "/dev/vda2" from volume group "cl" root@centos8:~ # pvremove /dev/vda2 Labels on physical volume "/dev/vda2" successfully wiped. root@centos8:~ # pvs PV VG Fmt Attr PSize PFree /dev/vdb cl lvm2 a-- <20.00g 1.00g
Reboot the system to check that we haven't broke anything. The OS should boot without problems. As a result of our manipulations, the LVM OS migrated to the second disk, and the first disk sill contains only boot-related partitions. In the next step, we will format the first disk, and the "/boot" partition will be lost with all its data. Let's copy its contents to the root filesystem for safekeeping.
root@centos8:~ # df /boot Filesystem Size Used Avail Use% Mounted on /dev/vda1 976M 124M 786M 14% /boot root@centos8:~ # umount /boot root@centos8:~ # mount /dev/vda1 /mnt root@centos8:~ # yum install -y rsync root@centos8:~ # rsync -av /mnt/ /boot/
Messing with the boot procedure, you run into the risk of getting your system unbootable. Therefore, we must have some quick way of rescue. It is of course always possible to boot from the installation CD in rescue mode. But this time, we'll take advantage of KVM's ability for Direct Kernel Boot.
This is not a very common type of boot method. I saw its implementation in a "petitboot" in OPAL running on IBM Power servers, I've read that this is possible in "coreboot" and sure we'll do it in KVM now.
Locate the current kernel and its initrd image in the /boot directory and copy them from the VM to "kvmhost":
kvmhost:~ $ scp root@centos8:/boot/vmlinuz-4.18.0-193.el8.x86_64 /var/tmp/centos8.krl vmlinuz-4.18.0-193.el8.x86_64 100% 8705KB 291.3MB/s 00:00 kvmhost:~ $ scp root@centos8:/boot/initramfs-4.18.0-193.el8.x86_64.img /var/tmp/centos8.ird initramfs-4.18.0-193.el8.x86_64.img 100% 27MB 336.8MB/s 00:00
Find important boot kernel options:
root@centos8:~ # cat /proc/cmdline BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-193.el8.x86_64 root=/dev/mapper/cl-root ro resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
We need only information about root location, marked as red. Shutdown the VM and enable "Direct kernel boot" at "Boot Options" at virt-namager GUI. Alternatively, you can directly edit os section of VM configuration using virsh edit command:
root@kvmhost:~ # virsh edit centos8
..
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<kernel>/var/tmp/centos8.krl</kernel>
<initrd>/var/tmp/centos8.ird</initrd>
<cmdline>root=/dev/mapper/cl-root ro</cmdline>
<os>
..
Now turn on the VM to see the direct kernel boot. There is no grub menu during startup, and this may indicate that you have performed a direct kernel boot.
We will repeat these steps each time before starting a new exercise. Strip out all mounts related to "/boot" from /etc/fstab and shutdown VM.
root@centos8:~ # sed -e '/\/boot/d' -i /etc/fstab root@centos8:~ # poweroff
Re-create the first disk if needed:
root@kvmhost:~ # qemu-img create -f qcow2 $(virsh domblklist centos8 | awk '/vda/{print $2}') 20g
root@kvmhost:~ # virsh edit centos8
Edit VM configuration (if needed) according to desired setup:
<-- BIOS -->
..
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<kernel>/var/tmp/centos8.krl</kernel>
<initrd>/var/tmp/centos8.ird</initrd>
<cmdline>root=/dev/mapper/cl-root ro</cmdline>
<os>
..
<-- UEFI -->
..
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/centos8_VARS.fd</nvram>
<kernel>/var/tmp/centos8.krl</kernel>
<initrd>/var/tmp/centos8.ird</initrd>
<cmdline>root=/dev/mapper/cl-root ro</cmdline>
<os>
..
The "nvram" file used by QEMU to keep EFI variables. If you want to start over with default variables, you need copy it over from template:
root@kvmhost:~ # cp /usr/share/edk2/ovmf/OVMF_VARS.fd /var/lib/libvirt/qemu/nvram/centos8_VARS.fd root@kvmhost:~ # chmod 600 /var/lib/libvirt/qemu/nvram/centos8_VARS.fd root@kvmhost:~ # chown qemu:qemu /var/lib/libvirt/qemu/nvram/centos8_VARS.fd
<-- UEFI SecureBoot -->
..
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/centos8_VARS.fd</nvram>
<kernel>/var/tmp/centos8.krl</kernel>
<initrd>/var/tmp/centos8.ird</initrd>
<cmdline>root=/dev/mapper/cl-root ro</cmdline>
<os>
..
This "nvram" file is not compatible with previous and should be copied from other template:
root@kvmhost:~ # cp /usr/share/edk2/ovmf/OVMF_VARS.secboot.fd /var/lib/libvirt/qemu/nvram/centos8_VARS.fd root@kvmhost:~ # chmod 600 /var/lib/libvirt/qemu/nvram/centos8_VARS.fd root@kvmhost:~ # chown qemu:qemu /var/lib/libvirt/qemu/nvram/centos8_VARS.fd
Make a cleanup for LAB environment referring to BIOS setup and re-creating first drive.
Boot the virtual machine using direct kernel boot and log in. Format the first disk as MBR using the "fdisk" utility. Create the only /boot partition. The result should be similar to:
root@centos8:~ # fdisk -l /dev/vda Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x5610d1f9 Device Boot Start End Sectors Size Id Type /dev/vda1 * 2048 526335 524288 256M 83 Linux
Format the resulting partition, copy "/boot/" content to it and mount as "/boot". Also add it to fstab.
root@centos8:~ # mkfs.ext2 -j -m0 /dev/vda1 .. root@centos8:~ # mount /dev/vda1 /mnt root@centos8:~ # rsync -av /boot/ /mnt/ .. root@centos8:~ # umount /mnt root@centos8:~ # mount /dev/vda1 /boot root@centos8:~ # df Filesystem Size Used Avail Use% Mounted on devtmpfs 969M 0 969M 0% /dev tmpfs 985M 0 985M 0% /dev/shm tmpfs 985M 8.6M 977M 1% /run tmpfs 985M 0 985M 0% /sys/fs/cgroup /dev/mapper/cl-root 8.0G 1.6G 6.5G 19% / tmpfs 197M 0 197M 0% /run/user/0 /dev/vda1 240M 145M 92M 62% /boot root@centos8:~ # echo "/dev/vda1 /boot ext2 defaults 1 2" >> /etc/fstab
Next is about installing GRUB2 loader on first disk:
root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... done root@centos8:~ # grub2-install /dev/vda Installing for i386-pc platform. Installation finished. No error reported. root@centos8:~ # poweroff
Shut down the virtual machine and disable direct kernel boot, then start it again. It should boot well.
What's going on here? GRUB2 is installed into MBR (512 bytes) and continues after it, using a 2m gap before the first partition. This image can read ext2, which contains the rest of GRUB2 modules. There is also the grub.cfg configuration file. GRUB2 reads it and acts accordingly, loads modules, kernel, initrd and starts them.
When looking at the newly created /boot/grub2/grub.cfg, no any "menuentry" were found. The file is divided into several sections. The section responsible for booting the current Linux installation is called 10_linux. The section that takes care detecting other OSes installed on your computer is called 30_os-prober.
Section 10_linux now only includes a general set of hints on where to find the GRUB root (this is /boot, not to be confused with Linux root). It then sets some default values for the kernel parameters and calls the blscfg module (acronym for BootLoaderSpec). This module will generate menu items on the fly. The information about available kernels are provided by the kernel packages and placed in the /boot/loader/entries directory. Check the link for details. This change eliminates updating the grub.cfg every time a new kernel is installed.
You can disable this behavior and revert to the old hard-coded menuentries by adding GRUB_ENABLE_BLSCFG=false to /etc/default/grub. It is emphasized that in this case you will need to re-create the config file manually for each new kernel installation.
This innovation adds another place for changes to the kernel command line. After you changed GRUB_CMDLINE_LINUX in /etc/default/grub and updated the GRUB2 config file with "grub2-mkconfig", the only thing that changed is the "set default_kernelopts" line in the config file. The problem is that the BLS file uses the "$kernelopts" variable and not "$default_kernelopts". You should check /boot/grub2/grubenv additionally, because the real values might be there. Even more, the broken or empty "grubenv" file could cause booting problems.
The initrd image is now merged from two images, the first of which is called early_cpio. You can get its contents with the usual cpio command. If you want to check the real, second content of the initrd, you should use the additional utility lsinitrd, which can also be used to extract it.
root@centos8:~ # lsinitrd --help .. --unpack unpack the initramfs, instead of displaying the contents. ..
Make a cleanup for LAB environment referring to BIOS setup and re-creating first drive.
Log into the virtual machine and create an MSDOS Partition Table (MBR). Create one partition of any type (NTFS in my case). This partition will not be used, but it must exist, creating a gap between the MBR and its beginning. The result should be similar:
root@centos8:~ # fdisk -l /dev/vda Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x73200e96 Device Boot Start End Sectors Size Id Type /dev/vda1 2048 41943039 41940992 20G 7 HPFS/NTFS/exFAT
Then repeat the GRUB2 installation:
root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... done root@centos8:~ # grub2-install /dev/vda Installing for i386-pc platform. Installation finished. No error reported.
Check the resulting /boot/grub2/grub.cfg file. You will see that the 10_linux section now points to LVM.
Shut down the virtual machine and configure it to boot from the first disk. The boot will failed. This is because grub looks for "/vmlinux" and "/initrd" as it described at BLS files, when they are "/boot/vmlinux" and "/boot/initrd" in current situation. We will not fight with the system, but we will find a workaround. Boot the virtual machine either with grub, adding "/boot" before the kernel and initrd, or direct kernel boot. Then copy the existing BLS file and modify it:
root@centos8:~ # cd /boot/loader/entries/ root@centos8:/boot/loader/entries # ll total 8 -rw-r--r--. 1 root root 395 Aug 5 08:42 d4067e945b934d09acfb204c36603fb8-0-rescue.conf -rw-r--r--. 1 root root 323 Aug 5 08:42 d4067e945b934d09acfb204c36603fb8-4.18.0-193.el8.x86_64.conf root@centos8:/boot/loader/entries # cp d4067e945b934d09acfb204c36603fb8-4.18.0-193.el8.x86_64.conf 4.18.0-193.el8-boot.conf root@centos8:/boot/loader/entries # vi 4.18.0-193.el8-boot.conf root@centos8:/boot/loader/entries # cat 4.18.0-193.el8-boot.conf title CentOS Linux (4.18.0-193.el8.x86_64) 8 (Core) - Prepend vith /boot version 4.18.0-193.el8.x86_64 linux /boot/vmlinuz-4.18.0-193.el8.x86_64 initrd /boot/initramfs-4.18.0-193.el8.x86_64.img $tuned_initrd options $kernelopts $tuned_params id centos-20200508110711-4.18.0-193.el8.x86_64 grub_users $grub_users grub_arg --unrestricted grub_class kernel
Changes are shown in bold. Now try booting from disk again and it worked this time.
Why? How did the boot work without a single relevant partition on the boot disk? As already explained, the installation script prepares the GRUB2 image and installs it in the MBR and immediately afterwards until the next partition. If you do not create the partition at all, the installation script will fail, as there is no guaranteed gap for it to work.
The image itself is prepared for a specific system. That is, this our image already includes additional LVM and XFS modules to successfully read the rest of the information, such as the "grub.cfg" file.
Generally speaking, the BIOS does not know how to work with GPT disks, that is why the UEFI appeared. But most of modern UEFI can work in BIOS compatibility mode, and then you can get a situation when the BIOS do boot from a GPT disk.
There is no MBR and no gap up to the first partition for installing a GRUB2 image on GPT disk. A special partition was invented to make room for the GRUB2 installation.
Make a cleanup for LAB environment referring to BIOS setup and re-creating first drive.
Start the virtual machine and format /dev/vda with "parted", a common GPT tool:
root@centos8:~ # parted -a optimal /dev/vda GNU Parted 3.2 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary 1 3 (parted) name 1 grub (parted) set 1 bios_grub on (parted) print Model: Virtio Block Device (virtblk) Disk /dev/vda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 3.00MiB 2.00MiB grub bios_grub (parted) q Information: You may need to update /etc/fstab. root@centos8:~ # cat /proc/partitions major minor #blocks name 252 0 20971520 vda 252 1 2048 vda1 252 16 20971520 vdb 11 0 1048575 sr0 253 0 8388608 dm-0 253 1 1048576 dm-1 root@centos8:~ #
Then repeat the GRUB2 installation:
root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... done root@centos8:~ # grub2-install /dev/vda Installing for i386-pc platform. Installation finished. No error reported.
Re-creating /boot/grub2/grub.cfg is not strictly required because nothing has been changed here. Shut down the virtual machine, configure it to boot from the first disk and observe the OS boot. Use our custom "Prepend vith /boot" grub menu entry to successfully boot OS.
The version with a separate /boot partition will not be discussed here, because it looks too obvious and boring.
UEFI can boot from both GPT or MBR disk. It does not use boot sector magic, but looks for its own partition, usually called ESP (EFI System Partition). This FAT formatted partition should include any next stage ".efi" loader:
# file /boot/efi/EFI/Boot/bootx64.efi /boot/efi/EFI/Boot/bootx64.efi: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows
All possible bootloaders must be registered with UEFI and this information is stored in NVRAM as EFI variables. ESP can be used to store other files related to the bootloader, for example Microsoft will use about 100m of space for their files. In one of the examples below, ESP will be used as the "/boot" partition, so it should be large enough to hold at least three kernel versions. As opposite, the ESP could be an extremely small, when use the single GRUB2 loader which will not take more than 1m.
The EFI-compliant GRUB2 has a different format than the one used to boot with BIOS and is called "grubx64.efi". The installation script uses a generic image and ignores GRUB_PRELOAD_MODULES. It is common practice to use a separate easy accessible "/boot" partition containing the GRUB2 modules, kernel and initrd.
Make a cleanup for LAB environment referring to UEFI setup and re-creating first drive.
Create an ESP and /boot partition using fdisk utility and MBR partitioning scheme like:
root@centos8:~ # fdisk -l /dev/vda Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x017d19f1 Device Boot Start End Sectors Size Id Type /dev/vda1 2048 526335 524288 256M ef EFI (FAT-12/16/32) /dev/vda2 526336 1050623 524288 256M 83 Linux
Pay attention that ESP should be 0xEF type in MBR scheme.
Format the first partition as FAT, the second as ext2, copy content of "/boot" to second, mount it as /boot, mount first partition as /boot/efi and add corresponding lines to /etc/fstab:
root@centos8:~ # yum install -y dosfstools root@centos8:~ # mkdosfs /dev/vda1 mkfs.fat 4.1 (2017-01-24) root@centos8:~ # mkfs.ext2 -j -m0 /dev/vda2 .. root@centos8:~ # echo "/dev/vda2 /boot ext2 defaults 1 2" >> /etc/fstab root@centos8:~ # echo "/dev/vda1 /boot/efi vfat defaults 0 0" >> /etc/fstab root@centos8:~ # mount /dev/vda2 /mnt root@centos8:~ # rsync -a /boot/ /mnt/ root@centos8:~ # umount /mnt root@centos8:~ # mount /boot root@centos8:~ # mkdir /boot/efi root@centos8:~ # mount /boot/efi root@centos8:~ # df /boot /boot/efi Filesystem Size Used Avail Use% Mounted on /dev/vda2 240M 125M 112M 53% /boot /dev/vda1 256M 0 256M 0% /boot/efi
As you remember, when we created the LAB, we copied the original contents of "/boot/" to the root filesystem. Therefore, we now copy back when creating a separate "/boot" filesystem.
Install GRUB2 efi binary into ESP registering it as "centos" boot entry:
root@centos8:~ # yum install -y grub2-efi-x64-modules efibootmgr root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... Adding boot menu entry for EFI firmware configuration done root@centos8:~ # grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=centos Installing for x86_64-efi platform. Installation finished. No error reported. root@centos8:~ # find /boot/efi /boot/efi /boot/efi/EFI /boot/efi/EFI/centos /boot/efi/EFI/centos/grubx64.efi root@centos8:~ # efibootmgr -v Timeout: 0 seconds BootOrder: 0001,0000 Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331) Boot0001* centos HD(1,MBR,0x568a017f,0x800,0x80000)/File(\EFI\centos\grubx64.efi)
Poweroff VM, make it boot from first disk instead of direct kernel boot and turn it on.
UEFI works with GPT disks in the same way as with MBR disks - it looks for an ESP and launch a registered bootloader that it finds there. Just because repeating the previous exercise with such minor changes seemed too boring to me, I decided to make it more interesting by merging the contents of the "/boot" filesystem with the ESP content (just like Microsoft does).
Make a cleanup for LAB environment referring to UEFI setup and re-creating first drive. Do not forget to restore default UEFI variables in "nvram" file.
Turn on the VM and make GPT partition using parted tool:
root@centos8:~ # parted -a optimal /dev/vda GNU Parted 3.2 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary fat32 1 399 (parted) name 1 "EFI System Partition" (parted) set 1 esp on (parted) print Model: Virtio Block Device (virtblk) Disk /dev/vda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 399MiB 398MiB fat32 EFI System Partition boot, esp (parted) quit Information: You may need to update /etc/fstab.
Then format the ESP as vfat, copy the content of "/boot" to it and mount it as /boot:
root@centos8:~ # mkdosfs /dev/vda1 mkfs.fat 4.1 (2017-01-24) root@centos8:~ # mount -t vfat /dev/vda1 /mnt root@centos8:~ # rsync -av /boot/ /mnt/ root@centos8:~ # umount /mnt root@centos8:~ # echo "/dev/vda1 /boot vfat defaults 0 0" >> /etc/fstab root@centos8:~ # mount /boot root@centos8:~ # df /boot Filesystem Size Used Avail Use% Mounted on /dev/vda1 300M 125M 176M 42% /boot
Installing GRUB2 is not much different, just adjust the options like:
root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... Adding boot menu entry for EFI firmware configuration done root@centos8:~ # grub2-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=grub2 Installing for x86_64-efi platform. Installation finished. No error reported. root@centos8:~ # efibootmgr -v Timeout: 0 seconds BootOrder: 0001,0000 Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331) Boot0001* grub2 HD(1,GPT,99bc1cdb-19cb-4ae5-8692-97106120b12b,0x800,0xc7000)/File(\EFI\grub2\grubx64.efi)
Poweroff VM, make it boot from first disk instead of direct kernel boot and turn it on.
It has already been explained that in a UEFI environment, the GRUB2 installation script does not generate a custom GRUB2 image, but uses a generic one. This is why there are no success stories about UEFI boot from "/boot" hosted on LVM. But there is no such limitation when using lower-level commands and performing actions manually. We will create our own custom "grubx64.efi" image in this exersize.
NOTE: This may be a bug in the current TianoCore implementation for KVM. Or maybe this is the usual behavior for UEFI firmware. But!! GRUB2 will not see a hard drive unless it is marked as a candidate to boot. In this chapter, GRUB2 will boot the kernel and initrd from LVM located on the second disk, then it should "see" it. You must add second disk to the boot list as the first or second disc - the order does not matter.
Make a cleanup for LAB environment referring to UEFI setup and re-creating first drive. Do not forget to restore default UEFI variables in "nvram" file.
Format the first disk with the parted tool, this time we'll make the ESP tiny.
root@centos8:~ # parted -a optimal /dev/vda GNU Parted 3.2 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary fat32 1 3 (parted) name 1 "EFI System Partition" (parted) set 1 esp on (parted) print Model: Virtio Block Device (virtblk) Disk /dev/vda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 3.00MiB 2.00MiB fat32 EFI System Partition boot, esp (parted) quit Information: You may need to update /etc/fstab. root@centos8:~ # mkdosfs /dev/vda1 mkfs.fat 4.1 (2017-01-24) root@centos8:~ # echo "/dev/vda1 /boot/efi vfat defaults 0 0" >> /etc/fstab root@centos8:~ # mount /boot/efi
First we need an additional GRUB2 configuration file that will be inserted into the image.
root@centos8:~ # cat early-grub.cfg set root='lvm/cl-root' # Point to root LV set prefix=($root)/boot/grub2 # Where grub modules resides configfile $prefix/grub.cfg # Switch execute original grub.cfg
As you can see, there is no insmod command, as most of the modules will be in our custom image. Let's create this custom image:
root@centos8:~ # df /boot /boot/efi Filesystem Size Used Avail Use% Mounted on /dev/mapper/cl-root 17G 1.6G 16G 10% / /dev/vda1 2.0M 0 2.0M 0% /boot/efi root@centos8:~ # mkdir /boot/efi/custom root@centos8:~ # grub2-mkimage -c early-grub.cfg -o /boot/efi/custom/grubx64.efi -O x86_64-efi -p /boot/grub2 \ disk lvm diskfilter xfs normal configfile root@centos8:~ # find /boot/efi /boot/efi /boot/efi/custom /boot/efi/custom/grubx64.efi
We should register new UEFI boot entry manually:
root@centos8:~ # efibootmgr -c -l '/custom/grubx64.efi' -L custom -d /dev/vda -p 1 Timeout: 0 seconds BootOrder: 0001,0000 Boot0000* UiApp Boot0001* custom root@centos8:~ # efibootmgr -v Timeout: 0 seconds BootOrder: 0001,0000 Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331) Boot0001* custom HD(1,GPT,90cd8499-235b-4e9d-8014-c98b1a69f566,0x800,0x1000)/File(\custom\grubx64.efi)
Check that you have grub modules at /boot/grub2/x86_64-efi/ location. If not, replicate them and do not forget to rebuild grub.cfg:
root@centos8:~ # rsync -av /usr/lib/grub/x86_64-efi/ /boot/grub2/x86_64-efi/ root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg
Now, poweroff the VM, disable direct kernel boot and turn it on. Use our custom "Prepend vith /boot" grub menu entry to successfully boot OS.
By the way, a side effect of enabling Secure Boot is to disable hibernation ability.
Make a cleanup for LAB environment referring to UEFI SecureBoot setup and re-creating first drive. Do not forget to restore default UEFI variables in "nvram" file.
Now we will repeat all the steps to prepare a classic UEFI boot from a GPT disk and a separate "/boot" partition. We have done this so many times that a description is no longer required, a list of commands will be enough.
root@centos8:~ # parted -a optimal /dev/vda GNU Parted 3.2 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary fat32 1 99 (parted) name 1 "EFI System Partition" (parted) set 1 esp on (parted) mkpart primary ext2 100 399 (parted) name 2 "Linux /boot" (parted) print Model: Virtio Block Device (virtblk) Disk /dev/vda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 99.0MiB 98.0MiB fat32 EFI System Partition boot, esp 2 100MiB 399MiB 299MiB ext2 Linux /boot (parted) quit Information: You may need to update /etc/fstab. root@centos8:~ # mkdosfs /dev/vda1 mkfs.fat 4.1 (2017-01-24) root@centos8:~ # mkfs.ext2 -j -m0 /dev/vda2 .. root@centos8:~ # mount /dev/vda2 /mnt root@centos8:~ # rsync -a /boot/ /mnt/ root@centos8:~ # umount /mnt root@centos8:~ # echo "/dev/vda2 /boot ext2 defaults 1 2" >> /etc/fstab root@centos8:~ # echo "/dev/vda1 /boot/efi vfat defaults 0 0" >> /etc/fstab root@centos8:~ # mount -a root@centos8:~ # df /boot /boot/efi Filesystem Size Used Avail Use% Mounted on /dev/vda2 282M 129M 154M 46% /boot /dev/vda1 98M 0 98M 0% /boot/efi root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... Adding boot menu entry for EFI firmware configuration done root@centos8:~ # grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=centos Installing for x86_64-efi platform. Installation finished. No error reported. root@centos8:~ # efibootmgr -v Timeout: 0 seconds BootOrder: 0003,0001,0002,0000 Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331) Boot0001* UEFI Misc Device PciRoot(0x0)/Pci(0x2,0x3)/Pci(0x0,0x0)N.....YM....R,Y. Boot0002* UEFI Misc Device 2 PciRoot(0x0)/Pci(0x2,0x6)/Pci(0x0,0x0)N.....YM....R,Y. Boot0003* centos HD(1,GPT,4acfd403-f06d-47fb-bac2-ed096cb62dcd,0x800,0x31000)/File(\EFI\centos\grubx64.efi)
Looks good, and if SecureBoot were disabled the UEFI boot would be successful. However, after the usual poweroff of the virtual machine, disabling direct kernel boot and turning on the VM, we will see the message:
BdsDxe: loading Boot0003 "centos" from HD(1,GPT,4ACFD403-F06D-47FB-BAC2-ED096CB62DCD,0x800,0x31000)/\EFI\centos\grubx64.efi BdsDxe: failed to load Boot0003 "centos" from HD(1,GPT,4ACFD403-F06D-47FB-BAC2-ED096CB62DCD,0x800,0x31000)/\EFI\centos\grubx64.efi: Access Denied BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x3)/Pci(0x0,0x0): Not Found BdsDxe: failed to load Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x2,0x6)/Pci(0x0,0x0): Not Found BdsDxe: No bootable option or device was found. BdsDxe: Press any key to enter the Boot Manager Menu.
What's going on here and what's missing? Secure Boot is a trusted chain, and the EFI bootloader does not trust the grubx64.efi file. A middleware called shim is invented for trusted key management and verification.
Reboot the virtual machine using direct kernel boot and install (reinstall/update) the shim-x64 package (-x64 is part of the name). Its content is self-describing:
root@centos8:~ # yum install -y shim-x64 .. root@centos8:~ # rpm -ql shim-x64 /boot/efi/EFI/BOOT/BOOTX64.EFI /boot/efi/EFI/BOOT/fbx64.efi /boot/efi/EFI/centos/BOOTX64.CSV /boot/efi/EFI/centos/mmx64.efi /boot/efi/EFI/centos/shimx64-centos.efi /boot/efi/EFI/centos/shimx64.efi
The first line files are helpers to register the correct entry at boot manager. The last file is a real shim, signed by Microsoft via Verisign. It should be loaded instead of the GRUB2 bootloader. Delete the registered boot record but there is no need to register a new one, it will be added automatically by the mentioned helpers.
root@centos8:~ # efibootmgr -v Timeout: 0 seconds BootOrder: 0003,0001,0002,0000,0004,0005,0006,0007,0008 Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331) Boot0001* UEFI Misc Device PciRoot(0x0)/Pci(0x2,0x3)/Pci(0x0,0x0)N.....YM....R,Y. Boot0002* UEFI Misc Device 2 PciRoot(0x0)/Pci(0x2,0x6)/Pci(0x0,0x0)N.....YM....R,Y. Boot0003* centos HD(1,GPT,4acfd403-f06d-47fb-bac2-ed096cb62dcd,0x800,0x31000)/File(\EFI\centos\grubx64.efi) Boot0004* UEFI QEMU DVD-ROM QM00001 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,65535,0)N.....YM....R,Y. Boot0005* UEFI PXEv4 (MAC:525400F34C4A) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400f34c4a,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y. Boot0006* UEFI PXEv6 (MAC:525400F34C4A) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400f34c4a,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y. Boot0007* UEFI HTTPv4 (MAC:525400F34C4A) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400f34c4a,1)/IPv4(0.0.0.00.0.0.0,0,0)/Uri()N.....YM....R,Y. Boot0008* UEFI HTTPv6 (MAC:525400F34C4A) PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400f34c4a,1)/IPv6([::]:<->[::]:,0,0)/Uri()N.....YM....R,Y. root@centos8:~ # efibootmgr -b3 -B Timeout: 0 seconds BootOrder: 0001,0002,0000,0004,0005,0006,0007,0008 Boot0000* UiApp Boot0001* UEFI Misc Device Boot0002* UEFI Misc Device 2 Boot0004* UEFI QEMU DVD-ROM QM00001 Boot0005* UEFI PXEv4 (MAC:525400F34C4A) Boot0006* UEFI PXEv6 (MAC:525400F34C4A) Boot0007* UEFI HTTPv4 (MAC:525400F34C4A) Boot0008* UEFI HTTPv6 (MAC:525400F34C4A)
Shim is controlled by mokutil tool (MOK - Machine Owner Key). Its manual talks about the --disable-validation option, which should effectively disable the continuation of the secure boot process after shim boots. However, I have not been able to prove it works, so we will stick to a secure boot strategy. This tool is also useful for checking if Secure Boot is enabled:
root@centos8:~ # mokutil --sb-state SecureBoot enabled
Installing shim-x64 is not enough. It will boot well, but in the next step, GRUB2 boot fails because the previously used "grub2-install" command had installed an unsigned version of grub. Therefore, as a last resort, the shim will run a file named mmx64.efi, which is a boot time mokutil. We will skip this test and install (reinstall/update) the grub2-efi-x64 package.
root@centos8:~ # yum reinstall grub2-efi-x64 .. root@centos8:~ # rpm -ql grub2-efi-x64 /boot/efi/EFI/centos/fonts /boot/efi/EFI/centos/grub.cfg /boot/efi/EFI/centos/grubenv /boot/efi/EFI/centos/grubx64.efi /boot/grub2/grubenv /boot/loader/entries /etc/grub2-efi.cfg
The grub.cfg file mentioned here is empty, but should be able to load the main source configuration file. Let's create the correct one:
root@centos8:~ # cat /boot/efi/EFI/centos/grub.cfg set pager=1 # will help at interactive mode set root='hd0,gpt2' # our /boot is on second partition GPT formatted first disk set prefix=($root)/grub2 # prefix defines the place GRUB2 will search for its modules configfile $prefix/grub.cfg # use this file as current config file
Now let's test the entire boot procedure, shutdown the virtual machine, remove the direct kernel boot, and start the VM. Boot seams work, but SecureBoot refuses to load kernel due to unknown signature:
error: ../../grub-core/loader/i386/efi/linux.c:215:(hd0,gpt2)/vmlinuz-4.18.0-193.el8.x86_64 has invalid signature. error: ../../grub-core/loader/i386/efi/linux.c:94:you need to load the kernel first. Press any key to continue...
The final step is to add the kernel signature to the shim database using mokutil. The kernel certificate comes with the kernel package and can be found in /usr/share/doc/kernel-keys/$(uname -r)/kernel-signing-ca.cer. Boot the VM using direct kernel boot and fix this by:
root@centos8:~ # mokutil --import /usr/share/doc/kernel-keys/$(uname -r)/kernel-signing-ca.cer input password: input password again:
Enter any password you can remember. It will be used only once on the next reboot. Power off the virtual machine, remove direct kernel boot, and power on the VM. A blue mokutil menu will appear, select Enroll MOK to continue, you will be prompted for a password as final confirmation. Then select Reboot from the menu.
The OS has finally booted using SecureBoot.
The previous part was easy, wasn't it? We will now create our custom "grubx64.efi" image and sign it with mokutil to enable SecureBoot for the custom image.
Make a cleanup for LAB environment referring to UEFI SecureBoot setup and re-creating first drive. Do not forget to restore default UEFI variables in "nvram" file.
We will repeat the UEFI boot: GPT disk without /boot partition work with small changes at the end. Please do not forget the previous NOTE. The second disk should present in boot sequense.
root@centos8:~ # parted -a optimal /dev/vda GNU Parted 3.2 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary fat32 1 9 (parted) name 1 "EFI System Partition" (parted) set 1 esp on (parted) print Model: Virtio Block Device (virtblk) Disk /dev/vda: 20480MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 9000kB 7952kB fat32 EFI System Partition boot, esp (parted) quit Information: You may need to update /etc/fstab. root@centos8:~ # mkdosfs /dev/vda1 mkfs.fat 4.1 (2017-01-24) root@centos8:~ # echo "/dev/vda1 /boot/efi vfat defaults 0 0" >> /etc/fstab root@centos8:~ # mount /boot/efi
There are changes in our "early-grub.cfg". As you can see, a grub's superuser has been added. You cannot get the GRUB2 command line or change the menu item without authentication when SecureBoot is enabled. This "grub.pbkdf2.sha512.10000.very-long-line" could be achieved with the command grub2-mkpasswd-pbkdf2 .
root@centos8:~ # cat early-grub.cfg set superusers="root" export superusers password_pbkdf2 root grub.pbkdf2.sha512.10000.very-long-line # <- Should be replaced with your line ! set root='lvm/cl-root' # Point to root LV set prefix=($root)/boot/grub2 # Where grub modules resides configfile $prefix/grub.cfg # Switch execute original grub.cfg root@centos8:~ # df /boot /boot/efi Filesystem Size Used Avail Use% Mounted on /dev/mapper/cl-root 17G 1.6G 16G 10% / /dev/vda1 2.0M 0 2.0M 0% /boot/efi root@centos8:~ # mkdir /boot/efi/custom
I've added a lot more modules to the custom image in current case. It is not enough to sign the "grubx64.efi" image, all additional modules must be signed. Instead, I put all the necessary modules in my image to save effort.
root@centos8:~ # grub2-mkimage -c early-grub.cfg -o /boot/efi/custom/grubx64.efi -O x86_64-efi -p /boot/grub2 \ disk lvm diskfilter xfs normal configfile minicmd ls loadenv blscfg linux password_pbkdf2 root@centos8:~ # find /boot/efi /boot/efi /boot/efi/custom /boot/efi/custom/grubx64.efi root@centos8:~ # grub2-mkconfig > /boot/grub2/grub.cfg
We have to put the shim itself in our "custom" directory. The easiest way is to extract it from RPM.
root@centos8:~ # yum install -y yum-utils root@centos8:~ # yumdownloader shim-x64 Last metadata expiration check: 0:36:45 ago on Sun Aug 23 05:40:38 2020. shim-x64-15-15.el8_2.x86_64.rpm 2.5 MB/s | 666 kB 00:00 root@centos8:~ # (cd /tmp && rpm2cpio ~/shim-x64*rpm | cpio -id) 10261 blocks root@centos8:~ # cp -v /tmp/boot/efi/EFI/centos/shimx64.efi /boot/efi/custom/ '/tmp/boot/efi/EFI/centos/shimx64.efi' -> '/boot/efi/custom/shimx64.efi' root@centos8:~ # cp -v /tmp/boot/efi/EFI/centos/mmx64.efi /boot/efi/custom/ '/tmp/boot/efi/EFI/centos/mmx64.efi' -> '/boot/efi/custom/mmx64.efi'
Install the package pesign if not installed yet, then use mokutil to approve hash of our custom grub image. Add a kernel certificate too at the same time:
root@centos8:~ # yum install -y pesign root@centos8:~ # pesign --hash -i /boot/efi/custom/grubx64.efi /boot/efi/custom/grubx64.efi ea838a514ec702c319268d4f090af56ab286172187066d90901d44da4b2cf39b root@centos8:~ # mokutil --import-hash ea838a514ec702c319268d4f090af56ab286172187066d90901d44da4b2cf39b input password: input password again: root@centos8:~ # mokutil --import /usr/share/doc/kernel-keys/$(uname -r)/kernel-signing-ca.cer input password: input password again:
Use the same password for both imports. This is a one time password will be used on next reboot to approve imports.
Since we installed everything manually, the boot into UEFI was not registered, you must do this with the command:
root@centos8:~ # efibootmgr -c -l '/custom/shimx64.efi' -L "custom secure" -d /dev/vda -p 1
Now, poweroff the VM, disable direct kernel boot and turn it on. Use our custom "Prepend vith /boot" grub menu entry to successfully boot OS.
The BIOS has the option to enable the boot ROM built into the expansion card. This is a way to enable booting from a technique completely unknown to the BIOS, such as Fiber Channel or iSCSI. Almost all NIC manufacturers ship their cards with PXE (Pre-eXecute Environment) boot support. PXE is built around the well-known at that time protocols : DHCP (or BOOTP) for providing boot information to the client and TFTP for transferring kernel and initrd binaries. Then other protocols were used to complete the bootstrap, such as HTTP or NFS, but they are not strictly part of PXE.
Setting up a PXE server is described in tons of articles on the Internet, I have at least three articles on my site for different architectures or OS. Almost all of them use the PXELINUX bootloader (syslinux). This time we will look at using GRUB2 as a PXE loader.
Make a cleanup for LAB environment referring to BIOS setup and re-creating first drive. Probably, you will need to remove the "nvram" file, if the test VM was UEFI just before. Then you should connect your VM to PXE network. I cannot describe this in two words, then just skip this step.
This command will create and populate the "grub2" directory in your "/tftpboot" tree:
root@centos8:~ # grub2-mknetdir --net-directory=/tftpboot --subdir=/grub2 -d /usr/lib/grub/i386-pc Netboot directory for i386-pc created. Configure your DHCP server to point to /tftpboot/grub2/i386-pc/core.0
Transfer the resulting /tftpboot/grub2 directory with its content to your TFTP server in PXE.
Then create an entry in your dhcpd.conf:
..
host centos8 {
hardware ethernet 52:54:00:ab:a0:2b;
filename "/grub2/i386-pc/core.0";
}
..
Do not forget restart DHCP service. You can boot your target and see the grub prompt. This is because it cannot find the grub.cfg file. Lets create one:
root@dhcpserver:~ # cat /tftpboot/grub2/grub.cfg set pager=1 # will help in interactive mode insmod biosdisk # Load BIOS info about hard disks insmod lvm # Our root on LVM insmod xfs # CentOS8 default is XFS now set root='lvm/cl-root' # Point to root LV set prefix=($root)/boot/grub2 # Where grub modules resides configfile $prefix/grub.cfg # Switch execute original grub.cfg
Using the GRUB2 boot loader at PXE makes it possible to continue the boot process over a more reliable protocol than TFTP, such as HTTP. You can load the kernel and initrd just by specifying their URL.
UEFI has its own network stack, which does not depends on the NIC EPROM. As a bonus, you can even use PXE over a wireless interface. But this usually implements secure boot. Since we are SecureBoot experts already, this version will be shown here.
It is better to do this example exactly after UEFI Secure boot chapter, because all required files will be in place.
As you know, BIOS and UEFI binaries are not compatible, then we populate TFTP root with UEFI binaries:
root@centos8:~ # grub2-mknetdir --net-directory=/tftpboot --subdir=/grub2 -d /usr/lib/grub/x86_64-efi Netboot directory for x86_64-efi created. Configure your DHCP server to point to /tftpboot/grub2/x86_64-efi/core.efi
Transfer the resulting /tftpboot/grub2/ directory to your PXE server, however ignore last recommendation about "core.efi".
We will use SecureBoot enabled shim and grub from /boot/efi/EFI/centos/
root@dhcpserver:~ # mkdir /tftpboot/secureboot root@dhcpserver:~ # cd /tftpboot/secureboot root@dhcpserver:~ # scp centos8:/boot/efi/EFI/centos/\*.efi . root@dhcpserver:~ # cat grub.cfg set pager=1 insmod lvm insmod xfs echo "In TFTP grub.cfg" sleep 2 set root='lvm/cl-root' set prefix=($root)/boot/grub2 configfile $prefix/grub.cfg
Then create an entry in your dhcpd.conf:
..
host centos8 {
hardware ethernet 52:54:00:ab:a0:2b;
filename "/secureboot/shimx64.efi";
}
..