I plan to take the following steps:
For the needs of SBO, a high-performance computer is not required; almost any hardware will do here. I'm planning to use ZFS, known as memory hungry software, so at least 4G RAM is required. Redundancy requires at least two hard drives. This office is really small, so a computer with 8G RAM and two 220G SSDs is perfectly acceptable.
A minimal Fedora 37 server was installed with a slight modification to the disk layout. Please remember to set the new VG to maximum space and not "auto" size, otherwise you won't be able to create a mirror due to lack of space. I made root 5G in size and ext4 file system. Added 1G swap LV. /var and /var/log and /var/lib ext4 were also added, 1G each. After installation, I got the following result:
root@server:~ # lsblk -e 2,7,11,252 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 223.6G 0 disk ├─sda1 8:1 0 600M 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 222G 0 part ├─rootvg-slash 253:0 0 5G 0 lvm / ├─rootvg-swap 253:1 0 1G 0 lvm [SWAP] ├─rootvg-varlog 253:2 0 1G 0 lvm /var/log ├─rootvg-var 253:3 0 1G 0 lvm /var └─rootvg-varlib 253:4 0 1G 0 lvm /var/lib sdb 8:16 0 223.6G 0 disk root@server:~ # df -l -x tmpfs -x devtmpfs Filesystem Size Used Avail Use% Mounted on /dev/mapper/rootvg-slash 4.9G 946M 3.7G 21% / /dev/sda2 960M 166M 795M 18% /boot /dev/mapper/rootvg-var 974M 212K 907M 1% /var /dev/mapper/rootvg-varlib 974M 25M 883M 3% /var/lib /dev/mapper/rootvg-varlog 974M 18M 889M 2% /var/log /dev/sda1 599M 7.1M 592M 2% /boot/efiBold font indicates the command that you are typing, the rest is the output of the command.
Add some usefull packages:
root@server:~ # dnf install bash-completion rsync rsyslog gdisk
This particular server cannot be switched from EFI to Legacy BIOS (Compatibility Support Mode). GRUB2 itself is implemented as an EFI application and resides on the EFI filesystem. I checked this particular grubx64.efi installed with Fedora 37 for its builtins modules. Found that it can work with mirrored LVM directly. So, let's move the contents of the /boot partition to the LVM volume to mirror it later.
root@server:~ # lvcreate -n boot -L256m rootvg Logical volume "boot" created. root@server:~ # mkfs.ext4 -j -m 0 /dev/rootvg/boot .. root@server:~ # mount /dev/rootvg/boot /mnt root@server:~ # umount /boot/efi root@server:~ # rsync -av /boot/ /mnt/ ..
Update the /etc/fstab to replace /boot be LVM instead of partition:
root@server:~ # umount /mnt /boot root@server:~ # vi /etc/fstab root@server:~ # grep boot /etc/fstab /dev/rootvg/boot /boot ext4 defaults 1 1 UUID=CC14-D6D2 /boot/efi vfat umask=0077,shortname=winnt 0 2 root@server:~ # systemctl daemon-reload root@server:~ # mount -a root@server:~ # df | grep boot /dev/mapper/rootvg-boot 230M 127M 100M 57% /boot /dev/sda1 599M 7.1M 592M 2% /boot/efi
Note that although we copied the data to the LVM volume, the boot process still uses the original partition according to the /boot/efi/EFI/fedora/grub.cfg file located on the EFI partition. Let's change it to use new LV:
root@server:~ # echo 'configfile (lvm/rootvg-boot)/grub2/grub.cfg' > /boot/efi/EFI/fedora/grub.cfg
Then restart the server to check its loading. If the reboot works, but you're still not sure if the boot process is using LVM, you can mount the original partition somewhere and rename its grub.cfg to a different name.
It's time to format the second disk according to the planned scheme
root@server:~ # parted -a optimal /dev/sdb GNU Parted 3.5 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit mib (parted) mkpart primary fat32 1 20 (parted) name 1 "EFI System Partition" (parted) set 1 esp on (parted) mkpart primary 20 100% (parted) set 2 lvm on (parted) print Model: ATA TS240GSSD220S (scsi) Disk /dev/sdb: 228937MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 20.0MiB 19.0MiB fat32 EFI System Partition boot, esp 2 20.0MiB 228936MiB 228916MiB primary lvm (parted) quit Information: You may need to update /etc/fstab.
The first section is for EFI and we will deal with it later. The second partition will be formatted as PV and added to the VG.
root@server:~ # pvcreate /dev/sdb2 Physical volume "/dev/sdb1" successfully created. root@server:~ # vgextend rootvg /dev/sdb2 Volume group "rootvg" successfully extended root@server:~ # pvs PV VG Fmt Attr PSize PFree /dev/sda3 rootvg lvm2 a-- 221.98g 212.73g /dev/sdb2 rootvg lvm2 a-- <223.55g <223.55g
It is important that both volumes have free space to create not only the data, but also the journal. If you missed the recommendation to set a maximum VG size during installation, then you must enlarge the partition, otherwise it will not be possible to create mirrors.
Convert all existing LVs to mirror ones.
root@server:~ # cd /dev/rootvg root@server:/dev/rootvg # ls boot slash swap var varlib varlog root@server:/dev/rootvg # for lv in * ; do lvconvert -m1 -y /dev/rootvg/$lv ; done Logical volume rootvg/boot successfully converted. Logical volume rootvg/slash successfully converted. Logical volume rootvg/swap successfully converted. Logical volume rootvg/var successfully converted. Logical volume rootvg/varlib successfully converted. Logical volume rootvg/varlog successfully converted.
Once the mirrored LVs are created, the existing initrd needs to be rebuilt or the reboot will fail.
root@server:~ # dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
When booting from a broken mirror, the dracut initrd activates only those LVs that are specified on the kernel command line. To fix this behavior, replace CMDLINE in /etc/default/grub to only include "rd.lvm.vg=rootvg". This option will activate the entire VG rootvg during the boot phase. Then re-create the grub.cfg.
root@server:~ # grep CMDLINE /etc/default/grub GRUB_CMDLINE_LINUX="rd.lvm.vg=rootvg" root@server:~ # grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... .. Adding boot menu entry for UEFI Firmware Settings ... done
Reboot the server to make sure it boots normally.
The second EFI partition must be created and copied over from the first. This data is mostly static, and rarely gets updated. Create a FAT32 file system and copy the original EFI content to its backup:
root@server:~ # mkdosfs /dev/sdb1 mkfs.fat 4.2 (2021-01-31) root@server:~ # mkdir /boot/efib root@server:~ # mount -t vfat /dev/sdb1 /boot/efib root@server:~ # rsync -av /boot/efi/ /boot/efib/
Add the copied EFI partition to fstab for mounting. Since we want the system to boot without any of the disks, add the "nofail" option to fstab. Add the same nofail mount option for the sda1 drive.
/dev/sda1 /boot/efi vfat nofail,umask=0077,shortname=winnt 0 2 /dev/sdb1 /boot/efib vfat nofail,umask=0077,shortname=winnt 0 2
Reboot the server to make sure it loads well. Then disable the first drive in the BIOS or unplug it to make sure the OS able boots from the only surviving drive. Then reconnect or turn on the drive back, as we will need work on it in the next chapter.
In this step, we need to destroy everything that exists on the original sda drive and rebuild the mirror. The same procedure will be relevant for rebuilding the mirror in the event of a real disk failure.
Use wipefs command to remove all filesystems and GPT on original /dev/sda.
root@server:~ # wipefs -a -f /dev/sda3 /dev/sda3: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31 root@server:~ # wipefs -a -f /dev/sda2 /dev/sda2: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42 root@server:~ # wipefs -a -f /dev/sda1 /dev/sda1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20 /dev/sda1: 1 byte was erased at offset 0x00000000 (vfat): eb /dev/sda1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa root@server:~ # wipefs -a -f /dev/sda /dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54 /dev/sda: 8 bytes were erased at offset 0x6fc86d5e00 (gpt): 45 46 49 20 50 41 52 54 /dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa root@server:~ # reboot
After the reboot, we should have the same situation as if we had just replaced the failed sda drive. First we need to create a new partition table on disk. We will use the "gdisk" tool to replicate the existing partition table to the target disk. Run gdisk with the source disk, activate the expert command menu, and write to the target disk.
!! This procedure will copy GPT from /dev/sdb -> to /dev/sda !! root@server:~ # gdisk /dev/sdb GPT fdisk (gdisk) version 1.0.9 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): x Expert command (? for help): u Type device filename, or press <Enter> to exit: /dev/sda Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to /dev/sda. The operation has completed successfully. Expert command (? for help): q
Fix VG by removing missing PV and adding a new one for a replacement.
root@server:~ # pvs WARNING: Couldn't find device with uuid N4aGoj-IyCD-jScK-Cj91-wyEs-U6Ed-23VviJ. WARNING: VG rootvg is missing PV N4aGoj-IyCD-jScK-Cj91-wyEs-U6Ed-23VviJ (last written to /dev/sda3). PV VG Fmt Attr PSize PFree /dev/sdb2 rootvg lvm2 a-- <223.55g 214.27g [unknown] rootvg lvm2 a-m 221.98g <212.71g root@server:~ # vgreduce rootvg --removemissing --force .. Wrote out consistent volume group rootvg. root@server:~ # pvs PV VG Fmt Attr PSize PFree /dev/sdb2 rootvg lvm2 a-- <223.55g 214.27g root@server:~ # pvcreate /dev/sda2 Physical volume "/dev/sda2" successfully created. root@server:~ # vgextend rootvg /dev/sda2 Volume group "rootvg" successfully extended root@server:~ # pvs PV VG Fmt Attr PSize PFree /dev/sda2 rootvg lvm2 a-- <223.55g <223.55g /dev/sdb2 rootvg lvm2 a-- <223.55g 214.27g
Then run repair for LVs:
root@server:~ # cd /dev/rootvg root@server:/dev/rootvg # for lv in * ; do lvconvert --repair /dev/rootvg/$lv -y ; done Faulty devices in rootvg/boot successfully replaced. Faulty devices in rootvg/slash successfully replaced. Faulty devices in rootvg/swap successfully replaced. Faulty devices in rootvg/var successfully replaced. Faulty devices in rootvg/varlib successfully replaced. Faulty devices in rootvg/varlog successfully replaced.
Copy EFI partition content manually:
root@server:~ # mkdosfs /dev/sda1 mkfs.fat 4.2 (2021-01-31) root@server:~ # mount /dev/sda1 root@server:~ # df | grep efi /dev/sdb1 19M 7.1M 12M 38% /boot/efib /dev/sda1 19M 0 19M 0% /boot/efi root@server:~ # rsync -av /boot/efib/ /boot/efi/
You can find installation instructions on the OpenZFS website. Briefly, it looks like this.
root@server:~ # dnf install -y https://zfsonlinux.org/fedora/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
root@server:~ # dnf install -y kernel-devel
root@server:~ # dnf install -y zfs
root@server:~ # modprobe zfs
root@server:~ # echo zfs > /etc/modules-load.d/zfs.conf
When upgrading the kernel, you must rebuild the ZFS DKMS module. This is usually done with the command (you not need run this now):
root@server:~ # dkms install -m zfs -v $(rpm -q --queryformat %{VERSION} zfs)
The next step is to create a zpool that should use the redundant LV. So first we need to create such LV.
root@server:~ # lvcreate -m1 -n zpool -L100g rootvg Logical volume "zpool" created.
Then create zpool on it
root@server:~ # zpool create -o ashift=12 -m none z /dev/rootvg/zpool
root@server:~ # zpool status
pool: z
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
z ONLINE 0 0 0
zpool ONLINE 0 0 0
errors: No known data errors
Finally, create filesystem with some flags usefull for SAMBA
root@server:~ # zfs create -o mountpoint=/home/samba -o casesensitivity=mixed z/samba