HOWTO use shared NFS root to boot multiple RedHat (6) oracle servers.

Two trends, consolidation and decentralization, met again at Data Arena. Decentralization won this round. Every Oracle DB will get it's own dedicated VM (consolidation won previous round). However, every server will be as thin as possible, preferrably unmanaged and sharing most resources provuded by NetApp's NFS share.

VM with minimal configuration (1CPU, 2Gram) booting from net will be created. If more resources required, then they can be added on-line due to RH6 hot-plug ready. The server should mount NFS root as read-only, only /etc and /var should be writtable and customized for particular server.

Preparing NFS share

Use already existing RH6 installation with required architecture (mine x64) to populate NFS root. Create NetApp NFS volume and give permissions like:

/vol/oradb_rh6_nfs      -sec=sys,ro,rw=192.168.0.10,root=192.168.0.0/24

Mount it on parent server:

 # df /mnt/net
Filesystem            Size  Used Avail Use% Mounted on
netapp:/vol/oradb_rh6_nfs   10G  128K  9.9G   1% /mnt/net

Now populate /mnt/net with required packages:

# yum --releasever=6Server --installroot=/mnt/net install yum

Create script to easy work within chroot environment, copy yum repository info, resolv.conf, and enter to chrooted environment:

# cat > /mnt/net/command_mount_chroot << EOF
mount -o bind /proc proc
mount -o bind /sys sys
mount -o bind /dev dev
chroot . /bin/bash -i
umount dev
umount sys
umount proc
EOF
# cp -v /etc/yum.repos.d/* /mnt/net/etc/yum.repos.d/
# cp -v /etc/resolv.conf /mnt/net/etc/
# cd /mnt/net
# sh command_mount_chroot

Install everything else you may need for server:

# yum --releasever=6Server install dracut-network at dhclient lsof man man-pages \
nfs-utils ntp openssh-clients openssh-server sendmail sudo tcpdump vixie-cron wget \
rsync compat-db compat-gcc-34 compat-gcc-34-c++ compat-libgcc-296 compat-libstdc++-296 \
cvs elfutils-libelf-devel elfutils-libelf-devel-static expect gcc gcc-c++ ghostscript \
ghostscript-fonts glibc-devel libaio libaio-devel libXp libXtst make openmotif \
poppler-utils sharutils sysstat unixODBC-devel urw-fonts vnc-server which xorg-x11-server-utils \
xorg-x11-twm xorg-x11-utils xterm glibc-devel.x86_64 glibc-devel.i386 unixODBC-devel.i386 pam.i386

Configure required things:

# echo "NETWORKING=yes" > /etc/sysconfig/network
# rm /etc/mtab && ln -s /proc/mounts /etc/mtab
# cat > /etc/fstab << EOF
192.168.0.1:/vol/oradb_rh6_nfs  /    nfs     nolock,nfsvers=3,tcp  0 0
tmpfs                   /tmp         tmpfs   size=1g,mode=1777       0 0
tmpfs                   /dev/shm     tmpfs   defaults        0 0
devpts                  /dev/pts     devpts  gid=5,mode=620  0 0
sysfs                   /sys         sysfs   defaults        0 0
proc                    /proc        proc    defaults        0 0
tmpfs                   /var         tmpfs   size=512m        0 0
EOF

/var have to be read-write, therefore it will be tmpfs populated every boot time. Prepare this:

# (cd / && rsync -av var/ vart)
# find /vart -type f -delete
# rm -rf /vart/lib/yum /vart/cache/yum
# (cd /vart && tar zcvf /var.tar.gz)

Insert following lines into /etc/rc.d/rc.sysinit immediately after "Mounting local filesystems: " section:

--- /etc/rc.d/rc.sysinit        2012-09-06 08:14:18.000000000 +0000
+++ etc/rc.d/rc.sysinit 2013-02-21 13:53:05.665240000 +0000
@@ -502,6 +502,9 @@
        action $"Mounting local filesystems: " mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2 -O no_netdev
 fi
 
+# Populate /var rw tmpfs
+(cd /var && tar zxpf /var.tar.gz )
+
 # Update quotas if necessary
 if [ X"$_RUN_QUOTACHECK" = X1 -a -x /sbin/quotacheck ]; then
        action $"Checking local filesystem quotas: " /sbin/quotacheck -anug

Generate hosts SSH keys from within chroot environment. Edit /etc/ssh/sshd_config to change SSH port to something free. Then just start SSH by /etc/init.d/sshd start and stop it. Return port definitions back.

Create initrd using dracut. Go to /boot and detect which latest kernel installed inspecting its content, then:

# cat  > /etc/dracut.conf << EOF
dracutmodules+="nfs network base"
omit_dracutmodules+="xen btrfs dm dmraid dmsquash-live lvm mdraid multipath"
add_dracutmodules+="network nfs"
add_drivers+="vmxnet3"
mdadmconf="no"
lvmconf="no"
EOF
# dracut nfsrh6.ird 2.6.32-279.22.1.el6.x86_64

Now copy vmlinuz-XXX and nfsrh6.ird to PXE server (setting of PXE server is out of scope of this paper). Here is an example of PXE config file:

# cat /tftpboot/pxelinux.cfg/nfsrh6 
DEFAULT nfsrh6
PROMPT 0
TIMEOUT 0

LABEL nfsrh6
 KERNEL nfsrh6.krl
 APPEND initrd=nfsrh6.ird root=nfs:182.168.0.1:/vol/oradb_rh6_nfs rd.shell rd.ip=auto rd.debug vmalloc=192MB single
 IPAPPEND 2

/tftpboot/pxelinux.cfg/ # ls -l 01*
lrwxrwxrwx 1 root root    6 Feb 21 14:54 01-00-50-56-88-0b-0c -> nfsrh6

Boot your VM ; welcome to localhost !!.

Next idea is to use script that set correct hostname, and different /etc/sysconfig/network-scripts/ifcfg-eth[0-9]* for each needs. Start network script will pick relevant settings according to existing MAC address. This work will be done next time.