This example based on two VMs running on KVM host. A pacemaker/corosync cluster by CentOS 7.3 will manage our resources. DRBD will emulate shared disk. KVM fencing will resolve brain-split.
Following steps should be performed on both nodes:
Decide on names, including cluster node names (here node01 and node02), cluster name (here ha) and VIP name (here hajnk).
Important NOTE: Names have to be DNS complient (for example underscores are unacceptable).
We are disabling for now selinux and firewall. Probably, I'll investigate later how to turn them back, not sure.
# getenforce Disabled
If you see other output, fix /etc/sysconfig/selinux
# cat /etc/sysconfig/selinux SELINUX=disabled
Uninstall firewall:
# for rpm in iptables-services firewalld ; do rpm -e $rpm ; done
Disable IPv6: Edit /etc/default/grub, fix GRUB_CMDLINE_LINUX line, replacing rngb quiet with net.ifnames=0 biosdevname=0 vga=0x314 ipv6.disable=1 rd.shell. Here is a result:
# grep GRUB_CMDLINE_LINUX /etc/default/grub GRUB_CMDLINE_LINUX="rd.lvm.lv=rootvg/root rd.lvm.lv=rootvg/swap vga=0x314 net.ifnames=0 ipv6.disable=1 rd.shell"
After fixing /etc/default/grub, generate new /boot/grub2/grub.cfg using:
# grub2-mkconfig -o /boot/grub2/grub.cfg
Remove RPMs:
# rpm -e $(rpm -qa | grep NetworkManager) biosdevname
It is nice to reboot now, at this time. The network interfaces names becomes named back to ethX and no IPv6 addresses assigned.
Configure IP address in /etc/sysconfig/network-scripts/ifcfg-eth0 if not done already.
All used IPs and network names should be resolvable either by DNS or by /etc/hosts. Even if DNS works well, put everything in /etc/hosts, this will speed up cluster performance.
# echo -e "$(hostname -i)\t$(hostname -f) $(hostname -s)" # >> /etc/hosts
Do not forget to add info about Virtual IP (VIP). Put both short and FQDN formated names. Once /etc/hosts full of data, copy it between both nodes.
Both nodes have to be synchronized by time. Using NTP preferable. Install RPMs:
# yum install -y net-tools ntp ntpdate sysstat eject wget rsync
Fix /etc/ntp.conf with correct NTP-SERVER-NAME-OR-IP (also remove IPv6 related line restrict ::1,otherwice service will fail to start) and start NTP synchronization.
# ntpdate -b NTP-SERVER-NAME-OR-IP # hwclock -w # systemctl enable ntpd.service # systemctl start ntpd.service
After some time ( >5min ), check progress with ntpq -p command
It is preferrable that both nodes will have same host's SSH keys, otherwise switching service between nodes can cause problems for SSH based services (scripts). It is already true for my installation, because both VMs cloned from same template. If you have independent images, please copy /etc/ssh/ssh_host* from one node to another and restart SSH service.
SSH passwordless access for root between nodes should be configured. Probably this is not a MUST requirement anymore, because most of cluster related interconnections done by corosync protocol with it's own keys and signatures. But this is very handfull.
Generate root SSH key on one node:
root@node01:~ # ssh-keygen -t rsa -b 2048 -C "root@ha" .. root@node01:~ # cd .ssh root@node01:~/.ssh # cp id_rsa.pub authorized_keys
Add host keys to known_hosts file:
root@node01:~/.ssh # ssh-keyscan $(hostname -s) | \
sed -e 's/'$(hostname -s)'/'$(hostname -f),$(hostname -s)'/' | \
awk '/'$(hostname -s)'/ {a=$1;gsub("01","02",a);print a","$0;}' > known_hosts
Copy credentials to other node:
root@node01:~/.ssh # cd root@node01:~ # scp -pr .ssh node02:
Install on both nodes:
# yum install -y pcs pacemaker fence-agents-all openssl-devel
Install jenkins.rpm on both nodes as it described at Jenkins Wiki Disable Jenkins from autostarting:
# chkconfig jenkins off # chkconfig --del jenkins
Installing DRBD software involving compiling from sources. This can be done on any one node or even other CentOS same version server with exact same kernel version. The resulting RPMs could be copied to cluster nodes and installed.
Install development packages, required to compile software:
# mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
# yum -y install gcc make automake autoconf libxslt libxslt-devel flex rpm-build kernel-devel
Download and compile latest DRBD tar balls from Linbit:
# cd /tmp /tmp # wget -q -O - http://oss.linbit.com/drbd/8.4/drbd-8.4.8-1.tar.gz | tar zxvfB - /tmp # cd drbd-8.4.8-1 /tmp/drbd-8.4.8-1 # make km-rpm
Download latest DRBD utils tar balls from Linbit:
# cd /tmp /tmp # wget -q -O - http://oss.linbit.com/drbd/drbd-utils-8.9.6.tar.gz | tar zxvfB - /tmp # cd drbd-utils-8.9.6
Patch to drbd.spec.in should be applied, otherwice resulting RPMs will fail to install.
--- drbd.spec.in 2017-03-27 13:59:28.930525786 +0300 +++ drbd.spec.in 2017-03-27 13:59:16.973397350 +0300 @@ -31,6 +31,7 @@ # conditionals may not contain "-" nor "_", hence "bashcompletion" %bcond_without bashcompletion %bcond_without sbinsymlinks +%undefine with_sbinsymlinks # --with xen is ignored on any non-x86 architecture %bcond_without xen %bcond_without 83support
/tmp/drbd-utils-8.9.6 # ./configure --with-pacemaker /tmp/drbd-utils-8.9.6 # make rpm
If everything was OK, the resulting RMPs are here:
# find /root/rpmbuild -type f -name "*.rpm" /root/rpmbuild/RPMS/x86_64/drbd-km-3.10.0_514.6.1.el7.x86_64-8.4.8-1.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-utils-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-udev-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-xen-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-debuginfo-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.9.6-1.el7.centos.x86_64.rpm /root/rpmbuild/RPMS/x86_64/drbd-km-debuginfo-8.4.8-1.x86_64.rpm
A kernel module compiled for current kernel version. Be aware to recompile DRBD module once updating kernel (or, in other words, forget about kernel updates).
Install resulting RPMs, copy to other node and install them too.
# rpm -ihv /root/rpmbuild/RPMS/x86_64/*rpm # scp /root/rpmbuild/RPMS/x86_64/*rpm node02:/tmp # ssh node02 rpm -ihv /tmp/\*rpm
Disable autostart service drbd on both nodes as it be managed by clusterware.
# systemctl disable drbd.service # systemctl start drbd.service
Starting service without defined resources just load drbd kernel module. This is vital for initialization drdb device in next chapter.
Disable LVM write cache (according to DRBD recommendations) on both nodes:
# sed -i -e 's/write_cache_state.*/write_cache_state = 0/' /etc/lvm/lvm.conf
Create LV (on both device) that will be used as DRBD back device
# lvcreate -n jenkins -L10g /dev/rootvg
Create /etc/drbd.d/jenkins.res resource file with similar content:
resource jenkins {
meta-disk internal;
device /dev/drbd0 ;
disk /dev/rootvg/jenkins ;
net { protocol C; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; }
on node01 { address 192.168.122.21:7791 ; }
on node02 { address 192.168.122.22:7791 ; }
}
Copy resource file to second node:
root@node01:~ # rsync -av /etc/drbd.d/jenkins.res root@node02:/etc/drbd.d/jenkins.res
Initialize (on both nodes) new resource:
# /sbin/drbdadm -- --force create-md jenkins
# /sbin/drbdadm up jenkins
# cat /proc/drbd
version: 8.4.8-1 (api:1/proto:86-101)
GIT-hash: 22b4c802192646e433d3f7399d578ec7fecc6272 build by root@node01, 2017-01-20 01:23:12
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404
This status shows that both nodes resources are up (Connected), the data itself is not synchronized (Inconsistent on both nodes) and DRBD does not know select primary (both are Secondary). Let's help it to select primary. Because this disk is still empty, the initial full resync could be skipped:
root@node01:~ # /sbin/drbdadm -- --clear-bitmap new-current-uuid jenkins/0
root@node01:~ # /sbin/drbdadm primary --force jenkins
root@node01:~ # cat /proc/drbd
version: 8.4.8-1 (api:1/proto:86-101)
GIT-hash: 22b4c802192646e433d3f7399d578ec7fecc6272 build by root@node01, 2017-01-20 01:23:12
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:298436 nr:0 dw:298436 dr:1404 al:69 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
This is normal working state of DRBD. Both nodes are up (Connected), the data is UpToDate on both nodes and one of them Primary. The resource will be managed by cluster software, therefore we should make resource Secondary on both nodes for now:
root@node01:~ # while ! /sbin/drbdadm secondary jenkins ; do sleep 2 ; done
root@node01:~ # cat /proc/drbd
version: 8.4.8-1 (api:1/proto:86-101)
GIT-hash: 22b4c802192646e433d3f7399d578ec7fecc6272 build by root@node01, 2017-01-20 01:23:12
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:298436 nr:0 dw:298436 dr:1404 al:69 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Set complex password for user hacluster (on both nodes), make it non-expired. Start service (on both nodes):
# systemctl enable pcsd.service # systemctl start pcsd.service
Authorise service (on one node):
root@node01:~ # pcs cluster auth node01 node02 Username: hacluster Password: node02: Authorized node01: Authorized
If you want to make cluster formation automatically via script or hate having unnecessary passwords in system, you can do pcsd authorization in alternative way.
Start and stop pcsd on one node, then inspect content of /var/lib/pcsd directory. There should be certificate files in it. Create pcs_users.conf file with similar content:
[
{
"creation_date": "Wed Feb 08 16:40:54 +0200 2017",
"username": "hacluster",
"token": "00000000-0000-0000-0000-000000000000"
}
]
The token is hexadecimal string, probably can be any string, not checked. In this example every symbol replaced to zero to force you generate your own token instead of copy-paste it.
Then create file tokens with content:
{
"format_version": 2,
"data_version": 2,
"tokens": {
"node01": "00000000-0000-0000-0000-000000000000",
"node02": "00000000-0000-0000-0000-000000000000"
}
}
Replace zeroes with token you had generated in previous file. Use same token for both nodes. If your cluster will have more than two node, add all of them here.
Copy whole /var/lib/pcsd directory to other node, then enable and start pcsd on every node.
# systemctl enable pcsd.service # systemctl start pcsd.service
root@node01:~ # pcs cluster setup --start --enable --name ha node01 node02 --transport udpu Destroying cluster on nodes: node01, node02... node01: Stopping Cluster (pacemaker)... node02: Stopping Cluster (pacemaker)... node02: Successfully destroyed cluster node01: Successfully destroyed cluster Sending cluster config files to the nodes... node01: Succeeded node02: Succeeded Starting cluster on nodes: node01, node02... node01: Starting Cluster... node02: Starting Cluster... node01: Cluster Enabled node02: Cluster Enabled Synchronizing pcsd certificates on nodes node01, node02... node02: Success node01: Success Restarting pcsd on the nodes in order to reload the certificates... node02: Success node01: Success
Check resulting status :
root@node01:~ # pcs cluster status Cluster Status: Stack: corosync Current DC: node02 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Sat Jan 21 12:14:41 2017 Last change: Sat Jan 21 12:14:28 2017 by hacluster via crmd on node02 2 nodes and 0 resources configured PCSD Status: node01: Online node02: Online
Ensure fencing/STONITH disabled for configuration time:
root@node01:~ # pcs property set stonith-enabled=false root@node01:~ # pcs property list Cluster Properties: cluster-infrastructure: corosync cluster-name: ha dc-version: 1.1.15-11.el7_3.2-e174ec8 have-watchdog: false stonith-enabled: false
Cluster configuration done from one of nodes using pcs command. The changes propagated via corosync and pcsd services to other nodes. All nodes chould be avaliable to changes be successed.
A shared disk (DRBD) will be initialized as LVM device, three Logical Volumes (LV) will be created and mounted where jenkins want them. A Virtual IP (VIP) and jenkins service will finish set of resources.
First we add DRBD resource becasue the rest depends on it:
root@node01:~ # pcs resource list drbd ocf:linbit:drbd - Manages a DRBD device as a Master/Slave resource systemd:drbd root@node01:~ # pcs resource describe ocf:linbit:drbd
No much parameters exist. A master/slave set should be defined too:
root@node01:~ # pcs resource create jenkins-drbd ocf:linbit:drbd drbd_resource=jenkins
root@node01:~ # pcs resource master master-jenkins-drbd jenkins-drbd master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
root@node01:~ # pcs status
Cluster name: ha
Stack: corosync
Current DC: node01 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Sat Jan 21 18:54:30 2017 Last change: Sat Jan 21 18:54:25 2017 by root via cibadmin on node01
2 nodes and 2 resources configured
Online: [ node01 node02 ]
Full list of resources:
Master/Slave Set: master-jenkins-drbd [jenkins-drbd]
jenkins-drbd (ocf::linbit:drbd): FAILED node02 (blocked)
jenkins-drbd (ocf::linbit:drbd): FAILED node01 (blocked)
Failed Actions:
* jenkins-drbd_stop_0 on node02 'not configured' (6): call=6, status=complete, exitreason='none',
last-rc-change='Sat Jan 21 18:52:11 2017', queued=0ms, exec=45ms
* jenkins-drbd_stop_0 on node01 'not configured' (6): call=6, status=complete, exitreason='none',
last-rc-change='Sat Jan 21 18:52:12 2017', queued=0ms, exec=43ms
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
It is recommended to define both base resource and it's master/slave definitions simultaniously. It is done by putting all definitions in one file then apply it. I'd configured on live system, therefore resource marked "FAILED". No problem with it:
root@node01:~ # pcs resource cleanup jenkins-drbd
Cleaning up jenkins-drbd:0 on node01, removing fail-count-jenkins-drbd
Cleaning up jenkins-drbd:0 on node02, removing fail-count-jenkins-drbd
Waiting for 2 replies from the CRMd.. OK
root@node01:~ # pcs status
Cluster name: ha
Stack: corosync
Current DC: node01 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Sat Jan 21 18:55:20 2017 Last change: Sat Jan 21 18:55:13 2017 by hacluster via crmd on node01
2 nodes and 2 resources configured
Online: [ node01 node02 ]
Full list of resources:
Master/Slave Set: master-jenkins-drbd [jenkins-drbd]
Masters: [ node02 ]
Slaves: [ node01 ]
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
According to status, master (primary) DRBD run on node02. You can verify this by cat /proc/drbd
A shared disk here is /dev/drbd/by-res/jenkins/0 let's initialize it (do this on node where primary DRBD run, node02 as following from previous output):
root@node02:~ # pvcreate --dataalignment=4k /dev/drbd/by-res/jenkins/0 root@node02:~ # vgcreate jenkinsvg /dev/drbd/by-res/jenkins/0 root@node02:~ # for lv in cache log lib ; do lvcreate -L2g -n $lv jenkinsvg mkfs.ext3 -j -m0 /dev/jenkinsvg/$lv done root@node02:~ # vgchange -an jenkinsvg root@node02:~ # pvs PV VG Fmt Attr PSize PFree /dev/drbd0 jenkinsvg lvm2 a-- 10.00g 4.00g /dev/vda2 rootvg lvm2 a-- 19.75g 3.75g
Thus, disk was initialized by LVM and contains Volume Group (VG) named "jenkinsvg". Lets create cluster resource for it.
root@node02:~ # pcs resource describe ocf:heartbeat:LVM
According to description, /etc/lvm/lvm.conf should be fixed, again. The following lines should be changed:
use_lvmetad = 0 volume_list = [ "rootvg" ]
A "rootvg" is name of my current rootvg. It should be always available. Copy lvm.conf to another node and repeat the following commands on both nodes. A new initrd should be produced to reflect these changes:
# systemctl disable lvm2-lvmetad.service # mkinitrd -f /boot/initramfs-$(uname -r).img $(uname -r) && reboot
Once cluster nodes comes back, add resource and check /var/log/messages for successful resource activation.
# pcs resource create jenkins-vg ocf:heartbeat:LVM volgrpname=jenkinsvg exclusive=true
You will see new resource failed on other node (where DRBD slave), but successed on master node.
# pcs resource describe ocf:heartbeat:Filesystem # pcs resource create jenkins-fs-cache ocf:heartbeat:Filesystem device=/dev/jenkinsvg/cache directory=/var/cache/jenkins fstype=ext3 # pcs resource create jenkins-fs-lib ocf:heartbeat:Filesystem device=/dev/jenkinsvg/lib directory=/var/lib/jenkins fstype=ext3 # pcs resource create jenkins-fs-log ocf:heartbeat:Filesystem device=/dev/jenkinsvg/log directory=/var/log/jenkins fstype=ext3
Observe all file systems mounted on one node. Fix permissions on this node for future use:
# chown jenkins:jenkins /var/cache/jenkins /var/lib/jenkins /var/log/jenkins # chmod o-rx /var/cache/jenkins /var/log/jenkins
# pcs resource list ipaddr # pcs resource describe ocf:heartbeat:IPaddr # pcs resource create jenkins-ip ocf:heartbeat:IPaddr ip=192.168.122.20
# pcs resource list lsb # pcs resource create jenkins-service lsb:jenkins
Groupping resources add constraints to run this group of resources on same node and also define the sequence of start and stop procedure. Therefore, the order of resources is important here. First we activate VG, then mount FS, then start service. A VIP have to be before service.
# pcs resource group add jenkins jenkins-vg jenkins-fs-cache jenkins-fs-lib jenkins-fs-log jenkins-ip jenkins-service
All these should happen only where master DRBD copy run, then add the following constraint:
# pcs constraint colocation add jenkins master-jenkins-drbd INFINITY with-rsc-role=Master # pcs constraint order promote master-jenkins-drbd then start jenkins
Clean error status from previous run and see the result:
# pcs resource cleanup --node node01
# pcs resource cleanup --node node02
# pcs status
Cluster name: ha
Stack: corosync
Current DC: node02 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Sat Jan 21 20:15:19 2017 Last change: Sat Jan 21 20:13:39 2017 by root via cibadmin on node02
2 nodes and 8 resources configured
Online: [ node01 node02 ]
Full list of resources:
Master/Slave Set: master-jenkins-drbd [jenkins-drbd]
Masters: [ node02 ]
Slaves: [ node01 ]
Resource Group: jenkins
jenkins-vg (ocf::heartbeat:LVM): Started node02
jenkins-fs-cache (ocf::heartbeat:Filesystem): Started node02
jenkins-fs-lib (ocf::heartbeat:Filesystem): Started node02
jenkins-fs-log (ocf::heartbeat:Filesystem): Started node02
jenkins-ip (ocf::heartbeat:IPaddr): Started node02
jenkins-service (lsb:jenkins): Started node02
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
There is no reason to test cluster, espetially with DRBD without fencing. These two nodes run on KVM host, therefore fence-virt will be used. Here is a very good guide to make fencing for our POC environment.