RHEL 5.9 HA-LVM cluster install and setup guide
version 201600508 created by Seamus Murray
Prerequisites 2 machines either physical or virtual
RHEL OS packages RHEL Cluster packages
these can be sourced from Note write guides for each of these methods * RedHat directly RHN * Local YUM Repository * Local Satellite/Spacewalk server * Local media cdrom/dvdrom/usb drive/iso image etc.
In this guide I will be using the ISO of the RHEL 5.9 install This ISO was downloaded from RHN and contains the following Repositories
- Server
- Cluster
- ClusterStorage
- VT
for the initial OS install we are just going to use the Server repo
Boot from the DVD/ISO Step though the installer note the package choice is dependant on what you want to do with the servers later Eg if you want to run a GUI and web browser you will have to install those Groups personally i always try to install the minimum number of packages to get the specific job done.
@base
@core
@editors
@text-internet
Once you install has completed and you have booted into your new OS.
setup the hosts to connect to a
RHEL OS packages RHEL Cluster packages
these can be sourced from * RedHat directy RHN * Local YUM Repository * Local Satellite/Spacewalk server * Local media cdrom/dvdrom/usb drive/iso image etc.
Install the required cluster utils for this example it is just * cman * rgamanger * openais
Note: for a HA-LVM cluster these is no need for any distributed lockmanager GFS etc... if these are installed its best to remove them now
Disable the cluster from starting up... until you have finished the config
chkconfig --levels 2345 cman off
chkconfig --levels 2345 rgmanager off
chkconfig --levels 2345 ricci off
chkconfig --levels 2345 luci off
Before we setup the cluster.. * ensure your hostnames IP addresses and interfaces are in order. __in this exaple each node has 2 physical interfaces 1. eth0 the public side, this is where the service that will be clusterd will run from https,nfs,ftp etc 2. eth1 the private interface that is just used for cluster communication broadcast,multicast or unicast * ensure the host name is correct and a FQDN in /etc/sysconfig/network * ensure hostname is not in any of the ifcfg-ethx files * ensure hostname is not listed against loopback address in /etc/hosts #RedHat installer put this in by default * restart network services, run hostname command manually or reboot if necessary* * ensure the hostnames for all IPs on all the cluster nodes are listed in the /etc/hosts file
sample /etc/hosts file
192.168.56.101 rhel59halvmn1.local rhel59halvmn1
10.10.100.101 rhel59halvmn1.private rhel59halvmn1-priv
192.168.56.102 rhel59halvmn2.local rhel59halvmn2
10.10.100.102 rhel59halvmn2.private rhel59halvmn2-priv
192.168.56.103 rhel59halvm-vip.local rhel59halvm-vip
Setup ssh keys between the nodes to make things easier
# ssh-copy-id -i /root/.ssh/id_rsa.pub rhel59halvmn2
Ensure the time on all cluster nodes are in sync
you may need to manually sync the time
alias REMOTETIME='ssh rhel59halvmn1 -C date'
date -s "`REMOTETIME`"
service ntp stop
ntpdate 192.168.1.1
ntpdate 192.168.1.1
service ntp start
Configure iptables to allow the cluster traffic through##
iptables -I INPUT -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 16851 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 21064 -j ACCEPT
Create a /etc/cluster/cluster.conf file
__note personaly I start with an existing complete cluste.conf file, edit it to match the desired config then start the cluster __if you want a better understanding of the config follow the steps below
<?xml version="1.0"?>
<cluster name="rhel59halvm" config_version="1">
<clusternodes>
<clusternode name="rhel59halvmn1.private" nodeid="1" />
<clusternode name="rhel59halvmn2.private" nodeid="2" />
</clusternodes>
<fencedevices />
<rm>
<failoverdomains />
<resources />
</rm>
</cluster>
Note: this is a very simple cluster.conf file there is no fencing resources services etc..
Start the cluster to see if everything works
[root@rhel59halvmn1 cluster]# service cman start
Starting cluster:
Loading modules... done
Mounting configfs... done
Starting ccsd... done
Starting cman... done
Starting daemons... done
Starting fencing... done
Tuning DLM... done
[ OK ]
[root@rhel59halvmn1 cluster]# clustat
Cluster Status for rhel59halvm @ Wed Jun 26 17:59:12 2013
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
rhel59halvmn1.private 1 Online, Local
rhel59halvmn2.private 2 Online
Add your fencing device
Note: because we didnt specify a fence method with the original cluster/conf version 1 we will now have to add the fence device and a fence method for each node. Personally I'd rather do it by hand using a text editor but... in this example im using the fence_manual device note this is not a real fence device do not usse this in production see man fence_manual for more info__ If you want you can use the ccs_tool to delete the individual node's config then re-add the node's config with the fence method
ccs_tool addfence manual fake-parameter
Note: because the fence_manual is a fake agent it doesnt require any parameters but the ccs_tool requires atleast 1 ccs_tool addnode rhel59halvmn1 -n1 -f manual ccs_tool delnode rhel59halvmn2.private ccs_tool addnode rhel59halvmn2 -n2 -f manual cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="rhel59halvm" config_version="7" name="rhel59halvm">
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="rhel59halvmn1" votes="1" nodeid="1">
<fence>
<method name="single">
<device name="manual"/>
</method>
</fence>
</clusternode>
<clusternode name="rhel59halvmn2" votes="1" nodeid="2">
<fence>
<method name="single">
<device name="manual"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="manual" agent="fake-parameter"/>
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
sync the cluster.conf file using sthe ccs_tool
ccs_tool update /etc/cluster/cluster.conf
Configure the private/heartbeat nic
*add multicast route for private/heartbeat
/etc/sysconfig/network-scripts/route-eth1
239.0.0.0/4 dev eth1
Ensure the that the RedHat certificate is installed # not installed by default
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
if not you may get errors like...
Public key for mkinitrd-5.1.19.6-79.el5.i386.rpm is not installed
Check if the repo access is working
yum clean all
yum list
Edit firewall rules to allow cluster traffic
#/opt/firewall-rules
iptables -I INPUT --protocol tcp --dport 22 -j ACCEPT
# Cluster
iptables -I INPUT -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 11111 -j ACCEPT
iptables -I INPUT -p udp -m state --state NEW -m multiport --dports 50007 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 50008 -j ACCEPT
# turn off firewall for private nic
iptables -I INPUT -i eth1 -j ACCEPT
if you want to use luci warning due to the way the RedHat Satellite server is configured you cannot perform rpm or yum group installs you have to specify the packages individually, because of this you can not use ricci/luci to create a cluster config You can use ricci/luci to import an existing cluster Therefore if you really need the GUI create a simple cluster with no resources or services first then import it into lucci
Starting ricci/luci
if you have not previously configured a luci account run " /usr/sbin/luci_admin init" and set a password start up ricci on both servers and luci on one server setup putty forwarding from the rdp jump host local port 80 to 127.0.0.1:8084 on the host you will run luci
start browser window https://127.0.0.1:80
Simple cluster.conf file to get you started
<?xml version="1.0"?>
<cluster alias="cluster1" config_version="1" name="cluster1">
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="hostname1.private" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="ipmi-hostname1"/>
</method>
</fence>
</clusternode>
<clusternode name="hostname2.private" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="ipmi-hostname2"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1">
<multicast addr="239.192.0.1"/>
</cman>
<fencedevices>
<fencedevice agent="fence_ipmilan" auth="password" ipaddr="192.168.1.1" login="test-user" name="ipmi-hostname1" passwd="test-password" delay="30"/>
<fencedevice agent="fence_ipmilan" auth="password" ipaddr="192.168.1.2" login="test-user" name="ipmi-hostname2" passwd="test-password"/>=
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="FailDomain" ordered="1" restricted="1">
<failoverdomainnode name="hostname1.private" priority="1"/>
<failoverdomainnode name="hostname2.private" priority="2"/>
</failoverdomain>
</failoverdomains>
<resources>
</resources>
<service autostart="1" domain="FailDomain" exclusive="0" name="servicel">
</service>
</rm>
</cluster>
scp cluster.conf hostname1:/etc/cluster/cluster.conf
ccs_tool update cluster.conf
Configure fencing
- Setup ipmi profile on the hardware
-
Test to see if fencing works
echo -e "ipaddr=192.168.1.1 \nlogin=
\npasswd= \naction=status" | fence_ipmilan -
Add the fencing details to the cluster.conf file
service rgmanager start
service cman start
clustat cman_tool status tail -f /var/log/messages
-
manual fence override fence_ack_manual -e -n hostname1.private
- manaul service relocation clusvcadm -r ServiceName
- ifconfig will not display a VIP you have to run
ip address show
you may want to disable the acpi daemon otherwise your server may not switch off fast enough
chkconfig --level 234 5 acpid off
chkconfig --levels 2345 cman off
chkconfig --levels 2345 rgmanager off
chkconfig --levels 2345 ricci off
chkconfig --levels 2345 luci off
setup HA-LVM
...refer to section F.2. Configuring HA-LVM Failover with Tagging from RHEL 5 Cluster admin guide
edit /etc/lvm/lvm.conf
-
Ensure that the locking_type parameter in the global section of the /etc/lvm/lvm.conf file is set to the value '1'.
-
Edit the volume_list field in the /etc/lvm/lvm.conf file. Include the name of your root volume group and your hostname as listed in the /etc/cluster/cluster.conf file preceded by @. The hostname to include here is the machine on which you are editing the lvm.conf file, not any remote hostname. Note that this string MUST match the node name given in the cluster.conf file. Below is a sample entry from the /etc/lvm/lvm.conf file:
volume_list = [ "vgsys", "@rhel59halvm1.private" ]
Create the PV VG LV and filesystems on one of the nodes
pvcreate /dev/mapper/mpath1
vgcreate vg_shared_pc /dev/mapper/mpath1
lvcreate -l 100%FREE -n lv_shared_pc vg_shared_pc
mkfs.ext3 /dev/vg_shared_pc/lv_shared_pc
mkdir /mnt/pc