Configuring a two node serviceguard Cluster (part 2)

In the first part of this tutorial I showed the basic serviceguard cluster configuration. Now I am going to show how to configure package inside the cluster. A package is an application and all the application’s resources required to execute properly.

Resources for packages may include:

– Disk Resources ( Volume Group(s), Disk Group(s) )

– Network Resources ( IP Address(es) )

Packaging Files include:

– Package Configuration File ( pkg.conf )

– Package Control Script ( pkg.cntl )

As I said earlier I will configure Oracle Database server as a package of service guard. The database instance will run on the first node only. In the event of a failure it will run on the second node.

At first we need to create file system on the shared storage. Oracle will use these shared file system for storing data files as well as binary. To create file system we need to create physical volume, volume group and logical volume. Here is the commands:

Physical Volume (PV) Creation:

# pvcreate -f /dev/rdsk/c4t0d1 Physical volume "/dev/rdsk/c4t0d1" has been successfully created. # pvcreate -f /dev/rdsk/c4t0d2 Physical volume "/dev/rdsk/c4t0d2" has been successfully created.

VG Creation:

Check the device number of the existing VGs.

# ls -l /dev/*/group crw-r----- 1 root sys 64 0x000000 Oct 4 11:27 /dev/vg00/group crw-r--r-- 1 root sys 64 0x010000 Oct 4 15:16 /dev/vglock/group

Create group file for new volume groups:

# mkdir /dev/vgappl # mknod /dev/vgappl/group c 64 0x020000 # mkdir /dev/vgoradata # mknod /dev/vgoradata/group c 64 0x030000

Create the required volume groups:

# vgcreate -s 32 vgappl /dev/dsk/c4t0d1 Increased the number of physical extents per physical volume to 6399. Volume group "/dev/vgappl" has been successfully created. Volume Group configuration for /dev/vgappl has been saved in /etc/lvmconf/vgappl.conf # vgcreate -s 32 vgoradata /dev/dsk/c4t0d2 Increased the number of physical extents per physical volume to 6399. Volume group "/dev/vgoradata" has been successfully created. Volume Group configuration for /dev/vgoradata has been saved in /etc/lvmconf/vgoradata.conf

Create the logical volumes (LV):

# lvcreate -L 204000 -n lvappl /dev/vgappl Logical volume "/dev/vgappl/lvappl" has been successfully created with character device "/dev/vgappl/rlvappl". Logical volume "/dev/vgappl/lvappl" has been successfully extended. Volume Group configuration for /dev/vgappl has been saved in /etc/lvmconf/vgappl.conf # lvcreate -L 204000 -n lvoradata /dev/vgoradata Logical volume "/dev/vgoradata/lvoradata" has been successfully created with character device "/dev/vgoradata/rlvoradata". Logical volume "/dev/vgoradata/lvoradata" has been successfully extended. Volume Group configuration for /dev/vgoradata has been saved in /etc/lvmconf/vgoradata.conf

Create file system / mount point:

# newfs -F vxfs -o largefiles /dev/vgappl/rlvappl version 5 layout 208896000 sectors, 208896000 blocks of size 1024, log size 16384 blocks unlimited inodes, largefiles supported 208896000 data blocks, 208826960 free data blocks 6375 allocation units of 32768 blocks, 32768 data blocks # newfs -F vxfs -o largefiles /dev/vgoradata/rlvoradata version 5 layout 208896000 sectors, 208896000 blocks of size 1024, log size 16384 blocks unlimited inodes, largefiles supported 208896000 data blocks, 208826960 free data blocks 6375 allocation units of 32768 blocks, 32768 data blocks

Mount the file systems:

mount /dev/vgappl/lvappl /appl mount /dev/vgoradata/lvoradata /oradata

Up to this stage, two file systems /appl and /oradata are created. Now we will install oracle database on these mount points. /appl will contain the oracle binary and /oradata will be used for storing the data files. I will not show the oracle installation. Just keep one thing in mind: The mount points /appl and /oradata are shared mount points. So, for anything that resides on these two mount points, you don’t need to copy it to the second node. But for any other files (i.e /etc/oratab, /var/opt/oracle/oratab etc) you need to manually send the files to the second node at the same location. Another important thing is, user id and group id of oracle user and oinstall group must same across all cluster node. Actually its applicable for any user, not only oracle.

After installing oracle, log on as oracle user, set the environment variables (ORACLE_HOME, ORACLE_SID etc) and start the database and listener to check that the installation is good. If everything is fine, then stop the database and listener.

Now you will,
1. unmount the file system on 1st node
2. deactivate the VGs and export the VGs from 1st node to a map file
3. send the map file to 2nd node
4. import the VGs on 2nd node from the map file
5. activate the VG and mount the file systems on 2nd node.
6. start the database and listener
7. If everything is OK then stop database, unmount file systems and deactivate VG on 2nd node.

Here is the commands:
On first node:

# umount /appl # umount /oradata # vgchange -a n vgappl Volume group "vgappl" has been successfully changed. # vgchange -a n vgoradata Volume group "vgoradata" has been successfully changed. # vgexport -p -v -s -m /etc/lvmconf/vgappl.map vgappl Beginning the export process on Volume Group "vgappl". /dev/dsk/c4t0d1 # vgexport -p -v -s -m /etc/lvmconf/vgoradata.map vgoradata Beginning the export process on Volume Group "vgoradata". /dev/dsk/c4t0d2 # rcp /etc/lvmconf/vgappl.map node02:/etc/lvmconf/ # rcp /etc/lvmconf/vgoradata.map node02:/etc/lvmconf/

On 2nd node:

# mkdir /dev/vgappl # mkdir /dev/vgoradata # mknod /dev/vgappl/group c 64 0x020000 # mknod /dev/vgoradata/group c 64 0x030000 # ls -l /dev/*/group crw-r----- 1 root sys 64 0x000000 Oct 4 13:00 /dev/vg00/group crw-r--r-- 1 root sys 64 0x020000 Oct 5 10:05 /dev/vgappl/group crw-r--r-- 1 root sys 64 0x010000 Oct 4 15:22 /dev/vglock/group crw-r--r-- 1 root sys 64 0x030000 Oct 5 10:05 /dev/vgoradata/group # vgimport -v -s -m /etc/lvmconf/vgappl.map vgappl Beginning the import process on Volume Group "vgappl". Logical volume "/dev/vgappl/lvappl" has been successfully created with lv number 1. Volume group "/dev/vgappl" has been successfully created. # vgimport -v -s -m /etc/lvmconf/vgoradata.map vgoradata Beginning the import process on Volume Group "vgoradata". Logical volume "/dev/vgoradata/lvoradata" has been successfully created with lv number 1. Volume group "/dev/vgoradata" has been successfully created. # mkdir /appl # mkdir /oradata # mount /dev/vgappl/lvappl /appl # mount /dev/vgoradata/lvoradata /oradata

Now start the database, listener and check. After successful checking shutdown db, unmount and deactivate the VG on the 2nd node.

The next step is package configuration. We will do it from 1st node.

Configuring package on servicegurad is a 3 step procedure:

1. Build the Package Configuration File

2. Build the Package Run/Halt Script

3. Compile and Distribute the Binary File

Step 1: Build the Package Configuration File

The first step in building the package configuration file is to run the cmmakepkg command to create the package configuration template:

# mkdir /etc/cmcluster/orapkg # cd /etc/cmcluster/orapkg # cmmakepkg -v –p orapkg.conf Begin generating package template... Done. Package template is created. This file must be edited before it can be used.

The next step is to edit / customize the file with the vi editor:
After editing the file will look something like below:

# cat /etc/cmcluster/orapkg/orapkg.conf PACKAGE_NAME orapkg PACKAGE_TYPE FAILOVER FAILOVER_POLICY CONFIGURED_NODE FAILBACK_POLICY MANUAL NODE_NAME node01 NODE_NAME node02 AUTO_RUN YES LOCAL_LAN_FAILOVER_ALLOWED YES NODE_FAIL_FAST_ENABLED NO RUN_SCRIPT /etc/cmcluster/orapkg/orapkg.ctrl RUN_SCRIPT_TIMEOUT NO_TIMEOUT HALT_SCRIPT /etc/cmcluster/orapkg/orapkg.ctrl HALT_SCRIPT_TIMEOUT NO_TIMEOUT SERVICE_NAME orapkg_db1 SERVICE_FAIL_FAST_ENABLED NO SERVICE_HALT_TIMEOUT 300 SUBNET 10.10.96.0

Here the value of RUN_SCRIPT and HALT_SCRIPT option is /etc/cmcluster/orapkg/orapkg.ctrl. Actually this is the script that will start and stop the application. In our case it will start/stop the database and listener. We will generate and modify this script on next step.

Step 2: Build the Package Run/Halt Script
The procedure for building the package control script is to create a template with the command:

# cd /etc/cmcluster/orapkg # cmmakepkg -v -s orapkg.ctrl Begin generating package control script... Done. Package control script is created. This file must be edited before it can be used.

Before going to the next step, copy all files from /opt/cmcluster/toolkit/oracle/ to /etc/cmcluster/orapkg/ directory. The mentioned directory contains HP supplied sample start-up/monitoring script for oracle. HP also supplies sample script for other application including apache, samba, NFS, tomcat etc.

#cp /opt/cmcluster/toolkit/oracle/* .

The next step is to edit/customize the script with the vi editor. This file is self explanatory. You need to specify the VG, LV, mount point and mount options for all package file systems (/appl and /oradata in this case). You also need to specify package IP address (ip address on which application will connect), SERVICE_NAME (a name for your service: any relevant name will suffice) and SERVICE_CMD (The full pathname of the command used to start the service process). After editing, the modified portion of the file will look like the following. Leave the rest of the file as it was.

VG[0]=vgappl VG[1]=vgoradata LV[0]=/dev/vgappl/lvappl; FS[0]=/appl; FS_MOUNT_OPT[0]="-o rw" LV[1]=/dev/vgoradata/lvoradata; FS[1]=/oradata; FS_MOUNT_OPT[1]="-o rw" IP[0]=192.168.100.50 function customer_defined_run_cmds { /etc/cmcluster/orapkg/toolkit.sh start test_return 51 } function customer_defined_halt_cmds { /etc/cmcluster/orapkg/toolkit.sh shutdown test_return 52 }

Now edit the /etc/cmcluster/orapkg/haoracle.conf file and set ORACLE_HOME and SID_NAME. This file will look like following:

ORACLE_HOME=/appl/oracle/product/10.2.0 ORACLE_ADMIN=oracle SID_NAME=orcl LISTENER=yes LISTENER_NAME=LISTENER LISTENER_PASS= USE_SPFILE=yes set -A MONITOR_PROCESSES MONITOR_PROCESSES[0]=ora_pmon_${SID_NAME} MONITOR_PROCESSES[1]=ora_dbw0_${SID_NAME} MONITOR_PROCESSES[2]=ora_ckpt_${SID_NAME} MONITOR_PROCESSES[3]=ora_smon_${SID_NAME} MONITOR_PROCESSES[4]=ora_lgwr_${SID_NAME} MONITOR_PROCESSES[5]=ora_reco_${SID_NAME} MAINTENANCE_FLAG=yes MONITOR_INTERVAL=30 TIME_OUT=30

You have completed package configuration. Before going to next step edit the cluster configuration file /etc/cmcluster/crmcluster.ascii and add the following lines:

VOLUME_GROUP /dev/vgappl VOLUME_GROUP /dev/vgoradata

Transfer the /etc/cmcluster/orapkg to the 2nd node.
Step 3. Compile and Distribute the Binary File

Here is the last configuration step: creating and distributing the package binary file. Execute the following commands one after another:

# cd /etc/cmcluster # cmcheckconf -v -P orapkg/orapkg.conf -C crmcluster.ascii Checking cluster file: crmcluster.ascii Note : a NODE_TIMEOUT value of 2000000 was found in line 127. For a significant portion of installations, a higher setting is more appropriate. Refer to the comments in the cluster configuration ascii file or Serviceguard manual for more information on this parameter. Checking nodes ... Done Checking existing configuration ... Done Volume group /dev/vgappl is configured differently on node node01 than on node node02 Volume group /dev/vgappl is configured differently on node node02 than on node node01 Volume group /dev/vgoradata is configured differently on node node01 than on node node02 Volume group /dev/vgoradata is configured differently on node node02 than on node node01 Volume group /dev/vglock is configured differently on node node01 than on node node02 Volume group /dev/vglock is configured differently on node node02 than on node node01 Parsing package file: orapkg/orapkg.conf. Checking for inconsistencies Modifying configuration on node node01 Modifying configuration on node node02 cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration.

The above command will check correctness and consistency of configuration file. If it completes successfully then run the following command:

# cd /etc/cmcluster # cmapplyconf -v -P orapkg/orapkg.conf -C crmcluster.ascii Checking cluster file: crmcluster.ascii Note : a NODE_TIMEOUT value of 2000000 was found in line 127. For a significant portion of installations, a higher setting is more appropriate. Refer to the comments in the cluster configuration ascii file or Serviceguard manual for more information on this parameter. Checking nodes ... Done Checking existing configuration ... Done Volume group /dev/vgappl is configured differently on node node01 than on node node02 Volume group /dev/vgappl is configured differently on node node02 than on node node01 Volume group /dev/vgoradata is configured differently on node node01 than on node node02 Volume group /dev/vgoradata is configured differently on node node02 than on node node01 Volume group /dev/vglock is configured differently on node node01 than on node node02 Volume group /dev/vglock is configured differently on node node02 than on node node01 Parsing package file: orapkg/orapkg.conf. Checking for inconsistencies Modifying configuration on node node01 Modifying configuration on node node02 Modify the cluster configuration ([y]/n)? y Marking/unmarking volume groups for use in the cluster Completed the cluster creation

Package is compiled successfully. You have the start the cluster using cmruncl command and then you can use cmrunpkg <package_name>command to start the package.

Here is some useful service guard cluster commands:

Cluster Management Commands
cmruncl
cmhaltcl
cmviewcl

Package Management Commands
cmrunpkg
cmhaltpkg
cmmodpkg

Node Management Commands
cmrunnode
cmhaltnode

This is the end of the two part tutorial. This tutorial may seem too abridged, but I am always available to explain/elaborate any part of it. Just leave a comment, I will answer as soon as possible.