In the
first part of this tutorial I showed the basic serviceguard cluster configuration. Now I am going to show how to configure package inside the cluster. A package is an application and all the application’s resources required to execute properly.
Resources for packages may include:
– Disk Resources ( Volume Group(s), Disk Group(s) )
– Network Resources ( IP Address(es) )
Packaging Files include:
– Package Configuration File ( pkg.conf )
– Package Control Script ( pkg.cntl )
As I said earlier I will configure Oracle Database server as a package of service guard. The database instance will run on the first node only. In the event of a failure it will run on the second node.
At first we need to create file system on the shared storage. Oracle will use these shared file system for storing data files as well as binary. To create file system we need to create physical volume, volume group and logical volume. Here is the commands:
Physical Volume (PV) Creation:
# pvcreate -f /dev/rdsk/c4t0d1
Physical volume "/dev/rdsk/c4t0d1" has been successfully created.
# pvcreate -f /dev/rdsk/c4t0d2
Physical volume "/dev/rdsk/c4t0d2" has been successfully created.
|
VG Creation:
Check the device number of the existing VGs.
# ls -l /dev/*/group
crw-r----- 1 root sys 64 0x000000 Oct 4 11:27 /dev/vg00/group
crw-r--r-- 1 root sys 64 0x010000 Oct 4 15:16 /dev/vglock/group
|
Create group file for new volume groups:
# mkdir /dev/vgappl
# mknod /dev/vgappl/group c 64 0x020000
# mkdir /dev/vgoradata
# mknod /dev/vgoradata/group c 64 0x030000
|
Create the required volume groups:
# vgcreate -s 32 vgappl /dev/dsk/c4t0d1
Increased the number of physical extents per physical volume to 6399.
Volume group "/dev/vgappl" has been successfully created.
Volume Group configuration for /dev/vgappl has been saved in /etc/lvmconf/vgappl.conf
# vgcreate -s 32 vgoradata /dev/dsk/c4t0d2
Increased the number of physical extents per physical volume to 6399.
Volume group "/dev/vgoradata" has been successfully created.
Volume Group configuration for /dev/vgoradata has been saved in /etc/lvmconf/vgoradata.conf
|
Create the logical volumes (LV):
# lvcreate -L 204000 -n lvappl /dev/vgappl
Logical volume "/dev/vgappl/lvappl" has been successfully created with
character device "/dev/vgappl/rlvappl".
Logical volume "/dev/vgappl/lvappl" has been successfully extended.
Volume Group configuration for /dev/vgappl has been saved in /etc/lvmconf/vgappl.conf
# lvcreate -L 204000 -n lvoradata /dev/vgoradata
Logical volume "/dev/vgoradata/lvoradata" has been successfully created with
character device "/dev/vgoradata/rlvoradata".
Logical volume "/dev/vgoradata/lvoradata" has been successfully extended.
Volume Group configuration for /dev/vgoradata has been saved in /etc/lvmconf/vgoradata.conf
|
Create file system / mount point:
# newfs -F vxfs -o largefiles /dev/vgappl/rlvappl
version 5 layout
208896000 sectors, 208896000 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles supported
208896000 data blocks, 208826960 free data blocks
6375 allocation units of 32768 blocks, 32768 data blocks
# newfs -F vxfs -o largefiles /dev/vgoradata/rlvoradata
version 5 layout
208896000 sectors, 208896000 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles supported
208896000 data blocks, 208826960 free data blocks
6375 allocation units of 32768 blocks, 32768 data blocks
|
Mount the file systems:
mount /dev/vgappl/lvappl /appl
mount /dev/vgoradata/lvoradata /oradata
|
Up to this stage, two file systems /appl and /oradata are created. Now we will install oracle database on these mount points. /appl will contain the oracle binary and /oradata will be used for storing the data files. I will not show the oracle installation. Just keep one thing in mind: The mount points /appl and /oradata are shared mount points. So, for anything that resides on these two mount points, you don’t need to copy it to the second node. But for any other files (i.e /etc/oratab, /var/opt/oracle/oratab etc) you need to manually send the files to the second node at the same location. Another important thing is, user id and group id of oracle user and oinstall group must same across all cluster node. Actually its applicable for any user, not only oracle.
After installing oracle, log on as oracle user, set the environment variables (ORACLE_HOME, ORACLE_SID etc) and start the database and listener to check that the installation is good. If everything is fine, then stop the database and listener.
Now you will,
1. unmount the file system on 1st node
2. deactivate the VGs and export the VGs from 1st node to a map file
3. send the map file to 2nd node
4. import the VGs on 2nd node from the map file
5. activate the VG and mount the file systems on 2nd node.
6. start the database and listener
7. If everything is OK then stop database, unmount file systems and deactivate VG on 2nd node.
Here is the commands:
On first node:
# umount /appl
# umount /oradata
# vgchange -a n vgappl
Volume group "vgappl" has been successfully changed.
# vgchange -a n vgoradata
Volume group "vgoradata" has been successfully changed.
# vgexport -p -v -s -m /etc/lvmconf/vgappl.map vgappl
Beginning the export process on Volume Group "vgappl".
/dev/dsk/c4t0d1
# vgexport -p -v -s -m /etc/lvmconf/vgoradata.map vgoradata
Beginning the export process on Volume Group "vgoradata".
/dev/dsk/c4t0d2
# rcp /etc/lvmconf/vgappl.map node02:/etc/lvmconf/
# rcp /etc/lvmconf/vgoradata.map node02:/etc/lvmconf/
|
On 2nd node:
# mkdir /dev/vgappl
# mkdir /dev/vgoradata
# mknod /dev/vgappl/group c 64 0x020000
# mknod /dev/vgoradata/group c 64 0x030000
# ls -l /dev/*/group
crw-r----- 1 root sys 64 0x000000 Oct 4 13:00 /dev/vg00/group
crw-r--r-- 1 root sys 64 0x020000 Oct 5 10:05 /dev/vgappl/group
crw-r--r-- 1 root sys 64 0x010000 Oct 4 15:22 /dev/vglock/group
crw-r--r-- 1 root sys 64 0x030000 Oct 5 10:05 /dev/vgoradata/group
# vgimport -v -s -m /etc/lvmconf/vgappl.map vgappl
Beginning the import process on Volume Group "vgappl".
Logical volume "/dev/vgappl/lvappl" has been successfully created
with lv number 1.
Volume group "/dev/vgappl" has been successfully created.
# vgimport -v -s -m /etc/lvmconf/vgoradata.map vgoradata
Beginning the import process on Volume Group "vgoradata".
Logical volume "/dev/vgoradata/lvoradata" has been successfully created
with lv number 1.
Volume group "/dev/vgoradata" has been successfully created.
# mkdir /appl
# mkdir /oradata
# mount /dev/vgappl/lvappl /appl
# mount /dev/vgoradata/lvoradata /oradata
|
Now start the database, listener and check. After successful checking shutdown db, unmount and deactivate the VG on the 2nd node.
The next step is package configuration. We will do it from 1st node.
Configuring package on servicegurad is a 3 step procedure:
1. Build the Package Configuration File
2. Build the Package Run/Halt Script
3. Compile and Distribute the Binary File
Step 1: Build the Package Configuration File
The first step in building the package configuration file is to run the cmmakepkg command to create the package configuration template:
# mkdir /etc/cmcluster/orapkg
# cd /etc/cmcluster/orapkg
# cmmakepkg -v –p orapkg.conf
Begin generating package template... Done.
Package template is created.
This file must be edited before it can be used.
|
The next step is to edit / customize the file with the vi editor:
After editing the file will look something like below:
# cat /etc/cmcluster/orapkg/orapkg.conf
PACKAGE_NAME orapkg
PACKAGE_TYPE FAILOVER
FAILOVER_POLICY CONFIGURED_NODE
FAILBACK_POLICY MANUAL
NODE_NAME node01
NODE_NAME node02
AUTO_RUN YES
LOCAL_LAN_FAILOVER_ALLOWED YES
NODE_FAIL_FAST_ENABLED NO
RUN_SCRIPT /etc/cmcluster/orapkg/orapkg.ctrl
RUN_SCRIPT_TIMEOUT NO_TIMEOUT
HALT_SCRIPT /etc/cmcluster/orapkg/orapkg.ctrl
HALT_SCRIPT_TIMEOUT NO_TIMEOUT
SERVICE_NAME orapkg_db1
SERVICE_FAIL_FAST_ENABLED NO
SERVICE_HALT_TIMEOUT 300
SUBNET 10.10.96.0
|
Here the value of RUN_SCRIPT and HALT_SCRIPT option is /etc/cmcluster/orapkg/orapkg.ctrl. Actually this is the script that will start and stop the application. In our case it will start/stop the database and listener. We will generate and modify this script on next step.
Step 2: Build the Package Run/Halt Script
The procedure for building the package control script is to create a template with the command:
# cd /etc/cmcluster/orapkg
# cmmakepkg -v -s orapkg.ctrl
Begin generating package control script... Done.
Package control script is created.
This file must be edited before it can be used.
|
Before going to the next step, copy all files from /opt/cmcluster/toolkit/oracle/ to /etc/cmcluster/orapkg/ directory. The mentioned directory contains HP supplied sample start-up/monitoring script for oracle. HP also supplies sample script for other application including apache, samba, NFS, tomcat etc.
#cp /opt/cmcluster/toolkit/oracle/* .
|
The next step is to edit/customize the script with the vi editor. This file is self explanatory. You need to specify the VG, LV, mount point and mount options for all package file systems (/appl and /oradata in this case). You also need to specify package IP address (ip address on which application will connect), SERVICE_NAME (a name for your service: any relevant name will suffice) and SERVICE_CMD (The full pathname of the command used to start the service process). After editing, the modified portion of the file will look like the following. Leave the rest of the file as it was.
VG[0]=vgappl
VG[1]=vgoradata
LV[0]=/dev/vgappl/lvappl; FS[0]=/appl; FS_MOUNT_OPT[0]="-o rw"
LV[1]=/dev/vgoradata/lvoradata; FS[1]=/oradata; FS_MOUNT_OPT[1]="-o rw"
IP[0]=192.168.100.50
function customer_defined_run_cmds
{
/etc/cmcluster/orapkg/toolkit.sh start
test_return 51
}
function customer_defined_halt_cmds
{
/etc/cmcluster/orapkg/toolkit.sh shutdown
test_return 52
}
|
Now edit the /etc/cmcluster/orapkg/haoracle.conf file and set ORACLE_HOME and SID_NAME. This file will look like following:
ORACLE_HOME=/appl/oracle/product/10.2.0
ORACLE_ADMIN=oracle
SID_NAME=orcl
LISTENER=yes
LISTENER_NAME=LISTENER
LISTENER_PASS=
USE_SPFILE=yes
set -A MONITOR_PROCESSES
MONITOR_PROCESSES[0]=ora_pmon_${SID_NAME}
MONITOR_PROCESSES[1]=ora_dbw0_${SID_NAME}
MONITOR_PROCESSES[2]=ora_ckpt_${SID_NAME}
MONITOR_PROCESSES[3]=ora_smon_${SID_NAME}
MONITOR_PROCESSES[4]=ora_lgwr_${SID_NAME}
MONITOR_PROCESSES[5]=ora_reco_${SID_NAME}
MAINTENANCE_FLAG=yes
MONITOR_INTERVAL=30
TIME_OUT=30
|
You have completed package configuration. Before going to next step edit the cluster configuration file /etc/cmcluster/crmcluster.ascii and add the following lines:
VOLUME_GROUP /dev/vgappl
VOLUME_GROUP /dev/vgoradata
|
Transfer the /etc/cmcluster/orapkg to the 2nd node.
Step 3. Compile and Distribute the Binary File
Here is the last configuration step: creating and distributing the package binary file. Execute the following commands one after another:
# cd /etc/cmcluster
# cmcheckconf -v -P orapkg/orapkg.conf -C crmcluster.ascii
Checking cluster file: crmcluster.ascii
Note : a NODE_TIMEOUT value of 2000000 was found in line 127. For a
significant portion of installations, a higher setting is more appropriate.
Refer to the comments in the cluster configuration ascii file or Serviceguard
manual for more information on this parameter.
Checking nodes ... Done
Checking existing configuration ... Done
Volume group /dev/vgappl is configured differently on node node01 than on node node02
Volume group /dev/vgappl is configured differently on node node02 than on node node01
Volume group /dev/vgoradata is configured differently on node node01 than on node node02
Volume group /dev/vgoradata is configured differently on node node02 than on node node01
Volume group /dev/vglock is configured differently on node node01 than on node node02
Volume group /dev/vglock is configured differently on node node02 than on node node01
Parsing package file: orapkg/orapkg.conf.
Checking for inconsistencies
Modifying configuration on node node01
Modifying configuration on node node02
cmcheckconf: Verification completed with no errors found.
Use the cmapplyconf command to apply the configuration.
|
The above command will check correctness and consistency of configuration file. If it completes successfully then run the following command:
# cd /etc/cmcluster
# cmapplyconf -v -P orapkg/orapkg.conf -C crmcluster.ascii
Checking cluster file: crmcluster.ascii
Note : a NODE_TIMEOUT value of 2000000 was found in line 127. For a
significant portion of installations, a higher setting is more appropriate.
Refer to the comments in the cluster configuration ascii file or Serviceguard
manual for more information on this parameter.
Checking nodes ... Done
Checking existing configuration ... Done
Volume group /dev/vgappl is configured differently on node node01 than on node node02
Volume group /dev/vgappl is configured differently on node node02 than on node node01
Volume group /dev/vgoradata is configured differently on node node01 than on node node02
Volume group /dev/vgoradata is configured differently on node node02 than on node node01
Volume group /dev/vglock is configured differently on node node01 than on node node02
Volume group /dev/vglock is configured differently on node node02 than on node node01
Parsing package file: orapkg/orapkg.conf.
Checking for inconsistencies
Modifying configuration on node node01
Modifying configuration on node node02
Modify the cluster configuration ([y]/n)? y
Marking/unmarking volume groups for use in the cluster
Completed the cluster creation
|
Package is compiled successfully. You have the start the cluster using cmruncl command and then you can use cmrunpkg <package_name>command to start the package.
Here is some useful service guard cluster commands:
Cluster Management Commands
cmruncl
cmhaltcl
cmviewcl
Package Management Commands
cmrunpkg
cmhaltpkg
cmmodpkg
Node Management Commands
cmrunnode
cmhaltnode
This is the end of the two part tutorial. This tutorial may seem too abridged, but I am always available to explain/elaborate any part of it. Just leave a comment, I will answer as soon as possible.
