HP-UX - Serviceguard Working with Package Maintenance Mode

[admin]

HP-UX Serviceguard - Working with Package Maintenance Mode

Serviceguard version A.11.19 introduced the ability to start a package in maintenance mode. How is it invoked and cleared?

The following explanation uses an example package named AP_PKG , to help demonstrate commands that will clarify how to use package maintenance mode.

Package maintenance mode can be used to terminate package startup at an interim point of normal package startup, to enable the admin to manually troubleshoot succeeding operations that are not working as expected in automated mode or otherwise experiment.

Package maintenance mode involves two commands; cmmodpkg and cmrunpkg .

To use package maintenance mode with these commands, the admin must learn the names of the package modules and the order in which they are started.

Example : If the admin wants to start an Oracle database package AP_PKG such that the system resources are assigned but the database is not started, learn the names of the modules and startup order by inspecting of the modular package configuration file:

$ cmgetconf -p AP_PKG | grep -E '^op|^module_name' | sed 's/ / /g'
module_name sg/basic
module_name sg/priority
module_name sg/failover
module_name sg/monitor_subnet
module_name sg/package_ip<- TAKE NOTE
module_name sg/service
module_name sg/volume_group
module_name sg/filesystem
module_name sg/dependency
module_name ecmt/oracle/oracle
module_name sg/all
module_name sg/weight
module_name sg/generic_resource
module_name sg/resource
module_name sg/pev
module_name sg/external_pre
module_name sg/external
module_name sg/acp
operation_sequence $SGCONF/scripts/sg/external_pre.sh
operation_sequence $SGCONF/scripts/sg/volume_group.sh
operation_sequence $SGCONF/scripts/sg/filesystem.sh
operation_sequence $SGCONF/scripts/sg/package_ip.sh <- TAKE NOTE
operation_sequence $SGCONF/scripts/ecmt/oracle/tkit_module.sh
operation_sequence $SGCONF/scripts/sg/external.sh
operation_sequence $SGCONF/scripts/sg/service.sh
operation_sequence $SGCONF/scripts/sg/resource.sh

The same listing can be retrieved from the cluster binary file:

$ cmviewcl -v -f line | grep AP_PKG | grep -E 'sequence=|module_name=' | cut -d\| -f3-

Noticed that the last module that runs before the Oracle toolkit gets involved to startup the database is sg/package_ip . Therefore, to start the AP_PKG package such that it does not start the Oracle database module, use these commands:

$ cmmodpkg -v -m on AP_PKG # enable maintenance mode for this package
$ cmrunpkg AP_PKG -m sg/package_ip # starts AP_PKG, up through IP assignment

In this example, the volume groups are activated, file systems mounted and relocatable IPs are assigned. At this point, the admin can manually start the database or perform other maintenance as needed.

A package cannot be switched to non-maintenance mode while running.

Example :

# cmmodpkg -v -n rxh17u09 -m off AP_PKG

Package AP_PKG must be cleanly halted before it can be removed from maintenance mode.

However the package can be switched from standard mode to maintenance mode while it is running, if the admin wants the package to ignore failure triggers.

See cmmodpkg(1M) man page for details and requirements.

To shut down a package in maintenance mode, first remove all dependencies on the package resources that were started after maintenance mode startup, such as halting a database. Then run these commands:

$ cmhaltpkg AP_PKG
$ cmmodpkg -v -m off AP_PKG # turns off maintenance mode for this package

Since package maintenance disables package failover, use these commands to re-enable package failover:

$ cmmodpkg -e AP_PKG -n hqoffmed1 # enables Node_switching on hqoffmed1
$ cmmodpkg -e AP_PKG -n hqoffmed4 # enables Node_switching on hqoffmed4
$ cmmodpkg -e AP_PKG # enables AUTO_RUN flag (and starts the package)
$ cmmodpkg -d AP_PKG # disables AUTO_RUN flag (this command is for reference only)

NOTE: cmrunpkg also enables the admin to exclude a module operation during package startup. Review the -e option in the cmrunpkg(1M) man page.

Maintenance node is reported in cmviewcl :

>cmviewcl -p AP_PKG

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
AP_PKG down maintenance disabled unowned

NOTE: The STATE changes from halted to maintenance .

The admin can also combine the -m and -e option with cmrunpkg to get the package to run modules up to a particular one, and exclude an interim module.

Example :

# cmrunpkg -v -m sg/package_ip -e sg/volume_group AP_PKG
Running package modular on node hqoffmed1
Successfully started package modular on node hqoffmed1
cmrunpkg: All specified packages are running

The package log shows this special treatment:

# tail -2 /var/adm/cmcluster/log/AP_PKG.log
Aug 9 10:22:46 root@hqoffmed1 master_control_script.sh[8013]: ###### Starting package AP_PKG up to
/etc/cmcluster/scripts/sg/package_ip.sh and excluding /etc/cmcluster/scripts/sg/volume_group.sh ######
Aug 9 10:22:47 root@hqoffmed1 master_control_script.sh[8013]: ###### Package start completed for AP_PKG ######

NOTES :

Maintenance mode is only available for modular packages.

Because Serviceguard does not monitor for failures of a package in maintenance mode, it will not fail over in the event of a failure though it may be configured to do so. Status of some package values may change to unknown , such as services.

Example :

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service unknown 2 0 xload