GT 5.0.0 GRAM5 : System Administrator's Guide

GRAM5 depends on a local mechanism for starting and controlling jobs. GRAM5 includes a fork local resource manager which requires no special software to execute jobs on the local host. GRAM5 can also be configured to use additional batch facilities and schedulers such as condor, torque, or LSF. Install and configure any local resource managers you intend to use prior to configuring GRAM5.

GRAM5 depends on LRM adapters to execute jobs described by GRAM5 RSL documents on a resource managed by a LRM.

LRM adapters included in the GT 5.0.0 release are:

	Note
	The pbs LRM adapter may work with other systems that implement POSIX batch environment services.

For configuration details, see "Configuring LRM adapters" in the Configuring section.

To deploy GRAM5 to be started by inetd, define a service entry for the gsigatekeeper service in /etc/services file if it is not already present. The ICANN-assigned port number for the service is 2119, so add an entry like:

gsigatekeeper   2119/tcp

1.1. Deploying GRAM5 via inetd

To deploy GRAM5 to be started via inetd, modify /etc/inetd.conf (or your operating-system specific path to this file) by adding the following entry (on one line):

gsigatekeeper stream tcp nowait root /usr/bin/env env LD_LIBRARY_PATH=GLOBUS_LOCATION/lib GLOBUS_LOCATION/sbin/globus-gatekeeper -conf GLOBUS_LOCATION/etc/globus-gatekeeper.conf

After doing so, run the operating-system specific command to have inetd reload its configuration.

Important

This example assumes that the LD_LIBRARY_PATH environment variable is needed to be able to run the GRAM5 programs and scripts. This is operating-system specific and may need to be changed. Consult the documentation for the operating system's runtime linker in the case of runtime problems. The GLOBUS_LOCATION string in the above example must be replaced with the actual path where GT has been installed.

	Note
	When deploying GRAM5 via xinetd, be sure to include the command-line option -inetd in the gatekeeper configuration file; otherwise, the gatekeeper will log a warning each time it is started.

1.2. Deploying GRAM5 via xinetd

To deploy GRAM5 to be started via xinetd, create a file in the /etc/xinetd.d/ directory (or the operating-system specific path to xinetd configuration files) called gsigatekeeper with the following contents:

service gsigatekeeper
{
    socket_type = stream
    protocol = tcp
    wait = no
    user = root
    env = LD_LIBRARY_PATH=GLOBUS_LOCATION/lib
    server = GLOBUS_LOCATION/sbin/globus-gatekeeper
    server_args = -conf GLOBUS_LOCATION/etc/globus-gatekeeper.conf
    disable = no
}

After doing so, run the operating-system specific command to have xinetd reload its configuration.

Important

This example assumes that the LD_LIBRARY_PATH environment variable is needed to be able to run the GRAM5 programs and scripts. This is operating-system specific and may need to be changed. Consult the documentation for the operating system's runtime linker in the case of runtime problems. The GLOBUS_LOCATION string in the above example must be replaced with the actual path where GT has been installed.

	Note
	When deploying GRAM5 via xinetd, be sure to include the command-line option -inetd in the gatekeeper configuration file; otherwise, the gatekeeper will log a warning each time it is started.

To deploy GRAM5 to be started as a daemon, run the globus-gatekeeper at boot time. This may be done in an init script, via cron, or other system-specific methods. The typical command-line to run is

globus-gatekeeper -conf $GLOBUS_LOCATION/etc/globus-gatekeeper.conf

	Note
	When deploying GRAM5 as a daemon, be sure to include the command-line option -f in the gatekeeper configuration file; otherwise, the gatekeeper will log a warning each time it is started.

GRAM5 can be configured to use the globus-job-manager-event-generator program to monitor job state changes. This is often more efficient than using a LRM adapter's poll method. This program is configured when the LRM-specific bundle is installed. However, by default, the job manager does not use the SEG. It must be explicitly enabled by adding the -seg-module LRM option to the job manager configuration.

To start the globus-job-manager-event-generator program, add the following command to your system init scripts or crontab to be run at boot time:

$GLOBUS_LOCATION/sbin/globus-job-manager-event-generator -scheduler LRM -background -pidfile PIDFILE

This will start the globus-job-manager-event-generator to monitor jobs for the resource named by LRM in the background. The process id of the command will be written to the file named by PIDFILE, so that processes can check if it is running and kill it if necessary.

	Important
	If the job manager is configured to use a SEG module but the globus-job-manager-event-generator is not running for that LRM, jobs will appear to hang. It is important that the program be running whenever GRAM5 jobs might be run.

To stop the SEG, kill the globus-job-manager-event-generator process. The -pidfile option makes it easy to know which process to kill. When the SEG terminates, it will remove that file.

% globus-job-manager-event-generator -scheduler pbs -background -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid
Running in background (78258)
% kill `cat $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid`

GRAM5 includes mechanisms to provide access to audit and accounting information associated with jobs that GRAM5 submits to a local resource manager (LRM) such as PBS, LSF, or Condor.

	Note
	Remember, GRAM is not a local resource manager but rather a protocol engine for communicating with a range of different local resource managers using a standard message format.

In some scenarios, it is desirable to get general information about the usage of the underlying LRM, such as:

The following three use cases give a better overview of the meaning and purpose of auditing and accounting:

Audit logging in GRAM5 is done when a job completes.

While audit and accounting records may be generated and stored by different entities in different contexts, we make the following assumptions in this chapter:

The audit record of each job contains the following data:

The rest of this chapter focuses on how to configure GRAM5 to enable Audit-Logging. A case study for TeraGrid can be read here, which also includes more information about how to use this data to get accounting information of a job, query the audit database for information via a Web Services interface, etc.

Audit logging is turned off by default. To enable GRAM5 audit logging, in the job manager, add the command-line option -audit-directory {audit-directory} to the job manager configuration in either $GLOBUS_LOCATION/etc/globus-job-manager.conf to enable it for all job manager services, or in $GLOBUS_LOCATION/etc/grid-services/LRM_SERVICE_NAME to enable it for a particular job manager service for a particular LRM.

The globus-gram-audit program reads GRAM5 audit records and loads those records into an SQL database. This program is available as part of the globus_gram_job_manager_auditing package. It must be configured by installing and running the globus_gram_job_manager_auditing_setup_scripts setup package via gpt-postinstall. This setup script creates the $GLOBUS_LOCATION/etc/globus-job-manager-audit.conf configuration file described below and creates database tables needed by the audit system.

The globus-gram-audit program support three database systems: MySQL, PostgreSQL, and SQLite.

The GRAM protocol test suite tests the implementation of the GRAM protocol library which is used by the job manager and GRAM clients to process messages. The following examples shows how to run the test suite.

% cd $GLOBUS_LOCATION/test/globus_gram_protocol_test
% grid-proxy-init
Your identity: /DC=org/O=example/OU=grid/CN=Joe User
Enter GRID pass phrase for this identity:
Creating proxy ................................... Done
Your proxy is valid until: Thu Nov 12 23:28:05 2009
% ./TESTS.pl
globus-gram-protocol-allow-attach-test...........ok
globus-gram-protocol-error-test..................ok
globus-gram-protocol-io-test.....................ok
globus-gram-protocol-pack-test...................ok
pack-with-extensions-test........................ok
create-extensions-test...........................ok
unpack-message-test..............................ok
unpack-with-extensions-test......................ok
unpack-job-request-reply-with-extensions-test....ok
unpack-status-reply-with-extensions-test.........ok
All tests successful.
Files=10, Tests=42,  1 wallclock secs ( 0.37 cusr +  0.24 csys =  0.61 CPU)

The GRAM client test suite tests the interactions between the GRAM client API implementation and the job manager service. These tests include authentication, callback, signal, job, and cancellation tests.

Example 7.2. Running the GRAM Client Test Suite

% cd $GLOBUS_LOCATION/test/globus_gram_client_test
% grid-proxy-init
Your identity: /DC=org/O=example/OU=grid/CN=Joe User
Enter GRID pass phrase for this identity:
Creating proxy ................................... Done
Your proxy is valid until: Thu Nov 12 23:28:05 2009
% ./TESTS.pl
globus-gram-client-activate-test................ok
globus-gram-client-callback-contact-test........ok
globus-gram-client-cancel-test..................ok
globus-gram-client-nonblocking-register-test....ok 1/4Failed submitting job request because an authorization operation failed
globus_xio_gsi: gss_init_sec_context failed.
GSS Major Status: Unexpected Gatekeeper or Service Name
globus_gsi_gssapi: Authorization denied: The name of the remote entity (/DC=org/O=example/OU=grid/CN=Joe User), and the expected name for the remote entity (/DC=org/O=example/OU=grid/CN=Joe UserX) do not match
.
globus-gram-client-nonblocking-register-test....ok
globus-gram-client-refresh-credentials-test.....ok
globus-gram-client-register-test................ok
globus-gram-client-register-callback-test.......ok 1/4Failed submitting job request because an authorization operation failed
globus_xio_gsi: gss_init_sec_context failed.
GSS Major Status: Unexpected Gatekeeper or Service Name
globus_gsi_gssapi: Authorization denied: The name of the remote entity (/DC=org/O=example/OU=grid/CN=Joe User), and the expected name for the remote entity (/DC=org/O=example/OU=grid/CN=Joe UserX) do not match
.
globus-gram-client-register-callback-test.......ok
globus-gram-client-register-cancel-test.........ok
globus-gram-client-ping-test....................ok
globus-gram-client-status-test..................Made 3961 calls to status in 60.416390 seconds
globus-gram-client-status-test..................ok
globus-gram-client-two-phase-commit-test........ok
globus-gram-client-register-ping-test...........ok
globus-gram-client-stdio-size-test..............job manager returned 0 (Success) when I expected it to still be streaming output
globus-gram-client-stdio-size-test..............ok
version-test....................................ok
All tests successful.
Files=14, Tests=33, 791 wallclock secs (32.10 cusr +  5.34 csys = 37.44 CPU)

	Note
	Some of the test cases display messages that look like errors when running. This is to be expected. The only concern should be the final lines indicating if the tests are successful or not.

	Note
	By default, this suite uses tests against a personal gatekeeper running the fork LRM. To run against another service, set the environment variable CONTACT_STRING to the service contact string prior to starting the tests.

The GRAM job manager test suite tests the features provided by the LRM scripts, including detecting failures, staging files, and submitting different types of jobs. The following example shows how to run the job manager tests.

% cd $GLOBUS_LOCATION/test/globus_gram_job_manager_test
% grid-proxy-init
Your identity: /DC=org/O=example/OU=grid/CN=Joe User
Enter GRID pass phrase for this identity:
Creating proxy ................................... Done
Your proxy is valid until: Thu Nov 12 23:28:05 2009
% ./TESTS.pl
job-manager-script-test..................ok
globus-gram-job-manager-stdio-test.......ok
globus-gram-job-manager-submit-test......ok
globus-gram-job-manager-failure-test.....ok
globus-gram-job-manager-rsl-size-test....ok
globus-gram-job-manager-user-test........ok
All tests successful.
Files=6, Tests=137, 200 wallclock secs (32.10 cusr +  5.34 csys = 37.44 CPU)

	Note
	This test requires a GridFTP server to be running on the host running the test suite. If one is not running, then the following test cases will fail: globus-gram-job-manager-stdio-test subtests 2, 6, 9, 12, 13, 16, 17, 18, 19.

	Note
	By default, this suite uses tests against a personal gatekeeper running the fork LRM. To run against another service, set the environment variable CONTACT_STRING to the service contact string prior to starting the tests.

In case you run into problems you can do the following

The following usage statistics are sent by default in a UDP packet (in addition to the GRAM component code, packet version, timestamp, and source IP address) at the end of each job.

The following information can be sent as well in a job status packet but it is not sent unless explicitly enabled by the system administrator:

In addition to job-related status, the job manager sends information periodically about its execution status. The following information is sent by default in a UDP packet (in addition to the GRAM component code, packet version, timestamp, and source IP address) at job manager start and every 1 hour during the job manager lifetime:

Also, please see our policy statement on the collection of usage statistics.