Configuring

1. Typical Configuration

1.1. Configuring sudo

When the credentials of the service account and the job submitter are different (multi user mode), then GRAM will prepend a call to sudo to the local adapter callout command. Important: If sudo is not configured properly, the command and thus job will fail.

As root, here are the two lines to add to the /etc/sudoers file for each GLOBUS_LOCATION installation, where /opt/globus/GT4.2.0 should be replaced with the GLOBUS LOCATION for your installation:

# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-gridmap-and-execute \
    -g /etc/grid-security/grid-mapfile \
    /opt/globus/GT4.2.0/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-gridmap-and-execute \
    -g /etc/grid-security/grid-mapfile \
    /opt/globus/GT4.2.0/libexec/globus-gram-local-proxy-tool *

The globus-gridmap-and-execute program is used to ensure that GRAM only runs programs under accounts that are in the grid-mapfile. In the sudo configuration, it is the first program called. It looks up the account in the grid-mapfile and then runs the requested command. It is redundant if sudo is properly locked down. This tool could be replaced with your own authorization program.

1.2. Configuring Scheduler Adapters

The GRAM4 scheduler adapters included in the release tarball are: PBS, Condor and LSF. To install, follow these steps (shown for pbs):

% configure --prefix=$GLOBUS_LOCATION --enable-wsgram-pbs ...
% make
% make install

Using PBS as the example, make sure the scheduler commands are in your path (qsub, qstat, pbsnodes).

For PBS, another setup step is required to configure the remote shell for rsh access:

% cd $GLOBUS_LOCATION/setup/globus
% ./setup-globus-job-manager-pbs --remote-shell=rsh

The last thing is to define the GRAM and GridFTP file system mapping for PBS. A default mapping in this file is created to allow simple jobs to run. However, the actual file system mappings for your compute resource should be entered to ensure:

  • files staging is performed correctly

  • jobs with erroneous file path directives are rejected

Done! You have added the PBS scheduler adapters to your GT installation.

2. Non-default Configuration

2.1. Credentials

To run the container using just a user proxy, instead of host creds, edit the $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml file, and either comment out the credentials section...

<?xml version="1.0" encoding="UTF-8"?>
<securityConfig xmlns="http://www.globus.org">
<!--
<credential>
<key-file value="/etc/grid-security/containerkey.pem"/>
<cert-file value="/etc/grid-security/containercert.pem"/>
<credential>
-->
<gridmap value="/etc/grid-security/grid-mapfile"/>
<securityConfig>

or replace the credentials section with a proxy file location...

<?xml version="1.0" encoding="UTF-8"?>
<securityConfig xmlns="http://www.globus.org">
<proxy-file value="<PATH TO PROXY FILE>"/>
<gridmap value="/etc/grid-security/grid-mapfile"/>
<securityConfig>

Running in personal mode (user proxy), another GRAM configuration setting is required. For GRAM to authorize the RFT service when performing staging functions, it needs to know the subject DN for verification. Here are the steps:

% cd $GLOBUS_LOCATION/setup/globus
% ./setup-gram-service-common --staging-subject=
"/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720"

You can get your subject DN by running this command:

% grid-cert-info -subject
            

2.2. GridFTP server

By default, the GridFTP server is assumed to run as root on localhost:2811. If this is not true for your site then change it by editing the GridFTP host and/or port in the GRAM and GridFTP file system mapping config file: $GLOBUS_LOCATION/etc/globus_wsrf_gram/globus_gram_fs_map_config.xml.

2.3. Container port

By default, the globus services will assume 8443 is the port the Globus container is using. However the container can be run under a non-standard port, for example:

% globus-start-container -p 4321

2.4. Gridmap

If you wish to specify a non-standard gridmap file in a multi-user installation, two basic configurations need to be changed:

  • $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml

    • As specified in the gridmap config instructions, add a <gridmap value="..."/> element to the file appropriately.
  • /etc/sudoers

    • Change the file path after all -g options

      -g /path/to/grid-mapfile

      .

Example: global_security_descriptor.xml

...
<gridmap value="/opt/grid-mapfile"/>
...

sudoers

...
# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-gridmap-and-execute \
    -g /opt/grid-mapfile \
    /opt/globus/GT4.2.0/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-gridmap-and-execute \
    -g /opt/grid-mapfile \
    /opt/globus/GT4.2.0/libexec/globus-gram-local-proxy-tool *...

2.5. RFT deployment

RFT is used by GRAM to stage files in and out of the job execution environment. In the default configuration, RFT is hosted in the same container as GRAM and is assumed to have the same service path and standard service names. This need not be the case. For example, the most likely alternative scenario is that RFT would be hosted seperately in a container on a different machine. In any case, both the RFT and the Delegation Service endpoints need to be adjustable to allow this flexibility. The following options can be passed to the setup-gram-service-common script to affect these settings:

--staging-protocol=<protocol> \
--staging-host=<host> \
--staging-port=<port> \
--staging-service-path=<RFT and Delegation factory service path> \
--staging-factory-name=<RFT factory service name> \
--staging-delegation-factory-name=<name of Delegation factory service used by RFT>

For example:

    % setup-gram-service-common \
    --staging-protocol=https
    --staging-host=machine.domain.net
    --staging-delegation-factory-name=machine.domain.net
        

will internally cause the GRAM service code to construct the following EPR addresses.

http://machine.domain.net:8444/wsrf/services/ReliableFileTransferFactoryService
http://machine.fakedomain.net:8444/wsrf/services/DelegationFactoryService
        

[Important]Important

Jobs submitted by Condor-G will fail if RFT is not located in the same container as GRAM4, because Condor-G assumes that RFT is located in the same container like GRAM4.

2.6. Authorization

Note that, by default, two authorization checks are done during an invocation of WS-GRAM:

  1. One authorization check is done by the container as configured by the defaultAuthzParam element in $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml
  2. Another check is done by WS-GRAM when it calls the Perl modules which are used for job submission to the underlying local resource manager. This is configured by the authzChain element which is, by default, set to gridmap in $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-factory-security-config.xml and $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-security-config.xml. This check is done for additional security reasons to make sure that a potentially hacked globus user account still can only act on behalf of the users who are defined in a grid-mapfile.

The second gridmap check can be avoided by adding a system property to the environment variable GLOBUS_OPTIONS before starting the container:

export GLOBUS_OPTIONS="$GLOBUS_OPTIONS -Dorg.globus.exec.disablegge=true"

This however does not mean, that no authorization check at all is done. The container still checks if the client is authorized as defined in $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml but there's no further authorization check when calling the Perl modules. It's up to the GT4 container administrator to decide whether or not to have that additional authorization check. Note that a change in the sudo configuration is required in that case because globus-gridmap-and-execute will not be executed. /opt/globus/GT4.2.0 should be replaced with the value of $GLOBUS_LOCATION for your installation:

# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.0/libexec/globus-gram-local-proxy-tool *

3. Configuration Details

3.1. JNDI configuration

The configuration of WSRF resources and application-level service configuration not related to service deployment is contained in JNDI files. The JNDI-based GRAM configuration is of two kinds:

3.1.1. Common job factory configuration

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml contains configuration information that is common to every local resource manager.

More precisely, the configuration data it contains pertains to the implementation of the GRAM WSRF resources (factory resources and job resources), as well as initial values of WSRF resource properties that are always published by any Managed Job Factory WSRF resource.

The data is categorized by service, because according to WSRF, in spite of the service/resource separation of concern, a given service will use only one XML Schema type of resource. In practice it is therefore clearer to categorize the configuration resource implementation by service, even if theoretically speaking a given resource implementation could be used by several services. For more information, refer to the Java WS Core documentation.

Here is the decomposition, in JNDI objects, of the common configuration data, categorized by service. Each XYZHome object contains the same Globus Core-defined information for the implementation of the WSRF resource, such as the Java implementation class for the resource (resourceClass datum), the Java class for the resource key (resourceKeyType datum), etc.

  • ManagedExecutableJobService

    • ManagedExecutableJobHome: configuration of the implementation of resources for the service.
  • ManagedMultiJobService

    • ManagedMultiJobHome: configuration of the implementation of resources for the service
  • ManagedJobFactoryService

    • FactoryServiceConfiguration: this encapsulates configuration information used by the factory service. Currently this identifies the service to associate to a newly created job resource in order to create an endpoint reference and return it.
    • ManagedJobFactoryHome: implementation of resources for the service resourceClass
    • FactoryHomeConfiguration: this contains GRAM application-level configuration data i.e. values for resource properties common to all factory resources. For instance, the path to the Globus installation, host information such as CPU type, manufacturer, operating system name and version, etc.

3.1.2. Local resource manager configuration

When a SOAP call is made to a GRAM factory service in order to submit a job, the call is actually made to a GRAM service-resource pair, where the factory resource represents the local resource manager to be used to execute the job.

There is one directory globus_wsrf_gram_<manager>/ for each local resource manager supported by the GRAM installation.

For instance, let's assume the command line:

% ls etc | grep globus_wsrf_gram_

gives the following output:

globus_wsrf_gram_Fork
globus_wsrf_gram_LSF
globus_wsrf_gram_Multi

In this example, the Multi, Fork and LSF job factory resources have been installed. Multi is a special kind of local resource manager which enables the GRAM services to support multijobs.

The JNDI configuration file located under each manager directory contains configuration information for the GRAM support of the given local resource manager, such as the name that GRAM uses to designate the given resource manager. This is referred to as the GRAM name of the local resource manager.

For instance, $GLOBUS_LOCATION/etc/globus_wsrf_gram_Fork/jndi-config.xml contains the following XML element structure:

<service name="ManagedJobFactoryService">
    <!-- LRM configuration:  Fork -->
    <resource
        name="ForkResourceConfiguration"
        type="org.globus.exec.service.factory.FactoryResourceConfiguration">
        <resourceParams>
            [...]
            <parameter>
                <name>
                    localResourceManagerName
                </name>
                <value>
                    Fork
                </value>
            </parameter>           
            <!-- Site-specific scratchDir
                    Default: ${GLOBUS_USER_HOME}/.globus/scratch
            <parameter>
                <name>
                    scratchDirectory
                </name>
                <value>
                    ${GLOBUS_USER_HOME}.globus/scratch
                </value>
            </parameter>           
            -->
        </resourceParams>
    </resource>
</service>

In the example above, the name of the local resource manager is Fork. This value can be used with the GRAM command line client in order to specify which factory resource to use when submitting a job. Similarly, it is used to create an endpoint reference to the chosen factory WS-Resource when using the GRAM client API.

In the example above, the scratchDirectory is set to ${GLOBUS_USER_HOME}/.globus/scratch. This is the default setting. It can be configured to point to an alternate file system path that is common to the compute cluster and is typically less reliable (auto purging), while offering a greater amount of disk space (thus "scratch").

3.2. Security descriptor

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-factory-security-config.xml contains the Core security configuration for the GRAM ManagedJobFactory service:

  • default security information for all remote invocations, such as:

    • the authorization method, based on a Gridmap file (in order to resolve user credentials to local user names)
    • limited proxy credentials will be rejected
  • security information for the createManagedJob operation

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-security-config.xml contains the Core security configuration for the GRAM job resources:

  • The default is to only allow the identity that called the createManagedJob operation to access the resource.

Note: GRAM does not override the container security credentials defined in $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml. These are the credentials used to authenticate all service requests.

3.3. Web service deployment descriptor

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/server-config.wsdd contains information necessary to deploy and instantiate the GRAM services in the Globus container.

Three GRAM services are deployed:

  • ManagedExecutableJobService: service invoked when querying or managing an executable job
  • ManagedMultiJobService: service invoked when querying or managing a multijob
  • ManagedJobFactoryService: service invoked when submitting a job

Each service deployment information contains the name of the Java service implementation class, the path to the WSDL service file, the name of the operation providers that the service reuses for its implementation of WSDL-defined operations, etc. More information about the service deployment configuration information can be found in Configuring Java WS Core.

3.4. Scheduler-Specific Scheduler Event Generator configuration files

In addition to the service configuration described above, there are scheduler-specific configuration files for the Scheduler Event Generator modules. These files consist of name=value pairs separated by newlines. These files are:

3.4.1. $GLOBUS_LOCATION/etc/globus-fork.conf

Configuration for the Fork SEG module implementation. The attributes names for this file are:

log_path
Path to the SEG Fork log (used by the globus-fork-starter and the SEG). The value of this should be the path to a world-writable file. The default value for this created by the Fork setup package is $GLOBUS_LOCATION/var/globus-fork.log. This file must be readable by the account that the SEG is running as.

3.4.2. $GLOBUS_LOCATION/etc/globus-condor.conf

Configuration for the Condor SEG module implementation. The attributes names for this file are:

log_path
Path to the SEG Condor log (used by the Globus::GRAM::JobManager::condor perl module and Condor SEG module. The value of this should be the path to a world-readable and world-writable file. The default value for this created by the Fork setup package is $GLOBUS_LOCATION/var/globus-condor.log

3.4.3. $GLOBUS_LOCATION/etc/globus-pbs.conf

Configuration for the PBS SEG module implementation. The attributes names for this file are:

log_path
Path to the SEG PBS logs (used by the Globus::GRAM::JobManager::pbs perl module and PBS SEG module. The value of this should be the path to the directory containing the server logs generated by PBS. For the SEG to operate, these files must have file permissions such that the files may be read by the user the SEG is running as.

3.4.4. $GLOBUS_LOCATION/etc/globus-lsf.conf

Configuration for the LSF SEG module implementation. The attributes names for this file are:

log_path
Path to the SEG LSF log directory. This is used by the LSF SEG module. The value of this should be the path to the directory containing the server logs generated by LSF. For the SEG to operate, these files must have file permissions such that the files may be read by the user the SEG is running as.

4. GRAM4 and GridFTP file system mapping

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/globus_gram_fs_map_config.xml contains information to associate local resource managers with GridFTP servers. GRAM uses the GridFTP server (via RFT) to perform all file staging directives. Since the GridFTP server and the Globus service container can be run on separate hosts, a mapping is needed between the common file system paths of these 2 hosts. This enables the GRAM services to resolve file:/// staging directives to the local GridFTP URLs.

Below is the default Fork entry. Mapping a jobPath of / to ftpPath of / will allow any file staging directive to be attempted.

<map>
    <scheduler>Fork</scheduler>
    <ftpServer>
       <protocol>gsiftp</protocol>
       <host>myhost.org</host>
       <port>2811</port>
    </ftpServer>
    <mapping>
       <jobPath>/</jobPath>
       <ftpPath>/</ftpPath>
    </mapping>
</map>

For a scheduler, where jobs will typically run on a compute node, a default entry is not provided. This means staging directives will fail until a mapping is entered. Here is an example of a compute cluster with PBS installed that has 2 common mount points between the front end host and the GridFTP server host.

<map>
    <scheduler>PBS</scheduler>
    <ftpServer>
       <protocol>gsiftp</protocol>
       <host>myhost.org</host>
       <port>2811</port>
    </ftpServer>
    <mapping>
       <jobPath>/pvfs/mount1/users</jobPath>
       <ftpPath>/pvfs/mount2/users</ftpPath>
    </mapping>
    <mapping>
       <jobPath>/pvfs/jobhome</jobPath>
       <ftpPath>/pvfs/ftphome</ftpPath>
    </mapping>
</map>

The file system mapping schema doc is here.

5. GRAM4 - RFT interaction

If a job description contains a fileStageIn, fileStageOut or a fileCleanUp element, or if globusrun-ws submits a job in streaming mode, GRAM4 uses RFT to perform transfer or delete requests. There are two ways GRAM4 can call RFT:

  1. WS calls
  2. Local Java calls, if RFT is located in the same container like GRAM4

By default GRAM4 in GT 4.2 is configured to use RFT located in the same GT4 container and to do local Java calls to call it. This improves performance in jobs with staging significantly. No additional security check is done in the call to RFT in that case; RFT trusts that a client had already been authenticated and authorized properly by GRAM4. We don't consider this to be a security risk because the client already passed security in the call to GRAM4, and because another security check is performed when RFT calls the GridFTP server(s) on behalf of the client using the delegated credentials.

If an admin wants to have different authorization for GRAM4 and RFT, or wants to configure GRAM4 to use RFT in a different container, or just does not want local Java calls, then they must be disabled by setting the parameter enableLocalInvocations to false in ${GLOBUS_LOCATION}/etc/globus_wsrf_gram/jndi-config.xml.

If ${GLOBUS_LOCATION}/setup/globus/setup-gram-service-common is used to configure GRAM4 to use RFT in a different container, then local Java calls will be disabled automatically.

6. Defining a default local resource manager

A client can submit a job without specifying the local resource manager (LRM) that should execute the job. By this it does not need to know which LRM is used by GRAM4: Condor, LSF, PBS, ...

To enable this there is a configuration parameter in the JNDI configuration that defines the default LRM if the client didn't specify it itself. By default it's set to Fork, but it can be changed by modifying the parameter defaultLocalResourceManager in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml. The following example illustrates PBS as default local resource manager:

<resource name="homeConfiguration"
    type="org.globus.exec.service.factory.FactoryHomeConfiguration">
    <resourceParams>
    ....
        <parameter>
            <name>
                defaultLocalResourceManager
            </name>
            <value>
                PBS
            </value>
        </parameter>
        ....
    </resourceParams>
</resource>

A client can specify a local resource manager though. In that case the explicitly specified LRM is used in job submission.

7. Disabling an already installed local resource manager adapter

When GRAM4 is initialized during startup of the GT container the JNDI configuration is checked for configured local resource manager adapters. If you want to disable an already installed local resource manager adapter you have to make sure that it's removed from the JNDI configuration. The following explains a way how this could be done:

Say, you installed support for PBS and want to disable PBS now. A listing of the GRAM4 related directories in $GLOBUS_LOCATION/etc will look like this:

[martin@osg-test1 ~]$ cd $GLOBUS_LOCATION/etc && ls | grep globus_wsrf_gram
globus_wsrf_gram
globus_wsrf_gram_Fork
globus_wsrf_gram_Multi
globus_wsrf_gram_PBS

All you have to do is to remove globus_wsrf_gram_PBS, or better, create an archive before removing it in case you want to enable PBS support at a later time again. After doing that the output of the above command should look like this:

[martin@osg-test1 ~]$ cd $GLOBUS_LOCATION/etc && ls | grep globus_wsrf_gram
globus_wsrf_gram
globus_wsrf_gram_Fork
globus_wsrf_gram_Multi
globus_wsrf_gram_PBS.tar.gz

After restarting the GT server users won't be able to submit jobs to PBS anymore.

8. Job lifetime configuration

For a general introduction see section Job Lifetime in the GRAM4 approach.

There are 2 parameters in GRAM4’s JNDI configuration in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml that have impact on lifetime of job resources:

maxJobLifetime

Max lifetime a client can specify in the initial job submission and in subsequent setTerminationTime calls. Default value is 1 year. A negative value means that there is no limit.

jobTTLAfterProcessing

Amount of time a job resource keeps on existing after the job has been fully processed and is in a final state Done, Failed, UserTerminateDone, UserTerminateFailed, and the client did not specify a job lifetime. Default value is 24h. A negative value means that the job resource does not expire.

Values are specified in seconds.

The parameters are exposed as resource properties of the factory resources to enable a client to query the values of the configuration parameters.

If a client does not specify any lifetime at all the job will run to completion in any event. After processing the lifetime will be set to (now + jobTTLAfterProcessing)

9. Registration with default WS MDS Index Service

9.1. Auto-registration

With a default GT 4.2.0 installation, the GRAM4 service is automatically registered with the default WS MDS Index Service running in the same container for monitoring and discovery purposes.

This is how auto-registration is configured:

There is a jndi resource defined in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml as follows :

 <resource name="mdsConfiguration"
    type="org.globus.wsrf.impl.servicegroup.client.MDSConfiguration">
    <resourceParams>
    <parameter> 
    <name>reg</name>
    <value>true</value>
    </parameter>
    <parameter> 
    <name>factory</name>
    <value>org.globus.wsrf.jndi.BeanFactory</value>
    </parameter>
    </resourceParams>
</resource>

To configure the automatic registration of GRAM4 to the default WS MDS Index Service, change the value of the parameter <reg> as follows:

  • true turns on auto-registration; this is the default in GT 4.2.0.
  • false turns off auto-registration.

9.1.1. Configuring resource properties

By default, the GLUECE: resource property (which contains GLUE data) is sent to the default Index Service:

You can configure which resource properties are sent in GRAM4's registration.xml file, $GLOBUS_LOCATION/etc/globus_wsrf_gram/registration.xml. The following is the relevant section of the file (as it is set by default):

<Content xsi:type="agg:AggregatorContent"
    xmlns:agg="http://mds.globus.org/aggregator/types">        
    <agg:AggregatorConfig xsi:type="agg:AggregatorConfig">
        <agg:GetResourcePropertyPollType
            xmlns:glue="http://mds.globus.org/glue/ce/1.1">
            <!-- Specifies that the index should refresh information
                    every 60000 milliseconds (once per minute) -->
            <agg:PollIntervalMillis>60000</agg:PollIntervalMillis>
            <!-- specifies the resource property that should be
                    aggregated, which in this case is the GLUE cluster
                    and scheduler information RP -->
            <agg:ResourcePropertyName>glue:GLUECE</agg:ResourcePropertyName>  
        </agg:GetResourcePropertyPollType>
    </agg:AggregatorConfig> 
    <agg:AggregatorData/>
</Content>

9.2. Manual registration

If a third party needs to register an GRAM4 service manually, see Registering with mds-servicegroup-add in the WS MDS Aggregator Framework documentation.

10. Configuring thread-pools

GRAM4 has two thread-pools that regulate internal processing.

One pool is responsible for processing all jobs in all states. The size of this pool limits the load GRAM4 produces in the GT container, and can be configured by modifying the parameter runQueueThreadCount in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml. The default value is 10.

The second pool defines how many threads process job events dispatched to GRAM4 by the Scheduler Event Generator (SEG) of each configured local resource manager. The default number of threads is 1, i.e. there's one thread that forwards events of job state changes for all jobs of all configured local resource managers. This should be fine for almost all scenarios. If a SEG however provides a lot of events, maybe even periodical events, or if the container persistence directory is located on a slow NFS disk, setting the parameter segEventProcessingThreadCount in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml to a higher value might make sense.