GT 5.0.2 GRAM5: Developer's Guide

New Features new since 5.0.1

Other Standard Supported Features

Removed Features

Tested platforms for GRAM5:

Protocol changes in GRAM since GT5.0.1 series:

GRAM depends on the following GT components:

No special security considerations exist at this time.

In the GRAM Client API, all functions that involve sending messages over the network have both blocking and nonblocking variants. These are useful in different programming situations.

The blocking variants, such as the globus_gram_client_job_request function require less application code, but will prevent subsequent instructions from executing until the request has been sent and the reply parsed. In a non-threaded environment, other callback functions registered with the Globus event driver may be invoked while the blocking function is running. In a threaded environment, other events may occur in other threads while the function is blocking, but the current thread will be blocked until the response is parsed.

The nonblocking variants, such as globus_gram_client_register_job_request require the application to include a callback function which will be called by the Globus event driver when the reply has been parsed. In a non-threaded environment, applications must poll the event driver using functions from the globus_poll or globus_cond_wait families of functions. In a threaded environment, the callback function may be invoked in another thread than the one calling the non-blocking function, even before the non-blocking function has returned. Application writers must be careful in using synchronization primitives such as globus_mutex_t and globus_cond_t when using non-blocking functions.

An application writer should use the non-blocking variants if the application will be submitting many jobs concurrently or requires custom network or security attributes. Using the non-blocking variants allows the Globus event driver to better schedule network I/O in these cases.

GRAM uses three types of contact strings to describe how to contact different services. These service contacts are:

GRAM clients and learn about a job's state in two ways: by registering for job state callbacks and by polling for status. These two methods have different performance characteristics and costs.

In order to receive job state callbacks, a client application must create an HTTPS listener using the globus_gram_client_callback_allow or globus_gram_client_info_callback_allow functions. A non-threaded application must then periodically call a function from either the globus_cond_wait or globus_poll families in order to process the job state callbacks. Additionally, the network must be configured to allow the GRAM job manager to send messages to the port that the client is listening on. This may be difficult if there is a firewall between the client and service.

The GRAM service initiates the job state callbacks, and thus they are usually sent very shortly after the job state changes, so clients can be notified about the state changes quickly.

In order to poll for job states, a client can call either the blocking or nonblocking variant of the globus_gram_client_job_status or globus_gram_client_job_status_with_info functions. These functions require that the network be configured to allow the client to contact the network port that the GRAM service is listening on (the Job Contact).

The client intiates these polling operations, so they are only as accurate as the polling frequence of the client. If the client polls very often, it will receive job state changes more quickly, at the risk of increasing the computing and network cost of both the client and service.

The GRAM5 protocols all use GSSAPIv2 abstractions to provide authentication and authorization. By default, GRAM uses an SSL-based GSSAPI for its security.

The client delegates a credential to the gatekeeper service after authentication, and the GRAM job manager service uses this delegated credential as both a job-specific credential and for subsequent communication with GRAM clients.

If a client or clients submit multiple jobs to a gatekeeper service, they will eventually all be handled by a single job manager process. This process will use whichever delegated credential will remain valid the longest for accepting new connections and connecting to clients to send job state callbacks. When a client delegates a new credential to a job, this credential may also be used as the job manager's credential for future connections.

GRAM5 jobs are described using the RSL language. The GRAM client API submits jobs using the string representation of the RSL, rather than the RSL parse tree. Clients can, if they need to modify of construct RSL at runtime, use the functions in the RSL API to do so.

This example shows how to use a gatekeeper "ping" request to determine if a service is running and if the client is authorized to contact it. It takes a gatekeeper service contact as its only command-line option. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int rc;

    if (argc != 2)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT\n", argv[0]);
        rc = 1;

        goto out;
    }

    printf("Pinging GRAM resource: %s\n", argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Ping the service passed as our first command-line option. If successful,
     * this function will return GLOBUS_SUCCESS, otherwise an integer
     * error code.
     */
    rc = globus_gram_client_ping(argv[1]);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to ping service at %s because %s (Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
    }
    else
    {
        printf("Ping successful\n");
    }
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_ping_example.c */

This example shows how to use the "version" command to determine what software version a gatekeeper service is running. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"
#include "globus_gram_protocol.h"

#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char *argv[])
{
    int rc;
    globus_hashtable_t extensions = NULL;
    globus_gram_protocol_extension_t * extension_value;

    if (argc != 2)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT\n", argv[0]);
        rc = 1;

        goto out;
    }

    printf("Checking version of GRAM resource: %s\n", argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Contact the service passed as our first command-line option and perform
     * a version check. If successful,
     * this function will return GLOBUS_SUCCESS, otherwise an integer
     * error code. Old versions of the job manager will return 
     * GLOBUS_GRAM_PROTOCOL_ERROR_HTTP_UNPACK_FAILED as they do not support
     * the version operation.
     */
    rc = globus_gram_client_get_jobmanager_version(argv[1], &extensions);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to get service version from %s because %s "
                "(Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
    }
    else
    {
        /* The version information is returned in the extensions hash table */
        extension_value = globus_hashtable_lookup(
                &extensions,
                "toolkit-version");

        if (extension_value == NULL)
        {
            printf("Unknown toolkit version\n");
        }
        else
        {
            printf("Toolkit Version: %s\n", extension_value->value);
        }

        extension_value = globus_hashtable_lookup(
                &extensions,
                "version");
        if (extension_value == NULL)
        {
            printf("Unknown package version\n");
        }
        else
        {
            printf("Package Version: %s\n", extension_value->value);
        }
        /* Free the extensions hash and its values */
        globus_gram_protocol_hash_destroy(&extensions);
    }

    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_version_example.c */

This example shows how to submit a job to a GRAM service. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int rc;
    char * job_contact = NULL;

    if (argc != 3)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT RSL\n", argv[0]);
        rc = 1;

        goto out;
    }

    printf("Submitting job to GRAM resource: %s\n", argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Submit the job request to the service passed as our first command-line
     * option. If successful, this function will return GLOBUS_SUCCESS,
     * otherwise an integer error code.
     */
    rc = globus_gram_client_job_request(
            argv[1], argv[2], 0, NULL, &job_contact);

    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to submit job to %s because %s (Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
        if (job_contact != NULL)
        {
            printf("Job Contact: %s\n", job_contact);
        }
    }
    else
    {
        /* Display job contact string */
        printf("Job submit successful: %s\n", job_contact);
    }

    if (job_contact != NULL)
    {
        free(job_contact);
    }
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_submit_example.c */

This example shows how to submit a job to a GRAM service and then wait until the job reaches the FAILED or DONE state. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

struct monitor_t
{
    globus_mutex_t mutex;
    globus_cond_t cond;
    globus_gram_protocol_job_state_t state;
};

/*
 * Job State Callback Function
 *
 * This function is called when the job manager sends job states.
 */
static
void
example_callback(void * callback_arg, char * job_contact, int state,
        int errorcode)
{
    struct monitor_t * monitor = callback_arg;

    globus_mutex_lock(&monitor->mutex);

    printf("Old Job State: %d\nNew Job State: %d\n", monitor->state, state);

    monitor->state = state;

    if (state == GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED ||
        state == GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE)
    {
        globus_cond_signal(&monitor->cond);
    }
    globus_mutex_unlock(&monitor->mutex);
}

int
main(int argc, char *argv[])
{
    int rc;
    char * callback_contact = NULL;
    char * job_contact = NULL;
    struct monitor_t monitor;

    if (argc != 3)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT RSL\n", argv[0]);
        rc = 1;

        goto out;
    }

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }

    rc = globus_mutex_init(&monitor.mutex, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing mutex\n");
        goto deactivate;
    }
    rc = globus_cond_init(&monitor.cond, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing condition variable\n");
        goto destroy_mutex;
    }

    monitor.state = GLOBUS_GRAM_PROTOCOL_JOB_STATE_UNSUBMITTED;

    globus_mutex_lock(&monitor.mutex);

    /*
     * Allow GRAM state change callbacks 
     */
    rc = globus_gram_client_callback_allow(
            example_callback, &monitor, &callback_contact);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error allowing callbacks because %s (Error %d)\n",
                globus_gram_client_error_string(rc), rc);
        goto destroy_cond;
    }
    /*
     * Submit the job request to the service passed as our first command-line
     * option. 
     */
    rc = globus_gram_client_job_request(
            argv[1], argv[2],
            GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED|
            GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE,
            callback_contact, &job_contact);

    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to submit job to %s because %s (Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
        /* Job submit failed. Short circuit the while loop below by setting
         * the job state to failed
         */
        monitor.state = GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED;
    }
    else
    {
        /* Display job contact string */
        printf("Job submit successful: %s\n", job_contact);
    }

    /* Wait for job state callback to let us know the job has completed */
    while (monitor.state != GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE &&
           monitor.state != GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED)
    {
        globus_cond_wait(&monitor.cond, &monitor.mutex);
    }
    rc = globus_gram_client_callback_disallow(callback_contact);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error disabling callbacks because %s (Error %d)\n",
                globus_gram_client_error_string(rc), rc);
    }
    globus_mutex_unlock(&monitor.mutex);

    if (job_contact != NULL)
    {
        free(job_contact);
    }

destroy_cond:
    globus_cond_destroy(&monitor.cond);
destroy_mutex:
    globus_mutex_destroy(&monitor.mutex);
deactivate:
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_submit_and_wait_example.c */

This example shows how to submit a job to a GRAM service and then wait until the job reaches the FAILED or DONE state. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int rc;
    int status = 0;
    int failure_code = 0;

    if (argc != 2)
    {
        fprintf(stderr, "Usage: %s JOB-CONTACT\n", argv[0]);
        rc = 1;

        goto out;
    }

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Check the job status of the job named by the first argument to
     * this program.
     */
    rc = globus_gram_client_job_status(argv[1], &status, &failure_code);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to check job status because %s (Error %d)\n",
                globus_gram_client_error_string(rc), rc);
    }
    else
    {
        switch (status)
        {
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_UNSUBMITTED:
                printf("Unsubmitted\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_STAGE_IN:
                printf("StageIn\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING:
                printf("Pending\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_ACTIVE:
                printf("Active\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_SUSPENDED:
                printf("Suspended\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_STAGE_OUT:
                printf("StageOut\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE:
                printf("Done\n");
                break;
            case GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED:
                printf("Failed (%d)\n", failure_code);
                break;
            default:
                printf("Unknown job state\n");
                break;
        }
    }
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_poll_example.c */

This example shows how to cancel a job being run by a GRAM service. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int rc;

    if (argc != 2)
    {
        fprintf(stderr, "Usage: %s JOB-CONTACT\n", argv[0]);
        rc = 1;

        goto out;
    }

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Cancel the job named by the first argument to
     * this program.
     */
    rc = globus_gram_client_job_cancel(argv[1]);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to cancel job because %s (Error %d)\n",
                globus_gram_client_error_string(rc), rc);
    }
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_cancel_example.c */

This example shows how to refresh a GRAM job's credential after the job has been submitted by some other means. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

int
main(int argc, char *argv[])
{
    int rc;

    if (argc != 2)
    {
        fprintf(stderr, "Usage: %s JOB-CONTACT\n", argv[0]);
        rc = 1;

        goto out;
    }

    printf("Refreshing Credential for GRAM Job: %s\n", argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }
    /*
     * Refresh the credential of the job running at the contact named
     * by the first command-line argument to this program. We'll use the
     * process's default credential by passing in GSS_C_NO_CREDENTIAL.
     */
    rc = globus_gram_client_job_refresh_credentials(
            argv[1], GSS_C_NO_CREDENTIAL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to refresh credential for job %s because %s (Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
    }
    else
    {
        printf("Refresh successful\n");
    }
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_refresh_example.c */

This example shows how to submit a series of GRAM jobs using the non-blocking function globus_gram_client_register_job_request and wait until all submissions have completed. This example throttles the number of concurrent job submissions to reduce the load on the service node. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

struct monitor_t
{
    globus_mutex_t mutex;
    globus_cond_t cond;
    int submit_pending;
    int successful_submits;
};

#define CONCURRENT_SUBMITS 5

static
void
example_submit_callback(
    void * user_callback_arg,
    globus_gram_protocol_error_t operation_failure_code,
    const char * job_contact,
    globus_gram_protocol_job_state_t job_state,
    globus_gram_protocol_error_t job_failure_code)
{
    struct monitor_t * monitor = user_callback_arg;

    globus_mutex_lock(&monitor->mutex);
    monitor->submit_pending--;
    if (monitor->submit_pending < CONCURRENT_SUBMITS)
    {
        globus_cond_signal(&monitor->cond);
    }
    printf("Submitted job %s\n",
            job_contact ? job_contact : "UNKNOWN");
    if (operation_failure_code == GLOBUS_SUCCESS)
    {
        monitor->successful_submits++;
    }
    else
    {
        printf("submit failed because %s (Error %d)\n",
                globus_gram_client_error_string(operation_failure_code),
                operation_failure_code);
    }
    globus_mutex_unlock(&monitor->mutex);
}

int
main(int argc, char *argv[])
{
    int rc;
    int i;
    struct monitor_t monitor;

    if (argc < 3)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT RSL-SPEC...\n",
                argv[0]);
        rc = 1;

        goto out;
    }

    printf("Submiting %d jobs to %s\n", argc-2, argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }

    rc = globus_mutex_init(&monitor.mutex, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing mutex %d\n", rc);

        goto deactivate;
    }

    rc = globus_cond_init(&monitor.cond, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing condition variable %d\n", rc);

        goto destroy_mutex;
    }
    monitor.submit_pending = 0;

    /* Submits jobs from argv[2] until end of the argv array. At most
     * CONCURRENT_SUBMITS will be pending at any given time.
     */
    globus_mutex_lock(&monitor.mutex);
    for (i = 2; i < argc; i++)
    {
        /* This throttles the number of concurrent job submissions */
        while (monitor.submit_pending >= CONCURRENT_SUBMITS)
        {
            globus_cond_wait(&monitor.cond, &monitor.mutex);
        }

        /* When the job has been submitted, the example_submit_callback
         * will be called, either from another thread or from a 
         * globus_cond_wait in a nonthreaded build
         */
        rc = globus_gram_client_register_job_request(
                argv[1], argv[i], 0, NULL, NULL, example_submit_callback,
                &monitor);
        if (rc != GLOBUS_SUCCESS)
        {
            fprintf(stderr, "Unable to submit job %s because %s (Error %d)\n",
                    argv[i], globus_gram_client_error_string(rc), rc);
        }
        else
        {
            monitor.submit_pending++;
        }
    }

    /* Wait until the example_submit_callback function has been called for
     * each job submission
     */
    while (monitor.submit_pending > 0)
    {
        globus_cond_wait(&monitor.cond, &monitor.mutex);
    }
    globus_mutex_unlock(&monitor.mutex);

    printf("Submitted %d jobs (%d successfully)\n",
            argc-2, monitor.successful_submits);

    globus_cond_destroy(&monitor.cond);
destroy_mutex:
    globus_mutex_destroy(&monitor.mutex);
deactivate:
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_nonblocking_submit_example.c */

This example shows how to submit a job and delegate a full credential to the job. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module functions
 * and GRAM Client API functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"

#include <stdio.h>

struct monitor_t
{
    globus_mutex_t mutex;
    globus_cond_t cond;
    globus_bool_t done;
};

static
void
example_submit_callback(
    void * user_callback_arg,
    globus_gram_protocol_error_t operation_failure_code,
    const char * job_contact,
    globus_gram_protocol_job_state_t job_state,
    globus_gram_protocol_error_t job_failure_code)
{
    struct monitor_t * monitor = user_callback_arg;

    globus_mutex_lock(&monitor->mutex);
    monitor->done = GLOBUS_TRUE;
    globus_cond_signal(&monitor->cond);
    if (operation_failure_code == GLOBUS_SUCCESS)
    {
        printf("Submitted job %s\n",
            job_contact ? job_contact : "UNKNOWN");
    }
    else
    {
        printf("submit failed because %s (Error %d)\n",
                globus_gram_client_error_string(operation_failure_code),
                operation_failure_code);
    }
    globus_mutex_unlock(&monitor->mutex);
}

int
main(int argc, char *argv[])
{
    int rc;
    globus_gram_client_attr_t attr;
    struct monitor_t monitor;

    if (argc < 3)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT RSL-SPEC...\n",
                argv[0]);
        rc = 1;

        goto out;
    }

    printf("Submiting job to %s with full proxy\n", argv[1]);

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }

    rc = globus_mutex_init(&monitor.mutex, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing mutex %d\n", rc);

        goto deactivate;
    }

    rc = globus_cond_init(&monitor.cond, NULL);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error initializing condition variable %d\n", rc);

        goto destroy_mutex;
    }
    monitor.done = GLOBUS_FALSE;

    /* Initialize attribute so that we can set the delegation attribute */
    rc = globus_gram_client_attr_init(&attr);

    /* Set the proxy attribute */
    rc = globus_gram_client_attr_set_delegation_mode(
        attr,
        GLOBUS_IO_SECURE_DELEGATION_MODE_FULL_PROXY);

    /* Submit the job rsl from argv[2]
     */
    globus_mutex_lock(&monitor.mutex);
    /* When the job has been submitted, the example_submit_callback
     * will be called, either from another thread or from a 
     * globus_cond_wait in a nonthreaded build
     */
    rc = globus_gram_client_register_job_request(
            argv[1], argv[2], 0, NULL, attr, example_submit_callback,
            &monitor);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to submit job %s because %s (Error %d)\n",
                argv[2], globus_gram_client_error_string(rc), rc);
    }

    /* Wait until the example_submit_callback function has been called for
     * the job submission
     */
    while (!monitor.done)
    {
        globus_cond_wait(&monitor.cond, &monitor.mutex);
    }
    globus_mutex_unlock(&monitor.mutex);

    globus_cond_destroy(&monitor.cond);
destroy_mutex:
    globus_mutex_destroy(&monitor.mutex);
deactivate:
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_attr_example.c */

This example shows how to programmatically add environment variable definitions to an RSL prior to submitting a job. The source to this example can be downloaded.

/*
 * These headers contain declarations for the globus_module,
 * the GRAM Client, RSL, and protocol functions
 */
#include "globus_common.h"
#include "globus_gram_client.h"
#include "globus_rsl.h"
#include "globus_gram_protocol.h"

#include <stdio.h>
#include <strings.h>

static
int
example_rsl_attribute_match(void * datum, void * arg)
{
    const char * relation_attribute = globus_rsl_relation_get_attribute(datum);
    const char * attribute = arg;

    /* RSL attributes are case-insensitive */
    return (relation_attribute &&
            strcasecmp(relation_attribute, attribute) == 0);
}

int
main(int argc, char *argv[])
{
    int rc;
    globus_rsl_t *rsl, *environment_relation;
    globus_rsl_value_t *new_env_pair = NULL;
    globus_list_t *environment_relation_node;
    char * rsl_string;
    char * job_contact;

    if (argc != 3)
    {
        fprintf(stderr, "Usage: %s RESOURCE-MANAGER-CONTACT RSL\n", argv[0]);
        rc = 1;

        goto out;
    }

    /*
     * Always activate the GLOBUS_GRAM_CLIENT_MODULE prior to using any
     * functions from the GRAM Client API or behavior is undefined.
     */
    rc = globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Error activating %s because %s (Error %d)\n",
                GLOBUS_GRAM_CLIENT_MODULE->module_name,
                globus_gram_client_error_string(rc),
                rc);
        goto out;
    }

    /* Parse the RSL string into a syntax tree */
    rsl = globus_rsl_parse(argv[2]);
    if (rsl == NULL)
    {
        rc = 1;
        fprintf(stderr, "Error parsing RSL string\n");
        goto deactivate;
    }

    /* Create the new environment variable pair that we'll insert
     * into the RSL. We'll start by making an empty sequence
     */
    new_env_pair = globus_rsl_value_make_sequence(NULL);
    if (new_env_pair == NULL)
    {
        fprintf(stderr, "Error creating value sequence\n");
        rc = 1;

        goto free_rsl;
    }
    /* Then insert the name-value pair in reverse order */
    rc = globus_list_insert(
            globus_rsl_value_sequence_get_list_ref(new_env_pair),
            globus_rsl_value_make_literal(
                strdup("itsvalue")));
    if (rc != GLOBUS_SUCCESS)
    {
        goto free_env_pair;
    }

    rc = globus_list_insert(
            globus_rsl_value_sequence_get_list_ref(new_env_pair),
            globus_rsl_value_make_literal(
                strdup("EXAMPLE_ENVIRONMENT_VARIABLE")));
    if (rc != GLOBUS_SUCCESS)
    {
        goto free_env_pair;
    }
    /* Now, check to see if the RSL already contains an environment 
     * attribute.
     */
    environment_relation_node = globus_list_search_pred(
            globus_rsl_boolean_get_operand_list(rsl),
            example_rsl_attribute_match,
            GLOBUS_GRAM_PROTOCOL_ENVIRONMENT_PARAM);

    if (environment_relation_node == NULL)
    {
        /* Not present yet, create a new relation and insert it into
         * the RSL.
         */
        environment_relation = globus_rsl_make_relation(
                GLOBUS_RSL_EQ,
                strdup(GLOBUS_GRAM_PROTOCOL_ENVIRONMENT_PARAM),
                globus_rsl_value_make_sequence(NULL));
        rc = globus_list_insert(
                globus_rsl_boolean_get_operand_list_ref(rsl),
                environment_relation);
        if (rc != GLOBUS_SUCCESS)
        {
            globus_rsl_free_recursive(environment_relation);
            goto free_env_pair;
        }
    }
    else
    {
        /* Pull the environment relation out of the node returned from the
         * search function
         */
        environment_relation = globus_list_first(environment_relation_node);
    }

    /* Add the new environment binding to the value sequence associated with
     * the environment relation
     */
    rc = globus_list_insert(
        globus_rsl_value_sequence_get_list_ref(
                globus_rsl_relation_get_value_sequence(environment_relation)),
        new_env_pair);
    if (rc != GLOBUS_SUCCESS)
    {
        goto free_env_pair;
    }
    new_env_pair = NULL;

    /* Convert the RSL parse tree to a string */
    rsl_string = globus_rsl_unparse(rsl);

    /*
     * Submit the augmented RSL to the service passed as our first command-line
     * option. If successful, this function will return GLOBUS_SUCCESS,
     * otherwise an integer error code.
     */
    rc = globus_gram_client_job_request(
            argv[1],
            rsl_string,
            0,
            NULL,
            &job_contact);
    if (rc != GLOBUS_SUCCESS)
    {
        fprintf(stderr, "Unable to submit job to %s because %s (Error %d)\n",
                argv[1], globus_gram_client_error_string(rc), rc);
    }
    else
    {
        printf("Job submitted successfully: %s\n", job_contact);
    }

    free(rsl_string);

    if (job_contact)
    {
        free(job_contact);
    }
free_env_pair:
    if (new_env_pair != NULL)
    {
        globus_rsl_value_free_recursive(new_env_pair);
    }
free_rsl:
    globus_rsl_free_recursive(rsl);
deactivate:
    /*
     * Deactivating the module allows it to free memory and close network
     * connections.
     */
    rc = globus_module_deactivate(GLOBUS_GRAM_CLIENT_MODULE);
out:
    return rc;
}
/* End of gram_rsl_example.c */

The core syntax of the RSL syntax is the relation. Relations associate an attribute name with a value, eg the relation executable=a.out provides the name of an executable in a resource request. There are two generative syntactic structures in the RSL that are used to build more complicated resource descriptions out of the basic relations: compound requests and value sequences. In addition, the RSL syntax includes a facility to both introduce and dereference string substitution variables.

The simplest form of compound request, utilized by all resource management components, is the conjunct-request. The conjuct-request expresses a conjunction of simple relations or compound requests (like a boolean AND). The most common conjunct-request in Globus RSL strings is the combination of multiple relations such as executable name, node count, executable arguments, and output files for a basic GRAM job request. Similarly, the core RSL syntax includes a disjunct-request form to represent disjunctive relations (like a boolean OR). Currently, however, no resource management component utilizes the disjunct-request form.

The last form of compound request is the multi-request. The multi-request expresses multiple parallel resources that make up a resource description. The multi-request form differs from the conjunction and disjunction in two ways: multi-requests introduce new variable scope, meaning variables defined in one clause of a multi-request are not visible to the other clauses, and multi-requests introduce a non-reducible hierarchy to the resource description. Whereas relations within a conjunct-request can be thought of as constraints on the resource being described, the subclauses of a multi-request are best thought of as individual resource descriptions that together constitute an abstract resource collection; the same attributes may be constrained in different ways in each subclause without causing a logical contradiction. An example of a contradiction would be to constrain the executable attribute to be two conflicting values within a conjunction. Currently, however, no resource management component utilizes the disjunct-request form.

The simplest form of value in the RSL syntax is the string literal. When explicitly quoted, literals can contain any character, and many common literals that don't contain special characters can appear without quotes. Values can also be variable references, in which case the variable reference is in essence replaced with the string value defined for that variable. RSL descriptions can also express string-concatenation of values, especially useful to construct long strings out of several variable references. String concatenation is supported with both an explicit concatenation operator and implicit concatenation for many idiomatic constructions involving variable references and literals.

In addition to the simple value forms given above, the RSL syntax includes the value sequence to express ordered sets of values. The value sequence syntax is used primarily for defining variables and for providing the argument list for a program.

Each RSL string consists of a sequence of RSL tokens, whitespace, and comments. The RSL tokens are either special syntax or regular unquoted literals, where special syntax contains one or more of the following listed special characters and unquoted literals are made of sequences of characters excluding the special characters.

The complete set of special characters that cannot appear as part of an unquoted literal is:

These characters can only be used for the special syntactic forms described in the section and in the section or as within quoted literals.

Quoted literals are introduced with the " (double quote) or ' (single quote/apostrophe) and consist of all the characters up to (but not including) the next solo double or single quote, respectively. To escape a quote character within a quoted literal, the appearance of the quote character twice in a row is converted to a single instance of the character and the literal continues until the next solo quote character. For any quoted literal, there is only one possible escape sequence, eg within a literal delimited by the single quote character only the single quote character uses the escape notation and the double quote character can appear without escape.

Quoted literals can also be introduced with an alternate user delimiter notation. User delimited literals are introduced with the ^ (carat) character followed immediately by a user-provided delimiter; the literal consists of all the characters after the user's delimiter up to (but not including) the next solo instance of the delimiter. The delimiter itself may be escaped within the literal by providing two instances in a row, just as the regular quote delimiters are escaped in regular quoted literals.

RSL string comments use a notation similar to comments in the C programming language. Comments are introduced by the prefix (*. Comments continue to the first terminating suffix *) and cannot be nested. Comments are stripped from the RSL string during processing and are syntactically equivalent to whitespace.

arguments = "Hello. Welcome to ""The Grid"""

arguments = 'Hello. Welcome to "The Grid'

arguments = ^!Hello. Welcome to "The Grid"!

RSL strings can introduce and reference string variables. String substitution variables are defined in a special relation using the rsl_substitution attribute, and the definitions affect variable references made in the same conjunct-request (or disjunct-request), as well as references made within any multi-request nested inside one of the clauses of the conjunction (or disjunction). Each multi-request introduces a new variable scope for each subrequest, and variable definitions do not escape the closest enclosing scope.

Within any given scope, variable definitions are processed left-to-right in the resource description. Outermost scopes are processed before inner scopes, and the definitions in inner scopes augment the inherited definitions with new and/or updated variable definitions.

Variable definitions and variable references are processed in a single pass, with each definition updating the environment prior to processing the next definition. The value provided in a variable definition may include a reference to a previously-defined variable. References to variables that are not yet provided with definitions in the standard RSL variable processing order are replaced with an empty literal string.

The RSL syntax is extensible because it defines structure without too many keywords. Each Globus resource management component introduces additional attributes to the set recognized by RSL-aware components, so it is difficult to provide a complete listing of attributes which might appear in a resource description. Resource management components are designed to utilize attributes they recognize and pass unrecongnized relations through unchanged. This allows powerful compositions of different resource management functions.

The following listing summarizes the attribute names utilized by existing resource management components in the standard Globus release. Please see the individual component documentation for discussion of the attribute semantics.

The following are some simple example RSL strings to illustrate idiomatic usage with existing tools and to make concrete some of the more interesting cases of tokenization, concatenation, and variable semantics. These are meant to illustrate the use of the RSL notation without much regard for the specific details of a particular resource management component.

Typical GRAM5 resource descriptions contain at least a few relations in a conjunction:

(* this is a comment *)
& (executable = a.out (* <-- that is an unquoted literal *))
  (directory  = /home/nobody )
  (arguments  = arg1 "arg 2")
  (count = 1)

& (rsl_substitution  = (TOPDIR  "/home/nobody")
                       (DATADIR $(TOPDIR)"/data")
                       (EXECDIR $(TOPDIR)/bin) )
  (executable = $(EXECDIR)/a.out
        (* ^-- implicit concatenation *))
  (directory  = $(TOPDIR) )
  (arguments  = $(DATADIR)/file1 
        (* ^-- implicit concatenation *)
                $(DATADIR) # /file2
        (* ^-- explicit concatenation *)
                '$(FOO)'            (* <-- a quoted literal *))
  (environment = (DATADIR $(DATADIR)))
  (count = 1)

& (rsl_substitution  = (TOPDIR "/home/nobody")
                       (DATADIR "/home/nobody/data") 
                       (EXECDIR "/home/nobody/bin") )
  (executable = "/home/nobody/bin/a.out" )
  (directory  = "/home/nobody" )
  (arguments  = "/home/nobody/data/file1"
                "/home/nobody/data/file2"
                "$(FOO)" )
  (environment = (DATADIR "/home/nobody/data"))
  (count = 1)

The following is a modified BNF grammar for the Resource Specification Language. Lexical rules are provided for the implicit concatenation sequences in the form of conventional regular expressions; for the implicit-concat non-terminal rules, whitespace is not allowed between juxtaposed non-terminals. Grammar comments are provided in square brackets in a column to the right of the productions, eg [comment] to help relate productions in the grammar to the terminology used in the above discussion.

Regular expressions are provided for the terminal class string-literal and for RSL comments. These regular expression make use of a common inverted character-class notation, as popularized by the various lex tools. Comments are syntactically equivalent to whitespace and can only appear where the comment prefix cannot be mistaken for the trailing part of a multi-character unquoted literal.

The first thing to determine when debugging unexpected failures is to determine whether the gatekeeper service is running, reachable from the client, and properly configured.

First, determine that the gatekeeper is running by using a tool such as telnet to connect to the TCP/IP port that the gatekeeper is listening on. From the GRAM service node, using a default configuration, use a command like:

% telnet localhost 2119
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'

An error message like the following indicates that the gatekeeper service is not starting:

telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host

If the telnet command exits immediately, then the gatekeeper service is being started but not running. Check the gatekeeper log (by default $GLOBUS_LOCATION/var/globus-gatekeeper.log) to see if there is an error message. A common error is having a missing library path environment variable in the gatekeeper's environment or having a malformed configuration file. See the globus-gatekeeper for information on the configuration options.

The next recommended diagnostic is to run the same telnet command from the machine which is acting as the GRAM client if it is distinct from the GRAM service node. Be sure to replace localhost with the actual host name of the GRAM service. Again, check for log entries in the case of immediate exit or refused connection. If the connection does not work, then there may be some network connectivity or firewall issues preventing access.

Next use a tool like globusrun to diagnose whether the client is authorized to contact the gatekeeper service. This is done by using the -a command-line option. For example:

% globusrun -a -r grid.example.org

GRAM Authentication test successful

If you do not get the success message above, then check the gatekeeper log to see if there is a diagnostic message. A common problem is that the identity of the client is not in the grid mapfile used by the gatekeeper.

The next test is to use the -dryrun option to globusrun to verify that the job manager service is properly configured. To do so, try the following:

% globusrun -dryrun -r grid.example.org "&(executable=/bin/sh)"
globus_gram_client_callback_allow successful
Dryrun successful

If you do not get the success message above, first check the error number in the GRAM5 Error codes table to determine how to proceed. If the result is unclear, check the job manager log (default $HOME/gram_DATE.log) to see if there are any further details of the error.

The final test is to submit a test job to the GRAM5 service and wait for it to terminate, such as this example shows:

% globus-job-run grid.example.org /bin/sh -c 'echo "hello, grid"'
hello, grid

If the process appears to hang, it might be that the job manager is unable to send state callbacks to the client. Check that there are no firewalls or network issues that would prevent the job manager process from connecting from the GRAM service node to the client node.

The methods described in this section are intended for debugging problems in the GRAM code, not in the user environment.

% $GLOBUS_LOCATION/libexec/globus-job-manager \
    -conf $GLOBUS_LOCATION/etc/globus-job-manager.conf -type fork

In case you run into problems you can do the following

The GRAM Protocol is used to handle communication between the Gatekeeper, Job Manager, and GRAM Clients. The protocol is based on a subset of the HTTP/1.1 protocol, with a small set of message types and responses sent as the body of the HTTP requests and responses. This document describes GRAM Protocol version 2 as used by GRAM5. This is compatible with with the GRAM Protocol parsers in GRAM2 with extensions.

rsl: &( executable = "/bin/echo" )
 ( arguments = "hello" )

rsl: "&( executable = \"bin/echo\" )
  (arguments = \"hello\" )"

POST ping/job-manager-name HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version

POST job-manager-name[@user-name] HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
job-state-mask: mask
callback-url: callback-contact
rsl: rsl-description

POST job-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size
protocol-version: version

"status"

POST job-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
"register mask callback-contact"

POST job-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
"unregister callback-contact"
        

POST job-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
"cancel"
        

POST job-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
"signal"

POST callback-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
job-manager-url: job-contact
status: status-code
failure-code: failure-code
    

POST callback-contact HTTP/1.1
Host: host-name
Content-Type: application/x-globus-gram
Content-Length: message-size

protocol-version: version
"renew"

1.4. Job State Model

As the GRAM service processes a job, the job undergoes a series of state transitions. These states and their meanings follow:

Table 10.1. GRAM Job States

State	Meaning
GLOBUS_GRAM_PROTOCOL_JOB_STATE_UNSUBMITTED	Initial job state
GLOBUS_GRAM_PROTOCOL_JOB_STATE_STAGE_IN	Job staging in progress
GLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING	Job submitted to LRM, awaiting execution
GLOBUS_GRAM_PROTOCOL_JOB_STATE_ACTIVE	Job executing
GLOBUS_GRAM_PROTOCOL_JOB_STATE_SUSPENDED	Job made progress executing but is now suspended
GLOBUS_GRAM_PROTOCOL_JOB_STATE_STAGE_OUT	Job staging in progress after job completed
GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE	Job completed successfully
GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED	Job was canceled or failed

Figure 10.1. GRAM State Transitions