The Globus Toolkit uses an asynchronous event model. Details of this model are contained in the remainder of this text but it will be helpful to take a few examples of other popular models.

Applications existing in event heavy environments, such as graphical user interfaces (GUIs), IO, or inter-process signaling, must implement some event model. Events are characterized by changes in the environment at an undetermined time. There are several different popular models used to handle such events. We provide examples of them here, and then describe in detail the asynchronous event model used by the Globus Toolkit.

main()
{
    while(!done)
    {
        ~ other processing ~
        data = ReadData();
        ~ process event ~
    }
}
    

main()
{
    while(!done)
    {
        if(EventIsReady())
        {
            ~ process event ~
        }
        else
        {
            ~other processing ~
        }
    }
}
    

event_handler()
{
    process event
    register_next_event();
}

main()
{
    ~ other processing ~
     resiter_event()
      ~ other processing ~
    while(!done)
    {
        wait_for_events();
    }
}
    

The heart of the Globus event model is the callback library. This API provides a user with functionality for asynchronous time events. In order to use the API for events, the user must implement a function (the callback) that is called when the event has occurred and processes it.

There are two fundamental functions that explain the API:

globus_result_t
globus_callback_register_oneshot(
    globus_callback_handle_t *          callback_handle,
    const globus_reltime_t *            delay_time,
    globus_callback_func_t              callback_func,
    void *                              callback_user_args);

globus_result_t
globus_poll();

The first function is fairly clear. It registers the callback callback_func with the system that will be called once the time specified by delay_time has expired.

The more interesting of the two is globus_poll(). Semantically this function is used to briefly turn control over to the Globus event system for processing. What this means is that globus_poll() must be called often enough for the Globus event system to function. This is recognized as a rather ambiguous statement. Therefore, a look at what happens with globus_poll() should assist in explanation. In threaded builds of Globus this globus_poll() simply results in a call to thread_yield() where control can be switched to a background thread dedicated to event processing. In non-threaded builds, a list of events is maintained by the system. A call to globus_poll() finds ready events in the list and dispatches the associated callback to the user within the same call stack.

In Example 1 a use of these two functions is displayed. The function user_callback is registered for execution after 1 second has elapsed.

In a non-threaded build, there is a single thread of execution. In the main loop, the call to globus_poll() invokes the Globus event process code. The code checks internal data structures for any ready events. If found, the user callbacks associated with the events will be called in the same call stack.

In a threaded build a user would see two threads (possibly more, but for the sake of clarity two will be used): the main thread that is executing the loop in main() and an internal Globus thread that is handling polling of events. The Globus thread is created when the user calls globus_module_activate(GLOBUS_COMMON_MODULE). This function must be called before any API function in the the globus_common package can be used. This is another common theme in Globus: all modules must be activated before use and deactivated when finished. The event thread polls all events and as they become ready the functions associated with them are dispatched.

Another important concept to note in this API is the use of the void * user_arg parameter. This is a simple but important part of the model. On registration of an event, a user can pass in a void pointer and this pointer will be threaded through to their event callback. The pointer can point to any bit of memory the user likes. Typically it points to some structure that allows the user to maintain state throughout a series of event callbacks. This memory is completely managed by the user, so if the memory is used in the event callback the user needs to be careful to not free the memory until the callback occurs. For a more complicated example of this see Example 2.

The first thing to look at in understanding the Globus event model is the thread abstraction layer. Globus can be built in a variety of ways with regard to the underlying thread system. It can be built with pthreads, win32 threads, or non threaded depending on the user's preferences and the available packages on the system. All builds present the same API. This thread API is very much akin to pthreads. If the reader is not familiar with pthreads, we recommend reading the pthread manual. The more notable API interface is presented below:

int
globus_thread_create(
    globus_thread_t *           thread,
    globus_threadattr_t *       attr,
    globus_thread_func_t        func,
    void *              user_arg);

int
globus_mutex_lock(
    globus_mutex_t *            mutex);

int
globus_mutex_unlock(
    globus_mutex_t *            mutex);

int
globus_cond_wait(
    globus_cond_t *         cond,
    globus_mutex_t *            mutex);

int
globus_cond_signal(
    globus_cond_t *         cond);

It is important to note that this is not a complete set of necessary functions to properly use the threaded API. However, for the purposes of this text, they will serve for an explanation.

In most cases the thread layer abstraction is a very thin pass through to the underlying thread package.

The notable exception is the non-threaded build. The Globus Toolkit has created a non-threaded, semantically equivalent implementation of all the functions described above (and of most in the pthreads API) with the exception of globus_thread_create(). In the non-threaded case this is a no-op. However the model of asynchronous programming used in the Globus Toolkit, globus_thread_create() is rarely needed or used.

In the Globus model, the callback code and the thread abstraction are coupled. Example 2 shows how this coupling works:

In the threaded build, globus_cond_wait() and globus_cond_signal() are simple passes through to the underlying thread packages, and as described previously, a background thread delivers the event.

In the non-threaded build, globus_cond_wait() will call globus_poll() and the non-threaded polling code takes over. For this reason, it is often not necessary to call globus_poll() in non-threaded builds. globus_cond_wait() tends to be used often enough to satisfy the needs of the event system.

In many ways, the asynchronous programming model is the most difficult of the three presented. The blocking model is clearly the easiest, because everything happens in-line, and when the event function (like a read or a write) returns, the event has completed and all data is available. Events in this model are treated just like any other function call and are therefore easily dealt with by programmers with modest logic skills.

The non-blocking model is a bit more complicated than blocking, but not much. The only twist is that a user must check to see if the event completed and, if so, how much of it completed. This still allows for in-line processing; it only requires an additional if statement. Even when event polling is multiplexed (for example, posix select) the processing is still inline. The user must add some branches to determine what event is ready and then process it. The most difficult challenge of the non-blocking model is making use of the idle time when no events are ready.

In both non-blocking and blocking, the user has easy, in-line control over when an event is processed. If there is any logic that must occur before the event, the user simply needs to complete that processing before calling either the blocking function or the non-blocking function which checks for ready events. The asynchronous model removes this luxury. In the asynchronous model events can occur at any time. This can complicate the logic of keeping critical sections of code safe. Further complication is caused by the fact that they come in via their own handlers. This removes the luxury of maintaining state on the local stack. Instead all state must be packed into heap allocated structures which are passed to the callbacks via void * pointers (see the monitor structure in Example 2).

The upside to the asynchronous model is that it forces cleaner, more well thought out code. The non-blocking model does not scale well. As more events are managed, the event processing code becomes unmanageable, typically resulting in a single function that is far too long and far too interdependent for practical maintenance. Since users can use local variables, the tendency is to use many different flags to control state instead of a clean, well thought out state machine. This is especially true with software that evolves over time, growing in complexity.

In contrast the asynchronous model scales very well. Every event has a clean separation of being scoped to a user handler function. All shared states among events must be encapsulated into a data structure. A side effect of these two characteristics is that it is easier for a user to define and follow a state machine then it is to create spaghetti logic based on many flags.

Example 3 shows a proper use of the asynchronous model. This example simulates the game of Craps. Craps is a dice game, the rules of which can be found with a simple web search, but the following state diagram should explain the rules well enough for this example.

Figure 4.1. State Diagram

Example 3 follows this state diagram. In the example rolls of the dice are considered events. For the sake of simplicity the example only uses a one shot event and then gets the data by calling random(); If this were a real world event, the values for the dice would come in as part of the event function. Notice how each time the event occurs the state is checked and, if needed, advanced to the next state. In the main function the program waits until the state machine comes to the final stage, where it signals the wait and allows for the program to end.

The Globus Toolkit is middleware for the grid. Because grid infrastructure often depends heavily on both push and pull notifications (remote events), the callback style event handling model the Globus Toolkit provides is essential. It allows entire APIs within the toolkit to be designed with asynchronous functions that use the event handling model. Once an API provides that asynchronous functionality (such as XIO), software that builds on top of it can leverage this functionality. This eases the burden of the application programmer, as they need only to implement a callback function to handle possibly many notification events efficiently, instead of stopping execution until one is received, or managing multiple threads.

In the Globus Toolkit, because of the thread abstraction it provides, threads are managed by the underlying code base, so that the developer can be ignorant of using threads but still be able to get their benefits, simply by specifying a compile time switch. Overall, this flexibility is quite powerful, which is why we encourage the use of this model when designing and developing your own software components using the Globus Toolkit.