Table of Contents
The GridWay Metascheduler enables large-scale, reliable and efficient sharing of computing resources (clusters, computing farms, servers, supercomputers...), managed by different LRM (Local Resource Management) systems, such as PBS, SGE, LSF, Condor..., within a single organization (enterprise grid) or scattered across several administrative domains (partner or supply-chain grid).
- Advanced Scheduling Capabilities
GridWay implements several state-of-the-art Grid-aware scheduling policies, comprising job prioritization policies (fixed priority, urgency, share, deadline and waiting-time) and resource prioritization policies (fixed priority, usage, failure and rank).
These policies are combined with:
Adaptive Scheduling, to periodically adapt the schedule considering applications' demands and Grid resource characteristics.
Adaptive Execution to migrate running applications in terms of resource availability, capacity or cost, and new application requirements or preferences.
- Transparent Grid Access
GridWay interfaces infrastructures with different middleware stacks. With GridWay, users can access heterogeneous resources in a transparent way. For example, it can access resources configured with both GRAM pre web services and GRAM web services. It also permits the use of different grids with different software stacks, for example, Teragrid with Globus and EGEE with gLite.
- Flexible Deployment Capabilities
GridWay supports multiple-user operation mode, and does not require additional middleware installation (apart from standard Globus services). Globus installation is not required in each end-user system.
GridWay allows different Grid deployment strategies, like Enterprise Grids, Partner Grids or Utility Grids.
- Different Application Profiles
GridWay executes different Grid application profiles:
Array (Bulk) jobs, for parameter sweep applications
DAG Workflows
Single-site MPI applications
- Fault Detection and Recovery
GridWay is able to detect several problems that can occur when executing a remote job. It also implements mechanisms that make the execution more reliable. It can detect a remote system crash, a job failure (via the job exit code) or even a network disconnection (using the polling mechanism) and migrate the problematic job to another resource.
GridWay also performs periodic saves of its state in order to recover from local failure.
- Reporting and Accounting
GridWay provides detailed statistics of Grid usage. In this way, the Grid administrator can properly plan usage policies and forecast workload. In addition, these statistics can be used by the scheduler to predict (per user) Grid resource response time.
- Standard Compliance
GridWay is an open-source project, flexible and completely based on standards to leverage its usability and interoperability. For example, users can describe their jobs using JSDL. Similarly, programmers can build grid enabled applications using the DRMAA standard.
- User Interface
GridWay provides end-users with a familiar environment similar to that found on classical LRM systems. So GridWay CLI eases the adoption of Grid technologies.
GridWay version shipped with GT4.2.1 is based on GridWay 5.4, a new stable version with increased functionality, including support to access the NorduGrid infrastructure, new bindings for DRMAA and support for DAGMAN workflows.
New to 4.2.1: Since this is an incremental release, only bug fixes have been included.
The following problems and limitations are known to exist for GridWay at the time of the 4.2.1 release:
- Bug 5308: gwd doesn't recover AIDs and TIDs
- Bug 5626: GridWay should offer the ability to start from a specified job ID
- Bug 5722: Autotool for the new doc
- Bug 5727: Improve DB checks in autoconf
- Bug 5735: Update development guide with the binding 1.0 for DRMAA JAVA
- Bug 5883: MAX_RUNNING_RESOURCE should be configurable for each host
- Bug 6037: implementation of pre_wrapper option in libdrmaa
- Bug 6334: Customized Gridway JOB IDs
- Bug 6343: improve gridway error messages and logging
- Bug 6346: MADs should check for java executable if used
GridWay uses different Globus services to perform the tasks of information gathering, job execution and data transfer.
GridWay depends on the following GT components for job execution:
- GRAM: GridWay can interface with both GRAM 2 (pre web services) and GRAM 4 (web services)
GridWay depends on the following GT components for data staging:
- RFT
- GridFTP
GridWay depends on the following GT component for information gathering:
- MDS
GridWay relies on the Globus basic security infrastructure for authentication and authorization. On top of that, GridWay can use other Globus services and components to complement this infrastructure:
- Delegation Service
- MyProxy
Tested platforms for GridWay:
GridWay builds successfully for the following platforms:
- Linux
- Tru64
- Mac OS X
- Solaris
- Aix
In addition, GridWay has been tested with the major Grid infrastructures. Click the following links to find more information on how to use GridWay with EGEE,TeraGrid,OSG and NorduGrid.
The following information regards compatibility with the previous stable version of GridWay included in GT 4.0 series:
API changes since GT 4.0:
- Added DRMAA scripting language bindings for Python, Ruby and Perl.
CLI changes since GT 4.0:
- Added gwdag, a tool to run Condor DAGMAN compatible workflows.
Configuration interface changes since GT 4.0:
- None
GridWay is an open project, flexible and completely based on standards to leverage its usability and interoperability. GridWay supports several standards developed by the Open Grid Forum., namely:
- Distributed Resource Management Application API Working Group (DRMAA-WG)
The DRMAA working group has developed an API specification for the submission and control of jobs to one or more Distributed Resource Management (DRM) systems. The goal is to facilitate the direct interfacing of applications to today's DRM systems by application's builders, portal builders, and Independent Software Vendors (ISVs). The Distributed Resource Management Application API (DRMAA) provides a generalized API to distributed resource management systems (DRMSs) in order to facilitate integration of application programs.
The scope of DRMAA is limited to job submission, job monitoring and control, and retrieval of the finished job status. DRMAA provides application developers and distributed resource management builders with a programming model that enables the development of distributed applications tightly coupled to an underlying DRMS. DRMAA preserves flexibility and choice in system design.
GridWay provides support for the DRMAA API (JAVA and C bindings) to develop distributed applications. OGF document GFD.104: GridWay DRMAA 1.0 Implementation - Experience Report provides details of the implementation.
- Job Submission Description Language (JSDL-WG)
The goal of the JSDL working group is to produce a language that describes the requirements of jobs for submission to Grids. JSDL 1.0, published as OGF recommendation GFD-R-P.056 is an XML-based language that focuses mainly on computational jobs. The JSDL-WG is working on extending this language to address a wider class of jobs, including Web service invocations.
Last GridWay release provides support for JSDL, the POSIX Application profile and HTC Profile schemas can be used to describe jobs.
Click here for more information about this component.