If you just want the "rules of thumb" on getting started (without all the details), the following options using globus-url-copy will normally give acceptable performance:

globus-url-copy -vb -tcp-bs 2097152 -p 4 source_url destination_url

The source/destination URLs will normally be one of the following:

file:///path/to/my/file if you are accessing a file on a file system accessible by the host on which you are running your client.
gsiftp://hostname/path/to/remote/file if you are accessing a file from a GridFTP server.

1.1. Putting files

One of the most basic tasks in GridFTP is to "put" files, i.e., moving a file from your file system to the server. So for example, if you want to move the file /tmp/foo from a file system accessible to the host on which you are running your client to a file name /tmp/bar on a host named remote.machine.my.edu running a GridFTP server, you would use this command:

globus-url-copy -vb -tcp-bs 2097152 -p 4 file:///tmp/foo gsiftp://remote.machine.my.edu/tmp/bar

	Note
	In theory, `remote.machine.my.edu` could be the same host as the one on which you are running your client, but that is normally only done in testing situations.

1.2. Getting files

A get, i.e, moving a file from a server to your file system, would just reverse the source and destination URLs:

	Tip
	Remember `file:` always refers to your file system.

globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://remote.machine.my.edu/tmp/bar file:///tmp/foo

1.3. Third party transfers

Finally, if you want to move a file between two GridFTP servers (a third party transfer), both URLs would use gsiftp: as the protocol:

globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://other.machine.my.edu/tmp/foo gsiftp://remote.machine.my.edu/tmp/bar

1.4. For more information

If you want more information and details on URLs and the command line options, the Key Concepts gives basic definitions and an overview of the GridFTP protocol as well as our implementation of it.

2. Accessing data in...

2.1. Accessing data in a non-POSIX file data source that has a POSIX interface

If you want to access data in a non-POSIX file data source that has a POSIX interface, the standard server will do just fine. Just make sure it is really POSIX-like (out of order writes, contiguous byte writes, etc).

2.2. Accessing data in HPSS

The following information is helpful if you want to use GridFTP to access data in HPSS.

Architecturally, the Globus GridFTP server can be divided into 3 modules:

the GridFTP protocol module,
the (optional) data transform module, and
the Data Storage Interface (DSI).

In the GT 4.1.3 implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.2.1. GridFTP Protocol Module

The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.

2.2.2. Data Transform Functionality

The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.

2.2.3. Data Storage Interface (DSI) / Data Transform module

The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN).

The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..

Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.

[TODO: pointer to DSI development docs]

2.2.4. HPSS info

Last Update: August 2005

Working with Los Alamos National Laboratory and the High Performance Storage System (HPSS) collaboration (http://www.hpss-collaboration.org), we have written a Data Storage Interface (DSI) for read/write access to HPSS. This DSI would allow an existing application that uses a GridFTP compliant client to utilize an HPSS data resources.

This DSI is currently in testing. Due to changes in the HPSS security mechanisms, it requires HPSS 6.2 or later, which is due to be released in Q4 2005. Distribution for the DSI has not been worked out yet, but it will *probably* be available from both Globus and the HPSS collaboration. While this code will be open source, it requires underlying HPSS libraries which are NOT open source (proprietary).

	Note
	This is a purely server side change, the client does not know what DSI is running, so only a site that is already running HPSS and wants to allow GridFTP access needs to worry about access to these proprietary libraries.

2.3. Accessing data in SRB

The following information is helpful if you want to use GridFTP to access data in SRB.

Architecturally, the Globus GridFTP server can be divided into 3 modules:

the GridFTP protocol module,
the (optional) data transform module, and
the Data Storage Interface (DSI).

In the GT 4.1.3 implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.3.1. GridFTP Protocol Module

2.3.2. Data Transform Functionality

2.3.3. Data Storage Interface (DSI) / Data Transform module

The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..

[TODO: pointer to DSI development docs]

2.3.4. SRB info

Last Update: August 2005

Working with the SRB team at the San Diego Supercomputing Center, we have written a Data Storage Interface (DSI) for read/write access to data in the Storage Resource Broker (SRB) (http://www.npaci.edu/DICE/SRB). This DSI will enable GridFTP compliant clients to read and write data to an SRB server, similar in functionality to the sput/sget commands.

This DSI is currently in testing and is not yet publicly available, but will be available from both the SRB web site (here) and the Globus web site (here). It will also be included in the next stable release of the toolkit. We are working on performance tests, but early results indicate that for wide area network (WAN) transfers, the performance is comparable.

When might you want to use this functionality:

You have existing tools that use GridFTP clients and you want to access data that is in SRB
You have distributed data sets that have some of the data in SRB and some of the data available from GridFTP servers.

2.4. Accessing data in some other non-POSIX data source

The following information is helpful If you want to use GridFTP to access data in a non-POSIX data source.

Architecturally, the Globus GridFTP server can be divided into 3 modules:

the GridFTP protocol module,
the (optional) data transform module, and
the Data Storage Interface (DSI).

In the GT 4.1.3 implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.4.1. GridFTP Protocol Module

2.4.2. Data Transform Functionality

2.4.3. Data Storage Interface (DSI) / Data Transform module

The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..

[TODO: pointer to DSI development docs]

Prev		Next
GT 4.1.3 GridFTP : User's Guide	Home	Command line tools