Table of Contents
GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.
Features new in GT 5.0.2:
New feature to synchronize datasets:
Only transfer files where the destination does not exist or differs (size, timestamp, checksum) from the source.
New globus-url-copy option that will glob/recurse/sync-check and dump the list of files to be transferred into a file:
All files that would be transferred are enumerated and dumped to the specified file. Resulting file is the same format as the -f input file. Note: It performs no write operations on the destination. If you intend to use this file as input for a future transfer, the -create-dest option will be required if any destination directories do not already exist.
New capability to put the server in offline mode
All new connections can be disabled and custom message can be configured to display to clients.
Features that continue to be supported from previous versions
- Improved failure restart capability in globus-url-copy
- Stall detection
- Client-side host aliasing
- GridFTP over UDT
- SSH security for GridFTP control channel
- Running the GridFTP server with GFork GridFTP
- Multicasting / Network overlays (EXPERIMENTAL)
- Netlogger's bottleneck detection for GridFTP transfers
- GSI security: This is the PKI based, de facto standard security system used in Grid applications. Kerberos is also possible but is not supported and can be difficult to use due to divergence in the capabilities of GSI and Kerberos.
- Third-party transfers: Very common in Grid applications, this is where a client mediates a transfer between two servers (both likely at remote sites) rather than between the server and itself (called a client/server transfer).
- Cluster-to-cluster data movement or Striping: GridFTP can do coordinated data transfer by using multiple computer nodes at the source and destination.
- Partial file access: Regions of a file may be accessed by specifying an offset into the file and the length of the block desired.
- Reliability/restart: The receiving server periodically (the default is 5 seconds, but this can be changed) sends “restart markers” to the client. This marker is a messages specifying what bytes have been successfully written to the disk. If the transfer fails, the client may restart the transfer and provide these markers (or an aggregated equivalent marker), and the transfer will pick up where it left off. This can include “holes” in the file.
- Large file support: All file sizes, lengths, and offsets are 64 bits in length.
- Data channel reuse: Data channel can be held open and reused if the next transfer has the same source, destination, and credentials. This saves the time of connection establishment, authentication, and delegation. This can be a huge performance difference when moving lots of small files.
- Integrated instrumentation (Performance Markers).
- Logging/audit trail (Extensive Logging in the server).
- Parallel transfers (Multiple TCP streams between a pair of hosts).
- TCP Buffer size control (Protocol supports Manual and Automatic; Only Manual Implemented).
- Server-side computation (Extended Retrieve (ERET) / Extended Store (ESTO) commands).
- Based on Standards: RFC 959, RFC 2228, RFC 2389, IETF Draft MLST-16 , GGF GFD.020.
Other Supported Features
- On the client side we provide a scriptable tool called globus-url-copy. This tool can take advantage of all the GridFTP protocol features and can also do protocol translation between FTP, HTTP, HTTPS, and POSIX file IO on the client machine.
- We also provide a set of development libraries and APIs for developers wishing to add GridFTP functionality to their application.
Deprecated Features
- None
- Error message displayed when the connection limit is exceeded at the server has been changed
- Bug GRIDFTP-68 - sshftp broken when password/passphrase entry is required
- Bug GRIDFTP-57 - SITE commands not individually disableable
- Bug GRIDFTP-62 - Disk read for CKSM uses unexpected block size
- Bug GRIDFTP-54 - Displaying the hostname while asking for password in SSH GridFTP
- Bug 6928 - AF_LOCAL vs. AF_UNIX
The following problems and limitations are known to exist for GridFTP at the time of the 5.0.2 release:
GridFTP depends on the following GT components:
- Non-WS (General) Authentication & Authorization
- C Common Libraries
- XIO
GridFTP depends on the following 3rd party software:
- OpenSSL (version is included in release)
Tested platforms for GridFTP
- i386 Linux
- ia64 Linux (TeraGrid)
- AIX 5.2
- Solaris 9
- PA-RISC HP/UX 11.11
- ia64 HP/UX 11.22
- Tru64 Unix
- Mac OS X
While the above list includes platforms on which we have tested GridFTP, it does not imply support for a specific platform. However, we are interested in hearing reports of success or bug reports on any platform.
Protocol changes since GT 5.0.1
- None
API changes since GT 5.0.1
- None
Exception changes since GT 5.0.1
- Not Applicable (GridFTP is not Java-based)
Schema changes since GT 5.0.1
- Not Applicable (GridFTP is not SOAP-based)
Associated standards for GridFTP:
- RFC 959 Base FTP protocol
- RFC 2228 gssapi security extensions for FTP RFC 2389 FEAT, OPTS, etc.
- extensions to FTP (IETF FTP Working group draft) for structured directory listings, SIZE, MDTM commands.
- GFD.020 GridFTP extensions
See GridFTP for more information about this component.