Introduction
Batch Replicator is intended to provide a general utility for replicating data files by locating physical file names in the Replica Location Service (RLS), transfering the data files using the Reliable File Transfer (RFT) service, and then registering the new physical file locations in the RLS again. In order to use the Batch Replicator, the RLS and RFT must also be installed. In order for Batch Replicator to locate your files, you must first register them in the RLS. Once you have registered your files you may use the Batch Replicator to replicate them to locations throughout your data Grid.
The Batch Replicator provides a set of command-line tools to create replication requests, to start/suspend/resume/stop replications, and to find status information on replications. The usage scenarios in this guide provide examples for basic usage of the Batch Replicator.
Table of Contents
- 1. Managing Sets of Replicas (WS)
- DataRep Commands
- globus-replication-create - This tool is used to create a replication resource by submitting a replication request to the designated replication service.
- globus-replication-start - This tool starts the replication activities.
- globus-replication-stop - This tool stops the replication activities.
- globus-replication-suspend - This tool suspends the replication activities.
- globus-replication-resume - This tool resumes the replication activities.
- globus-replication-finditems - This tool queries the replication resource to return the status of individual replication item activities.
- 2. Troubleshooting
- Index
Table of Contents
This section describes a few key usage scenarios and provides examples of using the Batch Replicator command-line tools.
Before using any of the tools, a user must generate a valid user proxy. Use grid-proxy-init.
% $GLOBUS_LOCATION/bin/grid-proxy-init
Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA.mymachine/OU=mymachine/CN=John Doe
Enter GRID pass phrase for this identity:
Creating proxy ................................. Done
Your proxy is valid until: Tue Oct 26 01:33:42 2004
Once you have generated a valid proxy you must create a delegated credential. Your delegated credential will be used by the Batch Replicator to make secure calls to other services (e.g., RLS, RFT, etc.) in order to perform the data replication. It is important to ensure that you give your delegated credential enough lifetime to support the running time of your replication activities. To delegate your credential use globus-credential-delegate(1).
% $GLOBUS_LOCATION/bin/globus-credential-delegate -s https://localhost:8443/wsrf/services/DelegationFactoryService mycredential.epr Delegated credential EPR: Address: https://localhost:8443/wsrf/services/DelegationService Reference property[0]: <ns1:DelegationKey xmlns:ns1="http://www.globus.org/08/2004/delegationService">b90c6c90-fcaa-11da-afde-c9bab5290e15</ns1:DelegationKey>
A key parameter for any replication request is the request file. The replication request file is a text file containing CRLF-terminated rows of tab-delimited pairs of Logical Filename (LFN) names and destination (URL) locations. An example of such a file is shown.
% cat testrun.req
testrun-1 gsiftp://myhost:9001/sandbox/files/testrun-1
testrun-2 gsiftp://myhost:9001/sandbox/files/testrun-2
testrun-3 gsiftp://myhost:9001/sandbox/files/testrun-3
testrun-4 gsiftp://myhost:9001/sandbox/files/testrun-4
testrun-5 gsiftp://myhost:9001/sandbox/files/testrun-5
![[Note]](/docbook-images/note.gif) | Note |
|---|---|
The LFNs in the left column of the request file (e.g., |
The initial step for any replication is to create the replication resource. Creating the resource depends on the availability of a Batch Replicator service, a delegated credential, and a properly formatted replication request file. The replication request file must be specified by its URL. Currently supported URL schemes for the request file include file, http, and ftp. If the replication client is run local to the service the file scheme is appropriate, whereas if the client is remote than the latter schemes must be used. It is a good practice to specify a filename to save the replication resource's endpoint reference. The endpoint reference is required for all other operations on the resource, such as getting resource properties, starting/stopping, and destroying it. Numerous options are available to influence the behavior of the data replication activities (see globus-replication-create(1)). One option of particular interest is the --start option, which immediately starts the replication activities following creation of the replication resource. An example of using the globus-replication-create(1) tool is shown.
% $GLOBUS_LOCATION/bin/globus-replication-create -s \ https://myhost:8443/wsrf/services/ReplicationService \ -C mycredential.epr -V myreplicator.epr file:///scratch/testrun.req
This command does not write to stdout when successful unless the --debug option is specified.
Once a replication resource has been create, the replication activities may be started. As mentioned in Create replication resource the replication may be immediately started after it is created. If the immediate start option is not specified, the globus-replication-start(1) tool must be used to start the replication.
% $GLOBUS_LOCATION/bin/globus-replication-start -e myreplicator.epr
No output is expect from this command when successful.
Throughout the lifecycle and after the completion of the replication resource, it will be important to lookup its Resource Properties. The standard WS-RF port types are supported and the supplied tools (e.g., wsrf-get-property) may be used with the Batch Replicator and its resources.
% $GLOBUS_LOCATION/bin/wsrf-get-property -e myreplicator.epr \
"{http://www.globus.org/namespaces/2005/05/replica/replicator}status"
<ns1:status xmlns:ns1="http://www.globus.org/namespaces/2005/05/replica/replicator">
Active</ns1:status>
And,
% $GLOBUS_LOCATION/bin/wsrf-get-property -e myreplicator.epr \
"{http://www.globus.org/namespaces/2005/05/replica/replicator}count"
<ns1:count xmlns:ns1="http://www.globus.org/namespaces/2005/05/replica/replicator">
<ns1:total>10</ns1:total>
<ns1:finished>0</ns1:finished>
<ns1:failed>0</ns1:failed>
<ns1:terminated>0</ns1:terminated>
</ns1:count>
Throughout the lifecycle and after the completion of the replication resource, it may be helpful to find individual replication items in the replication resource to inspect the detailed status of the replication activities. The globus-replication-finditems(1) tool is used to find replication items. The following example demonstrates the usage when finding a limited number of items, offset into the lookup results set, for a specified status.
% $GLOBUS_LOCATION/bin/globus-replication-finditems -e myreplicator.epr -S Pending -O 1 -L 2
<ns1:FindItemsResponse xmlns:ns1="http://www.globus.org/namespaces/2005/05/replica/replicator">
<ns1:items xsi:type="ns1:ReplicationItemType" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns1:uri xsi:type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema">testrun-2</ns1:uri>
<ns1:priority xsi:type="xsd:int" xmlns:xsd="http://www.w3.org/2001/XMLSchema">1000</ns1:priority>
<ns1:status xsi:type="ns1:ReplicationItemStatusEnumerationType">Pending</ns1:status>
<ns1:destinations xsi:type="ns1:DestinationType">
<ns1:uri xsi:type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
gsiftp://myhost:9001/sandbox/files/testrun-2</ns1:uri>
<ns1:status xsi:type="ns1:DestinationStatusEnumerationType">Pending</ns1:status>
</ns1:destinations>
</ns1:items>
<ns1:items xsi:type="ns1:ReplicationItemType" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns1:uri xsi:type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema">testrun-3</ns1:uri>
<ns1:priority xsi:type="xsd:int" xmlns:xsd="http://www.w3.org/2001/XMLSchema">1000</ns1:priority>
<ns1:status xsi:type="ns1:ReplicationItemStatusEnumerationType">Pending</ns1:status>
<ns1:destinations xsi:type="ns1:DestinationType">
<ns1:uri xsi:type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
gsiftp://myhost:9001/sandbox/files/testrun-3</ns1:uri>
<ns1:status xsi:type="ns1:DestinationStatusEnumerationType">Pending</ns1:status>
</ns1:destinations>
</ns1:items>
</ns1:FindItemsResponse>
When the replication is complete, the replication resource may be destroyed. Destroying the replication resource will help to free up system resources (namely, memory), especially in the case that the replication entailed a large amount of individual replication activities (i.e., many files specified in the replication request file). The standard WS-RF port types are supported and the supplied wsrf-destroy tool may be used to destroy the Batch Replicator resource.
% $GLOBUS_LOCATION/bin/wsrf-destroy -e myreplicator.epr
Destroy operation was successful
The Batch Replicator provides a set of command-line tools to control the creation and lifecycle of a given replication request. These command line tools are available on Unix and Windows platforms and will work in the same way (of course within the platform rules - the path syntax, variable definitions, etc.).
Table of Contents
- globus-replication-create - This tool is used to create a replication resource by submitting a replication request to the designated replication service.
- globus-replication-start - This tool starts the replication activities.
- globus-replication-stop - This tool stops the replication activities.
- globus-replication-suspend - This tool suspends the replication activities.
- globus-replication-resume - This tool resumes the replication activities.
- globus-replication-finditems - This tool queries the replication resource to return the status of individual replication item activities.
Name
globus-replication-create — This tool is used to create a replication resource by submitting a replication request to the designated replication service.
Synopsis
globus-replication-create
Tool description
Use this tool to create replication resources (also refered to as "Replicator" resources). You must specify the URL of the ReplicationService where the resource will be created.
You must submit the filename of a file containing an Endpoint Reference (EPR) to a delegated credential resource, which you must have previously created. Finally, you must submit
the URL of a request file specifying the desired data replications. If the client is running local to the service container the URL may be a file:// URL,
whereas if the client is remote the URL may be a http:// or ftp:// URL. The request file adopts a table format
structure where each line in the file represents a source-destination pair delimited by a single tab character. The source should be a logical filename (LFN)
as found in a Replica Location Service (RLS) Replica Location Index (RLI) service. The destination should be a URL acceptable to the GridFTP server. Most likely, you will want to
specify a filename in order to save the newly created Replicator resource's EPR. You may use the EPR for starting the resource and querying its resource properties.
Command syntax
globus-replication-create [options] request-file
Table 1. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| --binary <boolean> | Specifies binary data transfer |
| --blockSize <int> | Block size for data transfer |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -C,--delegatedCredential <file> | Loads Delegated Credential EPR from file |
| --concurrency <int> | Concurrency of data transfer |
| -d,--debug | Enables debug mode |
| --dataChannelAuth <boolean> | Data channel authentication for transfers |
| --destinationSubject <name> | Destination subject name for data transfer |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| --parallelStreams <int> | Parallel streams for data transfer |
| -s,--service <url> | Service URL |
| -S,--start | Starts the Replicator resource immediately |
| --sourceSubject <name> | Source subject name for data transfer |
| --subject <name> | Subject name for data transfer |
| --tcpBufferSize <int> | TCP buffer size for data transfer |
| --userName <name> | User name for data transfer |
| -V,--saveEpr <file> | Save EPR of newly created Replicator to file |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Name
globus-replication-start — This tool starts the replication activities.
Synopsis
globus-replication-start
Tool description
Replication resources created with the globus-replication-create tool may be "started" by using this tool and passing
the filename of the saved EPR as a parameter to the tool. The tool will indicate an error condition if the user attempts to start a resource that has been previously started.
Command syntax
globus-replication-start [options]
Table 2. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -d,--debug | Enables debug mode |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| -s,--service <url> | Service URL |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Name
globus-replication-stop — This tool stops the replication activities.
Synopsis
globus-replication-stop
Tool description
Replication resources created with the globus-replication-create tool may be "stoped" by using this tool and
passing the filename of the saved EPR as a parameter to the tool. The tool will indicate an error condition if the user attempts to stop a resource that has not been
previously started, a resource that has been suspended, or a resource that has terminated or been destroyed.
Command syntax
globus-replication-stop [options]
Table 3. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -d,--debug | Enables debug mode |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| -s,--service <url> | Service URL |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Name
globus-replication-suspend — This tool suspends the replication activities.
Synopsis
globus-replication-suspend
Tool description
Replication resources created with the globus-replication-create tool may be "suspended" by using
this tool and passing the filename of the saved EPR as a parameter to the tool. The tool will indicate an error condition if the user attempts to suspend a
resource that has not been previously started, a resource that has been suspended, or a resources that is done or has been destroyed.
Command syntax
globus-replication-suspend [options]
Table 4. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -d,--debug | Enables debug mode |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| -s,--service <url> | Service URL |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Name
globus-replication-resume — This tool resumes the replication activities.
Synopsis
globus-replication-resume
Tool description
Replication resources created with the globus-replication-create tool may be "resumed" by using
this tool and passing the filename of the saved EPR as a parameter to the tool. The tool will indicate an error condition if the user attempts to resume a
resource that has not been previously suspended, or a resource that is done or has been destroyed.
Command syntax
globus-replication-resume [options]
Table 5. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -d,--debug | Enables debug mode |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| -s,--service <url> | Service URL |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Name
globus-replication-finditems — This tool queries the replication resource to return the status of individual replication item activities.
Synopsis
globus-replication-finditems
Tool description
This tool provides the ability to query the status of individual replication items (e.g., replication of a specific file or files) managed by the given Replication resources. It is possible to query for the status of a specific named item or to query for the status of multiple items based on a particular status (e.g., Pending, Finished, Failed). In addition, to reduce potentially large overhead of returning a large results set to the client, the client may specify an offset and limit for the results set to be returned. The "name" or "status" option must be specified.
Command syntax
globus-replication-finditems [options] {-N name | -S status}
Table 6. Options
| -a,--anonymous | Use anonymous authentication. (requires either -m 'conv' or transport (https) security) |
| -c,--serverCertificate <file> | A file with server's certificate used for encryption. Used in the case of GSI Secure Message encryption |
| -d,--debug | Enables debug mode |
| -e,--eprFile <file> | Loads EPR from file |
| -f,--descriptor <file> | Sets client security descriptor. Overrides all other security settings |
| -g,--delegation <mode> | Performs delegation. Can be 'limited' or 'full'. (requires -m 'conv') |
| -h,--help | Displays help |
| -k,--key <name value> | Resource Key |
| -l,--contextLifetime <value> | Lifetime of context created for GSI Secure Conversation (requires -m 'conv') |
| -L,--limit <num> | Limit on the size of the result set. |
| -m,--securityMech <type> | Sets authentication mechanism: 'msg' (for GSI Secure Message), or 'conv' (for GSI Secure Conversation) |
| -N,--byName <name> | Finds item by the Logical Filename (LFN) name. |
| -O,--offset <num> | Offset into the results set. Indexed by 0. |
| -p,--protection <type> | Sets protection level, can be 'sig' (for signature) can be 'enc' (for encryption) |
| -S,--byStatus <status> | Finds item(s) by status. Valid status values include "Pending", "Finished", "Failed", and "Terminated". |
| -s,--service <url> | Service URL |
| -z,--authorization <type> | Sets authorization, can be 'self', 'host' or 'none' |
Table of Contents
The following section provides information about common errors for end users. For a list of common errors in GT, see Error Codes. You can also find information on sys admin logs in Chapter 5, Debugging.
Table 2.1. Batch Replicator Errors
| Error Code | Definition | Possible Solutions |
|---|---|---|
Authorization failed. Expected <hostname1> target but received <hostname2> | Did not receive expected hostname | When authorization is enabled on the container, you may need to use the proper hostname when referencing the Batch Replicator service rather than using localhost. |
org.globus.wsrf.ResourceException: Failed to create Replication: /scratch/testrun (No such file or directory) | Cannot find the request file | Ensure that the request file's filename is correct, that it is reachable by the Batch Replicator service, and that it has the appropriate permissions for the Batch Replicator service to access it. |
org.globus.wsrf.ResourceException: Failed to create Replication: String
index out of range: -1 | The request file is malformed (for example by using spaces instead of a delimiting tab character) which is resulting in a runtime exception. | Make sure your request file is in the correct form as described here. |
D
- delegating user credentials, Delegate user credentials
E
- errors, Errors
G
- generating a valid proxy, Generate a valid proxy
R
- replica sets, Managing Sets of Replicas (WS)
- replication request file, Replication request file
- replication resource
- creating, Create replication resource
- destroying, Destroy replication resource
- finding replication item status, Find replication item status
- getting replication resource properties, Get replication resource properties
- starting, Start replication