Big improvements in Globus performance on Amazon S3 endpoints
November 11, 2014 | Vas Vasiliadis
Our first implementation for transferring files to/from Amazon S3 endpoints (buckets) was a good proof of concept but performance was, at best, average. In our latest release, we’ve upped the stakes significantly and used multiple techniques to improve performance: (1) using parallel TCP file streams, (2) quickly retrying on transient faults, and (3) using multipart-upload only for files greater than 100MB.
Initial tests show throughput peaks of 5.2Gbps; in one test case we transferred 1.2 terabytes in 48 minutes. These transfers were performed with using Globus Connect Personal on Amazon EC2 instances with the following configurations: Instance type: m3.xlarge (network performance: High/1000 Mbps); Instance type: cr1.8xlarge, (network performance: 10Gbps). Sustained throughput on the m3 instance was approximately 880Mbps; on the cr1 instance it was approximately 3.5Gbps.