endpoint

Our first implementation for transferring files to/from Amazon S3 endpoints (buckets) was a good proof of concept but performance was, at best, average. In our latest release, we’ve upped the stakes significantly and used multiple techniques to improve performance: (1) using parallel TCP file streams, (2) quickly retrying on transient faults, and (3) using multipart-upload only for files greater than 100MB.

In the Transfer Web Interface (Web GUI), when you first access an endpoint you are prompted for your MyProxy username and passphrase. Once you provide this information, you can browse your files and perform transfers to and from the endpoint. This blog post is about what happens behind the scenes to make that happen.