AWS S3 File Transfer with AWS CLI

AWS S3 File Transfer with AWS CLI

Transferring files to and from Amazon S3 is one of the most common tasks for users of AWS. The AWS CLI (Command Line Interface) offers a straightforward and efficient way to manage S3 operations, especially when handling large files or extensive data sets. In this blog post, we will cover how to install and configure AWS CLI for optimal performance, discuss the commands needed to transfer files, and provide recommendations to ensure fast and smooth data transfers.

AWS CLI Installation

Before you can start transferring files to Amazon S3, you need to have the AWS CLI installed on your local machine. AWS CLI is supported on a variety of operating systems, including Windows, macOS, and Linux. Here’s a quick guide to installing AWS CLI on different platforms:

• Windows: You can install AWS CLI using the Windows Installer from the official AWS documentation.
• macOS: You can install AWS CLI using brew by running:

brew install awscli

• Linux: You can install AWS CLI using the package manager appropriate for your distribution. For example, on Ubuntu,

sudo apt install awscli

For a complete guide on installation for various operating systems, please refer to the official AWS documentation.

AWS CLI Configuration

Once you have installed AWS CLI, the next step is configuring it to communicate with your AWS account. To configure the AWS CLI, run:

aws configure

This command will prompt you to enter the following details:

• AWS Access Key ID
• AWS Secret Access Key
• Default region (e.g., us-east-1)
• Default output format (e.g., json)

To obtain your access key and secret key, you can generate them from the AWS IAM console. It’s essential to protect these credentials as they provide access to your AWS resources.

Optimizing File Transfers

By default, AWS CLI uses multipart uploads to transfer large files to S3, breaking them into smaller chunks. You can tweak the configuration to optimize performance, especially when dealing with large files. Two main settings to consider are:

• max_concurrent_requests: Increases the number of parallel requests, improving transfer speed.
• multipart_threshold: Adjusts the threshold size for splitting large files into smaller parts.

To modify these settings, edit the AWS configuration file located in ~/.aws/config (Linux/macOS) or C:\Users\USERNAME\.aws\config (Windows). Add the following:

[default]
s3 =
  max_concurrent_requests = 10
  multipart_threshold = 64MB

This setup will allow 10 concurrent requests and enable multipart uploads for files larger than 64MB, significantly reducing transfer times for large files.

Based on the official AWS S3 configuration documentation, here is an enhanced look at the parameters you can modify and the net effect of changing their values.

Key Parameters for Optimizing AWS S3 File Transfers

1. multipart_threshold
   • Definition: The minimum file size at which AWS CLI automatically switches to multipart uploads.
   Default Value: 8 MB
   • Usage:
     • Files larger than this value will be split into smaller parts and uploaded in parallel. This can dramatically improve upload speeds for large files.
     Effect of Higher Values:
     • Increasing this value (e.g., to 64 MB or more) means that files need to be larger before the multipart upload kicks in. This is suitable if you’re dealing with fewer large files and have limited bandwidth, as multipart can introduce overhead for small files.
     Effect of Lower Values:
     • Lowering this value (e.g., to 5 MB) ensures that more files are uploaded in parts, even if they are smaller. This can improve performance when transferring many medium-sized files or when you want to minimize the risk of failures during upload.

[default]
s3 =
  multipart_threshold = 64MB

2. multipart_chunksize
   • Definition: The size of individual parts when performing a multipart upload.
   • Default Value: 8 MB
   • Usage:
     • This defines how large each chunk of a file will be during a multipart upload.
   • Effect of Higher Values:
     • Using a larger chunk size (e.g., 16 MB or 64 MB) means fewer parts, but each part will take longer to upload. This can reduce overhead but might delay the failure detection for larger chunks.
   • Effect of Lower Values:
     • Using smaller chunks (e.g., 5 MB) means more parts and faster upload of individual parts. This can help with faster retries if a single part fails but increases the number of requests, which could introduce more overhead for very large files.

[default]
s3 =
  multipart_chunksize = 16MB

3. max_concurrent_requests
   • Definition: The number of parallel requests made by AWS CLI during multipart uploads or downloads.
   • Default Value: 10
   • Usage:
     • This controls how many parts of a file are uploaded or downloaded at the same time.
   • Effect of Higher Values:
     • Increasing this value (e.g., 15 or 20) allows more parts to be uploaded in parallel, improving transfer speed if you have a fast internet connection or want to maximize throughput.
   • Effect of Lower Values:
     • Reducing the number of concurrent requests (e.g., to 5) is suitable for slower or limited bandwidth environments, as fewer parts are uploaded simultaneously, reducing the load on the network.

[default]
s3 =
  max_concurrent_requests = 15

4. max_queue_size
   • Definition: The maximum number of tasks (parts) that can be queued for uploading or downloading in a multipart transfer.
   • Default Value: 1000
   • Usage:
     • This parameter defines how many upload or download tasks can be queued at one time during multipart transfers.
   • Effect of Higher Values:
     • Setting a higher value allows more tasks to be queued, which can improve performance if you are transferring many files or very large files.
   • Effect of Lower Values:
     • Reducing the queue size can help if you are on a constrained system with limited memory or network capacity, as it reduces the number of pending tasks waiting for processing.

[default]
s3 =
  max_queue_size = 500

5. max_bandwidth
   • Definition: Limits the amount of bandwidth that the AWS CLI can use for S3 transfers.
   • Default Value: Unlimited
   • Usage:
     • By setting this, you can cap the bandwidth used by AWS CLI to avoid saturating your network connection.
   • Effect of Higher Values:
     • If you don’t set a cap, or set a high limit (e.g., 50MB/s), AWS CLI will use as much bandwidth as it can, which is ideal when network saturation isn’t a concern.
   • Effect of Lower Values:
     • Lowering this value (e.g., to 5MB/s or 10MB/s) helps prevent S3 transfers from consuming too much bandwidth, which can be useful if you need to reserve bandwidth for other activities or users.

[default]
s3 =
  max_bandwidth = 20MB/s

6. use_accelerate_endpoint
   • Definition: Enables the use of the S3 Transfer Acceleration feature, which speeds up file transfers by routing them to the nearest AWS edge location before reaching S3.
   • Default Value: False
   • Usage:
     • S3 Transfer Acceleration is a paid service that increases the speed of data transfers over long distances by using AWS edge locations.
   • Effect of Enabling:
     • When set to true, transfers to and from S3 can be much faster, especially for long-distance data transfers, though there will be an additional cost associated with the acceleration.
   • Effect of Disabling:
     • If not using transfer acceleration, transfers will occur over the default S3 endpoints, which may be slower for geographically distant transfers.

[default]
s3 =
  use_accelerate_endpoint = true

AWS CLI Commands for S3 File Transfer

AWS CLI provides a rich set of commands to interact with Amazon S3. The basic syntax for transferring files is straightforward.

Uploading Files to S3

To upload a file or a directory to an S3 bucket, use the following command:

aws s3 cp /path/to/local/file s3://bucket-name/path/to/s3/

For uploading entire directories, you can use the --recursive option:

aws s3 cp /path/to/local/directory s3://bucket-name/ --recursive

This command copies all files in the directory to the specified S3 bucket.

Downloading Files from S3

To download files from S3 to your local machine, you use a similar command:

aws s3 cp s3://bucket-name/path/to/s3/file /path/to/local/directory/

To download an entire directory:

aws s3 cp s3://bucket-name/ /path/to/local/directory --recursive

This command will download all the files from the S3 bucket to the specified local directory.

Syncing Local and S3 Directories

One of the most powerful features of AWS CLI is the sync command, which only uploads or downloads files that have changed, making it more efficient for regular data transfers.

To sync a local directory with an S3 bucket:

aws s3 sync /path/to/local/directory s3://bucket-name/

To sync files from an S3 bucket to a local directory:

aws s3 sync s3://bucket-name/ /path/to/local/directory

This command will compare files in both locations and only transfer those that are different, saving bandwidth and time.

Recommendations for Fast and Efficient Transfers

To ensure that your file transfers to and from S3 are as fast and efficient as possible, keep these recommendations in mind:

1. Use the sync command: This is particularly useful when regularly updating files in your S3 bucket, as it only transfers changed files. But you may encounter performance issues, if the files are continuously generated in the folder where “aws sync” needs to copy to and from. Also when aws sync is running don’t interrupt the execution because of journaling and bookkeeping operations.
2. Enable multipart uploads: For large files, ensure that multipart uploads are enabled and fine-tuned as discussed in the configuration section.
3. Use max_concurrent_requests setting: Increasing the number of parallel requests can drastically improve performance, especially with high-bandwidth connections.
4. Choose the correct AWS region: To minimize latency, always store your data in an S3 bucket located in a region geographically close to your users or clients.

Example AWS CLI Configurations for Faster S3 File Transfer

Here are some example configurations for faster and efficient file transfer with AWS CLI.

Example Configuration for 50MB to 100MB sized File’s Transfers

s3 =
  multipart_threshold = 32MB
  multipart_chunksize = 16MB
  max_concurrent_requests = 40
  max_queue_size = 1500
  use_accelerate_endpoint = true
  max_bandwidth = unlimited

Example Configuration for 300MB to 400MB sized File’s Transfers

s3 =
  multipart_threshold = 128MB
  multipart_chunksize = 64MB
  max_concurrent_requests = 50
  max_queue_size = 2500
  use_accelerate_endpoint = true
  max_bandwidth = unlimited

Example Configuration for 10GB sized File’s Transfers

s3 =
  multipart_threshold = 500MB
  multipart_chunksize = 128MB
  max_concurrent_requests = 60
  max_queue_size = 3000
  use_accelerate_endpoint = true
  max_bandwidth = unlimited

Conclusion

AWS CLI is a powerful tool for managing and transferring data to Amazon S3. By following the steps outlined in this blog post, you can install and configure AWS CLI, optimize your file transfers for performance, and use the various commands to upload, download, and sync data. Taking the time to configure your CLI settings correctly will save you significant time and effort, especially when dealing with large data sets.

For more detailed documentation, you can always refer to the official AWS CLI documentation.