How to transfer data from Amazon S3 buckets to Azure Blob store using AzCopy

  Here is the guide to transfer files/folders from Amazon S3 buckets to Azure Blob store using AzCopy v10

Pre-Req: 

• Python 3.7 and above
• AzCopy v 10
• AWS CLI
• Any   Linux flavor Machine

Install AWS CLI  

[user@host]$ python3 -m pip install awscli --upgrade --user 

Add the installed location of AWSCLI onto PATH env variable

[user@host]$ export PATH=$PATH:~/.local/bin

Check if AWSCLI is installed properly
[user@host]$ aws --version
aws-cli/1.16.243 Python/3.7.3 Linux/2.6.32-754.15.3.el6.x86_64 botocore/1.12.233




Download and Install AzCopyv10

wget https://aka.ms/downloadazcopy-v10-linux
 tar -xvzf azcopy_linux_amd64_10.2.1.tar.gz
 cd azcopy_linux_amd64_10.2.1

Set Environment Variables  

[user@host ~/azcopy_linux_amd64_10.2.1]$ export AWS_ACCESS_KEY_ID=<S3_Access_id>
[user@host ~/azcopy_linux_amd64_10.2.1]$ export AWS_SECRET_ACCESS_KEY=<S3_SECRET>
[user@host ~/azcopy_linux_amd64_10.2.1]$ export PATH=$PATH:.


Use AWSCLI to list files from S3 bucket 

aws s3 ls s3://<bucket_name>/<folder> --recursive --human-readable --summarize
2019-02-12 13:02:52    3.5 MiB data/20190101/file.json

Now try to upload a sample file from S3 bucket to Blob container

azcopy copy "https://s3.amazonaws.com/<bucket_name>/<folder>/file.json" "https://<storage_account_name>.blob.core.windows.net/<blob_container_name>?<SAS_TOKEN>" 

Job cd62d161-94dd-1640-7457-a5903847b9d9 summary
Elapsed Time (Minutes): 0.0334
Total Number Of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 3696045
Final Job Status: Completed


Now that we ensured copy of single file is successfult, lets copy an entire bucket from S3 to blob store

azcopy copy "https://s3.amazonaws.com/<bucket>/<folder>" "https://<storage_account>.blob.core.windows.net/upload?<SAS_TOKEN>



Reference
https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-s3
https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html