Amazon S3 Bucket Migration between AWS Account

Amazon S3 Bucket Migration between AWS Account

Overview

Imagine you have 5000 audio files in your Amazon S3 bucket and you want to move it to a new AWS Account. But there is one caveat you cannot zip the audio folder download it to your local machine and upload it to the new AWS Account.

So what you would do? Don’t worry you are in the right place for the solution

In this post, you will learn how to migrate the data/objects of S3 Bucket from one AWS account to another AWS account

Objectives

  1. What is Amazon S3

  2. Getting the Destination AWS Account Number

  3. Attaching the policy in Source S3 Bucket

  4. Attaching the policy in Destination AWS Account IAM User

  5. Installing the AWS CLI in the Local Machine

  6. Configuring the Access Key ID and Secret ID

  7. Syncing S3 objects to Destination AWS Account

Prerequisites

1. What is Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. It can be used by all sizes of industries which can store and protect data for a range of use cases such as websites, mobile apps, backup, and restore enterprise applications.

2. Getting the Destination AWS Account Number

  • Sign in to the AWS Management Console account for the destination account

  • Once signed in the navigation bar click support and then click Support center

Once the Support center is clicked you can see a screen like below

On the left-hand side, you can see the Account number for your destination AWS Account Copy that number.

3. Attaching the policy in Source S3 Bucket

Now go to your source AWS account and then select S3 Bucket

Select the Bucket which you want to migrate

Click on the permission tab and select Bucket policy like below

In the policy editor add the below configuration

If your bucket has public access then do like below

4. Attaching the policy in Destination AWS Account IAM User

Goto your Destination AWS account and select IAM user like below

Click on the IAM option which will navigate to the screen like below

Click on the Policies on the left-hand side which would open a screen like below

Click on the Create policy and then click on the JSON tab

Now replace the policy with the below code

Once the policy is added click on the Review policy like below

Which will open a new page where you can fill in the policy name like below

Now click on the Create Policy button. Once the policy is created click on the User on the left-hand side like below

Now click Add User and then you can see a screen like below

Fill in the user name select the Access type as Programmatic access and then click Next Permissions

Now you can see a page like below and select Attach existing policies directly

Attach existing policies directly

In the Filter, policies type the policy name which we created earlier below

Click Next Tags and then in the last section you can see the credentials below

Download the credentials to your local machine because it will be shown only once

5. Installing the AWS CLI in the Local Machine

Now we will install the AWS CLI in the Local Machine for that open your terminal and type the below command (Mac and Linux Machine)

sudo pip install awscli

Once it is installed check whether AWS CLI is installed properly with the command

aws help

Now you could see an output like below

6. Configuring the Access Key ID and Secret ID

Now we will configure the AWS Destination account Access key and Secret key in the AWS CLI

Open the terminal and give the command

aws configure

which would ask for the Access key and secret ID like below

Configuring Access Key ID and Access Key

Add the Access Key ID, Secret Access Key and enter the default region of your Destination AWS account and then type table to show the output in table format else press Enter which will use the Default JSON option

Once the configuration is done it’s time for migration

7. Syncing S3 objects to Destination AWS Account

To Sync the S3 objects from Source to Destination AWS account use the below command in your terminal

aws s3 sync s3://YOUR-SOURCE-BUCKET-NAME-HERE s3://YOUR-DESTINATION-BUCKET-NAME-HERE --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

Replace the source bucket name, destination bucket name, and source, and destination region like below

aws s3 sync s3://my-us-west-2-bucket s3://my-ap-south-1-bucket --source-region us-west-2 --region ap-south-1

Now click enter which will start copying the files from your Source Bucket to Destination Bucket :)

I have some doubts

Question #1

Q: I have multiple folders in my Source S3 Bucket and i want to copy only one folder to the Destination S3 Bucket for that what shall i do

A: It is very simple just change the command like below and click Enter which will copy only one folder

aws s3 sync s3://my-us-west-2-bucket/folder1 s3://my-ap-south-1-bucket/folder1 --source-region us-west-2 --region ap-south-1

Question #2

Q: I have added new audio in my Source S3 Bucket and i want to copy that one to the Destination S3 Bucket. If i give the above command will it copy all the 5001 audio files or just the 1 audio file

A: It will copy/sync only the 1 audio file and not the other audio files from your Source S3 Bucket

Question #3

Q: I have renamed a file in Source S3 Bucket and now i am giving the above command will it replace the existing file in the Destination S3 Bucket?

A: No it will create as a new file in Destination S3 Bucket keeping the old one as it is.

Congratulations!

You have learned how to Copy/Sync files from one AWS Account to another AWS Account. Happy Learning!!!!