Setting up AWS CLI and Downloading S3 Files using command line interface

1. Install the AWS Command Line Interface (CLI) using:

$ pip install awscli 

For more details such as upgrade, check the amazon webpage here.

Note: if installation was not successful, reinstall forcefully. Check here

The next step is to configure. AWS requires the following items for this:
(how to find these?)

1. Access key ID
2. Secret Access key
3. Region name
4. Valid output format

2. Configuration

Now, run the following command and answer the prompts:

$ aws configure

AWS Access Key ID [None]: <enter the access key>
AWS Secret Access Key [None]: <enter the secret access key>
Default region name [None]: <enter region>
Default output format [None]: <text>

3. Copying files from AWS S3 folder to Local Desktop 

once this is done, we can start using AWS CLI for the file operations. For example, if it want to copy WBB3097.gz file in my AWS account to my local desktop, which is under "qgen" username and in the folder MIX7341, I do the following

$ aws s3 cp s3:URI  <Local Desktop Location>

$ aws s3 cp s3://qgen/MIX7341/WBB3097.gz ~/Desktop/AWS

4. List all files in AWS S3 folder 

Similarly to list all the files in the MIX7341, we can use the following command

$ aws s3 ls s3://qgen/MIX7341/

2018-08-02 11:03:58  459407360 WBB3066.gz
2018-08-02 11:03:01  337848320 WBB3067.gz
2018-08-02 11:02:57  327219200 WBB3068.gz
2018-08-02 11:02:57  298680320 WBB3069.gz
2018-08-02 11:02:57  280514560 WBB3070.gz

5. Copy multiple files using BASH for loop

Off hand, I had to download multiple files from AWS account. To click each file and download was taking time. So, I just listed the files (step1 below) and used "for loop" in bash to download the files (step2 below).

$ aws s3 ls s3://qgen/MIX7341/ | awk '{print $NF}' >files_in_AWS.list ##step 1

$ for d in $(cat files_in_AWS.list); do echo $d; aws s3 cp s3://qgen/MIX7341/$d . ; done ##step 2

border:solid green;border-width:.1em .1em .1em .8em;padding:.2em .6em;
Python ; Pastie ; www. hilitie.me