.. _aws-s3_collect:

==================================================
Pulling your files from an AWS S3 bucket
==================================================

Overview
===================================================

If your data is stored in AWS S3 we can configure AgileData to collect it directly from your storage bucket


Steps
===================================================


1. Sign in to AWS S3
------------------------------------------

To get started, open `AWS S3 Console <https://us-west-1.console.aws.amazon.com/s3/buckets?region=us-west-1>`_

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/agiledata-filedrop-bucket.png
    :width: 800px

If you haven't already created a bucket to hold the files you want to transfer to AgileData, then use the "Create Bucket" option to create one now.

In the screen capture above I've created a bucket called agiledata-filedrop which I'll use for this guide.


2. Setup access to your storage bucket
------------------------------------------

Once your bucket has been created, you can setup the access permissions so the files can be transfered to AgileData.

To get started, open `Security Credentials (IAM) <https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials>`_

Scroll down to Access Keys

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/amazon-setup-access-keys.png
    :width: 800px

Use the Create Access Key option to create a new access key.

Because I am using my root account to create a key there are a number of warnings. 

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-access-key.png
    :width: 800px

Good practice is to create an account eg 'filedrop-bucket' with minimal privledges.

For example - access to the bucket we created above, to read, write and delete files, and nothing else.

On successful key creation you will see 

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/aws-access-key-created.png
    :width: 800px

Copy the access key and secret access key and keep them secure until the next step when we will use them to create the transfer job in Google.


3. Create a storage transfer job
------------------------------------------

To get started, open `Storage Transfer Console <https://console.cloud.google.com/transfer/create>`_

Use the Create Transfer Job option create a new transfer job

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job.png
    :width: 800px

Fill in the source type and scheduling settings

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-2.png
    :width: 800px

Fill in the source bucket name and access key values we created earlier

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-3a.png
    :width: 800px


Fill in the destination bucket name (see below if you dont know the bucket name)

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-4a.png
    :width: 800px

To get the name of the AgileData bucket your files will be transfered to, navigate to the Collect > Upload Files page

Expand the Advanced Options to see your tenancies filedrop bucket name, this is the value you will copy into the transfer job configuration.

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/agiledata-target-filedrop-1.png
    :width: 800px

Set the frequency of the data transfer

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-5a.png
    :width: 800px

Set the name of the data transfer job, and set the source files to be deleted once transferred (important), then click Create

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-6a.png
    :width: 800px

On successful transfer job creation (and initial sync) you will see

.. image:: https://storage.googleapis.com/docs-agiledata-io/product-guild/collect/amazon-s3/create-transfer-job-success.png
    :width: 800px