AWS S3

Introduction

Amazon S3 (Simple Storage Service) is a scalable, high-speed, web-based cloud storage service designed for online backup and archiving of data and applications. It offers secure, durable, and highly-scalable object storage, making it ideal for a wide range of use cases, from data lakes and mobile applications to backup and restore operations.

Follow the guide below to set up an account and get IAM credentials on AWS S3:

Getting Started with S3

Getting Started with AWS S3

1. Create an AWS Account:

Visit aws.amazon.com, click “Create an AWS Account”, and follow the steps to register. Provide payment information and verify your identity to activate the account.

2. Access S3:

Log in to the AWS Console, search for S3, and navigate to the S3 dashboard.

3. Create a Bucket:

• Click “Create Bucket” in the S3 dashboard.

• Provide a unique bucket name and choose a region (e.g., us-east-1).

• Keep public access blocked unless specific use cases require otherwise.

• Finalize the setup by clicking “Create Bucket”.

---

Setting Up IAM User for S3 Access

1. Open IAM Service:

In the AWS Console, search for IAM and go to Users.

2. Create a User:

• Click “Add Users”, provide a username (e.g., s3-bytenite-user), and enable Programmatic Access.

• Attach the AmazonS3FullAccess policy or create a custom policy for specific bucket access.

3. Generate Credentials:

• Complete the user creation process and download the Access Key ID and Secret Access Key.

• Save these credentials securely; they will not be displayed again.

---

S3 Secret

secretType : s3

If your S3 bucket requires authentication for read or write access, set up a secret to store your S3 credentials securely with ByteNite (see 🔐 Setting up secrets)

Here's an example of a request body of the Secrets endpoint for saving S3 keys:

POST /auth/secrets

{
    "secret": {
        "id": "my_aws_secret",
        "secretType": "s3",
        "expiresAt": "2025-12-29T18:02:27.140Z", 
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "name": "My AWS Full Bucket Access"
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

---

S3 Data Source Object

dataSourceDescriptor : s3

@type : type.googleapis.com/bytenite.data_source.S3DataSource

Set up your data source with S3 using the your previously configured S3 secret and the following params :

@type string

Description:

Use the type.googleapis.com/bytenite.data_source.S3DataSource params type.

bucketName string

Description:

The name of your S3 bucket.

Example:

"my-app-data-bucket-12345"

cloudRegion string

Description:

The S3 bucket's region name.

Example:

"us-east-2"

name string

Description:

• Usage for Data Sources: The path to your input file following the bucket name.

• Usage for Data Destinations: The path to the output folder following the bucket name. Note: a path will be created if it doesn't exist.

Example:

• Data Source: "/vids/big_buck_bunny.mp4"

• Data Destination: "/vids/encoded/"

secret_id string

Description:

The ID of an existing s3 secret.

Example:

"my_aws_secret"

Here is an example S3 data source and destination request body:

POST /customer/jobs/{jobId}/datasource

{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "my_aws_secret"
        }  
    },
    
    "dataDestination": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/encoded/",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "my_aws_secret"
        }  
    }
}