Introduction to AWS S3

Amazon S3 (Simple Storage Service) is a highly scalable and secure cloud storage service provided by Amazon Web Services (AWS). It's widely used for storing and retrieving data, making it a popular choice for most cloud-based workflows.

Signing Up for AWS S3

To use AWS S3 with ByteNite, you'll need an AWS account and an S3 bucket. Here's a quick guide to getting started:

1. Visit the AWS Website: Head over to the AWS website.
2. Create an AWS Account: Click the "Create an AWS Account" button and follow the on-screen instructions.
3. Sign In to the AWS Management Console: Once your account is ready, log in at the AWS Management Console.
4. Create an S3 Bucket: In the AWS Management Console, go to the S3 dashboard, click "Create bucket", and follow the prompts to set up your bucket.

Generating S3-Compatible Credentials

To integrate AWS S3 with ByteNite, you'll need S3-compatible credentials. Here's how to generate these credentials:

1. Access IAM (Identity and Access Management): In the AWS Management Console, go to "Services" and select "IAM" under the "Security, Identity, & Compliance" section.
2. Create a New User: In IAM, click "Users", then "Add user".
3. User Name: Name your user (e.g., "ByteNiteUser").
4. Attach Permissions Policies: Assign custom policies to your user, including read and write access to your bucket, or use ready-made policies like "AmazonS3FullAccess".
5. Review and Create: Check the details and click "Create user."
6. Generate Keys: On the user's Summary page, click "Create access key". Choose "Third-party service" under Use Case, and you'll receive an "Access key" and "Secret access key". These are your S3 credentials. Be sure to save them securely.

With your AWS S3 bucket and credentials in hand, you're all set to connect AWS S3 with ByteNite!

For more info on AWS S3, refer to the official AWS S3 documentation.

Data Origin ⬇️

You can use input files from an AWS S3 bucket. If your bucket is private, make sure you have the necessary permissions and credentials. Specifically, for data origins, you must grant your credentials at least read access to the designated bucket.

Configuration

dataSource object:

Example

{
  "dataSource": {  
    "dataSourceDescriptor": "S3",  
    "params": {  
        "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
        "name": "/path/to/file.obj",  
        "accessKey": "your_access_key",  
        "secretKey": "your_secret_key",  
        "bucketName": "your_bucket_name",  
        "cloudRegion": "selected_region"
    }  
	}
}

Data Destination ⬆️

You can upload your output files to an AWS S3 bucket. If your bucket is private, make sure you have the necessary permissions and AWS credentials. Specifically, for data destinations, you must grant your AWS credentials at least write access to the designated S3 bucket.

Configuration

dataDestination object:

Example

{
  "dataDestination": {  
    "dataSourceDescriptor": "s3",  
    "params": {  
        "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
        "name": "/output/folder/",  
        "accessKey": "your_access_key",  
        "secretKey": "your_secret_key",  
        "bucketName": "your_bucket_name",  
        "cloudRegion": "selected_region"
    }  
	}
}