AWS S3

Read and write files from an AWS S3 or a S3-compatible storage

Introduction to AWS S3

Amazon S3 (Simple Storage Service) is a highly scalable and secure cloud storage service provided by Amazon Web Services (AWS). It's widely used for storing and retrieving data, making it a popular choice for most cloud-based workflows.

Signing Up for AWS S3

To use AWS S3 with ByteNite, you'll need an AWS account and an S3 bucket. Here's a quick guide to getting started:

  1. Visit the AWS Website: Head over to the AWS website.
  2. Create an AWS Account: Click the "Create an AWS Account" button and follow the on-screen instructions.
  3. Sign In to the AWS Management Console: Once your account is ready, log in at the AWS Management Console.
  4. Create an S3 Bucket: In the AWS Management Console, go to the S3 dashboard, click "Create bucket", and follow the prompts to set up your bucket.

Generating S3-Compatible Credentials

To integrate AWS S3 with ByteNite, you'll need S3-compatible credentials. Here's how to generate these credentials:

  1. Access IAM (Identity and Access Management): In the AWS Management Console, go to "Services" and select "IAM" under the "Security, Identity, & Compliance" section.
  2. Create a New User: In IAM, click "Users", then "Add user".
  3. User Name: Name your user (e.g., "ByteNiteUser").
  4. Attach Permissions Policies: Assign custom policies to your user, including read and write access to your bucket, or use ready-made policies like "AmazonS3FullAccess".
  5. Review and Create: Check the details and click "Create user."
  6. Generate Keys: On the user's Summary page, click "Create access key". Choose "Third-party service" under Use Case, and you'll receive an "Access key" and "Secret access key". These are your S3 credentials. Be sure to save them securely.

With your AWS S3 bucket and credentials in hand, you're all set to connect AWS S3 with ByteNite!

For more info on AWS S3, refer to the official AWS S3 documentation.


Data Destination

You can upload your output files from ByteNite to an AWS S3 bucket that you have access to. If your bucket is private, make sure you have the necessary permissions and AWS credentials. Specifically, for data destinations, you must grant your AWS credentials at least write access to the designated S3 bucket.

Configuration

API Endpoints

Create a new job   •    Set job data source

dataDestination object:

KeyValueMandatory
dataSourceDescriptor"s3"*
params@type"type.googleapis.com/bytenite.data_source.S3DataSource"*
paramsnameThe output folder path within your S3 bucket. Note: if the output path doesn't exist, it will be created*
paramsaccessKeyYour AWS Access Key ID*
paramssecretKeyYour AWS Secret Access Key*
paramsbucketNameThe name of your AWS S3 bucket*
paramscloudRegionThe AWS region where your S3 bucket is located*

Example

{
  "dataDestination": {  
    "dataSourceDescriptor": "s3",  
    "params": {  
        "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
        "name": "/output/folder/",  
        "accessKey": "your_access_key",  
        "secretKey": "your_secret_key",  
        "bucketName": "your_bucket_name",  
        "cloudRegion": "selected_region"
    }  
	}
}