Data Sources

Data storage methods supported by ByteNite

If your app needs data to function (which most do), it’s crucial to configure your data connections properly. This guide walks you through securely storing credentials, testing connections, and setting up your data sources.

ByteNite jobs can read and write from popular storage services, including Amazon S3 and Google Cloud Storage buckets.

How it works

Secrets handle external credentials securely.
Data Sources connect your storage options to your ByteNite jobs.

Once configured, secrets and data sources work seamlessly across any job.

🔐 Setting up secrets

If you’re using an authenticated service like S3, you’ll store your credentials securely in your ByteNite account—keeping them out of your code.

Steps:

Use the secretType field to select your secret provider (e.g., AWS, Google Cloud Storage).
ByteNite supports multiple providers. See: ☁️ Supported data source options.
Send a request to the Secrets endpoint to securely create and store your secret in your account.
This unlocks access to your data sources without hardcoding credentials.

Configure your secret request with the following parameters:

id string

Description:

A given ID for your secret. Hint: Use a string that can easily help remember the credential's scope, permissions, and provider.

Example:

"aws_full_s3_access_key"

name string

Description:

A descriptive name for your secret.

Example:

"John's AWS S3 Full Access Key"

secretType string

Description:

The data source provider of this secret.

Supported Values:

Please see ☁️ Supported data source options

Example:

"s3"

accessKey string

Description:

Your data source's access key.

Example:

"AKIAIOSFODNN7EXAMPLE"

secretKey string

Description:

Your data source's secret key.

Example:

"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

expiresAt string

Description:

A given expiry date for your secret, in ISO 8601 format.

Example:

"2025-12-29T18:02:27.140Z"

Here's an example of a secret request body:

POST /auth/secrets

{
    "secret": {
        "id": "aws_full_s3_access_key",
        "name": "John's AWS S3 Full Bucket Access",
        "secretType": "s3",
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "expiresAt": "2025-12-29T18:02:27.140Z"    
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

🔗 Setting up data sources

To enable input and output with your current storage provider, configure your data sources. You’ll provide a dataSource object for both inputs (called Data Source) and outputs (called Data Destination). Once both are set, you can:

Test your dataSource object by sending a request to the /datasource/test endpoint.
Attach a Data Source to a job via the dataSource field in the /jobs endpoint.
Attach a Data Destination to a job via the dataDestination field in the /jobs endpoint.

About the dataSource Object

The fields below define the dataSource object. Please note:

The params object varies depending on the data source type.
For data sources requiring authentication (like S3), provide the secret ID for the required credentials in the params body.
Use the bypass data source descriptor if your app doesn’t require input or output (e.g., no input data or output files).

dataSourceDescriptor string

Description:

The data source type selector.

Supported Values:

Please see ☁️ Supported data source optionsfor all data source configuration parameters.

Use bypass to skip a data source.

Example:

"s3"

params object

Description:

The data source parameters.

Supported Properties:

@type A params object type. Please see ☁️ Supported data source options.
Other parameters. Please refer to the specific guides under ☁️ Supported data source options.

Example:

{  
    "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
    "name": "/vids/big_buck_bunny.mp4",
    "bucketName": "my-app-data-bucket-12345",
    "cloudRegion": "us-east-2",
    "secret_id": "aws_full_s3_access_key"
}

Here is a full dataSource object example:

POST /customer/jobs/datasource/info

{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "aws_full_s3_access_key"
        }  
    }
}

☁️ Supported data source options

Below is a list of currently supported data source connections and parameters for both inputs and outputs.

I/O data sources

Input-only data sources

Output-only data sources

PreviousByteNite Dev CLI NextAWS S3

Last updated 3 days ago

Was this helpful?

Data Sources

Data storage methods supported by ByteNite

ByteNite jobs can read and write from popular storage services, including Amazon S3 and Google Cloud Storage buckets.

How it works

Secrets handle external credentials securely.
Data Sources connect your storage options to your ByteNite jobs.

Once configured, secrets and data sources work seamlessly across any job.

🔐 Setting up secrets

If you’re using an authenticated service like S3, you’ll store your credentials securely in your ByteNite account—keeping them out of your code.

Steps:

Use the secretType field to select your secret provider (e.g., AWS, Google Cloud Storage).
ByteNite supports multiple providers. See: ☁️ Supported data source options.
Send a request to the Secrets endpoint to securely create and store your secret in your account.
This unlocks access to your data sources without hardcoding credentials.

Configure your secret request with the following parameters:

id string

Description:

A given ID for your secret. Hint: Use a string that can easily help remember the credential's scope, permissions, and provider.

Example:

"aws_full_s3_access_key"

name string

Description:

A descriptive name for your secret.

Example:

"John's AWS S3 Full Access Key"

secretType string

Description:

The data source provider of this secret.

Supported Values:

Please see ☁️ Supported data source options

Example:

"s3"

accessKey string

Description:

Your data source's access key.

Example:

"AKIAIOSFODNN7EXAMPLE"

secretKey string

Description:

Your data source's secret key.

Example:

"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

expiresAt string

Description:

A given expiry date for your secret, in ISO 8601 format.

Example:

"2025-12-29T18:02:27.140Z"

Here's an example of a secret request body:

POST /auth/secrets

{
    "secret": {
        "id": "aws_full_s3_access_key",
        "name": "John's AWS S3 Full Bucket Access",
        "secretType": "s3",
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "expiresAt": "2025-12-29T18:02:27.140Z"    
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

🔗 Setting up data sources

Test your dataSource object by sending a request to the /datasource/test endpoint.
Attach a Data Source to a job via the dataSource field in the /jobs endpoint.
Attach a Data Destination to a job via the dataDestination field in the /jobs endpoint.

About the dataSource Object

The fields below define the dataSource object. Please note:

The params object varies depending on the data source type.
For data sources requiring authentication (like S3), provide the secret ID for the required credentials in the params body.
Use the bypass data source descriptor if your app doesn’t require input or output (e.g., no input data or output files).

dataSourceDescriptor string

Description:

The data source type selector.

Supported Values:

Please see ☁️ Supported data source optionsfor all data source configuration parameters.

Use bypass to skip a data source.

Example:

"s3"

params object

Description:

The data source parameters.

Supported Properties:

@type A params object type. Please see ☁️ Supported data source options.
Other parameters. Please refer to the specific guides under ☁️ Supported data source options.

Example:

{  
    "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
    "name": "/vids/big_buck_bunny.mp4",
    "bucketName": "my-app-data-bucket-12345",
    "cloudRegion": "us-east-2",
    "secret_id": "aws_full_s3_access_key"
}

Here is a full dataSource object example:

POST /customer/jobs/datasource/info

{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "aws_full_s3_access_key"
        }  
    }
}