Link Your Data Source

Data storage methods supported by ByteNite

If your app requires data to function (as most do), it's essential that your data connections are configured correctly. You'll securely store credentials, test connections, and more in this guide.

Your ByteNite jobs can read and write from popular storage locations, including S3 and Google Cloud buckets.

You will set up external credentials using Secrets, and connect your storage options to jobs using Data Sources.

Once set up, your secrets and data sources can work seamlessly across any jobs. Find below the documentation

Setting Up Secrets

If you're using an authenticated connection like S3, you can securely store your credentials in your ByteNite account, keeping them out of your code.

Use the secretType field to choose your secret provider, such as AWS or Google Cloud Storage, from the supported options.

Send a request to the Secrets endpoint to securely create and store a secret in your account, enabling access to your data sources.

Configure your request by filling in these parameters:

id string

Description:

A given ID for your secret. Hint: Use a string that can easily help remember the credential's scope, permissions, and provider.

Example:

"aws_full_s3_access_key"

name string

Description:

A descriptive name for your secret.

Example:

"John's AWS S3 Full Access Key"

secretType string

Description:

The data source provider of this secret.

Supported Values:

Please see Supported Data Source Options

Example:

"s3"

accessKey string

Description:

Your data source's access key.

Example:

"AKIAIOSFODNN7EXAMPLE"

secretKey string

Description:

Your data source's secret key.

Example:

"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

expiresAt string

Description:

A given expiry date for your secret, in ISO 8601 format.

Example:

"2025-12-29T18:02:27.140Z"

Here's an example of a secret request body:

POST /auth/secrets

{
    "secret": {
        "id": "aws_full_s3_access_key",
        "name": "John's AWS S3 Full Bucket Access",
        "secretType": "s3",
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "expiresAt": "2025-12-29T18:02:27.140Z"    
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Setting Up Data Sources

Configure your data sources to enable input and output with your current storage provider. You'll need to provide a dataSource object for both inputs (also known as "Data Source") and outputs (referred to as "Data Destination"). Once both dataSource objects are set up, you can:

The fields listed below represent the dataSource object. Please note, the params object varies based on the data source type. For data sources requiring access/secret key authentication, provide the ID of the relevant secret within the params body.

Use the bypass data source descriptor to skip data source or destination requirements. This is useful if no input data is needed or when your apps or engines produce no output

dataSourceDescriptor string

Description:

The data source type selector.

Supported Values:

Please see Supported Data Source Optionsfor all data source configuration parameters.

Use bypass to skip a data source.

Example:

"s3"

params object

Description:

The data source parameters.

Supported Properties:

@type A params object type. Please see Supported Data Source Options.
Other parameters. Please refer to the specific guides under Supported Data Source Options.

Example:

{  
    "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
    "name": "/vids/big_buck_bunny.mp4",
    "bucketName": "my-app-data-bucket-12345",
    "cloudRegion": "us-east-2",
    "secret_id": "aws_full_s3_access_key"
}

Here is a full dataSource object example:

POST /customer/jobs/datasource/info

{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "aws_full_s3_access_key"
        }  
    }
}

Supported Data Source Options

Below is a list of currently supported data source connections and parameters for both inputs and outputs.

I/O Data Sources

Input-Only Data Sources

Output-Only Data Sources

PreviousBuild an App NextAWS S3

Last updated 16 days ago

Was this helpful?

Link Your Data Source

Data storage methods supported by ByteNite

If your app requires data to function (as most do), it's essential that your data connections are configured correctly. You'll securely store credentials, test connections, and more in this guide.

Your ByteNite jobs can read and write from popular storage locations, including S3 and Google Cloud buckets.

You will set up external credentials using Secrets, and connect your storage options to jobs using Data Sources.

Once set up, your secrets and data sources can work seamlessly across any jobs. Find below the documentation

Setting Up Secrets

If you're using an authenticated connection like S3, you can securely store your credentials in your ByteNite account, keeping them out of your code.

Use the secretType field to choose your secret provider, such as AWS or Google Cloud Storage, from the supported options.

Send a request to the Secrets endpoint to securely create and store a secret in your account, enabling access to your data sources.

Configure your request by filling in these parameters:

id string

Description:

A given ID for your secret. Hint: Use a string that can easily help remember the credential's scope, permissions, and provider.

Example:

"aws_full_s3_access_key"

name string

Description:

A descriptive name for your secret.

Example:

"John's AWS S3 Full Access Key"

secretType string

Description:

The data source provider of this secret.

Supported Values:

Please see Supported Data Source Options

Example:

"s3"

accessKey string

Description:

Your data source's access key.

Example:

"AKIAIOSFODNN7EXAMPLE"

secretKey string

Description:

Your data source's secret key.

Example:

"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

expiresAt string

Description:

A given expiry date for your secret, in ISO 8601 format.

Example:

"2025-12-29T18:02:27.140Z"

Here's an example of a secret request body:

POST /auth/secrets

{
    "secret": {
        "id": "aws_full_s3_access_key",
        "name": "John's AWS S3 Full Bucket Access",
        "secretType": "s3",
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "expiresAt": "2025-12-29T18:02:27.140Z"    
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Setting Up Data Sources

Test your data source by sending a request to the endpoint.
Attach a data source to a job by sending a request to the endpoint.

Use the bypass data source descriptor to skip data source or destination requirements. This is useful if no input data is needed or when your apps or engines produce no output

dataSourceDescriptor string

Description:

The data source type selector.

Supported Values:

Please see Supported Data Source Optionsfor all data source configuration parameters.

Use bypass to skip a data source.

Example:

"s3"

params object

Description:

The data source parameters.

Supported Properties:

@type A params object type. Please see Supported Data Source Options.
Other parameters. Please refer to the specific guides under Supported Data Source Options.

Example:

{  
    "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
    "name": "/vids/big_buck_bunny.mp4",
    "bucketName": "my-app-data-bucket-12345",
    "cloudRegion": "us-east-2",
    "secret_id": "aws_full_s3_access_key"
}

Here is a full dataSource object example:

POST /customer/jobs/datasource/info

{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "aws_full_s3_access_key"
        }  
    }
}