# Data Sources

If your app needs data to function (which most do), it’s crucial to **configure your data connections** properly. This guide walks you through securely storing credentials, testing connections, and setting up your data sources.

ByteNite jobs can **read and write from popular storage services**, including Amazon S3 and Google Cloud Storage buckets.

#### How it works

* **Secrets** handle external credentials securely.
* **Data Sources** connect your storage options to your ByteNite jobs.

Once configured, secrets and data sources work seamlessly across any job.

***

## 🔐 Setting up secrets

If you’re using an authenticated service like S3, you’ll store your credentials securely in your ByteNite account—**keeping them out of your code**.

#### Steps:

1. **Use the `secretType` field** to select your secret provider (e.g., AWS, Google Cloud Storage).

   ByteNite supports multiple providers. See: [#supported-data-source-options](#supported-data-source-options "mention").
2. **Send a request to the** [secrets](https://docs.bytenite.com/api-reference/authentication-api/secrets "mention") **endpoint** to securely create and store your secret in your account.

   This unlocks access to your data sources without hardcoding credentials.

Configure your secret request with the following parameters:

<details>

<summary><code>id</code>  <em><strong>string</strong></em></summary>

**Description:**

A given ID for your secret. \
\&#xNAN;***Hint:*** Use a string that can easily help remember the credential's scope, permissions, and provider.

**Example:**

"aws\_full\_s3\_access\_key"

</details>

<details>

<summary><code>name</code>  <em><strong>string</strong></em></summary>

**Description:**

A descriptive name for your secret.

**Example:**

"John's AWS S3 Full Access Key"

</details>

<details>

<summary><code>secretType</code>  <em><strong>string</strong></em></summary>

**Description:**

The data source provider of this secret.

**Supported Values:**

Please see [#supported-data-source-options](#supported-data-source-options "mention")

**Example:**

"s3"

</details>

<details>

<summary><code>accessKey</code>  <em><strong>string</strong></em></summary>

**Description:**

Your data source's access key.

**Example:**

"AKIAIOSFODNN7EXAMPLE"

</details>

<details>

<summary><code>secretKey</code>  <em><strong>string</strong></em></summary>

**Description:**

Your data source's secret key.

**Example:**

"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

</details>

<details>

<summary><code>expiresAt</code>  <em><strong>string</strong></em></summary>

**Description:**

A given expiry date for your secret, in ISO 8601 format.

**Example:**

"2025-12-29T18:02:27.140Z"

</details>

Here's an example of a secret request body:

{% code title="POST /auth/secrets" %}

```json
{
    "secret": {
        "id": "aws_full_s3_access_key",
        "name": "John's AWS S3 Full Bucket Access",
        "secretType": "s3",
        "accessKey": "AKIAXXEXAMPLEEXAMPLEX",
        "expiresAt": "2025-12-29T18:02:27.140Z"    
    },
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
```

{% endcode %}

***

## 🔗 Setting up data sources

To enable **input and output** with your current storage provider, configure your data sources. You’ll provide a **`dataSource` object** for both inputs (called Data Source) and outputs (called Data Destination). Once both are set, you can:

* **Test your `dataSource` object** by sending a request to the /datasource/test endpoint.
* **Attach a Data Source** to a job via the `dataSource` field in the /jobs endpoint.
* **Attach a Data Destination** to a job via the `dataDestination` field in the /jobs endpoint.

**About the dataSource Object**

The fields below define the dataSource object. Please note:

* The `params` object varies depending on the data source type.
* For data sources requiring authentication (like S3), provide the secret ID for the required credentials in the params body.
* Use the `bypass` data source descriptor if your app doesn’t require input or output (e.g., no input data or output files).

<details>

<summary><code>dataSourceDescriptor</code>  <em><strong>string</strong></em></summary>

**Description:**

The data source type selector.

**Supported Values:**

Please see [#supported-data-source-options](#supported-data-source-options "mention")for all data source configuration parameters.&#x20;

Use `bypass` to skip a data source.

**Example:**

"s3"

</details>

<details>

<summary><code>params</code>  <em><strong>object</strong></em></summary>

**Description:**

The data source parameters.

**Supported Properties:**

* `@type` \
  A params object type. Please see [#supported-data-source-options](#supported-data-source-options "mention").
* Other parameters. Please refer to the specific guides under [#supported-data-source-options](#supported-data-source-options "mention").

**Example:**

```json
{  
    "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
    "name": "/vids/big_buck_bunny.mp4",
    "bucketName": "my-app-data-bucket-12345",
    "cloudRegion": "us-east-2",
    "secret_id": "aws_full_s3_access_key"
}
```

</details>

Here is a full `dataSource`  object example:

{% code title="POST /customer/jobs/datasource/info" %}

```json
{
    "dataSource": {  
        "dataSourceDescriptor": "s3", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-east-2",
            "secret_id": "aws_full_s3_access_key"
        }  
    }
}
```

{% endcode %}

***

## ☁️ Supported data source options

Below is a list of currently supported data source connections and parameters for both inputs and outputs.

### I/O data sources

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h3>AWS S3</h3></td><td>secretType : <code>aws</code></td><td>dataSourceDescriptor : <code>aws</code></td><td>@type : <code>type.googleapis.com/bytenite.data_source.S3DataSource</code></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2FwAxpOVilXFrrQLBfMZ7w%2Faws-data-source-cover.webp?alt=media&#x26;token=c6e96b31-5393-4ff0-a6c8-7499d686f033">aws-data-source-cover.webp</a></td><td><a href="data-sources/aws-s3">aws-s3</a></td></tr><tr><td><h3>Google Cloud Storage</h3></td><td>secretType : <code>gcp</code></td><td>dataSourceDescriptor : <code>gcp</code></td><td>@type : <code>type.googleapis.com/bytenite.data_source.S3DataSource</code></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2FrJ7qYz7RzA2zPGn9i6xO%2Fgcp-data-source-cover.webp?alt=media&#x26;token=938e84bc-2404-4eaf-a0f7-95d744cd580f">gcp-data-source-cover.webp</a></td><td><a href="data-sources/google-cloud-storage">google-cloud-storage</a></td></tr><tr><td><h3>Storj</h3></td><td>secretType : <code>storj</code></td><td>dataSourceDescriptor : <code>storj</code></td><td>@type : <code>type.googleapis.com/bytenite.data_source.S3DataSource</code></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2F9o9ZbDMTo1Oa5JQISu3P%2Fstorj-data-sources-cover.webp?alt=media&#x26;token=cd5c532f-37b3-4c7e-83c1-7c1239c39dd3">storj-data-sources-cover.webp</a></td><td><a href="data-sources/storj">storj</a></td></tr></tbody></table>

### Input-only data sources

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h3>Generic HTTP</h3></td><td></td><td>dataSourceDescriptor : <code>url</code></td><td>@type: <code>type.googleapis.com/bytenite.data_source.HttpDataSource</code></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2F3ay21a7rJ03fjZAeoNck%2Fhttp-data-source-cover.webp?alt=media&#x26;token=81636a37-5f33-40f4-bad1-3c98ddd0a565">http-data-source-cover.webp</a></td><td><a href="data-sources/http">http</a></td></tr><tr><td><h3>File Upload</h3></td><td></td><td>dataSourceDescriptor : <code>file</code></td><td>@type: <code>type.googleapis.com/bytenite.data_source.LocalFileDataSource</code></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2FHFCywIR1svwTUXIgQAzX%2Ffile-upload-data-source-cover.webp?alt=media&#x26;token=a52cc0be-ae25-470f-a93a-169aa6a53c1b">file-upload-data-source-cover.webp</a></td><td><a href="data-sources/file-upload">file-upload</a></td></tr></tbody></table>

### Output-only data sources

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h3>Temporary Bucket</h3></td><td></td><td>dataSourceDescriptor : <code>bucket</code></td><td></td><td><a href="https://1271894904-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FC4oKwqyUo55Lw35MjazY%2Fuploads%2F4fKJ0u5ivGPRIpswSFpr%2Ftemporary-bucket-data-source-cover.webp?alt=media&#x26;token=cb144e37-8a62-4af7-af56-0bdad96e5fbf">temporary-bucket-data-source-cover.webp</a></td><td><a href="data-sources/temporary-bucket">temporary-bucket</a></td></tr></tbody></table>