Google Cloud Storage

Introduction

Google Cloud Storage offers a highly scalable, secure, and durable object storage service designed to handle unstructured data across various use cases. With a unified API and integration across Google Cloud services, it supports efficient data retrieval and storage management. Its tiered storage options—ranging from high-performance to cost-effective archival—make it a versatile solution. Beyond its native ecosystem, Google Cloud Storage is widely adopted for hybrid and multi-cloud strategies due to its interoperability and global availability.

Follow the guide below to set up an account and get HMAC credentials on GCP:

Getting Started with Storage

Setting Up Google Cloud Storage

1. Create a Google Cloud Account:

• Visit the Google Cloud website and click on “Get started for free.”

• Follow the prompts to set up your account, including verifying your email and providing billing information.

2. Access the Google Cloud Console:

• Once your account is active, log in to the Google Cloud Console.

3. Create a Storage Bucket:

• In the console, navigate to the Cloud Storage section.

• Click on “Create bucket.”

• Provide a globally unique name for your bucket.

• Select a location for your bucket (e.g., “US”).

• Choose a default storage class (e.g., “Standard”).

• Set access control to “Uniform” to manage permissions uniformly at the bucket level.

• Click “Create” to finalize the bucket setup.

---

Generating Google Cloud Storage Credentials

To allow ByteNite to interact securely with your Google Cloud Storage buckets, you need to create a service account and generate HMAC (Hash-based Message Authentication Code) credentials.

1. Create a Service Account:

• In the Google Cloud Console, navigate to IAM & Admin > Service Accounts.

• Click on “Create Service Account.”

• Provide a name (e.g., bytenite-service-account) and an optional description.

• Click “Create and Continue.”

2. Assign Permissions to the Service Account:

• Assign the Storage Object Admin role to grant full control over objects in your buckets.

• Click “Continue,” then “Done” to finish creating the service account.

3. Generate HMAC Credentials:

• In the Google Cloud Console, navigate to Cloud Storage > Settings.

• Open the Interoperability tab.

• Under Service Account HMAC, click “Create a key for a service account.”

• Select the service account you created earlier (bytenite-service-account).

• Click “Create Key.”

• The console will display the Access Key and Secret Key.

• Important: Save these credentials securely, as the Secret Key will not be displayed again.

Additional Notes:

• For comprehensive details about creating buckets, refer to the official Google Cloud documentation on creating buckets.

• For detailed information on managing HMAC keys, consult the documentation on HMAC keys.

---

Google Cloud Storage Secret

secretType : gcp

If your Google Cloud bucket requires authentication for read or write access, set up a secret to store your Service Account HMAC credentials securely with ByteNite (see 🔐 Setting up secrets)

Here's an example of a request body of the Secrets endpoint for saving Google Cloud keys:

POST /auth/secrets

{
    "secret": {
        "id": "my_gcp_secret",
        "secretType": "gcp",
        "expiresAt": "2025-12-29T18:02:27.140Z", 
        "accessKey": "GOOG1AB7QD3TY4NSFIZHD4KPB6LVB4F53UJGEZEMRJDXO5PUYDXAOSIKUFNI",
        "name": "GCP Bucket Admin Project 'My App'"
    },
    "secretKey": "aBcDeFgHiJkLmNoPqRsTuVwXyZ1234567890+/ExAmPlEkEy"
}

---

Google Cloud Storage Data Source Object

dataSourceDescriptor : gcp

@type : type.googleapis.com/bytenite.data_source.S3DataSource

Set up your data source with Google Cloud storage using the your previously configured gcp secret and the following params :

@type string

Description:

Use the type.googleapis.com/bytenite.data_source.S3DataSource params type.

bucketName string

Description:

The name of your Google Cloud bucket.

Example:

"my-app-data-bucket-12345"

cloudRegion string

Description:

The Google Cloud bucket's region name.

Example:

"us-west2-b"

name string

Description:

• Usage for Data Sources: The path to your input file following the bucket name.

• Usage for Data Destinations: The path to the output folder following the bucket name. Note: a path will be created if it doesn't exist.

Example:

• Data Source: "/vids/big_buck_bunny.mp4"

• Data Destination: "/vids/encoded/"

secret_id string

Description:

The ID of an existing gcp secret.

Example:

"my_gcp_secret"

Here is an example Google Cloud data source and destination request body:

POST /customer/jobs/{jobId}/datasource

{
    "dataSource": {  
        "dataSourceDescriptor": "gcp", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/big_buck_bunny.mp4",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-west2-b",
            "secret_id": "my_gcp_secret"
        }  
    },
    
    "dataDestination": {  
        "dataSourceDescriptor": "gcp", 
        "params": {  
            "@type": "type.googleapis.com/bytenite.data_source.S3DataSource",  
            "name": "/vids/encoded/",
            "bucketName": "my-app-data-bucket-12345",
            "cloudRegion": "us-west2-b",
            "secret_id": "my_gcp_secret"
        }  
    }
}