Building Blocks

Overview of ByteNite's core components and functionalities

ByteNite's building blocks let you assemble your code into reusable components that you can version and add to your code development workflows.

Job Lifecycle

We've separated a job lifecycle into three building blocks corresponding to its stages: Partitioning Engine, App, Assembling Engine.

An illustration of a ByteNite job's lifecycle, spanning from data ingestion to data export.

1

Data Pre-Processing - Partitioning Engine

For distributed jobs relying on data partitioning, sharding, or other similar techniques, a preprocessing step should occur at the very beginning of the job, when data is read from the source. This is what your Data Partitioning Engines are designed for. They let you define data pre-processing and segmentation logic that will run at the start of the jobs and that you can reuse across your apps, allowing you to write apps that execute simultaneously and efficiently.

Note that we provide a default passthrough partitioning engine, which lets you work with the raw data when data partitioning is not needed.

2

Task Execution - App

Each data chunk passed by the partitioner will be processed by the code provided in your app. Apps are where you run tasks, whether it's table operations, image processing, or complex ML linear algebra. Remember, your app code may map data to different cores on the same machine, but it shouldn't handle data ingestion or export as this is managed by Data Engines and Data Sources.

3

Data Post-Processing - Assembling Engine

Processed tasks may be assembled according to specific instructions or filtering scripts that combine them into a single data file. This can be as simple as zipping the task outputs, or involve more complex assembling workflows. You can accomplish all of this with Data Assembling Engines.

If your job doesn’t require a data assembling step, use the passthrough Data Assembling engine to directly output what's returned by your app, similar to the partitioner passthrough option.

Building Block Guides

Delve into the following guides to understand how to use Apps, Data Engines, Job Templates, and Parameter Schemas to build your applications.

Apps let you submit code to run on ByteNite's infrastructure.

Build a Data Partitioning Engine to tell your app how to pre-process input data.

Add a Data Assembling Engine to define chunk merging logic.

Use Job Templates to harmonize versions across apps and data engines.