Building Blocks

Overview of ByteNite's core components and functionalities

ByteNite's building blocks are designed to help you structure your distributed workloads into modular, reusable components. These components—Partitioners, Apps, and Assemblers—let you focus on your core logic without worrying about the complexities of distributed execution, fault tolerance, retries, or logging. We've got that covered.

At a high level, ByteNite breaks down a distributed job's lifecycle into three phases:

1. Partitioning Engine: Handles input downloading, pre-processing, and task creation.

2. App: Executes the core logic for each individual task.

3. Assembling Engine: Collects and merges the results from individual tasks.

Each component is fully customizable and versionable, enabling you to build flexible pipelines that fit your specific needs.

Explore the following guides to learn how to develop your own ByteNite Apps, Partitioning Engines, and Assembling Engines to create distributed workflows tailored to your needs.

Apps let you submit code to run on ByteNite's infrastructure.

Build a Data Partitioning Engine to tell your app how to pre-process input data.

Add a Data Assembling Engine to define chunk merging logic.

Use Job Templates to harmonize versions across apps and data engines.