Fondant architecture overview#
At a high level, Fondant consists of three main parts:
/coredirectory serves as the foundational backbone of Fondant, encompassing essential shared functionalities:
component_spec.py: Defines the component spec class which is used to define the specification of a component. Those specifications mainly include the component image location, arguments, columns it consumes and produces.
manifest.pyDescribes dataset content, facilitating reference passing between components. It evolves during pipeline execution and aids static evaluation.
schema.pyDefines the Type class, used for dataset data type definition.
/schemaDirectory Containing JSON schema specifications for the component spec and manifest.
/componentdirectory which contains modules for implementing Fondant components:
component.py: Defines the
Componentclass which is the base class for all Fondant components. This is used to defines interfaces for different component types (Load, Transform, Write) across different data processing frameworks (Dask, Pandas, ...). The user should inherit from those classes to implement their own components.
data_io.py: Defines the
DataIOclass which is used to define the reading and writing operations from/to a dataset. This includes optimizing the reading and writing operations as well as selecting which columns to read/write according to the manifest.
executor.py: Defines the
Executorclass which is used to define the execution of a component. This includes parsing the component arguments, executing the component and evolving/writing the manifest. Each executor subclasses a corresponding
Componentclass to implement the execution logic for a specific component type.
/pipelinedirectory which contains the modules for implementing a Fondant pipeline.
pipeline.py: Defines the
Pipelineclass which is used to define the pipeline graph and the pipeline run. The implemented class is then consumed by the compiler to compile to a specific pipeline runner. This module also implements the
ComponentOpclass which is used to define the component operation in the pipeline graph.
compiler.py: Defines the
Compilerclass which is used to define the compiler that compilers the pipeline graph for a specific runner.
runner.py: Defines the
Runnerclass which is used to define the runner that executes the compiled pipeline graph.
Additional modules in Fondant include:
cli.py: Defines the CLI for interacting with Fondant. This includes the
fondantcommand line tool which is used to build components, compile and run pipelines and explore datasets.
explore.py: Runs the explorer which is a web application that allows the user to explore the content of a dataset.
build.py: Defines the
buildcommand which is used to build and publish a component.
testing.py: Contains common testing utilities for testing components and pipelines.