Pipeline documentation

A pipeline in Antiloop Springbok consists of Extractors, Transformers, Loaders, and Hooks. Pipelines are defined in XML and can be contained in a single file or split across multiple files for better organization.

By default, pipeline files are stored in the app/config/Zed/springbok directory. This location can be customized by modifying the SpringbokConfig::PIPELINE_FOLDER setting in your project’s config_XXX.php file:

<?php

$config[SpringbokConfig::PIPELINE_FOLDER] = APPLICATION_ROOT_DIR . '/config/Zed/springbok';

Pipeline file example:

<springbok>
    <pipeline name="pipeline_name">
        <extractor>
            <csv-file filename="path/to/file.csv" rowEntryName="rowEntryName" bulkSize="100" />
        </extractor>
        <transformers>
            <copy from="currentName" to="transformedName" />
        </transformers>
        <loaders>
            <glossary keyField="glossary_key" />
        </loaders>
    </pipeline>
</springbok>

You can define multiple pipelines within a single file by adding additional <pipeline> elements. However, each pipeline must have a unique name, even if they are spread across different files.

A pipeline consists of one extractor, multiple transformers, and multiple loaders. Transformers can be reused multiple times within the same pipeline.

To execute a pipeline, use the following console command: console springbok:run *pipeline_name*

This command processes the defined data flow, extracting, transforming, and loading data based on the configured steps.

For more information about: Hooks