Skip to main content

Pipeline Builder

Build Pipelines Visually

Drag-and-drop canvas for medallion architecture. Add Ingestion, Cleaning, Aggregation, and Destination nodes: connect them, write SQL (or let AI do it), and deploy to Airflow.

Node Types

Each node represents a step in your data pipeline.

Ingestion Node

Raw

Ingests raw data from a connected data source into an Apache Iceberg table. Zero transformations: stores an exact copy with full history.

Configuration

  • Linked data source (GCS, BigQuery, REST API)
  • Target Iceberg table name
  • Write mode (append or overwrite)
  • Schema evolution settings

Cleaning Node

Clean

Runs SQL transformations against Ingestion data. Let the Analytics Engineer generate cleaning SQL from natural language.

Configuration

  • SQL transformation (manual or AI-generated)
  • Preview results before saving
  • Target table name
  • Deduplication keys

Aggregation Node

Metrics

Aggregates Cleaning data into business-ready metrics. Star schemas, KPIs, and dimensional models.

Configuration

  • Aggregation SQL
  • Incremental vs. full refresh
  • Target metrics table
  • Preview aggregated results

Destination Node

Export

Exports Aggregation layer data to an external system. Configure the target warehouse, storage bucket, or database for downstream consumption.

Configuration

  • Destination type (BigQuery, Snowflake, GCS, S3)
  • Output table or file path
  • Write mode (append or overwrite)
  • File format for storage destinations

Canvas Features

Everything you need to build, test, and iterate on pipelines.

Analytics Engineer
Describe what you want in plain English. The Analytics Engineer generates SQL, validates column references, and shows you a preview, all before saving.
Live Preview
Click Preview on any node to see sample results. The preview runs a CTE chain through all upstream nodes so you see accurate output.
Data Engineer
Describe a full pipeline in one prompt. The Data Engineer decomposes it into tasks, generates SQL per node, and lays out the canvas.

How SQL Generation Works

When you describe a transformation in natural language, the platform runs a multi-step pipeline to produce correct, safe SQL:

1

Schema Context

The Analytics Engineer reads your upstream table schema: column names, types, and sample data, so it knows what's available.

2

SQL Generation

Your natural language request is sent to the LLM with the full schema context. It generates a SQL transformation.

3

Validation

Every generated query goes through automated validation: column references, type safety, security rules, and syntax are all checked before execution.

4

Auto-Correction

If an error occurs at runtime, the platform automatically detects and corrects the issue, then retries. Most errors resolve without any manual intervention.

5

Preview

You see sample results before committing. Edit the SQL manually if needed, or re-prompt the Analytics Engineer.

Scheduling & Execution

Run once or schedule to run on a cadence.

One-Click Execution

Run the full pipeline immediately from the canvas toolbar.

Scheduled Runs

Set hourly, daily, or weekly schedules. OptimaFlo generates an Apache Airflow DAG behind the scenes.

Backfills

Re-process historical date ranges when you update transformation logic. Runs sequentially to avoid Iceberg write conflicts.

Execution Monitoring

See each run move through the Raw, Clean, and Ready layers. Run history shows status, duration, row count, and the error for any failed run.

What's Next

Explore the AI that powers the builder.

AI Agents

Meet the Analytics Engineer, Manager, and Data Engineer

BI & Dashboards

Turn Aggregation metrics into charts and dashboards

Build without the boilerplate

Drag, connect, deploy. That's it.

Build a PipelineContact

Visual pipeline building with AI-generated SQL and one-click Airflow scheduling.