What is data lineage?

Data lineage is a record of where a piece of data came from and what touched it along the way: which source it was ingested from, which transformations ran on it, and which downstream tables or metrics depend on it. It answers two questions: what feeds this table, and what breaks if I change it.

Why it matters

Lineage lets you check the blast radius of a change before you make it: if you rename a column or drop a table, lineage shows what else reads from it. It also shortens debugging: when a number on a dashboard looks wrong, lineage traces it back to the source table and the transform that produced it.

Table-level versus column-level

Most lineage tools, OptimaFlo included, track lineage at the table level: which tables and pipelines feed which other tables. Column-level lineage goes one step further, tracing which specific source column maps to which destination column. It's a finer grain and harder to keep accurate as transforms get more complex.

How OptimaFlo tracks lineage

OptimaFlo tracks lineage at two levels: which pipelines feed a given metric, and which SQL files read from or write to a given table. Both are computed from data already in the platform, pipeline node outputs and SQL file source and target tables, rather than a separate lineage graph that can drift out of sync. This is table and pipeline-level lineage, not column-level: it won't show that a specific source column maps to a specific destination column. The table lineage view is available on the Scale plan and above.

Related terms

• Data pipeline: What lineage traces: the sequence of steps that produced a table.
• Apache Iceberg: The table format Raw, Clean, and Ready tables are stored in, which lineage traces across.
• Time travel queries: A related but different capability: querying a table's history by snapshot, not tracing what fed it.