What is a CDC pipeline?

A CDC pipeline is a pipeline architected around a stream of database change events, inserts, updates, and deletes, rather than periodic full-table reads. Each change becomes a discrete event that flows through extraction, transformation, and load stages as it happens, instead of waiting for the next scheduled batch.

How the pattern works

A CDC pipeline typically starts with a log reader that tails a source database's transaction log and turns each row-level change into an event on a stream, often Kafka or a similar broker. Downstream consumers subscribe to that stream and apply changes to a target table incrementally, event by event, instead of re-reading the source. The pipeline shape is push-based and continuous rather than pull-based and scheduled.

Why teams build pipelines this way

A CDC pipeline keeps a target system's data close to real time without the cost of repeatedly scanning a large source table, and it gives downstream consumers a clean, ordered record of exactly what changed and when. That matters most for tables that update constantly and where a target system, a search index, a cache, a replica, needs to stay in sync within seconds rather than hours.

Where OptimaFlo fits this pattern

OptimaFlo doesn't build CDC pipelines against source databases today: source connectors read through table reflection and SELECT queries on a schedule, not by tailing a write-ahead log or binary log, so there's no continuous change-event stream coming out of a connected Postgres or MySQL instance. Inside the platform, though, OptimaFlo applies the same pipeline-around-change-events idea to a different layer: a snapshot-diffing process compares Iceberg snapshots between medallion runs, and Clean and Ready rebuilds only process the rows that changed since the last run. That's the CDC pipeline pattern, building around what changed instead of a full rescan, applied to data already inside the lakehouse, not to the source connection itself.

Related terms

• Change data capture: The underlying technique a CDC pipeline is built on. See how it works and how OptimaFlo ingests source data today.
• Data pipeline: The general shape a CDC pipeline specializes: extract, transform, load, built around change events instead of scheduled batches.