What is data ingestion?

Data ingestion is the process of pulling data out of a source system, a database, an API, a file in cloud storage, and landing it somewhere it can be processed. It's the first step in any pipeline, and it can run as a one-time load, a scheduled batch, or a continuous stream.

Batch, scheduled, and streaming ingestion

A one-time load pulls everything a source has, once, usually for an initial backfill. A scheduled batch runs on a recurring interval, like every hour, pulling whatever's new or changed since the last run. A continuous stream keeps a connection open and ingests records as they arrive, which is what change data capture and message queues like Kafka are built for.

Why schema handling matters at this step

A source's shape isn't fixed. A new column shows up, a field gets renamed, a type gets widened. Ingestion is where that first gets noticed, and how it's handled there determines whether the rest of the pipeline breaks or adapts.

How OptimaFlo implements this

The Ingestion Engineer role connects to a source, reads its structure, and infers a schema automatically, handling auth, schema, and validation so a new source is live in minutes. OptimaFlo's source connectors read with a query against the source each run, a full table pull or a scoped query, rather than tailing a continuous change log. Whatever lands gets schema evolution applied automatically as part of the Raw-layer write, so a source adding a column doesn't break the run.

Related terms

• Data pipeline: The full system ingestion feeds into, extraction plus transformation and loading.
• Change data capture: A streaming technique for ingesting only what changed, instead of a full pull.
• Medallion architecture: The Raw layer is where ingested data lands, unchanged, before any transform runs.