Postgres to Iceberg: land PostgreSQL data as open Iceberg tables in your own AWS account

Connecting Postgres to Iceberg on AWS through OptimaFlo takes one thing beyond the usual: a read-only database user. There is no BigQuery dataset to provision, no GCP project to keep in sync, because Apache Iceberg in your own S3 is the finished destination, not a hop toward one. Initial setup runs about 15 to 20 minutes; after that, syncs are scheduled pipeline runs, not a one-time migration.

What actually happens

OptimaFlo lands the data as open Apache Iceberg tables in your own AWS account first (the Raw layer), and for this connector that is where it stays unless you choose to export further. The AI data team builds Clean and Ready models on top of Raw, the same layered process every source goes through. You get versioned, queryable history in Iceberg from the first run, owned entirely inside your own AWS account rather than a table living only inside a third-party warehouse's project.

The Postgres connector itself uses SQLAlchemy 2.0's reflection API (inspect()) to read column names and types directly from the database catalog, not from a hand-maintained schema file. It accepts either a full postgres:// connection string or discrete host, port, database, and credential fields, and it supports a raw SQL query as an alternative to a bare table name for sources that are not a single clean table. All synchronous database calls run on a thread pool so they never block the event loop.

Full refresh, not CDC

This is the honest limit worth stating plainly: the connector does not do log-based change data capture. Every run issues a fresh SELECT, either against the whole table or against whatever query you configured. For most analytics workloads (events, orders, daily snapshots) that is fine, especially on an incremental schedule. If you need row-level change tracking on a large, frequently-updated table, the practical pattern today is a custom query with a WHERE updated_at > ... filter, which the connector supports as a table alternative, not something the pipeline builder generates for you automatically.

Schema and identifiers

Schema inference requires either a selected table or a query. If neither is set, the connector fails loud with a clear error rather than silently returning nothing. Table and schema names are validated against a safe identifier pattern before they touch SQL, and the actual data read goes through SQLAlchemy's Table/select() expression API instead of string-interpolated queries, so a data source name with unusual characters cannot become a SQL injection vector.

If you still want a warehouse on top

Iceberg in your own AWS account does not rule out Snowflake or another database as a downstream target. Once Ready-layer tables exist, the same destination export every other connector uses can push a Gold table to Snowflake, another Postgres, or MySQL, in append, overwrite, or upsert mode. BigQuery is the one exception here: it needs a GCP project to land in, so it is only reachable if you also run or provision a GCP destination.

What is not supported

There is no built-in binlog-style or WAL-based replication, no automatic incremental watermarking, and schema-from-query is not implemented (a query-based source returns an empty schema rather than a guess). If your Postgres table needs sub-minute freshness in Iceberg, this is not that tool yet.