BYOC Documentation
AWS BYOC Setup Guide
Deploy OptimaFlo's BYOC (Bring Your Own Cloud) agent to your AWS account. Your data stays in your infrastructure. OptimaFlo never stores it.
New to BYOC? Start with the BYOC overview to see how it works and how it compares to a fully-hosted platform.
Quick Start Overview
Step 1: Connect Your AWS Account
AWS has no per-service "enable this API" step the way GCP does. Instead, OptimaFlo authenticates with a single cross-account IAM role that you create with a CloudFormation template. OptimaFlo assumes that role for short sessions (one hour maximum) to run Terraform, it never asks for or stores long-lived AWS access keys.
The trust policy is gated by a per-organization external ID (confused-deputy protection): only OptimaFlo's platform AWS account, presenting the exact external ID issued to your organization, can assume the role. A stolen role ARN alone cannot be used by anyone else.
How the connection flow works
- 1Start the connection from the BYOC dashboard. OptimaFlo generates a one-time external ID for your organization and shows you its platform AWS account ID.
- 2Download and deploy the CloudFormation template in your own AWS account, with the external ID and OptimaFlo's account ID as parameters.
- 3Paste the role ARN from the stack output back into the dashboard. OptimaFlo validates it with a real STS AssumeRole call before marking the connection as connected.
Deploy the CloudFormation Stack via the CLI
# Values shown to you in the BYOC dashboard after you start the connection
export OPTIMAFLO_ACCOUNT_ID="<shown in the dashboard>"
export EXTERNAL_ID="<shown in the dashboard>"
aws cloudformation deploy \
--template-file optimaflo-byoc-role.yaml \
--stack-name optimaflo-byoc \
--parameter-overrides \
OptimaFloAccountId=$OPTIMAFLO_ACCOUNT_ID \
ExternalId=$EXTERNAL_ID \
--capabilities CAPABILITY_NAMED_IAM
# Read the role ARN back out to paste into the dashboard
aws cloudformation describe-stacks \
--stack-name optimaflo-byoc \
--query "Stacks[0].Outputs[?OutputKey=='RoleArn'].OutputValue" \
--output textNote: the template downloads from /api/byoc/cloudformation-template in the dashboard so you always get the version OptimaFlo's backend currently expects. Deploying an old copy from a previous session can leave you on a stale template, see Common Issues below.
What the stack creates
The template creates two IAM roles in your account, both attached to the same managed policy:
optimaflo-byoc, the cross-account role OptimaFlo assumes via STS to run and manage your deployment.codebuild-optimaflo-role, the service role a CodeBuild project in your own account assumes to actually run Terraform. Terraform executes inside your account, not OptimaFlo's.
| Policy Statement | Actions | Purpose |
|---|---|---|
DeployPipeline | codebuild:*, sts:GetCallerIdentity | Run and monitor the CodeBuild project that executes Terraform |
Storage | s3:* | Create and manage the Iceberg warehouse bucket |
TerraformStateLock | dynamodb:DescribeTable, CreateTable, GetItem, PutItem, DeleteItem | Terraform's state-locking table (scoped to optimaflo-*-tf-lock) |
Compute | ecs:*, elasticloadbalancing:*, application-autoscaling:*, emr-serverless:*, elasticmapreduce:* | ECS Fargate agent and Polaris services, EMR clusters |
Database | rds:*, redshift-serverless:*, redshift:* | Aurora PostgreSQL for Polaris and Airflow metadata |
Orchestration | airflow:* | The Amazon MWAA environment |
Networking | ec2:* | VPC, subnets, security groups, NAT gateways |
Observability | logs:*, cloudwatch:* | CloudWatch log groups and alarms for every service |
Secrets | secretsmanager:*, kms:* | Aurora credentials, Polaris client secret, Airflow connections |
Catalog | glue:* | AWS Glue Data Catalog backing Iceberg and Athena |
IamForServiceRoles | iam:CreateRole, PassRole, CreatePolicy, AttachRolePolicy, CreateInstanceProfile, CreateServiceLinkedRole, ... | Terraform creates the IAM roles BYOC services run as (ECS task/execution, MWAA execution) |
Why so broad: the template's own description calls this out directly: permissions are intentionally service-level rather than scoped to individual resources for the first release, so Terraform can both create and destroy every part of the BYOC stack. Your data in S3 is never deleted by OptimaFlo: on teardown, the storage bucket is removed from Terraform state before destroy runs.
This is the kind of control layer SOC 2 and HIPAA audits check for. OptimaFlo is not currently certified under either. If a certification is a hard requirement for your team, confirm that with us directly before you commit.
Step 2: Choose Region & Deployment Tier
The guided wizard offers four AWS regions today:
| Region | Location | Notes |
|---|---|---|
us-east-1 | N. Virginia, USA | Most popular, best pricing |
us-east-2 | Ohio, USA | |
us-west-1 | N. California, USA | |
us-west-2 | Oregon, USA | |
eu-west-1 | Ireland | |
eu-west-2 | London, UK | |
eu-central-1 | Frankfurt, Germany | |
ap-northeast-1 | Tokyo, Japan | |
ap-southeast-1 | Singapore | |
ap-southeast-2 | Sydney, Australia |
The deployment_tier Terraform variable controls how much infrastructure gets provisioned. This is a separate setting from your OptimaFlo subscription plan (Starter, Growth, Scale, Enterprise), it only sizes the BYOC infrastructure inside your own AWS account.
| deployment_tier | What's Provisioned | Documented Estimate |
|---|---|---|
lite | ECS Fargate agent + S3, embedded DuckDB only | ~$35-60/mo |
starter | ECS Fargate agent + Polaris, Aurora, Amazon MWAA orchestration, DuckDB only | ~$165-270/mo |
production | Everything in starter, plus Athena warehouse compute | ~$415-620/mo |
enterprise | Agent, DuckDB, and Polaris run on a dedicated EC2 VM; MWAA and a dedicated EMR cluster | ~$1,000-2,000/mo |
Cost ranges above are the Terraform module's own documented estimates, not OptimaFlo subscription pricing. Actual AWS spend depends on usage and region.
What Gets Provisioned
Terraform composes the stack from separate modules based on the tier you pick. Everything below runs inside your own AWS account.
| Service | Purpose | Tier |
|---|---|---|
Amazon S3Warehouse bucket | Iceberg data, split into warehouse/bronze, warehouse/silver, warehouse/gold (Raw, Clean, Ready) and staging prefixes | All tiers |
AWS Glue Data CatalogCatalog metastore | Backs Iceberg tables and Athena queries | All tiers |
Amazon ECS FargateBYOC agent | Runs pipeline work and the Polaris catalog service | All tiers |
Amazon ECS Fargate + ALBPolaris REST catalog | Apache Iceberg REST catalog, backed by Aurora | starter+ |
Aurora PostgreSQL Serverless v2Metadata database | Polaris catalog metadata and Airflow metadata | starter, production |
Amazon MWAAManaged Airflow (3.2.1) | Orchestrates raw, clean, and ready pipeline DAGs | starter+ |
Amazon AthenaWarehouse compute | Queries your own Iceberg tables via Glue, 100GB-10TB. Not a data source. | production+ |
Amazon EC2Dedicated VM | Agent, DuckDB, and Polaris run together for SLA guarantees | enterprise only |
Compute routing: queries under 100GB run on DuckDB in the agent. Between 100GB and 10TB, OptimaFlo queries your Iceberg tables through Amazon Athena against the AWS Glue Data Catalog, this is a compute engine for data already in your lake, not a data source connector.
Connectors: the AWS-specific data source connectors currently registered are Amazon S3 and Amazon Redshift. Redshift is a read-only source, OptimaFlo does not write results back into it.
Step 3: Deploy via Dashboard
Once your AWS account is connected, deploy the BYOC agent using the same guided wizard as GCP. Pick AWS as the provider and the rest of the flow, region, tier, deploy, is identical.
- 1Connect AWS Account, deploy the CloudFormation stack and paste the role ARN back in
- 2Configure Deployment, select region and deployment tier
- 3Deploy, OptimaFlo assumes your role and runs Terraform through a CodeBuild project inside your own account
- 4Verify, the dashboard shows agent health and connectivity status