Terraform on AWS for enterprises: how to start, and the practices that keep it sane
A practical starting plan for running Terraform across many AWS accounts: Organizations and the landing zone, S3 remote state with native locking, OIDC for CI, parent and child modules, and environments/*.tfvars.
Most companies don't start Terraform on AWS with a plan. Someone writes a main.tf for a VPC, it works, a second team copies it, and a year later there are forty folders, three state buckets nobody owns, an IAM user called terraform-ci with AdministratorAccess, and a prod plan that occasionally wants to replace a database.
None of that is Terraform's fault. It's what happens when the code arrives before the account structure, the state design and the pipeline do.
This guide is the order I'd do things in if I were starting Terraform for an enterprise on AWS today. If you're new to Terraform itself (providers, variables, for_each, modules, remote state), read the Terraform best practices and module architecture guide first. This one assumes those basics and focuses on what changes when the cloud is AWS and the company is big.
The short version
- Accounts are your blast radius. One AWS account per workload per environment, grouped into OUs, created by AWS Organizations and a landing zone. Don't separate environments with tags or VPCs inside one account.
- State lives in S3 with native locking. Versioned, encrypted with KMS, public access blocked, one state file per stack per environment. DynamoDB locking is no longer needed.
- CI uses OIDC, not access keys. GitHub Actions (or GitLab) assumes an IAM role for a few minutes. A read-only role plans on pull requests; a separate role applies on
mainafter approval. - Child modules do one thing; root modules compose them. The same root module deploys dev, staging and prod, with the difference kept in
environments/<env>.tfvars. - The provider refuses the wrong account.
allowed_account_idsandassume_rolein every provider block. This one line has saved more prod accounts than any code review. - Guardrails sit above Terraform. SCPs and RCPs stop the things no pipeline should ever do, whatever the IAM role says.
Step 1: build the account structure before the code
The single best security and reliability decision on AWS is to split things into accounts. An account is a hard boundary for IAM, quotas, billing and mistakes. A developer with admin in the dev account can't touch prod, and a runaway terraform destroy in staging stops at the account edge.
Use AWS Organizations to create and group accounts, and AWS Control Tower (or your own landing zone, if you already have one) to set up the baseline: CloudTrail to a central log archive, AWS Config, IAM Identity Center for human sign-in, and a set of preventive controls. Control Tower is the faster path for most enterprises; it's opinionated, and those opinions are mostly right.
A structure that works for most companies:
| OU | Accounts | What lives there |
|---|---|---|
Security |
log-archive, security-tooling |
Central CloudTrail and Config logs, GuardDuty and Security Hub delegated admin |
Infrastructure |
network, shared-services, deployments |
Transit Gateway and egress, shared DNS and AMIs, Terraform state and CI roles |
Workloads/NonProd |
payments-dev, payments-staging, ... |
Application accounts, one per workload per environment |
Workloads/Prod |
payments-prod, ... |
The same, with stricter SCPs |
Sandbox |
One per engineer or team | Experiments, with a budget and auto-cleanup |
Keep the management account empty apart from Organizations, billing and Control Tower. Never run workload Terraform there, and never give CI a role in it.
Creating accounts with Terraform
Once you have more than a handful of accounts, creating them by hand gets slow and inconsistent. You have two good options:
- Account Factory for Terraform (AFT). AWS's own pipeline on top of Control Tower. You open a pull request with an account request, and AFT creates the account, enrols it in Control Tower and runs your baseline customisations. It's a lot of moving parts, but it's supported and it scales to hundreds of accounts.
aws_organizations_accountin a small, locked-down stack. Fine for a few dozen accounts if you're not using Control Tower. Run it from a dedicated account with delegated permissions, never from a developer laptop.
Either way, every new account should come out of the factory with the same baseline: the Terraform deployment roles (more on those below), a default-deny on the root user, budgets, and the standard tags.
Step 2: design state before the first apply
State is the most sensitive and most fragile thing Terraform owns. It can contain secrets in plain text, and if two people write it at once, it breaks. Get it right on day one.
Where state lives
Put state buckets in the deployments account (or a dedicated terraform-state account), not inside the workload accounts. That way the people who can change prod infrastructure aren't automatically the people who can read prod state, and deleting a workload account doesn't delete the record of what was in it.
Use one bucket per environment tier (for example acme-tfstate-nonprod and acme-tfstate-prod), so a nonprod pipeline role can be denied the prod bucket outright with one bucket policy, instead of fragile key-prefix rules.
Inside the bucket, one state file per stack per environment:
acme-tfstate-prod/
network/prod/terraform.tfstate
payments/app/prod/terraform.tfstate
payments/data/prod/terraform.tfstate
Small states plan faster, fail smaller and can have different owners. A single state for "all of prod" is the most common enterprise mistake I see.
The S3 backend with native locking
Since Terraform 1.10, the S3 backend can lock state with an S3 conditional write (use_lockfile = true). You no longer need a DynamoDB table for locking, and the DynamoDB option is deprecated. If you're migrating an older setup, turn on use_lockfile alongside the existing table first, then remove dynamodb_table once every pipeline is on a recent version.
Keep the backend block in the code minimal and pass the environment-specific part from a file, the same environments/<env>.backend.hcl pattern the generic guide uses:
# backend.tf
terraform {
backend "s3" {}
}
# environments/prod.backend.hcl
bucket = "acme-tfstate-prod"
key = "payments/app/prod/terraform.tfstate"
region = "eu-west-2"
encrypt = true
kms_key_id = "arn:aws:kms:eu-west-2:111111111111:alias/tfstate-prod"
use_lockfile = true
terraform init -backend-config=environments/prod.backend.hcl
The bucket itself
The state bucket needs:
- Versioning on, so a corrupted or deleted state file can be recovered. Add a lifecycle rule that expires old noncurrent versions after 90 days or so.
- SSE-KMS with a customer managed key, and a key policy that only the deployment roles and a break-glass role can use. Reading the object isn't enough without
kms:Decrypt. - Block Public Access on, and Object Ownership set to bucket owner enforced.
- A bucket policy that denies non-TLS requests and denies everything except the named Terraform roles and break-glass.
- Object Lock is optional. It's tempting, but it can get in the way of state migrations. Versioning plus a tight policy is usually enough.
The chicken-and-egg problem
The bucket that stores state has to be created by something. Keep a tiny bootstrap stack that creates the state buckets, KMS keys and deployment roles. Apply it once with local state, then add the backend block and run terraform init -migrate-state to move its own state into the bucket it just created. After that, it's a normal stack that changes rarely and needs two approvers.
Step 3: a repository layout that scales
The generic guide covers parent and child modules in detail. On AWS at enterprise scale, I use two kinds of repository:
- A module repository (or a few) with versioned child modules:
vpc,eks-cluster,rds-postgres,s3-bucket,iam-role. Each one is released with a git tag or published to a private registry (HCP Terraform, Terraform Enterprise, or a Git-based source with?ref=v1.4.0). - Live repositories per team or domain, containing root modules (stacks) that call those modules, one set of
.tfvarsper environment.
A live repository looks like this:
payments-infra/
stacks/
app/
main.tf # calls child modules
variables.tf
outputs.tf
providers.tf
backend.tf # empty "s3" block
versions.tf
environments/
dev.tfvars
dev.backend.hcl
staging.tfvars
staging.backend.hcl
prod.tfvars
prod.backend.hcl
data/
...same shape...
.github/workflows/terraform.yml
The important rule: the root module is identical across environments. Prod doesn't get a different main.tf. If prod needs three NAT gateways and dev needs one, that's a variable, and its value lives in prod.tfvars. When the code is the same everywhere, a change tested in dev is the change that reaches prod.
A child module, AWS flavoured
A child module should do one job, take a small number of meaningful inputs, and never configure its own provider:
# modules/vpc/variables.tf
variable "name" {
type = string
}
variable "cidr_block" {
type = string
}
variable "azs" {
type = list(string)
}
variable "nat_gateway_count" {
type = number
default = 1
validation {
condition = var.nat_gateway_count >= 1 && var.nat_gateway_count <= length(var.azs)
error_message = "nat_gateway_count must be between 1 and the number of AZs."
}
}
# modules/vpc/main.tf (excerpt)
resource "aws_vpc" "this" {
cidr_block = var.cidr_block
enable_dns_support = true
enable_dns_hostnames = true
tags = { Name = var.name }
}
resource "aws_subnet" "private" {
for_each = { for i, az in var.azs : az => i }
vpc_id = aws_vpc.this.id
availability_zone = each.key
cidr_block = cidrsubnet(var.cidr_block, 4, each.value)
tags = { Name = "${var.name}-private-${each.key}", Tier = "private" }
}
Use for_each keyed by something stable (the AZ name here), not count. Removing the middle item from a count list renumbers everything after it and Terraform will want to destroy and recreate those subnets.
Don't write your own VPC module if you don't have to. The community terraform-aws-modules collection is well maintained. Many enterprises wrap it in a thin internal module that pins the version and sets the company defaults (flow logs on, specific tags, no public subnets unless asked).
The root module that composes it
# stacks/app/versions.tf
terraform {
required_version = ">= 1.10"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
# stacks/app/main.tf
module "vpc" {
source = "git::https://github.com/acme/terraform-aws-modules.git//vpc?ref=v2.3.0"
name = "payments-${var.environment}"
cidr_block = var.vpc_cidr
azs = var.azs
nat_gateway_count = var.nat_gateway_count
}
module "database" {
source = "git::https://github.com/acme/terraform-aws-modules.git//rds-postgres?ref=v1.8.1"
name = "payments-${var.environment}"
subnet_ids = module.vpc.private_subnet_ids
instance_class = var.db_instance_class
multi_az = var.db_multi_az
deletion_protection = var.environment == "prod"
}
# stacks/app/environments/dev.tfvars
environment = "dev"
account_id = "222222222222"
vpc_cidr = "10.17.0.0/16"
azs = ["eu-west-2a", "eu-west-2b"]
nat_gateway_count = 1
db_instance_class = "db.t4g.medium"
db_multi_az = false
# stacks/app/environments/prod.tfvars
environment = "prod"
account_id = "333333333333"
vpc_cidr = "10.16.0.0/16"
azs = ["eu-west-2a", "eu-west-2b", "eu-west-2c"]
nat_gateway_count = 3
db_instance_class = "db.r7g.large"
db_multi_az = true
Pin module versions with ?ref= to a tag, and commit .terraform.lock.hcl so every run uses the same provider build. Upgrading a module in prod should be a pull request that changes one version number, nothing else.
Step 4: make the provider refuse the wrong account
This is the cheapest safety net in the whole setup:
# stacks/app/providers.tf
provider "aws" {
region = var.region
allowed_account_ids = [var.account_id]
assume_role {
role_arn = "arn:aws:iam::${var.account_id}:role/terraform-deploy"
session_name = "terraform-payments-app-${var.environment}"
}
default_tags {
tags = {
Environment = var.environment
Owner = "payments"
ManagedBy = "terraform"
Repository = "acme/payments-infra"
}
}
}
allowed_account_idsmakes Terraform stop before doing anything if the credentials point at a different account. Run a prod plan with dev credentials by mistake and you get an error, not a diff.assume_rolemeans the pipeline starts with an identity in thedeploymentsaccount and hops into the target account. The account ID in the tfvars decides where it lands.default_tagsputs ownership and environment on every taggable resource without anyone remembering to.
AWS provider v6 also lets most resources take their own region argument, which removes the need for a pile of aliased providers when a stack touches two regions (a CloudFront certificate in us-east-1, for example). Use it for the odd resource; if a whole stack belongs in another region, give it its own root module.
Step 5: CI with OIDC and two roles
No IAM users, no access keys in CI secrets. GitHub Actions, GitLab CI and most other platforms can issue an OIDC token that AWS exchanges for temporary credentials. I covered how that trade works in Delete the cloud key in your CI pipeline; here's how it fits Terraform.
Create two roles in each target account (the account factory should do this for you):
| Role | Who can assume it | Permissions | Used for |
|---|---|---|---|
terraform-plan |
Any branch or pull request of the infra repo | ReadOnlyAccess, plus read on the state bucket and KMS decrypt |
terraform plan on pull requests |
terraform-deploy |
Only the main branch, in a protected GitHub environment |
What the stacks in that account actually need, bounded by a permissions boundary | terraform apply after merge |
The trust policy for the deploy role pins the repository, the branch and the GitHub environment:
{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::333333333333:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/payments-infra:environment:prod"
}
}
}
Then the pipeline is simple:
- On every pull request:
terraform fmt -check,terraform validate,tflint, a security scan, andterraform planfor each affected environment using the plan role. Post the plan to the pull request. - On merge to
main: apply to dev, then staging, then prod. Prod waits for an approval in the GitHub environment, and re-runs the plan before applying so nobody approves a stale diff.
Read the plan before you approve it. Read the Terraform plan, not just the green check has a short routine for that.
Give humans the same split. Engineers get read-only in prod through IAM Identity Center, and a time-limited break-glass role for emergencies, logged and alerted on. If people can apply to prod from their laptops, the pipeline is a suggestion.
Step 6: guardrails that Terraform can't bypass
IAM roles say what an identity may do. Organization policies say what nobody in an account may do, including a misconfigured Terraform role. Manage them in their own stack, owned by the platform or security team:
- Service control policies (SCPs) on the
WorkloadsOUs: deny leaving the organization, deny disabling CloudTrail, GuardDuty and Config, deny regions you don't use, and deny changes to the Terraform roles except by the account factory. - Resource control policies (RCPs) to stop data leaving the organization, for example denying S3 and KMS access to principals outside your org. They protect the state buckets too.
- Tag policies to enforce the values of
EnvironmentandOwner, sodefault_tagscan't drift intoProd,productionandPRD.
Test SCP changes in a sandbox OU first. An overly broad deny can break every pipeline in the company at once, and the error messages don't always say an SCP was the cause.
Step 7: checks in the pipeline
You'll catch most problems before plan with a few cheap tools:
terraform fmt -checkandterraform validate, always.- TFLint with the AWS ruleset, which knows things like invalid instance types and deprecated arguments.
- A policy or security scanner such as Checkov or Trivy, for unencrypted volumes, public buckets and open security groups. Start in warn mode, fix the noise, then make it blocking.
terraform testfor your child modules, so a module release proves it still produces what it promised.- Cost estimation (Infracost or similar) on the pull request, so a change from
db.t4g.mediumtodb.r7g.4xlargegets a second look.
Step 8: day two
Infrastructure doesn't stay where you put it. Plan for that from the start:
- Drift detection. A scheduled pipeline runs
terraform plan -detailed-exitcodeagainst every stack nightly and alerts when exit code 2 shows a difference nobody committed. - Importing what already exists. Use
importblocks andterraform plan -generate-config-out=generated.tfto bring console-built resources under management, then clean up the generated code before merging. - Refactoring without destroying. When you rename a resource or move it into a module, add a
movedblock. Terraform updates the address in state instead of planning a replace. - Protecting the things that hurt.
deletion_protectionon databases and load balancers,lifecycle { prevent_destroy = true }on the few resources whose loss is unrecoverable, and a pipeline rule that fails any plan containing a destroy in prod unless a label is added. - Provider and module upgrades on a schedule. Renovate or Dependabot opening small upgrade PRs every week is far less painful than a once-a-year jump across major versions.
How to start: a realistic first month
If you're starting from nothing, or from a pile of unmanaged folders, this order works:
Week 1: foundations. Set up Organizations and Control Tower, create the Security, Infrastructure and Workloads OUs, the log archive and security tooling accounts, and IAM Identity Center for people. Turn on the default controls. Nothing is in Terraform yet, and that's fine.
Week 2: the Terraform platform. Create the deployments account. Write and apply the bootstrap stack: state buckets, KMS keys, the GitHub OIDC provider and the plan and deploy roles. Migrate its state into S3. Decide on the module repository and the first three modules you'll need.
Week 3: the first real workload. Pick one low-risk service. Build its live repository with environments/dev.tfvars and staging.tfvars, the pipeline, and the checks. Get it to the point where a merge to main deploys dev and staging with no human running Terraform.
Week 4: prod and the paved road. Add prod.tfvars, the approval gate, drift detection and the SCPs. Write a one-page "how to start a new stack" guide, ideally as a template repository, so the second team copies something good instead of something improvised.
After that, it's repetition: every new workload gets accounts from the factory, a live repository from the template, and modules from the shared repository.
Mistakes I'd avoid
- One AWS account with dev, staging and prod in it. Separating environments by tag or VPC isn't a security boundary.
- One giant state for a whole environment. Plans take ten minutes, every change locks everyone out, and one mistake has the widest possible reach.
- Terraform workspaces as the environment switch across accounts. They share a backend config and make it too easy to apply prod with the wrong variables. Separate tfvars and backend files make the target explicit.
- Long-lived access keys for CI or for the people running Terraform.
- Modules that configure providers or hard-code account IDs and regions. They can't be reused and they break
for_eachon modules. - Unpinned versions.
source = "...//vpc"with noref, orversion = ">= 5.0"on the provider, means next week's plan can differ from today's without a single commit. - Secrets in tfvars. Store them in Secrets Manager or SSM Parameter Store and read them with a data source, or better, let the application read them at runtime. Use
sensitive = trueon variables and outputs, and remember that state still holds the values, which is why the state bucket is locked down.
Further reading
- Terraform best practices and module architecture, the generic guide this one builds on
- S3 backend reference, including
use_lockfile - AWS Prescriptive Guidance: Terraform on AWS best practices
- AWS Control Tower Account Factory for Terraform
- Organizing your AWS environment using multiple accounts