What the State File Actually Is

Somebody deleted the state file. Every resource is still running, the configuration is untouched, and terraform plan now wants to create 214 things that already exist.

---

The Problem at Scale

That scenario is the whole lesson. The infrastructure is fine. The code is fine. What was lost is the only record of which cloud resource each block in your configuration refers to, and nothing else in the system holds it.

The cloud does not know that a particular load balancer is aws_lb.public. It knows an ARN. Your configuration does not know the ARN. The state file is the only place those two facts sit next to each other, which is why it cannot be regenerated and why deleting it does not break anything until the next plan, when it breaks everything at once.

Most treatments of Terraform describe state as an implementation detail you should keep in a bucket and otherwise ignore. It is closer to a small database whose consistency you are responsible for, and every serious incident in this course begins with something that happened to it.

---

How It Works

The three things it holds

The identity mapping. For each resource address, the provider's identifier for the real object. This is the essential part and the part that cannot be derived.

{
  "mode": "managed",
  "type": "aws_db_instance",
  "name": "primary",
  "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
  "instances": [{
    "schema_version": 2,
    "attributes": {
      "id": "orders-prod-db",
      "endpoint": "orders-prod-db.abc123.eu-west-1.rds.amazonaws.com:5432",
      "password": "hunter2",
      "...": "every other attribute, cached"
    },
    "dependencies": ["aws_security_group.db", "aws_subnet_group.private"]
  }]
}

A cached copy of every attribute. Used to resolve references between resources, to compute a diff, and to answer a plan run with -refresh=false. This is also where the secrets live, which is the next section.

The dependencies as they were at apply time. Recorded per resource instance, and this one is genuinely under-appreciated: destroy ordering comes from state, not from configuration. When you delete a resource block, there is no configuration left to derive an order from, so Terraform uses the dependency edges it recorded when the resource was created. That is the mechanism, and it explains a whole class of destroy ordering surprises later in the course.

Everything in it is in the clear

State contains every attribute of every resource, including the ones marked sensitive. A generated database password, a private key, a token returned by an API, all of it, in plaintext in the file.

Marking a variable sensitive affects output, not storage. It stops the value appearing in a plan or a log; it does nothing to the state file.

The operational consequence is blunt and worth stating plainly: read access to state is equivalent to read access to your production credentials. Encryption at rest and tight access control on the state backend are not hygiene, they are the same control as the secret store, and a state bucket readable by the whole engineering organisation is a credential store readable by the whole engineering organisation.

The two fields that make everything else work

Two pieces of metadata do more work than their obscurity suggests.

serial increments on every write. It is the optimistic concurrency mechanism: a backend that writes state checks the serial it is replacing, and a mismatch means somebody else wrote in between.

lineage is a UUID generated when the state is first created and carried forward through its whole life. It identifies this state's ancestry. If you point Terraform at a state whose lineage differs from the one it expects, it refuses, because two states with different lineages are two unrelated histories and merging them silently would be catastrophic.

terraform state pull | jq '{version, serial, lineage, resources: (.resources|length)}'
# {
#   "version": 4,
#   "serial": 1847,        <- writes since creation
#   "lineage": "b1f3...",  <- this state's identity
#   "resources": 214
# }

A rising serial with no applies is somebody writing state outside your pipeline. A changed lineage means the state was recreated, and whatever was in the old one is now unmanaged.

Local against remote, and what remote actually buys

The advice to use a remote backend is universal and usually given without the reason. Four things, and only the first is about durability:

Durability, since a laptop is not a storage system.

Shared access, so that two engineers and a pipeline see the same state.

Locking, which is the next lesson and the one that matters most.

Versioning, which is the recovery path in the lesson after that.

A remote backend without locking and without versioning has bought you one of the four. That is a common configuration and it is worth checking rather than assuming.

---

Building and Operating It

Look at your state, which most engineers never do.

# What Terraform believes it manages. If this list and reality
# disagree, everything downstream is built on the disagreement.
terraform state list | wc -l
terraform state list | head -20

# One resource, in full, including the attributes it cached.
terraform state show aws_db_instance.primary

Confirm the backend does all four things.

terraform {
  backend "s3" {
    bucket       = "org-tfstate-prod"
    key          = "platform/network/terraform.tfstate"
    region       = "eu-west-1"

    # Locking. Native S3 locking, which replaced the DynamoDB table
    # approach; DynamoDB based locking is deprecated. Next lesson.
    use_lockfile = true

    # Encryption at rest, because this file contains every secret
    # any resource ever generated.
    encrypt      = true
    kms_key_id   = "arn:aws:kms:eu-west-1:111122223333:key/..."
  }
}

# Versioning on the bucket is the recovery path. Without it there is
# no way back from a bad write.
aws s3api get-bucket-versioning --bucket org-tfstate-prod
# {"Status": "Enabled"}    <- anything else is a finding

Find out who can read your credentials.

# Everything with read access to the state bucket has read access to
# every secret in every resource this state manages.
aws s3api get-bucket-policy --bucket org-tfstate-prod \
  | jq -r '.Policy | fromjson | .Statement[]
      | select(.Effect=="Allow") | {principal: .Principal, actions: .Action}'

---

Tradeoffs and Decision Framework

Property	Consequence
Holds the identity mapping	Cannot be regenerated, only rebuilt by importing
Caches every attribute	Contains every secret, in plaintext
Records dependencies at apply time	Destroy ordering survives deleting the configuration
serial increments per write	Optimistic concurrency, and a drift signal for writes
lineage identifies the state	A change means the old state is now unmanaged

Three questions about any state you inherit. Is it versioned, since that is the only recovery path. Is it locked, which is the next lesson. And who can read it, because that list is the list of people with your production credentials.

Default: a remote backend with locking, versioning and encryption at rest, access restricted to the identity that applies, and never a hand edit.

---

Failure Modes and Common Mistakes

Treating state as a cache. It holds a mapping that exists nowhere else and cannot be derived.

Assuming sensitive protects the file. It affects output, not storage.

A state bucket readable by everybody. That is a credential store readable by everybody.

Remote backend without versioning. There is no way back from a bad write.

Hand editing. Every safe change has a command, and the unsafe ones fail obscurely.

Ignoring serial and lineage. They are the two fields that tell you somebody wrote outside your pipeline, or that the state was recreated.