Corruption, Recovery and the Backup You Do Not Have
The apply failed halfway. The next plan says the database already exists and cannot be created. Somebody suggests restoring yesterday's state file, which would be the second incident.
The Problem at Scale
State goes wrong in five distinguishable ways, and the reason to distinguish them is that the recoveries are different and applying the wrong one makes things worse.
lost deleted or unreachable
stale an older version is in place
incomplete a resource exists and state does not record it
wrong a bad edit, a bad rm, an import of the wrong id
unreadable written by a newer Terraform or provider
The instinct in every case is to restore the previous version, and it is correct in exactly one of them. In the others it converts a small problem into an orphaned estate, because restoring an older state un-records everything created since, and everything un-recorded is unmanaged forever.
Restore when the state is wrong about many things and you know a good point. Import when it is wrong about a few. That single distinction decides most state recoveries correctly, and getting it backwards is the common way a routine problem becomes a large one: rolling back a state file to fix one missing resource orphans every resource created after that snapshot, and none of them will ever appear in a plan again.
How It Works
The interrupted apply, which is the common one
Terraform writes state incrementally as an apply proceeds, so a cancelled or crashed apply has usually recorded what it finished. The gap is narrow and real: between the provider creating the object and Terraform recording it.
provider: CreateDBInstance -> accepted, id assigned
...process killed here...
terraform: write state -> never happened
the database exists
state does not mention it
next plan: create it
next apply: "DB instance already exists"
The window is small for fast resources and wide for slow ones, which is why this happens on databases, load balancers and managed clusters rather than on security group rules.
Recovery is an import of that one resource, and it is undramatic. What makes this an incident is reaching for a state restore instead.
Why restoring is usually wrong
state at 09:00, serial 1840 the version you would restore
applies at 10:00 - 14:00 created 23 resources, serial now 1863
incident at 14:30 one resource missing from state
restore serial 1840
-> state no longer records the 23 resources
-> they exist, they bill, nothing plans them
-> you have traded 1 orphan for 23
Restoring is right when the state itself is broken: a bad hand edit, a state rm that removed the wrong address, a push that overwrote with something incorrect. Those are cases where the recent history is what you want to discard.
It is wrong when the state is fundamentally sound and disagrees with reality about a small number of resources. That is what import is for.
The unreadable case
State records the Terraform version that wrote it, and newer versions may write a format older ones will not read.
Error: Unsupported state file format
The state file was created by Terraform v1.15.0, which is newer than
the version currently running, v1.11.4.
The usual cause is one engineer with a newer local Terraform running an apply. The state is now upgraded, everybody else is locked out, and the pipeline fails until it is upgraded too. Nothing is corrupted and nothing is lost; the effect is a forced upgrade at a time nobody chose.
The fix is prevention, which is Module 8: pin the version in required_version and let the pipeline be the only thing that applies.
The recovery ladder
In order of preference, because the cheapest correct option should always be tried first.
Restore the object version. Bucket versioning, one command, seconds. Correct for a bad write.
Pull, correct, push. For a state that is nearly right. The serial must be handled: a push of a state whose serial is not ahead of the current one is rejected, and -force exists and is the thing to be careful with.
Import the specific resources. For an incomplete state, which is the common case.
Reimport everything. The nuclear option, appropriate after a total loss with no versioning. Days of work for a large estate, and the argument for versioning being non negotiable.
The backup you do not have
Bucket versioning is the backup, and it is worth being explicit that it is not the same as a backup of the bucket. Versioning protects against a bad write. It does not protect against the bucket being deleted, the account being lost, or a lifecycle policy expiring old versions faster than you notice a problem.
Three checks, all cheap:
versioning enabled the recovery path exists
lifecycle policy on versions how far back you can actually go
replication or periodic copy survives losing the bucket
The second one catches people. A lifecycle rule expiring non current versions after seven days means your recovery window is seven days, which is shorter than the time it typically takes to notice a state problem.
Building and Operating It
Restore an object version. Know this before you need it.
# Versions of the state object, newest first. The one to restore is
# the last one written before the bad operation, which you identify
# by timestamp against the pipeline run that caused it.
aws s3api list-object-versions --bucket org-tfstate-prod \
--prefix platform/network/terraform.tfstate \
--query 'Versions[].{v:VersionId,mod:LastModified,size:Size}' --output table
# Take a copy of the CURRENT state first, whatever you are about to
# do. This is the step people skip and then need.
terraform state pull > /tmp/state-before-recovery.json
aws s3api copy-object --bucket org-tfstate-prod \
--key platform/network/terraform.tfstate \
--copy-source "org-tfstate-prod/platform/network/terraform.tfstate?versionId=$VERSION_ID"Import the resource an interrupted apply left behind.
# The undramatic fix for the common case. The id is whatever the
# provider uses, which the error message usually contains.
terraform import aws_db_instance.primary orders-prod-db
terraform plan # expect: no changes, or only what you intendedOr as configuration, which is reviewable and belongs in the repository:
# import blocks are visible in a plan and in a pull request, which
# is why they are preferable to the command for anything a team
# should see. Module 6 covers this properly.
import {
to = aws_db_instance.primary
id = "orders-prod-db"
}Pull, correct, push, when you genuinely must.
terraform state pull > state.json
# ... a minimal, specific correction ...
jq '.serial += 1' state.json > state-fixed.json
terraform state push state-fixed.json
# A push whose serial is not ahead is rejected. -force overrides
# that check, which is exactly the check you want.Verify the recovery window is what you assume.
# How far back you can actually restore. A rule expiring non current
# versions after 7 days is a 7 day recovery window, which is often
# shorter than the time it takes to notice a state problem.
aws s3api get-bucket-lifecycle-configuration --bucket org-tfstate-prod \
| jq '.Rules[] | {id: .ID, noncurrent: .NoncurrentVersionExpiration}'Take a copy of the current state before any recovery operation, including the ones you are confident about. It costs one command and it is the difference between a recovery you can retry and one you cannot, because the second attempt at a state fix is frequently the one that works and it needs the starting point that the first attempt destroyed. This is the same reasoning as taking a backup before a restore rehearsal, and it is skipped for the same reason, which is that the situation already feels urgent.
Tradeoffs and Decision Framework
| Problem | Recovery | Not this |
|---|---|---|
Bad write, hand edit, wrong state rm | Restore the object version | Import, one at a time |
| Interrupted apply, resource missing | Import that resource | Restore, which orphans everything since |
| Wrong id imported | state rm, then import correctly | Restore, unless it was the only change |
| State written by a newer version | Upgrade, or restore and pin | Editing the version field |
| Total loss, no versioning | Reimport everything | There is nothing else |
Three questions during a state incident. Is the state wrong about many things or a few, since that alone decides restore against import. What has been applied since the version you would restore, because every one of those resources becomes an orphan. And do you have a copy of the current state, as the second attempt usually needs it.
Default: bucket versioning with a retention window measured in months, a copy of current state taken before any recovery, import for a small disagreement, restore for a bad write, and a pinned Terraform version so the unreadable case does not arise.
Failure Modes and Common Mistakes
Restoring to fix a missing resource. You trade one orphan for every resource applied since the snapshot.
No copy of the current state. The second recovery attempt has nothing to start from.
Assuming versioning is a backup. It survives a bad write, not a deleted bucket or an expiry policy.
A short non current version expiry. The recovery window is shorter than the time to notice.
Editing the version field to fix a version error. The format genuinely differs; this produces a worse failure later.
Reaching for state push -force. The check it overrides is the one preventing a lost update.
Not knowing the restore procedure. It is three commands and the time to learn them is not during the incident.
An apply is interrupted after the provider created an RDS instance but before Terraform recorded it. The next plan proposes creating it and the apply fails with an already exists error. Twenty three other resources have been applied successfully since the last state snapshot you could restore. What should you do?
An apply died halfway and the resource exists but is not in state. What do you do, and what do you avoid doing?