Incidents and recovery

Incidents come in two shapes. Sometimes your infrastructure is on fire and you need the smallest, fastest change that resolves it. Other times a formae command itself is the problem: one you triggered by mistake, or one that is stuck. This guide covers both.

Fix a live incident with a patch

Patch mode is built for firefighting. It only creates or updates the resources you name and never destroys anything, so the blast radius is exactly the resource or two you are touching. See Apply modes for the full contrast with reconcile.

CLI
AI assistants

Write a focused forma

Include only the resources you are adding or fixing. The stack and target must already exist:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@aws/aws.pkl"
import "@aws/s3/bucket.pkl"

forma {
  new formae.Stack {
    label = "production"
    description = "Production infrastructure"
  }

  new formae.Target {
    label = "prod-target"
    config = new aws.Config {
      region = "us-east-1"
    }
  }

  new bucket.Bucket {
    label = "incident-logs"
    bucketName = "acme-incident-logs"
  }
}

The file does not need to live in your repo. Create it anywhere, for example /tmp/hotfix.pkl, apply it now, and fold it into your code later.

Apply the fix

formae shows the plan and asks you to confirm, then streams the run in a live view:

formae apply --mode patch hotfix.pkl

  formae apply · patch                                                                                  hotfix.pkl
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  + 1 create

  ▌ Resources
  Operation ▲   Label                                                       Type
  + create      incident-logs                                               AWS::S3::Bucket

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓: select  space: expand  →←: column  s: sort  y: confirm  q: abort This operation will create 1 resource(s).  Do
you want to continue? (y/N)  ?: help

Add --yes to skip the confirmation prompt when you are certain and need speed.

With the formae MCP plugin connected, describe the fix and let the assistant patch it in:

Conversation

You

We need an S3 bucket for incident logs in the production stack right now, without touching anything else.

Assistant

I’ll patch it in so nothing else is affected, and simulate first: it creates one resource, the S3 bucket incident-logs in production, and changes nothing else. Apply it?

You

Yes.

Assistant

Applied. incident-logs is live in production, and the rest of the stack is untouched. formae keeps track of this patch so it will not be silently overwritten. After the incident, decide whether to absorb it into the stack’s desired state or revert it.

Settle up after the fix

A patch updates the cloud without automatically changing the reconcile baseline. Once the incident is over, review the stack’s out-of-band changes and choose whether to absorb or revert each actionable change. Other out-of-band changes in that stack are reviewed at the same time.

Absorb a change worth keeping

Tell your assistant:

Keep the incident-logs bucket in production. Review any other out-of-band changes with me.

The assistant retrieves the stack, presents the remaining choices, and simulates the combined reconcile. Once you confirm it, formae records the bucket as desired intent. A pure absorption does not need to create the bucket again. If you maintain a project, the assistant updates that selected project’s Pkl source afterward. If you do not, the next session retrieves the accepted stack from formae. The decision does not depend on keeping the incident’s temporary files. With the CLI, reconcile your complete stack declaration and select absorb for the patched resource in the review of out-of-band changes:

formae apply --mode reconcile main.pkl

Review the final combined plan before confirming. After resolution, use the printed extraction queries and deletion guidance to bring an optional codebase up to date. Do not reconcile a resource-only extraction as if it were the full production stack.

Revert a temporary workaround

Tell your assistant:

The incident-logs bucket was temporary. Revert that patch in production.

Reverting a newly created resource may delete it. The assistant shows that operation in the final simulation and asks for confirmation. Resolve other out-of-band changes that need a decision in the affected stack as part of the same review. In the CLI review of out-of-band changes, choose revert for the patched resource. The existing --force option is still available when you deliberately want to overwrite all out-of-band changes with your declaration; simulate that complete plan first:

formae apply --mode reconcile --force --simulate main.pkl

Restoring configuration does not restore destroyed data. Inspect the command’s actual outcome after applying, especially if part of the operation failed.

Remove a resource after its creation failed

A failed creation remains part of the stack’s desired state. If formae has no managed observation of the resource, omitting it from your next reconcile cannot establish that it is absent from the cloud. formae reports the unresolved resource and failed command instead of silently accepting the omission. Inspect the failed command and investigate the cloud outcome. Recover or reapply the resource’s declaration, then reconcile its removal once formae can manage it. You can still retrieve the desired stack to recover that declaration. Discovery can help investigate, but does not by itself remove the recorded intent, and --force does not bypass this limitation.

Get started

Guides

Concepts

Reference

Incidents and recovery

Fix a live incident with a patch

Settle up after the fix

Absorb a change worth keeping

Revert a temporary workaround

Remove a resource after its creation failed

See also

​Fix a live incident with a patch

​Settle up after the fix

​Absorb a change worth keeping

​Revert a temporary workaround

​Remove a resource after its creation failed

​See also

Fix a live incident with a patch

Settle up after the fix

Absorb a change worth keeping

Revert a temporary workaround

Remove a resource after its creation failed

See also