Release safety

Once a pull request merges, the code is on the base branch and whatever deploys it is under way. This page covers the optional steps in foreman.config.json that guard production after that point, what a halt is, and how you clear one. Read it before you add deploy-github to your ship steps. The steps before the merge are on How a story ships; the deploy itself is on GitHub deployments and Supabase.

With the Salesforce pack, production is never a deploy target, and the pack’s own release paths halt and never roll back; Production stays yours has what a connected production org may and may not be asked to do. The Salesforce pack is a preview: its release paths pass their own tests, but no story has yet shipped through them on a live org, and the Gearset path has not run against a live Gearset pipeline.

Smoke checks

The smoke step runs after the deploy and checks the live site answers. Each address in ship.smoke.urls is fetched once, with a 15 second limit, without following redirects: a redirect is the site answering. No answer, or a status of 500 or more, stops the release with exit 5 and writes HALT. ship.smoke.expect pins the exact status an address must return. Point it at a health route that touches the database, so “the page loads” is not mistaken for “the app works”; every other address keeps the “below 500” rule. With no addresses there is nothing to check and the step passes; foreman ship --preflight warns about that.

The same check is the smoke step’s health check, run by foreman health and foreman health --site before a halt clears, this time following redirects.

The watch

The watch step keeps an eye on production for a while after the deploy. It runs your own command, ship.watch.command, with sh -c in the project folder and your environment file loaded: once at the start, then every everySeconds (60 by default) until minutes (10 by default) are up. The command can query Vercel or Sentry for new errors, or curl your health route. Any non-zero exit, or a run longer than timeoutSeconds (120 by default), stops the release with exit 5 and writes HALT, with the last 300 characters of the command’s output in the reason, secrets blanked. A watch in ship.steps with no command is a config error. The step is left out when features.watchRollback is off.

ship.watch.minutes and ship.watch.everySeconds are also settings on the dashboard (Settings), shown while ship.watch is configured.

Rollback

rollback is not a step in the list; naming it in ship.steps is a config error. Foreman runs ship.rollback.command once, only when a step fails after deploy-github passed, so the new build is the one live. A failed deploy never rolls back, because production still runs the previous version and rolling back would put an even older one live. Nor does a failure with no deploy before it. With the Salesforce pack, once a later stage that people share has started, nothing rolls back: the halt says “Not rolled back” and you put it right by hand.

The command runs with sh -c in the project folder, your environment file loaded and FOREMAN_STORY set to the story being shipped (vercel rollback --yes, say, with the Vercel CLI signed in). A run longer than timeoutSeconds (300 by default) counts as a failed rollback. Releases are frozen before the rollback starts and stay frozen: HALT is written first saying “Rolling back with <command>…”, so a crash in the middle still leaves releases frozen, then written again with what happened. The halt email says whether it worked: “Production was rolled back with <command>.” or “Rollback failed (<command> exited N); production may be broken.” The rollback has no health check: foreman health asks whether production is fine, not whether it can be rolled back. It is off with features.watchRollback.

foreman ship --preflight warns when watch is set without a rollback (a problem after the deploy then halts releases but leaves production as it is), and when a rollback is set with no deploy step, since it would never run.

"ship": {
  "steps": ["merge-main", "resources", "push", "pr", "ci", "mergeable-recheck", "merge", "deploy-github", "smoke", "watch"],
  "smoke": {
    "urls": ["https://acme.example", "https://acme.example/api/health"],
    "expect": { "https://acme.example/api/health": 200 }
  },
  // Exit non-zero when production looks wrong. Runs with your env file loaded.
  "watch": { "command": "./scripts/errors-since-deploy.sh", "minutes": 10, "everySeconds": 60, "timeoutSeconds": 120 },
  // Runs once, only on failure after a successful deploy. Needs the Vercel CLI signed in.
  "rollback": { "command": "vercel rollback --yes", "timeoutSeconds": 300 }
}

Checks on the merged code

With no ci step, merge-main runs your checks on the merged code whenever it brought in new commits from the base branch, and a failure stops the release with exit 2 before the pull request merges. The merge stays on the story branch and the story gets a fix round. A check is not run again when foreman checks already ran it, in the session’s own environment, on the same merged tree (the working tree is clean and its record is for exactly that tree): that record’s result counts, and the release says which checks it reused. foreman doctor says which applies: the step that tests the merged code, Foreman’s own run of the checks, or nothing at all.

Halts

A halt freezes releases. It is the file HALT in the state folder (.foreman/HALT by default), one line: the time and the JSON result of the step that failed, with its stage, reason, the pull request and whether production was rolled back. foreman ship writes it when:

• a step marked as halting fails: deploy-github, smoke, watch, and the Salesforce pack’s org guard, deploy, verify, deploy-watch and Gearset promote steps;
• any step fails after a point of no return (the merge, or a hosted migration that changed the live database), whatever that step’s own code would have been;
• the result is exit 4 (the migration failed) or exit 5 (the deploy or a check after it failed).

If HALT cannot be written, the event feed says so loudly (“releases are NOT frozen”) and the result carries haltWritten: false.

While HALT exists, the run shows Halted with the reason, and the dashboard’s alert says to fix the cause and press “I’ve fixed it” under Controls. You get one email, “Factory halted”, naming the story and the step, with what happened and what to do; when the failure was at the deploy, the smoke check or the watch and production was not rolled back, the email adds ship["deploy-github"].rollbackHint, your own words on how to roll back. Nothing new starts. Any other story that reaches its release exits 8 and its slot is held with “Committed; releases are frozen by a halt.”; the story that caused the halt is held with “This story stopped the release line; waiting for you to fix the cause.” A story session that finds HALT before it starts stops as halted. In GitHub Actions, each scheduled job ends in seconds with “halted: a release failed; foreman fixed once the cause is fixed”.

Foreman never retries, reverts or repairs production on its own after a halt. The story session blocks the story and adds an urgent owner task with the ship result, so the halt also appears under Waiting on you.

“I’ve fixed it” and foreman fixed

When you have put the cause right, press I’ve fixed it under Controls on the dashboard (the button exists only while the run is halted), or run foreman fixed from a terminal. The run then runs foreman health. Only if its checks pass does the halt clear: HALT is removed, the event feed says “Halt cleared: <what the checks said>” (or “Halt cleared; no health checks are configured.”), and the held slots are released oldest first. The story that caused the halt is started again in after-halt mode; the others carry on to their release. If a check fails, the run stays halted, the event feed says “Still halted: <reason>”, and the dashboard shows what still fails under the button. Deleting the file by hand skips the check: the other held stories resume, but the slot of the story that caused the halt stays held.

foreman health

foreman health runs the health check of every configured ship step that has one, and prints one JSON line { ok, reason, checks: [{ step, ok, reason }] }, exit 0 when every check passes and 1 otherwise. The checks are:

Step	What its health check asks
migrate-supabase	The hosted database has exactly the base branch’s migrations, read from the Supabase Management API with SUPABASE_ACCESS_TOKEN. It names migrations missing from the live database and migrations the live database has that the base does not.
deploy-github	The base branch’s newest commit has a successful deployment in the configured environment, any of them, since a redeploy makes a new one. A story merged but never deployed is not live, so it is not fixed: “Redeploy it, then press “I’ve fixed it” again.”
smoke	Every address in ship.smoke.urls answers, following redirects, below 500 or with its expect status.
watch	The watch command, run once, reports no problem. Left out when features.watchRollback is off.

foreman health --site runs only the smoke step’s check; the story session uses it to confirm a merged story is live. With nothing to check, both pass with the reason “no health checks are configured”. The command loads Foreman’s own keys from .env.foreman and the project’s environment file for the checks, and prints neither.

After a halt

Once the halt clears, the run starts the story that caused it as /foreman:story <id> after-halt. In that mode the session does not build or ship again. If the pull request merged and foreman health --site exits 0, it closes the story as done, noting what now works, the pull request, and “merged; the live site answers” (or “merged; no live site is configured” when no health check is set). Otherwise it takes the failed path and ends the story as failed, with what the health check said. The other held stories resume where they stopped: their branches are committed, so they go straight back to foreman ship.