When things go wrong

Foreman is built to keep running, so most of what goes wrong shows up as a line on Waiting on you or in foreman status rather than a dead process. This page is for the person running a project when that line appears. For each problem it gives what you see, in Foreman’s own words, why it happens, and what to do. The limits behind the numbers are on Settings; the pause, resume and stop controls are on Run, pause and stop.

Not building

What you see. An item under Waiting on you titled “Not building for N minutes” (“less than a minute” and “1 minute” at first, “N hours” past two hours), with one of three reasons. foreman status reads “running, but stuck: <reason>”. You get one stuck email per episode, and the item clears by itself when work moves again, with the event “Making progress again.”

Reason	After	Why
“the supervisor fails every loop: <text>”	5 minutes (watchdog.errorMinutes)	Every pass of the run’s loop hits the same error. Digits in the text are ignored, so a counter or a pid does not restart the clock.
“ACME-12 can be built and a slot is free, but nothing has started”	10 minutes (watchdog.idleMinutes)	The picker found a story and a free slot, and still nothing launched: usually a slot whose environment keeps failing to prepare, or a story whose session will not start.
“ACME-12 in w2 has shown no activity since 14:05 (no stage change, log output or CPU)”	20 minutes (watchdog.activityMinutes)	The oldest running session has produced nothing at all.

After the computer sleeps, every clock starts again at the wake, so a sleep alone is never reported as a stall.

Slow is not stuck. A session that is still active but has been in the same stage for 45 minutes (watchdog.progressMinutes) is slow. It shows as an FYI, “slow: ACME-12 in its fix round for 1 h 45 min (129 steps)”, with no email. The phrase names the stage: its design, its build, its fix round, its tests, review, its commit, its release. A slow story is left alone.

What to do.

• For the loop error, read the text. It is usually a tool that stopped answering, a story source that will not open, or a folder that cannot be read. Fix the cause; if the item does not clear, foreman stop and foreman run.
• For a story that can be built but does not start, look at the slot cards: a slot that is down or parked says why (Slots and environments), and foreman doctor names anything missing on the machine.
• For a session with no activity, open it in the dashboard’s command centre to see the last thing its agents did. You rarely need to act: the watchdog stops a session quiet for 30 minutes (watchdog.quietMinutes) or without progress for 2 hours (watchdog.buildMinutes) and starts the story again from its last finished stage. A story still building at 4 times the build limit while making progress is parked for you with “ACME-12 was still building after 8 h (4 times the build limit) while still making progress. Resume carries on from its last finished stage; Restart begins it again from main.” If the same story stalls again, Skip it (foreman slot w2 skip) and look at its brief.

Auto-pause after three misses

What you see. “Foreman paused itself” on Waiting on you, with a Resume button and one of these reasons; foreman status reads “paused: <reason>. Nothing new starts until you resume.” and you get one paused email. Stories already being built finish first.

• “Three stories in a row did not ship: ACME-10, ACME-11 and ACME-12.”
• “ACME-12 did not ship three times in a row.”
• “Three tries in a row did not ship: ACME-10 twice and ACME-12.”

Why. A miss is a story the pipeline rejected: the tester or a reviewer still said no after its last fix round, so it failed and is parked as a draft pull request. Three misses in a row pause the run, because the problem is probably not in the stories. Only a ship, or a release that passed every check and parked, clears the count. A story that stops to wait on you, one that waits on other stories, a deferral and an empty pick neither count nor clear.

Environment endings are counted apart. A session that crashed with no result, one the watchdog stopped, and a story ended with foreman story end and --cause environment (a test file that would not load, CI that never started, a conflict Foreman could not settle) are the environment’s fault, never a miss. Three of those in a row make Foreman wait 15 minutes, then 30, then 60, each time with an event such as “Three endings in a row were the environment, not the stories: ACME-10 ended without a result; ACME-11 was stopped by the watchdog; ACME-12 failed: npm ci exited 1. Waiting 15 minutes before carrying on.” A fourth run of three, after the last wait, pauses with the same words and “after waiting 15 minutes, 30 minutes and 1 hour”. A ship clears all of it; a story miss breaks the row without clearing the waits. Learned traits are never rolled back for a failure the environment caused.

What to do. Open the stories that did not ship: each failed email and draft pull request carries the open findings. Fix the brief or the code, or close the story. Then press Resume or run foreman resume; the counter starts from nothing. For environment endings, fix the machine first: foreman doctor, the slot’s environment, disk and tools.

Rate limits, credits and pauses that resume on their own

A rate limit or an outage kills a session with an API error (a 429, a 5xx, “overloaded”). That is nobody’s fault: the story is queued to resume from where it stopped, or held if it had already committed, with the slot reading “held until the API is back (retry at 14:05)”. The run waits 15 minutes, then 30, 60 and 120, and stays at 120; a limit or outage never pauses the run. Waiting on you and foreman status show the FYI “The Claude API is limited or down (wait 2); Foreman tries again at 14:05 by itself.” Any session ending that is not a limit means the API answered and starts the ladder again at 15. Resume lifts the wait sooner. Nothing to do.

Platform credits run out. Sessions start on the Console API key (ANTHROPIC_API_KEY) when there is one. When one dies because the key’s balance is empty (“Credit balance is too low”) or the key reached the usage limit set on it, and claude.fallback is subscription (the default), the story is requeued with no miss and Foreman looks for a subscription sign-in: CLAUDE_CODE_OAUTH_TOKEN in .env.foreman, or the claude.ai login claude auth status reports, asked within 30 seconds. Every start waits for the answer. With one, Foreman records the switch in .foreman/claude-signin.json, starts every later session without the key, and the story carries on at once. You get one warn event, one credits email and an FYI that lasts while the switch does: “Platform credits ran out at 14:05; now using your Claude subscription.” or “The Console API key reached its usage limit at 14:05 (access returns …); now using your Claude subscription.” Sessions still on the key when it switched die the same way and are requeued, with no second email. Foreman never switches back by itself: run foreman claude use-key, or press “Use the API key again” on the Set up tab, after topping up. foreman doctor says which sign-in the sessions use.

With no subscription sign-in to switch to, or with claude.fallback: none, the run pauses instead, saying so: “Platform credits ran out at 14:05, and no subscription sign-in was found to switch to (set CLAUDE_CODE_OAUTH_TOKEN in .env.foreman or run claude auth login). Top up the credits or add a sign-in, then press Resume.” The subscription’s own usage limit (“You’ve hit your limit”) pauses the run too: “The account is out of credit or hit its usage limit. It resets <time>.”

These pauses resume on their own. A credit pause records when to try again: the reset time Claude printed, else an hour on. Waiting on you reads “Paused: <reason>. Foreman tries again at 14:05 by itself; Resume starts it sooner.” and the paused email carries the same sentence. When the time comes, Foreman runs one cheap one-turn probe in the sessions’ own sign-in; if Claude answers, the pause clears with one event and the queued stories start. A probe that fails tries again every 30 minutes. A reset time more than 35 days ahead is not believed, and an hour is used instead.

Expired Claude sign-in

What you see. A session dies with “Failed to authenticate”, an expired OAuth session or token, invalid_grant or a 401. Foreman reads that only from the session’s own error, never from the story’s output, so a Salesforce invalid_grant or the app’s own 401 does not trip it. The story keeps its claim and checkpoint and is queued to resume. The run then checks the sign-in itself: claude auth status --json, then a one-turn probe, in the sessions’ own sign-in environment, and every start waits for the answer.

• When the check passes, the stories carry on at once, with one warn event naming them. A story may die this way 3 times in a row; the 4th death counts as a plain crash, with the event “ACME-12: its session died on the Claude sign-in 4 times in a row while the sign-in check passed; counting it as a crash.”
• When it fails, the run pauses with “Claude sign-in expired: run claude to sign in again.” and one paused email. It probes again about once a minute, and at once after a restart, and clears that pause by itself when a probe passes. It never clears a pause you asked for.

Why. The claude.ai login on the machine was refreshed or expired, or a token was revoked. No probe output is stored, logged or shown.

What to do. On the machine the run is on, run claude and sign in again as the same account, or save a fresh CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) on the Set up tab’s Claude step. The next probe clears the pause. foreman doctor’s “claude sign-in” check names the sign-in the sessions use and what to do when it is not ok.

Config invalid

What you see. “config invalid since 14:05: <reason>; running on the last good config” on Waiting on you and in foreman status, one warning event, and one config email with the subject “foreman.config.json is invalid”. foreman status reads the last good config and says so on stderr.

Why. An invalid foreman.config.json at start stops foreman run, and every other command refuses it. One that turns invalid while a run is going does not stop it: the run carries on with the last config that passed, saved as .foreman/config.last-good.json (the team layer only, without your local settings), and keeps showing the stories from it. The reason is cut to 300 characters.

What to do. Run foreman config check to see what is wrong, fix the file, and nothing else: the run reads the fixed file at its next pick, with no restart. A local setting in .foreman/settings.json that is not valid is simply ignored, and foreman doctor names it.

After a crash or reboot

What you see. Start-up events such as “Start-up: resuming ACME-12.”, “Start-up: closing ACME-10.” or “Start-up: letting go of ACME-11; its release is parked and finishes later.”, and the stories a dead run was building appear in the queue again. With foreman service install, the run comes back by itself at log-in and after a crash; otherwise foreman run starts it.

Why. Before the supervisor picks anything new it reconciles what it finds, in this order, each story in one action at most:

1. A release cut off after its point of no return (the ship lock left at merge or migrate-supabase) is finished, not redone.
2. A story whose pull request merged but which is not marked done is closed.
3. A parked release’s stale claim is let go; foreman release watch finishes it later.
4. Stories a dead slot was building are resumed, in priority order, and not pinned to their old slot: a free slot asks the picker first and starts a story it ranks higher before resuming one. Each old folder is freed first (unfinished work committed on the story branch, the folder detached at the base); a folder that cannot be freed keeps its story in that slot.
5. Stale locks are cleared and a stale ship lock broken.

A story started again carries on from its last finished stage. Slots of a dead run are idle again unless they were parked, held or down, which wait for you. A run stopped with Ctrl-C, a kill or a closed terminal ends its sessions as interrupted; their claims stay, so the next start resumes them the same way. foreman run --dry-run prints what each slot would do without doing it.

What to do. Usually nothing. Check foreman status once the start-up events have passed; a slot that is parked or down says why on Waiting on you.

Leftover branches

What you see. A warn event: “Slot w1: ACME-12 left changes in its folder; they are on the branch leftover/w1-acme-12-20261109, and the folder is clean again.” Sometimes a second one: “Slot w1: ACME-12 left key files in its folder (.env.test); they are not on any branch and were deleted.”

Why. When a story ends, or you Skip, Take over or Restart it, Foreman commits whatever its session left uncommitted (staged, unstaged and untracked; not ignored files) to a local branch leftover/<slot>-<story>-<YYYYMMDD> (-2, -3 when the day already has one), then resets the folder and detaches it at the base, so the next story starts clean. The commit carries its own identity and skips your git hooks, so neither stops the save. The branch is never pushed. New key files (an .env* file other than the templates, a .pem, an SSH key, the config’s env file names) are deleted, not committed. Changes only under the generated paths are dropped with no branch. At the pick, a dirty folder still on the branch of a story nothing holds any longer is saved the same way.

When a dirty folder cannot be saved (“Slot w1: could not save the leftover changes of ACME-12 (…); the folder is left as it was.”), or holds changes on no story’s branch, the slot parks with “Its folder has changes that do not belong to a story.” and the event names the branch it is on. You get a parked email, and the card offers Resume as is and Discard changes and reset.

What to do. The branches live in your repository, so list them from the main checkout with git branch --list 'leftover/*'. Cherry-pick what you want to keep and delete the rest with git branch -D. For a slot parked on a dirty folder, look in .foreman/worktrees/wN, then choose Resume as is to keep the changes or Discard changes and reset to drop them.

Who stopped the run

What you see. foreman status reads “Foreman: stopped by your terminal at 09:12 UTC”, and while the run still finishes its stories, “Stop requested by <who>: the run finishes the stories in progress, then stops.” The event says the same with the detail: “Stop requested by ACME-9’s session at 04:01 UTC (pid 4821, parent “node …”, in /…/worktrees/w2): finishing the stories in progress.”

Why. foreman stop, the dashboard’s Stop and the cloud runner’s stop each leave a record in .foreman/STOP saying who asked, with the command line and folder cleaned of anything that looks like a secret. Who is one of: your terminal, the dashboard (a browser), a request to the dashboard (with its user agent), the cloud runner, a Foreman run process, a story’s session, or an unknown process. Once taken, the record is kept in .foreman/last-stop until the next fresh start, so a cloud run’s stop stays named. A stop that came from anywhere other than your terminal, a browser on the dashboard or the cloud runner, for example a story’s session, sends an unexpected-stop email at once: “Foreman was asked to stop, not by you”. The run still finishes the stories in progress, then stops, and stays stopped until foreman run. A run that stops normally sends a stopped email with how many stories shipped; one that ran with --once does not.

What to do. If you did not mean it to stop, foreman run. If a session stopped it, look at that story’s last commands in the command centre before you start again.

A halt stopped releases

What you see. foreman status reads “halted”, Waiting on you shows the halt at the top with an “I’ve fixed it” button, held slots read “Committed; releases are frozen by a halt.” or “This story stopped the release line; waiting for you to fix the cause.”, and you get a halted email naming the step and the reason. foreman ship exits 8 with “Releases are frozen by a halt.” while it lasts.

Why. The ship runner writes .foreman/HALT when a step marked as halting fails, when any step fails after a point of no return (the hosted database changed, or the merge happened), for a failed migration, and for a failed deploy or smoke test. When a step after a successful deploy failed and the project has a rollback command, it rolls production back first and the reason says whether that worked. Releases stay frozen so no other story merges on top of a broken one.

What to do. Read the reason, fix the cause, then press “I’ve fixed it” or run foreman fixed. That runs foreman health: the health check of every configured ship step that has one (the hosted database has exactly the base’s migrations, the base’s newest commit is deployed, the live site answers below 500, your watch command reports nothing). Only when they pass does the halt clear (“Halt cleared: …”, or “Halt cleared; no health checks are configured.”) and the held stories ship, oldest first; otherwise the run stays halted and the event reads “Still halted: <reason>”. foreman health --site runs the live-site check alone. Release safety has the steps, exit codes and the rollback.