How Foreman learns

Foreman learns from its own mistakes. When the same kind of problem keeps coming back, it writes a short rule (a learned trait) or a how-to (a learned skill) for the agents, uses it from the next story, and tells you on Waiting on you. It watches each change on trial, takes it out again by itself if stories get worse, and you can undo any change with one click. This is on by default; see The kill switches below to turn it off. This page is for anyone who runs Foreman on a project and wants to know what it writes, where, and how to take it back.

What counts as a lesson

foreman learning scan --dry-run lists the lessons found in the last 60 stories (a story’s place in the window is its latest finding or run end). Without --dry-run the scan is written to <state folder>/lessons.json, a derived local file, for the learner to read. There are five kinds:

• Findings that repeat in meaning. Review findings that say the same thing in other words (the words their rules are about, compared within one stage family, review findings together), on 2 or more stories with a blocker or major finding (findings fixed in an earlier round count too), or on 5 or more stories at any severity.
• A failed story. A run record whose result is failed, unless the environment ended it (CI that never started, a conflict, a test file that would not load).
• A fix that did not fix: the same finding stopping one story in two rounds in a row.
• Your retry note: the “Note for the architect” you give when you retry a story.
• A repeated tool error: the same failed command or tool call (paths, ids and numbers made placeholders) in 2 or more stories, or 3 or more times in one session. Only the story session’s own tool calls reach its log, so a subagent’s tool errors are not seen.

Each lesson has a stable key, so it can be ignored, applied, or covered by a learned item already in force. A lesson Foreman took out itself after a trial is not sent to the learner again. The scan lists at most 5 pieces of evidence per lesson, each field cut to 300 characters.

Secrets are blanked before a lesson is made: the values in .env.foreman and the project env file, Foreman’s own key variables, anything in learning.redact, and token-shaped text such as JWTs and NAME_KEY=value. Before the learner sees anything, emails and your home folder are blanked as well, and paths are made relative to the project.

What Foreman writes

For each lesson, the learner, a short read-only Claude session, proposes what it should become. It runs claude -p in safe mode, with no CLAUDE.md, hooks or plugins of its own (the input carries the project’s CLAUDE.md), with only the Read, Grep and Glob tools, and with Read denied on ~/.ssh, ~/.aws, ~/.gnupg, ~/.config, ~/.docker, ~/.kube, ~/.claude, ~/.claude.json, ~/.netrc, ~/.npmrc, ~/.git-credentials, /etc and every .env* file. It gets the lesson, the diffs of the commits that fixed the where files (at most 20 KB), the project’s CLAUDE.md, the current traits and the skills with their descriptions, all scrubbed, and a budget of what is left of the day’s spend cap. It runs in the project, without the run’s FOREMAN_* variables, with Foreman’s own session keys; once the platform credits have run out it runs without ANTHROPIC_API_KEY, as the story sessions do. Foreman’s own code checks the proposal before anything is written. The result is one of two things:

• A learned trait: one rule of at most 400 characters for one role (or for every role). The agents read it after your own traits, under “Learned by Foreman (CLAUDE.md and the traits above win)”, so CLAUDE.md and your traits always win (Trait order). Learned traits are capped at 3 KB per role; past that the newest are left out, with a warning.
• A learned skill: a Claude Code skill whose description starts “Use when” (one line, at most 300 characters), with 1 to 12 steps and an example taken from the real finding and fix, at most 6 KB, under a slug of lower-case words joined by dashes, at most 40 characters. It is copied into each slot as .claude/skills/foreman-learned-<slug>/ and listed under “Learned skills” in the brief of each role it is for. Foreman never touches a skill skills-lock.json lists.

A proposal can also merge a lesson into a learned skill already in force, which becomes a new version of that skill, on trial again. A proposal that fails a check (too long, the wrong shape, a slug already used, text that looks like a secret) or hits a cap is recorded as a refused line in the ledger, with the reason and never the text.

Each learned item has an id, L-1, L-2 and so on, and the agents see it tagged [L-n]. When a learned rule is wrong, the tester and reviewers raise a finding with its [L-n] in the rule, and Foreman rolls that item back.

Where the files live

All of it is in the state folder (.foreman/ unless you set stateDir), never in your project’s git history:

• .foreman/learning.jsonl: the ledger, one line per change (add, update, keep, rollback, undo, refused), never rewritten. Each line names the item, its version after the line, the roles, the file, the text’s hash before and after, the lesson it came from, the reason, who made it (foreman or owner) and the numbers behind a decision.
• .foreman/learned/traits/<role>.md: the learned traits of a role, each tagged [L-n] (all.md for every role).
• .foreman/learned/skills/foreman-learned-<slug>/SKILL.md: a learned skill.
• .foreman/learned/.history/L-<n>/v<k>.md: the text of every version, written once, so any change can be undone. An undo checks the version it restores against the ledger’s hash and refuses when the file is missing or changed.
• .foreman/learner.jsonl: what each learner session cost, with its tokens.
• .foreman/lessons.json: the last scan, a derived file that is not kept on a state branch.
• .claude/skills/foreman-learned-* in each slot: the copies of the learned skills. They are excluded from git, so no story commit holds them.

Each change takes .foreman/learning.lock and fails with “another Foreman process is changing learned items; try again” when a live process holds it. With the state on a branch, each change is one commit there (Run state on a branch).

Trial and rollback

A new or changed item is on trial. Foreman watches the next learning.window stories that use it (5 by default), or 14 days when fewer come, and compares them with the 10 stories before the change on the item’s roles. Only a story that shipped, is ready to promote or failed counts; one that was stopped, crashed, rate-limited or blocked says nothing about the item, nor does a failure the environment caused, nor a story that ran under another item Foreman later rolled back. In order:

• It rolls the item back at once when a finding since the change names its [L-n], or when 2 of the stories on trial fail.
• Otherwise it waits until the window is full, or 14 days have passed with at least one story.
• When the trial is over, it rolls the item back if stories got worse (the average rounds rose by 0.5 or more, the average serious findings by 1 or more, or the failure rate by 20 points or more) while the problem it was meant to fix did not drop.
• It retires the item as “no effect” when that problem came back on 2 or more of the stories.
• Otherwise it keeps the item, as “Proven over N stories”.

A rollback or retire goes back one version, or removes an item that was new, and you get an FYI on Waiting on you (“Foreman rolled back L-n for <roles>” or “Foreman retired L-n for <roles>”, answered with Got it). When two items on trial share a story and both would be rolled back, the newer goes first and the older waits for the next pass, so one bad item does not take the others down with it. foreman learning show <id> has the numbers behind each decision.

Undo

Each change Foreman makes comes with an FYI on Waiting on you, “Foreman learned L-n for <roles>” or “Foreman changed L-n for <roles>”. Undo the change takes it out at once (back one version, or removed when it was new). Keep the change ends its trial and keeps it. Foreman applies the answer on its next loop, within seconds, even while stories are building; the answer acts only when the item’s latest change is still the one the note was about. Under the answered decisions, the Learned list shows every item with its version, status and reason, with Undo on each one still in force.

From the command line:

• foreman learning list: every item, with its kind, roles, version, status, path and reason.
• foreman learning show L-3: its history and the text in force.
• foreman learning undo L-3: back one version, or removed when it was new.
• foreman learning keep L-3: keep the version in force and end its trial.

Both undo and keep refuse in a story session on the run’s project. foreman status reads, for example, “Learning: 2 active, 1 in evaluation”.

The kill switches

• foreman settings set learning.auto false: Foreman writes nothing by itself any more, and foreman learn proposes traits for you to approve again (see below). Turning it off during a tick also drops the tick waiting behind it.
• foreman settings set learning.apply false: the agents stop using what was learned. The learned traits are left out of their prompts and the learned skills are removed from the slots. The files stay, so turning it back on restores them.

Both are settings, so the local layer, the team layer and the dashboard’s Settings tab, Learning group, all work (Settings).

Caps and cost

Every setting is in the learning block of foreman.config.json, or on the dashboard’s settings panel under Learning. Each applies from the next story.

Name	What it does	Default
learning.auto	Learn by itself, after each story.	true
learning.apply	Use what was learned from the next story.	true
learning.model	The model that decides what a lesson becomes: haiku, sonnet or opus.	haiku
learning.skillModel	The model that writes a learned skill: haiku, sonnet or opus.	sonnet
learning.window	The stories watched before an item is kept or rolled back. At least 3.	5
learning.maxPerRole	The most learned traits one role holds.	8
learning.maxSkills	The most learned skills in force. At the cap, a lesson that calls for a new skill waits: it is refused without asking the skill model, and asked about again once the lesson has another story and a skill slot is free. A merge into a learned skill in force is still written.	10
learning.maxEvaluating	The most items on trial at once.	2
learning.maxPerDay	The most changes Foreman makes in one day (UTC).	3
learning.maxUsdPerDay	A daily spend cap, in US dollars, for the learner sessions. Each session gets what is left of the day as its budget, and none starts when less than a cent is left. Each session’s cost is recorded in .foreman/learner.jsonl.	see the Settings tab
learning.redact	Extra words to blank before the learner sees anything. An entry written as /pattern/flags is a regular expression; anything else is literal text, for example ["Acme Corp", "/ACME-[0-9]+/"].	none

A cap of 0 is a cap already reached: learning.maxPerRole: 0 adds no traits, learning.maxSkills: 0 no skills.

Foreman runs the learner in the background after each story’s run record (foreman learning tick), one tick at a time, with at most one more waiting behind it however many stories finish meanwhile; a scheduled cloud job runs it at its end (foreman cloud tick, see Run in the cloud). A tick first evaluates the items on trial, then scans and sends each new lesson to the learner. A tick that fails logs one warning in the run’s events, “The learning tick failed (exit N); run foreman learning tick to see why”, and the stop of a run waits for a tick still going but never kills it. foreman learning run <key> --dry-run asks the learner about one lesson from the scan and prints its checked proposal without learning anything; without --dry-run it records the call’s cost but still writes no learned item.

Doctor and report

foreman doctor has three checks, each ok or a warning: ‘learning’ says whether learning is on, the active and evaluating counts, and the rollbacks and refusals of the last 7 days, and warns when the ledger cannot be read; ‘learning caps’, only with learning.auto on, warns at each cap reached (a role’s traits, learned skills, on trial, changes today, learner spend today); and ‘learned skills and pack tags’ warns about each learned skill that names a pack skill whose tag moved to another commit or left skills-lock.json. They print ids, names and tags, never a learned item’s text. foreman report adds a Learning section with the learner’s calls and spend and the counts of changes added, kept, rolled back, retired and undone, and its token report counts the learner as a role (Costs and reports).

foreman learn

foreman learn runs after each shipped story (one at a time, in the background, with a ship during a learn queued for the next), when the run reaches standby and when a run stops, each within 60 seconds and never holding the loop. For each shipped story with minor findings still open, it asks on Waiting on you whether to create a follow-up story (“Follow-ups from <ID>”) that fixes them, with the story’s checks as its Done when. Answer “Create a follow-up story” or “Ignore”. features.followUps turns the proposals off, and features.followUpsOfFollowUps decides whether a follow-up’s own findings get a follow-up at all.

With features.autoCreateFollowUps on (off by default), foreman learn creates each follow-up story at once instead of asking, with priority Low on Jira where the project allows it, and tells you in one FYI, “Foreman created N follow-up stories”, with “Undo and close them” (closes the ones not started yet; the rest are left and named) and “Keep them”. While proposals wait on you, the standby line says how many instead of “nothing to build”.

With learning.auto off, it also asks about rejections: when the same blocker or major finding sends three stories back to one builder (--min-stories N changes the number), it asks whether to add the rule as a trait for that builder. Answer “Add as a trait” or “Ignore”; either way you are not asked about that rule again, and Foreman’s own learning stays off that lesson too. If you ignored a proposal and change your mind, foreman learn reopen <decision id> takes it back and the next foreman learn asks again. foreman learn --dry-run prints what it would ask, foreman learn apply <decision id> and foreman learn ignore <decision id> are what the two answers run, and foreman learn undo <decision id> is “Undo and close them”. Its ledger is .foreman/learn.json.