How Foreman picks

Every tick, each free slot (a worktree of the project that builds one story at a time) asks which story to build next. This page is for anyone who wants to know why a story started, why another did not, and what the words “deferred”, “kept for it” and “Taken by you” mean. The rules are the same for every story source: markdown files, Jira issues and Salesforce Cases.

The order

A story can be picked when all of this holds:

• its status is todo, in-progress, or blocked with a blocked_by that starts with story ids (“ACME-3, ACME-4 must land first”) that are all done;
• every story in its depends_on is done, whatever group it is in;
• it is not an owner task, not skipped for this run, not claimed by another slot, and not retired or done.

Among those, the order is: a story left in-progress by a run that died first, then the story’s order (the order field of a markdown story, the position in the Jira filter’s ORDER BY, the position in the Case query’s orderBy), then the lowest number at the end of the id, then the id. The same ranking decides which interrupted stories resume first after a restart. A story that fails to pick for a reason below is simply left out, and the next one in line is taken.

foreman story next prints the story the picker would take and foreman story plan plays the whole run forward: the order the factory would build, every open story it cannot reach yet and what each waits on, and the open owner tasks. Skip on the command centre leaves a story out for the rest of the run; foreman next <id> (Build next) asks for one story first. That request is tried first at every pick, stays open until the story has started, and after 3 failed starts shows on Waiting on you as “ACME-12 cannot start next” with the reason. Only a story that can be built now is taken.

When nothing can be picked, the run stands by and reads the story source again each tick. A story not shipping is a miss only when the pipeline rejected it (failed after every build round): three misses in a row pause the run with “Three stories in a row did not ship: ACME-1, ACME-2 and ACME-3.” A ship or a parked release clears the count. A story that is blocked, deferred, waiting on you, an empty pick, a crash, a watchdog stop or an ending the environment caused neither counts nor clears. Three environment endings in a row wait 15, 30 and 60 minutes before trying again; only a fourth row pauses. Resume is on the command centre or foreman resume; see Run, pause, stop.

Dependencies and depends_on proposals

depends_on is the list of stories that must be done before this one starts. In a markdown story it is a frontmatter field; in Jira it is the issue’s “is blocked by” links; in Salesforce it is the field named dependsOnField. A dependency outside the Jira filter or the Case query is still checked: done counts as met, open holds the story, and foreman story plan says “outside the filter, not done”. foreman story set <id> --depends-on <ids> replaces the list by hand; it refuses a story depending on itself or a cycle. A dependency Foreman adds or removes is written back to the source (a Jira link, the Case field) whatever the comment switches say.

A story is also held when its blocked_by names stories, which is how the architect blocks a story that must wait: foreman story block <id> --reason "ACME-3 must land first". The picker takes it again once those stories are done. A reason in words, such as a question only you can answer, waits for you instead.

When the architect finds a story cannot be built until other stories land, and those ids are not yet in its depends_on, Foreman proposes the fix. With features.dependsOnProposals on (default on), the session asks on Waiting on you: “Add depends_on: ACME-3, ACME-4 to ACME-12?”, with Apply (recommended: “ACME-12 waits until ACME-3, ACME-4 are done, then starts by itself”) and Ignore (“Nothing changes: its list of what must finish first stays as it is”). Apply adds the ids to the story’s depends_on, keeping the ones it had. With the switch off nothing is proposed, and the story stays blocked until you act.

With features.autoApplyDependsOn on as well (default off), the session applies the fix at once with foreman story depend <id> --on <ids> --reason <text>: the ids join depends_on, the story goes back to todo with the progress note “Now waits for ACME-3: <reason>”, and one FYI on Waiting on you, “ACME-12 now waits for ACME-3”, carries Undo (takes the dependency off again) and Keep (clears the note). The command refuses when the switch is off. The agents never edit depends_on themselves. Both switches are on Settings and switches.

Two stories never change the same file

With two or more stories running at once, two of them never change the same file. Each design lists the files its story will create, change or delete (files in the architect’s plan, as paths from the repository root, tests included). Before the build, the slot claims them with foreman lock files <id> --plan <file>: one lock per file at .foreman/locks/file-<hash>.json, naming the slot and the story. Files every story touches are never claimed: the globs in fileLocks.ignore of foreman.config.json (["CHANGELOG.md"] by default; your list replaces it) and every generated path.

When another running story holds one of those files, the claim fails (exit 7), nothing is kept, and the story is deferred: the session puts it back to todo with the reason, such as “waiting for ACME-1 to ship: it is changing src/a.mjs” (at most 200 characters), and stops. A deferral is not a miss. Foreman writes a wait record, .foreman/waits/<id>.json, with the holders, the held files, every file the plan claims, and how often the story was deferred on each holder.

The picker then leaves the deferred story out while any of this holds:

• a running story holds a file its wait record or saved plan names;
• a running story holds an exclusive resource its saved plan names;
• a running story’s design names a file the record needs, even before that story has claimed it;
• it was deferred twice on a story that still runs, until that story ends;
• a free file it needs is kept for another story (next section).

A deferred story is not designed again at a cost while it waits: its plan is saved in .foreman/plans/<id>.json and the picker reads it. When the story comes back to a slot, after a deferral, a stop or a restart, it keeps that design. The plan records the commit the base branch was at when it was written, and the architect runs again only when one of the files the plan names has changed on the base branch since. A plan saved before Foreman 0.10.0 records no such commit, so it is always redone, and the new plan records one. Either way the session’s stderr says which it did: “reusing the plan saved for ACME-2” or “not reusing the plan saved for ACME-2”, with the reason, such as which of its files changed. The board’s recent list shows the story as deferred with the reason, and foreman status lists each one under “Waiting on running stories”: “ACME-2 waits for ACME-1: src/a.mjs”. A story’s file locks go with its slot when the story ships, blocks, fails, defers or finds nothing to build, so the waiting story starts by itself. A fix round after review claims its files again the same way.

Files kept for the top waiting story

A file a story lets go of does not go to whichever story plans first. Just before a slot’s locks are removed, each freed file is kept for the highest-priority story whose wait record names it, that was held up by nobody but the story letting go, and that does not itself give way to a higher waiter. The reservation is .foreman/locks/reserve-<hash>.json and lasts 30 minutes (a constant, not a setting); the event reads “Kept src/a.mjs for ACME-2, the highest-priority story waiting on it.”

While it lasts, another story that claims the file is refused with “waiting for ACME-2 to start: src/a.mjs is kept for it”, and the picker leaves out a lower-priority story whose plan or wait record names a file that a higher-priority waiter’s record names. The reservation is dropped when the kept story claims its files or its session ends. When that story starts, the free files its record names are kept for it first, so a story started in the next slot on the same tick cannot take them. A Build next request never gives way to a higher waiter; only the automatic pick does.

Waiting on you and foreman status name the top waiting story and the story holding its file: an FYI titled “ACME-2 waits for ACME-1” whose detail reads “The highest-priority story waiting on files: ACME-2 waits for ACME-1: src/a.mjs”, or “… src/b.mjs (kept for it)” when the file is free but kept.

Exclusive resources

An exclusive resource is something only one story may change at a time, such as a shared database schema. Name them in the resources block of foreman.config.json:

"resources": {
  "schema": {
    "paths": ["prisma/schema.prisma", "prisma/migrations/**"],
    "describe": "The database schema and its migrations. One story at a time."
  }
}

A name is a slug (lower-case letters, digits and hyphens). paths are the globs the resource covers and describe tells the architect what it is: the architect sets resources in its plan from that list, or leaves it empty. Each resource is one lock, .foreman/locks/resource-<name>.json; the one named schema is the schema token, kept at .foreman/locks/schema.json so a project can move to Foreman from the earlier factory with a live token (an older resource-schema.json is moved there).

When a plan names resources, the session takes each with foreman lock take <resource> <id>. When one cannot be taken, it releases the ones it took, puts the story back to todo with “waiting for <resource> to be free”, and stops deferred; the supervisor then keeps that story, and any story whose group needs a resource (stories.resourcesByGroup in a markdown source), out of the pick while the token is held, and takes the token for it when it starts. foreman lock held <name> says whether this slot holds a resource and foreman lock release <name> frees it; a lock is never taken from another slot. Every lock a slot holds goes with its story’s ending.

The release step resources is the safety net: when the branch changed a file matching a resource’s paths without holding it, the step takes the resource if it is free and stops with code 7 when another live slot holds it. See How a story ships.

Taken by you

You can take a story out of the run and keep it. “I’ll take it” is one of the answers to “ACME-12 could not pass review”, and Take over is a slot action on the command centre while a slot is building or held. Either way Foreman commits any work in progress on the story branch (“ACME-12 WIP: handed to the owner”), pushes it, claims the story for you and blocks it with blocked_by “Taken by the owner” and a progress note saying where the work is. Its branch and draft pull request stay, its locks are released, and the slot moves on. From a terminal, foreman lock owner-claim <id> makes the same claim.

The claim is a file, .foreman/claims/<id>.json, marked as yours; it has no process behind it and never goes stale, so a restart leaves it alone and the picker never offers the story. The dashboard lists these stories under Taken by you with a Hand back button, and foreman handback <id> does the same: the claim is removed, the story goes back to todo with blocked_by cleared and the note “Handed back to the factory from the command centre.”, and the run can pick it again. The slot actions are on Slots and the commands on Every command.