Build orgs

A build org is the Salesforce org a story is built and tested in before it is released. The Salesforce pack gives each slot (one copy of the repository that one story builds in) an org of its own, in one of two ways: a sandbox leased from a pool you list in salesforce.build, or a scratch org made per story through a Dev Hub and deleted when the story closes. This page is for whoever chooses between them and wants to know what Foreman does to the org before, during and after a story. The salesforce block has the keys; Slots has what every slot gets whatever the environment. The Salesforce pack is a preview: its release paths pass their own tests, but no story has yet shipped through them on a live org, and the Gearset path has not run against a live Gearset pipeline.

Whichever mode you pick, Foreman asks the org SELECT Id, IsSandbox FROM Organization before it uses it: it must be a sandbox (a scratch org reports itself as one) with the org ID the config gives it. Production is never a target. Every sf call runs through Foreman’s own sf home (What the pack adds); git and gh run with your real home, where their credentials are. The tester’s dry run goes to the slot’s build org unless salesforce.test names a stage.

The sandbox pool

With salesforce.build set to one alias or a list of aliases, the pack sets env.adapter to sandbox-pool. Slot w1 leases the first alias, w2 the second, and so on: one slot per build org, so slots may not be more than the list is long. When stage 1’s org is also a build org, that org is the only build org and slots is 1, because one org cannot build one story while it takes the release of another. The environment takes no options of its own; it needs sf, git and gh.

Before each story’s session starts, Foreman prepares the slot’s org in this order, and stops at the first refusal, which puts the slot down with a notice (When things go wrong):

1. The lease. The slot’s build org from salesforce.build, or a refusal when it has none (slot w3 has no build org in salesforce.build).
2. The identity check. The org must be a sandbox with the configured org ID; acme-dev1 is not a sandbox; production is never a target or acme-dev1 is org 00D..., not 00D... stops everything else.
3. The default org. target-org is set in the worktree’s own .sf/config.json, never globally, so the agents give no target-org flag and reach nothing else.
4. Source tracking. A tooling query of SourceMember checks the org tracks source changes; an org that does not (source tracking is off in acme-dev1) is refused. Set up on Salesforce says how to turn it on.
5. The clean-up. Described below. It runs only when the slot has a story, and only when the last story built in that org was a different one whose pull request did not merge.
6. Tracking reset. sf project reset tracking, so changes already on the org, the clean-up’s included, are never taken for the builder’s.
7. The record. The org’s record now names the slot’s story, written before the session’s first deploy.

The record of each build org lives in the state folder at salesforce/orgs/<org ID, 15 characters>.json as { alias, orgId, story, branch, at }. A failed clean-up keeps the old record, so the next prepare tries again; a prepare with no story leaves it alone, so the next one that has a story still cleans up after the last. Prepare answers reset when the clean-up deployed something, else up.

A pool sandbox keeps what a merged story deployed to it. That is deliberate, and it is what lets a build org also take stage 1. When you refresh a sandbox it gets a new org ID: put the new ID in salesforce.orgs, connect the org again on the Set up tab, and expect the doctor to start a new baseline, because a saved baseline for another org ID is ignored.

Scratch orgs

With salesforce.build set to { "scratch": { "devHub": "<alias>", "definition": "<file>" } }, the pack sets env.adapter to scratch. Each story gets a scratch org of its own, made from the definition file through the Dev Hub, which must be in salesforce.orgs and is often the production org, marked "production": true. The Dev Hub is used only to create, list and delete scratch orgs; it is never a build org or a stage. Scratch mode sets no slot limit. The environment needs sf.

Preparing a slot takes minutes, because a scratch org does, and goes like this:

1. A story is needed. A slot with no story is refused without an sf call: scratch orgs are made per story. The org’s alias is foreman-<story id, lower case>.
2. Reuse. When the story already has a ready record naming that alias, and the org behind it is still a sandbox with the recorded org ID, the slot reuses it and answers up, so a story that resumes after a crash or a credits pause keeps its org. A record that fails that check, or a creating record a dead run left, is deleted first and a new org made. A creating record with a job ID and no org ID is resumed once first (sf org resume scratch --wait 0), so an org sf can still sign in is recorded and deleted, not left to count against the Dev Hub; one still being made is kept and the slot refuses rather than make a second.
3. The Dev Hub’s limits. sf org list limits on the Dev Hub. With no DailyScratchOrgs left, the slot refuses with no create call (<devHub> has used its N scratch orgs for today); resume the slot once the Dev Hub has more. With no ActiveScratchOrgs left, Foreman first deletes, at once, the orgs of stories that are in no slot, then reads the limits again, and refuses if there is still none.
4. The record, then the create. A creating record is written before sf org create scratch --async, so a run killed while sf signs the org in still leaves the org’s username on record and the isolated home accepts it. The org is made for 2 days.
5. The wait. sf org resume scratch --wait 0 every 30 seconds, for up to 20 minutes. A timeout keeps the record for the next prepare or the sweep; any other failure deletes the org.
6. The identity check, as in the pool: a sandbox with the org ID resume gave.
7. Ready. The record turns ready, target-org is set in the worktree’s .sf/config.json, and prepare answers reset: a fresh org.

The records live in the state folder at salesforce/scratch/<story id>.json as { story, slot, alias, devHub, state, jobId, username, orgId, createdAt, readyAt? }, with state either creating or ready. The isolated sf home reads them so a story’s scratch org is allow-listed for that story only: by org ID once it has one, by exact username while it is being made, and only while the record names the configured Dev Hub and the Dev Hub lists the org as active. Outside scratch mode the records are ignored, so a pool project’s home never widens.

The sweep. Every minute, while running, paused or halted, the supervisor deletes the scratch org of every story that has been in no slot for 5 minutes: when a story ships, is parked, is rejected or is stopped and nothing picks it up again, its org goes within about six minutes. A creating record is resumed once first, and skipped without a warning while its org is still being made. A delete that fails leaves a warning and the record, so the next sweep tries again. The org’s record goes with the org; an org left in the home with no record is a stray, and the next run refuses to start until it is signed out.

Messages about scratch orgs name only aliases, story ids, the Dev Hub and org IDs; every word with an @ in sf’s own output becomes “an org”, so no username reaches a notice.

Source tracking and clean-up

Source tracking is what tells the builder’s changes apart from what was already in a build org. Both environments set the slot’s default org in the worktree’s .sf/config.json, so every sf command in the session targets that org and nothing else, and the pool resets tracking before each story so the org’s history starts clean for that story.

The clean-up of a pool sandbox

A pool sandbox keeps a story’s deploys after the story leaves it. When the story merged, that is right. When it did not (parked, rejected or stopped), the next story in that org would build on top of changes that are not in the base branch. So, when a slot’s next story is a different one and the previous story’s pull request did not merge (gh pr list --head <branch> --state merged finds nothing), Foreman gives the org the base branch’s version back:

• It fetches the base branch (up to 3 tries) and works out the reverse delta, from the old story’s head to the base, with sf sgd source delta --generate-delta, in a temporary worktree of the base, so the slot’s own worktree is never touched.
• It deploys that delta with sf project deploy start, the manifest as the package and the components the story added as post-destructive changes, --test-level NoTestRun, waiting up to 30 minutes. It is the one deploy the pack runs without tests, and it only ever goes to a build sandbox.
• It then lists each deleted type in the org (sf org list metadata) and refuses if any removed component is still there (acme-dev1 still holds ApexClass AcmeService after the clean-up).

A delta with nothing in it means nothing to clean. A story whose branch is gone is skipped. A clean-up failure keeps the record, so the next prepare tries again.

The baseline

An org usually fails some tests before any story touches it, and its coverage may already be under what a deploy needs. The baseline records that, once per org, so a story is never blamed for an inherited failure. The doctor’s salesforce baseline <alias> check starts it: sf apex run test --test-level RunLocalTests --code-coverage on the org, without waiting, saving the test run ID. Later doctor runs read the run through the tooling API until it completes, then save the failing tests (each Class.method, sorted, at most 200) and the org-wide coverage as a whole percent, and keep that until you refresh it. The file is salesforce/baseline/<alias>.json in the state folder, as { alias, orgId, testRunId, startedAt, state, finishedAt?, failing?, coverage? }, with state one of running, done or aborted; nothing else is copied from sf. A record for another org ID (a refreshed sandbox gets a new one) is ignored.

Every org with a role gets one: the build orgs and each stage’s org, a shared org once. For a stage org, failing tests or coverage under 75% warn that RunLocalTests cannot pass there; a build-only org warns only on failing tests, because no deploy with tests goes to it.

foreman sf baseline prints what each org already fails, as saved, and runs no sf. With --story, which the tester runs before its dry run, it adds when the current branch was created and lists an org’s failing tests only from a baseline that finished and was written before then, so a baseline saved during the story never counts. With --refresh it drops each saved baseline and starts the tests again, sandboxes only. The release steps read the same records: sf-validate and sf-deploy call a failure inherited when the only failures are tests the baseline lists, plus an org-wide coverage warning while the baseline coverage was already too low. Salesforce ship steps has that side.