Agentforce and OmniStudio

This page is for teams whose Salesforce project holds Agentforce agents or OmniStudio components. It says what Foreman does with each when a story is built and released, what the builder must write for an agent to ship, and which parts of those products the pack does not support. The Salesforce pack is a preview: its release paths pass their own tests, but no story has yet shipped through them on a live org, and the Gearset path has not run against a live Gearset pipeline.

foreman init detect reports whether the source holds either kind of metadata: OmniStudio when a package folder has omniScripts, omniIntegrationProcedures, omniDataTransforms or omniUiCard, Agentforce when it has aiAuthoringBundles, bots, genAiPlannerBundles, genAiPlugins or genAiFunctions. Two keys in the salesforce block then say what to do: agentforce (auto or off, default auto) and omnistudio (auto, standard, managed or off, default auto). See The salesforce block.

Agentforce: publish, activate, evaluate

Unless salesforce.agentforce is off, the pack lists a third builder beside the admin and developer builders: agentforce, the foreman:sf-agentforce agent, for “Agentforce: agent bundles, their actions and the Apex or Flows behind them”. The architect sends a story to it when that is the work. The builder writes the agent’s Agent Script bundle at force-app/main/default/aiAuthoringBundles/<Api>/, the actions it calls (an Apex class with one invocable method, or a Flow), the Apex tests first, a permission set for a service agent, and one eval spec per changed agent. It validates the bundle in the slot org with sf agent validate authoring-bundle and never publishes or activates: the release does that, in each stage’s org. It also never runs sf agent test create or run, because create writes a test definition into the package folders and run-eval in this CLI refreshes its sign-in in a loop until the org locks out logins; the Bash guard denies run-eval for the same reason, and the tester previews a changed agent with sf agent preview instead.

In a release, sf-delta first refuses, with code 2, any AiEvaluationDefinition in the package manifest, and checks that every AiAuthoringBundle in the manifest has its eval spec at the story’s commit. Then, after the deploy to each stage, the sf-verify-<stage> step runs the Agentforce check. The agents are the bundles in the story’s delta; with none the check passes with no sf call. An agent’s API name must be a letter followed by up to 79 letters, digits or underscores, because it goes into a query. Before any sf call the check fails when the Agentforce switch is off (below), when a spec is missing or malformed, when the story changed more agents than one release may test, or when the org has no room for one more eval suite per agent. Then, in the stage’s org:

1. every bundle is validated with sf agent validate authoring-bundle
2. for each bundle in order: Foreman reads the agent’s versions and remembers the active one as the rollback target (none for a new agent); publishes with sf agent publish authoring-bundle --skip-retrieve, so the new Bot and GenAi metadata never land in the ship worktree; reads the versions again, where the new highest is the story’s; records the agent on the story’s release record before activating, so a later failure can be rolled back; activates with sf agent activate --version <n>; and reads once more to confirm that version is active. A publish has 10 minutes.
3. Foreman opens one agent session for the org and, for each agent, finds the planner of the activated version (exactly one GenAiPlannerDefinition named <Api>_v<version>), records one suite on the counter, runs the spec’s cases, and re-runs once only the cases whose routing checks failed.

A routing check that fails again on the re-run fails the whole check with code 5 and halts; the agents are already on the release record, so a stage 1 rollback puts them back (each previous version reactivated, or an agent the story created deactivated). An outcome the judge scored below its threshold, and a routing case that passed only on the re-run, do not fail the release: they become notes in the story’s manual steps, such as “Check agent Acme_Support in qa: the judge scored case order-status below 3”. The response and a JUnit file for each run are kept at .foreman/salesforce/evals/<story id>/<stage>-<Api>.json and .junit.xml.

A publish is a deploy, so once one succeeded the check tells sf-verify that it changed the org, and the step reads the org’s newest deploy again and takes the publish as Foreman’s; see sf-deploy and sf-verify.

Licences and limits

Foreman runs the eval through the same API the CLI uses, with the named-user token that sf agent preview and publish use, and a planner session, which an employee agent needs. The reported limit is 10 eval suites in 10 hours per org, and Foreman keeps to it with a counter at .foreman/salesforce/agent-evals.json, one suite per agent per release. A story that changes more than 10 agents can never ship in one release; the reason asks you to split it. When the org has no room, the reason says how many suites ran in the last 10 hours and when the next can run. No eval call is retried: an answer of 429, or one naming a rate or request limit, fails the check so the release can be run again later, and an org that answers “login rate exceeded” is cooled down for an hour, during which Foreman makes no sf call to it. Cases are sent in batches of at most 5, in parallel, and a batch has 10 minutes. The session token and the named-user token never appear in a reason, a file or a log.

foreman doctor has a salesforce agentforce check. Unless salesforce.agentforce is off, it asks each org with a role (the build orgs and each stage’s org) whether Agentforce is on, by querying GenAiPlannerDefinition, the object the eval needs, and whether the org has an active Data Cloud permission set licence. When one org lacks either, the doctor turns Agentforce off with the reason, written to .foreman/salesforce/agentforce.json, and warns “Agentforce is off: <alias> has no Data Cloud licence”; the release then refuses any story that ships an agent bundle, with that reason. When every org has both again, the next doctor run turns it back on. An org that failed its own org check, or a query that failed, only warns and leaves the switch as it was. The doctor’s salesforce tools check also warns when the @salesforce/plugin-agent CLI plugin is missing, unless agentforce is off.

The eval spec

An eval spec is the builder’s test plan for one agent, at tests/<Api>-eval.json, outside the package folders. The release reads it at the story’s commit with git show. Its shape:

{
  "agent": "Acme_Support",
  "tests": [
    {
      "id": "order-status",
      "utterance": "Where is order ACME-1001?",
      "routing": [
        { "field": "lastExecution.llmEvents", "operator": "contains", "expected": "'agent_name': 'Order Status'" },
        { "field": "lastExecution.invokedActions", "operator": "contains", "expected": "'name': 'Get_Order_Status'" },
        { "field": "invokedActions[0][0].function.input.orderNumber", "operator": "equals", "expected": "ACME-1001" }
      ],
      "outcome": { "expectation": "The reply gives the order's status.", "threshold": 3 }
    }
  ]
}

The rules, each checked before any sf call:

Field	Rule
agent	Must equal the bundle’s API name, which is also the file name
tests	A non-empty list
tests[].id	1 to 80 letters, digits, _ or -, unique in the file
tests[].utterance	Text of 1 to 2000 characters
tests[].routing	A list (may be empty when there is an outcome)
routing[].field	lastExecution.llmEvents, lastExecution.invokedActions or invokedActions[0][0].function.input.<input>
routing[].operator	contains or equals
routing[].expected	Non-empty text
tests[].outcome	Optional; expectation is non-empty text, threshold a whole number from 1 to 5, default 3
each test	Needs a routing check or an outcome

A routing check runs as a string assertion and blocks the release when it fails twice. The outcome is a response rating: the judge scores the reply against the expectation, and a score below the threshold is a note, not a failure. A subagent check names the subagent’s label, not its API name; an action check names the action’s definition name. The builder covers each subagent and action the story adds or changes, with at least one case that should not reach the new action. A failed, erroring or missing routing check, a case with errors and a case with no result all count as routing failures, and the expected value in a message is always the spec’s, never what the API returned.

OmniStudio modes

salesforce.omnistudio accepts auto, standard, managed or off. datapacks is refused with “not supported yet”. Today the key does not change what Foreman does: no OmniStudio builder ships, so the architect gives OmniStudio work (Data Mappers, Integration Procedures, OmniScripts, FlexCards) to the developer builder whatever the key says, unless you add your own OmniStudio agent under roles.builders with a use line, in which case the architect picks it. OmniStudio components deploy with the rest of the delta as ordinary metadata.

What the pack does do for OmniStudio: the code reviewer reads changed OmniStudio components against two rules, that the dependency order holds (Data Mappers, then Integration Procedures, then OmniScripts and FlexCards) and that nothing is hard-coded by ID, and it runs the omnistudio-dependencies-analyze skill on them. The sf-verify-<stage> step has no OmniStudio check yet; the code says one joins the Agentforce check later.

Not supported: DataPacks, DevOps Center

• DataPacks. Foreman has no step that exports or deploys OmniStudio DataPacks, and "omnistudio": "datapacks" is refused by the config check as not supported yet. OmniStudio metadata in the package folders travels with a story as ordinary metadata.
• DevOps Center. "release": "devops-center" is refused as not supported yet; the release paths are sf, ci and gearset. DevOps Center’s own commands count as deploys and are refused against a production org (sf devops promote, devops promotion complete|validate, project deploy pipeline *), and a story session’s Bash guard denies project deploy pipeline start outright through its -c flag, so no pipeline promotion starts from a run.
• OmniStudio checks after a deploy. None run yet, as above.
• Agent tests through the CLI. sf agent test create and sf agent test run are never run, and AiEvaluationDefinition files are refused in the package folders; the eval spec replaces them.