Scenarios
Open in CMSThe Scenarios tab of the Conversation Simulation page is where you write, organize into folders, and run scenarios: test conversations that check how your agent answers. Found in the agent sidebar under Simulation → Scenarios. The page's second tab, Results, is described in Results.
AI-written content in this feature (suggested criteria, scenario names, the agent's answers and the match outcomes) can differ from one run to the next, so treat an AI suggestion as a draft to check.
Folders Pane
Shown on the left of the Scenarios tab. On a small screen it opens from a Folder: <name> button above the list, as a sheet that slides up from the bottom.
| New Folder | Opens Create New Folder. Enter a Folder Name and click Create. |
| All folders | Shows every scenario, whatever its folder. The number next to it is the total count. |
| Folder | Click a folder to show its scenarios. The number is how many it holds, including its subfolders. Whether the list also shows the subfolders' scenarios is set by Include Subfolders in the folder header. Use the arrow to expand or collapse its subfolders. Folders can be nested up to 3 levels. |
| ⋮ menu (or right-click a folder) | Opens the folder actions below. |
| Empty state | No folders yet ("Create a folder to organize your scenarios") with a New Folder button, shown until you create your first folder. |
| Retry | Shown with "Folders could not be loaded." when the folder list fails to load. |
If your agent has no folders at all, opening the form to create a scenario creates a folder named Default for you, so a scenario always has a place to go.
Folder Actions
| New Subfolder | Opens Create New Subfolder inside this folder. Not offered on a folder that is already at the third level. |
| Rename | Opens Rename Folder. Change the Folder Name and click Save. |
| Move to… | Opens Move "<folder>". Choose Top level or another folder as the new parent and click Move. Destinations that would break the 3-level limit are disabled. You confirm the move before it happens. |
| Run All | Runs every scenario in the folder after you confirm, then opens the run result. It follows Include Subfolders: when that is on, the scenarios of its subfolders are run too (the question says "... and its subfolders"). Disabled when there is nothing to run, and when it would run nothing it says "<folder> has no scenarios to run." |
| Delete | Opens Delete Folder. Deleting a folder also deletes its subfolders and the scenarios inside them, together with any private mock sets they own. This cannot be undone. Disabled when deleting it would leave no folder at all. |
Folders have no Duplicate action. Duplicating is only available on scenarios.
Folder Header
The card above the list, titled with the selected folder (or All folders).
| Include Subfolders | A switch, on by default. Shown only when the selected folder has subfolders that contain scenarios. When on, the scenario list shows the scenarios of those subfolders as well, and Run All (here and in the folder menu) runs them too. Turn it off to see and run only the scenarios placed directly in the folder. The choice is remembered while you stay in this browser session. |
| Run All (n) | Shown when a folder is selected and holds at least one scenario. Runs them all as one run after you confirm. The number follows Include Subfolders. |
| New Scenario | Opens the Create New Scenario form, placed in the selected folder. |
| Name | Filter: shows only scenarios whose name contains the text you type. |
| Apply / Reset | Applies the name filter, or clears it. |
Scenario List
| Sort By | Orders the list by Updated Date (default, newest first) or Name, Ascending or Descending. |
| Grid View / List View | Shows the scenarios as cards or as a table (columns Scenario, Steps, Folder). |
| Scenario card or row | Shows the name, its folder path and the number of steps. Click it to open the Edit Scenario form. |
| Checkbox | Selects the scenario for Run Selected. In List View the checkbox in the header selects every scenario shown. The selection is cleared when you select another folder, turn Include Subfolders on or off, or change the filter or sort. |
| Run | Runs this scenario after you confirm, then opens its result. See Run Confirmation. |
| Run Selected (n) | Shown when at least one scenario is checked. Runs all checked scenarios together as one run, after you confirm. |
| ⋮ menu (or right-click) | Duplicate (after you confirm, creates a copy named "<name> (Copy)"), Move to (opens Move Scenario to Folder: choose a Folder and click Move), Delete (after you confirm; cannot be undone). |
| Load More | Shown when more scenarios are available. Adds the next batch. |
| Empty state | No Scenarios Yet with a Create Scenario button, or No Scenario matches "<text>" with the hint "Try a different name, or reset the filter." when the name filter finds nothing. |
Run Confirmation
Every Run, Run Selected and Run All asks you to confirm first. The confirmation asks the question (for example "Are you sure you want to run the scenario ...?") and lets you choose which copy of the agent to test.
| Run against | Choose Published or Draft. The choice you made last time is remembered for this agent. |
| Published | Hint: "What your users talk to now." This is the default. |
| Draft | Hint: "Unpublished changes" when the draft differs from what is live, or "Same as published right now" when it does not. Disabled with the hint "No separate draft yet." when the agent has no separate draft. Choosing it shows a note: "Uses the draft's settings: prompt, model, tools, plugins, skills. Knowledge, documents and plugin data are shared with the live agent." |
| Escalation note | Shown for a Draft run of a single scenario (not Run Selected or Run All) when the scenario allows the escalation tool to run live: "Escalations from this run go to the draft escalation inbox." |
| Blocked tools list | Shown only when you run a single scenario (Run on a card or row, or Run in the scenario form), not for Run Selected or Run All, and only when the run would call tools that are blocked. It says "This run will hit n blocked tools. A blocked tool returns a policy error instead of running, so any step that needs it will fail." and lists each tool. The list follows the version you chose. |
| Mock It | A link next to each listed tool, shown when you run from the scenario form and the mocks feature is on. It opens a mock set editor seeded with that tool. Otherwise the hint tells you to cancel, open the scenario and mock the tools (or allow them live), or choose Run Anyway. |
| Yes / Run Anyway / Cancel | Yes starts the run. When blocked tools are listed, the button reads Run Anyway. Cancel closes the dialog. |
| Notes about the list | "Couldn't load all of the draft version's tools; this list may be incomplete." (or published) and "The agent's tool policy couldn't be loaded, so this list may include tools it runs live and miss tools it blocks." |
After you confirm, the page shows "Running scenario ...", then "... execution started successfully", and opens the run's result.
Scenario Information
In the Edit Scenario form this is a card at the top, titled Scenario Information. When you create a new scenario, it appears as a dialog with the same title after you click Save, once your steps are valid, with a name already suggested by AI from your steps. In the dialog, Continue moves on to "Are you sure you want to create this scenario?", and Cancel closes the dialog.
| Scenario Name | Required. The name shown in the list and in results, for example "Login Flow Test". |
| Folder | Required. The folder this scenario belongs to. |
Scenario Steps
A card titled Scenario Steps (n), or Scenario Step (1) when there is exactly one. All steps belong to the same conversation, just like a real chat with the agent. When there are no steps yet it says "No Steps added yet. Click "Add Message" to create your first Step."
| Collapse All / Expand All | Closes or opens every step card at once. On a narrow screen it is an icon with the same tooltip. |
| Step | One turn of the conversation. A closed step shows its number, its User Message (or "Step n" while empty) and small chips. Drag a step by its handle ("Drag to reorder steps") to change the order. |
| Step chips | n criteria (hover to read them), No criteria and No message (in red: required), Has errors (expand the step to fix the fields that block saving), n mock sets (this step's own sets) or inherits n (the step uses the scenario's sets). |
| User Message | Required. What the user says to the agent in this step. |
| Expected Response | Required. The answer you expect from the agent. |
| Evaluation Criteria | See Evaluation Criteria. |
| Tool Mocks (n) | Mock sets for this step only. Closed by default. A line explains what the step uses, for example "This step uses the scenario's sets." or "No mock sets for this step or the scenario yet." Expand it to see the step's sets. Add Mock Set is shown when the mocks feature is on. A "This step changes:" notice lists the tools this step mocks differently from the rest of the scenario. |
| What will each tool do in this step? | A link in the step's Tool Mocks section that opens the tools panel on that step. See Agent Tools. |
| Remove step (trash icon) | Removes the step after you confirm; its message, expected response, criteria and mock sets are lost. Disabled when only one step is left, because at least one step is required. |
| Collapse step / Expand step | The arrow at the right of the step card. |
| Add Message | Adds a new step at the end. |
Evaluation Criteria
Each step has an Evaluation Criteria (n) box (Evaluation Criterion (1) when it has exactly one). The run checks the agent's actual answer against these criteria, one by one.
| Suggest Criteria / Rewrite Criteria | Lets AI write the criteria for this step from the Expected Response. Disabled until the Expected Response is filled in. Shows Rewrite once the step already has criteria. |
| Add Criterion | Adds an empty criterion you can type yourself. |
| Criterion field | One thing the answer must contain or do. The refresh button inside the field asks AI to reword just that criterion. |
| Delete Criterion (trash icon) | Removes the criterion. |
Advanced Setup
Optional setup for the whole scenario: Date and Time, Tool Mocks and Agent Tools, in that order. The scenario runs without it. Its layout follows the width of the form: a pane on a wide screen, a card below the steps on a narrower one.
| Setup strip | On a wide screen the setup starts collapsed, as a narrow strip at the left of the form. The Setup button (tooltip Show Advanced Setup) opens it. Below it, one button per section (Date and Time, Tool Mocks, Agent Tools) opens the pane on that section. Hover a button to read the section's one-line summary. Tool Mocks and Agent Tools carry a count. |
| Setup pane | Titled Advanced Setup, with the line "Optional. The scenario runs without it." It stays in view while you scroll the form. Hide Advanced Setup (the double-arrow button) collapses it back to the strip. Each section can be folded to a one-line summary. |
| Advanced Setup card | On a narrower screen the setup is a card below the steps, closed by default, headed Advanced Setup · optional with a one-line summary of the date and time, the mock sets and the tool outcomes. Open it for one row per section: Date and Time expands in place, Tool Mocks opens in a side sheet (a bottom sheet on a phone) and Agent Tools opens the tools panel directly. |
| Date and Time | Not shown when the mocks feature is off. |
Agent Tools
Shows what the agent's tools will do when this scenario runs, as a count per outcome. It is a section of Advanced Setup and appears in full only after the agent's tool list and tool policy have loaded.
| Outcome bar | A coloured bar with one chip per outcome that applies: Mocked n, Partly mocked n, Live n, Simulated n and Blocked n. Hover a chip to read "Show only mocked tools" (or the outcome it names); clicking it opens the tools panel on just that outcome. |
| Summary line | When the section is folded, a line such as "Nothing blocked · 2 mocked · 1 live · 3 simulated", or "n will be blocked" in red when some tools would fail ("n blocked" on a narrower screen). |
| Review Tools | Opens the tools panel, What the agent’s tools will do. |
| Loading and error lines | "Loading tool catalog...", "Failed to load the tool catalog. The rest of the form is still usable." (with Retry), "Checking tool policy…", "The agent's tool policy couldn't be loaded, so what each tool will do can't be shown. Runs still follow the saved policy." (with Retry) and "This agent has no tools with mockable policies." |
| Allow Live settings line | "1 Allow Live setting is for a tool no longer on this agent." or "N Allow Live settings are for tools no longer on this agent." |
What the agent’s tools will do (tools panel)
A panel that opens from the right on a wide screen, or from the bottom on a phone. Its subtitle names the agent and the version the scenario runs against, for example "Libraría · runs against the published version".
| Edit Tool Policy → | Opens the Tool Policy page in a new tab. |
| Whole Scenario / Step n | Tabs that choose the scope. On a step tab, the step's own sets are checked first, and a note says so: "A step’s own sets are checked before the scenario’s; “Mock It” here adds a step mock set." |
| Coverage line | For example "4 action tools · 1 mocked · 2 will be blocked". Shown once the tool policy is known. |
| Outcome chips | The same chips as the outcome bar. Click one to show only that outcome; click it again to show all. |
| All / Needs attention · n | Switches between every tool and the tools that need attention (a call can still be blocked by the tool policy, or the tool is only partly mocked). |
| Search tools… | Filters the list by tool name or description. "No tools match your search." or "No tools in this view." when nothing is left. |
| Tool group headings | Tools are grouped by source: Custom API, Plugins, MCP, Other agents, A2A and Built-in, each with the category, instance or server name after a dot when there is one. |
| Tool row | The tool's icon, name, a Draft Only or Published Only chip when it exists in just one version, its description, its outcome chip and, when needed, a caption and a warning. |
| Outcome chip | One word: Mocked, Partly mocked, Live, Simulated, Blocked, or Disabled for a tool switched off in both versions (a read-only row with no actions). A small icon on the chip shows where the outcome comes from: a mock set, the Tool Policy, the tool's built-in default or this scenario's Allow Live. Hover it to read the sentence, for example "Blocked by its default. Its tool policy setting has never been changed." |
| otherwise chip | On a Partly mocked row, after the word "otherwise": what happens to the calls no mock case answers (Blocked, Simulated or Live). |
| n steps override | On a Mocked row, "1 step overrides" or "n steps override": steps whose own mock sets take the tool over. Hover it for the step numbers. |
| Caption and warning | A caption such as "Allowed live" or "Allowed live · takes effect after you save", or on a step tab "This step’s set answers calls no case matches". A warning, "Unmatched calls will reach the real system.", on a partly mocked row whose other calls run live. |
| Mock It | On a tool whose unmatched calls would be blocked. Opens a mock set editor seeded with that tool, so its calls get a scripted answer. Only shown when the mocks feature is on. |
| Allow Live… | Lets this scenario run a blocked or simulated tool for real, after you confirm ("Real data will change.", and for some tools a note that supervisors will be notified or that the other agent really runs). The change takes effect after you save the scenario. Disabled, with the tooltip "Only organisation Owners and Admins can allow a tool to run live", for everyone else; a lock note repeats it at the bottom. |
| Revoke Live | Takes back this scenario's Allow Live for the tool. Hover it to read what the unmatched calls go back to (blocked, simulated or live). |
| Check Order / Hide Check Order | On a tool that attached mock sets mock. Opens Checked in this order: every case and every Otherwise in the order they are checked, grouped by mock set, then a last line saying what answers the calls no case matches (an Otherwise, or the tool policy). A crossed-out case is never reached, with the reason. |
| Try a Call / Hide Try a Call | On a tool that an attached set mocks with something that can answer. Opens one field per input, with a Fields / JSON switch to type the arguments yourself, and a Test button. The result says which case or Otherwise would answer, and why the cases checked before it did not match. On the whole-scenario tab it reads as a step with no mock sets of its own. |
| Mismatch banner | "What’s shown here may differ from runs: the CMS and the simulation service are on different mock-matching versions right now." (and "Try a call may differ from runs." inside Try a Call). Shown only while the two services are out of step. |
| Allow Live for tools no longer on this agent | A list of leftover Allow Live settings for tools the agent no longer has, each with Remove. |
| Footnote | Says which tools exist only in the draft or only in the published agent, depending on the version you run against. |
Tool Mocks
A section of Advanced Setup, in both the create and edit form. Mock sets give your agent's tools scripted answers during a simulation, so a run does not depend on (or change) real systems. Shared mock sets are built on the Tool Mocks screen (Simulation → Tool Mocks), described in Tool Mocks. Each step also has its own Tool Mocks (n) section, closed by default.
When several attached sets mock the same tool, every set's cases are checked first, in the order of the list (a step's own sets, then the scenario's sets, top to bottom): the first case that matches answers. Only when no case matches does the first set whose Otherwise responds answer, and if none does, the tool policy decides.
| Mock sets attached (n) | The mock sets used by every step of this scenario. A step's own mock sets are checked before these. "No mock sets attached yet." when there are none. |
| Drag handle and number | Shown only when two sets in the list mock the same tool. Drag a row's handle ("Drag to change the order") to change the order; the number shows the order the sets are checked in. |
| Mock set row | Shows the name, a count such as "2 tools · 5 cases", the tool names, a library (shared) or lock (private) icon, and Unsaved changes when a private set has edits not saved yet. Hover the icon to read "Library: shared, can be reused in other scenarios" or "Private: only this scenario uses it". A crossed-out tool name means this set can never answer that tool in this list: hover it to read why, for example that a set above answers every call first. A tool marked "not found in the draft or published agent" no longer exists on the agent. |
| Overlap notice | "n sets mock <tool>." Open its details: "When a call matches cases in more than one, '<set>' answers because it’s higher in the list. Calls no case matches get the Otherwise of the highest set whose Otherwise answers. Drag to change the order." |
| Add Mock Set | Opens a menu: From Library… ("Reuse a shared set.") opens the Add from library panel, where you search, tick sets and click Add n mock sets; Create New ("Private to this scenario.") opens the editor. Only shown when the mocks feature is on. |
| Edit | On a private set, a pencil that opens the set in the editor. On a library set, an icon that opens the set in Tool Mocks in a new tab, because changing a shared set changes every scenario that uses it. Only shown when the mocks feature is on. |
| ⋮ menu on a row | Make a Private Copy (library sets), Save to Library (private sets; disabled with the tooltip "Save this scenario first — this set has unsaved changes" until the scenario is saved), Remove. |
| Mismatch banner | The same banner as in the tools panel, when the two services are on different mock-matching versions. |
Date and Time
A section of Advanced Setup. It is shown only when the mocks feature is on.
| Fixed date & time | A switch. Off, the line under it reads "Uses real time" and the agent sees the real date and time. On, you choose the date and time the agent sees. |
| Date & time | Appears when the switch is on. The date and time the agent sees on every run. |
| Time zone | Appears when the switch is on. The time zone for that date and time. |
| Summary | When folded, Real time or "Fixed · <date and time>". |
The section adds the note "Keeps “tomorrow” and other date-based tool arguments stable on every run."
Saving and Running
| Suggest All Criteria (x/y Steps ready) | Shown instead of Save while some steps still have no criteria. Lets AI write criteria for every step that has none, with progress "Processing n of m steps...". All steps need a User Message and Expected Response first. |
| Save | Shown once every step has at least one criterion. In create mode it opens the Scenario Information dialog; in edit mode it asks "Are you sure you want to save your changes?" and saves after you confirm. |
| Cancel | Leaves the form without saving and returns to the list. |
| Run | Edit mode only (top right). Runs the saved scenario after you confirm, as described in Run Confirmation. If the form has unsaved Allow Live changes, it asks you to save first, because a run uses the saved scenario. |
| Delete | Edit mode only (trash icon, top right). Deletes this scenario after you confirm. |
Related guide: Write Good Scenarios, Run Simulations, Tools in Simulations