From c0f310a574fdc63791f8afa77a8d5c237f422148 Mon Sep 17 00:00:00 2001 From: Stephen Smith Date: Thu, 27 Aug 2026 09:17:59 -0700 Subject: [PATCH 1/4] Expanded on how to configure an AI tester --- fern/docs.yml | 3 + fern/observability/simulations-advanced.mdx | 24 +-- .../simulations-configure-ai-tester.mdx | 172 ++++++++++++++++++ fern/observability/simulations-manage.mdx | 5 +- fern/observability/simulations-overview.mdx | 3 + fern/observability/simulations-quickstart.mdx | 7 +- 6 files changed, 193 insertions(+), 21 deletions(-) create mode 100644 fern/observability/simulations-configure-ai-tester.mdx diff --git a/fern/docs.yml b/fern/docs.yml index d1d280101..0c27f77d1 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -582,6 +582,9 @@ navigation: - page: Advanced path: observability/simulations-advanced.mdx icon: fa-light fa-flask-vial + - page: Configure an AI tester + path: observability/simulations-configure-ai-tester.mdx + icon: fa-light fa-robot - page: Manage path: observability/simulations-manage.mdx icon: fa-light fa-sliders diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index 4015d7ef8..cf6980642 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -1,21 +1,21 @@ --- title: Simulations advanced subtitle: Mock tools, send lifecycle webhooks, and reuse structured outputs in simulations. -description: "Configure advanced simulations with AI tester settings, variables, tool mocks, lifecycle webhooks, and reusable structured outputs for consistent testing." +description: "Configure advanced simulations with variables, tool mocks, lifecycle webhooks, and reusable structured outputs for consistent testing." slug: observability/simulations-advanced --- -Advanced simulation options let you configure the AI tester, set variable values, mock tool responses, trigger lifecycle webhooks, and reuse structured outputs. Use them after you complete the [**Simulations quickstart**](/observability/simulations-quickstart) and need more control over behavior or test conditions. +Advanced simulation options let you set variable values, mock tool responses, trigger lifecycle webhooks, and reuse structured outputs. Use them after you complete the [**Simulations quickstart**](/observability/simulations-quickstart) and need more control over behavior or test conditions. ## How it works -Advanced options belong to each simulation in a suite. They control how the AI tester behaves, which values the [**assistant**](/assistants) or [**squad**](/squads) receives, what mocked tools return, and which lifecycle events send webhooks. +Advanced options belong to each simulation in a suite. They control which values the [**assistant**](/assistants) or [**squad**](/squads) receives, what mocked tools return, and which lifecycle events send webhooks. Open a suite and select **Edit**, then **Next**. The review step contains the **Success criteria**, **Variables**, and **Tool mocks & webhooks** tabs. - - Change the AI tester's model, transcriber, and voice. + + Define the AI tester's scenario, behavior, model, transcriber, and voice. Supply dynamic-variable values without editing the assistant or squad. @@ -34,20 +34,6 @@ Open a suite and select **Edit**, then **Next**. The review step contains the ** -## Configure the AI tester - -The **AI tester** drives the simulated conversation, and its configuration lives in the personality. On a simulation's **Personality** tab, expand **Advanced settings** to change how the AI tester reasons, listens, and speaks: - -- **Model**: Choose the provider and model that control how the AI tester reasons. Use a provider and model you can access. -- **Transcriber** (optional): Override the AI tester's speech-to-text provider, language, and model. If you leave it unset, the AI tester falls back to a built-in default. -- **Voice**: Choose the provider and voice for the AI tester. Select a configured voice or enter a custom voice ID. - -Transcriber and voice apply to voice simulations. Chat simulations use the model only. - - - Editing a built-in **(Default)** personality saves it as a new personality you own; the built-in default is left unchanged. - - ## Mock tool responses During a simulation, the [**assistant**](/assistants) or [**squad**](/squads) under test runs its real [**tools**](/tools). Mock a tool to return fixed text instead, so that tool never calls its live service or API. A mock intercepts only the tool it names; every other tool the assistant or squad calls still runs for real. diff --git a/fern/observability/simulations-configure-ai-tester.mdx b/fern/observability/simulations-configure-ai-tester.mdx new file mode 100644 index 000000000..4bf201f92 --- /dev/null +++ b/fern/observability/simulations-configure-ai-tester.mdx @@ -0,0 +1,172 @@ +--- +title: Configure an AI tester +subtitle: Define the AI tester's goal and control how it behaves, reasons, listens, and speaks. +description: "Configure what a simulation AI tester should accomplish and how it behaves, including its model, transcriber, voice, turn-taking, and fallbacks." +slug: observability/simulations-configure-ai-tester +--- + +Configure the scenario and personality together to control an AI tester. The scenario defines what the tester should accomplish, while the personality defines how it behaves and communicates. + +## Scenario and personality roles + +Each simulation pairs one scenario with one personality. Keep their responsibilities separate so you can reuse the same customer behavior across different test cases: + +| Configuration | What it controls | Example | +| -- | -- | -- | +| **Scenario** | The AI tester's goal, relevant context, actions, and stopping condition | Dispute an unexpected charge, provide an account ID when asked, and end after receiving a resolution | +| **Personality** | The AI tester's behavior, tone, conversation timing, model, transcriber, and voice | Act impatient, interrupt long answers, and ask for concise explanations | + +Write the scenario as instructions for the AI tester, not as the expected outcome. Define pass or fail conditions separately under **Success criteria**. + +## Dashboard and API settings + +The available settings depend on whether you configure the AI tester in the Dashboard or through the API: + +| Area | Dashboard settings | Additional API settings | +| -- | -- | -- | +| **Scenario** | Name and intent | Target-assistant overrides, webhook destinations, transcript and message inclusion, and recording inclusion | +| **Behavior** | Behavior and who starts first | First message, start-speaking plan, stop-speaking plan, duration limit, background sound, and background-speech denoising | +| **Model** | Provider and model | Temperature, maximum tokens, fallback models, tools, prompt caching, knowledge base, and model-specific reasoning | +| **Transcriber** | Provider, model, and language | Language detection and hints, endpoint delay, vocabulary, contextual hints, and fallback transcribers | +| **Voice** | Provider and voice ID | Speed, version, language, pronunciation dictionaries, chunking, formatting, and caching | + +Transcriber and voice settings apply to voice simulations. Chat simulations use the model only. + +## Configure the AI tester + + + + + + + Open **Simulations**, select **Suites**, and create or edit a suite. Select the simulation you want to configure. + + + + On the **Scenario** tab, enter a scenario name and intent. Include the AI tester's goal, information it can provide, actions it should take, and when it should end the conversation. + + Keep behavioral traits such as tone, patience, and interruption style out of the scenario. Configure those traits in the personality so you can reuse them with other scenarios. + + + + Open the **Personality** tab. Enter the instructions that define how the AI tester should act. Under **Who starts first?**, select **AI tester** or **Assistant or squad**. + + + + Expand **Advanced settings**. Under **Model**, select the provider and model that control how the AI tester reasons. + + + + For a voice simulation, configure these optional settings: + + - Under **Transcriber**, select the speech-to-text provider, language, and model. + - Under **Voice**, select the provider and voice, or enter a custom voice ID. + + + + Select **Next**, configure the success criteria and other advanced simulation options, then save or run the suite. + + + + + + +Update the scenario's `instructions` to define what the AI tester should accomplish: + +```bash +curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Dispute an unexpected charge", + "instructions": "You are calling about an unexpected $40 charge. Provide account ID ACCT-1234 when asked. Ask the assistant to explain the charge and remove it if it is invalid. End the conversation after the assistant provides a resolution or next step." + }' +``` + +The `instructions` field sets the AI tester's goal and test conditions. Configure evaluations, target-assistant overrides, tool mocks, and lifecycle hooks on the same scenario. See [**Update Scenario**](/api-reference/simulation-scenarios/scenario-controller-update) for the complete schema. + +A personality's `assistant` field defines how the AI tester behaves. First, retrieve the existing personality so you can preserve its current assistant fields: + +```bash +curl -X GET "https://api.vapi.ai/eval/simulation/personality/" \ + -H "Authorization: Bearer $VAPI_API_KEY" +``` + +Then update the personality with `PATCH`. The following request shows representative settings for the model, Soniox transcriber, Vapi Voice, and conversation behavior: + +```bash +curl -X PATCH "https://api.vapi.ai/eval/simulation/personality/" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "assistant": { + "model": { + "provider": "openai", + "model": "gpt-4.1", + "messages": [ + { + "role": "system", + "content": "Act as an impatient customer who wants a concise answer." + } + ], + "temperature": 0.2, + "maxTokens": 500, + "fallbackModels": ["gpt-4.1-mini"] + }, + "transcriber": { + "provider": "soniox", + "model": "stt-rt-v5", + "languages": ["en", "es"], + "languageHintsStrict": false, + "maxEndpointDelayMs": 800, + "customVocabulary": ["Vapi", "Acme"] + }, + "voice": { + "provider": "vapi", + "voiceId": "Clara", + "version": "2", + "speed": 1.05, + "language": "en-US", + "cachingEnabled": true + }, + "firstMessage": "I need help with an unexpected charge.", + "firstMessageMode": "assistant-speaks-first", + "startSpeakingPlan": { + "waitSeconds": 0.6 + }, + "stopSpeakingPlan": { + "numWords": 2, + "voiceSeconds": 0.2, + "backoffSeconds": 1 + }, + "maxDurationSeconds": 600, + "backgroundSound": "office" + } + }' +``` + +Within a personality's `assistant` configuration, `assistant-speaks-first` means the AI tester starts the conversation. Use `assistant-waits-for-user` when the assistant or squad under test should start. + +The available fields vary by provider. See [**Update Personality**](/api-reference/simulation-personalities/personality-controller-update) for the complete schema. + + + The `assistant` field is a full nested configuration. Include every existing assistant field you want to preserve when you update a personality. + + + + + + + Editing a built-in **(Default)** personality in the Dashboard saves it as a new personality you own. The built-in personality remains unchanged. + + +## Next steps + + + + Configure variables, tool mocks, webhooks, and reusable structured outputs. + + + Edit suites, review and rerun results, and maintain test coverage. + + diff --git a/fern/observability/simulations-manage.mdx b/fern/observability/simulations-manage.mdx index 3805b78de..c72a84300 100644 --- a/fern/observability/simulations-manage.mdx +++ b/fern/observability/simulations-manage.mdx @@ -201,6 +201,9 @@ Personalities, scenarios, and simulations have their own `DELETE` endpoints: `/e Create and run your first simulation suite. - Configure AI testers, variables, tool mocks, webhooks, and structured outputs. + Configure variables, tool mocks, webhooks, and structured outputs. + + + Define the AI tester's scenario, behavior, model, transcriber, and voice. diff --git a/fern/observability/simulations-overview.mdx b/fern/observability/simulations-overview.mdx index f424a7383..7758459fa 100644 --- a/fern/observability/simulations-overview.mdx +++ b/fern/observability/simulations-overview.mdx @@ -91,6 +91,9 @@ Choose Simulations when you want to see how your assistant or squad performs aga Run your first simulation against an assistant or squad. + + Define the AI tester's scenario, behavior, model, transcriber, and voice. + Test assistant or squad logic with scripted mock conversations. diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index c7c3b8dd8..99f9e3879 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -65,6 +65,8 @@ You create Riley and the structured output during this quickstart. Open the **Personality** tab and select a built-in personality marked **(Default)**. Keep its behavior and advanced settings unchanged for this first simulation. + To customize what the AI tester does and how it behaves, see [**Configure an AI tester**](/observability/simulations-configure-ai-tester). + - Configure AI testers, variables, tool mocks, webhooks, and structured outputs. + Configure variables, tool mocks, webhooks, and structured outputs. + + + Define the AI tester's scenario, behavior, model, transcriber, and voice. What Simulations are and when to use them over Evals. From 37d7d02b9af40398fdcb1ecccab6eec5a1ff962f Mon Sep 17 00:00:00 2001 From: Stephen Smith Date: Thu, 27 Aug 2026 12:35:17 -0700 Subject: [PATCH 2/4] docs: address AI tester review feedback --- .../simulations-configure-ai-tester.mdx | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/fern/observability/simulations-configure-ai-tester.mdx b/fern/observability/simulations-configure-ai-tester.mdx index 4bf201f92..f3f825d36 100644 --- a/fern/observability/simulations-configure-ai-tester.mdx +++ b/fern/observability/simulations-configure-ai-tester.mdx @@ -30,7 +30,7 @@ The available settings depend on whether you configure the AI tester in the Dash | **Transcriber** | Provider, model, and language | Language detection and hints, endpoint delay, vocabulary, contextual hints, and fallback transcribers | | **Voice** | Provider and voice ID | Speed, version, language, pronunciation dictionaries, chunking, formatting, and caching | -Transcriber and voice settings apply to voice simulations. Chat simulations use the model only. +Chat simulations ignore transcriber and voice settings. Behavior, model, and turn-taking settings still apply. ## Configure the AI tester @@ -85,14 +85,29 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/" \ The `instructions` field sets the AI tester's goal and test conditions. Configure evaluations, target-assistant overrides, tool mocks, and lifecycle hooks on the same scenario. See [**Update Scenario**](/api-reference/simulation-scenarios/scenario-controller-update) for the complete schema. -A personality's `assistant` field defines how the AI tester behaves. First, retrieve the existing personality so you can preserve its current assistant fields: +A personality's `assistant` field defines how the AI tester behaves. First, retrieve the existing personality so you can preserve its current assistant fields and check whether it is built in: ```bash -curl -X GET "https://api.vapi.ai/eval/simulation/personality/" \ - -H "Authorization: Bearer $VAPI_API_KEY" +curl -sS -X GET "https://api.vapi.ai/eval/simulation/personality/" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -o personality.json + +jq '{ id, orgId, name }' personality.json +``` + +If `orgId` is `null`, the personality is built in and cannot be updated with `PATCH`. Create an organization-owned copy, then use the returned `id` for subsequent updates: + +```bash +jq '{ name: "Custom impatient customer", assistant: .assistant }' \ + personality.json > personality-copy.json + +curl -X POST "https://api.vapi.ai/eval/simulation/personality" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + --data @personality-copy.json ``` -Then update the personality with `PATCH`. The following request shows representative settings for the model, Soniox transcriber, Vapi Voice, and conversation behavior: +If `orgId` contains your organization ID, update the existing personality. The following request shows representative settings for the model, Soniox transcriber, Vapi Voice, and conversation behavior: ```bash curl -X PATCH "https://api.vapi.ai/eval/simulation/personality/" \ @@ -136,7 +151,6 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/personality/" }, "stopSpeakingPlan": { "numWords": 2, - "voiceSeconds": 0.2, "backoffSeconds": 1 }, "maxDurationSeconds": 600, From 443673efcfec51e3f80bc1748ccf20e6b69c1153 Mon Sep 17 00:00:00 2001 From: Stephen Smith Date: Thu, 27 Aug 2026 12:39:18 -0700 Subject: [PATCH 3/4] docs: clarify AI tester interruption settings --- fern/observability/simulations-configure-ai-tester.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fern/observability/simulations-configure-ai-tester.mdx b/fern/observability/simulations-configure-ai-tester.mdx index f3f825d36..bb382c8db 100644 --- a/fern/observability/simulations-configure-ai-tester.mdx +++ b/fern/observability/simulations-configure-ai-tester.mdx @@ -161,6 +161,8 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/personality/" Within a personality's `assistant` configuration, `assistant-speaks-first` means the AI tester starts the conversation. Use `assistant-waits-for-user` when the assistant or squad under test should start. +In `stopSpeakingPlan`, `voiceSeconds` only applies when `numWords` is `0`. When `numWords` is greater than `0`, the AI tester waits for that number of transcribed words instead. See [**Stop speaking plan**](/customization/voice-pipeline-configuration#stop-speaking-plan) for details. + The available fields vary by provider. See [**Update Personality**](/api-reference/simulation-personalities/personality-controller-update) for the complete schema. From c74617805440cd455b4b134a3f44065493792473 Mon Sep 17 00:00:00 2001 From: Stephen Smith Date: Thu, 27 Aug 2026 13:21:30 -0700 Subject: [PATCH 4/4] docs: refine AI tester terminology and tab/field bridges - Use 'assistant or squad under test' instead of 'target-assistant' - Map Behavior/Model/Transcriber/Voice settings to the Personality tab - Bridge the Dashboard 'intent' field to the API instructions field - Use 'success criteria' consistently in the cURL section Co-Authored-By: Claude Opus 4.8 --- fern/observability/simulations-configure-ai-tester.mdx | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/fern/observability/simulations-configure-ai-tester.mdx b/fern/observability/simulations-configure-ai-tester.mdx index bb382c8db..93bdbdbbe 100644 --- a/fern/observability/simulations-configure-ai-tester.mdx +++ b/fern/observability/simulations-configure-ai-tester.mdx @@ -24,12 +24,14 @@ The available settings depend on whether you configure the AI tester in the Dash | Area | Dashboard settings | Additional API settings | | -- | -- | -- | -| **Scenario** | Name and intent | Target-assistant overrides, webhook destinations, transcript and message inclusion, and recording inclusion | +| **Scenario** | Name and intent | Overrides for the assistant or squad under test, webhook destinations, transcript and message inclusion, and recording inclusion | | **Behavior** | Behavior and who starts first | First message, start-speaking plan, stop-speaking plan, duration limit, background sound, and background-speech denoising | | **Model** | Provider and model | Temperature, maximum tokens, fallback models, tools, prompt caching, knowledge base, and model-specific reasoning | | **Transcriber** | Provider, model, and language | Language detection and hints, endpoint delay, vocabulary, contextual hints, and fallback transcribers | | **Voice** | Provider and voice ID | Speed, version, language, pronunciation dictionaries, chunking, formatting, and caching | +In the Dashboard, Scenario settings are on the **Scenario** tab, and the Behavior, Model, Transcriber, and Voice settings are on the **Personality** tab. + Chat simulations ignore transcriber and voice settings. Behavior, model, and turn-taking settings still apply. ## Configure the AI tester @@ -43,7 +45,7 @@ Chat simulations ignore transcriber and voice settings. Behavior, model, and tur - On the **Scenario** tab, enter a scenario name and intent. Include the AI tester's goal, information it can provide, actions it should take, and when it should end the conversation. + On the **Scenario** tab, enter a scenario name and intent (the scenario's `instructions` field in the API). Include the AI tester's goal, information it can provide, actions it should take, and when it should end the conversation. Keep behavioral traits such as tone, patience, and interruption style out of the scenario. Configure those traits in the personality so you can reuse them with other scenarios. @@ -83,7 +85,7 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/" \ }' ``` -The `instructions` field sets the AI tester's goal and test conditions. Configure evaluations, target-assistant overrides, tool mocks, and lifecycle hooks on the same scenario. See [**Update Scenario**](/api-reference/simulation-scenarios/scenario-controller-update) for the complete schema. +The `instructions` field sets the AI tester's goal and test conditions. Configure success criteria, overrides for the assistant or squad under test, tool mocks, and lifecycle hooks on the same scenario. See [**Update Scenario**](/api-reference/simulation-scenarios/scenario-controller-update) for the complete schema. A personality's `assistant` field defines how the AI tester behaves. First, retrieve the existing personality so you can preserve its current assistant fields and check whether it is built in: