docs: Experimentation v2 — draft for team review#7316
Draft
talissoncosta wants to merge 9 commits intomainfrom
Draft
docs: Experimentation v2 — draft for team review#7316talissoncosta wants to merge 9 commits intomainfrom
talissoncosta wants to merge 9 commits intomainfrom
Conversation
Replaces the existing "bring your own analytics" A/B testing guide with a v2 proposal covering three flows: data warehouse connection, create experiment wizard, and reading results. All images are placeholders — screenshots will land before merge. Opening as a draft PR to align on flow, terminology, and scope before the backend work starts. Complements the prototype at #7195 where the same flows are live to click through. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
…URLs Docusaurus' MDX compiler rejects markdown images whose paths don't resolve to an existing file, failing the Link Check Docs job. Screenshots haven't been captured yet, so swap every placeholder to a text callout with the target filename preserved. Flips back to real images once screenshots land. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Warehouse setup is conceptually a third-party integration, not an Experimentation concept — sits naturally alongside Amplitude, Segment, and the other analytics destinations. Experimentation page keeps a brief pointer and links across. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Flips the top-5 placeholders to real images: experiments list, wizard step 1 (Details), wizard step 6 (Review & Launch), full results dashboard, and the warehouse connected state. Remaining 9 placeholders stay as text callouts pending capture. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The two-button Test + Connect pattern is deferred to a later iteration per product planning. V1 ships with a single Connect action that validates on save, which removes one step, one screenshot, and the "stale test result" warning from the documented flow. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
talissoncosta
added a commit
that referenced
this pull request
Apr 23, 2026
Per product planning, the two-button Test + Connect pattern is deferred to a later iteration. v1 ships with a single Connect action that validates credentials on save. Removes the testStatus state, runTest helper, inline test-result banner, the stale-test-result clearing logic, and the associated SCSS rules + keyframes. Keeps the prototype aligned with the Experimentation docs draft (#7316). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Flips the remaining wizard, results, and warehouse config-form placeholders to real image references. Two placeholders remain for screenshots still to be captured: the hero image on the Experimentation page and the Organisation Integrations list shot. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Move warehouse setup from a numbered section to a Prerequisites callout, fix the 5-step wizard claim that previously enumerated 6 steps, and adopt the Overview / no-Step-N / inline-Note structure from the Backstage integration page for a consistent voice across the v2 docs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Add a lead-in paragraph stating the audience model (segment = audience, all matched users in experiment, weights decide variation), clarify that control takes a weight slot, and call out v1 limitations (no sample-size dial, one experiment per segment+flag, no mid-run weight changes). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Match the prototype's new audience model: targeting (optional segment filter), sample size (% of eligible users in the experiment), variation split (per-arm weights). Replaces the segment-as-audience framing where the segment was the audience and weights were the only knob. Updates the overview bullet and the segments-concept blurb to position segments as an optional eligibility filter, not the audience itself. Updates the launch confirmation copy. Renames the screenshot reference to wizard-audience.png to match. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Closes #7344. Part of the #7275 epic.
Contributes to the Experimentation v2 effort.
Draft for team alignment — opened early so we can discuss flow, terminology, and scope before the backend work starts.
Preview links
Replaces the existing "bring your own analytics" A/B testing guide (
docs/docs/experimentation-ab-testing.md) with a v2 proposal based on the prototype in #7195. Three flows are covered:Known open questions for the team
experimentation-ab-testing.mdwhich already uses bucketing terminology. Worth confirming we're happy to commit to this vocabulary in public docs.What's not in scope
How did you test this code?
cd docs && npm install && make serveto verify the page renders, or open the Vercel preview.TODOin the markdown. They reference/img/experimentation/*.png. Screenshots will be captured from the prototype and added before merge.Image capture plan
Screenshots to capture (most critical first):
hero-results-dashboard.pngwarehouse-connected.pngexperiments-list.pngwizard-details.png+wizard-review-launch.pngresults-comparison-table.pngRemaining 8 images can land before merge — full list in the draft markdown.
What kind of feedback is most useful