Experiment node
The Experiment node splits users across multiple paths and (optionally) picks a winner once you have enough data. Use it whenever you want to test two or more variants of a downstream flow - different subject lines, send times, message content, or even branching logic.
How it works at a glance
Each user that reaches the Experiment node is routed into exactly one path. The path determines what happens next on the canvas - every path has its own outgoing edge that you wire to whatever node should run for that variant.
The Experiment node owns three concepts:
- Path type - the routing strategy (Standard / Pick winner / Personalized).
- Path distribution - how traffic gets allocated across paths.
- Special path kinds - regular paths, control groups (no treatment), delay groups (hold for the winner).
Path types
Open the Experiment node config and pick one of three modes at the top of the Experiment tab.
Standard
A plain A/B/n split. Users get distributed across your paths by the percentages you set, indefinitely. No winner is ever picked - both variants run forever. Use this when you want to keep multiple variants live in parallel, or when you'll be the one comparing results.
Pick winner
Same starting allocation as Standard, but the system automatically declares a winner once it has enough evidence (or hits the test-window time limit). After that, 100% of new users are routed to the winning path. Use this for content tests where you want the system to decide and lock in.
When picking the winner, the system uses Bayesian P(best) - the probability that a given path is the true best, computed from the data so far. This is calibrated under continuous monitoring (the scheduler checks every 5 minutes without distorting the math). Once a path crosses your confidence threshold (95% by default), it wins.
Personalized
Each user is routed by an ML model that predicts their best-fit path based on their attributes and event history. No single winner is declared - different users get different paths, forever. Use this when "the best path" depends on who the user is.
A small holdout (20% by default) keeps getting the random A/B split so the system can measure how much lift the ML model actually delivers.
The ML model retrains daily at 03:30 UTC across all active workspaces. Until a model exists for your workspace, users fall back to a random split (the predictions silently fail and the worker uses hash-based bucketing).
Path distribution
The big block of cards in the middle of the modal. Each card is one path:
- Type a name (click the pencil to rename inline).
- Set its percentage. The bar at the top shows total; it must equal 100 before you can save.
- Click the trash to remove. Treatments need at least 2 to remain; control and delay can always be deleted.
Distribute evenly spreads the remaining traffic across treatment paths only - control and delay keep their carve-outs.
Control group
A control is a path that exits the experiment immediately with no treatment. Users routed here continue down the canvas as if the experiment didn't exist. Used to measure lift: "did the treatments actually move the needle compared to doing nothing?"
- Add via + Add control.
- Only one control per experiment.
- Excluded from winner picking (control is the baseline, not something to optimize for).
Delay group
Available in Pick winner and Personalized modes. Users routed here are parked until the winner is declared or the model is trained, then routed forward.
- Add via + Add delay (optional - only required if you want late-arrival safety).
- For Pick winner: parked users get routed to the winning path once it's declared.
- For Personalized: parked users get routed via the ML model once the test window closes.
- If the experiment ends without declaring a winner (Statistical-Significance fallback set to "mix"), parked users are released as completed and exit the canvas.
How big should delay be? Tradeoff:
- Low (0–10%) - most users get treatments immediately, faster learning.
- High (20–50%) - more users wait safely for the winner.
- Default suggestion: 0% (skip delay entirely) unless you have a specific reason to park users.
Pick-winner settings
Visible when path type is Pick winner (also shown for Personalized so you can tune the model's stop conditions).
| Setting | What it does |
|---|---|
| Winner is decided by | The metric the system optimizes: Open rate / Click rate / Conversion event / Custom event. |
| Test window | Maximum time to run before declaring. Counts from the first treatment user's entry. |
| Custom event | Required when winner metric is "Custom event". Pick from your workspace's tracked events. |
| Score by | For custom events: count of events (default) or sum of a numeric property (e.g. revenue). |
| Statistical significance fallback | At the time-window cap, if results aren't significant: Send best-performing path (declare best so far) or Continue mix (keep splitting, no winner). |
| Experiment mode | Standard (lock the winner once declared) or Bandit (continuously rebalance toward the leader). |
Bandit mode
A "real bandit" implementation using Thompson sampling with a sliding window.
- Every user, fresh sample - there's no winner-declared state. Allocation continuously shifts toward whichever path is currently winning.
- A learning window (default 7 days) ages out old data so the bandit adapts to drift. A "winter winner" stops dominating in summer as winter data falls outside the window.
- Tweak the window via Bandit learning window (days) in Additional settings.
Use Bandit when:
- Performance might shift over time (seasonal content, time-of-day-sensitive offers)
- You want continuous optimization rather than a frozen winner
Additional settings tab
| Setting | Default | Notes |
|---|---|---|
| Confidence level | 95% | The threshold P(best) must cross to declare a winner. Higher = more rigorous, slower to decide. |
| Minimum sample per path | 100 | Don't declare a winner until each treatment has at least this many users. Prevents premature wins on tiny samples. |
| Conversion window (hours) | 72 | For "Conversion event" / "Custom event" metrics: how long after entry to count conversions. |
| Bandit learning window (days) | 7 | Bandit-mode only. Only events in the last N days count toward Thompson sampling. |
Results tab
Once the Experiment node is saved inside a running journey, its config gains a Results tab. Results appear only after contacts start flowing through the node.
- Per-path stats - for each path: contacts entered, plus open / click / conversion rates.
- Leader / winner badge - highlights the currently leading path (and the declared winner once one is picked).
- Statistical-significance readout - shows the p-value so you can tell whether the difference between paths is real yet.
- Declare winner - a per-path action to manually lock in that path as the winner.
- Reset experiment - clears accumulated results and starts the test over.
Re-randomize on re-entry
A standalone checkbox visible in Standard mode only. By default, the same user gets the same path every time they re-enter the canvas (deterministic hash). Toggle this on to re-roll on each entry. Rarely needed; useful for tests where seeing different variants is part of the journey.
Analytics
Click the experiment node on an active canvas to open the analytics drawer. It shows:
- Status badge - "Continuously optimizing" for bandit, "Personalizing per user" for personalized, "Winner Declared" / "Collecting Data" / "Stopped" for standard.
- Distribution bar - proportional view of current allocation across paths.
- Stats table - per-path: entered, delivered, opened, clicked, current rate, P(best) column showing live Bayesian probability each path is the true best.
- Winner banner - when declared: which path, when, and how (statistical significance vs time-limit).
- Waiting in delay banner - operational signal showing how many users are parked.
- Recent release banner - appears for up to 24 hours after delay-users get released to the winner.
Cross-experiment dashboard
In Analytics → Dashboards, the new "Experiment" data source adds an experiment grid widget showing every experiment in your workspace at a glance: status, leader, P(best), winner, delay queue. Filter by status / path type / canvas, search by name. Refreshes every 30 seconds.
Common patterns
Subject-line A/B test (Standard). Two paths, 50/50 split, route both to email send nodes with different subjects. No winner picking - you'll evaluate results manually after a week.
Onboarding flow pick-winner. Three paths with different onboarding sequences, Pick winner mode, winner metric = "completed onboarding" custom event, test window 7 days. Add a 20% delay group to safely catch late-arriving users for the winner. Statistical significance fallback = "best" so you always get a decision.
Send-time bandit. Two paths with different send-time delays before the email. Bandit mode, winner metric = open rate, learning window 7 days. Allocation continuously shifts toward whichever time is performing best lately.
Personalized recommendations. Three paths with different product-category recommendations. Personalized mode, holdout 20%. The ML model learns per-user preferences over time.
Troubleshooting
The winner never gets declared. Check that each treatment has hit minimum sample size (default 100). Check confidence level isn't set too high (99% takes much longer than 95%). For tests with small effect sizes, you may genuinely need more traffic.
My personalized experiment is just doing random splits. The ML model may not have trained yet. It retrains daily at 03:30 UTC across active workspaces. You can also trigger a manual training run from the ML service if needed. Check that your workspace has enough historical event data - features need real signal to learn from.
Bandit is stuck on one path that's degrading. The default learning window is 7 days - older performance dominates. Try lowering the window (e.g. 3 days) so the bandit adapts faster.
The delay group has users that never get released. Confirm a winner can actually be declared: minimum sample met, test window not yet expired, statistical significance fallback set to "best" (not "mix"). If "mix" is selected and the test ends without significance, delay users exit as completed.
Total ≠ 100%. The Done button is disabled until you fix it. Use "Distribute evenly" for a quick reset.