Skip to main content

Statistical Significance for A/B Tests

When a campaign has 2 or more variants, Joryio automatically calculates whether performance differences are statistically significant - so you know if a variant is actually better, not just randomly ahead.

What You'll See​

Confidence Intervals​

Each rate metric (Open Rate, Click Rate) shows a range below it:

Open Rate: 24.90%
[21.3% - 28.8%]

This means the true rate is 95% likely to fall within that range. Narrower ranges = more reliable data.

Significance Badges​

Next to each rate, you'll see a colored badge:

  • Green +12.5% - This variant is significantly better than the baseline
  • Red -5.2% - This variant is significantly worse than the baseline
  • Gray p=0.234 - Not enough evidence yet (need more data)

A result is "significant" when p < 0.05, meaning there's less than a 5% chance the difference is due to random chance.

Leader & Winner Tags​

  • "Leading" (blue tag) - The variant with the highest rate on the primary metric
  • "Winner" (green tag + trophy icon) - A statistically confirmed winner that has been officially declared

Setting Up Statistical Analysis​

A/B Test Configuration​

When creating or editing a campaign with 2+ variants, you can configure the analysis:

SettingDescriptionDefault
Primary MetricWhich metric determines the winnerThe first metric the channel can actually report - see Metrics by channel
Min Sample SizeMinimum recipients per variant before significance is calculated100
Auto-WinnerAutomatically declare winner when significance is reachedOff
Auto-PromoteShift 100% traffic to winner after declarationOff

Available Primary Metrics​

MetricBest ForWhat It Measures
Opens (open rate)Subject line testsUnique human opens / delivered
Clicks (click rate)Content & CTA testsUnique human clicks / delivered
Conversion (conversion rate)End-to-end impactConversions of the campaign's primary conversion event (Conversion Tracking) / recipients
Revenue (per recipient)Order value / AOVSum of purchase amounts (the one canonical Order Completed event) in the attribution window / recipients
CTRIn-app messagesClicks / impressions

Conversion uses the campaign's primary conversion event from Conversion Tracking (set one there, or the winner can't be scored on conversion). Revenue is a continuous, amount-weighted metric - it has no significance test, so a revenue winner is declared by the highest revenue-per-recipient at the deadline (it doesn't early-stop).

API Configuration Example​

{
  "abTestConfig": {
    "primaryMetric": "openRate",
    "minSampleSizePerVariant": 200,
    "autoWinnerEnabled": true,
    "autoPromoteEnabled": false
  }
}

How the Baseline Works​

Every significance calculation compares a variant against a baseline:

  1. Control group - used as the baseline when one exists, it has recipients, and the primary metric is Conversion or Revenue. A control was sent nothing, so it has no opens or clicks to compare an open or click rate against.
  2. The first authored variant - in every other case. It is never the best performer: the observed leader is partly a leader because it got lucky, so testing the rest against it makes them all look worse than they are and hands the win to whichever arm ran ahead early. A reference arm has to be fixed before the data arrives.

All other variants show their relative difference compared to this baseline.

Auto-Winner Detection​

When autoWinnerEnabled is turned on:

  1. A background job checks every 5 minutes
  2. It verifies all variants have reached the minimum sample size
  3. If a variant has p < 0.05 on the primary metric → winner is declared
  4. If autoPromoteEnabled is also on → traffic is shifted 100% to the winner

What Happens After a Winner​

  • A green banner appears at the top of the variant comparison table
  • The winning variant gets a trophy icon and "Winner" tag
  • Original traffic weights are saved (for rollback if needed)
  • If auto-promote is on, all new recipients get the winning variant

When No Winner is Found​

If variants perform similarly and no significant difference emerges:

  • The "Leading" tag shows which variant is currently ahead
  • The gray p=... badges show how close each comparison is to significance
  • Consider running the test longer or increasing sample size

Understanding the Numbers​

Confidence Interval (CI)​

What it means: "We're 95% confident the true rate falls within this range."

Example:
Variant A: Open Rate 20.0% [16.7% - 23.7%]
Variant B: Open Rate 24.9% [21.3% - 28.8%]

The CIs overlap here, suggesting the difference may not be significant. When CIs don't overlap, the difference is almost certainly significant.

P-Value​

What it means: "The probability of seeing this big a difference (or bigger) by pure chance."

  • p = 0.03 → Only 3% chance this is random → Significant (green/red badge)
  • p = 0.23 → 23% chance this is random → Not significant (gray badge)
  • p = 0.001 → Very strong evidence → Highly significant

Relative Difference​

What it means: "How much better or worse is this variant compared to baseline, as a percentage."

Baseline: 20% open rate
Variant B: 25% open rate
Relative difference: +25% (not +5 percentage points)

Collecting Data Banner​

When variants haven't reached the minimum sample size yet, you'll see a yellow banner:

"Collecting data - each variant needs at least 100 samples before significance can be calculated."

This protects you from drawing conclusions from too little data.

Metrics by Channel​

The default is the first metric the channel can report, and a metric the channel cannot report is refused at save time rather than scoring every variant 0%.

ChannelDefault MetricAvailable
EmailOpen RateOpens, Clicks, Conversion, Revenue
SMSClick RateClicks, Conversion, Revenue
PushClick RateClicks, Conversion, Revenue
In-AppCTRClicks, Conversion, Revenue
WebhookConversion RateConversion, Revenue
WhatsAppOpen RateOpens, Clicks, Conversion, Revenue

Personalized Variant (vs. Winning Variant)​

A/B testing offers two modes, both choosing from the same targets - Opens · Clicks · Conversion · Revenue:

  • Winning Variant - measure, then promote. Sends all variants, scores them on the metric above, and promotes the single best one to everyone else (this whole page).
  • Personalized Variant - predict per person. For each recipient, the ML model picks the variant that person is most likely to act on for your chosen goal - no single winner. It learns from outcomes over time (use Warm up for brand-new variants), and Conversion uses the same campaign primary conversion event. Opens and Clicks are distinct goals; "Opens" is a weaker signal since Apple Mail Privacy Protection inflates open rates.

Tips for Reliable Results​

Sample Size Guidelines​

What You're TestingMinimum Per Variant
Open rate (large effect)100-500
Click rate500-1,000
Conversion rate1,000-5,000
Small differences (1-2%)5,000+

Common Pitfalls​

  • Peeking too early - Don't check results every hour and stop when you see green. Let the test reach your minimum sample size.
  • Too many variants - 2-3 variants is ideal. More variants = more data needed per variant.
  • Testing during anomalies - Avoid holidays, outages, or other unusual periods.
  • Ignoring the CI - A "significant" result with a very wide CI may not be practically meaningful.

Next Steps​