Statistical Significance for A/B Tests

When a campaign has 2 or more variants, Joryio automatically calculates whether performance differences are statistically significant - so you know if a variant is actually better, not just randomly ahead.

What You'll See

Confidence Intervals

Each rate metric (Open Rate, Click Rate) shows a range below it:

Open Rate: 24.90%
[21.3% - 28.8%]

This means the true rate is 95% likely to fall within that range. Narrower ranges = more reliable data.

Significance Badges

Next to each rate, you'll see a colored badge:

• Green +12.5% - This variant is significantly better than the baseline
• Red -5.2% - This variant is significantly worse than the baseline
• Gray p=0.234 - Not enough evidence yet (need more data)

A result is "significant" when p < 0.05, meaning there's less than a 5% chance the difference is due to random chance.

Leader & Winner Tags

• "Leading" (blue tag) - The variant with the highest rate on the primary metric
• "Winner" (green tag + trophy icon) - A statistically confirmed winner that has been officially declared

Setting Up Statistical Analysis

A/B Test Configuration

When creating or editing a campaign with 2+ variants, you can configure the analysis:

Setting	Description	Default
Primary Metric	Which metric determines the winner	Open Rate (In-App: CTR)
Min Sample Size	Minimum recipients per variant before significance is calculated	100
Auto-Winner	Automatically declare winner when significance is reached	Off
Auto-Promote	Shift 100% traffic to winner after declaration	Off

Available Primary Metrics

Metric	Best For	What It Measures
Opens (open rate)	Subject line tests	Unique human opens / delivered
Clicks (click rate)	Content & CTA tests	Unique human clicks / delivered
Conversion (conversion rate)	End-to-end impact	Conversions of the campaign's primary conversion event (Conversion Tracking) / recipients
Revenue (per recipient)	Order value / AOV	Sum of purchase amounts (purchase / Order Completed) in the attribution window / recipients
CTR	In-app messages	Clicks / impressions

Conversion uses the campaign's primary conversion event from Conversion Tracking (set one there, or the winner can't be scored on conversion). Revenue is a continuous, amount-weighted metric - it has no significance test, so a revenue winner is declared by the highest revenue-per-recipient at the deadline (it doesn't early-stop).

API Configuration Example

{
  "abTestConfig": {
    "primaryMetric": "openRate",
    "minSampleSizePerVariant": 200,
    "autoWinnerEnabled": true,
    "autoPromoteEnabled": false
  }
}

How the Baseline Works

Every significance calculation compares a variant against a baseline:

1. Control group - If you have a control group variant (no message sent), it's used as the baseline
2. Best performer - If there's no control group, the variant with the highest rate on the primary metric is used

All other variants show their relative difference compared to this baseline.

Auto-Winner Detection

When autoWinnerEnabled is turned on:

1. A background job checks every 5 minutes
2. It verifies all variants have reached the minimum sample size
3. If a variant has p < 0.05 on the primary metric → winner is declared
4. If autoPromoteEnabled is also on → traffic is shifted 100% to the winner

What Happens After a Winner

• A green banner appears at the top of the variant comparison table
• The winning variant gets a trophy icon and "Winner" tag
• Original traffic weights are saved (for rollback if needed)
• If auto-promote is on, all new recipients get the winning variant

When No Winner is Found

If variants perform similarly and no significant difference emerges:

• The "Leading" tag shows which variant is currently ahead
• The gray p=... badges show how close each comparison is to significance
• Consider running the test longer or increasing sample size

Understanding the Numbers

Confidence Interval (CI)

What it means: "We're 95% confident the true rate falls within this range."

Example:
Variant A: Open Rate 20.0% [16.7% - 23.7%]
Variant B: Open Rate 24.9% [21.3% - 28.8%]

The CIs overlap here, suggesting the difference may not be significant. When CIs don't overlap, the difference is almost certainly significant.

P-Value

What it means: "The probability of seeing this big a difference (or bigger) by pure chance."

• p = 0.03 → Only 3% chance this is random → Significant (green/red badge)
• p = 0.23 → 23% chance this is random → Not significant (gray badge)
• p = 0.001 → Very strong evidence → Highly significant

Relative Difference

What it means: "How much better or worse is this variant compared to baseline, as a percentage."

Baseline: 20% open rate
Variant B: 25% open rate
Relative difference: +25% (not +5 percentage points)

Collecting Data Banner

When variants haven't reached the minimum sample size yet, you'll see a yellow banner:

"Collecting data - each variant needs at least 100 samples before significance can be calculated."

This protects you from drawing conclusions from too little data.

Metrics by Channel

Channel	Default Metric	Available
Email	Open Rate	Opens, Clicks, Conversion, Revenue
SMS	Open Rate	Opens, Clicks, Conversion, Revenue
Push	Open Rate	Opens, Clicks, Conversion, Revenue
In-App	CTR	CTR, Conversion
Webhook	Click Rate	Clicks, Conversion
WhatsApp	Open Rate	Opens, Clicks, Conversion, Revenue

Personalized Variant (vs. Winning Variant)

A/B testing offers two modes, both choosing from the same targets - Opens · Clicks · Conversion · Revenue:

• Winning Variant - measure, then promote. Sends all variants, scores them on the metric above, and promotes the single best one to everyone else (this whole page).
• Personalized Variant - predict per person. For each recipient, the ML model picks the variant that person is most likely to act on for your chosen goal - no single winner. It learns from outcomes over time (use Warm up for brand-new variants), and Conversion uses the same campaign primary conversion event. Opens and Clicks are distinct goals; "Opens" is a weaker signal since Apple Mail Privacy Protection inflates open rates.

Tips for Reliable Results

Sample Size Guidelines

What You're Testing	Minimum Per Variant
Open rate (large effect)	100-500
Click rate	500-1,000
Conversion rate	1,000-5,000
Small differences (1-2%)	5,000+

Common Pitfalls

• Peeking too early - Don't check results every hour and stop when you see green. Let the test reach your minimum sample size.
• Too many variants - 2-3 variants is ideal. More variants = more data needed per variant.
• Testing during anomalies - Avoid holidays, outages, or other unusual periods.
• Ignoring the CI - A "significant" result with a very wide CI may not be practically meaningful.

Next Steps

• A/B Testing basics - How to create and run A/B tests
• Creating campaigns - Campaign setup guide
• Campaign analytics - Understanding your results