A/B Testing Campaigns

A/B testing allows you to test different versions of your campaigns to see which performs better and optimize your messaging.

New to control groups? Control groups explained covers all three - what each one receives, and what each test can tell you.

What is A/B Testing?

A/B testing (also called split testing) sends different versions of your campaign to different segments of your audience to determine which performs best.

Example:

• Variant A: "Get 20% off your first purchase"
• Variant B: "Save $10 on your first order"

Send each to 50% of your audience and see which has higher conversion.

When to Use A/B Testing

Subject Lines (Email)

Test different subject lines to improve open rates:

Variant A: "Your exclusive discount inside"
Variant B: "Save 20% today only"
Variant C: "Limited time offer for you"

Message Content

Test different message copy:

Variant A: Long-form explanation with benefits
Variant B: Short, punchy message with CTA
Variant C: Story-based narrative

Call-to-Action

Test different CTAs:

Variant A: "Shop Now"
Variant B: "Get My Discount"
Variant C: "Start Shopping"

Timing

Test different send times:

Variant A: Monday 9 AM
Variant B: Wednesday 2 PM
Variant C: Friday 6 PM

Sender Information

Email sender is per variant - each A/B variant sets its own sender, From, Reply-To and BCC in the variant editor's Settings tab (the Sender section). So you can test the From line, one of the biggest open-rate levers:

Variant A: From "John at Company" <[email protected]>
Variant B: From "Company Team" <[email protected]>
Variant C: From "Company" <[email protected]>

The senders, Reply-To and BCC addresses you can pick come from your workspace's managed lists - see Email Configuration.

Creating an A/B Test

A/B variants are authored inside the Compose step's VariantTabs shell - the same shell every channel uses. Add variants by clicking the + Add variant button at the end of the tab strip, then author each tab's body the same way you'd author a single-variant campaign.

Step 1: Open the campaign wizard

1. Navigate to Campaigns → New Campaign
2. Fill in the Details step
3. Configure Delivery (or, for in-app campaigns, the Delivery trigger picker)

Step 2: Add variants in Compose

The Compose step starts with a single variant tab labeled A. Click + Add variant to add a second tab (B), and again for additional variants. Each tab carries:

• A colored dot for quick visual identification
• A read-only weight chip (allocation lives in the A/B card below)
• An amber dot if the variant hasn't been authored yet

For email and in-app variants, use the 3-tile method picker (Drag & Drop / HTML editor / Use template) on each tab. SMS and push tabs accept their channel's body directly. WhatsApp tabs let you map a different template per variant.

Step 3: Set the weight allocation

The A/B Allocation card below the tabs has a % unit-addon input per variant. Total weight across all variants must sum to 100%.

Variant A: 50%
Variant B: 50%

Or split across more variants:

Variant A: 33%
Variant B: 33%
Variant C: 34%

If you've added a control group, its weight stays fixed when you add or remove message variants - only the non-control variants are rebalanced across the remaining budget. If you remove every message variant, the control stays put and the Total Weight validator surfaces the gap until you add a variant back.

Step 4: Choose stopping rule + optimization method

The A/B card also exposes:

• Stopping rule preset - Quick (4h deadline), Standard (24h deadline), or Custom. Both built-in presets early-stop on statistical significance.
• Optimization method - locked with a "Needs 2+ variants" pill until you've added a second variant. Dropping below two variants auto-resets the method to none.

Choosing the optimization method

Method	What it does	When to use
No Optimization	Plain A/B - split evenly, you read the results	You want to compare variants and decide manually
Winning Variant	Test on a slice, then auto-send everyone the best performer	A one-shot send where you want the winner picked for you
Personalized Variant	AI picks the best variant per user from their history	Recurring campaigns or reused content - it personalizes per person, not one global winner

About Personalized Variant (cold start / warming up): the AI learns from outcomes, and outcomes take time to arrive (hours for clicks, days for conversions). So a brand-new set of variants is not personalized on the first send - it runs as a normal A/B (even split + a random holdout for measuring lift) and personalizes automatically as results accrue. That makes it best for recurring campaigns and reused offers (the model is already warm); a one-shot blast of brand-new variants will simply be a standard A/B that seeds future learning. The campaign shows a "personalization is warming up" note until the model has learned its variants.

Step 5: Choose Winning Metric

Select which metric determines the winner:

• Open Rate (email, WhatsApp and Viber - the only channels that report a read) - % who opened the message
• Click Rate (every channel except webhook, which has nobody to click) - % who clicked any link
• Conversion Rate - % who completed goal event
• Revenue - Total revenue generated (requires conversion tracking)

Step 6: Test duration

The Quick / Standard presets ship with sensible deadlines (4h / 24h respectively), counted from when the test sends go out - not from when you set up the campaign, so a test scheduled for later still gets its full window. For longer-running tests, use Custom to set your own deadline and minimum sample size.

Step 7: Publish

Hit Publish in the wizard header. The A/B engine handles variant assignment via deterministic hashing (hash(userId + campaignId)), so the same user always sees the same variant across resends.

A/B availability by channel

The A/B card shows for every variant-based channel: email, SMS, WhatsApp, webhook, push, in-app. Push used to save a single body at the campaign level - legacy push campaigns hydrate into a single Main variant on edit, which is what enables push A/B testing now.

Example A/B Tests

Example 1: Subject Line Test (Email)

Goal: Improve open rate

Setup:
- Variant A: "Your monthly newsletter"
- Variant B: "5 tips to boost productivity"
- Traffic: 50% / 50%
- Winning metric: Open rate
- Duration: 24 hours

Results:
- Variant A: 15% open rate
- Variant B: 28% open rate ← Winner

Insight: Emoji and specific value proposition performs better

Example 2: Call-to-Action Test

Goal: Improve click-through rate

Setup:
- Variant A: "Learn More" button
- Variant B: "Get Started Free" button
- Variant C: "Try It Now" button
- Traffic: 33% / 33% / 34%
- Winning metric: Click rate
- Duration: 48 hours

Results:
- Variant A: 5.2% CTR
- Variant B: 8.7% CTR ← Winner
- Variant C: 6.1% CTR

Insight: Emphasizing "free" increases clicks

Example 3: Content Length Test

Goal: Improve conversion rate

Setup:
- Variant A: Long-form (500 words)
- Variant B: Short-form (100 words)
- Traffic: 50% / 50%
- Winning metric: Conversion rate
- Duration: 7 days

Results:
- Variant A: 2.1% conversion
- Variant B: 3.8% conversion ← Winner

Insight: Concise messaging converts better for this audience

Example 4: Timing Test (SMS)

Goal: Optimize engagement time

Setup:
- Variant A: Send at 9 AM
- Variant B: Send at 2 PM
- Variant C: Send at 6 PM
- Traffic: 33% / 33% / 34%
- Winning metric: Click rate
- Duration: Same day

Results:
- Variant A (9 AM): 12% CTR
- Variant B (2 PM): 8% CTR
- Variant C (6 PM): 18% CTR ← Winner

Insight: Evening messages perform best for this audience

Analyzing Results

View Test Results

1. Go to Campaigns
2. Click on your A/B test campaign
3. View Variants tab

Metrics Displayed

For each variant:

Email Campaigns:

• Sent count
• Delivered %
• Open rate %
• Click rate %
• Conversion rate %
• Revenue (if tracking)

SMS Campaigns:

• Sent count
• Delivered %
• Click rate %
• Conversion rate %
• Response rate %

Push Campaigns:

• Sent count
• Delivered %
• Click rate % (a tap on the notification is a click - push has no open to measure)
• Conversion rate %

Statistical Significance

Joryio automatically computes statistical significance using a two-proportion z-test and Wilson Score confidence intervals:

• Confidence intervals below each rate show the range where the true rate likely falls
• Significance badges show whether differences are real or random:
  • Green +12.5% - significantly better
  • Red -5.2% - significantly worse
  • Gray p=0.234 - not significant yet
• Leader/Winner tags highlight the best-performing and declared-winner variants

Sample Size Matters

For reliable results, each variant should have at least:

• 1,000 recipients for open/click rate tests
• 100 conversions for conversion rate tests

For full details on confidence intervals, p-values, and auto-winner detection, see Statistical Significance.

Declaring a Winner

Manual Declaration

1. Review variant performance
2. Click Declare Winner
3. Select winning variant
4. Optionally: Send winner to remaining audience

Automatic Declaration

If enabled, Joryio automatically:

1. Waits for test duration
2. Calculates statistical significance
3. Declares winner if confidence > 95%
4. Sends winner to remaining audience (if configured)

No Clear Winner

If variants perform similarly:

• Continue with either variant
• Run a longer test
• Test something different

Best Practices

1. Test One Variable

Test only one element at a time:

Good: Test subject line only

Variant A: "Get 20% off"
Variant B: "Save big today"
Same content, same CTA

Bad: Test everything

Variant A: Subject, content, CTA all different
Variant B: Subject, content, CTA all different
Can't identify what made the difference!

2. Use Sufficient Sample Size

Ensure meaningful results:

Good: 10,000 recipients = 5,000 per variant Bad: 200 recipients = 100 per variant (too small)

Minimum recommended:

• Email open rate: 1,000 per variant
• Click rate: 500 per variant
• Conversion rate: 100 conversions per variant

3. Run Tests Long Enough

Account for day-of-week and time-of-day variations:

Good: Run for 3-7 days Bad: Declare winner after 2 hours

Recommended duration:

• Email: 24-72 hours
• SMS: 4-12 hours
• Push: 6-24 hours
• Conversion-based: 7 days

4. Test Meaningful Differences

Make variants different enough to matter:

Good:

Variant A: "Save $10"
Variant B: "Get 50% off"
(Significantly different)

Bad:

Variant A: "Save today"
Variant B: "Save now"
(Too similar)

5. Document Learnings

Keep track of test results:

Test: Subject Line - Emoji vs No Emoji
Date: Jan 15, 2024
Winner: With emoji (28% vs 15% open)
Insight: Emoji increases open rate by 87%
Action: Include relevant emoji in future emails

Common A/B Testing Mistakes

1. Testing Too Many Variants

Don't: 5+ variants on small audience Do: 2-3 variants with enough traffic each

2. Stopping Tests Too Early

Don't: Declare winner after a few hours Do: Wait for statistical significance

3. Testing Everything at Once

Don't: Change subject, content, CTA, timing simultaneously Do: Test one variable at a time

4. Ignoring Segment Differences

Don't: Assume results apply to all segments Do: Test within specific segments

5. Not Implementing Learnings

Don't: Run tests but don't use insights Do: Apply winning tactics to future campaigns

Advanced A/B Testing

Sequential Testing

Test in stages:

Stage 1: Test subject lines (winner: emoji subject)
  ↓
Stage 2: Test CTA buttons (winner: "Start Free Trial")
  ↓
Stage 3: Test content length (winner: short form)
  ↓
Final campaign: Combine all winners

Segment-Specific Tests

Test within segments:

Test 1: Trial users
- Variant A: Emphasize features
- Variant B: Emphasize price
Winner: Price-focused

Test 2: Paid users
- Variant A: Emphasize features ← Winner
- Variant B: Emphasize price
Winner: Feature-focused

Insight: Different messaging for different segments!

Multivariate Testing

Test multiple elements (requires larger audience):

Test combinations of:
- Subject (A or B)
- CTA (X or Y)

Results in 4 variants:
1. Subject A + CTA X
2. Subject A + CTA Y
3. Subject B + CTA X
4. Subject B + CTA Y

Large Audience Required

Multivariate testing splits traffic across many variants. Need 10x larger audience than simple A/B test.

Conversion Tracking

To test conversion rates, set up conversion events:

1. Go to campaign settings
2. Enable Conversion Tracking
3. Select conversion event:
   • "Order Completed"
   • "Trial Started"
   • "Form Submitted"
4. Set attribution window (e.g., 7 days)

Joryio will track which variant drives more conversions.

Next Steps

• Statistical significance & auto-winner - Confidence intervals, p-values, and automatic winner detection
• Create your first campaign
• Email campaign best practices
• Campaign analytics
• Conversion tracking setup