Control groups explained
A control group is a slice of your audience that you deliberately treat differently, so you have something to compare your results against.
Without one you only know what happened. With one you know what your work changed.
If you send a campaign and 4% of people buy, that 4% is not the value of the campaign. Some of those people were going to buy anyway. A control group is how you find out how many.
The one rule
There are four different control groups in Joryio, and people often assume they work the same way. They don't - but they all follow one rule:
A control group gets whatever would have happened if you hadn't done the clever thing.
So the question is always: what is the clever thing here, and what does "without it" look like? The answer differs, which is why the three behave differently.
The four control groups
| Where | The control group receives | It answers |
|---|---|---|
| Global control group | Nothing at all, permanently | Is all of my marketing worth anything? |
| Campaign A/B test | Nothing at all, for this send | Is sending this better than not sending it? |
| AI-optimized campaign | A randomly chosen message | Is the AI better than just picking something? |
| Journey experiment (Personalized) | Your configured path split | Is personalizing better than my own split? |
The label in the app tells you which one you're looking at - each says what that group receives.
The first one is different in kind from the other three: it is not attached to a campaign at all, it applies to everything you send, and it stays the same people for months. It has its own page - see Global control group.
1. Campaign A/B test: "Control - no message"
What it is. When you build an A/B test you can add a control group alongside your message variants. People in it receive nothing.
Example. You're sending a win-back email to 10,000 lapsed customers.
- Variant A goes to 4,500 people
- Variant B goes to 4,500 people
- Control goes to 1,000 people, who get no email
Two weeks later:
| Group | Bought something |
|---|---|
| Variant A | 6.0% |
| Variant B | 5.5% |
| Control (no email) | 4.0% |
What you learn. Variant A beat Variant B - but more importantly, both beat doing nothing. Your campaign added about 2 percentage points. Without the control you'd have celebrated a 6% conversion rate without knowing that 4% of it would have happened anyway.
When to use it. Any time you want to justify a campaign existing at all - especially recurring sends, discount offers, and anything where people might have bought regardless.
What to expect. Your headline numbers will look worse with a control group, because you're now measuring your real contribution instead of total activity. That's the point.
The trade-off. Those 1,000 people get nothing. If it's a genuinely useful message, that's a real cost. Keep the control small (5-10% is normal) and don't leave one on every campaign forever.
2. AI-optimized campaign: "Control group % - gets a random pick"
What it is. In an AI-optimized campaign, an AI decides per person which message and channel to send. There is no split you configured - the AI is the whole thing.
So "without the clever thing" doesn't mean sending nothing. It means picking something at random. People in this control group do get a message - just a randomly chosen one.
Example. You have four offers and an AI-optimized campaign to 20,000 people, control group set to 10%.
- 18,000 people get the AI's pick for them
- 2,000 people get one of the four offers chosen at random
| Group | Bought something |
|---|---|
| AI's pick | 7.2% |
| Random pick | 5.9% |
| AI's contribution | +1.3 points |
What you learn. The AI is worth about 1.3 points over picking blindly. If those two numbers were the same, the AI isn't earning its keep yet - usually because it hasn't seen enough results to learn from.
When to expect a difference. Not immediately. The AI learns from outcomes, and outcomes take time - hours for clicks, days for purchases. On a brand-new set of offers it starts out effectively random and improves as results arrive. Judge it after several sends, not the first one.
One thing that surprises people. Someone in this control group can get a different message each send. That's deliberate - "picking at random" means rolling the dice each time. If they got the same message forever, you wouldn't be comparing against random picking, you'd be comparing against one arbitrary list.
3. Journey experiment, Personalized mode: "Control group % - keeps your split"
What it is. In a journey you can add an Experiment node that splits people down different paths - and you set the percentages. Switch it to Personalized and a model starts picking the path per person instead, ignoring your percentages.
So here "without the clever thing" means your split. People in this control group keep getting the percentages you configured.
Example. An experiment node with three paths and your split set to 40 / 30 / 30. Personalized mode on, control group 20%.
- 80% of people get the path the model picks for them
- 20% of people keep plain 40 / 30 / 30
| Group | Completed the journey |
|---|---|
| Model's pick | 31% |
| Your 40/30/30 split | 26% |
| What personalizing added | +5 points |
What you learn. Personalizing is beating the split you'd otherwise have run. That is exactly the decision you care about - not "is the model good in the abstract", but "is it better than what I was already doing".
Why this control group keeps the same path each time. Because your split does. Someone your 40/30/30 put on the discount path stays on the discount path. The control has to behave like your split, or you're not comparing against your split.
When to expect a difference. Same as the AI campaign - the model needs results to learn from. A brand-new experiment runs as a plain split at first and personalizes as data arrives. The node shows a warming-up note until then.
What a holdback is (and isn't)
You'll also see the word holdback in A/B tests. It is not a control group and it is not compared against anything.
When you run an A/B test with automatic winner selection:
- The test goes to part of your list - say 30%
- Joryio waits and works out which variant won
- The winning message then goes to the remaining 70%
That 70% is the holdback. They're not being held out of anything - they're waiting for the best version. They all get a message in the end.
Quick way to keep it straight: a control is a group you compare against. A holdback is a group you send to later.
Choosing a size
Bigger control group = more reliable answer, less audience receiving your best treatment. The usual balance:
| Audience size | Suggested control |
|---|---|
| Under 5,000 | 15-20% - small audiences need a bigger slice to say anything confidently |
| 5,000 - 50,000 | 10% |
| Over 50,000 | 5% |
Two practical warnings:
- Too small and you learn nothing. A 2% control on a 3,000-person list is 60 people. One or two purchases swing the result. See Statistical significance for how Joryio decides a result is real.
- Don't compare across different tests. A control group belongs to the campaign or experiment it was defined in. Numbers from one test's control aren't a baseline for a different test.
Common questions
Do control group members know they're in one? No. They simply receive nothing (A/B), or a normal message chosen a different way (the other two).
Is it the same people every time? Membership is stable, yes - which is what lets you watch the same group over time. In the AI campaign the message they receive changes each send, but who is in the group doesn't.
If I have a global control group too, do the percentages add up? No - they nest. A contact held out globally is excluded from the campaign before its control group is drawn, so they are never counted twice. With a 10% global holdout and a 20% campaign control, 1,000 contacts split as 100 global, 180 campaign control (20% of the remaining 900) and 720 receiving the campaign. That is 280 receiving nothing, not 300. See Global control group.
Can I look at a control group's results on their own? They're most useful as a comparison, not on their own. "The control converted at 4%" means little; "the campaign beat the control by 2 points" is the number worth reporting.
My control group is doing better than my campaign. What now? Take it seriously - it can happen and it's worth knowing. Check first that both groups are large enough for the gap to be real rather than noise. If it holds, the message is genuinely underperforming: fatigue, wrong audience, or wrong offer. That's a finding, not a failure of the test.
Do I need one on every campaign? No. Use one when you're deciding something: is this campaign worth running, is the AI paying off, is personalizing better than my split. For routine sends where the answer is already known, the control is just an audience you didn't message.