Predictive A/B Testing: What a Prediction Before Launch Can and Cannot Tell You

01What Does Predictive A/B Testing Mean?

When a vendor says its tool predicts, the first question is when. A rating before the test exists tells you whether to build it. A signal while the test runs changes how traffic is split or when the test stops. A figure after the test tells you which variant to keep. All three get called prediction, and they are worth very different things.

The third meaning is the one most testing tools use. Shoplift, for example, uses “Bayesian statistical methods” and reports a main metric called “Probability to Win” once a test has traffic. That is a readout, not a forecast. The second meaning has its own trade-offs, covered in multi-armed bandits vs A/B tests and sequential testing. This page is about the first meaning: a rating of an idea before anyone builds it.

02Why Would You Want a Prediction Before Launch?

Start with the base rate. Per Ronny Kohavi, 10 to 20 percent of experiments win. Academic work points the same way: Berman and Van den Bulte (Management Science, 2022) analysed 2,766 Optimizely experiments and estimated that about 70 percent of the tested effects were true nulls, changes that did nothing measurable. In plain words, the most likely outcome of any test idea is that nothing happens.

People are not good at spotting the exceptions. In GoodUI’s 2022 analysis of 70,149 guesses on GuessTheTest, people picked the winner of two variants 59 percent of the time, a little better than a coin. Milkman and colleagues (Nature, 2021) asked experts to rank behavioural interventions and found they “failed to predict which interventions would be most effective”.

That is the case for a prediction before launch. A shop with forty ideas and room for four tests a month has to choose. If the choice rests on taste, most of the slots go to ideas that do nothing. If it rests on what happened when comparable shops tested comparable ideas, more of the slots go to ideas with a record.

03How Do Vendors Predict Test Outcomes Today?

*Likelihood to Win is one percent figure per test idea, shown with how many tests back it. It ranks an idea against the shop’s own backlog. It is not a win probability in the strict sense.

Apex by DRIP: Likelihood to Win

Apex by DRIP calls its method Prediction-Based Experimentation. The plain version: we know what shops like yours tested and which tests won. Apex has three layers: a test memory of 4.3 million A/B tests from 151,000 shops, collected since 2011; a testing tool that builds, launches and evaluates tests directly in the shop; and managed execution with the DRIP team. Every test idea gets a Likelihood to Win before launch, one percent figure per idea shown with how many tests back it. The backlog classifies ideas automatically and scores them.

Apex scores each idea against 4.3 million observed A/B tests from 151,000 shops and shows how many tests back each score. The nearest published figure in this comparison is Kameleoon’s Ideate, trained on more than 20,000 experiments; Varify cites 10,000 or more test patterns; no other vendor in this comparison publishes a count of tests behind its recommendations (vendor pages checked October 2026).

Apex has published a holdout figure and a pre-registration. In a holdout check within a shop, the top third of ideas won 1.4 to 1.6 times as often (press release, September 29 2026; vendor-stated). Predictions for 33 brands were scored and frozen in early September 2026, and the results are published from the launch on. The full product is described in what Apex by DRIP is.

Kameleoon: PBX Ideate

Kameleoon’s Ideate agent is part of PBX 2.0, launched April 16 2026. It “analyzes your website to identify the highest-impact experimentation opportunities” and suggests “up to eight test ideas for each page”. The Ideate page says it is “Trained on 20,000+ experiments” and names the “Lever Framework, the CRO methodology developed by GAIN and refined over 15 years through more than 10,000 experiments”. The launch post gives a different figure: “a database of over 25,000 experiments built over more than a decade by CRO specialists”.

Kameleoon also publishes an accuracy figure: “63% accuracy on high-confidence predictions, compared to an industry average of around 20%”. The launch post says Ideate ideas are “three times more likely to convert than the industry average”. Both are vendor statements. Before you rely on them, ask what counts as a high-confidence prediction and what share of ideas receive one. The side-by-side is in Apex vs Kameleoon.

Varify: AI ideas from test patterns

Varify, a German testing tool, added a CRO AI Audit on February 14 2026 that produces test ideas and a site assessment. Its AI page says “Recommendations are based on CRO patterns from real experiments” and cites “10,000+ test patterns”, and states that “Over 1,370 teams use Varify AI”. Varify has also published the counterpoint: “No model can reliably predict user behavior on your specific site” (varify.io, checked September 2026). We agree with that sentence. A pattern is a starting point, and the test on your shop decides. Details in Apex vs Varify.

Shoplift: Lift Assist

Shoplift describes Lift Assist as “Shoplift’s recommendation engine which dynamically provides theme sections”. It duplicates the template and inserts a section styled with the shop’s imported Brand Styles; it “does not automatically add content to this section”. For its data, Shoplift says it “analyzes millions of shopper sessions to recommend what converts” and “Leverages your brand and test ideas from hundreds of top brands”. It describes its data in shopper sessions rather than in tests, and we found no published accuracy figure for the recommendations. Details in Apex vs Shoplift.

Wingify (VWO and AB Tasty): agents on proprietary data

VWO and AB Tasty relaunched as Wingify on September 16 2026, “the Agentic Experience Optimization Platform”. Its Hypothesis Agent “turns evidence into ranked, testable ideas and drafts the variations”, and “recommendations are presented for human review”. The data is described as “trained on AB Tasty’s proprietary data and user insights” and “Wingify’s proprietary data and UX principles”, with no corpus size and no published win prediction. A Synthetic Testing Agent “tests a change on synthetic visitors built from your real segments”. Details in Apex vs VWO.

Shopify SimGym: simulated shoppers

SimGym is a separate Shopify app, described as “a Shopify app in AI Research Preview that uses AI-powered shoppers to simulate buyer behavior on your online store”. It compares two themes or analyses one. Shopify says: “By learning from real commerce sessions across our platform, we have trained models to simulate real customer personas who can approximate what shoppers might do and feel.” This is a different kind of prediction. It does not look up how past tests ended; it sends AI agents through your store and reports what they did.

Shopify is open about the limit. Its product lead Aaron Glazer said on the Unofficial Shopify Podcast (December 2025): “It’s not fully testing”, and positioned SimGym for stores without enough traffic for split tests. On the App Store, SimGym has 2.8 of 5 from 40 reviews; one reviewer writes “there’s no way to export simulation results, and there’s no way to customize simulation shoppers” (checked October 2026). Details in Apex vs Shopify Rollouts.

Autonomous CRO tools

A newer group of tools proposes, builds and sometimes launches tests with agents. Dalton proposes tests ranked by projected revenue and says “nothing goes live without your approval”. Spar ranks ideas and promises “Paste a URL. Get a prioritized backlog in 10 minutes”, with “No autopilot”. CRO Agent scores ideas on “Proof, Ease, Cost, Time, Impact” and runs no tests itself. Keak states a “73% win rate” (press release, January 15 2025). These are vendor statements; on the pages we read (checked October 2026) we found no method behind the headline figures. The wider field is covered in AI A/B testing tools.

04What Can a Prediction Tell You, and What Can It Not?

What a prediction can tell you

• Which ideas to run first. A rating sorts a backlog. With limited traffic, order matters more than volume.
• Which ideas to drop. An idea that has rarely worked in comparable shops can wait. Every test you do not build saves design, development, QA and traffic.
• How much evidence sits behind a rating. A score backed by many observed tests means something different from a score backed by a handful. Ask to see the count.

What a prediction cannot tell you

1. How your audience will react. No model knows your visitors. That is why you still run the test. What a prediction changes is which tests you run.
2. Anything about a new idea. An idea nobody has tested has no record. It should score as unknown, not as bad. Keep some capacity for ideas like that, or the program stops learning.
3. Whether past wins were real. Training data is noisy. Berman and Van den Bulte estimate that 18 to 25 percent of results called significant at the 5 percent level are false discoveries. A model trained on reported wins inherits some of that noise.
4. Effects inside one segment. A change can be flat overall and strong for mobile visitors or returning customers. A rating is a summary, and summaries lose detail.
5. Whether the test is run well. A well-chosen test that breaks on one browser, tracks the wrong event or gets called early produces nothing useful. The rating belongs in the planning meeting, not in the stopping rule.

For context, our own record, not a promise for your shop: across DRIP’s own tests, the win rate went from 27 percent in 2024 (808 tests) to 56 percent between April and June 2026 (278 tests). The industry baseline is 10 to 20 percent, per Ronny Kohavi. A win rate depends on how a win is counted and on which ideas reach a test, and loose statistics can produce a high one, so read any vendor’s win rate, ours included, with its period, its test count and its definition. More benchmarks are in A/B testing statistics.

05How Do You Check a Vendor’s Prediction Claim?

1. When does the prediction happen? Before the test is built, while it runs, or after? Only the first one changes what you test.
2. What is it computed from, and how was that counted? Observed tests, written case studies, test patterns, shopper sessions and purchases are different units. A figure in one unit cannot be compared with a figure in another.
3. What does the number mean? A probability that this test wins, a rank within your backlog, or a projected revenue figure? Each needs a different reading.
4. Was it checked on data the model had not seen? A holdout check compares predictions with results the model was not trained on. Without one, an accuracy figure can describe the training data rather than your future tests.
5. Were predictions frozen before the results? Scores recorded before tests run, then compared with what happened, are harder to tune after the fact.
6. What does the vendor say it cannot do? A real method comes with limits. If a vendor names none, that tells you about the marketing, not the method.

Applied to Apex, the answers on record are these. The rating happens before launch. It is computed from 4.3 million observed A/B tests from 151,000 shops, and each score shows how many tests back it. The number ranks an idea against the shop’s own backlog; it is not a win probability in the strict sense. The holdout check within a shop found the top third of ideas won 1.4 to 1.6 times as often. Predictions for 33 brands were frozen in early September 2026, with results published from the launch on. Ask us for the details behind each of these in a call, the same way you would ask any other vendor.

06Which Approach Fits Which Shop?

Where Apex is not the right choice

• A team that already picks winning tests well gets less from the prediction layer.
• A shop that cannot ship a test a week yet should build that habit on a lighter tool first; buying prediction before you can execute is the wrong order.
• Anyone who needs a published price list before a conversation. Apex’s pricing is not published; a card-free free tier for one store exists, and the next step is to book a call.

For the full picture beyond prediction, with fifteen tools compared on what they test, statistics, shop systems, data location and price, read A/B testing tools for online shops.