01Which AI A/B Testing Tool Fits Which Job in 2026?
The quick verdict, one line per job:
- Agents across the whole cycle, with an approval gate: Wingify’s Wingz (VWO and AB Tasty) and Kameleoon’s PBX 2.0.
- Already on Optimizely: its Opal agents generate ideas, build complete tests and report on the program.
- AI that builds and checks tests on Shopify: Intelligems documents a building agent, a QA agent and an MCP server.
- Working from ChatGPT or Claude: check the permissions. Elevate’s connection is read-only; Intelligems, ABlyft and Convert document building tests.
- Software that runs tests for you: Dalton, Spar and pagent document a human approval step; Keak, Variant and OptiMonk do not publish one.
- Tests that keep coming back flat: the job is ranking ideas before you build them. Apex by DRIP gives every test idea a Likelihood to Win, backed by 4.3 million observed A/B tests from 151,000 shops.
Where we write “none found”, we did not find the feature in the documentation we read; that does not prove it is missing. Full profiles of fifteen tools are in our comparison of A/B testing tools for online shops.
02What Does AI Do in an A/B Testing Tool?
- Ideas. The AI reads your pages, analytics or session data and proposes what to test.
- Ranking. The AI orders ideas by expected impact before anything is built.
- Building. The AI turns a prompt, a design file or a saved idea into a variant.
- Checking. The AI reviews the setup, or simulates visitors, before a test reaches real shoppers.
- Reading. The AI summarises a result, explains it or recommends whether to ship.
- Autonomy. The AI launches tests and applies winners, with or without a person approving each step.
Two questions separate the offers: what data sits behind the job (your traffic, a library of past experiments, or a general language model), and who approves before a test reaches shoppers. A third is new in 2026: whether the tool connects to ChatGPT or Claude, and with which permissions.
| Job for the AI | Tools that document it | What to check |
|---|---|---|
| Suggest test ideas | Wingify, Kameleoon, Optimizely, Intelligems, Varify, ABlyft | More ideas are not better ideas |
| Rank ideas before building | Apex, Kameleoon, Intelligems, Wingify | What data the ranking uses |
| Build variants from a prompt | Wingify, Kameleoon, Optimizely, Intelligems, Visually, Varify | Which test types it can build |
| Check a test before launch | Wingify, Intelligems, Dalton, Shopify SimGym | Simulated shoppers are not real buyers |
| Read results | Apex, Optimizely, Kameleoon, ABlyft, Varify | The statistics method underneath |
| Work from ChatGPT or Claude | Apex, Intelligems, Wingify, Visually, ABlyft, Convert | Read-only, or can it launch tests |
| Run tests without a person | Keak, Variant, OptiMonk | No published human review step |
| Tests keep coming back flat | Apex by DRIP | Price not published; book a call |
03Which Tools Use AI to Find Test Ideas?
Wingify’s Insight Agent “reads funnels, replays, forms, and surveys”, and its Hypothesis Agent “turns evidence into ranked, testable ideas and drafts the variations”. Kameleoon’s PBX Ideate “analyzes your website to identify the highest-impact experimentation opportunities”, with “up to eight test ideas for each page”. Optimizely added an Idea Builder in April 2026 that “generates experiment ideas based on your page, goals, and research”.
Intelligems’ Smart Test Recommendations (beta, April 2026) “surfaces high-impact test ideas automatically”. Varify launched a CRO AI Audit in February 2026. ABlyft produces “five AI-generated experiment ideas with hypotheses” for 100 credits. Elevate offers “AI-powered test suggestions” from its Standard plan, Visually lists “AI recommendations”, and ABConvert added an “Autopilot (beta)” inbox of ranked recommendations in July 2026.
Among start-ups, Spar and Dalton rank ideas from store data, and CRO Agent, from the UK agency Blend Commerce, scores ideas on “Proof, Ease, Cost, Time, Impact” without running tests.
In Apex by DRIP, the Apex chat in the app answers what to test next, and the backlog classifies ideas automatically and gives each a score.
04Which Tools Rank Ideas Before a Test Runs?
A ranking is as good as the outcomes it has seen. What the vendors publish:
- Kameleoon PBX Ideate: the Ideate page says “Trained on 20,000+ experiments”; the April 2026 launch post says “over 25,000 experiments”. Kameleoon states “63% accuracy on high-confidence predictions” (vendor-stated).
- Wingify: “trained on AB Tasty’s proprietary data and user insights” and “Wingify’s proprietary data and UX principles”; “recommendations are presented for human review”. No corpus size found.
- Intelligems Atlas: “ranks them by estimated profit impact”. Training data: not published.
- Shoplift Lift Assist: “Leverages your brand and test ideas from hundreds of top brands”.
- Varify: “Recommendations are based on CRO patterns from real experiments”, citing “10,000+ test patterns”.
- Start-ups: none publishes a database of observed tests behind its ranking.
Apex by DRIP: a Likelihood to Win for every test idea
Apex by DRIP is an A/B testing platform for companies selling online that predicts which tests will win before they go live, built on one of the largest A/B test databases in e-commerce. DRIP calls the method Prediction-Based Experimentation. The plain version: we know what shops like yours tested and which tests won.
The test memory holds 4.3 million A/B tests from 151,000 shops, collected since 2011. Every test idea gets a Likelihood to Win before launch: one percent figure per test idea, shown with how many tests back it. In a holdout check within a shop, the top third of ideas won 1.4 to 1.6 times as often (press release, September 29, 2026). DRIP scored and froze predictions for 33 brands in early September 2026; the results are published from the launch on.
Apex scores each idea against 4.3 million observed A/B tests from 151,000 shops and shows how many tests back each score. The nearest published figure in this comparison is Kameleoon’s Ideate, trained on more than 20,000 experiments; Varify cites 10,000 or more test patterns; no other vendor in this comparison publishes a count of tests behind its recommendations (vendor pages checked October 2026).
05Which Tools Build Variants With AI?
The incumbents: Wingz, PBX Build and Opal
“Wingz interprets the design intent and builds the variation” from text, Figma, a screenshot or a PDF; variations are “visible and editable before anything ships”. Kameleoon’s PBX Build creates variations from chat prompts and Figma designs, and PBX Configure “launches tests through conversational prompts”, paid in prompt credits.
Optimizely shipped a Variation Development agent in March 2026 and a Build agent in September that “turns a saved or draft idea into a complete A/B test”. Opal uses Gemini and Claude, states “Your data is never used to train the model or shared across customers”, and runs on credits with unpublished prices.
The Shopify apps and mid-market tools
- Intelligems: the Atlas Experience Agent builds content, theme, template, redirect and JS/CSS tests from text; “Offers, checkout, post-purchase, price, and shipping experience types are not yet supported.”
- Shoplift: Lift Assist inserts a theme section styled by your Brand Styles and “does not automatically add content to this section”.
- Visually: a Copilot editor changes variants by natural language, with a choice of Gemini, Claude and GPT models; a generator turns “plain English prompts (or images) into components”.
- Varify: prompt-based variants (beta, March 2026) on Pro, paid with AI credits or your own key.
- ABlyft: “Lumen generates the CSS and JavaScript code for your experiment variation”, for 50 credits.
- GrowthBook: an AI mode in the visual editor generates variations from prompts.
Apex by DRIP: building from code, a visual builder or a chat assistant
Apex builds, launches and evaluates tests directly in the shop. Experiments can be configured as code, edited in a code editor or built in a visual builder called Operator, with CLI preview links. The MCP connection lets a shop owner build and read tests from ChatGPT or Claude without a terminal; the CLI is the developer path.
06Which Tools Use AI to Check a Test Before Launch?
Wingify’s QA step “reviews configurations before launch; flags potential risks”, then a person signs off, and its Synthetic Testing Agent “tests a change on synthetic visitors built from your real segments”. Intelligems’ Atlas QA agent checks the setup before launch on all accounts. Dalton checks its variants in a browser before you approve them.
Shopify’s SimGym is “a Shopify app in AI Research Preview that uses AI-powered shoppers to simulate buyer behavior on your online store”. It compares add-to-cart rates between themes or recommends changes by “conversion impact”. Shopify adds that “results might differ from actual buyer behavior”. After a monthly allowance it costs $0.75 per single-theme simulation. For Shopify Rollouts itself we found no agents.
Apex shows clients four QA checks: all devices, tracking and targeting, responsive, and matches design, with QA via preview. Once a test runs, Guardian stops it automatically at a revenue loss threshold.
07Which Tools Use AI to Read Test Results?
- ABlyft: an AI agent recommends “Ship, Don’t Ship, or Keep Running”, for 50 credits.
- Optimizely: a Program Overview agent reports on the whole program, on top of the sequential Stats Engine.
- Kameleoon: PBX Analyze builds a “Takeaways” library of completed tests, searchable by uplift, page or metric.
- Wingify: Evi Explore gives revenue projections with confidence intervals on Bayesian SmartStats.
- Varify: AI experiment analysis since August 2026, on a default “frequentist method with one-sided tests”.
- Intelligems: a Slack bot sends summaries.
- Visually: the MCP server answers questions “about your A/B tests, revenue, and store performance”.
In Apex, the chat answers why a test won and how visitors moved, and its answers cite sources. Underneath sit fixed-horizon statistics with confidence, profit per test on Shopify, and heat maps per variant.
08Which A/B Testing Tools Connect to ChatGPT and Claude?
| Tool | ChatGPT | Claude | What the assistant can do |
|---|---|---|---|
| Apex by DRIP | Yes | Yes | Build and read tests |
| Wingify (Wingz) | Yes | Yes | Connects Wingz agents; scope not detailed |
| Kameleoon | None found | Yes, connector and Claude Code | 50+ tools: experiments, flags, segments, goals |
| Optimizely | None found | Yes | Experimentation data via OAuth |
| Intelligems | Yes | Yes | Drafts, starts, pauses, ends tests after confirmation |
| Visually | Yes | Yes | Answers questions on tests and revenue |
| Elevate | None found | Yes, Desktop and Code | Read-only; cannot create or start tests |
| Varify | Not stated | Yes, Claude connector | Create, start and pause experiments |
| ABlyft | Yes | Yes | Read and write, 17 tools |
| Convert | Yes | Yes | Read results, run audits, build tests |
| ABConvert | None found | Yes | Read and write, Growth plan and up |
| GrowthBook | None found | Yes | Create flags and experiments, read results |
| Shoplift | Via HTTP API | Via HTTP API | No MCP server found |
| pagent | Yes | Yes | MCP server for its testing agent |
| Shopify Rollouts | None found | None found | No agent connection found |
Behind the cells: Intelligems’ tools ask before they act (“Your AI client will ask you to confirm before running them”). Elevate’s server “cannot create, start, pause, or end an experiment”. Kameleoon’s PBX Ship uses MCP to turn winning variants into production code. Shoplift pitches “Command Your CRO Program From Claude or ChatGPT” over “plain HTTP”.
Apex’s MCP connection lets a shop owner build and read tests from ChatGPT or Claude without a terminal. A server that can start tests needs the same review rules as a person who can.
09Which Tools Run Tests Without Human Approval?
Wingify calls autonomy “a dial, not an all or nothing”, from “propose-and-approve” to autopilot, and says “the safeguards never switch off: an approval gate, guardrail metrics, traffic caps, a kill switch, and a permanent holdout”. Intelligems takes the other default: “Activation is a manual step.”
The autonomous start-ups sell the whole loop:
| Tool | What the AI does | Approval before launch | Entry price |
|---|---|---|---|
| Keak | Generates and launches variants; Auto Pilot | None found | $0.005 per impression |
| Dalton | Proposes, builds and browser-checks tests | Yes, nothing goes live unapproved | $74 a month, annual billing |
| Spar (formerly Cuped.ai) | Audits Shopify stores, ranks ideas, writes variants | Yes, “No autopilot” | $500 a month Pro (website) |
| pagent | Hypotheses, variants as code, runs tests | Yes, people approve each variant | Custom quote by monthly users |
| CRO Agent | Scored idea list; runs no tests | Not applicable | Not announced; waitlist |
| Coframe | Agents design, code and deploy tests | None found | Not public |
| Fibr AI | Agents personalise every URL as overlay | None found | None found |
| Sherpa | Studies behaviour, builds variants, keeps testing | None found | None found |
| Variant | Three agents; deploys experiments automatically | None found | $199 to $599 a month |
| Parallel | Self-optimising storefront with “Agentic A/B” | None found | $199 to $1,399 a month plus credits |
| Evolv AI | Reinforcement learning on live experiences | None found | Not public |
| OptiMonk Smart Product Page Optimizer | Writes, tests product pages; applies winners | No, winners applied automatically | From $198 a month |
Spar and Variant are Shopify apps; Dalton runs on one snippet across shop systems, and pagent on a script tag.
What none of the start-ups shows
Absences in public material, checked October 2026, not judgements:
- None of the start-ups publishes a database of observed tests behind its predictions.
- None offers price, offer or shipping tests; human-run apps such as Intelligems and Elevate do.
- Human review by people appears where an agency sits behind the tool: Blend Commerce for CRO Agent, and Dalton’s $699 tier with a CRO strategist.
- Keak and Spar publish a statistics method (sequential); Dalton’s two pages contradict each other.
- None publishes a measured win rate beyond headline figures such as Keak’s “73% win rate” (vendor-stated).
Apex’s third layer is managed execution: tests built, checked, launched and read together with the DRIP team. An autopilot mode is not part of the Apex product description this comparison uses.
10What Does Each Main Tool Offer, at a Glance?
Apex by DRIP: best for shops whose tests keep coming back flat
- Best for: shops whose tests keep coming back flat: the constraint is choosing which tests to run, not building them.
- Can test: pages and page elements, parallel tests with optional exclusion groups, A/A tests, redirect tests, global holdout, feature flags and staged rollouts.
- Shop systems: installs as a Shopify app (two-click install via link, server-side tracking, first-party data).
- Data location: hosting in the EU plus UK; GDPR in place; ISO 27001 and SOC 2 in progress, no certification yet.
- Entry price: pricing is not published; a card-free free tier for one store exists; book a call.
- Main strength: the memory of millions of observed tests behind every idea, so selection improves before anything is built.
- Main limitation: the price is not published.
Documented AI: a Likelihood to Win per test idea, the Apex chat, a backlog with automatic classification and a score, an MCP connection for ChatGPT and Claude, and a CLI. Not the right choice for a team that already picks winning tests well, for a shop that cannot ship a test a week yet (buying prediction before you can execute is the wrong order), or for anyone who needs a published price list before a conversation.
Wingify (VWO and AB Tasty): best for agents across the whole test cycle
- Best for: teams that want agents from research to rollout, with autonomy they can raise step by step.
- Can test: pages and features with staged rollouts; price testing “coming soon” on the VWO Shopify page.
- Shop systems: VWO Shopify app since March 2024, script not on checkout pages.
- Data location: not covered in the sources we read.
- Entry price: not published; a third party reports $665 a month for VWO Testing Growth, annual, at 100,000 tracked users; 30-day trial (checked October 2026).
- Main strength: Wingz agents for every step, with safeguards that “never switch off”.
- Main limitation: no public prices and no corpus size behind the idea ranking.
VWO and AB Tasty relaunched as Wingify, “the Agentic Experience Optimization Platform”, on September 16, 2026. Not the right choice for Shopify shops that want price, offer or checkout tests today. Head to head: Apex vs VWO.
Kameleoon: best for prompt-built tests plus server-side flags
- Best for: teams that want prompt-built tests, server-side flags and an MCP server in one platform.
- Can test: any page element the snippet can change, plus server-side features; price tests: none found.
- Shop systems: JavaScript snippet; the Shopify integration is documented for Shopify Plus.
- Data location: region not stated; ISO/IEC 27001:2022 and SOC 2 Type 2.
- Entry price: PBX Starter “Starting from $495/month”; 30-day free trial (checked October 2026).
- Main strength: client-side tests, server-side flags and an MCP server that turns winners into production code.
- Main limitation: Shopify support is documented for Plus, and the App Store listing has no reviews.
Documented AI: PBX 2.0 agents for Ideate, Build, Configure, Analyze and Ship, an MCP server and a Claude connector. Not the right choice for Shopify stores below Plus or for small budgets. Head to head: Apex vs Kameleoon.
Optimizely: best for enterprise statistics with agents built in
- Best for: enterprise teams that want sequential statistics, server-side testing and building agents in one product.
- Can test: anything in the page through the snippet, server-side logic through SDKs; Shopify checkout is measured, not changed.
- Shop systems: snippet in the theme; no testing app on the Shopify App Store.
- Data location: “Your data will be stored in EU data centers”; ISO 27001 and SOC 2 Type 2.
- Entry price: not published; Vendr reports a median of $81,447 a year (checked October 2026).
- Main strength: a broad statistics and agent stack in one product.
- Main limitation: no public pricing; a third-party median of about $81,000 a year.
Documented AI: Opal with Idea Builder, Variation Development, Build and Program Overview agents, plus an MCP server. Not the right choice for small budgets or Shopify checkout tests. Head to head: Apex vs Optimizely.
Intelligems: best for AI-built tests next to price and shipping tests
- Best for: Shopify merchants who test prices, shipping and offers and want agents for content tests.
- Can test: prices, discounts, bundles, shipping, themes, templates, content, checkout blocks (Plus), post-purchase offers.
- Shop systems: Shopify, every plan; no other shop system.
- Data location: region not published; data “will be transferred outside of Europe”.
- Entry price: Smart Content $69 a month; AI on every paid plan (checked October 2026).
- Main strength: prices, shipping, offers, checkout and content in one app, with profit and COGS reporting.
- Main limitation: price testing needs a team-built integration and a 3-month minimum.
Documented AI: Atlas recommendations, Experience Agent and QA agent, and an MCP server for Claude, ChatGPT, Gemini and Grok. Not the right choice for buyers who need EU data documentation up front. Head to head: Apex vs Intelligems.
Shoplift: best for theme and template tests on Shopify
- Best for: Shopify brands that test themes and templates in the Theme Editor.
- Can test: templates, the theme, prices (Advanced and Pro); not checkout pages.
- Shop systems: Shopify and Shopify Plus; headless “not currently supported”.
- Data location: data “is transferred to and processed in the United States”.
- Entry price: Core $99 a month up to 50,000 visitors; 14-day trial (checked October 2026).
- Main strength: tests run on Shopify’s own themes and templates.
- Main limitation: no headless, checkout or A/B/n testing.
Documented AI: Lift Assist and an HTTP API for Claude and ChatGPT; no MCP server found. Not the right choice for headless shops or tests with three or more variants. Head to head: Apex vs Shoplift.
Visually: best for an AI editor plus personalisation on Shopify
- Best for: Shopify shops that want testing, personalisation and a natural-language editor in one app.
- Can test: “themes, discounts, prices & shipping”; price tests need 600 or more monthly orders on Scale.
- Shop systems: Shopify and Shopify Plus; headless via Hydrogen.
- Data location: region not published; SOC 2 and GDPR programs “in the process”.
- Entry price: $19 a month up to 200 orders; 14-day free trial (checked October 2026).
- Main strength: testing, personalisation, recommendations and upsells in one app.
- Main limitation: client-side changes with no documented server-side option.
Documented AI: the Copilot editor, a component generator, AI recommendations and an MCP server for reporting questions. Not the right choice for buyers who need a finished SOC 2 report. Head to head: Apex vs Visually.
Elevate: best for price and shipping tests at a low entry price
- Best for: Shopify stores that want price, shipping and checkout tests on a small budget.
- Can test: pages, themes, prices, shipping, offers, product images, checkout components (Plus).
- Shop systems: Shopify; headless via an SDK.
- Data location: hosting region not published; DPA and certifications: none found.
- Entry price: Basic $49 a month; AI suggestions from Standard at $99 (checked October 2026).
- Main strength: price, shipping and checkout tests from $99 a month.
- Main limitation: no published hosting region, DPA or security certification.
Documented AI: test suggestions, image generation and a read-only MCP server. Not the right choice for buyers who need a signed DPA before purchase. Full profile in our comparison of 15 tools.
Varify.io: best for AI-built tests at a flat price on any shop system
- Best for: teams that want prompt-built tests at a flat price per domain.
- Can test: product, collection, cart, home and landing pages; Shopify Plus checkout via a separate app.
- Shop systems: any shop that loads JavaScript.
- Data location: AWS in Frankfurt plus Cloudflare.
- Entry price: Growth €199 a month billed annually; AI and MCP from Pro at €249, credits extra (checked October 2026).
- Main strength: flat price per domain with unlimited traffic.
- Main limitation: default statistics are one-sided frequentist tests.
Documented AI: CRO AI Audit, prompt-based variants, bring your own key, an MCP server and AI experiment analysis, all shipped in 2026. Not the right choice for teams that need sequential testing or bandits, both marked “SOON”. Head to head: Apex vs Varify.
Full profiles of the other tools named above: Apex vs ABlyft, Apex vs Convert, Apex vs ABConvert, Apex vs GrowthBook and Apex vs Shopify Rollouts.
11How Should You Choose an AI A/B Testing Tool?
Questions to put to every vendor:
- What data sits behind the idea ranking, and how many observed tests?
- Which test types can the AI build, and which not?
- Who approves before a test reaches shoppers?
- Can the ChatGPT or Claude connection change tests, or just read them?
- Which statistics sit under the AI summary?
- Is AI included, billed in credits, or run on your own API key?
A shop whose tests keep coming back flat has a different constraint: choosing which tests to run. That is where Apex by DRIP fits, with a Likelihood to Win for every test idea, the testing tool and managed execution in one place.
