| #1 Apex by DRIP | MixedFrequentist engine with deliberately conservative defaults: fixed-horizon testing with Holm-Bonferroni correction and a sample-ratio-mismatch gate, plus optional always-valid (mSPRT) sequential analysis. Revenue significance is intentionally held back after internal certification measured 20%+ false-positive rates on naive revenue tests. On top: win-rate prediction from the 4.3M-test corpus lifts the average program win rate from 27% to 55%. | Shopify appShopify app that passed Shopify’s review (installs via OAuth from the dashboard; not App-Store-listed yet), plus WooCommerce, Shopware, Centra and custom storefronts. Unique in this index: real charged-price tests via Shopify Cart Transform, shipping-cost tests and post-purchase experiments. | UK/IE | Cookies or local storageUK (London) and Ireland; DPA relies on the UK adequacy decision. Cookie and localStorage visitor IDs last 365 days. Apex uses no fingerprinting, never stores visitor IPs, integrates with 8 consent managers and does not support IAB TCF.program managed in Drata, SOC 2 / ISO 27001 audit package prepared, audit not yet begun | Only inside DRIP. Book a call |
|---|
| #2 Convert Experiences | MixedThe deepest stats in this comparison: t-tests with configurable tails, Bonferroni/Sidak corrections, Bayesian with explicit risk thresholds, and real sequential testing via Asymptotic Confidence Sequences (Pro+). SRM check on every plan. | Shopify appNative Shopify app (free to install, launched Oct 2024, still 0 reviews) that injects the script without theme edits and is compatible with checkout extensibility, for tracking, not checkout-UI editing. Price testing and multi-arm bandits included. | EU | UnverifiedEU-hosted on AWS (security page: eu-west-1 Ireland; AI info page: Frankfurt, Convert’s own pages disagree). SOC 2 attested and ISO 27001 certified, free DPA. No documented cookieless delivery mode or Google Consent Mode integration. | Growth $399/mo ($299 paid annually), Pro $599/mo ($420 annually), both capped at 100K tested users/month; overage from $399 per 100K. USD, tax-exclusive. 15-day free trial, no credit card. |
|---|
| #3 Varify.io | MixedOne-sided frequentist by default (all plans); two-sided tests and Bayesian with runtime forecast are Pro-only. No sequential/always-valid inference and no in-product sample-size planner. | SnippetStorefront testing via snippet (no app needed); Shopify order/revenue import into the dashboard. Checkout testing needs the separate "Varify: Checkout A/B Testing" app, $119/month extra and Shopify Plus only. | DE | Cookieless availableGerman company, AWS hosting in Germany (Frankfurt). No cookies by default (local/sessionStorage), no personal data beyond IP, DPA with TOMs, documented Google Consent Mode V1/V2 setups. Not SOC 2 certified (roadmap: 54%). | Growth €249/mo (€199 paid annually), Pro €349/mo (€249 annually), Enterprise from €499/mo, unlimited tracked users on every plan, but only 1 domain. 30-day free trial, no credit card. |
|---|
| #4 Kameleoon | MixedFrequentist, Bayesian AND sequential testing side by side, plus CUPED, multiple-testing correction and SRM detection, all included at the entry tier. Caution: the reported "reliability" metric is 1 − p-value, not the probability the variation wins. | Shopify appShopify app carries the "Built for Shopify" badge but has 0 reviews after 4+ years (listed June 2021). Checkout testing needs Shopify Plus plus a developer-built Checkout UI extension and cart-attribute plumbing, not point-and-click. | EU | Cookies or local storageEuropean infrastructure without US hyperscalers: subprocessors are OVHcloud (France) and Hetzner (Germany); EU and North America data regions exist (EU residency is a choice, not automatic). Four documented consent modes, Delayed Mode renders experiments while writing and transmitting nothing until consent. ISO 27001:2022, SOC 2 Type 2, PCI DSS. | PBX Starter from $495/month (10 experiments, 50,000 tracked users), the only enterprise-class tool here with a public entry price. Enterprise is custom. 30-day free trial (3 experiments, no credit card). Feature management, personalization and the dedicated code editor are Enterprise-only. |
|---|
| #5 VWO | SequentialBayesian engine with sequential analysis that handles peeking and multiple comparisons; MDE, ROPE, false-positive rate and power are user-settable; automatic SRM, outlier and Simpson’s-paradox checks. Note: "SmartStats" is now legacy branding for non-Data360 accounts. | Shopify appShopify app (free to install, launched Mar 2024), 0 reviews after ~2.5 years. It deploys tracking everywhere except checkout, post-purchase and order-status pages: VWO cannot A/B test the Shopify checkout. Available from the Growth plan. | EU | Cookies or local storageReal EU data-residency option: customers can pin visitor data to GCP europe-west1 (Belgium); US and India regions also exist. Three documented consent modes (Completely/Partially/Do Not Block), DPA with SCCs, ISO 27001:2022, SOC 2 Type II. | No published prices, plans (Growth/Pro/Enterprise) are quoted per module and billed on Monthly Tracked Users. 30-day free trial. Procurement data (Vendr, 102 purchases) puts the median annual contract at ~$16,300 (range ~$7,800–$35,800). |
|---|
| #6 ABlyft | MixedFrequentist and Bayesian, selectable per experiment, with a configurable certainty level (50–99.9%) and a minimum-conversions threshold. No sequential testing and no sample-size calculator. | No appNo Shopify App Store app. Integration is snippet plus documented tracking wiring: a server-side order webhook (immune to ad blockers and missing consent) and/or the Customer Event Pixel. No checkout-UI testing. | DE | Cookieless availableThe strictest privacy posture in this comparison: no personal data stored at all, no IP, no user ID. German hosting, localStorage instead of cookies, self-hostable snippet, bring-your-own-tracker option. No SOC 2 or ISO 27001 claimed. | A genuine free plan (no credit card, no sales call) plus a quoted paid plan billed by tested users. No public numbers. |
|---|
| #7 Optimizely Web Experimentation | SequentialStats Engine: frequentist sequential testing with always-valid inference (mSPRT) and tiered Benjamini-Hochberg false-discovery-rate control, academically published and genuinely differentiated. Fixed-horizon and Bayesian options exist in beta (documented for Feature Experimentation). | No appNo first-party Shopify app for Web Experimentation, install is a manual theme.liquid snippet edit or a $29/month third-party app. Optimizely’s own docs: the snippet does not load on Shopify checkout pages; checkout events need a custom pixel. Revenue tracking leans on the separate paid ODP product. | EU | Cookies or local storageEU Data Hosting offered, but Optimizely states "some services aren’t in scope" without listing them, whether Web Experimentation is covered is unverified. Strong cert list (ISO 27001:2022, SOC 2 Type 2, PCI DSS, TISAX), public DPA with SCCs. Five consent options; no cookieless mode for Web Experimentation. | "Every Optimizely plan is individually packaged", no prices, no trial, no free tier for Web Experimentation. Procurement data (Vendr, 126 purchases) reports a median annual contract of ~$81,000; third-party estimates put entry around $36–40k/year. |
|---|
| #8 AB Tasty | BayesianWell-documented Bayesian engine (beta-binomial priors, ~1M Monte Carlo samples, automatic Bonferroni) for rate metrics; Mann-Whitney U test for revenue metrics. Sequential Testing Alerts are loser-detection only, AB Tasty itself advises against calling winners early with them. | Shopify appShopify app (free to install, launched May 2024): 4.9/5 from only 4 reviews, a thin footprint for an enterprise vendor. Whether the Shopify checkout can be tested is not documented anywhere public. | EU | Cookies or local storageFirst-party cookies only (13-month lifetime), visitor IPs "immediately deleted", three consent modes incl. Didomi support, DPA via dpo@abtasty.com. Runs on GCP/AWS with region-follows-client but no named EU region. Holds its own ISO 27001; SOC 2 is attributed to its cloud providers, not itself. | "No fixed plans. Just a custom proposal." Annual contracts, no self-serve trial (a 1–2 week proof of concept instead). Procurement data (Vendr) puts the median around $66,500/year. |
|---|
