Apex vs Building It Yourself With Claude Code: 2026 Comparison for Online Shops

01How does Apex compare with a split test Claude Code builds for you?

Note on Likelihood to Win: it is one percent figure per test idea, shown with how many tests back it. It ranks an idea against your shop’s own backlog. It is not a win probability in the strict sense.

02What does Claude Code do when you ask for A/B testing?

Amplifying asked Claude Code to add common capabilities to four new test projects, three runs each, and recorded what it chose. For feature flags, the building block of every split test, the agent wrote its own system in 55 of 80 picks (68.8 percent), picked LaunchDarkly in 17 (21.3 percent) and PostHog in 3 (3.8 percent) (Amplifying, What Claude Code Actually Chooses, February 2026, checked October 17, 2026).

Amplifying ran Claude Code with Sonnet 4.5, Opus 4.5 and Opus 4.6; newer models may choose differently. The study measured feature flags, not A/B tests for online shops, and we found no published figure for that exact request. The pattern matters anyway: when you ask a coding agent for testing, expect a few files of code that split visitors, not a testing product. That code can be fine. Whether it gives you a correct answer depends on what is around it.

03When is a homemade split test good enough?

• A switch, not a test. Turning a feature on for some visitors and off again if something breaks is what the Amplifying feature flags are for.
• A big, obvious change on a lot of traffic. When the difference is large, a simple comparison shows it, as long as you decided beforehand how long it runs.
• Checking the plumbing. An A/A test with two identical versions tells you whether assignment and tracking are even. If the two groups differ, fix that before any real test.
• Learning how testing works. Building one test yourself teaches you what a testing tool does for you.

04Where does a homemade split test give wrong answers?

1. Sample size. How many visitors a test needs depends on your conversion rate and the smallest effect worth finding. Without that number set before launch, a flat result means nothing. See how to calculate A/B test sample size.
2. Peeking. Looking at the numbers every day and stopping when the variant is ahead turns noise into winners. Fixed-horizon tests are read once, at the planned end. See the peeking problem.
3. Revenue and profit. A shop test is about money. A homemade test usually counts clicks or orders on the page where it runs. Tying each order, discount and refund back to the variant the buyer saw is the hard part, and the one that decides whether a winner pays.
4. Flicker. A test that swaps content in the browser after the page loads can show the original for a moment, then the variant. Visitors notice, and the measurement shifts with them.
5. QA. Each variant has to work on every device, keep tracking intact, respect cookie consent and keep bots and your own team out of the numbers. Uneven groups are a warning sign; see sample ratio mismatch.
6. Test memory. A year later, which tests ran, what they changed and why one won lives in old branches and chat threads. The next idea gets chosen without that history.

05How are winners chosen before launch?

Claude Code can suggest test ideas from your pages and your notes. It has no record of how those ideas ended in other shops. Apex scores each idea against 4.3 million observed A/B tests from 151,000 shops and shows how many tests back each score. In a holdout check within a shop, the top third of ideas won 1.4 to 1.6 times as often (press release, September 29, 2026). DRIP scored and froze predictions for 33 brands in early September 2026 and publishes the first results from November 2026.

06Can Claude Code run Apex instead of building from scratch?

In Claude Code: add the @drip-apex/mcp-server block with your APEX_API_KEY to a project .mcp.json, as the Apex docs show; Cursor uses the same local server. The hosted server at https://app.drip-apex.com/api/mcp, with OAuth sign-in, works as a custom connector on claude.ai, in ChatGPT’s developer mode and in Codex. Connecting Claude Code or Cursor to the hosted server is untested. Developers can also use the apex CLI (npm install -g @drip-apex/apex-cli).

The agent then works inside the shop you approved: it lists tests and reads results, creates a draft test, writes and builds the variant code, opens a preview link and captures QA screenshots, saves an idea to the backlog and reads its Likelihood to Win once scored. Approving QA and launching stay a click in Apex. Setup for every client is on drip-apex.com/agents.

07What does each option cost?

The cost of a homemade test is the time to build the split, the tracking, the revenue attribution and the QA, then to keep all of it working as the shop changes. Apex’s price is not published. A card-free free plan for one store exists; book a call to see whether your shop is a fit.

08When is building it yourself the better choice?

• You need a feature switch or a staged rollout, not a measured test. That is what the agent builds by default.
• You run one large test on a lot of traffic and set the sample size and end date before launch.
• You cannot ship a test a week yet. Build that habit first; buying prediction before you can execute is the wrong order.

09When is Apex the better choice?

• Your tests keep coming back flat. Every test idea gets a Likelihood to Win before launch, backed by the memory of 4.3 million A/B tests from 151,000 shops, so selection improves before anything is built.
• You judge tests on money. Apex reports runtime revenue per test and, on Shopify, profit per test, and Guardian stops a test at a revenue loss threshold.
• You want the agent and the safeguards. Claude Code builds and previews tests in Apex through MCP, while statistics, QA checks, heat maps per variant and the test record come with the tool.

Apex is not the right choice for every shop. A team that already picks winning tests well gets less from the prediction layer, and anyone who needs a published price list before a conversation will not find one.

10Apex or building it yourself: which should you pick?

If you need a feature switch or one big test and you set the sample size before launch, a split test written by Claude Code is enough. If your tests keep coming back flat, the constraint is choosing which tests to run, and that is what Apex’s Likelihood to Win is built for. If you judge tests on revenue and profit, Apex ties both to the variant and stops a test that loses money. Either way the agent stays in the loop: through MCP it builds and previews tests in Apex, and a person launches them.