# A/B Testing on Shopify

## Introduction

Implementing any new technology on Shopify should have a positive return on investment (ROI) and justifiable total cost of ownership (TCO). For product discovery solutions in particular, this validation often comes through A/B testing themes against each other, but not all metrics are created equal when measuring the impact of product ranking and relevance.

<br/>

## Key Metrics

### Revenue Per Visitor (RPV)

Revenue Per Visitor (RPV) measures the average amount of revenue generated for every unique visitor to a Shopify storefront. It directly reflects the overall monetary effectiveness of storefront traffic and shopper experience.

### Conversion Rate (CVR)

Conversion Rate (CVR) is the percentage of visitors who complete a purchase. It indicates how effectively the site persuades visitors to achieve this goal.

### Add-to-Cart Rate (ATC)

Add-to-Cart Rate (ATC) represents the percentage of visitors who add at least one item to their shopping cart during their session. This metric signals product interest and is a key step towards a potential conversion.

### Understanding These Metrics

While conversion rate (CVR) indicates if more visitors are making a purchase, **revenue per visitor (RPV)** is more important for product discovery A/B tests because it reflects the **total value generated**. Add-to-Cart rates can help indicate higher search relevance for specific queries or collection pages.

Product discovery solutions can increase CVR by guiding shoppers to cheaper or less relevant items, which could decrease overall revenue. RPV directly measures whether the changes are leading to higher-value engagement and purchases, providing a clearer picture of business impact and profitability.

!!!
AI Commerce Search focuses on increasing Revenue per Visitor (RPV) as a primary goal.
!!!

<br/>

## A/B Testing Shopify Themes

When evaluating AI Commerce Search, merchants can choose between two A/B testing approaches:

**Theme-Level A/B Testing**: The A/B test provider compares _all traffic and conversion metrics_ across two complete Shopify themes (Theme A with an existing solution vs. Theme B with AI Commerce Search on search and collection pages). This measures holistic theme performance, including direct product page visits, purchases, and all shopper journeys.

**Targeted A/B Testing**: The A/B test focuses specifically on visitors who interact with search and collection pages where AI Commerce Search is active.

### Theme-Level Test Challenges

While theme-level testing provides a comprehensive view, it makes isolating the specific impact of AI Commerce Search difficult. For example:

- If Theme B shows higher conversion rates, this could be due to increased traffic from external sources (Google Shopping, social media, email campaigns, etc.) where shoppers land directly on product pages and bypass search and collection pages where AI Commerce Search operates.
- Conversely, if Theme B underperforms, the issue might be unrelated to search quality (e.g., a checkout bug, poor mobile experience).

**Recommended Approach**: For the most accurate measurement of AI Commerce Search's impact, use an A/B testing partner that can segment results by shopper journey and specifically measure visitors who engage with search or collection pages. If this is not possible with existing A/B testing tools, merchants can compare the metrics offered by existing search tools alongside the Analytics page in the AI Commerce Search console.

### Advanced Measurement

If the existing product discovery solution can send User Events tagged with an **experiment ID** to Nimstrata's Storefront API, merchants can use the native AI Commerce Search experiments dashboard inside Google Cloud. These events can also be sent before A/B testing to accelerate AI model training. [Contact Nimstrata](https://www.nimstrata.com/contact) for advanced measurement support.

<br/>

## A/B Testing Partners

Nimstrata has partnered with several A/B testing solutions in the Shopify ecosystem to help merchants measure the impact of AI Commerce Search on a storefront using a neutral third party.

### Shoplift

[!ref target="blank" text="Get Started with Shoplift"](https://apps.shopify.com/shoplift)

### Intelligems

[!ref target="blank" text="Get Started with Intelligems"](https://apps.shopify.com/intelligems)

### Visually

[!ref target="blank" text="Get Started with Visually"](https://apps.shopify.com/visually-io)

### Convert

[!ref target="blank" text="Get Started with Convert"](https://apps.shopify.com/convert-experiences)