# A/B Testing on Shopify

# Introduction

Implementing any new technology on Shopify should have a positive return on investment (ROI) and justifiable total cost of ownership (TCO). For product discovery solutions in particular, this validation often comes through A/B testing themes against each other, but not all metrics are created equal when measuring the impact of product ranking and relevance.

# Key Metrics

# Revenue Per Visitor (RPV)

Revenue Per Visitor (RPV) measures the average amount of revenue generated for every unique visitor to your Shopify storefront. It directly reflects the overall monetary effectiveness of your traffic and user experience.

# Conversion Rate (CVR)

Conversion Rate (CVR) is the percentage of visitors who complete a purchase. It indicates how effectively your site persuades visitors to achieve this goal.

# Add-to-Cart Rate (ATC)

Add-to-Cart Rate (ATC) represents the percentage of visitors who add at least one item to their shopping cart during their session. This metric signals product interest and is a key step towards a potential conversion.

# Understanding These Metrics

While conversion rate (CVR) indicates if more visitors are making a purchase, revenue per visitor (RPV) is more important for product discovery A/B tests because it reflects the total value generated. Add-to-Cart rates can help indicate higher search relevance for specific queries or collection pages.

Product discovery solutions can increase CVR by guiding users to cheaper or less relevant items, which could actually decrease overall revenue. RPV, however, directly measures whether the changes are leading to higher-value engagement and purchases, providing a truer picture of business impact and profitability.

# A/B Testing Shopify Themes

When evaluating Vertex AI Search for Commerce, you can choose between two A/B testing approaches:

Theme-Level A/B Testing: Your A/B test provider compares all traffic and conversion metrics across two complete Shopify themes (Theme A with an existing solution vs. Theme B with Vertex AI on Search and Collection pages). This measures holistic theme performance, including direct product page visits, purchases, and all user journeys.

Targeted A/B Testing: Your A/B test focuses specifically on visitors who interact with Search and Collection pages where Vertex AI Search for Commerce is active.

# Theme-Level Test Challenges

While theme-level testing provides a comprehensive view, it makes isolating the specific impact of Vertex AI Search for Commerce difficult. For example:

• If Theme B shows higher conversion rates, this could be due to increased traffic from external sources (Google Shopping, social media, email campaigns, etc.) where users land directly on product pages—completely bypassing Search and Collection pages where Vertex AI operates.
• Conversely, if Theme B underperforms, the issue might be unrelated to search quality (e.g., a checkout bug, poor mobile experience).

Our Recommendation: For the most accurate measurement of Vertex AI Search for Commerce's impact, use an A/B testing partner that can segment results by user journey and specifically measure visitors who engage with Search or Collection pages. If this is not possible with your existing A/B testing tools, you can also compare the metrics offered by your existing search tools alongside the Analytics page in the Vertex AI Search for Commerce console.

# Advanced Measurement

If your existing product discovery solution can send User Events tagged with an experiment ID to Nimstrata's Storefront API, you can use the native Vertex AI Search for Commerce experiments dashboard inside of Google Cloud. These events can also be sent prior to A/B testing to accelerate training the AI models. Please contact us if you would like to take advantage of more advanced measurement capabilities.

# A/B Testing Partners

We've partnered with several leading A/B testing solutions in the Shopify ecosystem to help you measure the impact of Vertex AI Search for Commerce on your storefront using a neutral third party.

# Shoplift

# Intelligems

# Visually

# Convert