Introduction
The Bee’s Knees British Imports, founded in 2011 by English-born Donna and Lucinda, began as a shared appreciation of British heritage. Over time, it has grown from an online business to a beloved shop in West Acton, Massachusetts. Joined by fellow Brit Trish in 2019, the business thrived during the pandemic, leading to the opening of a larger storefront. The Bee’s Knees was awarded Best of Boston® Best Shop for Anglophiles in 2023 by Boston Magazine, all while providing top-quality British-made products and exceptional customer service.
In addition to their physical storefront, they operate a Shopify ecommerce store that caters to buyers around the world. Earlier this year, their Shopify agency, 36creative, approached them with an opportunity to test Google Cloud’s Vertex AI using Nimstrata’s Retail Cloud Connect app after seeing success with other clients like LOZURI.
Lucinda and Trish were excited about the opportunity to try this new technology and leveraged free trials of Google Cloud, Nimstrata’s app, and Shoplift to A/B test Google’s search algorithms on their website.
Implementation
Before this test, The Bee’s Knees used Boost AI for merchandising search and collection pages. To ensure a successful A/B test and implementation, the new solution needed to integrate seamlessly with the existing data format and customizations that The Bee’s Knees team had established to create their desired customer shopping experience.
Filter Trees
Considerable effort had been invested in creating filter trees using Shopify tags to filter specific brands with tag prefixes. For instance, each “Emma Ball” branded item has a tag such as “Emma Ball/Animals” to generate a corresponding filter option.
Nimstrata collaborated with The Bee’s Knees to import their existing Boost configuration into Nimstrata’s app settings, ensuring an identical shopping experience.
Dynamic Facets
Once facet and filter data is configured, Google Cloud’s Vertex AI Search for Retail can ensure that the most relevant filters are displayed based on the user’s search query, intent, and available inventory. By dynamically updating based on search results, dynamic facets prevent users from encountering dead-ends that yield no results.
App Block Styling
To run a fair and accurate A/B test comparing Boost with Vertex AI, Nimstrata prioritized creating a consistent shopping experience for users on both product discovery platforms. Achieving this consistency was crucial to ensure that any observed differences in performance could be attributed solely to the AI platforms themselves, rather than discrepancies in the user interface or shopping experience.
To this end, Nimstrata customized their default App Blocks using custom CSS to match the existing Boost AI experience as closely as possible. This meticulous customization effort was designed to eliminate any visual or functional differences that could potentially influence user behavior and, consequently, conversion rates.
As recommended by Shoplift, the emphasis on maintaining a uniform user experience highlights the importance of careful test design in A/B testing. Ensuring that both test variations are nearly identical, except for the variable being tested—in this case, the AI platform—allows for a more precise measurement of each platform's true impact on conversion rates.
The Initial A/B Test
Both Search and Collection Pages needed to be tested in tandem, meaning an A/B testing tool that could provide theme vs. theme testing like Shoplift, was required. The test was conducted over five weeks, splitting traffic evenly across all devices.
Google Cloud Vertex AI Search for Retail demonstrated a significant improvement in conversion rates. Shoplift measured a 20.58% increase in conversion rates with Vertex AI compared to Boost AI. Furthermore, the statistical analysis powered by Shoplift revealed a 99% confidence level, confirming the dramatic lift that Vertex AI provides to drive more conversions.
Testing Recommendations
After successfully completing the search and collections test, the team advanced to testing Vertex AI Recommendations on their product pages. They chose an "Others You May Like" model, optimized for click-through rates using the available user event training data.
In just nine days, they confirmed a significant 22.47% increase in conversion rates with a 95% confidence level, demonstrating the near-immediate value of integrating Google Cloud's AI technology into a Shopify store.
Conclusion
The Bee’s Knees British Imports’ journey with Google Cloud’s Vertex AI and Nimstrata’s Retail Cloud Connect app has been a resounding success. By leveraging advanced AI technology, they not only improved their search and collection pages but also significantly enhanced their product recommendations in days.
These results underscore the transformative impact of integrating AI technologies into ecommerce platforms alongside robust A/B testing. The Bee’s Knees has not only maintained its reputation for top-quality British-made products and exceptional customer service, but has also embraced cutting-edge technology to enhance their online shopping experience. This case study exemplifies how traditional businesses on Shopify can thrive in the digital age by adopting innovative solutions to meet and exceed customer expectations.
