How to Evaluate your E-Commerce Product Recommendations

Product recommendations play a very impactful role in the product discovery path of customers. Strong and relevant product recommendations do not only increase sales but also build strong customer relationship by giving them a sense of being understood and properly served.
New-age product recommendations are powered by different machine learning techniques like collaborative filtering, content-based filtering, and computer vision technology. Today retailers don’t want to lag behind in the cut-throat competition and are happily embracing these AI-powered recommendation systems into their online stores.
This serious shift into using AI systems is also because of the invent of cloud-based infrastructure and standalone systems. Retailers don’t need to invest into buying heavy processors and hiring machine learning industry experts to generate product recommendations. They just need to spend few cents to get intelligent recommendations and insights from third party recommendation providers. As a result, almost all the big brands and retailers provide product recommendations in their online store generated either by an in-house team or from a third party vendor.

Why measure ROI from product recommendations

Now, as a significant amount of investment is being done in product recommendations by retailers, it’s equally important to measure the return on this investment. There are many e-commerce metrics to be tracked for measuring conversions in an online store. In this blog, I am going to discuss some important metrics to measure the quantitative benefits of your store recommendations and track the ROI.
Technical folks use metrics like recall, accuracy, and precision to measure a machine learning system. But how these technical metrics maps to the business value of these systems. In the case of recommendation engines, you can use the business metrics discussed in this blog.

What is a metric and why it’s important with Product Recommendations

Before we deep dive into metrics, first let’s defines what is a metric and how important it is for any system.
A metric is a measuring system that quantifies a trend, a characteristics or a specific feature. For using any metric, two things are equally important. First is clearly defining a metric and second is measuring it properly to get the insights. Another important thing to consider is the time period. Any metric is measured within a pre-defined time line. Here are 4 ways to measure product recommendations:

4 Measuring Recommendations

To define recommendations metrics, we should first decide what are the things that affect the efficacy of any recommendation. On a high level, things that matter most are the location where recommendations are placed, how end users are interacting with them and how much sales is being driven by recommendations.
Thus, we can evaluate recommendations on impact, interaction, and conversion. Please note all these metrics are calculated for a specific period of time.

1. Impact

We often see that there are multiple places where recommendations are shown in an e-commerce store like the landing page, product page, and checkout page etc. The number of different places showing recommendations also affects their effectiveness.
The number of locations displaying recommendations is the number of different recommendation lists we are showing at these locations. These locations give you an idea of coverage from these lists of the e-commerce store. This is an important metric which can answer many things and can be linked with some other interaction or conversion metric to derive some actionable insights like which location performs well and how many lists should be used for desired results. We can do A/B testing to test put different scenarios.
Another important stat to measure is the total number of recommendations served in an e-commerce store in a specified time period. We should also calculate this number respective to certain locations. This is used in calculating the click through rate.

1. LOC – The number of locations where recommendation lists are being displayed
2. REC – Total number of recommendations served

2. Interaction

Interaction metrics define the level of interest and engagement, users are showing in the recommended items. We can measure the interaction level with the number of clicks in the recommended items. And this number of clicks is used to calculate the popular metric CTR (Click Through Rate).

1. CIR (Clicks in Recommendations) – The total number of clicks in the recommendations/recommended items
2. CTR (Click Through Rate) – CIR/REC (Number of clicks in recommendations/total number of recommendations served)

3. Conversion

This is the most important and relevant group of metrics which measures the conversions of a recommendation engine and defines its true business value.

1. The rate of orders with recommendations:
This metric calculates the proportion of orders which contain at least one item brought from recommendations. For this, we need the number of all the order which contain one or more than one recommended items and the total number of orders of the e-commerce store.

ROR = ORDERS_REC/ORDERS (Number of orders containing recommended items/Total number of orders in the store)

2. Rate of items recommended per order:
This metric calculates the proportion of recommended items in an order. This metric is calculated on an average basis. For this, we take an average number of recommended items in an order and average number of total items in an order.

RIRO = NRIO(Average)/NIO(Average) (Average number of recommended items in an order/ Average number of total items in an order)

3. The increase in average order value (AOV)
This metric tells the increase in average order value of the store through recommendations. A positive result of this metric also proves the fact that recommendations can be very effectively used for upselling and cross-sell.

(AOV of orders containing recommended items – AOV of the store) / AOV of the store

4. The increase in Revenue 
This is the most important metric which tells how much revenue is increased by adding the recommendation engine in the store. This metric can be linked with the number of locations for displaying recommendations to derive the most efficient number of locations. While calculating this metric, we should also keep in mind the seasonality factor of the sales cycle.

((Revenue generated by recommended items) / (Total revenue of the store – Revenue generated by recommended items))

Calculating the metrics:

Let’s take a simple example of an e-commerce store and calculate the metrics. Here are the data from an e-commerce store for a specific period of time.
REV_REC = $ 76000
REV = $ 475000


180000 16000 1500 400 2 5.5 34600 27500


From the above data, we can easily calculate the metrics:
CTR = (CIR/REC)*100 = (16000/180000)*100 = 8.88%
ROR = (ORDERS_REC/ORDERS)*100= (400/1500)*100 = 26.66%
RIRO = (NRIO(Avg)/NIO(Avg))*100 = (2 / 5.5)*100 = 36.66%
IAOV = ((AOV_REC – AOV) / AOV)*100 = ((34600-27500)/27500)*100 = 25.81%
IR = ((REV_REC) / (REV – REV_REC)) *100 = ((76000/(47500-76000))*100 = 19.04%
What these numbers mean:

  • Of the 8.88% recommendations that are presented, a user will click once.
  • From all the orders placed at the store, 26.66% of orders contained at least one recommended item.
  • On an average 36.66% items of an order were bought from recommendations.
  • There is an increase of 25.81% in the average order value of the store after adding recommendations.
  • And lastly, 19.04% of revenue is increased after adding the recommendations in the store.

Now, when we understand the metrics to evaluate recommendation engine on any e-commerce store, the bigger question arises around how to choose the best recommendation partner for your store.

Selecting the perfect recommendation partner:

While selecting the perfect recommendation partner, you should be very careful about the provider’s overall offering. Apart from the above-discussed metrics, you should also consider how flexible the recommendation provider is if you want to use these recommendations at multiple locations both inside and outside the store like newsletters, personalized promoted content, email or push notifications.
The provider should provide features like removing repeated recommendations for a specific user and allowing you to filter the recommendations based on your upsell and cross-sell strategies. You should also ask how much personalized are the provided recommendations and what are the data points being considered while personalizing. These data points should not interfere with your customer privacy policies.
For greater insights, A/B testing based on the discussed metrics works very well. You should give it a try. Always remember, recommendations should improve the UX of your store and help customers in their natural product discovery, not the other way round. I hope you find these metrics useful.

Image credit : 415Agency