Building Real-time Product Recommendations with Generative AI

The journey to implementing artificial intelligence and machine learning solutions in retail (and many industries, for that matter) requires solving a lot of common challenges that routinely crop up in digital systems: updating legacy systems, eliminating batch processes, and using innovative technologies grounded in AI/ML to improve the customer experience in ways that seemed like science fiction just a few years ago.

To illustrate this evolution, let’s follow a hypothetical contractor who was hired to help implement AI/ML solutions at a big-box retailer.

The problem

First day at BigBoxCo on the “Infrastructure” team. After working through the obligatory human resources activities, I received my contractor badge and made my way over to my new workspace. After meeting the team, I was told that we have a meeting with the “Recommendations” team this morning. My system access isn’t quite working yet, so hopefully IT will get that squared away while we’re in the meeting.

In the meeting room, it’s just a few of us: my manager and two other engineers from my new team, and one engineer from the Recommendations team. We start off with some introductions, and then move on to discuss an issue from the week prior. Evidently, there was some kind of overnight batch failure last week, and they’re still feeling the effects of that.

It seems like the current product recommendations are driven by data collected from customer orders. With each order, there’s a new association between the products ordered, which is recorded. When customers view product pages, they can get recommendations based on how many other customers bought the current product alongside different products.

The product recommendations are served to users on bigboxco.com via a microservice layer in the cloud. The microservice layer uses a local (cloud) data center deployment of Apache Cassandra® to serve up the results.

How the results are collected and served, though, is a different story altogether. Essentially, the results of associations between products (purchased together) are compiled during a MapReduce job. This is the batch process that failed last week. While this batch process has never been fast, it has become slower and more brittle over time. In fact, sometimes the process takes two or even three days to run.

Improving the experience

After the meeting, I check my computer and it looks like I can finally log in. As I’m looking around, our principal engineer (PE) comes by and introduces himself. I tell him about the meeting with the Recommendations team, and he gives me a little more of the history behind the Recommendation service.

It sounds like that batch process has been in place for about 10 years. The engineer who designed it has moved on, not many people in the organization really understand it, and nobody wants to touch it.

The other problem, I begin to explain, is that the dataset driving each recommendation is almost always a couple of days old. While this might not be a big deal in the grand scheme of things, if the recommendation data could be made to be more up to date, it would benefit the short-term promotions that marketing runs.

He nods in agreement and says that he’s definitely open to suggestions on how to improve the system.

Using generative AI to do the heavy lifting

So how can we build a real-time solution to this problem? Let’s start by thinking in hypothetical terms.

What if we took a GenAI approach, and had vector embeddings generated for all of our product titles and descriptions? That would enable us to identify similar products in real-time by using one product’s vector to perform an approximate nearest neighbor (ANN) operation. We would then have a list of similar products, based on the original product. With this functionality, we could at the very least offer some quick product recommendations.

But what about product recommendations based on customer purchases?

That’s a little tricker, but certainly not impossible. We could go through a customer’s product order history, and create a list of products that they have already purchased. We could also weight or influence the order of that product list to determine relevance to the customer. For example, we could take into account how many times a product was ordered, how much total quantity was ordered, or the price of the product (maybe we want to drive sales to more expensive products). Either approach would yield a list of products ordered by their relevance to that particular customer.

Let’s say we take the top three most relevant products to that customer. We could then run an ANN on each of those 3 products with a limit of 4, and build a total list of 12 products that we could use. These products (or some subset of them) would then be shown in the “you might also like” section of BigBoxCo’s website for product recommendations. These recommendations would also be driven by each customer’s unique ordering history.

Implementation

First, we’ll need to iterate through our entire product data set and generate vector embeddings based on the product’s title and description. We can then add that vector embedding to the existing product table. Fortunately, Cassandra has a native vector data type and a vector search library, so we do not need to find a vector database. Although we should probably check to make sure that it’s running on Cassandra version 5, and plan an upgrade to version 5 if it is not.

To generate the vector embeddings for our products, we will use an open source model. Specifically, we will use Hugging Face’s “all-MiniLM-L6-v2” sentence transformer. This model uses 384 dimensional vectors, which is large enough for us to do some basic similarity without sacrificing performance. We can adjust our products table to add a new, vector-enabled column like this:

ALTER TABLE bigboxco.products
ADD product_vector vector<float, 384>;

We’ll also need a storage attached index (SAI) to enable vector searching. So let’s create that index, too:

CREATE CUSTOM INDEX ON bigboxco.products(product_vector)
USING 'StorageAttachedIndex'
WITH OPTIONS = {'similarity_function': 'cosine'};

With our new vector column and index in place, we can now move on to generating vector embeddings for our products. For this step, rather than iterating against the database live, it’s probably easier to get a CSV (or other text-based export) from the products table of product_id, name, and description for all of our products. We can then build a batch job to iterate through the file, generate embeddings based on the name and description, and write them into our Cassandra cluster.

Note: A similar batch process can be found here: https://github.com/aar0np/TestHFEmbeddings

In our data access layer, our team still has some older CQL-based code that’s used with the DataStax Java Driver. This is not a problem, as we can build a simple method to execute our vector search query with a CQL prepared statement:

private PreparedStatement qvPrepared = session.prepare(
        "SELECT product_id, name, product_group, images, "
        + "product_vector, parent_id, category_id " 
        + "FROM product_vectors ORDER BY product_vector ANN OF ? LIMIT 3;");

public List<ProductVector> getProductsByANN(ProductVector originalProduct) {
    List<ProductVector> returnVal = new ArrayList<>();
    BoundStatement qvBound =
            qvPrepared.bind(originalProduct.getProductVector());
    ResultSet rsV = session.execute(qvBound);
    List<Row> ann = rsV.all();

    if (ann.size() > 1) {
        for (Row promo : ann) {
            ProductVector annPromoProd = new ProductVector(
                    promo.getString("product_id"),
                    promo.getString("name"),
                    promo.getString("product_group"),
                    promo.getSet("images", String.class),
                    promo.getObject("product_vector"),
                    promo.getUuid("parent_id"),
                    promo.getUuid("category_id"));
            returnVal.add(annPromoProd);
        }
    }

    return returnVal;
}

Next, we can then adjust our recommended products service method to build a list of the 3 most-relevant products for our customer (from their order history). With those 3 products, we can then call our above getProductsByANN() method with each of them, and build a list of products to suggest. By driving the recommendation with relevant products from the customer’s own order history, this new recommendation method can consistently respond with data that is up-to-date.

Conclusions and next steps

It’s worth noting that the first result included with each vector search will be the product that was used to initiate the search, scored with a 100% match. We will want to increase our CQL LIMIT clause to 4, and always skip the top result. Some additional heuristic filters may be necessary to add as well, like a “do not recommend” list. This is because there are some products that our customers will buy either once or infrequently, and recommending them will only take space away from other products that they are much more likely to buy on impulse. For example, recommending a purchase of something from our appliance division such as a washing machine is not likely to yield an “impulse buy.”

Fortunately, we did come up with a way to augment the existing, sluggish batch process using GenAI with vector search. Once we get a feel for how this system performs in the long run, we should consider shutting down the legacy batch process. The PE acknowledged that we made good progress with the new solution, and, better yet, that we have also begun to lay the groundwork to eliminate some technical debt. In the end, everyone feels good about that.

Learn more about GenAI and retail/ecommerce.