Should a retail web-site always return the same items for identical searches made by different users?

Question

We supply a large number of products for purchase through our web-site. There is a new initiative to apply a third-party AI product to hijack searches to return products based on both the search term and predictions from browsing history and other people's search history with successfully processed sales. If I now search for a keyword, I will get a different set of products returned than if someone else searches for the same thing. If I pass an URL of my search to a friend to compare the products, we will have different lists, so cannot discuss. The list, I find, changes day-to-day, on my own machine due to the AI's suggestions.

Is this a design "no-no"?

Should the AI be solely used for recommendations and not for the core search results?

Is there any guideline to cite that makes suggestions on this?

I have also put this on: Software Recommendations

Answer 1

There are 2 types of search: finding a specific answer or item or exploratory search i.e. discover the space of possibles.

For the 1st type of search, where there is only one good answer, results shouldn't change based on who the user is and what they did. This applies when you're looking for a specific information (the actors in a movie, the date of a battle, etc).

The 2nd type of search is most common on e-commerce: you have a rough idea of what you want and you search to know the different products that match your criteria. From your description, it seems your website search is closer to this 2nd type of search.

In this case, having search results tailored to a specific user based on their history and other users' actions will likely deliver more relevant results on top and thus increase conversion. Google already refines their results based on user location, past search history and activity. On Amazon, the list of results is different depending on whether you're logged in or not, because they take into account your past browsing and buying history and wish list.

You raise 2 issues with AI-tailored search results:

1- a user can't simply share a search url and expect their friend to see the same results.

2- a user might not find the same products in their next search.

You could mitigate these issues by providing a favorite / list system. Users could add one or multiple products to a list (buttons such as "save for later", "favorite", etc) and share the list with their friends.

This introduces more complexity, but you will obtain valuable information from the list users compile (with the ability to refine future recommendations based on listed items or even send tailored mailing that include listed products).

The UI element to save an item will serve as a signal to the user that this list is temporary and they should save an item if they want to find it again later.

Answer 2

I think the idea of custom search results awesome. But you hardly avoid if the user can't back to your site and make the same search and have the same result.

You need to think about how your user we feel when this happens. Do you think the user will think:

• "Oh, maybe they change the search logic, let me try another keyword alternative" OR
• "Oh no, they don't have the product anymore, let me Google it to find in another place"

Other points to thinking about:

1. How frequently your users share the result of a search using a link? If they doing it, why?
2. How do you can use this AI to improve the results equal for all?
3. Maybe you could use it to show on top of the common search results some suggested products.

Anyway, my vote is:

YES, the search result link should keep the same product if your user shares the link.

Answer 3

Just a keyword search is not enough unless it is a long and very specific keyword. Filters will be required to keep the search results in order. In addition to all these, the ability to sort them in a specific order. The search engine will need to be designed to consider and prioritise specific parameters from the index in displaying results.

Considering that the indexing time is 4 hours, the same search keyword and filters should not display different results within that period as the data-set remains unchanged. If the results are varying within that period, then the search just needs to be more rigid.

AI results will generally be different for two different people since it is based on individual activities. Search results are not expected to be overwritten by AI suggestions.

Answer 4

Keep in mind that the way you train the AI and configure the parameters for the search algorithm is still based on user requirements, so there is definitely nothing to say what the search result should look like - this should be based on what the user expectations are.

If the expectation of the user is that the 'search pool' is constantly changing, then for a given keyword the results should always return the latest set of results (e.g. if you add new products). This is the simplest case.

If the expectation of the user is that the search algorithm collects some information and use it to refine the search result, then for a given keyword the results can vary even if the 'search pool' remains the same. This is also a relatively simple case.

However, if you combine the two possibilities then you end up with a number of different permutations depending on what the user expectation is for each aspect of the search process. Without understanding more about the existing design or the users, I would say that you should take a conservative approach and not assume what the user expectations might be. It will be easier to enhance features compared to designing something complex and having to scale it back.

Answer 5

You will never always return the same results

This is important to remember. For various reasons, the SRP for any given terms is usually a slowly changing experience. The merchandise and inventory change, so the results should change. Maybe not today or tomorrow. But at some point, things are dropped and things are added.

So, if it's not an immutable list of constants, what level of change will your users tolerate. Chances are, more than you think.

The age of data science and machine learning

I tested an algorithmic SRP ranker nearly 10 years ago. It was pretty early for data science applications in mid-level e-comm applications, but the human intervention model was too "emotional". If we had a big day with lots of data (a few hundred thousand hits), the model might return something new the next day. It worked. Conversion rates saw a double digit lift.

10 years on, ML is everywhere. Digital products respond to people's needs. If the model is good (and remember, humans do write these models), then it will get things right more than it gets things wrong.

Users now expect their digital world to respond to their behavior and needs.

Incremental improvement

On the topic of search ranking methods, this data science article makes a keen observation:

When you reach a scale where incremental gains in relevance are more important than the rest of the factors, consider moving to an ML model. But do make sure you have a good answer to the “Optimization Metric” problem before you start working on a ML model.

The operative phrase there is "incremental gains". You need a lot of data coupled with a well-tuned model to slowly improve the performance (engagement, conversion, average order, etc). IOW, the results are not erratic.

Too big for human orchestration

And just to further emphasize a critical point with AI / ML: you are operating on massive data sets. As pointed out in this Forbes article, man + machine can do more.

… machine learning provides significant value by processing vast quantities of data to find patterns, drawing conclusions no human is capable of and enabling incredibly precise, data-driven activations for marketers.

Bottom line

If you have a large catalog, you probably have a lot of users, and even more data. ML-driven SRPs are likely to prove fruitful.

Answer 6

There is no definite yes or no answer to the question.

It depends on several factors, the same site can have very precise search and algorithm (AI/ML) assisted search.

Real world examples:

• When I search for an "Audi Q3" keyword in a car sale portal, I would expect to see results that contain ONLY Audi Q3s, sorted preferably by price or any other sorting criteria that I can filter. But when I search for "SUV" or "4X4", then the algorithm can assist the search.

• When I search for "expanded cork insulation" I should see exactly that, it's a product that you sell, you either have it, or not.

• The algorithm can assist when the search returns no results, or when results may contain too many irrelevant items. Algorithms would work well with abstract keywords such as "gift ideas" in a large marketplace (Etcy or eBay).