5 ways to increase result diversity at web-scale

This past Thursday, December 16th Daniel Wärnå and I gave a concluding talk of the 2021 at Haystack LIVE! Event organized by OpenSource Connections.

Photo by Daria Nepriakhina on Unsplash

We discussed the topic of result diversity at web-scale, starting with a discussion on what is diversity in search and when it may or may not be suitable.

I’d like to personally thank Daniel Tunkelang for providing feedback on our slides and for publishing https://dtunkelang.medium.com/thoughts-on-search-result-diversity-1df54cb5bf4a where he shares a few interesting points on how to go about reasoning on this very important topic. Also, big thanks to Aditya Jitta for generous help with slides and demo code. The idea of this work was inspired by Eric Pugh and his presentation on measuring diversity during Berlin Buzzwords 2021.

In a nutshell — precision of search can counterintuitively be traded for more result uncertainty — or diversity — that will give a larger perspective over products, documents, books — what have you in your online search engine. This in turn can increase engagement and average check by 17% (in e-commerce) or task accomplishment (in enterprise search).

Here is the recording of the presentation:

And here you will find slides:

Thanks to Charlie Hull for hosting us!