Session: Evaluation Embedding Based Retrieval for eCommerce
In the realm of e-commerce, search engines play a crucial role in connecting buyers with sellers, facilitating purchases through the retrieval of relevant items. Traditionally, search engines employed Token-based matching like BM25, but the advent of Transformers, particularly Sentence-Transformers, revolutionised machine learning tasks by enabling semantic matching between queries and listings. This ushered in Embedding Based Retrieval (EBR), allowing for the addition of semantically relevant listings to recall. The challenge lies in evaluating the performance of these embeddings, given the vastness of data in commercial e-commerce search engines. We present an embedding-based evaluation framework (E2val) to assess any embedding-based model against baseline/token-based retrieval, addressing the need for efficient evaluation without indexing all embeddings to production.
The talk outlines a two-fold approach to offline evaluation, recognizing the impracticality of evaluating the entire recall horizontally. To ensure comprehensive evaluation, the choice of query set is emphasised, particularly for queries likely to yield null or low results. The defined metrics for evaluation include Recall @k, Category Congruence @k, Entity Congruence @k and choice of querysets. These metrics collectively establish an internal benchmark for evaluating use-case specific embeddings in the e-commerce landscape, offering a means to compare various EBR models and streamline the selection process for A/B testing, ultimately optimising time and resource allocation.
Bio
With over a decade of experience in the tech industry, Shubhangi Tandon has established herself as a leader in applied research, machine learning, and software development. Currently serving as an Applied Researcher at eBay, Shubhangi has been instrumental in enhancing eBay's search capabilities, notably spearheading the development of the "Window Shopping" inspired search feature. Her expertise encompasses building ML models for visual search, defining inventory quality, and pioneering methods to improve search result diversity.
Prior to her tenure at eBay, Shubhangi was deeply immersed in natural language generation at the University of California, Santa Cruz's Natural Language and Dialogue Systems Lab. Here, she tackled end-to-end language generation challenges, utilizing Seq2Seq frameworks, dual RNN encoders, and innovative techniques to produce stylistically varied synthetic data.
In addition to her research endeavors, Shubhangi has showcased her pedagogical skills at UC Santa Cruz, managing and mentoring a class of over 400 students, with a curriculum centered on algorithms, data types, and Python programming. Her internships at VMware and her significant contributions at Goldman Sachs in the realms of communication compliance and anti-money laundering technology further underscore her versatility and commitment to technological advancement.