Scaling ML embedding models to serve a billion queries

Scaling embedding models to serve a billion queries Senthilkumar Gopal @sengopal

Search @ eBay How can we discover items without describing them? This is a problem across many domains where search is a core functionality. Question to ponder can we provide users with the ability to “discover” through visual cues instead? 3 © 2022 eBay. All rights reserved. https://unsplash.com/photos/2oUiUu5QAys

So what is an embedding then? Represents Semantic Similarity Similarity (sofa, couch) A real word example [R768] 6 © 2022 eBay. All rights reserved. https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html

So what is an embedding then? You shall know a word by the company it keeps - (Firth, J. R. 1957:11) Large Language Models - GPT 3 [175 B] ○ 45 TB text data - Wikipedia and books Neural network learns word associations from a large corpus. 7 ○ Detects synonymous words. ○ Suggests words for a partial sentence. © 2022 eBay. All rights reserved. https://en.wikipedia.org/wiki/Word2vec https://ruder.io/word-embeddings-1/

How do we “learn” an embedding? Text Encoder 12 © 2022 eBay. All rights reserved. R768 That’s how an embedding looks like!!! Image Encoder Devlin, Jacob, et al. “Bert: Pre-training of deep bidirectional transformers for language understanding.” arXiv preprint arXiv:1810.04805 (2018). https://en.wikipedia.org/wiki/Convolutional_neural_network

Why do we need ANN? All problems start with SCALE Exhaustive search curse of dimensionality ANN Approximate Nearest Neighbours 13 © 2022 eBay. All rights reserved. https://en.wikipedia.org/wiki/Proximity_analysis#/media/File:Euclidean_Voronoi_diagram.svg http://ann-benchmarks.com/index.html#algorithms

How does this function? Display all inventory matching my visual appeal Quickly pivot to entire inventory using a visual first cue 15 © 2022 eBay. All rights reserved. Is this a orange throw kilim pillow?

Data Engineering

Data Ingestion Challenges 19

Speed vs. resource trade off

Storage

Download errors

Challenges with Apache Airflow Challenge Solution Multiple Spark versions Define task level parameters Multiple Docker image versions Python virtual environment packages Different platforms, zones, and network Retries, system monitoring flakiness 24 © 2022 eBay. All rights reserved. Apache Airflow Logo

Evolution Model Drift • Seasonality • Aging of the models Actions • Metrics monitoring • Downstream evaluation • Retraining 27 © 2022 eBay. All rights reserved. Data Drift • Data Integrity • Data pipelines Actions • Fault tolerance • Monitoring of time, cpu, memory, disk

Scaling ML embedding models to serve a billion queries

Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

Slide 17

Slide 18

Slide 19

Data Ingestion Challenges 19

Speed vs. resource trade off

Storage

Download errors

Slide 20

Slide 21

Slide 22

Slide 23

Slide 24

Slide 25

Slide 26

Slide 27

Slide 28

Slide 29

Slide 30