Jul 2, 2022
Max Irwin on economics of scale in embedding computation with Mighty
With so much model research and development going on in the Python world, what options do programmers in other languages have? But also, how about that performance bit of transformer models with 768 dimensions — is this at all feasible to run in production and not pay a fortune for a GPU cluster?
Max Irwin, the luminary in the search engine world, a founder of max.io and a former managing partner at OpenSource Connections, sat down with me to talk about the economics of scale in the business of embedding computation.
Max is a big proponent of making deep learning available to more languages than Python: Rust, Go, JavaScript, Java. He actively develops Mighty, the embedding server (a blazing fast one) and various connectors (“starters”) for Qdrant, Pinecone, Weaviate and Solr. So if you felt lonely in your non-Python part of the programming world with respect to deep learning train, you will feel home listening to this podcast. Max has also presented on the same topic on Berlin Buzzwords this year. Check it out.
If you want to deploy Mighty to optimize costs by moving from GPU to CPU with comparable latency characteristics, you will find a discount code in the podcast shownotes.
Few topics we covered:
01:10 Max’s deep experience in search and how he transitioned from structured data
08:28 Query-term dependence problem and Max’s perception of the Vector Search field
12:46 Is vector search a solution looking for a problem?
20:16 How to move embeddings computation from GPU to CPU and retain GPU latency?
27:51 Plug-in neural model into Java? Example with a Hugging Face model
33:02 Web-server Mighty and its philosophy
35:33 How Mighty compares to in-DB embedding layer, like Weavite or Vespa
39:40 The importance of fault-tolerance in search backends
43:31 Unit economics of Mighty
50:18 Mighty distribution and supported operating systems
54:57 The secret sauce behind Mighty’s insane fast-ness
59:48 What a customer is paying for when buying Mighty
1:01:45 How will Max track the usage of Mighty: is it commercial or research use?
1:04:39 Role of Open Source Community to grow business
1:10:58 Max’s vision for Mighty connectors to popular vector databases
1:18:09 What tooling is missing beyond Mighty in vector search pipelines
1:22:34 Fine-tuning models, metric learning and Max’s call for partnerships
1:26:37 MLOps perspective of neural pipelines and Mighty’s role in it
1:30:04 Mighty vs AWS Inferentia vs Hugging Face Infinity
1:35:50 What’s left in ML for those who are not into Python
1:40:50 The philosophical (and magical) question of WHY
1:48:15 Announcements from Max
As usual, you can listen to the episode in audio form:
Spotify:
Apple Podcasts:
and on RSS: https://rss.com/podcasts/vector-podcast/522301/
Enjoy!
