Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance

Database history and the shift to NoSQL

0:00Hello, and welcome back to the Cognitive Revolution. Today my guest is Pete Johnson, a 30-year technology veteran now serving as field CTO of AI at MongoDB. We start with a brief history of database technology, going back to the 1970 introduction of SQL, and unpacking how the relative scarcity of disk space at that time informed data normalization as a design principle, and then how decades of Moore's Law and the resulting evolution of constraints

0:30ultimately led to the creation of MongoDB in 2007. This history puts today's acceleration into stark relief. As Pete recounts, in a remarkably short period of time, we've gone from being effectively forced to implement RAG pipelines by the very limited context windows of GPT-4 class models, to RAG is dead when we first got Million Token Contexts. To the brief token maxing window and subsequent budgetary fallout, to today, where RAG is again

1:02a major enterprise priority, as company data is increasingly available for AI systems to use, usage itself is scaling, and naively maxing out the context window costs multiple dollars each and every time. Pete's key point, above all, is that agent performance, and especially cost-adjusted agent performance, depends heavily on effective retrieval. With this in mind, MongoDB has continued to ship the improvements that developers need to take full advantage of AI. Vector search, sorting methods which combine traditional and vector search results

1:37into a single result set, a Rerank tool called Dollar Rerank, and their own embedding models, powered by the acquisition of Voyage AI, which have some pretty cool features, like a shared embedding space across model sizes, and a Matryoshka data structure. Interestingly, while MongoDB customers are free to use whatever embedding models they like, Pete insists that embeddings are, contrary to popular belief, not yet commoditized. From there, we go on to talk about how developers are using these tools to build memory systems,

2:13the write, change, recall, forget loop that Pete sees as the general pattern, and why forgetting is currently the hardest part. For now, Pete says it's important to remember that we're only 18 months or so into building AI agents, and as such, there is no established right answer, and nothing like a lamp stack for AI that enterprises can confidently buy and deploy without meaningful customization.

2:38We also touch on the lessons he's learned from talking to customers in India, Brazil, Mexico, and Europe, all just this year. And to my surprise, he reports that the most advanced companies he's spoken to in 2026 were in fact outside of the United States. Without further ado, I hope you enjoy this speed run through database history and high-level AI readiness strategy for enterprises and survey of the still-fast-evolving frontier of agent memory.

3:09With Pete Johnson, field CTO of AI at MongoDB. The Cognitive Revolution is brought to you by Mercury, the banking platform loved by over 300,000 entrepreneurs. I use Mercury's virtual cards, which make it super easy to set limits, expiration dates, category, and even merchant-specific spending controls to give my more autonomous AI agents, aid and clay, the ability to buy and test products. Recently, I asked if they could find a good way to split an AI-generated image into layers,

3:42separating the text from the background, and so on. Two of the products they found were behind paywalls. But using their Mercury virtual card, which is limited to SaaS purchases only, they bought a month's subscription, tested the products, allowed me to review the results, and then canceled the stuff we didn't need, all with functionally zero risk to me. This is already really powerful. And now, with spend, Mercury is making it possible to run an entire company's spending with the same level of ease and control.

4:14With spend, you can set granular budgets for every team, person, and all the agents you like. Plus, you can process receipts automatically and even temporarily auto-lock people's cards if there are ever any issues. The future of spending money is dynamic but controlled. So join me in the future of banking. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC.

4:47The I.O. card is issued by Patriot Bank, NA, member FDIC, pursuant to a license from MasterCard International Incorporated. Pete Johnson, field CTO of AI at MongoDB, welcome to the Cognitive Revolution. Thanks so much for having me, Nathan. I appreciate it. I'm excited for this conversation. I've been a database guy in the past, and now we're all kind of everything guys these days with our agents greasing the wheels for us. But the, I think one of the big themes that we're going to get into is the quality of data

5:20is still critically important to making things work. And I'm excited to learn everything I can from you over the course of the next hour or so.

SQL versus NoSQL in modern enterprises

5:30Maybe just for quick starters, I remember first encountering MongoDB like 15 years ago, maybe, when a young Guillermo Rauch, who's now the CEO of Vercel, was showing it off in the context of what at that time was like, seemingly like crazy futuristic Node.js applications. Like, wow, you can make JavaScript do these kinds of things? And there's a way, there's a database that like works in this different way where I'm not constrained by the schema that I already defined.

6:01Like, this was mind-blowing stuff. I'm not sure we've got a lot of technical people in the audience that say AI engineer is kind of our number one profile, but I'm not sure everybody knows kind of the broad lay of the land when it comes to database technology. So I thought maybe it would be helpful just to first kind of contextualize SQL versus NoSQL, schema versus schemaless. Give us kind of the high-level view so we know where Mongo sits in this obviously bigger database space. Sure. I like to think of this in terms of the history,

6:34in terms of the different kinds of problems we've had to try to solve with software over the last 60 years or so. And where it really starts for me is in June of 1970 is when E.F. Codd, who is an IBM researcher, wrote the original white paper that gave birth to SQL. So that was June of 1970. I happened to have been born in February of 1970. So I am as old as SQL, Nathan. And if you think about the kinds of applications we were building at the time

7:06as an industry and the kinds of hardware we had at our disposal at the time, the applications were very departmental in their nature, right? Not every company had a computer. And those that did, the people that were using it, were often on the same floor or certainly in the same building. They were working nine to five hours. It was okay to have downtime on the weekend. And if you think about the big three pieces of hardware at the time, memory, compute, and storage, storage was by far the most expensive of those three things in 1970.

7:41So if you think about how you might organize data, keeping in mind that the scarce resource was storage, you would, of course, do it in such a way that's now known as normalization. So you may have in university or in coding school, you may have learned things like first and third normal form, all about trying to lay out data on disk in the most efficient way so that you aren't storing any piece of data more than once. The simplest example I can give with that might be

8:12my wife and I share an address and maybe with a retailer might want to know what our address is. There's two people at that address. You might have in one table, the list of people. In another table, you might have the addresses. You might have a third table that links the people to the address. So these are only storing that address one time. And that's the most disk efficient way to do that with, like I said, some kind of normalization. Well, fast forward to 2007, which in October of 2007

8:43is when the first commit was made to MongoDB. And the world is a very different place in 2007. The world has internet. The world has cloud. The world has mobile devices. iPhone came out in the US for the first time that year. So it's a very different world. And the scarce resource in that world, because we had 47 years of Moore's law between those two dates, the scarce resource in that ecosystem became time. Because the kinds of applications that we would write that were in the cloud or that we would write for mobile devices,

9:15you could not have downtime on the weekends. It was not people who were confined to the same building. It was not nine to five use cases. So it was a very different world that required a very different kind of database. And that's what gave rise to NoSQL. Now, our flavor of NoSQL, which, like I said, that first commit was in October of 2007, uses it as its core JSON. So you mentioned the stuff with JavaScript. That's why that was so popular 15 years ago or so, is because what you get back from your API call is JSON.

9:48We're storing it in transit as JSON. We're storing it on disk and a binary form of JSON we call BSON. And that's part of why we're so fast is because we're keeping that format the whole time so that when you serialize it off a disk, when it goes through our servers, when it ends up in your client, it's essentially staying in the same format that whole time. So if you think about the kinds of applications that we've been building since 2007, like I said, disk space is no longer the scarce resource. If you take that address example,

10:21if you denormalize that data just a little bit, if you, instead of storing it across three tables, if you have a JSON structure that has my name and my address, and then a second one that's got my wife's name and the same address, you're duplicating the address. But what do you get in return? What you get in return is one disk read. So that's why it's faster. So instead of having to do three disk reads to get that same data, you can now do it in one disk read. Does that mean that every problem is solved by a NoSQL database

10:53or by MongoDB in particular? No. But it also doesn't mean that every problem has to be solved with SQL. And that's how that sort of evolved out of SQL was invented at a time where the problems were slightly different. And a lot of those problems still exist. SQL is certainly a valid solution to lots of different problems that we still have. But when you need speed, when you need scale, and increasingly, when you need better retrieval quality, that's really where MongoDB shines.

11:25Reminded of the old saying that the problem has faded, but the solution persists. Yes. I see that in a lot of parts of life these days. Tell me a little bit more. You were just kind of getting to it, but tell me a little bit more about, I mean, Mongo's a pretty big company, $30 billion market cap, obviously has made serious inroads into some of the biggest companies in the world as customers. Give me kind of the next level of detail in terms of before the AI moment,

11:56and then obviously we'll get into that in depth. What are the kind of big decision factors that would push an enterprise to go one way or the other with their database decision? And how are they, do they typically do it on like an application by application basis these days? Or do big companies feel that they need to have a single kind of consistent database technology for everything they're doing? Sure. It's a mix. Like you just said,

12:26we're in like 75% of the Fortune 500. But if you look at the overall market, you mentioned our market cap, we're about $2.5 billion is what we ended. That was our revenue number for our fiscal 26. We're currently in the middle of our fiscal 27. And that's about 2%, 3% of the $100 to $110 billion database market. Where most enterprises choose us is those first two things that I mentioned, when you need it fast or when you need it at higher scale. Because you're not storing disk,

12:57you're not storing that data across multiple places of disk. That's how you get the speed. That's how you can get better scale if you're storing. If you're going to retrieve JSON because every modern programming language can serialize JSON into some object format so that you can use it immediately in your business logic. And that's the other aspect of time. It's not just wall time for the transaction, but it's also the developer time of what do I have to do with this wad that I get back from my API call and then make some kind of useful business decision out of it. But what we see is right solution

13:28for the right problem. And increasingly what AI has done is it's opened up the number of problems that were a better fit for than a traditional SQL approach, especially those at higher scale. And that's why I mentioned the three things, the speed, the scale, the retrieval quality has really become an important factor. It continues to, especially later here in 2026, as you start to see people strategize around token maxing to evolve different kinds of agentic architectures

13:59in ways that we didn't see in the first half of 2026. Retrieval quality has become more important for AI, but it does end up being a mix. It depends on the corporation. Some of them will take it workload by workload. And for this workload, we'll use Mongo. And for a different workload, we'll use SQL. Others are taking an all-in strategy. It really varies based on, in some cases, how much greenfield versus brownfield do you have in your portfolio? In other cases, it's where's my talents already? I was just at, for a major automotive company

14:29here in North America, I was just at a hackathon where we trained up like 250 engineers and turned them loose for three days on a hackathon. And they've got some new product ideas that fell out of that. So there's no single right answer, right? There's lots of ways to sort of walk through that forest. But at the end of the day, it does come down to right problem for, a right solution for the right problem. And where we typically help people is with those modern applications that need that speed at scale and those gigantic use cases that increasingly need

15:00that better retrieval quality. I am really interested though in big three tech modernization. And I have a bit of a brush with it from time to time, just enough to know that there's a lot of appetite, but there's also a lot of friction. What was your experience of, and I'm sure a lot of these app concepts that people were working on are AI apps, right? Without naming names, what more can you tell us about the kind of gonzo on the ground experience in the big three? Some of it was customer experience

15:32kind of applications. Others was, how can we make the dealers more efficient? How can we make the mechanics more efficient? Some of it was, how can we make the in-car experience more efficient? It really spanned the gambit of different types of use cases that a big three automaker might have. But really it's, I mean, I'll go as far as to say, like our education system has a bias, has a historical bias that thou shalt always normalize. And for a lot of use cases, that's valid, but there are an increasing number

16:02of use cases where these big three that I just mentioned matter. And increasingly, that retrieval quality one is becoming a much bigger deal for agentic architectures as you start to try to take better advantage of the context window than just stuffing it with a million tokens every time. And we can get into some of the details as to why we're a good fit for that and why that's emerged like in the wake of what happened at Uber earlier in the year with them burning through their token budget for 2026 and 13 weeks. That grabbed a lot of headlines and there's been similar stories

16:33about where like maybe token maxing wasn't such a great idea to begin with. Yeah, it was a phase. We've all been through our phases. I was really attracted to the schema-less database model way back when because I'm just kind of a free-flowing person who loves to just make a mess now and tell myself I'll clean it up later. And so this is one of the things that I'm really interested in in terms of how DB interacts with AI applications

17:04because it seems like on the one hand it's more tempting than ever to do that, right? It's more valuable than ever to just be like let me just dump all the data in here and like maybe an AI will make use of it sometime later. But then you do have interesting challenges downstream of that. one with your example of like the two records with the same address now you've got to update it in two places. So that's something to keep track of obviously. I'm also kind of struck that like the advantage of latency

17:36you may have a different take on this but it feels to me like the advantage of latency is somewhat reduced because now like everything gets put through a slow step anyway and so the ability to like shave a little time off the DB call when I'm then going to have to wait for tokens maybe matters less and then I'm also thinking like if I do start just dropping all my data payloads into a sort of extra field now I'm going to be doing like exactly what you're saying I shouldn't be doing in terms of just bringing a million tokens into context all the time.

18:07So take that whatever direction you will what do you see as kind of the biggest trends in application development that have bearing on your database choices as we are AI-fying everything. Hey we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Granola the AI-powered notepad built for the way real people actually meet. Here's how it works

18:38you take rough notes like you normally would and in the background Granola securely transcribes the meeting then it turns everything into clean structured actually useful notes when the meeting ends. And the best part Granola works through your device's audio which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word

19:08and still walk away knowing exactly what was decided who's doing what and what comes next. When I had Granola co-founder Sam Stevenson on the show earlier this year he explained how Granola aims to provide a calming experience for people with crazy work days and as user of the app myself I have been struck by how streamlined even minimalist the Granola product experience is. That takes real discipline but the result is a product that works not just for AI early adopters but diverse teams of people

19:38who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stevenson for a master class in designing AI products for mass market adoption and try Granola for free at granola.ai slash TCR The Cognitive Revolution is brought to you by Diffusion the AI transformation specialists that help organizations from traditional SaaS businesses to defense companies to non-profits

20:09build software factories that can scale not just outputs but business outcomes. You probably know that the majority of enterprise AI projects fail. In general that's because leadership fails to realize that AI isn't like traditional software that you can just buy and install. On the contrary if you want AI to amplify your business's unique DNA you'll need to make a sustained effort to record understand simulate and optimize your business processes.

20:40Building these skills by trial and error takes years but your business problems can't afford to wait so here's how Diffusion can help. You identify your most important business problem fly to Silicon Valley for an intense week of problem solving with the Diffusion team and by the time you leave you'll have not only cracked a critical challenge but built the core skills needed to do it over and over again from home. Cognitive Revolution listeners receive a 25% service credit on their first engagement

21:10with Diffusion so visit diffusion.io slash TCR to learn more about how custom built software factories can scale critical outcomes for your business. that's diffusion.io slash TCR sure so let me so first let me push back on one thing you just said most people think that we're schemeless and that's not actually true it's not that we're schemeless it's that we're schema flexible that you can

21:41change the schema more easily over time because more of the data is denormalized and therefore centralized in a way it's not a strict requirement unless you specifically say so that every document in a collection so we don't have tables and rows right we have a think of a JSON WOD as a document and you have multiple JSON WODs put together into a collection not every document in a collection necessarily has to have the same shape you can have

22:12different shaped documents in the same collection so because the schemas are flexible and because you can have different document shapes in the same collection that's why people assume that we're schemeless but what it does is it gives us the ability to be malleable in a way that a traditional SQL schema is less malleable I won't say not malleable at all because it depends on how you're laying out the data but anybody who's ever had to change a SQL schema that's already in production

22:42and the cascading effect that has on multiple tables knows the pain of what I'm talking about that if you had some of that data denormalized it's far easier to add attributes to a document that's already there and do so selectively in a way that isn't possible nearly to the same extent in the SQL world and this is where how we've implemented like the history of how we implemented vector search and the impact that has on application architectures

23:12for agents and that the details matter here so if you'll allow me I'd like to talk a little bit about that history does that sound okay yeah please okay

Lexical, vector, and hybrid search evolution

23:21so for us it really started with in an unusual place and that is with lexical search so in 2020 we noticed that a common use case for MongoDB was for people to stand up their own Apache Lucene servers co-located with wherever their their MongoDB clusters might be and the reason they were doing is they wanted to be able to point that Lucene cluster to different text fields in a document and be able to do keyword retrieval off of them perfectly reasonable

23:51thing so we thought well as you might know there's there's three different versions of MongoDB there's community which is you're responsible for the support you're responsible for the operations there's enterprise advanced which most customers are using on prem where you're responsible for the operations but we're responsible for the support and then there's Atlas which is our managed service version of it which you can deploy your instances on any cloud hyperscaler data center you'd like amongst Amazon

24:22Google and Azure and in that form factor we will do the support and we'll do the operations so those are the different choices you have when you deploy MongoDB and regardless of which one you choose like I said we notice people standing up Lucene servers so that they can get keyword retrieval on what they have so in 2020 we introduced what's now known as Atlas search which is on the managed version of the managed version of MongoDB you

24:52automatically just you just get it as part of the instance that there's a lower priced tier where you just point it to attributes some text attributes that you already have in your data will automatically index them you can do keyword retrieval search on those then the next logical thing was a couple years later well if you're gonna have lexical search you might as well also have vector search so think about this what's a vector at the end of the day a vector is an array of floats right you take some piece of data

25:23and whether that piece of data is text or an image or audio or video you pass it to an embedding model of your choosing and what you get back is an array of floats so to us an array of floats is just an additional attribute to a document you already have so for us to implement vector search was just add an additional attribute to the flexible schemas that we already have and now build a vector index on top of that array of floats and that's your vector search and what it

25:53created was this notion of being able to do these powerful hybrid searches as well so if you think about the sort of the trivial example that most people learn MongoDB off of is suppose you have a book and a book might have a text field for the title and it might have a text field that is a URL that points to the cover of that book you might have an integer that's the number of pages you might have an integer that's the year of publish and you might have a text

26:24field that's the synopsis so if that's a standard record for a book and if that's a standard document for a book in MongoDB and that's put in a collection suppose now I want to create a vector index off of the synopsis I could take that text synopsis text I could pass it through an embedding model of my choice and I then store the array of floats that come back from that embedding model and if I wanted to suppose I had

26:5410,000 records I had 10,000 documents in my collection maybe I only want to do a vector search on the top thousand maybe I don't want to do it on all of them because of the flexibility of the schema is because you can have different shapes MongoDB not only allows that but thrives on it really well so we had lexical search we had vector search and then that enabled us to do this notion of hybrid search so suppose I wanted to do combine a lexical search of the title of the book with this now vector search

27:24that I have on the synopsis and what if I also wanted to pre filter because I've got some integer fields in here suppose I want to do a search where I do a lexical search on the subject and a vector search on the synopsis but I want to eliminate any documents any books that weren't published after that were public I only want to see books in my data set that were published after the year 2000 so now I have these three levers of query

27:55power that together are give me lots of interesting ways to now query my data that I combine this sort of pre-filtering based on the metadata with the lexical search with the with the vector search so once we had that in play by like 2023 we thought well this works with any embedding model you want but could we make this easier to develop if we had an embedding model that was part of it and that's why we purchased voyage in 2025

28:25which should create this better together story we can get into all kinds of things with the better together but the way that we go to market with voyage is you can use voyage on whatever vector database you want you can use any embedding model and use MongoDB as your vector database or you can use them together and there's a better together story that's a part of that so that's kind of our history like I said kind of unusual that it started with lexical search but it kind of went lexical search vector search hybrid search with pre-filtering and now this better together storage with

28:57the voyage embedding models that's how that built over time for us what is it like to query that today are you able to I think this is like a has been relatively recently advanced right that I can now issue a single query and put kind of in SQL terms I guess aware clause and like all three of those different kinds of data and the engine under the hood which filter to apply first like all that stuff kind of I don't have to worry about anymore is that right

Rank fusion and score fusion features

29:28that is so one of the advantages that we have in the way that we implemented vector search is it gets to stand on the shoulders of the base product that we already had so we've been running Atlas in production for a decade now so we already knew know how to replicate data across multiple data centers even across multiple clouds we already know how to do sharding whether you want to keep data close to specific users for business reasons or for legal reasons we already know how to do security not only query at rest or not

29:59only encrypted at rest and encrypted in transit but we can also do during the query we get to stand on the shoulders of all of that for the vector search and one of the things that we get to do on top of that is MongoDB has always had this notion of aggregation pipelines and the point of aggregation pipelines is instead of forcing you to do a query get some JSON wad back of your result set and then have to massage that before you use it instead with the aggregation pipelines you can give extra

30:30you can give extra instructions to the back end so that that data gets sorted filtered massaged in different ways so that what you get back from that API call you can now use immediately and you don't have to do any other further massaging so because we already had that mechanic in the core product what you're referring to is we just announced a pair of features one's called rank fusion and one's called score fusion and what that allows you to do is in absence of an aggregation pipeline in these new aggregation

31:01pipeline stages you would have to make the vector search and then independently you would have to make the lexical search and then you would have to combine them yourself now with those two aggregation pipeline stages you can do it in one API call so you only have one round trip call to our servers and then on the back end we'll do both of those calls for you and then we'll combine the results either based on their rank with rank fusion or based on their score with score fusion so that what you get

31:31back is an already sorted list based on which one of those functions in the aggregation pipeline you chose so that you could then use that JSON wad that result set immediately I'd love to learn a little bit more about how that works because this is something that I have run into and I've come up with pretty simple solutions for myself right but like you sort of have the filter and then I'll just be like okay I guess I'll sort by the

32:02vector match score subject to the filter right or we'll kind of tier them based on different filter values it's always struck me that there should be smarter ways to do that that I've come up with that you guys have some so what are the kind of galaxy brained approaches for getting the best final sort order from these kind of fundamentally different things so for now what most people are doing is and this is the reason why we have

32:32the pipeline stages that we do it's it's usually based on our anchor by score and if you also throw in the pre-filtering that lets you that lets you reduce the data set before you do the queries and then you get those queries back and you can interleave them based on either score or rank for now that's what most people are trying to do but my joke for this Nathan is like there's two kinds of developers there's those that love MongoDB and there's ones that haven't tried us like we're we're always in a pretty tight loop of feedback where we the whole reason we came up with rank fusion and

33:03score fusion is because the community told us they wanted it in very much the same way that why did we start doing Atlas search why didn't we make lexical search part of the platforms because we saw people were using it we found a way we saw a way to make their lives easier if it's so that you don't have to spend the operational cycles managing your own Lucene servers what could you do with the cycles that we give back to you and it's the same thing with rank fusion score fusion so that you don't have to manage that code base yourself so that if you could just throw

33:34it in as a verb in the query that you already have okay now what are you going to do with that with that time that we're giving back to you so where it goes community is going to tell us what other kinds of things they want with that read more novels is one candidate answer for what I'll do with my time savings and going back to your book example yeah another thing I've struggled with at times is let's say I have the book but let's say I actually have

34:05all the contents of the book too right now I've got hundreds of pages of text associated with this kind of one entity now I can break that down into I just read a novel that had acts so it had like four different acts and then of course there were chapters within the acts and then there were sections perhaps and paragraphs and obviously you're down to sentences at some point how do you think about I think there's some new work on this too although I confess I don't grok it fully how do you think

34:35about like how that raw text should be chunked or otherwise divided up to be represented in vector space is it like am I looking for a single answer to that or is it do you think of it more like a trunk and branch and leaf kind of thing I've experimented with that kind of stuff a little bit as well but I don't I guess I I don't really know what the state of the art is in that realm

35:06today hey we'll continue our interview in a moment after a word from our sponsors you're listening to Deep Grim Flux TTS different voices same model all ready to speak Flux TTS is a streaming text-to-speech model built for voice agents Flux reads the room it holds context across the conversation turn after turn with consistent tone interruption handling and plenty of personality so your agents keep it flowing and customers can just keep talking try Deep Grim Flux TTS free now until September 12th

35:38visit deepgram.com slash keep talking terms apply today's episode is brought to you by Anthropic by now you know my story Claude drafts my intro essays and I rewrite them not because the drafts are bad but so I can stand behind everything I publish well I have an important update Claude Fable 5 is the first model to have me rethinking my rule today I now think co-authorship not sole ownership should often be

36:08the goal where the model excels rewriting its work can be more about vanity or a misplaced sense of duty than integrity I feel it most in songwriting I'm no lyricist but I'm good with a song concept and Fable writes some amazing verses I give it feedback on its misses and I push it to aim for higher inspiration add layers of meaning optimize syllable density and above all write a hit song these days I get compliments on just about

36:38every song we write together Claude is the AI for problem solvers it's the collaborator that understands your entire workflow and thinks with you not for you whether you're debugging code at midnight building a financial model or strategizing your next business move Claude extends your thinking to tackle the problems that matter for problems worth solving get started with Claude at Claude dot AI slash TCR that's Claude dot AI slash TCR and check out Claude Pro which

37:09includes access to all of the features mentioned in today's episode once more that's Claude dot AI slash TCR fair so

Chunk size tradeoffs and contextualized chunking

37:18there's a couple of more broadly speaking and I promise I'll answer your question more broadly speaking the use case for this level of search for this kind of semantic or vector searching is typically rag pipelines for agents or for a more sophisticated agentic memory for agents that's when you typically see vector search start to enter the conversation when it comes into an agentic architecture in some way specifically what you're talking about year is chunking and chunk size tends to be more in that rag

37:51pipeline use case I've got some external data I've got some proprietary data I want to inject that proprietary data into my LLM of choice without having to go through an expensive fine-tuning or training process because the LLMs are trained on public data they're not trained on my proprietary data so how do I inject that in a safe way it still gives me some data fidelity so that I'm not I'm not giving away my proprietary data but I can still get it to focus on my business problems

38:21instead of every LLM knows who the last 30 Oscar winners were and that's not super helpful when I'm trying to get an agent to help me figure out what my NOC should do next or my my IT ticketing should do next so that's the use case so if you've got a bunch of documentation that someone who is a human professional in that world you now need to figure out how big of the how big that should the chunks be that I place in the individual documents and what's the fidelity of that and when you're the application developer trying

38:52to figure this out there's this trade-off that you're looking at so if I make my chunks too small let's say to the fidelity of a sentence if I make my I potentially lose the context in which that sentence appears just if you just read one sentence from a novel doesn't tell you a whole lot but if you then also read the paragraph or the page it tells you a lot more about what the context of that sentence is so that's an argument for well maybe I should make my chunks

39:22bigger there comes a point though where if my chunks are too big my storage costs start to go up and my retrieval quality starts to go down because if I've got three pages let's say and I'm trying to get to the fidelity of a sentence I now have too much information and I lose the fidelity of that sentence so what's the right answer I've been doing this a long time Nathan the answer is always it depends right so you have to go through as a developer you have to go through this iterative cycle like I chunk to a

39:54certain size I run some tests what's my retrieval quality okay now I try it again I try a different chunk size I do some tests I see what my retrieval quality is and I have to do that three or four times that for my use case to try to find out what the right balance is between chunk size storage costs and retrieval quality so one of the reasons we bought Voyage is because like those guys like their side hustle is teaching people at Stanford how to how to build LLMs those guys are all geniuses and they came up with this new

40:26concept last summer called contextualized chunking and we've got a blog article on this that we can potentially link through here but the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes there comes a point in a graph where if you've got retrieval quality on your exit on your y-axis and you've got chunk size on your x-axis with a traditional embedding model there comes a point where at lower chunk sizes you have zero context so the retrieval quality is low and it builds but

40:57then at some point it flattens and then it starts to degrade if your chunk sizes become too big and that's why you have to go through these iterations to find the right combination but what if you did what if instead of sending that L as one big text blob what if instead you send it as two what if you sent the fidelity of the sentence you actually want and then as a second string you send the size you send the other contextual informations where the name contextualized chunking comes from and we will figure out for you what the right

41:28chunk size is for that combination and we will give you a vector array that comes back that balances those things for you when you flip that script it turns out you can get better retrieval quality with a smaller chunk size which otherwise is not possible if you're doing it the traditional way you the way to get better chunk size way to get better retrieval quality was with higher chunk size but if you use this contextualized chunking we released version three of that last summer we just released four of that in the last six weeks you can

41:59flip that script now there still are use cases where you need to control the chunk size for different things but if that's something that you wanted to not worry about or have to learn about and the way I think about this is more developers will build AI agents in the next three years than did in the last three years and in order to make that possible we have to lower the learning curve we have to make it easier for the developer ecosystem to learn how to do this so if you don't want to have to learn how to go through the iterations of figuring out what the right balance of chunk size and retrieval quality

42:31is instead if you could use contextualized chunky we'll figure it out for you and you still get good retrieval quality out of it so that's that's one of the benefits that voyage offers that no other embedding model out on the market offers that ease of use feature that gets you better retrieval quality I definitely appreciate not having to worry about it but I do want to learn a little more about it it sounds like this happens in sort of a test free way at the level of

43:03the developer so like I don't have to bring a bunch of eval suite work to the table so how at the level of kind of principles like how is it working in the background and you said also I'm getting one vector back right so I pass in the sort of the chunk that I think I would really want to be able to zero in on and context and that gets converted into a single vector back that represents both of those things in I guess

43:35some sort of superposition yes that's

Matryoshka dimensions and embedding model quality

43:38exactly how it works so you get back one array of floats exactly how you did before it's just now you don't have to go through the iterations of figuring out the size of the chunks you need and similarly there's another voyage feature that all the voyage all the voyage embedding models have called Retroiska reasoning because the other place you have to make a decision about storage cost versus retrieval quality is with number of dimensions so what do I mean by that everybody knows what two dimensions is if you've taken high school

44:08level algebra right xy right but in an embedding space you tend to have at least 256 dimensions and sometimes as high as 2048 and the more dimensions you have so each dimension is represented by one of those floats in that array of floats the more dimensions you have the richer your embedding space is and the better retrieval quality you get but at higher dimensions that doesn't come for free you've got a storing 256 floats

44:41takes up less space on disk in an index and memory than it would for 2048 so you again have to go through this iteration of what's the right number of dimensions for my use case given my what my storage costs might be all the voyage models have a feature in them called Retroiska reasoning it comes from the Russian nesting dolls and getting about how Russian nesting dolls work right you've got you've got one doll and you open it up and there's another one exactly the same but smaller a smaller fidelity inside and then you keep doing that and over and over so what what

45:14the how the how the voyage models work is when we generate let's suppose you did at 1024 suppose you wanted 1024 dimensions you run some tests now you want to try 512 with a traditional model you have to run your entire corpus of data through a second time at 512 but with voyage models you don't have to do that when you've run it through at 1024 vectors those floats they're ordered so if you want to try 512 you just lop off the

45:45last 512 and that's now you immediately can begin testing on the remaining 512 so again it doesn't completely solve the problem of figuring out what the right combination of retrieval quality and storage space is but it helps you get to the answer faster so these are there when you take some of these things and some there would be three or four of these kind of features that make it easier to use and help you get to your final answer more quickly and

46:15the idea there is to give you time back in your day that you can work on your business logic instead of figuring out the plumbing yeah that's cool i'm a huge fan of matryska anything uh when does this stuff become necessary so for me i'm basically a business of one and i try to be an early adopter of everything that i can and i do have a pretty well working i call it deep context but basically a retrieval system that

46:48allows my agent to go into kind of all of my history from the last five years essentially which is emails and slack messages and everything i publish online and dms across all kinds of channels the podcast of the transcript of the podcast i should say diarize so it knows what i've said and what the guest has said adds up to about a gigabyte in my case and i haven't really optimized it much at all i just kind of let the agent throw it into a database of its choosing and put whatever

47:20optimizations on it it felt like it needed to and then we did get a at one point it was like well yeah we could probably do better than keyword so we've got an embedding layer on there as well i think full disclosure i believe i used the gemini embedding model for that uh but it's like not very well optimized how would i know if i'm really missing out on something i don't have like a huge eval suite right it's just like i'm kind of vibing it with it doesn't seem to be working well is it a matter of like data scale scale of users

47:52is it about like cost why i want to optimize my inference cost and that's where i really need to get serious about how much data is is being returned like what are the what are the thresholds that people or obviously larger organizations cross where they're like okay we can't really do it the let the agent choose its own adventure way anymore we really need to get serious about some of these optimizations well i'd return to the three things i mentioned in our first 20 minutes or so it's it's speed scale and retrieval quality are the main three things

48:26what most people do and you're not going to offend me if this is what you did i mean most people start with postgres and pg vector and then they choose they choose their embedding model with whatever cloud they're using like gemini is prominent if you're going to use if you're going to be on google just like open ai's embedding model is is pretty popular over on azure because of the historic relationship that those two companies have but there comes a point in when you're when you're doing a demo when you're doing a poce it doesn't always show itself but there comes a point where when do milliseconds matter

48:59to your use case when does scale matter to your use case and that typically comes depending upon chunk size that typically comes at about a hundred thousand vectors is when is what i mean by scale and when does retrieval quality matter if you look at voyage ai model hugging face has a benchmark out there called r tab that voyage ai models are typically at the top of and we can get as much as 14 percent improvement compared to some of those other embedding models that we just mentioned

49:30so are there use cases for which a 14 percent difference in embedding model quality is the difference between a hallucination and a correct answer and that's before you even start putting re-rankers on it which is another way that you can boost retrieval quality without having to do anything special to your data so like i said it's it's speed it's scale and by scale i typically mean in the neighborhood of a hundred thousand vectors and retrieval quality out of your embedding model most

50:00people think embedding models are commoditized and that is not true there is a very big difference that you can get in retrieval quality based on what embedding model you choose anthropic does not have an embedding model in market they recommend us it's a great recommendation you mentioned re-rankers and there was also this kind of earlier concept of sorting results out of the database this brings to mind this concept of bitter lesson engineering that i think is kind of growing in prominence which from

50:33my simple point of view is just like every so often you should probably kind of go through your stack and look at all the kludgy extra things that you did to make things work and say like which of these do i no longer need because the model got smarter or the embedding model got better and so things are just kind of naturally working now or could naturally work now where in the past i had to kind of do all these artisanal craft sort of things to to make sure

51:06that they worked are you seeing examples of that in the retrieval space broadly where things are in some ways getting easier or have you not really seen the bitter lesson applied to these pipelines these production environments yeah i think there's two places that i've seen that recently and one of them is with the re-ranking that we were just talking about so in the spree we released dollar re-rank which is a companion to the score fusion and rank fusion that we already talked about so

51:38those two relate to doing a hybrid search the dollar re-rank again you would in a historical way if you would have to do your vector search once send your results to a re-ranker to get them reordered in a way that is most optimal for a rag use case to then pop them into the context window just like we did with the score fusion and the rank fusion we've now got a uh a stage where if you do dollar re-rank you call the api once we'll do both of them for you on the back end so that you only have to make one round trip

52:10to the server for that so that's one place that we see some additional ease of use the other one is a feature that we released recently called auto embeddings which is you tell us this is a better together story you tell us what which collection you tell us which attribute on documents in that collection and you tell us which voyage model and how many dimensions you want and we'll take care of the rest of it anytime an existing document with that attribute in it changes we will automatically take

52:43that new let's say it's text put it through the embedding model update the vector in the document update the index and memory and we'll do all that for you if a new document shows up in to that collection that has that attribute we'll go through the same cycle for you so again trying to remove some of the plumbing so that you get some time cycles back as a developer so you don't have to you don't have to go craft your own and maintain over time your own embedding pipeline you just pass us

State of the art in agent memory systems

53:14one json wad and we'll do it for you zooming out a little bit from the database to the the broader systems that wrap around them you stuff about memory for ai systems and this is another thing where i've seen kind of the great location and then somewhat simplification probably simplification has been enabled by things that enterprises are maybe a little reluctant to pay for at scale like when

53:46i get a million token context window i'm like great now i don't have to worry about all this stuff i can let the model handle a lot more of it but i do take your point that obviously you got thousands of users that can add up how what would you describe as and of course it depends but trying to abstract away from some of the details and give me kind of a center of the distribution description what would you say is like the state of the art in what a good memory system looks like today sure so i again history

54:18matters here right when chat gpt came out in 2022 in the end of 2022 it had a very simple application architecture you took you took the query you put it in a context window the llm would process it and give you a result end of story that's how it worked and by spring of that year by the next spring you would see headlines like chat gpt passes the bar exam and it would have these very cool things it could do we already talked about why you need rag because those llms even today are only trained on public data

54:53if you want to inject proprietary data into it you need to to specifically do that with a rag pipeline what we saw in 2025 though was the second problem that we see with with the core llms and that gave rise to the need of memory and that is like in spring of 23 if you had asked chat gpt what's mongo db stock price today it couldn't tell you and the reason it couldn't tell you was the llms have a knowledge cutoff day there's a date after which they don't know anything about the world because the model vendors at some point have to stop training and put them into inferencing mode and the way we overcame that in

55:272025 or so was with tools and with mcp and being able to have to grant to the llm hey here's this api or here's this tool we could call to go get additional information that's why whether you're a cloud person or a gemini person or a chat gpt person that's why today if you ask hey what's mongo db stock price it can tell you because it can find a tool in its arsenal go make a web search get that information and report it back to you the other thing we saw in 2025 though is this notion of

55:58looping where the result of one loop of that agent then became the input of the last one the context window is still very dung right the context window is still you have a fresh context window with every time you pass it to the llm and that's when we needed agentic memory and the early passes we saw of agentic memory was short-term memory was typically okay take every response that we got within this session and cram it into the context window so that we can see what happened earlier in this conversation and then long-term memory was okay now do the same thing for every session we've had

56:33for the last three days and cram that in there as well and then had two negative side effects that i now see people starting to push against negative side effect one we already talked about is token maxing right if you if every agentic loop puts a million tokens in the context window even con even context you don't necessarily need your cost starts to add up but there's also a functional cost to that there's a couple academic studies that show that like the first 7k and like the last 7k tokens are the most important and the stuff that's in between can end up just muddying and confusing the llm as it's

57:08trying to come up with an answer so instead of thinking of how could i maximize how could i cram a hundred a million tokens into the context window every time what if i could choose just the right 200 000 for this agentic loop so we're starting to see more sophisticated short-term memory that doesn't take everything in this session but just the things in the session that are relevant for this pass and the same thing with long-term and we're starting to see some new memory types especially in fortune 500s there's a new memory type called taxonomic memory so think about either in a

57:44vertical so like automotive we were talking about before or in a specific company there's some terminology that is specific to verticals or specific to a particular company that a lay person who is not in that vertical those words may mean different things so you might have a list of let's say a hundred of those terms but for this agentic loop what are the five that are relevant for this loop of the agent and then do that again for the next loop what are the five that are relevant for

58:16that loop of the agent so again instead of taking all hundred terms and putting them in the context window every time when not all of them are going to be relevant what if you could instead have a more sophisticated memory type memory system that instead of just taking everything use vector search or semantic search or some combination of vector and lexical with some pre-filtering and just got the just the right just there's some perfect you'll never get there but there's some perfect

58:47context window for this pass and that perfect window doesn't use all of it it uses just the stuff that it needs to those are the kinds of things that we're starting to see in enterprises both to combat the cost of the tokens and to combat the relevance of the tokens so you can get the best answer out of this call to the llm and do you have any tips for maintaining these memory systems i mean this kind of goes back to some of the challenges with and maybe there's good solves within mongo db for

59:20some of this if so definitely highlight them for me but i sort of struggle with the lack of normalization or my failure to observe the dry rule in my own memory system sometimes i'll have like you know i have these like raw exports that combine my email and dms and everything else right into these kind of monthly log files and these are typically like a couple hundred thousand tokens for a month then i'll like summarize those into a monthly summary and then i'll take the monthly summaries

59:55and i'll put those into a yearly summary and then i've got like a wiki type carpathy inspired structure that sits on top of all that and is like more entity based and they kind of link to each other and some of this lives in a database some of it lives in a file system but now i've got a challenge where i'm like some new stuff has happened okay what do i do what do i change how do i make sure that it's changed everywhere it needs to be changed i sometimes also have instances where i've had really funny i mean the models are getting better with this is less of a problem these days but

1:00:30in setting this up in the first place i had some problems where the models would see a project that i like experimented with a little bit or like was maybe going to pursue and then didn't and it will like keep it as like an open thread for months and months and i'm like dude i never actually did that you can just forget about it so forgetting is also sometimes a real virtue a memory system um how what what have you learned what tips would you offer in terms of how to maintain these things so they're accurate not just when you first create it but on an ongoing basis this is the hardest problem

1:01:06with agents right now i think and the way that i put this to people is we as an industry we've been building databases for 60 years we've been building agents for like 18 months man like nobody knows all the answers yet we're all sort of collectively figuring it out but what i do see with these more sophisticated memory types is that as the application author that you have two responsibilities with these memory systems where with the prior one you only had one really so not only do you call

1:01:36this memory system hey here's my query give me the best maybe you give it a token budget is the way some of these work so that you don't you don't have to figure out how many memory types there are and how to query them you're just like give me the best 50k tokens or give me the best 100k tokens you give it a token budget and then it gets it back from the mix of different memory types it might have and then you put that in the context window you get your answer and then your second responsibility is to send that answer back to the memory system so the memory system can then curate it and then write it into the memory system and when it writes it into the memory system the more

1:02:11sophisticated ones also have this notion of our back so that if you and i are in the same job type that we get to share the memories so that if you come up with a really good memory and then i get to reuse it later then that ends up being beneficial for both of us so that's the second responsibility you have but the hardest part is what you're talking about here the way that one co-worker just wrote this to me yesterday i want to quote him write change recall forget because these things have a a half-life to them right like things that are more recent are more important than ones that

1:02:47that took place weeks if not months ago maybe you can tell me about any like mongo specific tricks or advantages for these sort of graph structures that i seem to keep making i have people and the organizations they work for and the ideas that that i sort of associate them with and these things all kind of point to each other but they do it in a pretty loose way where i'm i'm right now kind of

1:03:18trusting the agent to hopefully notice those pointers and to the degree i'm like running maintenance hopefully follow those pointers and do the necessary maintenance i don't have a lot of guarantees and i suspect that there are better technologies that i could be building on that would give me a lot more robustness well there's there's a couple different ways to tackle this like i said

Graph structures and retrieval optimization

1:03:39the forget part is the hard part of it but what we see our customers doing is this is where like the retrieval quality of those memory types that's where that non-commoditized embedding model can make a difference and where a re-ranker can make a difference you can get depending upon the use case you can get like a five to ten percent boost at retrieval quality just by using a re-ranker on top of whatever your embedding model of choice is and that's true of architecturally that's why we made it so easy to implement the re-ranker on top of the voyage embedding models when you're doing that vector

1:04:15search but there are use cases where maybe you have so much data that you might have to take a hybrid approach where maybe you use maybe use a graph structure for i don't know two to six levels of data and then you get down to a leaf and you do a vector retrieval inside that leaf we see people doing that as well there's all kinds of information you can find on our website about how you can because we're json based you can use myligo db to build uh graph structures into your data so that you only have to go one place for a graph database a core database a vector database the embedding

1:04:50re-ranking all in one place so you don't have to try to stitch together multiple tools yourself and have to maintain that over time but we do see people doing that with graph structures it's typically not as deep as like you would traditionally think of as a as a graph database need but like i said it's pretty common to go i don't know half a dozen or maybe maybe a dozen layers that maybe for bigger corpuses of data that are more heavily categorized like some of our retail customers do that with

1:05:21product databases if you think about how a hierarchy of products might appear on a website or on a mobile app they might segment that first by product category using a using more of a graph style and then once you get to a particular product category then do some vector searching within that individual category node are there any applications that you would point to as just being great examples of memory well implemented well our biggest customer right now they

1:05:53haven't been very public about how they did it there's a company called 11 labs out there they have multiple agents per customer this sort of micro agent approach where you get multiple smaller agents at the disposal for individual customers and given the number of customers they have so they 11 labs started life as a model provider that was doing way sophisticated speech to text and text to speech and they built a platform on top of that that is more of like an audio editing suite that they do for that and as part of that they they have a bunch of agents doing all kinds of editing and

1:06:27transformation kinds of things for their customers if you think about the kind of context memory that they have there an important way to do that as well i would worry about any memory architecture that instead of relying on lower costs embedders and re-rankers is relying on multiple passes of the llm to help you categorize and shrink the the corpus of data that you might want to then put into the context window for the larger sort of more functional llm call because then that's just adding up tokens as well i mean that why

1:07:02things like embedders and re-rankers have a lower token cost is again right tool for the right problem yeah that adds a lot of latency too at this point even the small models can reason for quite some time before you actually get your answer from them they can and so all those things add up over time and it's not just tokens but you're you're right the overall latency for the decision loop that you're into makes a

Enterprise build versus buy strategies

1:07:26big difference i think you also have some interesting takes on build versus buy analysis i won't even try to summarize it just give me your your hot takes on how people should be thinking about building versus buying what is mature enough in the ai realm to buy and what do you even if you kind of buy part of it what do you still have to expect that you're going to end up building or customizing enough that it kind of feels like building yeah so i have a good fortune in my job i've been to seven countries this year

1:07:58to talk to probably a hundred different customers about where they are in their ai journey and they tend to fall into sort of three camps camp one is i bought a license for this one tool and i'm done right like my ai strategy is done i bought one thing and while that could be a good starting point it typically is not specific enough to help solve the problems of your overall enterprise then you have a group of people who have done some pocs and maybe a couple of production deployments with

1:08:29varying degrees of success on roi that have struggled with roi i always argue probably picked the wrong problem picking the right problem is really important in this space and the way that i encourage customers to think about this is like what are the top 10 to 15 problems that you have going on in your business right now and of those what do you have good data for i mean you made a comment earlier in this conversation like data quality is a big deal things like bad data quality and bad security posture don't get solved by ai they get amplified by ai so what's your what's your biggest problems

1:09:04what do you have good data for and then what do you already have metrics for around the problem and that's the biggest difference that people that tend to skip over when it comes to problem selection because if you don't already have metrics for how something is performing you won't know if it got better so call center use cases are very popular low-hanging fruit in enterprises because i already know what cost per call is i already know what call volume is based on how i'm already bonusing people in

1:09:35those jobs so if i introduce ai into their workflow and i see those numbers jump i can attribute that change to the ai and i can do a back of the envelope roi we see the same thing with software delivery lifecycle early in the year we saw all kinds of bragging about well now because i've got clod code or i've got ggravity or if i got codex now i can produce five times as much code as i did before anybody who's been doing it any length of time knows that lines of code is a terrible metric to judge the productivity of a set of developers off of instead how how fast are you getting from idea to correction

1:10:12deployment that's the metric that matters they're not not necessarily lines of code so the metrics there matter and that's the difference between the folks that are in that middle state either stuck in sort of poc purgatory and haven't quite gotten to production deployments the biggest difference isn't in how they're applying the tech it's what problems they chose to try to solve and then you've got people that are sort of more advanced that are looking at some of these more sophisticated memory types and things where they're trying to optimize the systems that they have and in some cases they're they're

1:10:43making purchases of larger platforms to help them do that other times they're doing homegrown and like i said we're we're like 18 24 months into this there's there's no there's no one way to do this yet there's there's no lamp stack for agents yet in the way that we have with web development we will get there having lived through that having lived through that life cycle we'll eventually get there but like there's there's no react in angular there's no lamp stack for agents right now you reach in call centers and this is the second time today that this has come to mind i just ordered

1:11:18a pizza last friday night from the local pizza place in my neighborhood shout out once again to greg's pizza in detroit i think it's just a one location business and who answers the phone detroit style pizza is underrated by the way it is this actually is a traditional circular round pizza but i agree after auto it's detroit's biggest export to the world is pizza but even just this one location business right i call and who answers but an ai agent and it was a pretty impressive

1:11:51experience i have to say like very natural conversation not flawless there were a couple little things where especially as i started to get a little adversarial in my testing of it like i asked it what company is powering this ai agent experience that i'm having and it took that as it somehow thought i was ordering pizza for my company so you know there's some kink still but i overall was like very impressed like the happy path i thought worked really well and this has me thinking like boy there must be just a ton of transformation happening at call center businesses

1:12:24how how much are you seeing like there's always this kind of waves of change right where like the first wave is like we give our agents a tool where they can like get help faster and be more accurate but then there's this like next wave or there might be several waves but certainly one that seems like it's coming is you just call and talk to ais right how far along that journey are the customers that you are talking to how much like actual disruption have you seen in that part of people's businesses so far most the fortune 500s that i talked to are doing employee facing use cases with human in

1:13:03the loop for exactly the reasons that you just laid out and for some of the roi reasons that i laid out before so if you think about the employee facing use case you already know what you're bonusing your different employees are and what kpis what key performance indicators you're using to judge whether or not they're doing a good job so like i said i used the example of call centers before but you can think of any job whether you're an insurance adjuster or a line worker in a manufacturing plant or whatever your job might be everybody's got some key performance indicators and if you introduce ai into whatever

1:13:37that workflow is and you see those jump that means that you're you're you figured out some roi but there's also security a data security issue for that as well if you and i are employees for the same company is it great if my salary accidentally leaks to you no but it's way better to have that than if we are two customers of the same company and i get to see your data that is a vp has to get on a plane for that kind of scenario so like the risk reward is higher for customer facing use cases than it is for employee

1:14:13facing use cases so at least where we are now in the ecosystem where i mostly see the fortune 500 the exceptions of course but i see a lot more work going into those employee facing use cases with human in the loop than i see going full autonomous ai with customer facing use cases for you and what you're doing at and for mongo db are you you of course you've got traditional documentation right that human developers can go and read but i'm sure you're also thinking a lot about how do you make

1:14:46your documentation ai friendly and then how do you have an ai that can proxy into that documentation and then potentially you're like packaging up and distributing skills and trying to think about like how to become the database that is like recommended by the agents when somebody's doing something for the first time how is how is your role as like bringing this technology to the world evolved in light of all the different ways that people are discovering and starting their implementation process

The voyage acquisition and shared embedding spaces

1:15:17so all the things you just mentioned are certainly things that not just us but every software company is involved in one way or another the two that i would shine spotlight on are the mcp tools to make it easier for agents to talk to your data in your mongo db instances we had an mcp server for a while and we were now kind of self-hosting some of those as part of atlas as part of the ecosystem and then the other one is the agent skills so think of that is kind of curated system prompts for doing things like data

1:15:50modeling doing things like optimizing operations things that we have well-organized playbooks for that previously we would have as documentation but now we have them as a series of doc of markdown files that you can feed to an agent of your choice to give it sort of the lowdown on different ways that it should approach different tasks and there's maybe the i think it's agent skills we just dropped a couple of months ago has like the top eight it's like it's six to eight different agent skills that we have for things

1:16:23like operations and data modeling to help you do those sorts of things so yeah that's become pretty commonplace these days and we're certainly part of that ecosystem you mentioned this acquisition that mongo db did of voyage a couple of times and i had a kind of just a couple like theoretical questions about that okay one is it's striking that mongo built its own vector db right and there's

1:16:54like a bunch of vector db startups out there so like one could have imagined an alternative scenario where there was an acquisition of a vector db startup but there wasn't there was a acquisition of the model startup and then also just look at the relative value and you said these guys are geniuses and i don't doubt that but the price at even at 220 million dollars which is i'm old enough to remember when that was serious money you know it's it's less than a tenth of a percent of mongo db's overall market cap so i'm like or less than one percent i should say so i'm just kind of

1:17:31wondering like what should we infer from these observations about value and defensibility in the software business over whatever passes for the long term in your mind you know you could tell a story about like infrastructure is the big winner models get commoditized incumbents can defend themselves against startups like what do you think are the right macro lessons to draw from this experience i mean fundamentally what we've always been about has been how can we make the day in a life

1:18:05of a developer easier so we can make it easier for them to build their business logic and spend less time worrying about the plumbing lower in the stack i went through a description earlier in this conversation about how we implemented vector search and for us because we already had the json based it was it was relatively straightforward for us to just okay you add an additional attribute that attribute is your array of floats you generate an index based on that array of flows so that was pretty straightforward for us to add to the existing product and take advantage of the things like

1:18:39the charting the security and the data replication that we already had in the base product that some of the net new vector database companies out there have to catch up and struggle with a little bit so that's why we chose to sort of do vector search the way that we did is because of the flexibility based into built into the document model from the very beginning it was it was pretty straightforward for us to do that but where it was not as straightforward and why we made the voyage acquisition was today you can still use

1:19:10whatever embedding model you want as long as it generates that array of floats you put that array of flow to your document you go create your index and you're off and running but we saw an opportunity to a that the market was seeing embeddings and re-rankers as a commodity whereas we we saw voyage really standing out and like i said before anthropic agrees with us um and we could create these better together stories like the auto embeddings that i mentioned before another one i didn't mention is being able in atlas to manage all your api keys in one place so that you don't have to go to one console for

1:19:42your core data another console for your vector data a third console for your embedding models and a fourth console for your re-rankers you can get all that in one place the management of it is easier over time so if you look at the history of mongodb features and really what our focus has been we're all developers at heart and it's it's about making it easier for the developer ecosystem to learn and operate this these new techniques that we see in application architectures around agents whether it's a rag

1:20:15pipeline or whether it's the agentic memory so how do you make how do you make vector search more approachable how do you reduce that lower lower that learning curve so that more people can learn it more quickly and start to participate in this ecosystem another story that i've heard that i want to just get your reaction to and see if you think it's true in your experience is i was speaking to a founder of a vector db startup one time and i kind of made the case or put it to him that jesus seems to me like the incumbents are going to be able to add a vector

1:20:51aspect to what they're doing before you're going to be able to kind of replace everything that they're doing and so it seems like they're going to have a hard you're going to have a hard time like really displacing them and his answer was well that may be true but most of the data that is coming into our vector database has never been in a database before at all right it's just been sitting out in some data lake or data warehouse or just kind of unstructured piles of documents and now it's coming into a higher level of infrastructure and it's being made more valuable in a way that just

1:21:25wasn't happening at all before sure are you seeing something like that and and what does that look like i also noticed that there's the multimodal embedding model that supports video so video would be potentially a great candidate for the kind of thing that has never been in a database before are you seeing this like phase shift of just like a much greater universe of data coming into mongo than in previous eras we are seeing we are seeing this broader ecosystem of data that that wasn't indexed

1:21:56before because it didn't lend itself to a traditional lexical search the cool thing about the vector search is i mean vector search at its core is taking some piece of data and mapping it into in space into some geographical geometric space and then really all vector search is similarity like where what are the closest vectors to this new thing that i'm searching on and video is a great example audio is a great example unstructured data you just take a bunch of pdfs that you have sitting

1:22:29around in a sharepoint i mean those are good examples as well so yeah i think that there's there's some truth to that that we're it's not just how do i take the data and put it into a database but how do i find it and how do i find in a way that is fast is scalable that's a good quality and that's why the way that we implemented vector search and being able to within the same platform have the combination of the pre-filtering the vector search and the lexical search you feel like that gives us advantage and gives our developer community more levers to pull from than what some of the

1:23:03alternatives are and like i said before because it's it stands on the shoulders of the core product with the with the atlas version you can already deploy that any hyperscaler data center you want with good data resiliency with good security and if you need to to shard that data so that it doesn't meet particular geographies we got all that for for essentially for free because of how we implemented vector search on top of the core product and yeah we're seeing all kinds of different kinds of data get put into those documents in a way that we didn't before

International adoption and global ai maturity

1:23:36you mentioned travels i think so far to seven countries this year where i'm so kind of myopically focused on what's going on in san francisco and silicon valley that i'm mindful that i may be missing important stories or differences in perspective that are going on around the world i try to fill that gap at least with it when it comes to china but it's a big world out there what has stood out to you in your travels this year in terms of differences of perspective on ai usage patterns values you name it could be anything but just kind

1:24:12of what do you think the u.s audience in our inward looking way is missing that the rest of the world is doing i think biggest thing if i turn question on its head just a little bit there's a presumption in other countries that the u.s is ahead and doing things that other people are not and i found the opposite to be true so i mean i live and work in out of cincinnati so u.s is one of those countries i've spent some time in both amsterdam and london earlier in the year but i just did a tour

1:24:45uh that included stops in toronto bingaluru mexico city and sao paulo and the two most sophisticated customers i talked to this year were in mexico city and sao paulo and they assumed when we started their conversation that they had u.s competitors that were doing things that they weren't and like i said the opposite was true i think we've reached a point with some of these technologies that like geographic barriers don't matter nearly as much as they did during like the web app era

1:25:19or or even during the cloud era because for the cloud era like if one of the hypersteelers didn't have a data center in your country yet you were kind of out of luck that's not true anymore like pretty much every country has at least one hypersteeler data center in it and by extension access to models and vector databases and embeddings and re-rankers like access for these things is far better than what i've seen with previous technology revolutions that we've seen so

1:25:50i think i understand why there would be an assumption that u.s companies would be ahead since so many of the bigger ai companies are u.s based but like i said the two coolest things i've seen this were in mexico city and in sao paulo so i think that those geographic barriers to being ahead in the market are starting to disappear what do you think are the barriers why is that i guess is it that the american companies are culturally too conservative to run as fast as some of these

1:26:26international companies are is it that is it like a leapfrog story where they sort of had the international companies i mean had kind of less recent technology investment that they would have to get comfortable replacing or what's what's driving that surprising observation well i think it has more to do with the lack of barriers outside the u.s than it does with any of the behaviors of the companies within the u.s it's just access is democratized now in a way that it wasn't that way that it wasn't for the cloud and for the mobile waves right like if you didn't have

1:27:01the radio towers yet an iphone's not going to do you much good but now we have all this infrastructure that's laid worldwide in a way that we didn't 15 years ago that that makes access to these things far easier than they were before so there's just there's just a wider set of people in a wider set of countries that have access now that they didn't before so i don't think it's any it's not an indictment of the way that i think u.s companies are thinking about it i just think the there that greater access erodes those geographic barriers that we've seen with previous technology

1:27:33waves any closing thoughts any anything you'd want to leave people with or anything i didn't get to that you would want to make sure we touch on well there's one technology thing that we have i didn't get a chance to talk about and that's shared embedding spaces and then there's there's also this notion of just like kind of where we are on this journey together the shared embedding spaces real quick is one of the other ease of use things that the voyage models provide that nobody else does is in january we released four versions of the of the version four of the text model

1:28:09everybody typically releases small medium and large and at those different price that each one has a different price point and a different retrieval quality that comes around with that but we also introduced something called the nano which is open weight and free for anybody to download in off of hugging space and what shared embedding spaces does is each one of those four models shares an embedding space meaning that embeddings that are generated with one of those models they're compatible with the other three and what that lets you set up is you could take your corpus of data

1:28:41and you could embed it using say the large and developments every other embedding model is going to force you to pay tokens during your development cycle to hit against their embedding model so as you're querying as you're taking your queries and you're you're passing them through the embedding model to do your searches but with us you can use the nano there you can run that on your laptop if you wanted to so you could if you wanted to have zero token costs aside from the embedding of the corpus of data

1:29:12but for your queries you could eliminate your your token costs during development in a way that wasn't possible before january so that's not for everybody you do take a little bit of a retrieval hit when you use a different retrieval quality hit when you use a different model but it for some customers that don't want to have to that want to lower their token costs during development this is one way that they can potentially do that so that's that's shared embedding spaces which we introduce in january um and more broadly i want to come back to the idea that we've been building databases for a

1:29:44really long time we have not been building agents for a very long time we're we're going to continue to see iterative improvement and it's going to happen fast like mc published mcp on like monday before thanksgiving two years ago and by march all of their competitors that embraced it as a protocol token maxing first got mentioned in like early march and by april it was over as like a conversation piece like the quickness the speed at which we have these conversations and we go through these

1:30:16cycles of these different features it just it's faster than it ever has been before so the fun part is there's still a lot we all have collectively to learn we're going to all collectively figure out how to make this easier for everybody and you'll see more and more agents in everyday life as a result well as field cto of ai at mongo dv i know you're going to be racing as hard as you can to keep up with all these cycles and we'll continue to look for new updates and more great

1:30:50progress from you p johnson thank you for being part of the cognitive revolution nobody keeps just one thing anymore we used to have to choose one picture for a whole december one letter we couldn't lose now there's a record of every hour boxes stacked to the beams a house so full of yesterday it's swollen at the seams cause forgetting is the hardest part so i held on to it all

1:31:25every summer every borrowed song and i can't find you at all i kept everything everything but you i kept everything everything but you i went looking for you sunday down the rows of all i own

1:32:02everything lies with its own kind everything but you has a home your life is in a box somewhere under years i never sorted through i swear i never lost a thing i just can't get to you cause forgetting is the hardest part so i held on to it all every summer every borrowed song and i can't find you at all

1:32:33i kept everything everything but you i kept everything everything but you last night looking for my keys i found that ferris wheel a ticket folded in a road map next

1:33:06to songs you used to steal nothing in this house is gone it all stays close to what it knows i quit digging started following and the power became a road now when i come looking i don't tear the whole house through i start with one bright summer and let it walk me back to you one thing beside another each one hands me one more down the roads right to your laughter right where it was before

1:33:45cause forgetting is the hardest part and i never needed it at all every summer every borrowed song is a light on down the hall i kept everything everything leads to you i kept everything everything leads to you everything leads to you everything leads to you down the road down the years everything leads to you

1:34:17everything leads to you everything leads me home to you

Show outro and network credits

1:34:29if you're finding value in the show we'd appreciate it if you'd take a moment to share with friends post online write a review on apple podcasts or spotify or just leave it to you If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine

1:35:02Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.