Milliseconds to Match: Criteo's AdTech AI & the Future of Commerce w/ Diarmuid Gill & Liva Ralaivola

Introduction

0:00Hello, and welcome back to The Cognitive Revolution. Today, my guests are Dermot Gill and Liva Ralevola, CTO and VP of Research and head of the AI lab at Criteo, the advertising technology company that powers much of the personalized advertising that we experience on the open internet. I'm also joined by my longtime friend and teammate, Alex Persky-Stern, who took over for me as CEO of Waymark some three years ago now, and has since formed a partnership with Criteo that brings Waymark's AI-powered commercial creation product to Criteo

Modern Digital Advertising

0:32advertisers. We begin with an explanation of how modern digital advertising works and the value that it creates for society. Personally, I tend to emphasize that without the commercial recommendation systems that allow businesses to affordably reach their target customers, a lot of the long-tail small businesses and niche products that we enjoy today simply wouldn't be viable at all. Dermot and Liva, for their part, focus on how ad tech delivers more relevant, engaging experiences and supports free access to information, as well as emphasizing how easy it is

How Ad Tech Works

1:06for individuals to opt out of personalization systems. From there, we dive into how it all works. Criteo has been in business for more than 20 years, and while their AI techniques have naturally evolved with the field, most fundamentally from the earlier era of handcrafted features to the modern era of deep learning, the one constant has been their need for incredible speed. From the time your browser requests a webpage, Criteo has just milliseconds to locate your profile among the billion or so in their

1:36system, and in light of what you're doing right now, decide which one of many millions of products to recommend, and also how much to bid in a real-time auction. It's a deeply challenging problem that

Criteo's AI Techniques

1:48requires lots of pre-computing, but the upshot is that they've developed a highly modular system, powered by multiple foundation models, that supports prolific experimentation on top of cached user and product embeddings. Beyond the core tech, we also discuss Criteo's new partnership with OpenAI, which though still in its infancy, they expect will complement ChatGPT's broad world knowledge with accurate, real-time product inventory information. They tell the story of the company's European roots and share their commitment to privacy, their sense that European compliance

2:21burdens are overstated, their decision to use the same Euro-compliant tech stack globally,

Partnership with OpenAI

2:26and their passionate belief in the European AI ecosystem and talent pool. They also explain why they're confident enough in their moats to publish a lot of their research, and how this helps them attract and retain talent well enough that they're still comfortable publishing the AI labs full 50-person roster to their website. We trade ideas regarding the role that generative AI will play in the expansion of the advertising market and evolution of personalized creative, and they share their admittedly speculative thoughts about how the fundamental value exchange of advertising might

2:59change as human time becomes more valuable and AI agents take on more product discovery and research work. Overall, I think this episode is both an informative look at how modern AI techniques are being used to make high-value commercial recommendations under extreme constraints, and a useful corrective for those who deny the ways in which cutting-edge advertising enriches modern life. With that, I hope you enjoy my conversation with Dermid Gill and Liva Ralevola of Criteo. Dermid Gill and Liva Ralevola, CTO and VP of

3:36Research and Head of the AI Lab at Criteo, welcome to the Cognitive Revolution.

Guest Introduction

3:41Thank you, Nathan. Pleasure to be here. Hi. Thank you. Excited for this conversation. Also excited to welcome my longtime friend and teammate, Alex Prisky-Stern, who's the CEO of Waymark, which we had been building together for a number of years before he took over for me as CEO a few years back. Longtime listeners have heard many asides about Waymark, and Alex is the guy running the show there now. Happy to have Alex here today because Criteo is obviously in the advertising business and bringing a lot of AI to the advertising business

4:12in various ways. And Waymark, under Alex's leadership, has partnered with Criteo to provide some creative solutions as part of that whole package and go into market together.

Data Collection and Usage

4:22Lots to get into. I wanted to start with something that I saw recently that kind of caught my attention and I'm going to get your take on it. It was, I'm sure you've seen this, a Bernie Sanders sit-down across the table with, I believe, Claude having a voice conversation. And it was a bit of a strange tone. It felt a little dated to me in some ways. But the subject of the conversation was Bernie saying to the AI, what do you think Americans need to know about how their data is being collected and how companies are profiling them and how that's all being used? And it kind of

4:57had a, you know, ominous overtone to the whole thing. I think there's probably still a lot of kind of misconceptions or misunderstandings out there about this. But this is part of, you know, always a significant part of the business that Criteo is in. So I would love to hear from you guys, you know, as, as folks who have built it and are doing it today, what, what do you think Americans need to know about how their data is being collected and how it's being understood and how it's being used? And what's the upside of that to, and maybe the downside as well, but what's the upside of that to businesses and consumers?

User Profiles and Data

5:30Yeah. It's a great question. And it's something that we in the industry need to do probably a better job of explaining and demystifying. For me, I think it all boils down to, you know, like a transparency. So, you know, explaining to users where, what data is collected. So for example, with Incredio, we don't collect any personal information. So it's really, you know, a random anonymous ID. And then there's some things around, you know, what products people are interested in, you know, kind of what they've seen, what they like, what they don't like, and so on. And it's all about,

6:04for me, about a value exchange, right? So relevancy. So, you know, a system that knows nothing about you is going to show you random stuff that's irrelevant. The brain has a really great way of filtering out irrelevant stuff, whereas something that's truly interesting for you is way more engaging, it's way more resonant. And for a user, that creates a better experience. Also, I think in advertising, one of the great things is advertising is very much the lubricant that keeps the internet open and free, right? You know, that provides, that allows service

6:37providers the ability to keep their services not behind paywalls. And so there's great utility in that. And advertising is what keeps that going. It's the revenue that those content and service providers get that allows them to provide those great services to the end users. And, you know, providing transparency to users can actually see what's happening and the ability to opt out. That's also something that's very, very important. You know, Cradio was a pioneer with the ad choices icon. So someone can click, and they can see why they saw this ad, and they give some

7:10the ability to opt out. Once you do that, then I think it provides great value to all the participants. We try to be like mentalists. We have very few information, very few cues, and we try to detect what is going to be the most relevant for each consumer and each end user so that in this value exchange, everyone's going to be happy. As a middle party, we have down the road, the ones who have the least data and most challenging tasks in terms of AI. And that's why, actually, on my part, I'm still

7:47because the challenge in terms of machine learning AI is really a big one, and it's the most interesting one.

Double Click on Aspects

7:55Could we do kind of a double click on a couple aspects of that? One being like, if I were to open up my file, I'm not even sure if that's quite the right way to think about it. I'd be interested to know, like, what's in there? And I have, you know, I've occasionally clicked this sort of ad choices thing and seen, oh, you're seeing this ad because you're interested in skincare. And I'm like, okay, my wife got me this one. But I don't really know, you know, that's kind of a high level sort of summary statement of why I'm seeing it. I don't know exactly what is under the hood. I'm also kind

8:27of confused about like, you know, of course, we go see these, you know, we go to websites all the time, and we get this pop up that says, you know, accept cookies, don't accept cookies, what's going on with those cookies. I know there was a big change to the industry. And I think it was kind of driven by Apple a few years ago, maybe it was driven by other parties as well, where the way in which information is gathered and the nature of that information was kind of changed. I think there were some winners and losers from that. I'm not quite sure how that really shook out or if we're back to essentially the status quo ante before those changes were made effectively. But I'd love to hear

9:04kind of just a little bit more concrete description of like, what the data is. And then that obviously feeds into what is the machine learning layer that sits on top of that data look like to make sense of it, you know, obviously, that data becomes the inputs. But I always like to get down to very brass tacks on like, what are the inputs and outputs of the models, so we can really understand like what it is that the AIs are doing for us. Sure. So there's a couple of different ways that I think it works. So first of all, I mentioned, you know, when you arrive on, say you're on a retailer or

9:35on a brand website, so they, you know, kind of using technology like Criteos, we can create a record on the computer called a cookie with a random ID. And the ID doesn't have any personally identifiable information. And so then afterwards, as you kind of continue browsing, so if you look at a product, then afterwards, when you leave that website, and you go browsing the web, they can know that you've shown an interest in that product, right? And they can show you ads for that product, specifically for the same one you've seen. Alternatively, what could also work is if, for

10:08example, you took a look at a mobile phone, well, then that could have you assigned to a group of people who are tech enthusiasts, right? And the type of phone that you look at could be interesting too. So iPhone users have a different profile from Android users from whatever else. So you could be part of a wider audience that could just be seen as Apple enthusiasts or tech enthusiasts and so on. And then when an opportunity, when you're browsing the web, and you're looking at web.com, internet.com, whatever, an opportunity comes for advertisers to bid to make it to pay the website

10:47owner to show the advertisement in front of you. And based on the information that they've got about your previous browsing history, they can then decide whether they want to take this opportunity to show you the same product or show you equivalent products. And then maybe I'll hand it over to Liva to kind of say how you actually do that bid, how you decide whether to show it and add or not. Yeah, precisely. One of the very important things is to being capable of valuing the expectation of

11:17revenue of a placement and knowing that if you are going to place an advertising in that placement, then there's a high probability for you for it to be clicked on or not. And you have to evaluate that. In order to do that, you're going to use machine learning and AI models that are precisely are going to evaluate whether a placement and given some product that we can put on is going to be something that is going to bring revenue. And for that, we precisely do a lot of, we collect all

11:49those data that Jeremy talked about. And there is a huge machinery that we put in place in order to learn from that data. And if I had to summarize the type of problem that we're doing, even though it's a bit more complicated, should we bid or should we not bid on that placement? And we learn a classifier from that. And it's, of course, there's this question that you started with, whether it's

12:21when you click and want to see your information. There's this question about utility and and read that exists. If you want to be very precise in terms of evaluating the value of the placement, then you have to use very sophisticated models. Probably have heard about deep learning models. And the more sophisticated the models are, the less easy it is to understand what they have computed. So there is this trade-off to have. Now we have all in the industry and in

12:55particular, we use deep learning models in order to assess whether placement is good and assess whether this product is going to be relevant for you. So it means that in a way, we have what we have gained in terms of precision and relevancy. We have something to make up in terms of explainability. And just so you know, because we talked a bit before, that's a big topic in terms of research, scientific research, AI research to provide explainability on those

13:28models that are doing crazy stuff. And one of the things that we are looking at as well, but it's not easy like to have both high utility and high explainability. And Jonathan, I have a question I'm interested in here. The idea of the user profile and what this person might be interested in is obviously super, super core. One thing that I think is really interesting is you guys have this open AI partnership, which is super cool. And I know very new. So some of these answers might not exist yet. But one thing that people talk a lot about, of course, is that the queries are much

14:03richer in the context of AI and chat. But something that I haven't heard people talking about is whether the profiles are meaningfully different. Now, if I'm working with quality or an agent, like it knows a lot about me, is that starting to change what how we understand the user? And where do you see that going? So many things. So I did with the first very first thing you said, okay, that's there's still a lot of things to unpack to uncover, we are precisely at this stage. Because of course, there are many questions about privacy, how the data data are defined. That's that question. So, so far, there is no

14:41answer yet. But one thing that I can answer is, though, is that those conversational agents, they provide with a new interface, before you just had essentially the websites, or some apps that you could use, and now they are have that. One thing that is very important is that those models, like, you know, etc, they are very, very good at general reasoning. So they can do some recommendation. And in some ways, you can think that they're going to be able to, and if you ask, okay, I would like to buy shoes, they are going to propose

15:16you shoes that make that are relevant. But one of the things that we do is commerce data, we are going to tell us about what exactly people are interested in. And the big challenge that we have today, is precisely to make the to have the both models, the LLM models that are behind all those conversational agents, with all the models that we have in Criteo that are capable of providing very accurate commerce information. And the challenge in terms of technical challenge is

15:47precisely to merge the two and make some ways, but that's not on our part, from the LLM. They are going to have some information that is going to encode. But one way is precisely not necessarily to have this information, but more to know and see how we can enhance or those models can enhance our commerce models that we've built for years. And that's where we sit as of today. To you, Dermit? Yeah, I think that's exactly right. So the thing about the LLMs, and it's amazing technology,

16:19we're super, super impressed by the power of all of these, I think everyone is. But when those companies, when they train their model, it is true and accurate at that moment in time, commerce data is actually way, way more dynamic, right? And so, for example, you know, kind of, they would not be able to know, for example, that there are flash pricing. So, you know, around Black Friday and so on, where things change very rapidly. They also wouldn't know, for example, things like ruptures in stock, right? So the way that they gather their information is by doing this massive crawling of

16:52the internet. And then at that point in time, when they've updated their model, very quickly, it starts becoming stale, at least from the product point of view. So Cradio has this massive network of 17,000 retailers. We ingest their product data on a daily basis, sometimes multiple times a day. And it means that we always have access to fresh data. So like Leva said, we did this hybrid architecture where an LLM, in partnership with technology provided by a company at Cradio, can ensure that when a user asks for a product, they not only get all the richness that an LLM

17:25can provide, but they can also ensure that it's up to date and it's accurate. Because from a user point of view, it's a very bad experience when you search for a product and it comes back with something, you click true, and the product's either a different price, or it's out of stock, or it's not what you were thinking about. So that's why that hybrid architecture makes so much sense. Yeah. So today, just to make sure I'm getting it, the process is ultimately pretty similar to what you'd have on the open web when you're in the chat interface, maybe with a richer query.

17:55AI is being inserted in a whole bunch of other different places in the stack, but that applies really across all surfaces. So I would say yes and no. In fact, I think where these tools like the LLMs have an ability to elevate the whole experience is in the area of product discovery. And so if you're in the market for a new product and you're in that kind of like trying to dig out a lot more. So I think for the first time ever, we have the ability to provide the end users the experience, the same experience you get when you go into a store and you've got this really kick-ass sales assistant who only cares about

18:30giving you a good experience. Who could answer your questions, who knows the full catalog, who's able to kind of tell you the good and the bad of each product. And then it leaves you with the experience of, you know, when you walk out the store, you feel like that person has really answered what you're looking for. So the LLMs in coordination with the accurate product information can do the same experience where you can actually query, you can ask extra questions, you can drill deep down and it doesn't get tired, it doesn't get bored. And it's always giving you real-time accurate information.

19:01And maybe there is something, maybe that's very technical, but that has been changing. It's how you're going to connect the tools and the thing that we provide with those LLMs. So you've probably heard of the agentic era, the fact that we have MCPs, those protocols that are going to help make like almost transparent the use of already made up tools. And that's something that makes it easier, like to combine those LLMs with what we provide. It's something that, of course, is

19:35technical, maybe nobody cares about that. But in terms of deploying something, it has been a lot easier because before you had to adapt to each surface, to each website, et cetera. But now with those protocols that are coming up, it has been made easier. So that's just our duty to make sure that you are compliant with those protocols. And that's actually what we do. It allows us to surface all the tools that we've built over the years and make them available to.

Core Models and Architecture

20:03Yeah, great point. Hey, we'll continue our interview in a moment after a word from our sponsors. Most billing platforms were built to send invoices and assume your pricing is simple and predictable. But if you're building an AI product, a fintech tool, or a developer platform in 2026, your pricing is anything but. Usage tiers, consumption billing, and bespoke enterprise contracts are now the norm. And you're probably managing it all across disconnected tools and fragmented systems. Sequence handles the entire revenue workflow from contract to cash. Quoting,

20:40invoicing, metering, revenue recognition, plus Sequence agents that automate the manual finance work that usually takes teams days each month, while also helping them to collect cash faster. Companies like Cognition, Incident I.O., Runway, and Open Router use Sequence to run their full revenue process between CRM and ERP without the spreadsheet mess. If your pricing has gotten more complicated than your current billing setup can handle, check out SequenceHQ.com and use the code Cognism in the

21:13source field when you book a public demo to save 20% off year one. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts,

21:45organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For tax season, I asked Claude to help me get organized. It went through my inbox, tracked down 1099s for all 10 of my part-time jobs, and built me a comprehensive report on my expenses and donations. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend

22:20reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind. But then I thought to ask Claude. And sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. So for problems worth solving, get started with Claude at Claude.ai

22:54slash TCR. That's Claude.ai slash TCR. And check out Claude Pro, which includes all of the features mentioned in today's episode. Once more, that's Claude.ai slash TCR. Can I dig in a little bit more on the core models that you guys are using to make predictions? And I'd love to understand the architecture of this better. I think for calibration, anybody who's listening to this feed is going to have a conversational familiarity, at least, with how large language models work. So we know that they're

23:31generating a token at a time. We know that the inputs get embedded. And we know the kind of mechanics of the forward pass and all that stuff, right? And we know it's autoregressive, blah, blah, blah. This strikes me as a very different world. And I don't have nearly as much intuition for what the models are that are driving these things. I do know that they have to be a lot faster because the ad's got to show up really quickly on the page. And then I know also that there's a pretty challenging matching problem in there somewhere because I've got millions of, you've got, we've got, society collectively has

24:07got millions of these profiles of individuals. And then also, as you said, like, into the tens of thousands of advertisers. And, you know, I don't know how much pre-computing is done or whatever, but, you know, it has to happen pretty quick on the load of a page. So could we kind of break down, like, how big are these models? You know, what do the inputs look like? You could imagine something very, very large and sort of very sparse set of inputs. But I guess it, you know, it doesn't seem plausible that it's

24:41like, here's all the websites and here's which ones this user visited, right? That doesn't seem like it works. So there's got to be some sort of tokenization or something that is kind of bringing the user profile into a manageable state size so that it can be used as an input. I'm not even sure if I'm quite asking the right questions here. So, you know, tell me what this looks like under the hood. Yeah. So maybe I can take a quick stab at it and then Liva can take it down into more detail. So Liva actually referenced this earlier. So every single time that we want to,

25:14when we get an opportunity to show an ad, so that opportunity actually goes to multiple different ad tech providers who are all acting as kind of delegates on behalf of the actual advertiser themselves, whether they're brands or advertisers. And so the amount that we bid is based on how valuable that opportunity is to the advertiser. Effectively, how likely the user is to click on that ad and go back to the website and buy the product. And the way we evaluate that is we,

25:45through the mechanism we talked earlier, we see what products the users are interested in, what they've looked at, what they clicked through, what they've seen, what they buy, what they don't buy, and so on. And so as the display opportunity comes up, so we see the ID that we mentioned in the cookie, and then we take a look at all of the different products that that person has seen or whatever audience segments they belong to. And each one, we can say, okay, based on all the different features we put into the model. So, you know, the products, the previous purchase history, the context of the

26:20website, the device they're on, a couple of other things that come in. And there's actually probably, I'm not sure, it's like 150 different features we can take in. And each of those go into this calculating as part of this massive equation, which will tell us the likelihood that person is to click, the likelihood they are to click through to the website and eventually do a purchase. And all of that comes out to a value which we bid. If we win the opportunity, then we have to say, well, which products do we show and how do we do all of that kind of stuff? All of that process gets

26:54done in milliseconds because we use a lot of caching, we've trained the models offline, then the inference happens at real time in really low latency. So one thing that is important regarding all the data that we have, like the websites, the products that were shown, the clicks, etc. And then the model, the question, as I said before, is kind of, let's reduce it to a classification, a binary classification problem. One of the main types for people who have tried to do some machine learning, how you're going to encode and how you're

27:26going to represent the data. So I'm going to do that in two steps. The first one is going to, I'm going to talk about the legacy, all models that we used to have and where we are now. And that's where we've been for a couple of years. Before, there was this question about, so all the products, the website that you visit, you have to encode them. And you have to encode them so that the way you encode to the vector you're going to use to represent all those past information still carries

27:57a meaning. If you just encode them in a silly way, you're going to lose a lot of information. So before that, there was a choice before, because of speed of computation and because you have like crazy intuition about the type of model. It was like a spark representation, like very huge vector with a 2 to the 12 number of inputs with a 1, 0, 1, 0, 1, 0, because you can do very fast

28:27computations on those sparse vectors. But that was a way to represent the data that we used to have. And we just learned from that vector, what is called a logistic regression model. So it's a linear model. You can think of just one neuron with a lot of neurons coming in, if you have the learning analogy. And we used to have that and we learned the model and it was very fast, even though it was sparse. There are many libraries to do like sparse matrices and sparse vectors. But then it was

29:01there was something that was very manual on the features before. And of course, the reasons why, for instance, the Credo iLab and the head of what's created was to say, okay, maybe it's not sustainable to have to craft new features and to think about how we're going to represent data each time, because the cookies can change the information that we have. And for instance, with LLM is going to change. So how can we proceed with more modern techniques? So it was the intent and the goal of the Credo iLab to

29:35bring deep learning. So it was created in 2018. And it was precisely the objective to say, okay, let's go to the next level, not have handcrafted features, but rather have them computed from the data. So now before we had like two to the 12 to the 20s, depending on the encoding sparse vectors. Now, essentially, we have something like between 200 to the 1000 features that are automatically computed by

30:07one of the proprietary algorithms that we have, which is called Deep KNN, that computes deep learning features, on top of which we do learn some other models that are going to do those classification tasks. So maybe the essential thing is to understand is we went from two to 12 sparse vectors to something that is a couple hundreds. And now we are at the next level. Again, we're going with all the things that we that are available with these models that you can download, we're going to the next level,

30:42trying to even be more adapted, adaptable to the data that we're going to process. And that's something that is good as we as we speak. So it's it's a way to find all the models that we have. So that's the types of model that we have. And of course, the thing that is very important is, and that's Jeremy, everything happens in milliseconds. So what we have as a challenge is not only to be accurate, but also to be fast. So that's, that's a nice challenge that we have. To be accurate, to be fast, and then also to do it billions of times a day, right, at huge levels

31:18of stability and reliability. And then the other thing that's a great challenge within our system is when you have the Black Friday, Cyber Monday, the busiest period of the year. So imagine any world in where your piece of technology, you are able to run it at like 300% of what you normally run. Imagine taking your car, and running it for one week at 300 miles an hour, and then back to normal for the rest of the year. And it's the same car, same machinery, and it has to perform exactly the same way.

31:50That sounds like a challenge. The engineers behind it, my hat goes off to them. It's the work they do. One thing for the people listening and for you is that one thing that is important for us is also to share how we work machine learning. You can access blogs that explain the deep K&N methodology, and that explains a little relevancy for the retail media business. And we want to go even deeper.

32:26We have more scientific papers, because we have researchers doing AI science and publishing in conferences. And everything ties up together. And if you want to know more, if you want to dig deeper, if you want to know the sizes of the models, how we train them, what are the losses that we use. We have a bunch of articles online that you can totally download and read to have more information.

32:55I was, of course, reading some of that ahead of this call. And it's super cool to see how you guys have made that both public, but also pretty easy to understand for, you know, I'm not a true technical person. And it was totally easy to follow. Hey, we'll continue our interview in a moment after a word from our sponsors. AI is rapidly moving from assistants to agents, and it's causing a sea change. AI isn't just helping anymore. It's taking action. And here's the reality. You don't get outcomes from agentic AI unless you trust it to operate at scale. That's why AvePoint is building a

33:29control layer for AI. This foundational layer helps you govern what agents can access, secure how they operate, make activity auditable, and recover when something goes wrong. All as one connected system. See every agent, app, and workflow, and what they touch. Govern with policy and guardrails that work at machine speed. And recover quickly so a mistake doesn't become an outage. That control layer creates trust. And trust is what unlocks the right outcomes, letting you automate more work, move faster, and deploy agents

34:03with confidence instead of hesitation. If you're scaling agents and want those outcomes by design, learn more about AvePoint at avpt.co.tcr. That's avpt.co.tcr. Can we talk a little bit more, and this probably is to some degree in the papers that you've put out,

Foundation Models and Embeddings

34:28but the sort of architecture of models and kind of what gets pre-computed. I'm thinking back to an earlier episode I did with the woman who leads AI at Stripe. And they had a pretty interesting strategy that I imagine you might have some similarities to where they train this foundation model for payments. And it's a huge model trained on, I don't know, like a trillion payments or something, you know, massive data set that they have. It strikes me that they can be pretty open about it

35:01because they have the data, everything's flowing through them, and that's not about to change. So they can be, you know, they can afford to be fairly open about techniques. And you guys might be in a similar spot where having the network is the moat. And so you can afford to, you know, tell more of the techniques than maybe other companies could. But one thing I thought was really interesting that they did with this foundation model for payments was instead of trying to use that model for all the different tasks that they have within the company, and there are many, they use the embeddings of a

35:34given payment as the input to other models that may have other inputs as well. But in sort of modularizing things this way, they were able to both like amortize the cost of this model across a ton of different use cases without necessarily having to even anticipate what those use cases would be, and also do something that really just allows the developers like across the company, a lot of freedom to say, okay, now I have, I know I have this really rich signal that I can kind of treat as a

36:08black box. But if I bring that into whatever machine learning task I'm trying to work on right now, they're just seeing like dramatically better results across the board because that signal is so rich. I hadn't really heard that in too many other places. But I wonder if you guys have kind of a similar structure where there are models that sort of, whose outputs or whose encodings, embedding, whatever, feed into a bunch of other models. All right, so many things. So first, before I talk about DeepCannon, and it's precisely one way to embed the

36:42data that we process because embedding is very, very, very important for people listening. Embedding is a way to transform the data that we process, a product or user timelines or a website into a vectors on which you're going to be able to do computation. And as you say, Nathan, the thing that is important is for those vectors capturing a lot of signal. That's very important. We want them to capture a lot of signals. And of course, that's something that the use that you mentioned, the

37:14fact that you have a way to encode all the data so that can be used for other tasks as inputs to other tasks is precisely one of the key to being able to use the most of the data. And to answer your question of foundation models, we do have this program in Crisio working on building the foundation model. And that's very important. As you said, one of the things that we are very open about that, because first, it's not easy to build a foundation model. So that's why all the companies can talk

37:44about the fact that they are doing that. It's not easy. We have the data. And then there is a question of the roadmap, how you're going to do that, whether you want from the get go, you're targeting a big models or you want models that are going to talk to one another. And our ways to approach that is precisely to have many foundation models, not that many, but three or four that are going to compute embeddings either on products or user timelines or etc. Then we're going to have a way to make

38:17those embeddings that are going to be computed by those foundation models available to everyone in the company. And it's just starting to fruition today because people are using that as a hot start, a warm start to train models. And just so you know, one month ago, we had a hackathon, and it's always a way for us to try things. And we made available those foundation model embeddings that were used by many other teams just to say, okay, now I'm going to learn something. I'm going to

38:52learn something. And I do not want to start from scratch. I know that somewhere in the company, there is like gems that can be used and like those vectors that are going to contain a lot of information. And I'm not going to start from scratch. And getting back to what I said before, regarding the manually computing features, it's totally automated. And you have that and we provide that and we have it's a huge, it's a huge project because the question is to being able to refresh

39:22them is being able to version them like, like a software, how you have a new version that is going to break up everything that is using them. And that's a big project. And we know that is the one corner piece that is going to feed all the AI models that we build. Yeah, maybe just to build on that. So, you know, kind of as part of the technology group and, you know, Aliva with his team in the AI lab. So one of the things that we empower is that feature and that product innovation across all of Gradio. So historically, you know, Gradio made its,

39:56you know, its first success as part of what we call retargeting, so lower funnel, but we're expanding our feature set. We're building new products in what we call the mid funnel, also customer acquisition, product discovery, multi-channel. So, you know, on open web, on social CTV, and even in the LLMs. And so the ability to go from zero to performance, right? So to be able to deliver high amounts of return on ad spend or ROAS as the advertiser calls it, anything we can do that can accelerate that process. So the hot start, as Aliva said,

40:30so those foundation models are exactly, instead of going from zero and building everything from scratch, you're already halfway there from a performance point of view. So that's why these models are so exciting for us. And maybe one thing very quickly on how we could, and that's the basics. It's really the basics of how you can use. So imagine you have a product and you embed them with those foundation models, because you have a lot of signals, you know that they were bugs, etc. You do not know at human how it was processed, but you learn something,

41:03you have a model, like a vector. And then, for instance, you want to recommend another piece of, another type of product that is very, normally, the embeddings are well defined, then it's very easy to have a similar product to be recommended, just looking at the similarity of the different vectors. But one thing that you can do is, for instance, you have some information about a user who has seen this product, this other product, this other product, and you have that

41:37for many users. Given one product, what you can do, given that you have this representation for one product, you also have those users, you can look at all the closest users from one product, and it builds you an audience, the people that you can target, just because you have similarities between the vectors that were computed for product, and the ones that were computed for users. And conversely,

42:08you have a user, and you can, it's encoded, let's say it's encoded by all the products that he saw, or she saw, and you're going to have all the products that are around, and you can recommend the product that they're around. So that's a recommendation. So it's very powerful to have those representations, as long as they are semantically meaningful. And that's what we work at. So everything, for the most part, it sounds like is kind of pre-computed. I mean, there's an interesting moment where, like, that most recent action of the user has got to be critical. So

42:44presumably, there's some marginal compute there that has to happen. But if I'm understanding correctly, you have sort of a base user profile, and base product encodings. And those are aligned such that at runtime, it's an inner product of those two vectors. Yeah, basically, everything is done except that comparison. But I guess there is probably some last second update of the user profile as well, based on where they are right now. Yeah, totally, totally. And that's, that's actually a

43:14challenge, how you recompute those embeddings, like live runtime. And that's actually an engineering and technical problem to do that very precise way. And it's it means like having other versions of the big models so that you can compute much faster, but still, you do not hinder the way the similarities are going to be computed. But that's precisely where we put a lot of effort on that's, that's, that's, you just nailed it. That's a question for this small adjustment, where you have to take into

43:47account the latest information, where you cannot rely on offline information. But doing that offline, online, sorry, is the key to what we do. And we've done a lot of experimentation with that over the years, right, trying to find the right balance, if you have the perfect amount of time, you can come up with the perfect result. But sometimes the fast result is good enough, right, in terms of the time constraints you have, especially when you talk about the constraints of real time bidding, where you have to answer in milliseconds. So it's the best possible answer in the times constraints that you have.

44:21Yeah, that's, that's really interesting. I mean, the architecture of this is, it has, it does have a lot in common with the Stripe system. And they have a, you know, a lot of, I guess makes sense, because they have a lot of similar constraints in terms of they've got to respond to, should you approve this transaction or not in? Yeah, so like an incredibly short amount of time. Yeah, yeah, it's, it's a great analogy. How about on the agentic? I don't know if agentic is maybe even the right word yet. But in in just like an open AI context, right, or in any sort of chat context,

44:51where now we've got this additional signal of like, what is the context of the conversation that the user is having? Does that also get treated essentially the same way you get? Do you get like raw text and embed it yourself? Or they send you over an embedded form of the text? And then it becomes like another one of these already embedded inputs that goes into the decision making process?

45:17Okay, I can take it. So, essentially, in Credo, we started to look at those conversational agents, like two years ago, three years ago, we were interested in knowing what was happening. And so I'm not going to talk about the partnerships, because it depends, and it's just building, etc. But in the way we think, and that's my job in the Credo lab, we are kind of envisioning all the different scenarios. One scenario is we have the conversation and work on that and we learn something. And like

45:52two years ago, there was a person in Credo who built kind of the conversational agent on Slack, using the messages that were there about how to use the account strategies to try and answer some questions from clients. And he built from scratch an agent and a model trying to and recommend the answers to troubleshooting problems. So it's, it was not a recording product,

46:23it was recommending a solution to a problem, but it's the same thing. And we also tried something where we actually had very, some kind of summary of the conversation, like just a vector trying to see whether it was as a signal that would help us to build another model and to see whether we could learn from that. It's, of course, less powerful than having conversation, but that's something that we've trying, we're trying. And that's something that actually we are in terms of what I do in the

46:55Creo Lab is we're trying to de-risk what we have to envision about what is going to come up. So, so far, in terms, and I'm going to let Jeremy talk about that, in terms of the partnerships, who we should work with, we're still investigating the right way and the right data to exchange. It's still early days, right? So, you know, kind of, we're very, very happy with the partnership we have with OpenAI. They're a really great team to work with. We're both very much privacy driven, right? We're both very much about user consent and so on. And so, you know, kind of, it's something

47:31that's a core principle of both companies and how, you know, kind of, we handle that. So, we're very much respectful of that. I know OpenAI are exactly like that too. So, you know, kind of, we need to just have only the information we need to be able to show an advertisement at a given time. It's not going to be all the time because not every context makes sense. So, and that's something that we're continuing to partner as we build out this network together. And maybe just a last word on that. You have a team or a bunch of people working on what is

48:07called a trustworthy machine learning. And I think that it has never been as important as today, because precisely the type of information, the volume of information that people are ready to share is like massive. And there's this question, and that's not just a research question anymore. People are like, it's very practical. And we do look at how we can, so, trustworthiness is fighting against hallucination, being sure that it's privacy safe,

48:38being sure that sometimes we're not, or sometimes we do understand that the people we work with are someone to be, that we should recommend to. So that's something that we're looking at. Maybe we're a bit like more on the upstream side, I think, but we know that at some point, yeah, something that is going to be very strong, either in terms of regulation, in terms of, I do not know, but we prefer to be on the same side, on the safe side, and we kind of prepare our weapons to be able to answer at the right time.

49:11And I think, yeah, I totally agree. And I think the fact that we kind of work based in Europe, right, and where, you know, kind of, there's a, definitely with things like GDPR, there's a lot of sensitivity around those. We've always had that as a core principle, as we build out these products. For advertising to be truly useful, it has to be true, trustworthy, right? So the user, you know, we don't want to do anything that's creepy, because if it's creepy, it won't work, right? It has to provide real utility. When the user sees it, they go, okay, that's interesting. That's engaging. If that's the case, and they're more likely to click on it, they're more likely to

49:45click on it, then it means that we're a better value provider for the advertiser. Can you tell a little bit more about, I mean, I think probably a lot of people are surprised to almost an hour into the conversation to hear the company has roots in Europe, is headquartered in Europe. You know, I tend to think of ad tech as a, you know, mostly American phenomenon. But how different is it really trying to operate in Europe versus the United States? And do you, are there things that you actually do differently across jurisdictions based on restrictions that may exist in Europe,

50:20or is it sort of the same approach globally? Like, I think people have a lot of sense that there's, you can't do AI in Europe is kind of what the, you know, I think the first order summary would be from a lot of people. So if you think that's wrong, disavuse us of that notion. So I would say that is fundamentally wrong. It's false because, so I'm from Ireland, and I moved over to France 11 and a half years ago. And the team here is just amazing, right? So the AI lab that Lever runs, the quality of the data scientists we have here is just off the charts.

50:55They're so, so talented, so, so dedicated to what they do. We're definitely being European born, what helped us was growing up in that environment where we had to be very, very careful. Like, it was the first principle for us in terms of how we handled the user data. We, you know, we've got like this contract, you know, this implied contract with the end users. In terms of, you know, kind of how we operate, the US obviously is one of our largest markets. So we do operate in many territories. Everywhere we work, we're very, very respectful of, you know,

51:27local regulations around data and what you can and can't do. And we always go the extra mile. We always are very, very careful to make sure that we're fully compliant. And then some with respect to regulations, because, you know, it's, it's super, super important. And we're a very global company. We have offices in Ann Arbor, in Michigan, in Toronto, you know, across other locations in Europe. So from a development and an AI point of view. So that part, you know, can really take in talent from all across the globe. Léva, maybe you could talk a bit about AI in Europe?

52:04Yeah. French person. I can. Yeah. One thing that I can say is that

52:12here in France, there is like a school of mathematics and computer science, where people are very talented. They are really well trained into doing that. And it's not just AI and in general, and they are very good. They are very good people. And it's just the question of having very good people trained is kind of enough to innovate. It's not a question that there's no question about France, not innovating. And you can hear about Credo, but you heard about Mistral, you heard about

52:45many companies coming from France. And one of the things that is very important in Europe are actually, maybe I'm going to talk about France. And you know, that part of my life, I was a professor at university. So I know students as well. The students, they are very good. They do learn with they do like, and maybe that's something that is not necessarily the best thing. But in France, like mathematics and computer science, I think that are very formal is very important. And maybe

53:16people tend to forget that. But the roots of AI, the reason why you can craft and build models that are meant to answer some specific questions, and you're capable of doing that, of crafting those models, relies on the ability to formalize to the model and to find the right technical tools, either in computer science, in engineering, in more mathematical parts of things. And that's that's, that's, that's just important to innovate. And we do have that in France. And

53:51I do not know the rest of Europe, I do know France, but I do that. In France, we have that. But also, one thing that struck me when I moved here was, if you think about it, like say, ancient mathematics was like a lot of that would come from Greece, but modern mathematics, when you think about it, when you study it, I did engineering in university, you've got all these names Laplace, Lazier, Fermat, all of these Galois, which underpin the whole modern mathematics, which is the basis for

54:22machine learning and AI. And so France has really, really great standing in that area, which companies like Criteo are massively benefited from. I was quite impressed and surprised, actually, in looking up the Criteo AI lab, first of all, just how many people are on the team. I think it's like, I think I saw like 50 faces on the website. And then specifically, I was like, very surprised to see all the faces on the website, because I was thinking, you know, geez, like, I don't see too many AI companies in the US doing that.

54:55I think they're all afraid that if they put their names and faces on the website, that Zuckerberg is going to come calling and the whole thing goes sideways, because they all get offers they can't refuse. Not that I want to complicate your lives. But how are you? How are you thinking about that? How are you able to build and retain a team like this in the age of the Zuckerberg old blank check? So I think I'll go first and leave it you can kind of layer on top. And within Criteo, there is

55:26obviously unbiased, but the culture here is amazing, right? I travel over an hour every day to come into the office because just there's so many cool people that were so good at what they do. And you know, there's something really engaging about that. If you can give these people some really interesting problems to work on in an environment where they're surrounded by like minded people, that's super, super engaging. You know, and the fact that a lot of people here have a long tenure, have been working there for quite a while, it creates that ecosystem that is very, very engaging.

55:58So even if we do have competition from the others, you know, we often welcome that because it puts us on our toes and makes sure that we in the leadership have to make sure that we continue to make Criteo a really, really great place to build your career. And we've been doing a great job for over 20 years now. So, you know, that's something we want to continue doing long into the future. Léva, maybe you can... One of the things that is, we have research scientists, and one of the things that the research scientists and I do, they publish. And one of the things that is true across all

56:32research scientists, I think, and even in other companies, is that they have to make their research

56:43reproducible. It has to be open, etc. And research and science like that. So that's something that you can see, it's kind of, you have to have a presence on the internet if you have, if you are a research scientist. That's more of what I'm talking about on the academic side of things. The people who are called scientists, usually they have their own website, they have their faces. And that's what we

57:15want them to have here in Criteo as well to do research as other scientists from universities, from other companies. So that's the reason why. And then there's the thing that Jeremy shared, is up to find challenging problems, challenging topics to work on, that they can say, okay, it's good to be here to try and do science in Criteo, because the problems are very few, they are not easy,

57:46and it allows them to connect something that might be very upstream to something that can be also deployed. Maybe not yet, not today, but in one year or two years. And that's very important. And maybe one other thing, just to build on that, Leva's team also works very close with a lot of the kind of the academic institutions in France and, you know, across Europe. And we sponsor PhD students, they'll come in, they'll work alongside the existing research team, they work on real world projects,

58:16and so it gives them real, concrete experience. And many of those PhD students actually become full-time employees. Not all do, but that's okay too. They publish their research, you know, which is good for their career. And this is a great way to keep that pipeline going, and it keeps us very well connected with the wider AI ecosystem as well. Yeah. Certainly the ability to publish one's work in today's world is a differentiated part of the offer. I've got one episode that has been recorded three plus months now with somebody at one of the, let's say, frontier companies that we just cannot get approved. And it's great stuff,

58:53brilliant work. We'll see if it ever sees the light of day. Maybe just one more beat on the regulatory environments. And I would separate here, following the rules, which you've, you know, I think clearly stated that you're committed to doing from advising on like what the rules should be. Do you think that there is like a, I guess, first of all, do you think there is a meaningful difference between the environments in Europe and the United States when it comes to like,

59:25an individual's rights, I guess is one way to think about it, but maybe rights, what I really am trying to get at is like, who has it better? You know, do people in Europe actually have meaningfully better protection that I should envy? Or do I get like meaningfully better ads that people in Europe should envy? Or is this all much ado about nothing? Who should be changing on what margins their rules to, you know, just to better serve their citizens? Do you have a point of view on that? So we actually engage quite a lot with the authorities, you know, kind of in the EU,

59:58with the different data protection offices, the DPOs in different parts of Europe, as well as in the US. And, you know, kind of, we're very much advocating on behalf of the end user, right, to ensure that it's about, you know, transparency, it's about user consent, ensuring that users have a way to be able to opt in and opt out. And that's why, you know, we had the cookie consent message long before it was even regulatory. And we also kind of push through on ensuring that we live in a system where there's

1:00:29a fair value exchange, right, so that the user feels that they have, you know, kind of proper, you know, kind of use of their data, and that they get value back, right, to free content, through free services, and so on. In a world where all of that disappears, right, then you take away all the value for the advertisers. So if an advertiser who's spending money to try and increase their sales, if they're not getting that value back, if their products are being shown in front of people who have no interest, then that does nothing for their business, and they're not

1:01:01going to spend. If they don't spend, then those people who provide that rich stream of, you know, of services and of websites and of content, they have to monetize some other way. So then what they do is they have to put up paywalls. That's not in the interest of the end user. So as long as there's fair value exchange, as long as we're transparent about what we do, then I believe that's really important. So back to the question about US versus EU, to a large degree, a lot of that stuff's been equalized, right? A lot of those things like, you know, in California, there was a lot of changes to

1:01:35CCPA and CCPR and so on. And they're very much inspired by each other. We try and build a global solution. So you know, kind of what we've done in Europe, we use the same approach globally. So it's not like, you know, we try and be looser elsewhere. We really believe that by having that principle and by being born in Europe, it means that we have a solution that can work pretty much everywhere. Cool. That's really interesting. Let's talk about creative a little bit. I think I'll invite you to help lead this part of the conversation. But it is interesting that we've

1:02:12made it this far and not really talked about creative. Obviously, there's a lot of different formats. I don't even know today if... I guess a couple of ways at least I would want to come at this. One is like, I think the number I saw was Criteo has 17,000 advertisers. In our experience at Waymark, we've kind of seen often lack of creative is one of the biggest barriers to new advertisers signing up with, you know, a platform like Criteo or for that matter, Meta or Google or what have you. I wonder if that's, if you see that similarly, like, is that a core barrier to

1:02:46market expansion? And then, obviously, we're in this moment of, like, cost of creative of some quality is dropping precipitously. And I don't know if there is a dynamic layer to the creative. Like, when we're doing these kind of matching auction prediction sort of things today, is creative an input to that decision making? Do advertisers, like, have multiple creatives that you're sort of scoring or, you know, embedding and kind of using to drive outcomes? Or is that

1:03:18still a frontier where perhaps because they don't have enough options, you can't do that in many cases? Or I don't know for what other reason, but it seems like in the future, we sort of imagine, like, everything's going to be highly personalized, you know, the ads are going to be, you know, much more talking to us as individuals. It seems like you have the infrastructure to do that, but maybe just the creative is not there. What do you think is the future of creative? Yeah, I think that's one of the most exciting areas in the whole generative AI space. Kind of

1:03:48the things that you can create with these next generation models is just insane. And I think they have the possibility to create even more engaging advertisements, right? You know, you talked to Nathan about the hyper-personalized. I think that's super, super powerful. And people will see it and they go, wow, okay, that's exactly the product I'm looking for. And it kind of makes it much, much more interesting and much more engaging for them. The other part as well is, you touched on the point there, with this technology, with the ability to democratize

1:04:21that content creation. So before where, you know, kind of the mid to long tail would have been very much cut out of the picture because it was just out of their reach, out of their means to be able to to create that high quality content that is now becoming more and more accessible. Gradio recently launched a self-service product called Gradio Gold. And this is like really making it easier and easier for those advertisers who are very often, sometimes much smaller, to be able to create a very engaging campaigns of really great creatives

1:04:55that will help drive and grow their business. That's super, super important. I mean, it's really, really enabled by platforms like Waymark, who's a great partner of ours. Thanks for the shout out. We love it. Yeah, on creative, so two things. First, there's something that we had been doing for years in Creo is dynamic creation with some templates, some visual assets, and we had to come up with by branch them online. So that was something like you have Legos and it's going to put you an ad,

1:05:28to craft you an ad. In terms of duration of those ads, and with Gerag TVI, we're not at the level of, in terms of speed, there is no way, unless you're ready to like, wait for five seconds or to the page to load. We're not there. So we have to find a way again to balance things between something that is going to be done offline and online. So we're going to kind of the Legos that I talked about,

1:05:58the visual assets, they can be generated either by us or by our partners with Gerag TVI. And then it's up to us to arrange that at runtime, depending, using our, your engine actually that is capable of arranging that online. But maybe at some point, one day, it's going to be very quick and maybe distribute something that is not going to be like by us. But one day in the future, it's going to be computed by maybe your TV or your mobile phone, like online, and it's going to be very

1:06:32quick, but we're not there yet. Because you've tried probably to use some of those that he trying to build up an image, it's not something that then you are from that, but we're not there yet. At some point, maybe in two years, three years, we're going to have to be able to have that. Yeah, do you have a perspective? I'm interested in the level of personalization that you see the infrastructure supporting, I think, you know, there's, of course, the level of like, the user experience that you've talked about a lot, which is super important, you don't want

1:07:05it to be super creepy. But when you think about the level of personalization, that's at least possible infrastructurally, do you think it's going to be literally at the individual level, we could tell you a story that is specific to you? Or do you imagine it being more audience and context level and kind of where do you see that living? For me, I think, I think you're right, I think it would be mostly more at the audience level, right? Because the point is going way too hyper localized. I'm not sure there's huge utility for the end user for the brand either, the brands and the advertisers will still want to retain

1:07:40levels of control over, you know, the look and feel and, you know, kind of how their brand is showing up. So within those kind of parameters, and I think it's mostly, you know, on an audience basis, I think that's kind of at that level, I can see it working down to hyper, you know, personalized, you know, where every single person sees a different thing. I'm not sure there's huge utility there from the point of view of driving more sales or getting more products in front of the end users. Something that I would like to see actually, at some point is to see those models to be

1:08:17to have sizes that allow them to be embedded in some like devices in glasses in. And if at some point we arrive at that level, then maybe the computation is going to be shared, we're going to provide something at level, and maybe on the personal device, something that is not going to be seen by us, but just the personal device is going to tweak the things at the very end saying, okay, actually, it was something that was proposed by I don't know us, but I know you,

1:08:49that's my device. And I have on that device something that is really personalized. And on that device, the very privacy sure privacy say, we're going to record something, but just on the device. So maybe so that's my dream. Yeah, I love it. Super interesting. Maybe there's going to be a share something that is going to be shared in terms of computation and maybe the personalization is not going to happen on our side, but at the end user side.

1:09:19Yeah, I would agree with that. Where I could see this going is where the action is actually triggered by the end user, the consumer, right? And so the good use case is, you know, kind of okay. So I get invited to a wedding in the south of France in the summer. I want to know what looks good. So I can ask, you know, the AI, can you recommend some outfits? And then you can see a virtual try on, stuff like that. The usual example, well, if you want to see how something would look in your home. So that's very much that's engaged by the user. So they're kicking it off themselves.

1:09:52And because they've done that, then it's not creepy. It's their action. That's it. That's that's initiated this. And that for me is actually where hyper personalization really comes back from where that really makes sense. And that's great. And again, go back to what I talked about earlier. It's the utility. It's trustful, it's useful, and it's providing value. And it could help give a far better experience for all involved. Yeah, that's super interesting. And it seems like it fits with what we were talking about earlier, too, like in a chat or agent context, where we're actually getting way more comfortable

1:10:25with what would be creepy in any other context, letting it do that personalization for you. It's super, super interesting. One character who's asking for it, then I think that's perfectly cool. Yeah. Yeah. Cool. Super interesting. One other thing I'm interested in kind of a long, it's a different angle on the personalization, but contextualization. I know CTV isn't your historical bread and butter, but a growing concern. And I think one of the best documented performance gains is when the creative of the advertising matches the creative of the context.

1:10:59I'm sure that's also true in some other contexts, but really true on the TV. Another place where you can imagine lots and lots of different variations that match the tone and style of the particular, not only the genre or the movie, but the very specific moment in the movie. Is that stuff that you have started to think about and bake into the way that you run your models? Yeah. I think for me, it has to be, you know, what's super important about advertising is it should not be intrusive to the user experience. It has to kind of feel kind of seamless and something

1:11:35that's getting in the way of the user experience of the content. And that also applies to websites as much as it does the connected TV. And, you know, I think we've all seen those demos of, you know, where you get advertisements inserted into the content, whether it's sports or it's your favorite sitcom and so on. And I don't know if we're quite ready for something like that yet. I think you have to have something where, you know, kind of the user feels, you know, kind of that it's relevant, that it's not interfering with their experience of the content. And so, you know, with video and connected TV, there's pre-roll, there's mid-roll, there's post-roll,

1:12:08which is, you know, kind of a different way of experiencing it. So as long as it's not intrusive, I think the users will be open. I think that's kind of more of the guiding principle there.

1:12:19Makes sense. I have one other question that I'm interested in, which is, so I think this is a really developing moment right now, as for the first time ever, creative can really be technically powered, where it's always had a very human-in-the-loop requirement. How are you guys thinking about, as a technical organization, what your role in the creative is, not only right now, but over these next few years? I think that, so as of today, the way we do approach those creatives so that we are sure that

1:12:54the quality of the thing that we provide is to build our own creative models, because they're very costly, they're very expensive, etc. But I think that it's something that is going to be, yeah, of course, it's going to be key. I don't know, it's not my own turf. So I think that it's that, yeah, we have a team that is dedicated to do that. They have a program there, like a not a one-year program, but like two years, two years, three-year program to integrate those

1:13:27creatives. But they used to be in the Creative by Lab, but now they're not anymore. So maybe I'm going to learn a bit on that one. Yeah, so for me, I think, ultimately, you know, kind of there's a few different constraints. So, for example, brand guidelines. So the brands themselves, the original advertiser or the retailer, they've got their own kind of look and feel and the way that they want these things to show up. As well, ultimately, we want to build advertising that works, advertising that brings value for the advertisers, so they feel that their money is well spent with us.

1:14:02And so, you know, kind of, I think, within those parameters, we try many different things. So Liva talked about, you know, kind of the way we did DCO before, which was really one of the great ingredients of our initial success. You layer on top the generative AI capabilities on top of that, and the possibilities are endless. It's really, really, the potential is just that amazing. Yeah. One, maybe just a word that you used, Alex, and it's true for everything that is going on with AI today,

1:14:33is human-in-the-loop.

1:14:38That's the key, actually. Not just as creative, but for everything that we've been talking about. And the big challenge is being sure that you put the human at the point. That's very, very important. You should not strip the user away from his right to decide. If you do that, to automate everything, it's not going to work. It's going to be a problem in terms of liability, or responsibility, et cetera, et cetera. But at the same time, you're going to feel as a human being

1:15:09allowed to make any decision. So just so you know, the question that we have every day, when we build the model is, where is the human? How do we learn so that, and how do we learn the model so that the human is still a place, and we have to spot the right place? And just missing that could be very detrimental to the project that we have. And it's true for creatives, for the bidding models, it's true for the agent stuff, et cetera. So that's the big thing that is very important to actually not just us, but everyone building new tools. It's the human. And that's why

1:15:42those conversion agents work so well. It's still heavily relying on human interaction. So that's very important. Yeah. Well, here's a small one, but a curious one, and it is kind of relevant to our business. So it's top of mind for that reason, but also in general, right? We've been talking about like all these different touch points. Where are we today on cross device understanding of who somebody is? I could imagine that that could be like very algorithmic and sort of deterministic,

1:16:13but it occurs to me that maybe that's another AI question in today's world. Who is one person and who's a different person when some devices are shared and some are on mobile networks and, you know, all the complication there. That was supposed to be the short one, by the way. So I'm just focusing that for, and I can resume it long, but I don't want to keep you guys long if you have something next. Yeah. Yeah. So there's a few different aspects to it. So again, it goes back to user consent, right? So the user needs to be okay with this because, you know, there's shared Wi-Fi at home and all this kind of stuff. And you want to make sure that

1:16:46that you're showing the advertisement to the right household member. So there's different ways. So, you know, kind of there's the Wi-Fi signal, but there's the network that you're connected to. There's also, if you're logged in on multiple devices and that, you know, kind of, for example, on social or on other ways. So that can be used deterministically to say, yes, it is the same person on multiple devices. The probabilistic way can be used for certain things, but only in the kind of very, very untargeted kind of way, because if somebody has opted out of advertising, if they've

1:17:16gone through that, then you don't want to re-show them advertising even through a different way. So that's something that is something we always kind of keep in front of mind. Cool. All right. Last big one for me, zooming out kind of as far as possible. I see, I mean, in general, with this whole AI wave, there's like dramatic uncertainty about what's going to happen, right? How powerful the AIs are going to get and how disruptive it's going to be. And, you know, do we need a whole new social contract, et cetera? Specifically in the advertising

1:17:49space, I feel like I see two trends that kind of counter each other. One is that if all goes well, we should all be a lot richer and the value of time should go up. And that would be really good for the advertising market in the sense that fundamentally you're competing to have some little share of people's time. And the more, the richer they are, the more valuable their time is, you know, the more that's going to cost. So that's an advantage, presumably. But then the other thing that people also sort of see is like, maybe search costs and matching costs could really

1:18:22drop in a lot of ways if we all have our AIs going out and like vetting us, you know, much broader portions of the world than we previously could, right? Today, I'm like, I can only evaluate so many shoes. But maybe in the AI era, I could have an AI that goes out and really evaluates, you know, in a comprehensive way, every possible shoe. And like, therefore, my product decision making is less about like, who was willing to bid on my time and more about how much time was I willing to have my AI invest on my behalf to go out and figure out what to do. I don't know. That's more

1:18:53of a prompt, I guess, than a question. But what do you think of those trends? Are there other big trends that you see as kind of being, you know, huge factors? And where do you think we are in advertising in, say, five years time, or the singularity, whichever comes first? I'm not sure about the singularity. I have my doubts about that one. So for me, where I see this new generation of AI really kind of helping to, you know, kind of increase the amount of value to the end user, you mentioned time, Nathan, you know, kind of maybe there's a value as well, is for

1:19:27that kind of product that you're looking for, right? If you're kind of a very specific thing in mind, you know, where you don't know where to get it, you're able to then query these LLMs or whatever interface, you know, whatever LLMs become, and partnering with that really kind of rich commerce data. You say, here's my criteria. I'm, you know, I don't care about the price. I just want the best product possible. And, you know, you explain what it is you want. And connecting all of these systems has the possibility to get a really, really, you know, great result when you get that great

1:20:02result and you're quite happy to buy it. I think that's the kind of utility where helping people discover, and not only just the product they're looking for, but what goes with it, how you can actually enhance it and so on. And that kind of, when you see the way that they're working today and where they're going and how fast they're improving, I think that's incredibly exciting, right? And it can provide a really, really great consumer experience for the retailers to be able to engage in that. That allows them to get their products in front of the end users in a way that they couldn't do before, you know, because some of the advertising before, but a lot of it was

1:20:36guesswork. Now it becomes really, really focused and you're really kind of making sure that you get all the information that the end user needs to be able to buy that product, whether it's the shoes you're looking for, or that new piece of tech that kind of you want to invest in. I think it really, really gives a very enhanced way for the end users to discover products. Yeah, just first, or the prompt, because it's very deep. One of the things is that

1:21:08there is something that is going to happen so far. In a way, people are exposed to advertising. You might make the choice not to be exposed, but you are exposed in a way. I think that with the advances in AI, those companions or those assistants that you're going to have with all those LLMs, platforms, et cetera, maybe at some point you're going to be wanting advertising.

1:21:39The exchange is going to change a bit because the advertising is also accessing or discovering the right products, et cetera. And maybe the things are going to happen behind the scene, like you're going to turn a knob, say to your assistant, okay, I want to go or to see shoes, but I want you as an assistant to be exposed or to look at 10 or 12 or a hundred different shoes. And I want you to select among those 12 shoes, six of them, but I want to be exposed to six of

1:22:19them. And I want to be exposed to advertising. And I want to choose from that. So the, there is a new actor that there might be a new actor, a new intermediary in the, in the advertising business. And people might be wanting to be exposed. So that again, to have the choice as a human to say, okay, I want to be, because having the choice of different shoes is having an advice, advertising in front. So maybe they want, they will be saying, I want to see, I want a trip. I want five different

1:22:52trips, show me five different trips, but I won't be, I want to be shown five different trips, not more, not less. And then I'm going to choose. So maybe it's going to change something in the way advertising is to be, is going to be, um, experienced just because, uh, the quality and the ability of the system to, uh, filter and to select. So maybe there is a change like that. So you say that in five years, maybe you will be in, in that situation where we're going to say, okay, show me, uh, show me some, some advertising. So, yeah.

1:23:25Yeah. Yeah. That's like the collapse of search and advertising. If the advertising is good enough, it might be better than search. Yeah. Yeah, exactly. Interesting. Cool. Well, that's a great note to end on Alex Prisky Stern, Jeremy Gill, and Liva Ralevola. Thank you for being part of the cognitive revolution. Great to be here. Thanks, Nathan.

1:23:55Thank you. Thank you. Thank you. Mentalist in the middle, reading smoke a mirror. No name, no face, just a cookie getting clearer. 17,000 shelves, fresh catalog, a river LLM knows the world, but the price we deliver La classe, la grunge, firma and gala

1:24:26Modern math on modern drums Ooh la la, voila, spas 2 to the 12th Now a dense hot start Deep in and in and begin Know the shape of your car Billions of bits a day, milliseconds on the clock 300 miles an hour, same car, never stop We the mentalists in the middle Read a signal through the riddles Best answer in the time we got Accurate fast, billions a shot Cognitive revolution on the wire French touch on the flames, dope the fire

1:24:58If it's creepy, it won't work, keep it clean Human in the loop in between Ask the agent for a wedding fit South of France in June Virtual triumph spinning like a daft punk tune Not hyperlocal audience, but sharper than before Mid-funnel, open web, CTV and more Trustworthy machinery, privacy on deck GDPR in the DNA, not a line we checked And over to Toronto, Paris to the bay

1:25:29Off the charts, the scientists, that's how we play Show me five different trips Let the agent decide Behind the scenes on your device Personalize inside We the mentalists in the middle Read a signal through the riddles Best answer in the time we got Accurate fast, billions a shot Cognitive revolution on the wire French touch on the flames, dope the fire If it's creepy, it won't work, keep it clean Human in the loop in between

1:26:00Where is the human, where is the hand Spot the right place where the people still stand Collapse of searching, advertising Merging in the glow Asked to be shown, that's the future we know Show me five, show me six Let me assist and choose Turn it up, turn it up We ain't here to We the mentalists in the middle Read a signal through the riddle

1:26:32Best answer in the time we got Accurate fast, billions a shot Cognitive revolution on the wire French touch on the flames, dope the fire If it's creepy, it won't work, keep it clean Human in the loop in between Signaling the static

1:27:04Mentalists in the middle If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai,

1:27:34or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.