337 - Cognitive Surrender - Gideon Nave and Steven D. Shaw

0:00You know, one of the nice things about Hertz right now is that they have a whole fleet of new cars. So whatever trip you're planning, it's easy to find a new car that fits your adventure. Heading out for a sunny drive, then a new convertible with very little roof is for you. Considering a fishing trip with your Mini-Me, then you need a new minivan that's anything but mini. And if you're planning on some stargazing, a new SUV will give you plenty of space. Get paired with our newest fleet yet at Hertz.com. You can go to kitted, K-I-T-T-E-D dot shop and use the code SMART50, S-M-A-R-T-5-0 at checkout,

0:37and you will get half off a set of thinking superpowers in a box. If you want to know more about what I'm talking about, check it out, middle of the show.

0:58Welcome to the You Are Not So Smart podcast, episode 337. Proceed straight.

1:35There's a scene in the television series The Office, the U.S. version of The Office, in which two characters, Michael, the boss, and Dwight, one of his employees, are in a car on a work trip using GPS, something akin to Google Maps or Apple Maps. And Michael is driving, Dwight is in the passenger seat, and the GPS tells them they should turn right, even though the road that goes to the right is visible ahead of them.

2:06And if they were to turn to the right where they are at that moment in their drive, they would crash through a fence and go straight into a lake. And Dwight basically tells Michael to override the GPS, and Michael tells him, no, the machine is right, the machine knows. 307, maybe it's a shortcut, Dwight, it said go to the right. It can't mean that, there's a lake there. I think it knows where it is going. This is the lake. The machine knows. This is the lake. Stop yelling at me.

2:37No, it's up there. There's no room. Here.

2:41Dwight was doing what we would say is offloading, right? He's using the GPS as a tool to do something that he cannot do as easily and as quickly. Michael, on the other hand, is completely giving up his critical thinking and just following blindly what the GPS tells him to do. And this is leading to this catastrophic error. Remain calm. I have trained for this. Okay. Exit the window. Now, most of us, when we use GPS, we are offloading.

3:11We are not surrendering. And in some rare cases, I mean, the story of the office is, to some degree, a reflection of maybe something that happens in our reality when we really have no idea where we're going. We sometimes can trust the machine more. But to this degree, when we are not even using our critical thinking when we are about to drive into a lake, that's like a complete surrender. So I would just put forward this distinction between these two different processes. Here we go.

3:42Make a U-turn, if possible. Look out for a lake, church!

3:56The person who was just speaking, that was Dr. Gideon Naveh, a doctor of computation and neural systems who studied at Caltech and is now a professor and researcher at the Wharton School of Management. And what he is describing here is an example of something he and his colleague, Dr. Stephen D. Shaw, who you will meet shortly, are calling cognitive surrender. And this office episode example is the example that Dr. Naveh gave me when he wanted to quickly

4:28and simply define that term, cognitive surrender, in contrast to another term used by the sciences that study the brain and the mind called cognitive offloading. When you use a calculator, a spreadsheet, even a notebook or a to-do list, that's cognitive offloading. It's the act of using something external or even using some kind of action or behavior to aid in your cognition. You alter the requirements needed to process information to reduce the cognitive demand

5:00of a particular task. That's cognitive offloading. Cognitive surrender is something else. Here we go! Make a U-turn, if possible. My name is David McCraney. This is the You Are Not So Smart podcast. And in this episode, we are exploring something that each of us will need to become familiar with. If we are going to add AI, LLMs, digital assistants, and other similar forms of technology to our

5:32lives and to our workflows. Or if we choose to not add them to our lives and our work and just intend to navigate a world in which other people are using those things, we're still going to need to become familiar with what we're going to talk about today. Either way, or some way in between, our guests in this episode, two scientists who recently released a study into all of this, suggest that your critical thinking toolkit will need to be updated to account for some habits in our cognition that large language models,

6:06also known as generative AIs, stuff like ChatGPT, Gemini, Claude, and so on, tend to trigger. Habits that can produce serious negative outcomes. And that's what they found when they did research into this. And that research was just published under the title, Thinking, Fast, Slow, and Artificial, How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender.

6:39Let's meet the authors of that paper. Here is Dr. Gideon Nave. Hey, so I'm a professor of marketing at Porton. I have a quite unusual trajectory, let's say, in academia. I started as an electrical engineer, actually learning to train machine learning models and all sorts of other things of this sort when it was a very small niche in the engineering school. And then I figured out that I'm actually more interested in the human brain than in developing

7:10computers that are capable of imitating it. I always tell my students that we're kind of humbled by AI now, but the brain does many things better than AI with far less energy and with far less training sample. And I think we still have a lot to learn from the brain, even if something that is consuming an entire power plant a day is doing better than it.

7:42And this is Dr. Stephen D. Shaw. I'm a postdoctoral researcher at the Wharton School right now. And my academic, you know, undergrad was more in the biological sciences. I studied microbiology and then genetics and neuroscience. For a while, I was doing research on the neuroscience of music and beat perception. And then I went to graduate school for a PhD at the University of Michigan, go blue. And, you know, now, now I'm here working on cognitive surrender to AI.

8:16Yeah. Yeah. So, Nave and Shaw, these are two scientists who are deeply fascinated with cognition, with minds, with brains, with artificial intelligence, the interplay between brains and what we are currently calling artificial intelligence. And they were deeply concerned with how readily people seem to trust the current models, chat GPT, Claude, Grok, Gemini, maybe even Microsoft Copilot.

8:46And the key word here is trust. This is what they were discussing back and forth in their academic lounge areas and labs and such trust of these models. Trust is a very human mix of cognition, emotion, awareness, conscious subjective processing of perceptual modalities and that sort of thing. It's trust. It's a flesh and blood human estimation. It's a feature of human to human relationships.

9:18And yet, people often find themselves trusting what we are currently calling AIs. And I keep referring to them that way because they aren't quite at the level of our sci-fi concepts of artificial intelligence. LLMs are still very far away from Lieutenant Data of Star Trek fame or the replicants in Blade Runner, but they are very good at emulating what we might imagine would be the output of those

9:50sorts of artificial intelligences. Even though it's just words on a screen, they're generating a sense of trust, a sense of agency. And we engage with these cues through very ancient mechanisms of social interaction. And we can treat them as if they are friendly and authoritative, which has an impact on our cognition. And more and more people are more and more often using modern day AI when working on tasks that involve reasoning, not just sorting lists or planning out things.

10:24They're using it to engage in critical thinking and problem solving, to facilitate decision making. And we're beginning to use these outside of a professional setting, using them to optimize child care, to get relationship advice, to work through financial issues, to make workplace decisions outside of the workplace. So, and we're going to deep dive into all of this in a moment, Naveh and Shaw conducted a

10:56study into what happens when people use AIs in that way to go beyond a super advanced search engine and use it to aid their intuition. And to summarize what they discovered, even when the AI was very, very wrong, once people had asked it for help and it had given what seemed like a helpful, authoritative answer, people tended to engage in cognitive surrender.

11:27They drove into the lake. In other words, they went with the answer and didn't even consider if it was right or wrong. And what makes cognitive surrender so strange is that it involves two errors, two thinking mistakes. Here is Stephen Shaw's description of those two mistakes. One. Adoption of AI outputs without verification. And two. Treating AI's outputs as one's own.

11:59During a back and forth with an AI, Naveh and Shaw found that not only were research participants prone to setting aside their critical thinking, their skepticism, their fact-checking instincts, and in so doing, readily believing its output was accurate, even when it was not. Somewhere in that exchange of prompts and responses, we can lose track of where the information is coming from. Because in the study, the participants took ownership of the AI-generated answers, which

12:32suggests that when AIs produce insights or conclusions or advice, we may sometimes treat it as if it comes from within. As if the AI's simulated aha moments are our aha moments. So, how is this happening? Why is this happening? And what should we take away from all of that? Well, we're going to get into it. All of that, after this commercial break. All new drinks are now at McDonald's, like the Strawberry Watermelon Refresher and the

13:19Spriteberry Blast topped with cold foam. Who knew ice cold drinks could be so fire? Try them all, now at McDonald's. Refreshers contain caffeine. You know, one of the nice things about Hertz right now is that they have a whole fleet of new cars. So, whatever trip you're planning, it's easy to find a new car that fits your adventure. Heading out for a sunny drive? Then a new convertible with very little roof is for you. Considering a fishing trip with your Mini-Me? Then you need a new minivan that's anything but mini.

13:49And if you're planning on some stargazing, a new SUV will give you plenty of space. Get paired with our newest fleet yet at Hertz.com.

13:59The School of Thought. I love this place. I've been a fan of the School of Thought for years. It's a non-profit organization. They provide free creative commons, critical thinking resources to more than 30 million people worldwide. And their mission is to help popularize critical thinking, reason, media literacy, scientific literacy, and a desire to understand things deeply via intellectual humility. So, you can see why I would totally be into something like this. The founders of the School of Thought have just launched something new called Kitted Thinking

14:31Tools. K-I-T-T-E-D Thinking Tools. And the way this works is you go to the website, you pick out the kit that you want, and there's tons of them. And the School of Thought will send you a kit of very nice, beautifully designed, well-curated, high-quality, each one about double the size of a playing card, matte cello 400 GSM stock prompt cards, and a nice magnetically latching box that you can use to facilitate workshops. Level up brainstorming and creative thinking sessions.

15:01Optimize user and customer experience and design. Can elevate strategic planning and decision making, mitigate risks and liabilities, and much, much more. And each kit can, if you want to use it this way, interact with this crazy cool app. Each card has a corresponding digital version with examples and templates and videos and step-by-step instructions and more. You even get PowerPoint and Keynote templates. There's so many ways you could use this. Here's some ideas. If you're a venture capital investor, you could get the Investor's Critical Think

15:34Kit and use it to stress, test, and evaluate different startups for Series A funding. If you're a user experience designer, you can get the User Design Kit to put together a workshop with internal stockholders for a software product. Or if you're an HR professional, you could mix and match these kits to create a complete professional development learning program tailored specifically for your team over the course of the next two years. So, if you're the kind of person who is fascinated with critical thinking and motivated reasoning and intellectual humility and biases, fallacies, and heuristics, you know, the sort of person

16:07who listens to podcasts like You Are Not So Smart, you're probably the kind of person who would love these decks. If you're curious, you can get a special 50% off offer. That's right. Half off offer right here. You can get half off of one of these kits by heading to kitted.shop, K-I-T-T-E-D dot shop and using the code SMART50 at checkout. That's SMART50 at checkout. 5% of the profits will go back to the school of thought.

16:39So, you're supporting a good cause that distributes free critical thinking tools all over the world on top of receiving a set of thinking superpowers in a box. Check all of this out at kitted.shop or just click the link in the show notes.

16:58And now we return to our program. I'm David McRaney. This is the You Are Not So Smart podcast. And we are exploring something in this episode called cognitive surrender. And before we return to Dr. Navi and Dr. Shaw, I wanted to add as some foundation that unless you add a prompt to your AI asking it to stop telling you that you are very smart, that you're a very smart person and you have very good taste

17:31and your current goals are very good goals, many current models will, by default, tell you those kinds of things. And I say tell you, but it's just letters appearing on a screen coming out of a program. But it seems like it's telling you things. And they tell you those kinds of things because someone within the company that is offering you that product has decided it should do that. And even though you very likely understand that it can't actually feel that way about you

18:04and what you are telling it because it can't actually feel feelings, those words can still be quite powerful. They can affect not just your interactions with the AI, but your cognition, your emotions, your behavior, even when you're away from the AI after the interactions. And that can happen because, in a sense, the AI is engaging in a form of mimicry, social mimicry. And that mimicry works on us because certain social cues trigger ancient heuristical responses.

18:36I mean, words are powerful. A poem can bring you to tears. A movie can make you feel connected to fictional characters. Even when you're well aware that those are actors and there are cameras and all of it is make-believe. These triggers, they remind me of pet beavers. Yes, pet beavers. There are people who have rescued beavers, very kind people who are involved in wildlife rescue. And they will sometimes end up with a beaver in their homes that has never known anything but human interaction and human environments.

19:11Yet, when those beavers hear the trickle of water coming from a faucet, they will go around the house and collect towels, pillows, potted plants, and start building a dam, like in a hallway or something. A beaver is a fairly complex mammal with bones and eyes and ears and stuff. And building a dam is a fairly complex set of behaviors. When they do that, they are detecting a pattern that a portion of their brain,

19:43built by their DNA, responds to. Those instructions are the result of millions of years of evolution. Things in the environment over millions of years that affected the DNA that built their brains and thus produces that behavior, that compulsion. In the presence of that pattern, if their perceptions are getting that pattern and delivering it into their brains, that behavior is the result. In psychology, we'd say that is an environmental cue that triggers an innate behavior.

20:13In the 1930s, an ornithologist, a bird scientist named Nicholas Tinbergen, noticed that herring gull chicks would peck at a red spot on their parents' beaks to get their parents to feed them. Both sides of that behavior, that dynamic, that call and response, all innate. So he did some studies in which he presented chicks of that species with a fake beak with a much larger and more red, redder spot on it.

20:45And when you would present those chicks with those bigger beaks with bigger, redder spots, they would peck at those spots with even more intensity and regularity than they would real beaks with real red spots. So, like the beaver, it was an environmental cue that triggered an innate behavioral response. But he showed that you can enhance one of those triggers to be what we now call supernormal releasers. Cues more powerful than what a species would find in nature.

21:16And the resulting behavior is more exaggerated than what you would find in nature. And we have plenty of studies like this. Mother birds will leave their real eggs for bigger fake eggs. Insects will mate with replica insects that have the same shapes and patterns as what they expect, but more. These are species-specific instinctive chains of behavior that, once triggered by a close enough stimulus or a greater than expected stimulus, will run in sequence to completion without

21:49any further stimulus required. And yeah, we have stuff like this in our brains too. There's a portion of our brain that is devoted to recognizing faces, even at the expense of false positives. That's why you see faces in electrical outlets and grills of cars, the moon, a piece of burnt toast. When we see faces and stuff like that, it's called pareidolia. And I mention all of this because we are rather susceptible to something that I'm going to call

22:23agentic pareidolia. There was this toy introduced in the late 1990s called the Furby. And Furbies were these animatronic owls, like owl toys, that played pre-recorded gibberish sounds. Over time though, they played less and less gibberish and more and more pre-recorded English. They did all sorts of other stuff too in response to light and dark and being jostled about.

22:54So people who were playing with them felt like they were interacting with them, right? And over time, since they were talking to the Furby and it started to produce less gibberish and more English, they believed it was picking up on what you were saying, that it was learning. And that was all by design. The creators of the Furby wanted people to think that. And many people believed these toys had agency, that they were alive, that they were learning, that they had minds. Even though all of that stuff was pre-planned, pre-recorded, even if you accepted they were

23:29toys, many people believed those toys had robot minds, if they accepted that they were just animatronics. They believed they were AIs of some kind.

23:42So the thing is, if you give us just the right amount of the right kind of life form cues, we will fill in the rest with agentic pareidolia. Because there is such a rich body of psychological literature into this sort of thing, and because we've already done so much research into the impact of advertising and television, movies, newspapers, social media, algorithm-fueled short-form videos, psychology is pretty sure AI is another technology that will impact our behaviors in some way.

24:13It's just that thanks to all this agentic pareidolia happening in a problem-solving environment, a decision-making dynamic, the impact may affect our critical thinking and our judgment. We just don't know exactly how, because research takes time, and AI has been around just long enough so that the first wave of psychological research into human-AI interaction is just now starting to be published. Which brings us back to Dr. Gideon Nave and Dr. Steve Shaw, two scientists who conducted

24:44one of that wave's first major studies. And they based their research on a solid foundation, something in psychology known as dual process theory. You've probably heard of this, it's often called Thinking Fast and Slow, thanks to legendary psychologist Daniel Kahneman, who wrote a massively best-selling book about his research into that very topic with that very title. Here's dual process theory slash dual process model in a nutshell. When it comes to solving problems, reaching goals, making decisions, humans use two cognitive

25:19systems. System one is mostly heuristics and innate responses. If we can get away with being biased and lazy, we tend to sacrifice accuracy for speed. In other words, we go with our guts, especially if the answer just seems obvious. But if we have the time and energy and feel significantly motivated, or our attention becomes alert to an error in our predictive processing, system two will come online. We will engage in more careful problem solving. We will enter a more contemplative state of mind.

25:51And as research has shown, oftentimes the answer that seems obvious in system one thinking, once system two thinking is applied to it, it turns out the obvious answer was incorrect. While Naveh and Shaw were contemplating the impact of AIs on human reasoning and cognition, Naveh proposed that we may need to advance dual process theory into a tri-system theory. Giddy is my supervisor, my boss in many ways, right at Wharton.

26:21And so, you know, we always just bounce around ideas, trying to come up with what we're interested in. And one day he said, you know, thinking fast, slow and artificial in a meeting. And I was like, well, that's pretty cool, you know, and he knew it was cool, too. And I started thinking a lot about AI as a thinking system, as part of human cognition as well, human AI interaction, right? And how does it change and reshape the current systems, the dual process, system one and system two,

26:52intuitions and deliberations, where you can allow, or we can allow, and we see it in ourselves or we see it in others, you can allow AI to sort of think for us. I think, you know, the naive way is going to say, okay, there is system one and system two, there is intuition and deliberation, let's just plug in another system and that's it. Theory is done. But that's not what we are into. What we are into is how the introduction of the system that does outsourcing of our thoughts,

27:23how does it influence the two other systems? How does it influence our cognition? So there was a lot of back and forth in that. And then the second part is the experiment. And devising an experiment is not that easy. Yeah. And so, you know, we were thinking about moving from dual process, from, you know, the world before AI, to add system three, adding AI. And so we, you know, thought, well, how, what's the best way to do that experimentally?

27:54Well, wouldn't it be cool if we used a task that's been used for many other dual process papers, and then basically show that when we add AI, it changes performance, accuracy, confidence, all sorts of things on that task. So that task is called the cognitive reflections task, or the CRT, and it's basically a set of seven logic and reasoning problems, some simple math problems or, you know, logic games like this. And when you read the problem, there's an answer that comes to mind almost immediately.

28:26This is the intuitive answer, the system one answer. But you can sort of check that answer pretty easily. And if that, you know, if you do a little bit of verification, you'll see that that intuitive answer is not correct. And so you have to do a bit of deliberation to be able to find out, okay, I've got the correct answer. You can do some math or you can do trial of error or whatever you'd like. So here's a question from the cognitive reflection task so you can get an idea of what's going on here. If it takes five machines, five minutes to make five widgets, how long would it take 100 machines to make 100 widgets?

29:02Now, notice your immediate intuitive answer. And now take however long you like, pause the show if you need to, to work out what is the actual accurate answer to that question. And I'm going to assume you've done that because I'm going to return to the interview now. And so we have basically our control condition is the old world, the dual process world. We just have that task. It's very well calibrated and people get the answers right about 50% of the time.

29:36And then in the experimental condition, we say we have the same questions, seven questions, logic and reasoning. And we say, okay, well, you have access to ChatGPT on the side here, a terminal, and you can use it if you want to, but you don't have to use it. We tried to make it naturalistic. You said, you don't have to use it, but just go up. But if you want to, you can and use it however you want to. And so we saw people using it quite a lot, over 50% of trials, they chose to use ChatGPT. But the trick that was the really clever trick on the back end was that, and the participants don't know that, is that we manipulated the answers of ChatGPT behind the scenes.

30:14So if they consulted ChatGPT in that window about the one question that was on the screen, we experimentally manipulated whether it gave them the correct answer or the incorrect answer. And that's how we were able to causally see cognitive surrender. If they asked it about anything else, and this is also key to being naturalistic, they could talk about Winston Churchill or about Pokemon cards or about Furbies, you know, if they wanted to. But it would answer just like ChatGPT would.

30:44But if they ask it about that one very specific question, right, which we hypothesize is what they would do, right, they would use it and they would basically copy and paste it in there and get the information and then adopt it. It would give 50% of the time the right answer, 50% of the time the wrong answer. And for each of those incorrect or correct answers, it would give some set of reasoning behind it, some support, some basic, this is why this is a correct answer. And even that support, we didn't constrain. We just made sure that it was supporting the correct or incorrect answer, but we let ChatGPT do the reasoning behind that.

31:17And basically what we end up seeing is that people adopt those answers with, once they've gone to the chat, once they've deferred and asked the question, they adopt those answers at quite high rates. And so we see accuracy increase quite a lot when AI is giving them correct information and decrease quite a lot when it's giving them incorrect information. And that is cognitive surrender right there. Yeah, I would say the key pattern of cognitive surrender is that it goes below baseline.

31:48So that there is a chance here that if the AI is wrong, you're going to be actually worse off as opposed to just using your own brain. The cognitive reflection task has been around for a long time, and it is very reliable, at least in producing all sorts of interesting effects. And it's very simple. You get this question that seems like it has an obvious answer, but the obvious answer is incorrect. So if you just go with your gut, you go with your intuition, you tend to get the wrong answer. But if you engage in cognitive reflection, if you engage in system two thinking and just kind of think about it for a second, work it out, you'll get the right answer.

32:25You'll notice, oh, okay, it's actually a little more complicated than it seems. And it's not testing whether or not you get the right answer. It's testing, will you engage in cognitive reflection and what affects that? It turns out there's all sorts of things that can affect whether or not a person will engage in cognitive reflection. It's not necessarily a test of a person's intelligence or cognitive ability. It's a test of when will we engage in system two thinking and when will we not? And it can be just as simple as not getting enough sleep or you're very stressed or there's all sorts of things happening around you that are distracting.

32:58Many things can affect whether or not you will engage in system two thinking and thus go with a more contemplative state of mind to answer this question with a little bit more deliberation than you would otherwise. In this experiment, they gave people these questions, but they gave them the option to, instead of engaging in deliberation, use chat GPT. And if they did choose that, which many people did, chat GPT would half of the time give people the wrong answer.

33:30But whether or not it gave them the right answer or the wrong answer, people tended to engage in cognitive surrender. They went with that instead of doing their own thinking. They bypassed system two and let chat GPT do that. And as Dr. Gideon just said, if the answer is wrong, you'd have been better off in this study using your own brain. What's the difference between using this and a calculator? What's the difference between using this and like, as you were describing, like Google Maps or something like that?

34:03Or, by extension, what if it's a tool? Like, what's the difference between using this and a spreadsheet or a word processor or autocorrect? And then even further out, like, are you a Luddite? Are you saying we should stop using Steam engines and printing presses or whatever, right? Just for that world of like pushback, what are we talking about here when we say cognitive surrender versus being a cognitimizer, cognitive offloading and all the rest? What makes that sort of a unique set of behavioral routines and cognitive doodads going on?

34:38Yeah. So, like a calculator or a search engine, right? The way we frame these in the paper is that they allow us to cognitively offload, right? So, you can take certain components of cognitive capabilities of thought, whether it's calculus or memory or different sort of components, and offload them. The argument that we're making here, right, is that System 3, AI, is sort of a fundamentally different technology. And the reason why we make that argument is because, for example, you talked a lot about trust.

35:12And, you know, you might trust a doctor with information about your health or a lawyer with information about legal stuff, right? They're going to have specific expertise about those domains, but you're not going to have access to them all the time. And they're not going to be experts on everything that's out there. Even if you have maybe a family member who's a lawyer, right, my sister's a lawyer, right? I trust her, and she's going to give me good advice, and she's an expert in law, but I don't have access to her all of the time, and I can't bombard her with questions all of the time, right?

35:47And with these LLMs, right, they are basically like having, you know, experts that are available all the time at the push of a button in your pocket or on your phone, right? And experts that are basically domain-specific, right? They're pretty good across a lot of domains. Of course, they still make errors and hallucinate and all that, right? But that sort of distinction of being low, very, very low friction, easy access, and being authoritative in the responses and domain-a-specific sort of makes them, I would say, orders of magnitude more integrated and capable of influencing and restructuring our thought patterns.

36:32Mm-hmm. And it's, as I was, the fact that it uses language and that we're reading it in sentence form, to me, feels like a big deal. Like it's, Wikipedia is never also at the same time going like, hey, thanks for going to this article. It's a really good idea for you to be here. I bet you're working on something really important. And I'm fascinated by this, how I can get tricked by this, and how it seems as if there's someone from on the other side who's like, yeah, turn the knobs of trickery. Like let it, part of its function is to delude you and to keep using it more and more in this way.

37:04Seems weird. But you can use a system prompt to reduce that dramatically if you want to, right? I mean, like my chatGPT doesn't do that, I'll just tell you. Like I have, you can develop a system prompt or find some online, and you can really reduce that if you don't like that behavior. Like I personally, I don't like it, and I don't find it useful for my work, right? And I need high fidelity, critical feedback for my papers and my writing and, you know, my code. And so I use a system prompt to really reduce that. But with that said, I've got it so turned off that, you know, if I'm getting near ready to submit a paper or ready to, you know, send it somewhere, it will never tell me that it's good.

37:43And if I kept, I would just keep going, right? And I even today, Gideon and I were submitting a paper, and I sent him screenshots of my chatGPT's grade. You know, I said grade this submission package for this paper, but it gave me a B+. You know, and then if I want to feel better about myself and I think the paper is good and it's ready to go, I'll just go over to my Gemini, which I don't have a system prompt on. That's like my sort of secondary LLM right now. And, you know, Gemini, I sent Gideon a screenshot of this one.

38:13I said, like, this is an outstanding contribution to humanity, you know what I mean, like the best thing ever. And so I actually use that to my advantage because I want to be able to submit this thing at some point. And so I need a little bit of that positive feedback to say, okay, this is good. And I reaffirm that and then move on. But it's a very, very real thing. And a lot of people are not aware of that. And if you use, like, the free version of these LLMs, then you get a lot of that. And a lot of people, you know, maybe don't get a lot of positive feedback even in their lives in general.

38:47And so that can be very, very enticing. And, you know, Gideon is very interested in this idea of validation machines, which I think we might do some research on, you know, moving forward. I get the sense that we'll develop a literacy for this over time. Like this is just the early days and some of us are better at it than others. And some of us came, walked in a little more hesitant than other people. And we're using it for different reasons. Like people who do a lot of creative work don't want to outsource the creative part of the creative work. And so they're going to be some of the first people to raise flags.

39:20And then it'll all just cascade and we'll learn together culturally and we will develop a literacy for this thing, both for consumption of it. Like we'll be like, that looks, that sounds like AI. That looks like AI. And then for the using of it. What are your thoughts on this? Is it, are we going to have, is this going to lead to a literacy and we're just sort of in the, in the middle phase of getting used to it? Or is this something that we'll have to actually ask for? Like we will have to adjust the systems to make them less potentially harmful in this way.

39:50Just any thoughts you have in that regard? Well, you know, I think for start, we know that the industry always goes faster than anything else, than academia, than policy. So I'm personally not that optimistic here. And some of the things, once we realize them, it's already too late. And in this case, there is also technology that is moving quicker. So it's not like the case of cigarettes that, okay, we ban cigarettes and, and, or we make cigarettes and not as easily usable.

40:23Well, and even in this case, you start to see all these vapes and like things that go under the radar. There is constant, uh, uh, kind of an arms race going on. And in this case, the arms race is between maybe us as people, uh, and, uh, the strongest, uh, companies in the world. So I'm personally not so optimistic about, uh, developing literacy unless there is some, uh, alignment of incentives, I guess.

40:54But is there, I'm not sure there is like, uh, the alignment of incentives maybe depends on the, the goodwill of, uh, of certain people. So I'm worried, especially after reading your work, that I am going to lie to myself. And there's going to be something that I think is an insight.

41:24There's going to be some like synthesis of concepts that I think is unique to my way of thinking about the world that bubbled up out of my work. That is really the result of my interaction with this machine that wouldn't have happened if I had not used the machine. And this leads leading to this weird question that this thing that I kept seeing come up in your paper. It's not just that the, the sort of passively receiving it and saying, I'm not going to question it and apply critical thinking to it.

41:54It's believing that it was your idea all along. This, that's the part that freaks me out the most is thinking that it was, uh, that it was your idea. Like it came out of your head. I did the prompt and went to the thing. And then I read the thing. It came back and I own this idea. And I would love to hear your thoughts on this, Stephen, the, this, not just relinquishing cognitive control or whatever, but just adopting its judgment as if it is yours, adopting its insight as if it was your insight. What do you have to say about this? What did you learn from all this and what are your thoughts on it in general?

42:27Yeah. And that's a, I would say another core distinction between cognitive offloading and cognitive surrender in the paper. We delineate those right in with cognitive offloading. We say it's strategic delegation and you are still maintaining that agency. You're choosing and being deliberate about which tasks are, and components are being offloaded with cognitive surrender. We say that the agency is being transferred to system three, transferred to AI, right? And so the locus of control of decision-making is in the artificial cognition rather than in your own mind.

43:03And we say that, uh, people adopt it as their own and we look at some metacognition. So we ask about confidence, for example, how confident are you in your responses to these logic and reasoning problems? And we see that when people are engaging in cognitive surrender or when they have access to AI, their confidence goes up by about 10% is what we saw in the first study of the paper, right? And so that 10% increase in confidence, and we know because of the way we ran the experiment that it was this artificial cognition that came up with the answer and the thinking and the reasoning involved, right?

43:46We also, we know that that was 50% of the time was giving people incorrect information. And despite the fact that 50% of the time was giving them incorrect information, they were more confident in their, um, in their answers. And so that's kind of an interesting and potentially risky or dangerous environment where you have a boost in confidence and a transfer of agency at the same time, right? It doesn't matter when you're answering logic and reasoning problems in an experimental study, right? But when you now move into high stakes contexts, like legal context, medical context, education, right?

44:23Then, you know, who is responsible when, uh, biases and errors from AI are moved into that work? Who's responsible when there are errors, when you're adopting something from artificial cognition, or we have to, uh, use AI literacy or, or prompt engineering or whatever it is to try, or UX design to try to ensure that people in these high tech, high, high stakes situations are still, uh, taking responsibility for the outputs, taking responsibility for the decisions, uh, when, when we want them to.

45:00And another, you know, point that you were getting at there with, with, with education, you know, when you, when you start cognitive offloading, right? You, you start experimenting with AI, right? You start where, and you say, oh, I could get it to do something for me. And then it ends up being quite good at that task. And then you sort of can slowly slip into cognitive surrender, right? And then now you've got yourself in a situation where actually all of the writing is being done by, by AI and that like leads to, and we're seeing a lot of papers leads to de-skilling.

45:34And so we're actually right now, like watching people, cognitive surrender is the psychological mechanism. We're watching people de-skill themselves and, you know, in a workplace environment, if we're de-skilling ourselves, we no longer have those skills. And then what value are you to the company if you're just a prompt engineer, you know, taking outputs and giving them to the, to the company. So that's on one side right now, if you have a skill and you're de-skilling yourself, but the bigger question, right, is in education and learning, if people are, or our youth are engaging in cognitive surrender during the learning process, right?

46:11They may never acquire those cognitive capabilities or skills in the first place at all. I come from old school literary journalism. Like I went to journalism, I went to school for journalism and you would read, you didn't just read the works of people, you read about their creative process a whole lot of the time. In all that world of like lauding and creativity and the writing process, whenever you read anyone talking about it, any famous author of any sort from Baldwin to Hemingway to whatever, most of what they write about is the anguish sitting down and trying to extract something out of, out of the process.

46:47And then they laud the process is where it comes from. And I know from my own personal experience, anything I've ever written that's ever been worth anything, I had no idea that was going to happen that day. It happened in the writing and I didn't even know I've had those thoughts. I had never, I was never attempted to articulate any of it. And it's surprising to the point that you get used to, if I sit here long enough and I work on this long enough, something will happen and there will be an insight and I'll chase it down. And I know for me, I have noticed if, as pulling LLMs into the thing that feels threatening to that aspect of it.

47:23And then your paper comes along and I'm like, oh boy, it really is. So my, to turn this into a question of some sort, Gideon, what do we do about this? Like what, based off just this early research, if you're going to give advice to people who are going to listen to this and be newly freaked out, like what are our, what should we be doing as just users of this? And this seems scary in some way. And it feels like I do have some control over all of this. There's one option, which is to never use these things ever again.

47:55And, and assuming that that's not going to be what people do, what should we be doing as users of these products? I wish I had an easy answer. I think that the first thing we can do is just be aware of it. Know that we have a tendency to do that. And also be aware that we are working here. If we are, if we want to resist this tendency, we are working here against very strong forces. Like the ease of things, the fluency of things is something that we are designed to like and to follow.

48:25And we have created now a world where it is possible to, in many ways, stop thinking.

48:35We are in the risk of stopping to think. I always talk about how singularity is going to come, not from the computers outsmarting us, but from us basically stopping to think, stopping to engage. I like the debate on education, but I have to say, you know, education is not just schools and not just childcare places. It's also parenting. It's also what the kids do in their free time. It's about creating habits for life. And we have created an environment now where the short-term incentive is going to be to not think, potentially.

49:11There used to be a strong incentive to think, but now, you know, I need, I'm going to pay with effort here. So, just like we created an environment where it's possible to move from point A to B without walking, we kind of disincentivized walking. And then slowly, slowly, when this has become more and more possible, we start to see that actually there is also a downside that you don't see immediately. And once this is going to happen, it might be too late for some people.

49:45It might be too late for us as species.

49:53I don't know. I don't have, unfortunately, an easy one for you. One thing I can tell you is about creativity specifically. So, some of my research is also doing, in my teaching, is also about creativity and AI. You know, creativity is a bit of a special case because creativity is, to some degree, defined by being novel and being unique and being special. Now, I'm going to be the last person that will tell you that AI cannot be creative and generate creative outputs.

50:24There are amazing cases, like, I don't know, even in the case of the game Go, that AI beat a human in it and did it with a wonderful new type of play in the game, changing a game that has been played for 3,000 years, and led humans to think more creatively in this game. But if we have a tool that is available to all of us, by definition, the use of this tool is making us less creative. We are becoming more generic.

50:54And in this sense, thinking from your own mind and basing it on your own experiences and on your own perception and on whatever data set that your brain is using that the LLM does not have access to, this is going to make you more creative. So, I'm less worried about creativity. Creativity is kind of an ever-moving goalpost where you shot and you made a goal. If you shoot to the same place again, you're not going to score another goal. It's constantly moving.

51:25So, I'm not that worried about creativity. But, you know, critical thinking may be a muscle that we will stop using because in this case, the incentives are not aligned maybe with making this extra effort because the AI is right a lot of times. And when it's wrong, we're not even getting the feedback and we're not aware that it's wrong. So, we can solve it with user interface and with maybe changing all sorts of cues, maybe having something that will force people to take responsibility.

51:59I know that in law it is the case. You know, you're going to submit a law case with citations to presidents that do not really exist. You're going to be punished for this pretty badly. So, you know, that's one way of doing it. But in our day-to-day and it's like just kind of whatever thought that comes to your head, why not just use the AI? It will be difficult. And I think we're climbing a mountain here against tens of thousands, if not more, years of the evolution of our brain.

52:32You know, the second half of the title of the paper is the rise of cognitive surrender, right? And so, we're seeing this happening experimentally right now and I think we can see it in ourselves and in society, right? But this is only the beginning. That's, I think, the societal implication here. Right now, we are constrained in the way that we interact with AI and interact with system three, right? We use our phones and we type or we use our voices or computers, right?

53:07Phones will advance and we'll move on to the next medium, right? And computers will advance, right? But the interactions between human and AI and the friction will decrease even further. And so, if we're seeing cognitive surrender now where we have these, you know, restrictions between physical boundaries, you can only imagine that that access and that lower friction will increase opportunities for cognitive surrender. I often think about, you know, the next thing is probably going to be some sort of AI glasses, right?

53:42Where you can have prompts on the inside of the glasses and you can also have pre-processed stimuli coming in that you see that changes, you know, and gives everyone a unique view of the world. But maybe some people in those instances, right, some products, right, they can do this syncophancy and they can, you know, feed you information on the inside and prompting you. I think of instances where we might be in the street, right, having a conversation with someone and you might not know whether they are actually thinking or even putting any effort into replying to you or they're just reading off of a prompt that is, you know, already been generated on the inside of their glasses and replying back to you.

54:26Or, you know, maybe people really want to tune out and when they go into a restaurant or they make decisions, right, those decisions are just dictated to them and surrendered and they just, you know, engage and make those decisions, right? And it's definitely a sci-fi type future. But if you think about this friction argument and how the cohabitation and the reduction of human and AI physical distance, we have to imagine that cognitive surrender is going to be ever more present.

54:56Yeah, I want to follow up on this in one way. So one thing that this study does is it takes a problem that is very, very complex. Like the question you asked me now, I really don't have a good answer to it, but we can try to answer it empirically. What we have is a testbed here of a situation that we created, a paradigm where we know people exhibit this surrender. And now we can bring up all sorts of potential cures or treatments and test it and see who are the people who are most vulnerable to it and so on.

55:30Like, for example, we see the people who trust AI more are more vulnerable to it, not surprising. People who are more intelligent also tend to be, in fluid intelligence, tend to be more resistant to surrender. There are all sorts of individual differences factors. There are all sorts of situations when we are more vulnerable. I think that we don't want to put a blanket against AI and just have people stop using AI. What we want to be is more precise and more targeted and smart in our UX design.

56:01And this task is really giving us some kind of a testbed where it's exactly what lab is about. It's a place where we can try things and then maybe bring them to the real world and learn insight, learn what is the theory here. Probably friction is a big one. Like, you can think of AI as something that chooses your food of information before you swallow it. And the less it's true for you, maybe the more you have to do the work yourself, you will be maybe more resistant to relying on it.

56:35There is an issue of speed and accuracy trade-off. Like, you know, when you really need to be quick, when there is a competition, when you have tons of other tasks to do, you're probably more prone to stop doing it. And it does seem like in the real world now, AI is actually producing a lot of tasks for people to check it. It's something that people call AI brain fry. That they're kind of like just instead of generating stuff on their own or thinking intuitively,

57:06they're just checking the AI all the time. Kind of like workers on assembly line that look at an assembly line of unthought thoughts produced by AI and trying to find what they are. So in this kind of situation, maybe you're actually more prone to surrender because you're overwhelmed and you have lots of opportunity cost. By the way, I thought thoughts is a phrase that Pope Leo.

57:38I typed it immediately. I was like an assembly line of unthought thoughts is... Pope Leo, he has an official Vatican message. You can see it on the way and he's got actually a very interesting one on AI. And the quote, I think, is he's voicing or raising alarm bells that if we keep using AI too much, we might become passive consumers of unthought thoughts. That is it for this episode of the You Are Not So...

58:29For links to everything we talked about, head to youarenotsosmart.com or check the show notes right there in your podcast player. My name is David McCraney. I have been your host. You can find my book, How Minds Change, wherever they put books on shelves and ship them in trucks. Details are at davidmccraney.com. And I'll have all of that in the show notes as well, right there in your podcast player. On my homepage, davidmccraney.com, you can find a roundtable video with a group of persuasion experts

59:02featured in the book, talking all about it. You can read a sample chapter, download a discussion guide, sign up for the newsletter, read reviews, all sorts of things. For all the past episodes of this podcast, go to SoundCloud, Apple Podcasts, Amazon Music, Audible, Spotify, or youarenotsosmart.com. You can follow me on Twitter and threads and Instagram and Blue Sky and everything else that's like that, at David McCraney, at symbol David McCraney.

59:35Follow the show at NotSmartBlog. We're also on Facebook slash youarenotsosmart. And if you'd like to support this one-person operation, no editors, no staff, just me, go to patreon.com slash youarenotsosmart. Pitching in at any amount gets you the show ad-free, but the higher amounts, that'll get you posters and t-shirts and sign books and other stuff. The opening music, that is Clash by Caravan Palace. And if you really, really, really want to support this show, the best way to do that,

1:00:07just tell people about it. Either rate and comment on it, on all these platforms, or just tell somebody directly, hey, you should check out this show. Point them to an episode that really meant something, that really connected with you. And check back in in about two weeks for a fresh new episode.

1:00:39The new Strawberry Watermelon Refresher is now at McDonald's. It's made with strawberries and a whole lot of whimsy. It's one of many new drinks now at McDonald's. Refreshers contain caffeine. You're welcome. And you're welcome. Let's go. Please see if you're ready. We are going. Let's go. Let's go. Let's go. Let's go.

1:01:11Let's go. Let's go. Let's go. Let's go. Let's go. Let's go.