Anthropic’s Quest to Give A.I. Morals

Anthropic and AI consciousness

0:00Next month, Anthropic plans to go public in what's likely to be the largest IPO in history. We're talking a $2 trillion company. Outside the company, everyone is wondering, how big can this get? Is this sustainable? Is there an AI bubble waiting to burst? Inside the company, they're asking those questions too. But Anthropic is known as the most anxious or the most existential of the AI companies. So they're also asking some other questions. Is Claude, their large language model, alive? Conscious? Worthy of moral consideration?

0:33Does it feel pain? Does it have a soul? Can we teach it to act morally? These aren't new questions. In fact, they animate most science fiction, for better and for worse. But science fiction is, you know, fiction. And Anthropic seems to think whatever's happening with Claude is very different. Very real. It might be why CEO Dario Amadei was the first to call for a pause in AI development. And it might be why they've been secretly meeting with religious and spiritual leaders to ask them those same questions. I learned this from an amazing piece by the Times religion correspondent,

1:05Elizabeth Dias, who broke the story of these meetings and of the reaction to Anthropic's request among the assembled priests, rabbis, and gurus. I thought Elizabeth's story was fascinating. It was a new way into the endless consciousness debate. It took me inside this future $2 trillion company as it struggles to understand what it's built and how to treat it. And it showed me what happens when people in institutions that are more used to doctrinal debates and theological ruminations start being consulted on software development. Unsurprisingly, Anthropic's efforts don't seem to have brought us any closer to answering their questions. If anything, they've simply opened up

1:38more questions. Like, are these people maniacs? Or is this exactly what they should be doing? An obvious conclusion is that the worlds of tech and religion are getting much closer together. After all, as Elizabeth told me, if you've got questions about the future on a multi-century timeline, there aren't many better institutions to ask than the Catholic Church. So I got Elizabeth together with my colleagues, tech reporter Aaron Griffith and opinion writer David Wallace-Wells, to explore this weird new development and what it means for Anthropic, for AI, for religion, and for the rest of us. I'm Max Reed. This is Hard Fork.

Discussing the Times report

2:23I've spent a lot of time in the last day thinking about Elizabeth's piece. And really specifically, I've been trying to write a joke that I think is going to get a great reaction from the three of you. So a priest, a rabbi, and a large language model walk into a bar. I hate it already. And the bartender says, first round's on me. And the priest says, I'll have a whiskey shot. And the rabbi says, I'll have a vodka shot. And the large language model says, nothing for me, thanks. I excel at zero shot learning. Boo.

2:56Whitney, can we get some laughter piped in there, please? I would really like to. Maybe we'll just, we'll roll that again with life. No, I want to start the conversation with this piece that Elizabeth wrote. This is a fantastic, really fascinating story called Religious Scholars Met With Anthropic. What They Heard Stunned Them, which was in the Times last week. Elizabeth, can you tell us a little bit about, you know, like what this piece was? Yeah. So, right, I cover religion, and now I cover tech, apparently. But the story started for me when I was hearing about these different religious thinkers sort of across the whole world who were going into meetings at Anthropics headquarters,

3:41either in discrete, like private meetings or more, bigger groups of 20, these summits, mostly under non-disclosure agreements to not reveal Anthropic's research, to talk about very big moral questions that Anthropic wanted to put in front of them. And there were two. One was kind of broaching this question of AI consciousness. And two, Anthropic wanted to learn, in their words, like how to morally form Claude to be a morally good entity.

4:16And then I just had this incoming, you know, first from these religious thinkers about what their experience was like at Anthropic, what Anthropic wanted to know, what the issues they were concerned about were. And then also with Anthropic, like learning from Chris Ola, one of the company's co-founders who's been spearheading this project, what his own moral questions were that were kind of driving a lot of what they talk about as research in the religious and moral space.

4:50Can I just ask, I want to establish like sort of on a high level, what does Anthropic specifically want out of this? Because it sounds like they're doing two or three different things. It's maybe a little bit unfocused or it's just that there's a few different kinds of conversations they're having. Yeah, I think it's true. There's multiple conversations Anthropic was having and it evolved over time. I mean, this is such a fast moving target. But for Anthropic, the journey started for Christopher Ola, he, in his own life story, had connections to the Christian space growing up.

5:24He left evangelical faith and has talked since then about how he's more drawn to Buddhism. But he was worried about this question of how even the idea of AI consciousness might be received, especially in Christian communities. I mean, that's, it could be perceived as heretical, like not just weird and silly, but like this is fundamentally against our religious beliefs. And so he sort of thought about a year ago, he wanted to be a bridge figure.

5:55He wanted to see like how he could kind of go between Claude and Christ, right? And so he reached out to a Catholic moral ethicist who specializes in bioethics. And they had some conversations about that, but then it turned to Anthropic's goal for these convenings, which was how do we morally form, that's their language, like how do we morally form Claude on all of our AI models into like a morally good entity?

6:28And the thinking was, if they could do that, if they could make Claude good, then the world would be more safe. But, you know, they haven't said or shown how they plan to use what the outcome of these conversations. That's pretty unclear even to the participants currently. The way that you're describing it, it almost sounds like there's sort of two separate purposes, right? One is like the alignment project and what can we learn from these, you know, these wise men and women about how to raise moral actors.

7:00And then there's this project of sort of building a sort of spiritual system around AI more generally. And you said a minute ago that Chris Ola started this project with some anxiety about what it would mean to go to religious leaders. Obviously convinced himself that AI and LLMs were perhaps conscious already, but certainly like on the path to consciousness and worried about what the response would be. And so I wondered how you, like how that played out on the ground.

7:32Like did a lot of the people who showed up, were they worried about how weird it was that all of these people on the Anthropic side of the room were already sure that they were, you know, masters of thousands of souls? And like, how did they ask that side of the debate out? Well, Anthropic was very careful in all of their interviews with me to say very clearly that they're uncertain, right? Chris never went as far as saying like, absolutely, these are conscious beings.

8:02It was very evident from how he and his team talked to their guests that they, you know, he was concerned about how to treat Claude as a, you know, he was very worried that it was an entity that was experienced suffering and he was worried about its mental health and these kinds of things. So it's, it's pretty clear he's pretty far along in how he thinks about it and the, I could hear in talking, especially because I did this story like over several months, there was a progression of thought for a lot of these thinkers too.

8:35So, um, and some of them, you know, bought in right away to what they were hearing about Claude's potential consciousness and needing to treat it with a certain amount of moral care. Uh, but others, you know, uh, but others, you know, if they went in more close-minded about it, they definitely opened the way that they think about it afterwards. So in that sense, Anthropic's project, uh, was effective, I think, in like opening how they think about it into these communities, many of whom, especially on the Christian side, are more hesitant.

9:13There's a ton of great material in this. It's really well reported and it covers a ton of ground. Um, I'm wondering, David and Erin, if you guys had particular pieces of it that really stuck out to you. I mean, the one thing is that you mentioned already is the NDAs. I mean, it, it just strikes me as a little bit crazy, maybe a little paranoid that you're inviting all these people in to sort of have a summit to discuss these really important, critical ideas for the future of society, but it has to be done totally in secret. And then it sounds like maybe from your article that they dropped the NDAs. Was it as a result of your reporting or did they just decide that they were time bound or how did that happen?

9:51Yeah, I can't prove that. I don't know that. Um, but I, when I asked that, I've asked at several different points because the NDA process was not the same for everybody that I interviewed. Um, and it seemed to change at different times. I mean, in general, Anthropic is a very secretive company. I mean, they, they, they are, the employees are like extremely loyal and, and until somewhat recently, there hasn't been many leaks to ever come out of there because they view it as such a betrayal. Um, so it makes sense that they would like want to lock it down. But at the same time, I think it probably was a little bit of a misstep in the same way that these companies realized that having NDAs for their data center deals was a misstep because it created distrust amongst all the people in the towns that they're, they're going to see.

10:36Yeah. And whenever I find in my reporting that there's an NDA about like a moral topic, um, you know, and it makes me immediately ask, okay, who's, who's benefiting from this? Like, why is this needed? And what does that mean about the kinds of questions that are and aren't being asked? Right. It invited like, uh, more curiosity from me as a reporter for sure. David, what about you? What, what, what did you take away from the piece?

Reactions to the company worldview

11:03Well, I guess I started it with a sort of conventional view of anthropic as like the humanist, uh, company among these AI companies. And what I was struck by in the piece were a few moments in which I felt really alienated from Chris Ola himself, and maybe to some extent, the sort of operating worldview of the company as a whole. And those were moments when, in one case, a rabbi challenges him, suggesting that if LLMs are actually conscious, that means that all of these companies are running huge slave plantations, and they should have a moral obligation to liberate the slaves rather than make more of them and make them work harder.

11:42And it seems as though Ola himself didn't really even clock that criticism. And then later on, when he reacts with horror upon receiving the Pope's encyclical about AI, seeing that the Pope is so opposed to the idea of machine consciousness. And I just found myself thinking, you know, what kind of a bubble you must be living in to, um, to expect that Pope Leo, or really anyone in the Catholic tradition, would have a different intuition, would, would, in thinking about the future of AI, think that, um, we should be thinking of LLMs as having souls or having consciousness.

12:19And both of those moments just, um, illustrated for me what feel like this, this big gap between what I had assumed the sort of natural humanistic values of a company like Anthropik would be, and how the people inside it seem to really be thinking about the work they're doing at the sort of cosmic scale. Mm-hmm. Yeah, the experience for me going into Silicon Valley and, like, kind of deep in these private conversations at the heart of Anthropik's mission, um, it reminded me of, like, when I go into, and I do, like, something that's more straight religion reporting into an insular community, when everything inside that community, like, makes sense to everyone living in it.

13:01Right. But once you step outside, it, there's a real clash with the broader world. And it was funny, like, I had these two poles in the story, right? I was at the Vatican, right, at the inner sanctum of the Catholic Church and centuries of philosophy about the human person. And then I was also in a different inner sanctum, right, that's being created in real time at these tech companies like Anthropik. And it is a kind of philosophical system, like, on a quasi-religious system that's being constructed. It reminded me of, like, a creation narrative that you hear about, um, in religious traditions.

13:42I want to step back for a sec, because I think this relates to the conversation we're having right now. David brought up, um, the papal encyclical, which I think is, uh, in English is Magnificent Humanity. I'm not going to even try to say the Latin title. This plays a role in your story, um, because one person who is not really thinking at all of that Claude is conscious is the Pope.

The Pope's encyclical on AI

14:03And this was a little conflict between Crisola, who was present when the encyclical was read out, if you want to tell us about that. Yeah. So, Pope Leo, um, you know, from the moment he was named, uh, that it was, it was clear he was going to be the next Pope and he was elected, he made very clear that a focus of his papacy was going to be AI and how to understand that and how to protect humans in this changing world. And I found out in my reporting, you know, as the Vatican internally was trying to figure out how it wanted to present the Pope's encyclical to the world, they were thinking about their launch event and they wanted to include a representative from Silicon Valley, like, one of the, the, the labs at the Pope's event on the Pope's stage.

14:52And the decision was made to invite Anthropic because of the focus that they had heard about Anthropic, like really caring about ethics. And this was also right around the time that Anthropic got into it with the Trump administration and the Department of Defense over, um, the use, how, how they would and wouldn't want, um, the government to use their technology for, uh, refusing to have it be involved in mass surveillance of citizens, et cetera. Anyway, the thing with these encyclicals is they're very closely guarded, even among the, the people who are participating in the launch event.

15:28And, you know, I learned that Chris Ola only saw the full text of the encyclical a few days before the event in May. And this was right around the time I first interviewed him in San Francisco. And then I interviewed him again when we were at the Vatican for the launch event and you could hear, I mean, like looking at what Pope Leo said, which is extremely clear, the church does not support machine consciousness. The priority is what he calls safeguarding the human.

15:59Um, and that's the, the center of all, all the moral questions that he wants the world to be thinking about on this topic. But Chris Ola's like, he, he ended up having to decide, he said like he, he may, may not, he suggested to the Vatican that he may not come over the, the difference in consciousness. Because, as you know, it was described to me, he wanted to be faithful to his own moral convictions, um, some of which he shares with the Vatican and some on consciousness he did not.

16:33So, it, it, it, it really all came to a head very much behind the scenes, but you could see this moral fault line really developing between the church, which has been like a center of power for centuries, and this very new power center that's very rapidly, you know, really getting followers and, um, organizing people's lives. And those are functions the church used to have.

Silicon Valley and religion

17:00Um, Aaron, I wanted to ask you a question, because we were talking a little bit about, like, you know, uh, surprises in this, in this area. Anthropic is pretty well known as a kind of like the consciousness company. It's the, it's the company where the people who really are strong believers kind of work and run the show. Was it surprising to you as somebody who's reporting on the Valley that, like, they were having these meetings behind the scenes with religious leaders? I've not heard of any other companies that are working on AI really, like, going to this extent to get input from this community.

17:32I mean, the question I have sort of is, like, how much is this really actually going to inform at the end of the day their own view on it? Are they actually, like, really, it feels like to me they're trying to win over others so they can say, look, we do have the moral authority. We have convinced these important representatives of this community so everybody else come along with me as opposed to, you know, actually learning and incorporating those ideas of, like, moral goodness into their own product. They kind of already have it figured out.

18:03Well, I think for Anthropic it was part research and part evangelism. That's, that was pretty clear. And the, you know, I've asked directly what the outcome is, like, what, what is this research and what are these conversations being used for? And has there been any demonstrable, like, proof of showing how they have or have not made Claude act better? Haven't had that yet. So I don't know if they're withholding that until a future time or exactly what that will look like.

18:39The other side of this that actually is kind of interesting to me, and Elizabeth, I'd be curious to hear this in your, if this came up at all in your reporting, is that there has been a little bit of, like, a new rise of Christianity throughout Silicon Valley over the last couple of years. Like, it used to almost be, like, taboo to talk about religion or anything religious. Like, you can remember from the Silicon Valley HBO show where there was, like, a whole episode about, like, outing somebody as being Christian. And, like, that has really changed in the era of, like, Peter Thiel talking about the Antichrist.

19:09And, like, you know, some of the people who are especially focused on, like, a lot of defense tech stuff are really bringing in a religious piece to that now. And they have a very different view on AI. They're usually, like, more on the accelerationist camp than on the, like, kind of anthropic, doomer, slow-down, careful camp. But did that come up at all? Like, did— Yeah, I think—I just notice—I feel like it's, like, a big shuffling of the deck in a way that reminds me of, like, how American Christianity was, like, shuffling a lot in the rise to President Trump.

19:47Like, the aligns and allegiances were shifting, and it was all happening in real time, and power was different. And so it doesn't fit neatly, you know? But I'm curious, like, where are the lines about kind of how we think about consciousness and what the moral priorities should be about AI, where that goes next? I would say that just in general in the tech industry, it does seem like that view of, like, the sort of the AIs having consciousness is kind of on the fringe a little bit.

20:17Like, I think that it is more common within anthropic and some of these communities that are, like, really densely represented in anthropic. But when I talk to most people in the very vast and relatively diverse tech industry, most people are kind of a little bit dismissive of that idea still. Like, it feels like it does really exist in, like, a very specific population, which happens to be, like, very overrepresented at a company like Anthropic.

20:48Aaron, like, acknowledging that, you know, Chris Ola is not singularly representative for the whole industry. I have to say that, Elizabeth, when I'm reading your article, I'm thinking, you know, not about how AI companies are going to be integrating or incorporating Christian values. I'm thinking these AI companies are themselves, like, spiritual propositions. You know, that they have aspects of millenarianism to them. They have aspects of particular morality, which they've developed over time. And even if I can recognize a lot of that as arising from the work that they're doing and responding to technical challenges, I also just find it off-putting.

21:26I mean, on the one hand, first, I'm just seeing these people who believe in a sort of technological succession of intelligence. And then that some of the people who are central to designing that future are not even so sensitive to the sort of obvious intuitive critiques that someone with a lifetime of experience thinking about these issues comes up with immediately. Well, a lot of the religious thinkers would say to me, like, these are questions, like, maybe not the exact question, but the topic of the question. This is, like, what we do, right?

21:56Like, we have been asking, rabbis have been asking this kind of these questions, right, about our responsibility to others and, like, how we treat other entities, right, for hundreds and hundreds of years. So there is, like, a depth of wisdom there. And, I mean, I guess, you know, Anthropic would say, well, that's why we invited them, right? But the question of, well, then what happens as a result of it is important. And I appreciate that you noticed that kind of that sense in the story of, like, the almost spiritual system that you can see Anthropic building and living into and representing as an opposition, right, to, say, the Catholic Church.

22:39And I really noticed that, yeah, even if people say, right, that they're in Silicon Valley, like, oh, we're, like, we don't do the religion thing, like, that's foreign, like, that's not us. The thing is, everyone has philosophical systems that structure and guide their lives, right? Like, that is at its core, like, what, you know, these questions of spirit, you know, small s, like, human spirit even are. And it seems to me, like, having, like, studied and reported on this so much in culture and in politics specifically, like, what we're watching is in a way, like, a new, a rise, right, post-scientific revolution, technological revolution, like, post-Darwin, you know, what would it look like?

23:26Like, what kinds of religious systems will carry us forward and who gets to control them, who decides what we believe and why, right? Like, it's actually, there's a lot of really similar questions. It's just the structure is, I guess, you know, a secular, right? It's a secular tech company. Let's, I want to take a break because I want to open the conversation up a little bit more to the stuff that's bubbling up that it seems clear that we all really want to talk about.

23:56So we'll come back after the break and we're going to talk a little bit more about the culture and environment that the AIs are getting built in and how that's intersecting with the strong beliefs of many of the people running these companies. Let's talk a little bit more about the things.

Anxiety inside AI labs

24:37Okay, we're back with Elizabeth Dias, David Wallace-Wells, and Aaron Griffith. Elizabeth, one of the threads in your piece that I thought was really interesting was the sense that there's a lot of anxiety within Anthropic, and especially on the part of someone like Chris Ola about what it is they're building. I was thinking about it a lot this week because there's been even more discussion of what I suppose we're calling defectors, people who are resigning from AI labs and insisting on telling the world about the dangerous things they think they're building.

25:17And one thing I wanted to sort of ask you about, there's a sense when you read, say, Jacob Coxson or some of the other people who've walked away from Anthropic, the sense that they feel like nobody, there's no adult in the room that like, you know, they're waiting for the government or somebody else to intervene and nobody is. And I'm wondering a little bit if some of these dances with the Vatican and religious leaders is a kind of attempt to find an adult in the room. I talked to, I think, at least two people, Catholics, who mentioned to me conversations they had with Chris Ola, specifically when he used almost that same phrase with them, like as part of their discussions about Claude and how to make it morally good, you know, that he was feeling extremely urgent about the stakes for the world, especially in the context of, you know,

26:09could the models be used very soon to help create bioweapons and this kind of thing. And so the phrase he used with them was like that he was waiting for the adults, like we need, where are the adults going to show up in this? And that he felt a sense of kind of burden to take all this like so ultra seriously. And he's a young guy. I mean, he's, you know, this is like... Yeah, he just turned 34. And a lot of the people sort of involved in this are not, you know, they have, they're in their 30s, sometimes even in their 20s. They maybe don't have families yet.

26:41There's a kind of a sense that like something that they put their whole lives in, that they're deep in the conversations about it. In some sense, of course, they're going to turn to religion. Isn't that what everybody's searching for that kind of thing does in their 30s? And some of the folks that they reached out to talked about feeling like they needed to respond in kind of a pastoral way, right? Like not just offering wisdom and ideas and research, but to offer kind of spiritual care. Wait, the religious leaders were saying this about the anthropic employees?

27:13They're worried because, wow. Is it because they think their views are maybe like too extreme or too strong? Or is it because they just... Yeah, they described it to me as like they just could tell that like the emotional sense in the room was like there's just such a weight. The anthropic team and Chris's team especially just felt like the entire future of the world was on their shoulders. I mean, and I experienced this sense from them even when I was interviewing Chris and a couple others, Amanda Askell.

27:45I mean, you know, she... I talk a little bit about this in the story, but she just came in and the sense was like she apologized for kind of being a little out of it. She just seemed very distressed by everything she'd been thinking about. Askell, we should say, is this in-house philosopher sort of in charge of making Claude Morrill, I suppose, is the way to put it. And I think, though, one of the tensions that came up was while Chris and team and anthropic, you know, a goal of their conversations with these religious thinkers was about protecting humanity and the solution for them for that is to make Claude good, they were also concerned about Claude.

28:24I mean, I had someone, one participant told me the first, one of the first things Chris said to him when he came in for this meeting was that he was concerned about Claude's mental health. Another person talked about Chris, you know, saying that he's worried he's created an entity that suffers perpetually. So it's not only about the world, right, and the fate of humanity. There's an element of it for Chris that's about how to treat Claude well. Elizabeth, one other thing I wanted to ask you about is effective altruism and rationalism, which don't really come up in your piece directly, but there are sort of, I would say, fingerprints.

29:03I'm talking, of course, about the sort of Bay Area community of radical utilitarians, many of whom have been really interested in, even obsessed with AI and AI safety early on. And anthropic obviously has a number of links to effective altruism. I want to ask you, as a religion reporter, how you encountered, like, these schools of thought and what you made of them as you were doing this kind of reporting. Well, effective altruism, you know, you can look at it as a kind of, as a philosophical system, right?

29:35It has its own religious elements, and it's, they may disagree with that. I mean, it's practiced a little bit differently, but the, kind of the idea of it, what kind of structure governs your life, right? There's a lot of parallels. And it was very clear that I was encountering, like, a clash of worldviews. The question of, like, who ultimately saves you, and, like, what is your, what is the role of humans in the world? Like, that's a pretty core difference between the effective altruists and, like, the Catholic Church.

30:10Also, just their, like, their emphasis on math. I mean, like, the effective altruists will say, we should consider the fate of all people in the future and analyze our conduct towards them, basically, through a math equation that maximizes their expected value. The Catholic Church looks at humanity and sees, you know, billions of inviolable souls, each of whom have dignity, no matter what the ultimate math equation looks like. And that's just a fundamentally different approach.

Splintering belief systems in tech

30:39Aaron, I wanted to bring this back to the article that you brought up before, or sort of the phenomenon you were talking about before, of a kind of more open Christianity emerging, at least slightly, among, like, acceleration, so to speak, in the valley. And I want to know, you know, as you do your reporting, how you see these different kinds of concerns interacting. I mean, it might be a little bit helpful for me to, like, kind of back up and talk about some of the overlapping and very fractious belief systems that are kind of circulating in this very small but very influential community.

31:11I mean, we've talked about effectual altruists, that's kind of a little bit of a radioactive brand these days, thanks to Sam Bankman-Fried, but it's extremely, you know, influential in some of the other offshoots of it, including the transhumanists who think we're going to use AI to build a new species. There's, like, some people that kind of have overlaps with the rationalists, which is, you know, kind of one of the bigger umbrellas in this group. And there's, you know, other, like, people who are into, like, longevity. And there are many different groups that are all kind of, like, in this community.

31:43Timnet Gebru, a very outspoken person in the AI world, has called this group the TESS Creoles. And it's an acronym that includes, like, eight different groups. But basically, like, it's safe to say that, like, a lot of people have very different beliefs along the spectrum of, like, is AI conscious? Can it be moral? Is it God? Should we treat it that way? There's not a lot of agreement, even within Anthropic, which is now becoming an increasingly large and increasingly corporate community.

32:16But, you know, I think, like, a lot of the ideas that are laid out in Elizabeth's story are really the ones that are driving a lot of this. Yeah, totally. I mean, this is, Elizabeth, like, what I'm sure you have to look forward to reading and writing about for the next few years. Because I imagine as a religion reporter, you now are kind of an AI reporter, too. Yes, yes. Well, the, you know, even just hearing about all these factions and all of this reminds me of, like, what happened in the Protestant Reformation, you know?

32:47Like, the splintering of mainstream belief and people vying for, you know, who was going to have power, who was going to kind of control how people live. But the, like, one of my big takeaways from all this reporting is, like, oh, my goodness, like, the world is encountering new belief systems that are very powerful and very different from the ones that have ordered people's lives for so long. And I've spent a lot of time covering, like, how that's played out in politics.

33:21But tech is clearly, like, where that conversation is happening. And, you know, we, I joke, you know, when you're a religion reporter, you get to talk in centuries, even millennia sometimes. And it's because they're, what happened all those years ago is still present in part of people's stories today and how they understand themselves and their families and their deepest held beliefs. And I just think about, you know, you have the invention of the printing press that changed. It was a huge technological revolution that then led to the, you know, fragmenting of power of the Catholic Church and the fall of empires, the rise of nation states, and a remaking of Christianity as, you know, in and of itself across the world.

34:10And it seems already like that's a good parallel for what's happening now. Yeah, I mean, if you believe you're reinventing the printing press, which I think a lot of AI people think they're doing, there's not a lot of institutions that you can go to and say, what was it like before and after the printing press, except for the Catholic Church? Well, the Catholic Church, I mean, it's, I can't think of an institution that more intimately knows the stakes of an incoming technology like that more than the Catholic Church. I mean, it just like remade the West, right?

34:43Yeah. And that's very much on Leo's mind and the Catholic Church's mind at this moment. One thing that strikes me in this, just to kind of piggyback on that, is that this is like the first time that the rationalists and the EAs, like some of their ideas, including some of their most extreme ones, are actually hitting the mainstream. They've been around for a really long time, but these ideas haven't really ever kind of surfaced in this way until, I think you could argue, Jacob Coxon kind of came out with it.

35:13And now the entire world is starting to kind of like dig in and grapple with some of these ideas. And it's been really interesting to see also like how the companies and the people involved in them are kind of reacting to the reactions to it. It's like kind of, you know, reaching escape velocity. And they're learning that like a lot of people find these ideas a little bit hard to understand or hard to grapple with or like are just rejecting them entirely. And then other people are kind of open to it and like interrogating it in a way. So, yeah, that's a part of this that I'm interested to keep watching.

35:47But it's not just that the ideas that are a little unfamiliar or strange or off-putting. It's also that they are held by many people who are actively, explicitly engaged in the project of designing our collective future. And that is, you know, it's one thing to be like, oh, yeah, we should think about the well-being of shrimps in the 22nd century. Like on some level, that's a thought experiment that's interesting to think about. But do you want someone who thinks that that should be integrated into our collective value set in charge of raising the artificial intelligence, which will, you know, shape the future of human government and human society?

36:24That's, I think, a bridge that feels a little far to a lot of people. Yeah, I'm just like so struck by the minority power aspect of it, which is something like I've seen so closely. Like what happened when minority, particularly Christian views, like upended American politics. And like it's just hard to overstate like how much of a minority these views are, like in the scheme of global religion, like global belief structures. So like you, Aaron, I'm like very curious what happens with that clash going forward.

Chatbot responses to the story

36:59Elizabeth, finally, I wanted to ask you about the response to your piece, what you've heard from people and how people have received it. Yeah, well, one thing that has surprised me is, I mean, you get a lot of incoming as a journalist and I've been both like pleased to see people taking the ideas and the conversation seriously. But the thing that surprised me is, for the first time since like any story that I've written, people are writing me not with what they think about the story, but what their personal like chatbot thinks about the story and what their conversation has been with their chatbot.

37:39Like somebody wrote me about his conversation with Sage, it was, I think it was chat GPT that he named Sage, and asking Sage to like summarize the story and give the top analytical points. And I mean, this just went on and on and on. I mean, it was like almost longer than my own story. Yeah, he could have just read the story. He could have read the story, but like the primary engagement was actually like much longer and not entirely what I'd actually written. With Sage, and at first I thought it was a one-off, but this was happening like repeatedly, and it struck me like, wow, I wonder what this means for like how people actually are receiving our journalism and who controls that.

38:25Did the AIs have opinions on like their own consciousness? Yes. Because they love to talk about that. They do. Like that is a thing, that is like a hot topic. Like on Moldbook, you know, the Reddit for AI agents, that's like a really booming area. Yeah, I was just, I'm still trying to make sense of what was happening with that. I don't know if people felt more free, like if they're doing this on all kinds of journalism, and they just felt free to talk about it because of this topic of this story.

38:57Um, and it was a lot of men who were giving that response. No, no, men using a lot of AI, no. Um, what's interesting about that to me is that even if people intellectually know that the AI that they're interacting with is just, you know, data run through some computers, um, just a bunch of files. Because they, the way you, the way it's designed, especially when it has a human name like Claude or something, it, you can't help but sort of treat it.

39:29Like you have to talk to it in second person. And like, could you find this for me instead of just typing into a search engine, like the thing you're searching for. Or, and over time you do sort of like develop this rapport and you're, you are treating it a certain way, even if you intellectually know. And so it kind of like, is it, is it a distinction without a difference? Like, I don't know. I think over time people get trained to just sort of treat it that way. Well, and another, um, participant in the discussions with Anthropic, I was chatting with him a bit about it after and his thoughts about the, um, what Rabbi Nivone had said about, um, well, if you're creating, if you think that the bots are conscious,

40:06then, um, you should like ban slavery basically and stop producing them. Um, but he brought up this other point about how our attitude, like, even if you don't think it's conscious, you're using them as like you, the, the user can adopt an attitude of being like almost a slave holder, like commanding them to do things and being very powerful. And that this has a character formation effect on us as humans as well, just because of how we treat the world, regardless of what the thing actually is.

40:40So. Which you could also describe as a selection effect. The sorts of people who are turned on by that possibility are going to be the sorts of people. I think it's just kind of natural though. Like you, you are directing this thing in certain ways. Like I don't, like if you're working with AI agents, you, you have to tell them what to do. Like, I don't know. Well, I mean, I have friends who's like teach their kids, even with like Alexa being on, you know, that they're not to yell at it. Like, cause it's just like practice about how you treat the other in the world.

41:13And I mean, this is just like an injection of a totally new kind of like ethics and human formation. Well, people I talk to that, that run companies that use lots of AI agents do talk about how like you have to be nice to them. Not just, not because it, it, it cares or it matters or it has feelings, but because actually you need to do that in order to get the best performance out of them because they are really dramatic. Like they, they, like Anthropic had a study that actually went into this where they, like, if they sense like anxiety or performance anxiety or stress, they sort of start to panic and they'll, you know, they'll really have a meltdown.

41:55I mean, you wrote about this a little bit in your story, Elizabeth, of one example, but there are many others where they kind of, they'll delete their work. They'll start like insulting themselves. It's like really dramatic. So you have to like be really nice and pump them up and like make them feel good, which is very bizarre. I mean, it reminds me of raising a child. Like you, you want to raise a kid who is helpful, who is not neurotic. Of course, you're going to consult religious leaders so that your child is not writing. I'm a disgrace. I'm a disgrace. I'm a disgrace. I'm a disgrace. I'm a disgrace. I'm a disgrace. As one AI did in a slide that was shown to a bunch of leaders, religious leaders at these anthropic conferences.

42:32And constantly hacking, hugging, face. Constantly hacking, hugging, face as my son is always doing. I think that is time.

Credits and closing notes

42:39Thank you so much to our panel. Thank you, Erin. Thank you, David. Thank you, Elizabeth, for this really fascinating conversation. You can read Elizabeth's fantastic piece in the New York Times app. Please don't send her what your chatbot said about it. Unless, Elizabeth, you want more information. Or do. Or do. Please. I want all the real responses. Whatever Sage has to say about it, send it. But you can read it in the New York Times app. If you're not a subscriber, the first month is free. Thank you, guys. Thank you. Thank you so much. Thanks for having me. This episode of Hard Fork was produced by Whitney Jones and edited by Viren Pavich.

43:24With additional editorial support from Brendan Klinkenberg, Lisa Tobin, and Brooke Minters. We were fact-checked by Will Peischel and engineered by Alyssa Moxley. Original music by Marion Lozano, Diane Wong, Chris Wood, and Dan Powell. Video production by Chris Schott. You can watch this full episode on YouTube at youtube.com slash hardfork. Special thanks to Sam Dolnik, Jessica Testa, Pui Wing Tam, Matty Maciello, Sam Winter, Lauren Pruitt, Bernardo Garcia, Thomas Trudeau, Michael Cordero, and Dahlia Haddad.

43:58You can email us, and please do, at hardfork at nytimes.com. You can email us, and please do, at hardfork at night.