AI Won’t Take Your Job (But It Might Change It)

Introduction

0:00Hello, and welcome to the Harvard Data Science Review podcast. I'm Liberty Vittert, feature editor of the Harvard Data Science Review, or at least the AI version of me.

0:15Yes, AI is reading this introduction, and yes, it even helped write it. But don't worry. Our interview is all real people, real voices, and real conversation. I'm joined, as always, by my co-host and editor-in-chief, Xiaoli Meng.

AI Impact on Job Market

0:32Today, we're looking at AI's impact on the job market, how it's affecting both young workers entering the workforce and those already established in their careers. Is AI taking jobs away, reshaping them, or creating opportunities we can't yet see? To explore these topics, we've invited two distinguished guests, Ben Waber, visiting scientist and lecturer at MIT, and Rafaela Saden, professor of business administration at Harvard Business School.

1:02So let's dive in with actual humans in conversation. In honor of AI and the workforce episode and how AI is transforming the workforce, we decided to generate all of our questions today completely using ChatGPT. So if it goes really well, then ChatGPT is working, I guess, and totally changing the workforce. But it was funny because we were talking about it, and something that would usually take us a couple hours

1:33to really put together the questions was about, you know, one minute. So we'll see how this goes. But we thought it would be good to do in honor. Did you generate the answers, Stu? You know what? In real time, if you want to generate the answers, there we go. Then we'll see. Then, you know, this would make this podcast easier. Then it's just the Ouroboros of AI just eating itself. But yeah, anyway. That's your job, yeah.

AI in the Workforce

1:56So when people talk about AI, you know, really transforming the workforce, a lot of times into a lot of people, it feels very abstract. And it's, you know, maybe something you've seen in the movies, or it's this, you know, in 10 years, no one's going to have a job. But from your perspectives, what's happening right now? What are these most sort of immediate and concrete ways that AI is already shaping how people work day to day beyond just generating interview questions?

Generative AI

2:27So let's talk about generative AI for a second, because I think that's what most people right now are talking about, things like large language models and things like ChatGPT, right? But what they've done mostly is give executives at companies permission to credibly cut large numbers of the workforce or give their existing workforce more work with not more compensation and juice their profit margins a bit by doing that. Now, I think there is going to be long-term costs to that in terms of performance of these

2:59organizations. And you've already seen some organizations, you know, go back on some of these moves in terms of saying they weren't going to hire any more people or getting rid of people and then rehiring. I think that's actually one of the most near-term things that's happening. A big reason for that, I think a lot of the experience that people have with these tools, which are damn impressive and like pretty cool, is using them in very short-term situations situations where, for example, like the idea of, you know, mocking up a mobile app in five

3:32minutes and something that, you know, as a former, you know, computer science student would have taken me three days in the past. And that's amazing. That's very cool. That's very different than building actually a working app. And I've actually, I'm doing some work right now with Matt Bean out of UC Irvine on this specifically. But this is really where companies are hurtling without fully understanding the technology. Maybe I'll stop my rant there and pass it off to Raphael to maybe have a probably different perspective than I do on this.

Expert Perspectives

3:59I'm so glad you started. So I think, you know, at a high level, there is one piece that we agree with. And I think it's the notion that everybody talks about AI and nobody really knows what they're talking about, maybe including ourselves, because this is such a, you know, this being a general purpose technology or at least having the potential of being a general purpose technology, where I differ from your perspective, Ben, is that there isn't necessarily an excuse in using AI in this hype term because we have some malicious intent.

4:31Maybe some companies are like that. But I think that this is a technology that necessarily, as it happened with past technological waves, requires a huge deal of firm-specific experimentation before we can really understand what it does. So you were asking, what is the most immediate use? I think that right now there are some very well-documented individual uses, which could be, you know, as AI basically as a co-pilot that helps us find information or summarize information

5:03or maybe visualize it better, specific tasks that AI can do, which is very far from the notion that AI substitutes entire jobs. By and large, we are still talking about individuals using AI, potentially seeing some benefit in terms of time, but I would say not yet super well-documented. And what we know is that there hasn't been the organizational transformation that I think is a potential for this technology, but still needs to be discovered.

5:35And I think this is, you know, in my mind, when I think about AI, I think immediately about the J-curve, which is a pretty well notion of, you know, what happens to productivity when you are in the wake of a new technological wave. There is a period of losses even in terms of productivity because you have to figure it out. And I think this experimentation, I will reiterate, is firm-specific. Nobody has the playbook yet. And if people tell you that they have the playbook, I don't trust them. I think companies need to figure it out. And then there will be organizations that I think will get stuck in this process.

6:10But there will be also, I think, potential organizations that are either native with this new technology or that will have the ability to learn and understand from different use cases. And most importantly, bring their organizations with them. Because that's another piece of AI that we don't talk about. But the systemic application of AI will require some fundamental changes to organization. And it's not clear to me that everybody wants to go there. Well, thank you to both of you. I think what you said, especially the J-curve, I certainly can relate.

6:43I'm still waiting for the J-curve to happen to me because I keep wasting more time. And so speaking of time, I also appreciated that we have these questions generated by the AI is, you know, very quick. And I'm going to read one of them. And the listeners will know that I'm reading because this one, you won't detect any chinglish in it. So, you know, I'm reading.

Job Displacement Debate

7:08There is an ongoing debate where AI mainly displays jobs or make works dramatically more productive. Where do you each see the balance tipping in the next decade? Raffaele, you go first. Yeah, I can tell that this is a question generated by Chanty, because it makes no sense, right? Okay, good. Even better. Yes. Tell me why it makes no sense. So let me tell you where I am with this. I think that it's impossible to talk about the effects of Gen AI in aggregate terms right

7:43now. It's really, I think we are going to see some pretty big heterogeneity or differences in how AI is adopted across different organizations. So it's possible that if you look at the aggregate data, you won't see much because you will see some organizations doing nothings and others really going out there, which I think is already happening. And there is some evidence of this, for example, in the census data. The Denmark paper recently also showed that. For example, right?

8:15So this is Anders Homelund, who is a Danish researcher who did a very nice study of the adoption of AI across Danish workers. And he found that there is a very large heterogeneity across workers that depended on the policies of the firms, how much they were investing, how much they were trading. So I think that's like, first, I would focus more on the variance than the aggregate in this phase. And second, what happens is a choice.

8:45There is no predetermined trajectory. If AI is going to replace workers, there will have been a choice of automating certain jobs, for example, which some jobs may be automated. If AI ends up creating new jobs, I think that this, again, will be a conscious choice that might come out of a discovery process, but it's not written anywhere. And so it's a little bit up to the creativity and the imagination and maybe even the courage of people to go out and find out new ways of doing business where we will end up.

9:19And this time, I'm glad you went first. I think you framed it much better than I could. I mean, I want to point out that one can still get a paper published in an economics journal talking about the employment effects of the introduction of the typewriter, okay? So the typewriter came out a while ago, and we still don't really know everything about the employment effects of that. So whenever people say, oh, first of all, it's not even a single technology when it comes to like AI today. But when people make these prognostications about certain things, not only is it very impossible

9:50to know, but as Raphael said, these are choices, you know, that we make around how we design these technologies, around how we adopt them, right? And I think that right now, first of all, not only is it very hard to play some of these things out, but a lot of the predictions that people are making are based on, I think, very fundamental misunderstandings of even what the current class of technology can do. You know, even people talking about, you know, summarization. Like, these things actually can't do reliable summarization, and we can have a long discussion about that, some great work by some folks out of UMass Boston. But part of the reason I think that this discussion has gotten, I think, framed not in this conversation,

10:25but in general incorrectly, obviously, you have people with certain incentives that want to frame it in a certain way, you know. But beyond that, a lot of the way is just how we tend to think about even experimentally testing new technologies. And people say, oh, like, ChatGPT is superhuman on math, which is an insane thing to say, right? Like, we don't say a knife has superhuman cutting ability. I mean, it's true, but it's just, like, you don't, I wouldn't use my hand to cut a sandwich. Like, that is incorrect. You know, the question, of course, is what is the correct tool for solving a problem?

10:56And again, how should we design those tools, as Raphael was saying. But I just want to use one example of, actually, addition, all right? So if you have people with a high school diploma in the U.S. and you have them do four-digit addition, they'll make an error about 0.7% of the time, okay? So now let's pretend I have a generative AI calculator, which you shouldn't do. But let's say I make one, right? I make a generative AI spreadsheet that's better than people on four-digit addition. It only makes an error 0.5% of the time. So you could say, oh, that's better than people, right? But just like I wouldn't have a person calculate every cell of a 1 million, you know, cell spreadsheet,

11:31you know, if I now give you a generative AI Excel and I say, oh, a random 0.5% of these cells are wrong, that is unusable. Like, that's actually unusable as a tool, right? And so I think we've really painted ourselves into a corner in terms of, first of all, how we compare these tools. The correct comparison is not a generative AI program versus just, you know, a naked human that you pull out of the forest. It's like, okay, using a calculator, right? And then beyond that, the question is, all right, like, how should we design these tools? I think it's fascinating that we've brought back command lines, right, as an interface,

12:03despite the fact that they're a terrible interface, but now we've also made them stochastic, right? Like, it's just interesting when you think about, like, the computer mouse or the keyboard. Like, actually, there's literally, and Raphael can correct me if I'm wrong, but I think there are zero papers written on the productivity effects of the computer mouse, right? Which is fascinating, right? No, no, because I actually looked at this. I've looked at this. I haven't checked the last month. That's shocking. Imagine I removed the mouse from every computer, you know, like, people's productivity would plummet, right? But the thing is, it doesn't exactly automate what people have done.

12:36Like, we can do new things, right? And I think this is where a lot of the interesting things in the AI space in general, I don't even just mean general AI. I think a lot of the interesting things in the AI space will happen much more around interfaces, right? I think a lot of the way that these systems are built, even things like cancer detection, it's like output a prediction, but we don't have to output a prediction. It could, you know, if we get an x-ray, it could highlight a part of the image and a person looks at it. Now, that's the exact same technology. It's a different design. I think that's where the interesting things are going to come. But I do think to the extent that we keep honing in on, okay, can this do better than a person at this task?

13:08We're going to keep getting, frankly, like a very limited, crappy instantiation of the technology versus what we could do if we think, you know, there's a whole field called human-computer interaction that does amazing things that I think people in the space have been ignoring. And I think that's where a lot of the interesting stuff is going to happen.

13:25Yeah. I mean, if I can, I think I agree with you, Ben. And I think that the fundamental issue, and, you know, you should know I teach strategy, so maybe this is my personal bias, but I think that the fundamental issue is focusing on the specific technological applications rather than thinking about the opportunities for value creation. And that, I think, is a different way to think about the technology. And I want to be specific here.

13:56What excites me about AI and generative AI in particular is this notion that, I think, David Autor from MIT, I've seen it from him first. Maybe other people said it before him. But it's this idea that generative AI can free tacit expertise. So, you know, we would think of experts as knowing people as experts. And, you know, these retrieval opportunities and generative AI applications can help us access expertise that might have previously been completely bottled in individuals.

14:31And so if you think about this type of application, I've actually run experiments that show in a control setting that this type of applications, which are not AI substituting humans, but it's, you know, specific tasks potentially being augmented by expertise that happens to be channeled by AI, could lead us in new places and new value creation. So, for example, I'm working on projects where we are trying to see whether I can help blue-collar workers solve problems in autonomy.

15:05Because, you know, usually to solve a problem on the line, you would have to call an expertise, an expert, an engineer. Whereas now there are organizations that are thinking about interfaces that allow workers to access this knowledge in a way that is much more intuitive and easier than before. And so those are the type of applications where I think, you know, if we can think in this frame and if we can validate that they actually add value, this could potentially be very exciting. I feel like, you know, this all makes a lot of sense to me, but, you know, I would say the general person,

15:39but I think even people with a lot of expertise in data science, data analytics, you know, when you hear these sort of statements about, you know, there's not going to be any jobs for people under 30 in five years, there's not going to be any, you know, in 10 years or five to 10 years, it creates a huge amount of fear. Is there any validity to these fears? There could be validity, but I do think that there's some short-term problems that are going to be very real for people. And I want to use an example actually from MIT. And again, I don't want to scoop myself because I'm writing a piece on this, but just to give a preview of it.

16:14So I was being a mentor at one of the hackathons there at the beginning of the summer. Okay. And these are mostly MIT undergrads, grad students, but they'd all have completed at least a year at MIT. Okay. And what they had to do is they were, you know, mocking up an app and they had to show working demo. And then if you won, you actually got like tens of thousands of dollars. Like it was for real stakes. Okay. And, you know, I'm walking around just trying to help some of the groups out. All the groups are using co-pilot technology to code up their apps.

16:46And again, like I said earlier, you can make these things that would take you days. You can make them in minutes, which is amazing. But they actually had to make something that worked, right? At least, you know, for like a demo purpose. And so I had a group come up to me and they said, all right, well, you know, we have an error in our app. Can you come look at like the code? And just to be totally clear, like, yeah, I have a BA and MA in computer science and I got my PhD, you know, in this. But I haven't coded in like 10 years, like seriously, right? So, and I told them this. I was like, guys, you really don't want me to look at this code. Like I haven't looked at things. And they're like, oh no, but, you know, it would really be helpful for us.

17:18So it was fine. So I start looking at this code and it's, I don't know if any of, you know, any of your listeners or you have actually looked at, you know, code generated by generative AI that's like a larger program, but it's inscrutable. Like it works, but it is, like, it's unreadable. Like you have no idea why it works basically, right? And so after playing around with for like 20 minutes, I'm like, listen, guys, this is going to take me hours to figure out. Like, let's just look at the error messages. It says there's a variable here that's got a problem with it. Let's, you know, try to debug this the regular way.

17:48So why don't you put it in a debug print statement just to figure out what's going on? And these, these MIT students, right, asked me, how do we do that? Like, if you have even taken an introductory programming class, like, you know how to write a debug print statement. So this is very concerning to me. So, okay. But first you could say, oh, well, that was just one group. Okay. Then another group came up to me and another group and another group. The exact same. And these are people, some in grad school, I mean, you know, who clearly had at least, you know, done passively well in college level classes with that kind of, you know, ability.

18:21And so, Liberty, to your question, I think this is the concern that I have is that in terms of what I actually think that in the near to medium term we're doing is de-skilling, or at least we're not allowing this, you know, core skills to develop in a whole class of students and workers. Then again, in the future, I think this will be corrected with better designs and other things, but I think in the near term it's going to be a really big problem. And so I don't think that this means that the technology itself is going to, like, replace the jobs, but I do think that there's going to be an awful lot of people who might be hired for jobs that on paper they look qualified for, but they will be very unqualified.

18:57And we're going to have some really big problems in the, you know, in the medium to long term, actually, for this cohort. I don't think it's exactly getting your question, but it is something that I'm extremely concerned about from the workforce perspective. Actually, even more than a lot of the, like, we automate away all the jobs. Yeah, I mean, I think that, you know, there have been some, you know, recent papers that have shown this decline in hires among the youngest cohorts and a corresponding increase in, for specific occupations that are more exposed to AI and increases in older, more experienced workers.

19:34So understanding what's going on, I think, is relevant. I don't know if it's an issue. I actually don't think that it's an issue of AI already substituting for these workers, because as Ben was saying, at least as far as I know, for many jobs, we are still very far from where, you know, you have entire jobs that are substituted and automated away. But what might be going on, which is entirely consistent with Ben, is that this is a technology, AI, that is most productive when you have contextual knowledge.

20:04And so you put AI in the hands of an engineer that knows the debug statement and, you know, you can do a lot of cool stuff. It could save time. It could maybe increase creativity. Who knows? But that person would also be able to understand when is it that AI is making a mistake or maybe go inside the code to understand exactly how to use it. And, you know, I have a friend, he has an AI company who was telling me, look, my senior engineers are very concerned about hiring young people, not because they believe that their jobs can be automated, but because if I give AI to an experienced person, they can destroy the code for the whole organization.

20:42Because they actually don't have these foundational skills that are needed to make good use of this technology. But then the implication, and I'm very sympathetic with Ben, and the problem that we have to figure out is how do we bring young people to the point at which they can make the best use of this technology or vice versa? How do we think about a way to be able to nurture people inside organizations, knowing that there is a technology that eventually will be very helpful for them?

21:12And that's a much more complicated argument than just saying there is no jobs for young people. Well, looks like the AI we used had anticipated your answer. So here's one. And again, I love the ideas that we humans say, hey, that does not make sense at all. Okay, let's see if this one makes sense. If you were advising a student or mid-career professional today, what specific skills or ways of working would you say are most future-proof in an AI-driven labor market?

Future-Proof Skills

21:46All right. I also don't – I don't really know. I think I always like – I mean, listen. I always like – I think statistics is always useful, but I don't know if that future-proofs it. It's more like I think having a statistics – at least a basic grounding in statistics is helpful because it helps folks interrogate claims that people make about whether that's technologies, whether that's about improvements more broadly. I think that's helpful. Whether that's specific to – because we have more AI introduced, like I don't know. I mean, I do think that in general, like what are people really good at?

22:17People are really good at being creative, coming up with new ideas, and communicating with other humans. We're very, very good at that. You know, the ability to collaborate and build relationships effectively with other people is always useful. I mean, that's all I got, which is not very sexy. But I don't think that I've got much more if that's any consolation. I think, you know, one piece, however, that I would really push for if somebody asked me that question – and I keep asking this question at education experts, and nobody really has one answer.

22:47There's a lot of uncertainty, and we just have to recognize that. But what I would prioritize is getting access to organizations that give you the permission to make decisions, to develop experience, to develop that contextual knowledge that I think will be really, really critical going forward. It's like, how do I apply the technology? How can the technology help me? And so perhaps where we will go is not so much the idea that you do college and you are a fully formed, skilled person, but actually maybe we have to consider ourselves more as apprentices.

23:25And that college experience is not going to be where we are fully formed. It usually isn't anyway. We learn a lot on the job. But I think it's going to be even more important to think about giving students time to get experience inside the real organizations and making real decisions. Because my suspicion is that that type of contextual knowledge will become very, very relevant to distinguish yourself from others. You know, Rafael, your research seems, according to ChatGPT, emphasize management quality as sort of a real key driver in predictivity.

AI and Management

24:03So how specifically will AI change what good management looks like? Because I feel like we've talked a lot about or people are talking about AI a lot for, you know, jobs for people under 30 or, you know, sort of these jobs that aren't what we would call management. So how could management really use this? Yeah. So, look, I mean, going back to what we discussed at the very beginning, think about AI as for companies is like taking a walk on the jaker.

24:33OK, now, at that period, when you experiment and when you have to figure out what to do, that is typically not a good time to put very specific and quantitative, you know, measurement or, you know, target revenues and so forth. So my sense is what you do and what's considered optimal depend very much on where you are in that stages of adoption. I think that at the early stages of adoption, it's more about leadership, giving people a direction.

25:03And I insist on the strategic direction, knowing more or less where we are going and giving people space to experiment and learn from this experimentation. Once you are over that and you have some understanding of how value can be created, that's where management and, you know, the creation of KPIs, the creation of metrics, monitoring how things are used, that could be helpful for the implementation stage. And I think that there are different stages of adoption, one that is more uncertain and one that could be more certain and more about scaling things that we've figured out.

25:42And for each of these stages, you might need very different types of different attention, attention to different things and perhaps different people, too. Well, Ben, you have studied, according to Judge Biddy, how behavioral data inside companies reviews hidden patterns in collaboration. How might AI accelerate or even automate organizational decision making based on these kinds of insights? Yeah, so I think that, again, it more or less got the gist.

26:16I think this is the thing. It's sort of like a horoscope is that you can say, you know, vague things that are mostly right. It's like fine. And then we're filling in the blanks and it's fine. But no, I mean, I think that I am very bullish on using large language models for data labeling, which is importantly different than summarization, right? But I do think that if in the past I would have paid people on, like, Mechanical Turk to, like, label, like, is this email happy or sad or something like that, I'm quite convinced that one can, you know, fine-tune an LM to do that.

26:52And I think there's some very interesting things that one can do with that, right? I think in the past, a lot of my work had been on looking at patterns of collaboration and how that relates to outcomes. For a lot of reasons, looking at things like content or other different types of patterns was a lot less predictive. But there was obviously a pretty hard limitation on some of the approaches one could use to try to characterize that kind of data. And I think there's some very interesting things that can be done on the labeling side. I mean, I really think actually that's where the vast majority of value from generative AI technology, at least in areas where the truth matters.

27:26I think creative tasks actually is a different thing. But in areas where the truth matters, I actually think that that application is going to be the biggest one. It's probably going to be a lot smaller than what a lot of people are saying in terms of, you know, automating jobs or whatever. But I do think that even as Raffaello was saying, really making use of that kind of capability does require really rethinking how one designs processes within organizations, you know, what sort of applications one has. And so there is some new work in that area that's just getting started. But that's sort of what's interesting to me. ChargPT was very smart here.

27:56It said the question spotlights Ben's expertise, then invite Raffaello to connect back to organizational structures. You know, it's give us the kind of a way to do it. So I don't know. Raffaello, do you want to say a few words in terms of how the structure, the organizational structures? Oh, yeah. I mean, yeah. So caveat is I think there are different models right now and, you know, companies are figuring it out. I think the optimal organizational structure to implement AI doesn't exist yet, or at least I haven't seen it.

28:31What I believe might be, you know, connected with what Ben was saying is one talent that will be really necessary here is to identify who helps you learn at this moment in time. And this is not trivial in the sense that what I'm seeing also in very advanced organizations is that people with very strong technical expertise may not be the best people to help you understand how to creatively use a technology for a new source, for a new way of creating value.

29:04Because, you know, they have been trained in a certain way, they've been using technology in a certain way for their whole time. They might have some justified reluctance to change what they're doing. And so I think that one specific piece that organizationally you need to figure out is who do you recruit at the very beginning of your adoption to understand what's technically feasible? And then who do you recruit to help understand what are the potential use cases? And typically one possibility would be to think about skunk teams, people who are, you know, far from the established organization so that they can invent.

29:42And then lastly is, again, how do you bring back this knowledge so that it gets the funding that it needs, it gets implemented, it gets scaled. And so organizationally, as you can see, there isn't just one model. I think it's really understanding how do you position yourself to be able to learn and select what's good for you and then enable the organization to adopt. Rafael, you were talking earlier about how we're going to see, you know, some companies really adopt this, some not do anything at all, and, you know, this sort of widespread.

30:18And so AI can really, I guess, sort of widen the productivity gap, not just in companies or in, you know, sectors, but also between countries. And I think that's one of the big questions. I mean, what do you see as the risks going forward for something like that? Oh, I'm seeing what I've seen 20 years ago with my first PhD, the first chapter of my PhD dissertation, and it was about the adoption of computers. In organizations, so the famous solo paradox, we see computers everywhere except the productivity statistics.

30:55We were talking about this stuff more than 20 years ago, 30 years ago. And it turns out, that was stuff that I studied and many other people studied too, that the productivity impact of computers was really dependent on the type of organizational and managerial practices that companies were using. In particular, this ability to be flexible about the ways in which you were thinking about talent and the ways in which you were allocating talent. Give you one concrete example that, again, has been studied a lot.

31:28The introduction of ATMs in banks created a very different ways. You know, these people that now greet you at the entrance of the bank, that was the substitution of the cashier and the introduction of new jobs that were much more based on human skills and not so much in these routine jobs. So, that's why I'm sort of bullish on the idea that we will see a ton of heterogeneity across firms. And that's why I also think that one potential risk is that you will end up seeing heterogeneity also at the macro level.

32:01And in particular, you can hear from my accent, I'm not British and I'm not American. And I'm actually Italian. And one concern I have for countries such as Italy, which are already behind from a digital adoption perspective, is that we will not catch up fast enough. Italy and other parts of continental Europe were already and are already behind in terms of digital skills for basic applications. And may risk not being agile enough to get into this continuous experimentation mode that I told you.

32:35And therefore, they may be passing on opportunities that this technology may provide to them. Think about the importance of manufacturing in some European countries. This could be fatal. You know, it could be really potentially very problematic. And in fact, the adoption gap, I think, for Europe is potentially much more problematic than the innovation gap or, you know, the creation of new LLMs or new AI companies. So, I see this as a real risk.

33:07And I think my hope is that there will be a way to spur experimentation much more, not just in the U.S., but also in other parts of the world. So, we have been talking about the human-machine interaction. I think what we call a human-computing interaction. That's a, you know, buzzword. So, here's a question from ChatGPT.

Human-AI Collaboration

33:27What does an ideal model of collaboration between humans and AI look like? There are examples today that give you optimism about how that relationship might evolve. Maybe, Ben, you go first. Sure. So, I also think, like, I think that is a flawed, like, that's not the correct model. It's not a person. It's not an agent. It's, like, we used to call things computer programs, and now we call them agents. It's the same thing. Like, that's, it's just, it sounds fancier. Like, it's a tool, right? And the question is, how does one use it effectively, right?

33:59But you don't say I'm collaborating with a calculator or I'm collaborating with Excel. Like, that doesn't make sense. And so, organizations that then, you know, implement it, this is Raviella's point, I mean, to the extent they're actually experimenting with it and actually measuring outcomes, you know, to the extent they do use it incorrectly, hopefully they would have metrics in place to understand that they're not getting the outcomes that they want. I do have a very large concern that most organizations do not have that. And the problem is they look at things that are very easy to measure that are short term, right? So, I know a very large technology company that shall remain nameless has mostly replaced

34:33their HR department with large language models, okay? But again, that doesn't mean that tomorrow they lose a billion dollars, right? And again, in terms of saving time, right? Like, just like writing the questions here, like, you all save time by writing those questions, which is, you know, whatever, it's fine. This is not, like, a big deal. But now let's say I get a large language model that writes the employee handbook for the company, right? 50 pages, let's say. Now, again, should a person read through those 50 pages and, like, check that everything's correct? Like, yeah, but listen, people are people.

35:03They're not going to do that. That's the whole reason you use this is to save yourself time. And so now let's say there's not a section in there about sexual harassment because most of these systems are fine-tuned to avoid sexual content. That doesn't mean that tomorrow you lose a billion dollars, but it could mean that in a couple of years something horrible happens, right? And then you get sued for hundreds of millions of dollars. Now, are you going to be able to go back and say, oh, that's because we used a large language model? No, you're not. That systemic risk is still in there. And so I think part of this, though, gets to how one thinks about using these tools. It's about being both a good programmer as well as having a good mental model for how

35:38these tools work. And the problem is the mental model that most vendors and even a lot of academics are promulgating of these tools is as people, but they're not people. And that's a problem because the way you would treat a person is just so fundamentally different from how I would use an Excel spreadsheet. But I think having an appropriate mental model is going to be important. And I think the mental model that I would suggest folks have of these tools is it's like a cookbook written by random people on the internet, okay?

36:09So you would never say that a cookbook knows how to make chocolate cake. Like, that doesn't make sense. Like, the information is there. But also, if you don't know how to make a chocolate cake, so let's say I turn to the recipe page for chocolate cake and I look at it. If I already know how to make a chocolate cake, I could discern, is this a good recipe? Probably. But let's say I haven't made it before, right? Now, this recipe could have been written by a great baker, but maybe it's written by someone who doesn't know what the hell they're doing and it's actually terrible. So I need to have the ability to actually discern that. But beyond that, now let's say I bake the cake, but I don't even taste it myself.

36:42I give it to someone else and I don't ask them what they think. And so this gets to the way that you should bring these tools, especially as currently constructed, into your workflows very much is going to require both people who can discern that those outputs are appropriate, that you have an understanding of actually how these tools function, right? And I worry that rather than doing this in a methodical, experimental way, I think most companies are trying to jam these in very quickly in a non-thinking way, again, without having

37:12metrics in place. And I think that's going to have big problems. I'm not actually sure that I really answered the question, but I at least answered the question that I wanted to answer. So that's hopefully OK. That's what a human does. So it's good. There we go. There we go. Exactly. And Rob Hallig, do you have any response? Yeah, I mean, I think the so your question is about the optimal human computer interaction, right? So where should we go? Yeah. So, you know, I think that that's where what I was saying before comes back and it connects

37:49also with what Ben just said. I think that first of all, it's hard for me to understand or to evaluate something that is, you know, taken off the shelf and just introduced in an organization, because there are many things that are very firm, specific about what is a good output. And I think that the way the good human computer interaction starts with the design of the technology, which I think if we believe in that promise that there is some expertise that

38:20can be disseminated and leveraged by others, you have to spend quite a bit of time thinking about and training the machine and making sure that it does have a good model of the expertise and it has, you know, some guardrails that allow you that, you know, prevent things from going really badly. I think that that's where you start. And then the second piece is, you know, human computer interaction. We need to understand interaction. What does it mean for a technology to add value?

38:52So first of all, you have to have hypotheses of how value might be created. And these hypotheses are necessary to form an experiment. And then you need to be able to measure and to have a good understanding of whether things went according to your hypothesis or not. And so what I'm getting with that is that I think that human computer interaction is one piece technology, but a lot of it is actually how you train the technology. And an even more important piece is how do you learn how this technology helps your people?

39:23And so I know that Ben really believes in that, and I believe in that too. Companies and organizations in general, you need to get much, much better at becoming experimenters. And this means formulating the hypothesis, monitoring, measuring, learning, and doing this continuously, not just once. Because that's, I think, at the basis of creating a productive human-computer interaction. Thank you to both of you so very much. And I'm sure we could have asked Judge B to generate another two hours questions.

39:56But we need to wrap up here. And as Liberty and I said, we were trying to experiment in here using AI to generate all these questions. But we do have one coming from us. That's the magical one question. So if you could wave a magical one and put the AI genie back into the bottle, so to speak, which part of what you put back? Okay. So it'll be a very specific thing. Well, I would... I don't know if this is putting it back in the bottle, but I would make it so that systems

40:30could not respond with terms like I or say I'm thinking. Yeah, I mean, it would also not have a text box interface. Because having an Omni tool makes no sense, right? Like, why do we have one thing to try to do ever? Like, that's just, you know, that's a minimal ask. There's probably bigger things that I could ask that would take way too long to expound on. But that's maybe my not so radical ask. I think it's still doable. Like, we could just... Even Raphael said this. We could choose to do that tomorrow. We're choosing not to. But we have full control over what we're building.

41:02Yeah, I mean, it's a tricky one. I'm going to go in a different place. I think I am intrigued at the same time scared by the ability of AI of making people feel heard. And I think that that's like, it's not so much putting it inside because I can see how this could actually help with loneliness or, you know, some aspects of solitude. But I'm very, very scared because at some point when we've seen it, unfortunately, there

41:34is a limit where you really need a human. And right now, that threshold of helping people feel a little bit less lonely, but, you know, really knowing when is it that they need another human being to be with them, there is no clarity. I think that the machine is not good there. And I think that that's actually something that concerns me because I see a lot of loneliness around. And if only we could use this technology to help alleviate this loneliness, but also to help people

42:08get the human help when it's needed, I think we would have, it would be terrific. But I don't know if that's where we are going. Thank you again to both of you for this really insightful conversation. And normally for a HDSR podcast, because we're facing, you know, the other data science community, we will have a question more about data science, but this is the AI generated. We didn't prompt it, so it doesn't have much in it.

42:38But I want to emphasize that as a statistician, I was so pleased to hear, Ben, your emphasis about everyone should know something about statistics. And also, Raffaele, your emphasis on everything started about thinking about hypothesis, you know, then experiment, then measurement. These are very much things we teach our students about how to do research, how to think about, you know, any problems. So I appreciate that. And with that, I want to close again by thanking both of you.

43:10And thank you so much. And Liberty, I know you are with your little baby, the future workforce. I know his grunts. He's like a green or not a green because he grunts away. That's the ultimate test is there. So again, thanks, everyone.

43:29I'm Liberty Vittert Capito. And on behalf of Shali Meng and our guests, thank you for joining us. And a special thanks to our producers, Rebecca McLeod and Tina Tobey-Mack. This was the Harvard Data Science Review. Everything data science and data science for everyone.