326. Rethinks: The Secrets to Getting the Information You Need from AI

Newsletter and sponsors

0:00Hi, Matt here. I am super excited to share that our September newsletter is now available. Our newsletter shares exciting things we have going on in the next month or so, and you'll learn practical, tactical skills for handling conflict in your life. That's Think Fast, Talk Smart on LinkedIn, or under resources at fastersmarter.io. And now a word from our sponsors. Their support covers the cost of production, allowing us to bring you this episode free of charge.

0:30The more time I spend thinking, teaching, and communicating clearly, the better I can do the work that really matters to me. And one thing that helps is spending less time just getting up to speed. Before I reply to an email, prepare for a meeting, or follow up on a conversation, there can be a lot of context to sort through. That's why Superhuman Go stood out to me. It's from the makers of Grammarly, which I've relied on for years. And unlike AI tools that require another tab and often require me to explain what I'm working on, Superhuman Go works where I already work.

1:01It's already up to speed and right there when I need it. I can make sense of a long email thread, prepare for a meeting, or draft a response without switching apps or losing my place. Go handles some of the most repetitive work that I do, so I can stay focused on the work that actually needs me. Try Superhuman Go from the makers of Grammarly. Find out more at superhuman.com.

AI as a thinking partner

1:28AI can generate answers in seconds, but the quality of those answers depends on the quality of the conversation. I'm Matt Abrahams, and I teach strategic communication at Stanford Graduate School of Business. Welcome to this Rethinks episode of Think Fast, Talk Smart, the podcast. We're revisiting my conversation with Jeremy Utley and Kian Gohar to explore how conversational AI can become a true thinking partner, helping individuals and teams solve problems more creatively and communicate more effectively.

2:00Enjoy this journey into our communication vault.

2:05The promise of AI is tremendous, yet most of us, if we're using it at all, are using it incorrectly. It's not about a transaction. It's about a conversation and interaction. I'm Matt Abrahams, and I teach strategic communication at Stanford Graduate School of Business. Welcome to Think Fast, Talk Smart, the podcast. Today, I am really excited to chat with Jeremy Utley and Kian Gohar.

2:37Jeremy is a repeat guest to Think Fast, Talk Smart. He's an adjunct professor specializing in creativity and entrepreneurship at Stanford and the author of Idea Flow, the only business metric that matters. Kian is founder of Geolab and former executive director at Singularity University. He's the bestselling author of Competing in the New World of Work. Welcome, Jeremy and Kian. Thanks for being here. Thanks for having me back. Such a joy to be here with you. All right. Shall we get going?

3:07Let's do it.

Study on AI and problem solving

3:08To get us started, Kian, I'm curious, what motivated you and Jeremy to look at the impact of AI on creativity and problem solving? So in my business, we coach teams to achieve complex business goals through team transformation practices. Sometimes this involves tackling a difficult innovation project. Sometimes it's struggling with change management, but it always boils down to this one issue, which is how can we solve this problem? And that usually entails human ideation and prioritization.

3:39And I wanted to see if we can bring a new technology tool into the mix to get better ideas. And given that I've been teaching AI in Silicon Valley to executives for a long time, that was always my hunch. But it wasn't until ChatGPT became publicly available that I had the aha. And I didn't quite know how it could be useful to the issue of problem solving on teams, but I had a hunch. And so we designed the study together to learn from real world practice of how teams might

4:09use AI to get better ideas to solving problems. And what we did was we recruited hundreds of participants from many companies in Europe and the US, and we asked them to identify a particular pain point within their organization or a problem that they needed to solve for, something that was real. And then we actually gave half of the participants in each of these problem solving ideation workshops had access to ChatGPT and the other half didn't have access to it. So they were just thinking on their own as humans.

4:39And then the other half had humans plus AI. And we went through a problem solving ideation exercise. And at the end of it, we asked the problem owners or the executives who shared with us the particular pain point to grade all the various ideas from A, being spectacular, all the way to D, not worth pursuing further. And so we ran this blind study to understand how generative AI can facilitate problem solving, ideation, and collaboration with real world examples.

5:12The other thing that we did was we asked participants before and after the session, how they felt about collaboration and problem solving and ideation, because we wanted to see what was the impact of just a standard brainstorming session on someone's attitude towards problem solving and collaboration and innovation? And what was the impact of a session that included generative AI? Did it have a differential impact on participant sentiment? And the research to me is fascinating. Jeremy, can you summarize the results of the research you and Kian did

5:44and relate it to the quality of idea generation and the feelings people have about the ideas that were created? Very broadly speaking, we found that teams, if they want to outperform using AI, they need to adhere to certain practices. And sadly, most teams do not. So despite the potential to dramatically outperform, most teams leave the vast majority of their innovation potential on the table when they use AI. When you approach AI like an Oracle or like a search engine, just looking for the answer,

6:18it may feel like magic, but the data shows you end up underperforming. If you want to get world-class results and outperform, you can't approach it like an Oracle. You need to instead approach AI like a conversation partner. And that doesn't feel nearly as magical. It feels more like work. And for that reason, the teams that outperform actually feel worse than the teams that underperform. It's very counterintuitive. I find that absolutely fascinating. And I also find that, interestingly, the ability to have a good conversation with AI is what

6:51makes the difference for creating valuable ideas and solving problems. Kian, can you go into a little more depth with the counterintuitive findings that Jeremy just shared? Why do you think it is that AI-assisted teams that delivered worse solutions actually felt better about their work and the AI-assisted teams who delivered better outcomes actually felt worse related to their non-AI-assisted counterparts? Because rewiring our brains to act differently takes practice. And we professionals have decades of experience or certain ways of doing things.

7:25And we're asking them to work differently now with a different workflow. And that's hard. It's like going to the gym after not having exercised for a long time. It really sucks. And, you know, particularly enjoy it at first, but it has long-term benefits. And so we found that teams that were able to use AI effectively, they actually had to rewire their workflow and how they went about to try to incorporate AI as a co-pilot on the team. And that's different. And it was exhausting and they felt like it was just a lot of work, but ultimately they had

8:00better responses. Would you add something to that, Jeremy? Part of the challenge is I almost picture a cartoon strip with a thought bubble that's like, but I was promised superpowers. The premise and the promise of AI, I think it's intimidating and it maybe strikes fear into a lot of people's hearts. That's one thing. I don't think there's a reason to be afraid. I think there's a lot of reason for enthusiasm and fun and seeking fluency, obviously. But I think there's a premise of this is going to be like magic. But when you start there and you think, all I have to do is ask a question and then I get

8:30answers. Well, guess what? You do. You ask a question and you get three pages of call it B plus documentation. And teams go, it is kind of magical. Well, let's go get coffee. Like we're kind of done, right? If you're expecting magic, you will get it, but you won't do that well. If you've gotten practice being a conversation partner to an AI co-pilot, what you realize is the first step that you get out isn't great unless you're really thoughtful about framing and providing context and digging deeper and pushing back and asking questions and treating

9:05it much more like a conversation. And then the more you treat it like a conversation, the less magical it feels and the more it feels like AI is actually getting you to work and pulling your best thinking out of you. And so I think part of it is just the mindset people have when they hear, oh, we get chat GPT. The people who go, oh, magic are almost always the underperformers. The people who roll out their sleeves and go, oh, now we have work to do. It's maybe a different kind of work to do, but I'm excited to do it. Those are the people who outperform. I would also add that there is a reason why chat GPT gives you generic answers and it's

9:42designed mathematically to give you the most likely answer that it thinks you want to hear. And so as a result, the first few suggestions it offers are supposed to be generic and average. And so unless you have this conversational back and forth with it, like you would with a colleague or a friend to push it and to give it context, you're just going to get average answers and average ideas. And that's really not good enough if you're trying to solve complex problems. It comes down to mindset shift and work. There is no magic. It's not for free. If you really want to solve problems well and get creative, you actually have to have a

10:15conversation.

The Fix It methodology

10:17So Jeremy, can you share your Fix-It methodology and describe the different components and how they work? Absolutely. So Fix-It is a five-step methodology to turbocharge your collaboration. F stands for you have a focused problem, right? So you don't want to boil the ocean, but you want to be very focused in the kind of challenge that you are bringing to chat GPT. So in this example, instead of saying what recommendations would you make for a human interfacing? I would say I'm joining a podcast and I'd like to give concrete, tangible suggestions to

10:52professionals who haven't had much experience with chat GPT. If you had three or four, or maybe I'd start with 10 suggestions for someone who's unfamiliar but wants to be more familiar, start by asking me three questions to better understand the kind of professional I'm describing, right? That's what we mean by F, kind of focused. So that's a very focused prompt, right? I is ideate individually. So before coming to the team, because remember our study was conducted in the context of teams trying to solve problems. Before coming to your team, and even before coming to chat GPT, think for yourself,

11:25about what do you know and what's your point of view. Again, this kind of hones the context that you bring to the conversation, right? It's critical. Research has shown that the best way to brainstorm is to alternate kind of individual ideation and group ideation. Well, the same is true actually in the context of collaborating with generative AI. You want to alternate between individual personal thinking and thinking assisted and amplified by AI, right? So that's the I. X is context. So providing sufficient background context.

11:57A lot of times we recommend that folks upload documents, right? When we were conducting our study, we would have a problem owner actually do a dossier that we would upload to chat GPT to provide context. If you don't know what your context is, here's an amazing hack. Ask chat GPT to ask you for the context. So as an example, I've got a friend who's negotiating a lease trying to build a gym and he's trying to come to an agreement with the property owner and there's a gap between where they need to be. And I said, hey, why don't you ask chat GPT for help?

12:28He said, well, how would I do that? I said, well, have chat GPT interview you about your objectives and then interview you about the counterparty's objectives and then make some recommendations. Well, that interview effectively becomes a means by which you can provide context. So the X is make sure that you're giving chat GPT context on the problem. The second I is interactive, iterative conversation, right? So you're never just taking the first response that chat GPT gives you. You want to be having a back and forth and a dialogue.

12:59So for example, if Matt and Kiana and I are going to try to title this episode, we might say to chat GPT, hey, we'd love 10 titles for an episode. And then chat GPT comes back with 10. Almost everyone's default is to think, which of these 10 do I like best? What Reid recommend is just immediately say, I'd like another 10. And then read all 20 and say, here are the ones that I like. Would you give me 10 more like these? Chances are you're not going to get as good. And then if you think, wait, why did I like number two? Oh, there was a funny alliteration. And number seven had a funny pun.

13:31And number nine made reference to, okay, use those as design principles for the next 10, right? That's an iterative back and forth that yields, we'd probably find that the fourth tranche of 10 title suggestions would be radically exponentially better than the first 10, right? But it's by going back and forth. F-I-X-I-T. T is for team incubation. So it's important to then bring the ideas that you've generated individually and with chat GPT or whatever LLM you're using back to your team.

14:05And then importantly, commission some experiments. So this is where this work dovetails with traditional innovation methodology, but you never just want to select one idea and move forward. It's impossible for an LLM or for a human to a priori know which solution is going to be the best fit. So as a team, you want to have a practice and a process around incubating or low resolution prototyping a handful of the high potential solutions that you've generated in order to determine which one actually solves the problem the best.

14:35This methodology, upon hearing it, makes a lot of intuitive sense, but I can definitely see the effort involved. I like that it involves individual work and collaborative work, not just with the chat GPT, LLM, but also with others on the team. Kian, not surprisingly, I'd like to dive deeper into the fourth step that Jeremy introduced us to, interactive conversations. We've spent a lot of time on this podcast talking about how to have better conversations, but

15:06of course, focusing on humans. What specific advice can you provide to us for improving our conversations and our communication with LLMs to make sure we maximize the potential goodness that can come from collaborating with a tool like ChatGPT? Yeah. So the first thing I'd say is to make sure you've downloaded a LLM app on your phone and interact with an LLM on the phone instead of doing it on the web browser.

15:37If you don't see an app on your phone, it's very unlikely that you'll use it on a consistent basis. And the more you use it, the more familiar you become with it, the more likely it'll actually be part of your workflow, whether it's individually or whether it's as a team. So that's really the first step. Download one of these apps on your phone. And part of that is because we have historically been for the last 20 plus years in the internet era and the browser era. We see a text box and we know exactly what to do with it. We type in a particular word or particular question.

16:09And now that we have large language models, the user interfaces look pretty much exactly the same, like the Google search engine box. And it's oftentimes difficult to figure out the right kind of question to ask it or to word it in the right way to get the right answer to suggestions. And so we actually think it's a lot better if you start interacting with these large language models through conversation, through spoken audio, rather than just trying to type it into a search engine box and trying to figure out the exact right prompt. The second thing is that you should be uploading your prompts with audio messages, voice messages,

16:44and talk to it just like you would talk to a friend. So whether you're talking to a friend about a particular problem or talking to a colleague about a particular problem, just record it literally on the phone, on audio to text. And then the large language model will transcribe that and then offer out some suggestions. And then the third thing I'd say is think about using different models. There are several different kinds of large language models that you can use from ChatGPT to Claude to Bing and to others. And they all have their different flavors and different kinds of personalities.

17:16And you might want to use one of those models for a particular exercise or activity. And then maybe you should go talk to another model, just like you would talk to two or three different friends and get their opinions or two or three different colleagues and get their opinions because they're going to give you different kinds of nuance and different kinds of answers. I find that really helpful advice. I mean, we approach the communication with technology based on the interface that we have with it. And if we move to our phones, which we're used to communicating with very differently than we are, let's say, a browser in a text box, it can really change it.

17:49And it strikes me that it's not just the actual interaction interface, but it's also the curiosity we bring to it. When I go to enter information into a search engine, I just want to get the answer. I don't necessarily see it as a conversation and a dialogue. So that curiosity that I bring with the follow-up questions or the doubting and the exploration and expansion, I think is really important. And I like the advice to check with different LLMs, just like you would check with different friends. It gives you different ideas and inputs.

18:20You can even feed different LLMs answers into one another, right? Oh my goodness. We can facilitate and broker a conversation amongst LLMs. Absolutely. I mean, I plug in ChadGPT's answer into Claude and say, what do you think of this? All the time. And vice versa. I'd say Claude's answer and plug it into ChadGPT. Ask ChadGPT what it thinks of it. So this is why Googling is such a bad metaphor for what we're talking about here, because you'd never query Google about a Google query to get meta for a second, right? But after a sales call, I hop on my ChadGPT voice app while I'm stretching for a run.

18:54Instead of sitting at the screen, I'm stretching for a run. I say, hey, I just talked to Matt and Kian about our research and a couple of follow-ups. Would you craft a quick memo that I could send to the team just thanking them for the time today and how much I enjoyed the conversation, right? Well, it's going to do it. Well, then my next thing is I just look at it and I say, if I were to upload this memo to you and I were to ask you for advice on how to make sure that they read it and respond, what three changes would you recommend? And then immediately, I had this happen the other day. I was doing a voice vomit on a post-sales call and I had ChadGPT write a memo.

19:29And then I asked ChadGPT, how would you criticize this memo? And ChadGPT said, it's far too long for today's busy professionals. No one's going to read this memo. And I said, would you please go ahead and shorten it to a point where you think that people actually read it? You can ask ChadGPT to evaluate its own work and it will do so dispassionately, right? And that's the fun, but that requires iteration and a back and forth. In many ways, the advice that you all are giving are advice that we give to people when they are engaged in empathetic dialogue and human-to-human interaction.

20:01It's about curiosity, questioning, about challenging in certain ways. And it changes the metaphor completely, as you mentioned, Jeremy. We'll be right back to finish our conversation. But first, a quick word from one of our sponsors. Their support allows us to bring you this show free of charge. This podcast is sponsored by Squarespace. If you have expertise to share, the right online presence can help turn what you know into real opportunities to teach, connect, find clients, or grow something of your own.

20:33That's what I like about Squarespace. You can create a professional site that actually reflects who you are, starting with one of their professionally designed templates or using Blueprint AI to help you shape the look and feel. And if your expertise is part of your business, Squarespace gives you tools to showcase and sell services like consultations, classes, or events, with scheduling and payments all in one place. You can also use Squarespace email campaigns to stay connected with the people you reach and keep the relationship going beyond your website.

21:05Go to squarespace.com slash thinkfast for a free trial. And when you're ready to launch, go to squarespace.com slash thinkfast to save 10% off your first purchase of a website or domain.

Getting started with AI

21:20So before we end, I like to ask my guests a series of questions. Two are similar to everybody and one that's very unique. So Jeremy, for those who haven't started with ChatGPT and large language models, where should they start? I think for a lot of people, if they say, where should I start? But we've talked to so many professionals who say, I've been meaning to try ChatGPT. I've been meaning to try generative AI. And that to us is a shame. Every single listener to this podcast could have at least 10 hours of ChatGPT under their

21:54belt. And if you find yourself going, oh man, I'm behind. Well, don't worry. You can get off speed quickly. Here's a simple place to start. This is something that every single listener can do right now. Think of an emotional human decision you're trying to make in your life. Something that you would ordinarily ask a partner or a friend or a spouse or a colleague about. Okay. Think about what that is. Go to ChatGPT and say, hey, I'm trying to make this decision.

22:24Will you please ask me three or four questions before giving me your recommendation? If every single listener will do that one simple activity, they're going to have what we call a personal epiphany that's going to have cascading impact. All of a sudden, they're going to start thinking, could ChatGPT do this? Could I ask ChatGPT about that? Right? But it must be emotional. It must be deeply personal. It must be the kind of thing that you would ordinarily ask another human being about.

22:57Get ChatGPT to ask you three or four questions about it before giving you advice and start the snowball rolling that way. Question number two and three. Kian, I'm going to ask you because Jeremy has previously answered these on an earlier episode. And I encourage everybody to listen to that episode where Jeremy and I talk about his book Idea Flow. So Kian, who's a communicator that you admire and why? And you can't say ChatGPT. I'm going to give you two answers. One is Peggy Newton, who is a columnist for the Wall Street Journal.

23:27And I so deeply admire her elegant prose and her personal anecdotes that make complex topics readily understandable. She was a speechwriter for George Bush Sr. And she developed the phrases that we became very familiar with during his presidency, like kinder, gentler nation, or read my lips, no new taxes. And the politics and the policy aside, the ability to translate big ideas into a few short

24:02words is just so masterful that I always enjoy reading her articles in the Wall Street Journal, regardless of the politics or the policy. And the second person I'd recommend is Sam Horn. She is an author of a book called Kung Fu, and she is a complete master at teaching which words to lose and which words to use to convey meaning without tripping over yourself and creating unintended arguments.

24:32And these two are role models of how I think about communication. Thank you. Both of those recommendations, I hear you talking about words and the ideas that those words bring about. Both of those are individuals who have mastered that craft. Because words are really conversations that are the seeds of innovation and how we think about solving big problems. And if we don't get the words, then we won't be able to actually get to the meaning and connection of what we're trying to solve for. Final question for you, Kian. What are the first three ingredients that go into a successful communication recipe?

25:06The first ingredient is something that might seem counterfactual when we're talking about a communication recipe. And that is listening. Listening to the environment, listening to the context, and listening to what your counterparty might be thinking. And so trying to understand the context by listening is super important. The second one is to know your audience. Who are you speaking to? Who's on the other side of the table? Who's in the room? What are their priorities? And what are their goals?

25:36And so that you can think about crafting your message in a way that resonates with them. And the third thing is to know your end goal. How do you want the audience of your intended communication to feel after the communication is over? Whether that's in person, whether that's in a meeting, or whether that's online. How do you want the audience receiving the communication to feel? And then you work backwards from that to structure the flow and the necessary ingredients to make

26:08your communication easily, relatable, understandable, and to land. We have certainly heard the notion of knowing your audience and being critical. I really appreciate and like this idea of backward mapping. Start from the outcome, and then we have to build and do and say and emote to get to that outcome. Thank you for that. And thank you both, Kian and Jeremy, for the conversation. I find your work fascinating and super helpful as generative AI becomes more and more commonplace.

26:43Further, I love how the concepts of effective communication and conversation and critical thinking can help our partnering with AI to be better, more creative problem solvers. To learn more about the results from Kian and Jeremy's work, go to howtofixit.ai and check out their article in HBR. Thank you so much. Thanks for having us.

Closing and production credits

27:08Thank you for joining us for another episode of Think Fast, Talk Smart, the podcast. It was great learning from Jeremy and Kian. Among the many things they shared, the one that stands out most to me is that AI is most powerful when we treat it as a collaborative thinking partner rather than simply a tool for finding answers. To learn more, please be sure to dive into our library of past episodes wherever you get your podcasts. Our back catalog is full of helpful communication tips and tools to help you be a better communicator.

27:42This episode was produced by Katherine Reed, Ryan Campos, and me, Matt Abrahams. Our music is from Floyd Wonder, with special thanks to Podium Podcast Company. Please find us on YouTube and wherever you get your podcasts. Be sure to subscribe and rate us. Also, follow us on LinkedIn, TikTok, and Instagram. Before we wrap up, I just want to say thank you for listening. It really means a lot to hear how people all over the world are using these ideas in their own lives. It inspires me and the whole team that brings you this show.

28:14If you want more episodes and resources, feel free to follow, subscribe, and explore past conversations. We're grateful for your support of Think Fast, Talk Smart. You're welcome.