#236: AI Answers - No Time for AI, AI Budgets, Vendor Terms & Data Risk, AI Disclosure & Vanishing Entry Level Roles

Introduction to AI Answers

0:00Where does management come from? What do managers do in the future if we don't train entry-level employees? So if we stop hiring as many entry-level staff, then we decimate the future managers and leaders of the company. Welcome to AI Answers, a special Q&A series from the Artificial Intelligence Show. I'm Paul Reitzer, founder and CEO of SmarterX and Marketing AI Institute. Every time we host our live virtual events and online classes, we get dozens of great questions from business leaders and practitioners who are navigating this fast-moving world of AI.

0:34But we never have enough time to get to all of them. So we created the AI Answers series to address more of these questions and share real-time insights into the topics and challenges professionals like you are facing. Whether you're just starting your AI journey or already putting it to work in your organization, these are the practical insights, use cases, and strategies you need to grow smarter. Let's explore AI together.

1:03Welcome to episode 236 of the Artificial Intelligence Show. I'm your host, Paul Reitzer, along with my co-host, Kathy McPhillips. Welcome, Kathy. Thank you. Kathy is our Chief Marketing Officer at SmarterX. If you are a regular listener to the podcast, you have heard Kathy many times before. So Kathy is not taking Mike's job. Mike and I are still doing the weekly every Tuesday, and Mike is the host of the AI Transformation series that we've started. We've done three of those now.

1:34And then Kathy and I do these special editions of what we call the AI Answers episodes. So these episodes are based on questions that we receive during our monthly Intro to AI and Scaling AI classes, as well as some questions that we get from virtual events. So these are all community-driven questions, and we just kind of go through. There's usually, I don't know, Kathy, what do we have, 15, 12 to 15 of these things? Yes, we do. Kathy prepares a brief that I look at as we start recording. It's a wonderful brief, but I just kind of dive in and we go.

2:05So, yeah, so today's 15 questions. They are based on the last two Intro to AI and Scaling AI classes that we did. The Intro to AI was July 29th, and Scaling AI was July 9th. So these are both from late July.

2:24And, yeah, so the interesting thing, and we'll kind of get into this as we go through these questions, is the Intro class I have been teaching since fall of 2021. So we've had north of 60,000 people register for that series. And back in 2021, pre-ChatGPT, you can imagine we were getting some very foundational questions, just very basic, trying to understand what the technology was and where it was going and how to figure out use cases. And the Intro questions have evolved.

2:57They have become much more advanced. So even the people that are attending the Intro class generally have really, really good questions. And then the Scaling AI, we have now done for probably going on two years, Kathy, I would say. We've been teaching that one. Yeah, just about. Yeah, we've done 18 of those. And so that's probably had 10,000 to 12,000 people, I would guess, go through that series. And that one teaches our five-step framework for scaling AI in an organization. So those questions tend to be more organizationally driven, team department driven.

3:28They're thinking bigger picture about how to move beyond just pilot projects and actually scale it in the organization. So if you've never had a chance to attend either of those, we do continue to teach them. You can look in the show notes and there's links to sign up for the next ones. I think we have one coming up at the end of September, maybe. Does that sound right, Kathy? We have September 10th is our next Scaling AI. And September 30th is our next Intro to AI. Okay. So apparently I'm teaching a class next week. All right. So September, you can join us for both of these if you'd like.

3:59All right. Actually, no, it's on Friday. Actually, it's on the 11th because you're not going to be in town on the 10th. So we have to shift that date. Yes. Okay. All right. So, okay.

Marketing AI Month overview

4:09So today's episode is brought to us by Marketing AI Month. This is a brand new thing. And I am going to let Kathy kind of tee this up for us. And then we will get into the Q&A. Also, do you think you ever need to say that people are worried I'm going to take Mike's job? I do because I think that these episodes with you, Kathy, are so popular. And people might dip in and just want to hear you and I talk. And they don't realize there's this broader show. So I don't know. I think I never know when someone enters into the artificial intelligence show world.

4:40And it might be their first experience with it is. Fair. And AI answers on like a YouTube clip. I don't know. So, yeah. I always try and set it up that way. I'm not taking Mike's job. I don't want Mike's job. We are all good. I don't want Mike's job either. So thank you for being here. This episode is brought to you, brought to us by Marketing AI Month. We launched that Tuesday of this week.

5:06So this month we are doing something really fun. We've got this amazing AI for Marketing course series as part of AI Academy. All month long. You can get it free. It's valued at $499. So you'll get the series. It's five expert-led sessions by Mike, practical tools, frameworks that we actually use as well, and a professional certificate when you are done. So that's free through September 30th. You do not need to complete it. By September 30th, you just need to enroll by that date. And to close out the month, which we're actually doing on October 1st, we are holding an AMA where,

5:37similar to this format, I will be peppering Mike and Paul with all of your questions. Anyone that has registered for that or enrolled in the course is eligible to attend. So if you only want the AMA, enroll anyways, but the course itself is amazing and I highly recommend it. I took it myself over the holidays and kind of kicked off my year really strongly. So I was really excited about that. So if you visit smarterx.ai slash marketing, you can enroll right on that page. Anything to add on that, Paul? No, I take advantage of it.

6:08It's our most popular course series. AI Academy has, I think, 22 professional certificate course series right now. So the idea here was like, let's just accelerate marketing, understanding and adoption because our experience with a lot of our AI Academy business account customers is that marketing is often the tip of the spear. It's the department that's leading on AI adoption, but then also an essential part of communication and strategy to bring other departments in and drive AI adoption and transformation through

6:42other departments being communicated through the marketing team. So marketing to us is a compounding value for the industry. The more marketers we can get to understand this stuff and be adept at using it in a responsible way, the faster we can spread that throughout organizations. So if you're a listener and you're not a marketer, make sure to send that link to your marketing peers within your organization because it, again, is open to everyone. And it's a great value for everybody. For sure.

Practical ways to use AI in marketing

7:12Which takes us to our first question. So let's jump right in. Number one, what are the best ways to use AI in marketing? I might turn this one back to you and say, how are you using it, Kathy, as our chief marketing officer? I'll keep it simple. And then if you have something to add, Kathy, I'd love to hear your perspective on this one. I just start with what do you do? Like I would look at the tasks and the workflows that you are expected to perform every day. I would look at the goals that you're expected to achieve. And then I would go in and just ask your favorite AI assistant.

7:42You could use our jobs GPT tool if you want, but you could just give this prompt to any of your favorite assistants and say, I am a director of marketing. My job is this. Here's the KPIs I'm responsible for. Help me figure out like the top three to five ways that I could be using AI today. These are the tools I have access to. Here's the data I can connect to. Just give it the information and let it help you figure that out. But the answer is it's personal to you and your role in the company is how I kind of simplify that. Agreed. I think I would add, yeah, I'm using it a lot for strategy.

8:15We are doing, we are rolling things out on a regular basis. So me getting a head start on getting all of the strategy out of my head onto paper, it's repurposing strategies that have worked in the past. It's building this knowledge base of all the things that we're doing. It infrequently helps me write because I do enjoy writing, but it does give me a gut check on some things. I'm using some of our paid advertising to help me figure out new ways to message, looking at data, looking at trends that I might not be able to surface with my non-analyst eyes.

8:49So one thing we've done on the marketing team is we have these documented processes for virtually everything that we're doing. So we're trying to, on a quarterly basis or whatever cadence we decide, running it through saying, can we do something more? Can we do something more? So having the processes, having the use cases documented, and just because things are changing so rapidly, updating that regularly has been really helpful for our team. Yeah, one example, and we don't have to get into great detail on this, but the marketing AI month that we just talked about, we decided last Wednesday that we were doing this.

9:21It was a very spur of the moment, hey, let's just give AI for marketing away. Let's just try and accelerate awareness and understanding in the marketing industry leading up to our MECON event, to our big conference in October, which is for marketers and business leaders. And so Kathy, I don't even know how she did it all, but we launched that campaign three business days later. It was an entire thing. And I'm sure there was ways Kathy leveraged AI to help in the planning and the production of assets, things like that.

9:51But to launch a major campaign in a matter of three business days is wild. Like, if you think, what could we have done historically? Now, I'm sure Kathy worked really long hours and it wasn't like she just hit a button and boom, here's your campaign. But even that, there's just no way you do that. We have a small team. Like, we don't have 20 marketers where Kathy is just like, all right, everybody go do your things. It's like, no, we get our hands dirty. We're actually in this, doing this stuff. So that's an example of like a very real, relevant thing that I'm sure we couldn't have

10:24pulled off without the support of AI. Right. And people. I mean, Jeremy Zimmer, our director of marketing for Academy, he was obviously integral in all of this. Macy are in social media. Everyone. And then I built this brief and like, here's all of the things we could be doing with my knowledge, with AI's knowledge. And I told the team, I'm like, I know I'm missing something. So please, as humans, go in and help me figure this out. What are all the pieces, parts that I'm missing? But yes, AI definitely gave me a very good roadmap to get it all done.

Carving out time for AI learning

10:54Okay.

Carving out time for AI learning

10:55Number two, I have a team that is mostly made up of AI beginners. They are curious and optimistic, but also time starved. The most common pushback I hear is, I believe that if I invest time in AI, it will eventually generate efficiencies in my day-to-day work. But how do I carve out the time for now? Sales targets and client deadlines won't pause while I do this. Are there any good tips to respond? I would do a forcing function of a workshop. It could be a one-hour workshop. It could be two hours. It could be half day, however you want to do it. But I would do an AI workshop where you teach them a framework.

11:27So we have a use case model we use. It's free. You can go, I mean, it's in our book. We teach it online. Like there's plenty of ways to go learn it and you can apply that framework to your own company. But basically, let's say it's, let's just pick 90 minutes. We are all going to leave here in 90 minutes with one to two AI use cases that we can implement starting tomorrow morning. That's going to save us at least three hours a week. Like just set some parameters. Because what it does is it forces you to look through your tasks and workflows that you do

11:57over any given week and say, which are the ones that take more than three hours? Okay. Now you've narrowed that down to like, let's say there's five things on that list. Um, which ones have a repetitive process that we could automate elements of, which ones require creation of strategy documents or analysis of data? Like you go look for the use cases that are a good fit and then you just pick one and you build a project, you build an agent, like you just build something. So make the first like, you know, 20 minutes setting the stage, make the next 25, 30 minutes

12:28brainstorming where people are prioritizing lists. Make the last 30 minutes or so sharing what you've done or actually building the thing. Um, so maybe extend it to two hours and now you can actually have a build session with it. So I would, you have to force it. Like you have to have something that says we aren't leaving here today until everyone has built a thing that's going to save them time. And then if that works, do it every other week or do it once a month. And like, then you just stack those things over time. And all of a sudden everybody's got five agents or apps that are, you know, saving them 10,

12:5915 hours a week. Yeah. And I think showing them how, you know, I think the jam sessions we're doing where people on our team are opening up their computer, sharing it on the TV. We're all in the same room. We're online. We're watching them click the buttons and doing the things because that is daunting. People are like, okay, I think I can do that. I probably have the brainpower and the capacity to do it. I just don't even know how, and I don't have time to really figure it out. So having someone show you how to do it is I think so valuable. Yeah. And I'll, I don't know, again, I haven't really looked at these questions, so I don't know if I'm going to get into this and I'm not even actually a hundred percent sure I want to

13:31share this yet. But like, I started a project this morning where I'm basically going through and trying to look big picture at where agents can be infused into my role. And as I was doing it, I was like documenting, journaling it like I normally would. Like, okay, I'm going in here, I'm creating this, I'm going to test this agent. And so I would go through and like this whole dialogue with myself. But then I realized, wait a second, maybe I'll just turn on screen recording and I'll just like record this in Zoom or Google or wherever, and then eventually take that and

14:03turn it into like demos for the team. But then I'm like documenting all the steps I'm going through, all the visualizations I'm seeing as well. So all of that is helpful, but I think you're right, Kathy. Like just, you need to infuse whether they're workshops, build sessions, jam sessions, whatever you want to call them, those have to just become part of the routine. And then it gives people the time and the permission to do these kinds of things. Right. The build sessions at Macon this year, not to plug Macon, but I will. I'm really excited about that because you bring your computer, you sit down and 45 minutes

14:35you leave with something tangible. And it's just so helpful, even if it needs to be, you know, sussed out after you get home, but just being there and watching someone else do it is pretty cool.

Using external AI consultants

14:45Okay. Number three, we are a small company. How should we think about using external consultants to help? The team is so busy doing day-to-day work that we are not making time to create agents and take us to the next level. We have lots of ideas, but need the time, skills. What thoughts do you have on that? Yeah. I mean, there's a decent chance you can find consultants who come in and help you analyze your workflows, look for ways to drive innovation, look for ways to drive, you know, automate things, build some agents.

15:18I think that that's a short-term fix, honestly. I just feel like this is going to be so critical to every organization's operational structure and the way they manage the company that you have to own it. Like, so I explained this a little bit on the podcast, but we're building Smarter X Labs within our organization. We're a small company. We have 19 employees. Um, I think of Labs as like the R&D unit and their job is going to be to go to each team

15:49or department or business unit and work with them to say, where are the barriers right now to achieving your goals? What are the workflows that, you know, take the longest or what are the major things that we should be doing as a company that we just don't have someone in that seat right now? Like we can't do it. Um, and so they're going to basically function as, um, forward deployed engineers. So that's the concept we talk in the podcast where you basically have people who go in, look for things to build problems to solve, and then they build them for them.

16:20But what I, I believe the reason we have to internalize this is because we have to maintain that, that IP. Like we have to maintain that know-how within the organization, because if we go in and we solve something with Kathy and her team on the marketing side, it may translate over to the sales side or the ops side. Like an agent we build here might just need a couple of tweaks and all of a sudden it's a sales agent too. And so because AI being infused into all these areas is so critical to what we're trying to build, I just see it as like absolutely fundamental skill we have to build internally.

16:55So if you're going to rely on consultants in the early going, I think that's fine. But I would, I would be training or developing people who can function in that role moving forward. And it could be a, you know, a marketer who's just, you know, also pretty technical, like very comfortable in the data and the systems that's trained to become like an AI ops person. And it doesn't, you don't have to go hire software development people. You don't have to go hire AI engineers. They can be business people, you know, knowledge workers with domain expertise who are just the

17:26most advanced users of the AI platforms you use. So that's how I would think about it. And you know, you think about subject matter expertise, you know, being, like you said, domain expertise, institutional knowledge, like a consultant can help you with most of that stuff. So, or do you, so do you start building notebooks and knowledge bases now, if you're going to start using someone on the outside? I mean, if you're, if you're going to bring in a consultant, they're just going to come in initially in like document processes, audit workflows, they're going to look at what the AI is going, and again, all could really be valuable.

17:57Like if you truly don't have the time to do this and like the alternative is no one does it, then yeah, like hire somebody to at least get this moving. But I don't know. I just, it's one of those things I think is so critical to every company that I would be very aggressively looking at what is that, like defining what that role is and seeing do we have someone on our team who could, this is a career opportunity for them. Like they would work really well. We've seen this in companies, Kathy, you and I both know people who are in AI ops roles today, who, you know, 12 months ago, 18 months ago were marketers or they were

18:30project managers or like whatever. So this to me is that whole creating, seeing an opportunity for a career path for yourself, defining it, and then going and building it. And when we start this internally, I would like to be tribute and I would like you to tear apart my department and tell me all the things that I will swallow my pride. And I can't wait to see what comes out of it. I think it is just like a totally collaborative thing. I think it's more of like, you have a wishlist, like you have roles you'd love to fill. And I think this division is like, okay, Kathy, what are the five roles you would put on your

19:00team today if you had the approval to hire them? And they might look and say, I think we can get you like 50% of the way there on three of these in like two weeks. It's not like, you're not gonna be your FTE, but like we could probably build something that'll at least give you that functionality. That's how I'm thinking about it. And then you kind of like go down from there. I'm ready. Okay.

Leadership for AI transformation

19:19Number four, who should be the point person for AI transformation in the enterprise? In some organizations, it's the CFO with project manager who understands the systems and knows where the data sets are hidden. Has this been your experience? I don't know that there's a right answer for this one yet. I do think it depends so much on the skillset of this person. So I've seen one where it was like a chief legal officer who became like the head of transformation because that person was just also extremely qualified on the AI side and

19:52understood the business deeply. I've seen it being driven. We had our first AI transformation spotlight that Mike did with Good Karma Sports. That is a CEO driven transformation. I mean, Ty, who did the interview, was telling he's a key, key driver of the whole thing. But that is being very much led from the top by Craig at that organization. And so this is a CEO who got into it, said, this is the future of the company. I'm all in. And he is like right there every week in every meeting about AI strategy.

20:23So in that case, I don't know that they have a chief AI officer, but it's probably Craig. Like he's the driver. I've definitely seen where the CIO does it. If that CEO has deep business knowledge, like if they're not just thinking about it from a technical perspective, we've definitely seen instances where like Dan Slagan, who's speaking at Macon this year, marketer, CMO. And Dan just sort of emerged into being an AI transformation leader at a major company because he just became the most adept at the technology.

20:53So I don't think that there is a right answer. I really do think it is, where is your organization at? Where is the CEO in the organization at? Because I am a believer that if the CEO is extremely AI forward, that is probably the person who should be at the top of the chain when it comes to spearheading the transformation. Obviously, he doesn't have the ability to be in every detail, but the CEO should be the driver. Um, that's probably going to be the most successful organizations is when it is a top three priority

21:25for the CEO and they are actively involved in everything. Yeah. It just opens up that permission and enablement when the CEO is in that role. Yeah. Like I actually, I, this is totally behind the curtain stuff. Um, I actually messaged our team today because I was working on something, uh, pretty significant for our organization. And so I just messaged, I was like, Hey, can you guys send me the excerpt from our AI policy? Regarding connectors because, uh, Tracy and Elizabeth, they're going to laugh when they hear this now. Um, because I'm, I may be instituting an R and D exemption.

22:00Uh, that's a term I just made up. I literally put in parentheses to them, um, and like approving things myself. Um, so I'm not bypassing our approval channels in our AI policy, but I am. I mean, it kind of seems like you are. I'm accelerating them. And so within like 10 minutes I had, I was copied on emails to our IT team and our lawyers saying that, you know, basically our CEO is planning to accelerate approvals. We'll call it our AI policy. Now let's think about companies.

22:31Think about companies. We know where the CEO has gone rogue, Paul. Let's not. Yes. Oh no, Tracy did her job. She was like, here's the 15 things that I'm basically like want to check before I like go back to Paul, but like, and they're all really smart things. It's like, yeah, I know. Like I, I'm not going to just like hit the buttons, but, um, yes, when the CEO is, has an urgency, like, Hey, I'm doing a presentation on this in seven days and I, I need these connections in place right now, then things happen and you do it within the safest way

23:04possible, but you have to also like an example, again, I don't want to like spend our whole time on this, but like Tracy's thing is okay, cool. You turn that on. Does everyone have access? Do we have to like communicate with everybody not to use the access you just granted? So while I want to move fast, I, I, that is why I reached out to her first and said, I'm going to do this within our guardrails. We have to set that example, but we also have to find a balance here because I have no choice. Like I, I have to do this in the next seven days. So how do we do this in, in the, the most, um, responsible way possible related to our guidelines

23:39and policies? Yeah. So you may have, you may have some access in your chat GPT account later today that doesn't look familiar. Um, okay.

Building and using AI agents

23:49Number five, are there generic agents everyone can use or for agency beneficial for us? We need to build them on our own. It depends on what platform you're in. So this is actually the initiative I'm, I'm referring to. I'm, I'm working deeply on integration of agents into smart directs. And I had this, I haven't talked about this internally. So again, I was, I, it's not even joke. Like people think I'm like making this up maybe when the podcast or I'm like, yeah, no one on our team knows literally no one on our team knows about some of the things I'm working on and they hear about it on the podcast. So a couple of weeks ago, I, I started, uh, a personal initiative, um, to deeply infuse

24:26agents by the end of this calendar year. And in essence, figure out what that looks like moving forward, all the connections, how we're going to structure them through the smart directs labs division, um, things like that. And so, um, it, I have a forcing function that I have a presentation next week related to an agentic enterprises. And then I'm teaching an AI executive workshop at Macon on October 13th, where I need to actually present a model for this. So I'm deep in the agent world at the moment.

24:56And what I'll say is like an example here would be, if you're in chat GPT, they have standard agent templates that you could go in and just start with those. Like when the pops might as like a marketing strategy agent, they've pre-built the skills for it. They've built the system instructions for it. All you do is go in and give it a knowledge base or connect it to like HubSpot or whatever it is you want to connect it to if you're allowed to. Um, so I guess the short answer is yes. Like there are generic agents if you're in co-pilot or you're, if you're in Google, um,

25:29chat GPT, Claude, they've pre-built some agents that are more universal, but most likely you're going to be building things that are specific to your role or to your organization, because for them to be really valuable, they're going to need, um, a knowledge base that is personalized to you or your company. And they're going to need connections to data sources that your administrators are going to have to approve. Got it. Okay. Number six. I love this question.

26:00Can you set up checks within an agent that require it to pause for human approval before taking specific actions? I wouldn't build an agent that you couldn't do that with. Um, so yeah, it just depends on how technical you are. Um, like where you're building these agents. But again, if you're working, I'm going to assume most people that listen to our podcast are probably not like advanced codex users, um, cursor users. Like they're probably not the, no, I may be wrong.

26:32Again, we don't really, it's hard to know your podcast audience. It's just all these people. It's quite opaque. Um, so I'm assuming that most of our listeners are more, uh, marketers, sales professionals, business leaders, um, that are not deeply technical coders, software developers. So we're, we, I'm just going to use the collective we here are likely we're Salesforce users and we're going to be trying to figure out how to use the Salesforce agents that are available to us. We're going to be building co-pilots in Microsoft. We're going to be building, um, chat GPT agents.

27:06There's instructions within there that where you're going to say, this is the checkpoint where you're going to, or you're going to give it read, read only access where it can't actually go and send something or do something. So the vast majority of us are likely only working with and building agents where humans are in the loop or in the lead. And yes, you should know when an agent would be taking an action and you should be the one that giving it permission to do that in low risks of scenarios, you may give it permission to do the actions without approval.

27:37Like example there, send Kathy a, um, a summary, uh, report every Sunday night of marketing performance data the week prior. And it goes to HubSpot, it gets the data, it pulls it, it summarizes it, it sends it to her. So it's going to take the action because it's just between me and Kathy. There's no risk of it sending to our customer base, that, that kind of thing. Okay, cool. Number seven, for organizations with many data sets containing personal and sensitive information, a complex enterprise architecture, legacy systems, shared services across very

28:12different departments and pockets of AI forward teams. How can agents be enabled for those teams without creating risk for the rest of the organization? With great care. That, that is like when you're talking about these agents that are accessing sensitive information, let's say like a healthcare system, um, you know, a law firm, a bank, wealth management company, like whatever, like there's just, we all have sensitive information. We can even say smarter acts. We have customer data, um, that we always obviously want to be extremely sensitive to. Um, so anonymizing data before you put it into these systems, things like that.

28:44Um, you, you have to have it and legal involved. Like it's just, you, you can design what you want to have the outcome be. You can build prototypes of agents, you can do all of these things, but when it's going to do things that present risk to an organization, especially if you're in a heavily regulated industry, you, you have to know in advance the approval systems you have to go through. You have to deeply understand your AI policies. And that's the example I was giving earlier with Tracy. Like Tracy just taught an entire course and we'll get to some AI policy stuff. I see it highlighted below. Um, she just taught a whole, uh, AI Academy live course on our AI policy, which is a very,

29:19um, comprehensive document. And if you're going to be building agents, you have to be familiar with that. You have to know what you're allowed to do and not allowed to do, what data you're allowed to put into it, what connections you're allowed to make. Um, and so what we're trying to do as an organization is these sort of like, I'm just going to make up a term here, like horizontal approvals. Like, okay, if we want to connect HubSpot to ChatGPT, let's approve it once so that if Kathy or Tamara or Dia or anybody wants to do something between ChatGPT and HubSpot, they

29:53don't have to come ask for the connection permission every time it's already been connected and then it's approved for read-only access for like all these different uses. So you do something once and then it, it, it permeates across the whole organization with permissions. And so I think you'd be in a situation like that. Like if you're in this kind of enterprise where you're going to be accessing personal sense of information, doing these complex architectures, try and try and solve for as many of those with a one-time permission and training process as you can.

30:23So that like, everybody's not coming and trying to get approvals are basically the same thing. Right.

Funding and budgeting AI investments

30:31Number eight, how should a company fund its AI investments over the next few years? And where should I expect the new spending to come from? Oh boy. Uh, it's gotta be dynamic. I don't know. Like, this is a really hard question to answer. Um, like I'm going to just like, I'm thinking out loud here, people is my first thing. So like when I'm thinking about smarter X, I'll just take it to make it personal. So for our company over the next three years, where am I going to put AI investment dollars? Um, my first investment is the people to help me figure out the answer to that question.

31:05Um, so the people who think 24 seven about our AI investments, that could be human capital. It could be token capital. It's like, where's our spend going to go? What platforms are we going to invest in? But first you have to have the right people to make those decisions. Um, second is going to be my, my platforms. So what are our token budgets? Where, how are we going to optimize those token budgets? Um, how do you even budget for token budgets? Yeah. Like how do you like teach people what that even means? And how do you manage them?

31:37Um, and then third would be probably more of an infrastructure thing. Like, do I need to be buying Mac minis? Do we need to be buying GPUs? Like, are there, are there business cases from a hardware perspective where we actually need to be like thinking differently about the company, um, to where we have access and do we need to be building? I guess the fourth would be, is there going to be an increasing argument that we should be training our own model at some point? You know, I think that a lot of companies are going to head in that direction where they're

32:07going to get these open weight, um, more efficient models. And they're just going to, you know, do reinforcement learning, you know, training on them to be adept at their specific industry and business case. So, yeah, I guess that's four. And again, I've never actually thought deeply about this question at all. Um, so that's, I, I, I reserve the right to change my answer at some point. So, but it is hard for, I met with the group this morning and they're on a three year cycle where they're trying to plan out the next three years.

32:37Like, how do you even do that? And as leadership, are you able to give your team permission to just as intelligently as you can make those decisions, knowing that they're going to change? Yeah, I think three years is hard, but every company's doing, every enterprise obviously thinks in three to five year roadmaps. Like they're all looking in that direction, I think you need to make decisions that are as, um, flexible as possible and as like evergreen as possible.

33:08So you're not like locking, I don't know, like one of the ways I think about this one is, you know, if we go really, really deep with ChatGPT as an example, and everything starts running through that and all the connections are for ChatGPT and then something happens to open AI or like their models or whatever. And it's down for two hours, two days, two weeks. What happens to the company? There's no redundancies, there's no backup plan. And so that, that's a, you know, betting on a single provider of intelligence

33:42isn't a super scalable thing probably. And so when you're looking at three years, a strategy would be like, we need intelligence redundancy and that, that scales, that like becomes a strategy that lives over three years and actually starts to affect the decisions you make because we've set a priority around redundancies. Um, so that thought, because I do have another question later that I don't want you to answer right now. Okay. All right. So let's end it right. Let's end that right there. That's the end. Okay. Number nine, um, kind of going into this a little bit beyond subscription costs.

34:16How are companies actually budgeting and forecasting total AI spend, including training and education time, set up and integration, internal token usage and token usage driven by external user or customers. Is anyone doing this well enough to have a real forecasting model or is everyone still finding out when the invoice arise? Are there any resources you would suggest? The best I've heard, I don't even know if best is the right term here, a model I have heard is literally just sending or setting a cap each month. Like here's our, here's our budget for the month.

34:47There are definitely more and more organizations trying to figure out the token optimization and management. There was just something from Google. I think it was on Monday. I saw this where they're starting to offer that kind of capability within their systems to help you manage it. There's startups emerging that do token optimization and basically route you to the, like the most efficient models. Um, I saw something, I think it was this week where open AI is now actually experimenting with

35:17outcome-based pricing. So part of this is we're developing systems for the way that AI is priced right now, but most of these labs and providers of the intelligence know that this is not a viable, scalable system. Like per token, utility-based pricing works sort of for developers. But once you start getting it into departments like marketing, sales, operations, HR, that's a really, really bizarre way to try and price stuff.

35:48Like if it's not seat-based, um, with unlimited usage, or it's not outcome-based, if like the outcome is very predictable and able to be attributed specifically to the model's work, then it gets, it gets kind of tricky. So, um, yeah, I don't, I don't know, like an awareness of how the tokens are being used, what the different models charge per token. Um, it's very early. I don't have a great answer for like, go look at these three companies because they're doing it perfect.

36:19I don't know anybody who's actually solved this yet, including, uh, some of the, the big tech companies that are responsible for building the models. I don't know that other than providing unlimited uses of tokens to their employees, but even Meta just changed. Like they've made changes to the models that they're using for this reason, because they, they couldn't manage the token budgets and they're a builder of their own models. Like, right. So. And if you do know somebody listener, reach out cause we will have them on the transformations.

36:49Yeah, totally. Okay.

Vendor approval and data policies

36:52The next three questions are from the Academy live. We had on Tuesday this week with Tracy, our COO, and then Samantha, our legal counsel. So these were, I thought these just fit in well with some of these discussions. So the first one is, um, what does a right-sized AI vendor approval process look like? How long should a review take? What happens to employees while they're waiting? And how do you catch it when a vendor update introduces new integrations or changes how an approved tool handles data? And I know that you are not doing, you know, that ops is doing this, but as the leader, what

37:24thoughts do you have on this? I would start early. I mean, we've, we've been in situations with some customers of ours where they got excited about instituting, you know, different AI technologies into the organization. They'd made the business case for it and then they go to procurement and they would be told there was like a multi-month wait time before procurement would even look at anything. So it starts with really just understanding your organization's procurement process and what kind of risk and security considerations need to be made as we start, you know, getting into

37:57the AI technology that might require access to certain data sets. And, um, yeah, I think it, it really is about an understanding of the process. And then from there, it's going through, um, and building that in. So if, if you know that it's going to be three months before you can get the system you want or the approval for the use case you want, or the connection you're looking for, you need a plan B about what that, um, workflow looks like without those tools. So, yeah, I, I think again, this one is very, um, subjective to the organization and what

38:31their approval processes look like and how they're defined within the AI policy. So I know in our case, we have to submit now through form requests to the operations team when you want a new vendor, if you want to make a connection to a data source. And then I think the team says like within 48 hours or something, they'll at least give a preliminary, Hey, we're going to have to look deeper into this. This is going to request kind of the example I gave with Tracy earlier. Like I put in a request, I get, we'll call it. Um, and then there's a, you know, within a few minutes, there's an email to the legal team.

39:05There's an email to the it team because for the ops team to make their decision, they need the, the expert perspective from those two bodies to make the informed decision around the vendor or the connection that's being requested. So yeah, it's going to vary by company to company, but the best thing you can do is deeply understand that process before you get started. So you don't run into that obstacle when you're ready to go. For sure. And in one instance that we had internally, the request was put in. It took longer than normal because it was brand new, brand new tech.

39:36And we were trying to figure out, well, ops was trying to figure all of it out with legal. And there were two, there were, there were some redundancies because things were being done twice, one, the old way and one trying to do it with a new tool. So there was a minute of double work, but in the long run, it's going to be so much better now that everything got approved the right way. Yep. And I will just say being on the other end of this, you know, we have our AI Academy, which is a technology platform. And so we have to go through procurement, especially for these bigger enterprises. And you have to meet some very stringent standards that are only getting stricter as we go.

40:11And so Tracy can attest, like, I mean, she spends a fair amount of her time getting through procurement systems and she could probably sit here all day telling you about all the new requirements over the last like 18 months as a result of AI. Yeah. There was one she did last week or two weeks ago that she was just like, this is the wildest form I've ever filled out. Yeah. But it's necessary. Yeah. OK, number 11, when reviewing an AI vendor's terms and data policies, what should companies look for regarding what happens to their data if the vendor is acquired, goes bankrupt or

40:43sells its assets? How can organizations manage or mitigate that risk? Or sells the data for training purposes?

40:53I'm like this one. I just honestly feel like my best answer is you have to rely on your legal team. Like everybody's legal team is going to have different appetites for risk. They're going to look at it through different lenses. It is just simply not a decision that a traditional business leader, department leader, you don't have the expertise and likely you don't have the permit in your company to make those decisions. You have to involve legal and whatever other channels are required.

41:24And hopefully, based on our previous question, that's all clearly documented in the AI policy. And if it's not, it needs to get documented so that you can follow these procedures. And then you know when it's a non-starter. So you're getting really excited about this new AI tool. You go in and pull up their terms of use. You throw it into ChatGPT and say, tell me how they handle our data. And immediately, it pops up with like three red flags for you. Another way you could do it is take your AI policy related to vendor data usage, drop that

41:57into ChatGPT thread or Gemini or wherever, and say, hey, here's our AI policy or build a project that has that AI policy baked into it. I'm going to give you a contract for a new vendor. Can you tell me how this relates to our terms of use requirements and flag for me what I should highlight for our attorneys? I do that kind of stuff all the time where I'm just like, here's this doc, here's this doc, compare them and tell me what I should ask the attorney. For sure. And then, you know, just thinking about when they change the terms and conditions or when

42:28something, I mean, there's so many, it's not like approve it once and it's done. It's like set up some normal cadence of just reviewing everything to make sure it's still. Yeah, like when Anthropic came out with Fable 5, they changed their data retention policy where they kept everything for 30 days. And a lot of people were up in arms about that. Right. Number 12, when should a company disclose externally that AI was used to create content or complete work? Is there a meaningful threshold or does it depend more on the nature of the work and its potential consequences?

43:00This gets in again to AI policies. Everybody needs to make these decisions for themselves. I feel like it's a bit of a sliding scale where more and more, it's just going to be kind of assumed and expected that AI played a role. And I think like tagging every single thing with like, AI helped here, AI did this. I don't think most people are going to care most of the time. So I think that there's some basic rules that at least I tried to live by and I hope our company tries to live by is like when authenticity matters, you should be doing the work.

43:34It should be human in the lead. Like it's okay if you're using AI to edit or verify ideas and things like that. But it should be you and you should stand behind that work. This came up on episode 234 or 35 of the podcast. We were talking about that one author who used it and got called out on it. And he's like, of course, I use it. I use it all the time. Like, what are you talking about? I generally like to default to if you're using it, it can be something as simple as like

44:04co-authored with Claude or Claude supported it in the editing and revisions of this document or Claude helped me with brainstorming, whatever. Like just disclose it because it doesn't do any harm. But I don't think we're at the point where we need to make like a huge deal of it anymore.

44:22So, yeah, again, it's you got to make these decisions in your policies and then provide that guidance to your team because everybody's wondering. It's like, I don't know if I'm supposed to tell somebody I didn't actually look at their email. Like my agent is now answering my emails for me. In that case, I would just say, yes, you should disclose it if your agent is answering your emails for you. But yeah, I think it's a very personal decision and it needs to be decided in the policies.

Training entry-level staff and future managers

44:45I was working on something last week and I can't even recall what it was, but I kind of went into it blind, not really knowing what I was going to be working, like what I was going to be doing. And I used AI a lot and I put that in the, when I distributed it to the team, I was like, this is Claude written with Kathy supporting it. Like, I honestly, I don't know. So this is my first pass I'm learning. And just so people looking at this, like this doesn't sound like her. It's so, it's so obvious when it's not, you know? But I just want people to know that where I was on that, like one example, and you could

45:21attest to this, Kathy, like sometimes we'll be working on a problem and I'll go into chat to people like, here's the exact problem. Here's the context you need. Here's what I'm trying to think through. Build a brief for me. Like, I just need to like, think about this and it'll build a brief and I will take 10 minutes. I'll read the brief. I'll give it some thought. I'm like, this is really good. I will send it to the team internally and say, this is completely unedited, but I have reviewed this and this is a really good starting point. And then I turn it over to the team. So they know I didn't do it, but they know I've signed off on like, this is a good direction. Come back to me when you all have made progress on this.

45:55So yeah, I just, I don't know. I don't think there's like rigid rules around this. I feel like it's just evolving as our use of the models changes. Okay. Number 13, entry-level staff have historically learned their trade by implementing task-based work assigned by their managers. If that work is now going to be assigned to AI, how do entry-level staff learn their trade and where do future managers come from? My theory is apprentice programs. I talked a little bit about this on the podcast this week.

46:25It was 235. I think it came up because there was that research that came out about apprenticeships were actually like the number one thing. That happens to be what I'm talking about for Macon. My opening keynote is the architect, the orchestrator, and the apprentice. It's a theory and it's just a theory on what the future of work looks like and what the different roles we're going to need within organizations are. My belief is that we can actually accelerate learning and expertise through apprenticeships with the support of guided learning from AI and personalized learning.

47:01But I'm not sure yet. I have six weeks to figure out what exactly that is. I've been thinking about this model since March of this year. I don't know. This has been my great debate. Where does management come from? What do managers do in the future if we don't train entry-level employees? So if we stop hiring as many entry-level staff, then we decimate the future managers and leaders of the company. And I don't know that, again, kind of like the earlier question, I don't know of an organization

47:32that has this figured out. I've talked to lots of companies and there are lots of AI transformation leaders and HR leaders at big companies that are just starting to ask these questions themselves. And nobody seems to really know where to go with it yet. Yeah. 41 days, in case you're wondering, until my con. No pressure. I only have three talks to create in that time period.

47:58Number 14. You described the actual mechanics of an LLM predicting the next word as a bit of a black box. Is it really? And if so, isn't that a little ominous? It is really a black box. There's a field of study called mechanistic interpretability where they try and understand why the models do what they do. The LLMs itself is making predictions based on proximity of words to each other within a graph. So this is a very oversimplified way to try and help people visualize this.

48:30Imagine all of human language as this three-dimensional cube. And within that cube is every word in the human language. And how close those words are to each other sort of shows a probability of them occurring next to each other. And so when ChatGPT starts writing something, it's in essence going through this dimensional graph. And it's like knowing that Kathy sat on the chair.

The black box of LLMs and platform risks

48:59And it knows that chair is a likely word connected to where Kathy would sit because people sit on chairs. Could be a bench. Could be the table. Could be all of these things. But in essence, like imagine in that cube, those words all appear near each other. Sat, chair, desk, whatever. That's kind of like how it works. The way to think about the language model, the closest proximity to it is the human brain. And that would be like saying, well, do we really not know why Paul picked the words he just picked when he was explaining this?

49:29Like, no, we actually don't. Like, you could put me under an MRI machine and it might see what neurons in my brain are firing when I'm talking, but it can't drill in and say, why did he choose the word he chose? It's likely just on my training and in my experience as a human of the words that appear near each other in the English language. So that's kind of it. Like, yeah, it's ominous. Yeah, it's weird. Like, we don't really understand this intelligence that we've created. All we know is that we give it more and better data and we give it more chips to train on that

50:06it seems to just get smarter and more capable of simulating human behavior and language. And that's kind of it. Like, that's the basis for the first scaling law of AI that has now stayed true for the last decade, which is more data, more computing power. The bigger the model, the better it seems to get. Um, there was a, this sort of ominous thing that Ilya Sutskova said back in the day at OpenAI, which was they, they just want to learn like these models just want more information and they just get smarter.

50:37Okay. Number 15. The nice thing about these questions is that I get to decide the question. So I actually dropped in my own question. My husband's been traveling for a week and I've had a lot of free time. So this is, these are the things I think about. Um, okay. When I worked with Joe Polizzi at the Content Marketing Institute, for years, he always emphasized the importance of building on land you own, your blog, your email list, your audience. We've seen, we've seen, um, what can happen when people go all in on rented land, like LinkedIn, Twitter, or another platform.

51:08And the algorithm changes, products, audiences, um, audience becomes harder to reach. The platform disappears. Now people are building products, audiences, and sometimes entire businesses on custom GPTs and other AI platforms. Are we watching the same story repeat itself with LLMs? And how should we think about the risk of building on rented land? That's a deep thought to have. I know a lot of free time, not really free, not really free time. Actually alone time. Yeah. I think it kind of relates back to the thing I mentioned earlier about, you know, if you

51:40build on a single platform, you're sort of a prisoner to what happens to that platform.

51:47I, I look at this a lot now with like how transferable, um, skills are between agents and platforms. The system instructions that are used to build a GPT can very easily be copied and pasted into Claude and you can build the same capability over in Claude. So I think that there's an inherent risk here. It's probably part of the reason why some organizations are now choosing to take open weight models and train their own models. It's just a more reliable system. The model lives on internal servers.

52:17You're not, um, you know, captive to a cloud service going down and you no longer have access to it. So I do think that more organizations will start to build redundancies and backup plans in, um, at a very surface level, you can start by making sure everything is documented. We've, we've done this again internally at SmarterX where we're now at least from a governance perspective, documenting when apps and agents are created by different people. Ideally you include what the system instructions are, what the knowledge base they have access

52:49to is so we can go back and audit it. And so let's say like worst case scenario, you build everything around ChatGPT, you've got 25 agents that are basically running the company. You've got 75 skills that those agents use to do what they do. And then ChatGPT goes down for five days. You could, worst case, go grab the skills, recreate them in Claude, or you just mirror everything in Claude. Like someone's job literally might just be maintaining a mirrored image of the ChatGPT

53:19instance in a second platform. So that if something goes down, you just flip a switch. Everybody moves over to Claude and continues their work. I would imagine there are organizations that are actively doing what I just described. I haven't talked to them. It's just one of those things you can kind of make an assumption that someone is ahead of the game doing this because we are talking about entire companies running on this. And now you have companies laying off thousands of people under the assumption that agents can do a lot of this stuff. Well, if those agents aren't available, work doesn't happen.

53:50So there needs to be redundancies built. Yeah, and I was talking to Claire, who is our producer of the podcast and works on our content. We were talking today about how Mike had actually done, talked about this in one of his courses for Academy about how to extract some of that from one tool to be able to upload it and put it in another tool or how to document it and what that process is. So it really is a critical step of all of this. All right, that is number, that is 15 questions. That's a good questions. I feel like those are all different questions, too.

54:22I don't know that I've answered any of those. Maybe the marketing use cases has probably been answered at different times, but it changes. But the rest of these are relatively original, really good. You don't look at this beforehand, but Claire and I do. Yeah, you did a good job. She does a lot of, has this been asked 20 times already? Let's not ask it again. So yeah. Yeah, good stuff. So thank you, Paul. You'll see Mike next Tuesday back on the podcast. If you are interested in learning more about Marketing AI Month or Macon, our big event

54:52coming up in October, please let us know. I'd love to help you get you to both of those things. Get registered. Come to Cleveland. We've got, oh, do the pod because we have the pod 100, which gets you a VIP lunch with me and Mike's. We're doing on that, the final day of the October 15th, a VIP lunch for anybody who uses the pod 100 when they register. So yeah, macon.ai. Get registered. We'd love to see you there. Pod 100. Thanks, Paul. Thanks for listening to AI Answers. To keep learning, visit smarterx.ai, where you'll find on-demand courses, upcoming classes,

55:26and practical resources to guide your AI journey. And if you've got a question for a future episode, we'd love to hear it. That's it for now. Continue exploring and keep asking great questions about AI. Thank you.