Stray Russian words and made-up style rules, with Daniel Heuman and Chris Ryder

The Hidden Flaws in AI Writing

0:00Grammar Girl here. I'm Mignon Fogarty, and today I am here with Daniel Heumann and Chris Ryder from Intelligent Editing. Guys, welcome to the Grammar Girl podcast. So good to be here. Thank you, Mignon. Lovely to be here. You bet, Daniel. I've known you for years, and I saw this great post you did on LinkedIn, which is why both of you are here today to talk about your work on editing software products. And, you know, I like that you're so connected with the world of editors. You know what is going on

0:32with editors so much that I always love talking with you. And you had this blog post about AI. And I think that we all, you know, recognize that raw AI writing style. And a lot of people feel like they can pick it out. We don't like it. But I think of AI as producing pretty much mechanically perfect writing. So much so that, you know, horrors, I see people talking about introducing errors to their writing to make it seem more human. Like, that's how much we all think it's like,

1:04quote unquote, perfect. But your blog post pointed out that like, there are all sorts of ways in which it is not perfect. And it was sort of revelatory for me. So I would love to go through these errors that you have identified that it can still make. Amazing. Thank you. And first off, I feel really bad that I wrote the article in this commercial way. I was rereading it today. And it's and it's selling our product perfect it. And I'm really grateful that you read past that. And saw that underneath there was a, you know, an important argument that we were trying to make. And I may have to go back and delete some of some of the

1:35commercial lines because I appreciate this. And I want to point out, like, this is not a sponsor, we will talk about your products, but this is not a sponsored post or anything. I just think that your work is interesting. So

Inconsistencies in Spelling and Rules

1:46let's talk about first one was US UK spellings and inconsistencies. I mean, this one, this one for me is always going to stand out because I can't do this at all, right? I am half British and half American. The accent for your more American audience is, you know, the best I can do to sound British, but but I am 50 50. And my spelling is all over the place. I've lived time, I've spent time in the US, I spent time in the UK, and I can't keep track of what is what. So I have spent more time thinking about those things. So I really notice if the AI

2:19produces that same mix that I'm afraid to say I produce. And it comes up especially with because there is no true definition of what is UK English, what is US English, someone has told the AI that, you know, use a UK English set. So spell words like connection, C O double N E X I O N, which is extremely rare in this country. Chris is PhD linguistics, he can maybe chime in on what could

2:49be my false impression. But my understanding is that if someone says produce this in British English, you wouldn't normally do that. But if you want to do a list of words that are accepted that a spell check will not flag, well, that is an accepted British variation that you wouldn't see so often in the US. So it does these, it doesn't do that all the time, either. It'll do these things sometimes. And it really stands out when you start seeing these spellings that don't quite fit. It goes even more wrong. When you hit Oxford spelling, I've not seen the AI yet that can get that right that mix of

3:23we're going to have I Z E endings with UK spelling. Most people when they do British spelling do an I S E ending on things. So the, the variation that comes in all our language is, I suppose, reflected in the variation that the AI produces, but it's not consistent. It's not doing it in a way that makes sense with what we expect or what we instruct. And that's why it just strikes me every time why it was the number one on my list, why I really struggle with, yes, most of it does seem completely correct. But these certain elements, jump out. It'll go back and forth. It won't always be consistent. Is that right?

3:57That's right. I mean, in more often, when it's doing British spelling, we'll do the I S E. But if you tell it to do Oxford spelling, I mean, it's gets very confused. Sorry, today, the 19th of August, as I say these words, it gets very confused. By the time this airs, it's quite possible that it will be fixed. It's changing really fast. Chris, you're the linguist who is, you know, doing work on all these software products, intelligent editing. What, you know, is British English versus American English a big challenge for you? How do you distinguish, you know, what that even means when you're trying

4:31to set down rules? Yeah, it's not easy for the reasons Daniel said, you know, there are sort of these prescriptive rules of this is British and this is American, but it's really not that simple. Like you say, connection with an X. No one spells it with an X. J or spelled G-A-O-L. I mean, unless you're writing a 19th century novel. No, it's J-A-I-L in British English as well. We all spell it that way. And so it's, it really does depend a lot on usage, not these sort of prescriptive rules and usage always changes. Um, so it's not that easy to pin down. And then I also think as

5:05well, we're talking about UK and US spelling, Canadian spelling, I would say, forget it because it's such a hybrid anyway. I think Canadian users must have a terrible time trying to get AI to do what they wanted to do. Yeah, absolutely. Okay. So, well, the next problem you found was with table headings. I mean, no one gets these right. No one gets this right at all. So why would we expect an AI to do any better than, than we can? Well, you would expect it to do better because you think machines are, you know, programmed a certain way that, that they will be consistent, but large language models aren't

5:40programmed that way. That's not, that's not the mechanism behind modern AI. So it is not consistent when you get to things like the top row of a table, the top column of a table, I guess we can call that the heading, the heading row, as opposed to the heading label. So when it does those, it's really hard for people to keep track of, well, this is a heading, am I capitalizing every term? And similarly, it's one that the AI will just go wrong on. It's really surprising. As I say, I'm questioning my own words that I wrote in that article, because it doesn't

6:14seem like it could be true. And yet I wrote that because I was editing a lot of AI texts and I, and I found these things that it was definitely getting wrong. And I, I can't explain that, but I certainly found it. I get it wrong because we get it wrong so often and it's trained on human writing. I think it has to be something like that. If you train these models on enormous amount of human writing, you're training them on an enormous amount of human errors. And people really struggle on that, on that capitalization of table headings and in different ways. So if we're not getting it right and it's trained on this data that isn't right, how is it going to get it right? Is my

6:48understanding of how AI works, but it's, but it, but it is remarkably strange. Yeah. I struggle with that too. I cleaned up a spreadsheet the other day and I noticed that all my headings were erratic. Some were capitalized, some weren't. I couldn't decide what I wanted. Yeah. It's, it's, it's hard. It is hard. And what about, and I guess also, uh, the same as for title capitalization, you know, like Chicago manual of style title capitalization rules are really complicated and they change. And so I think you noticed that not only is it not current,

7:22um, necessarily when AI tries to write Chicago manual of style, you know, headlines essentially, again, it struggles as much as we do because those are complicated and ever changing. I think that's the story. I think what I can't speak for the Chicago manual, but it seems to me that one of the reasons why they might have changed their rules between the 17th and 18th edition is that the rules were really complicated. The rule that we learned in school for headings is what the words that are less than three letters or four letters use lowercase, but that is not

7:55the actual rule. And I'm going to have to give way to Chris to, to fill us in on, on exactly how you break it down in terms of parts of speech. Cause I will, I will struggle, but it's for in Chicago 17th, I think words like under and underneath are treated the same way because they have the same meaning. Are we on prepositions here? I may just have made a fool of myself, but because if that is a preposition, then it doesn't matter that it was long. Whereas Chicago 18th switched the rule so that when it's the short version, it's can be lowercase, but the longer one is capitalized.

8:30I'm struggling. People struggle that AI cannot today, August, 2026, cannot get it right. That notion that we're all, and which would it follow anyway, unless you specifically tell it exactly what rule we're following in exactly what way, then it may do better. But if you just tell it, be consistent on title capitalization or follow the Chicago manual's rules on title capitalization, it will really struggle. Yeah. And this is something, one of the other problems you found is that it just, it can't follow Chicago style or AP style. And I made a little video for the Associated Press.

9:05It might not be out yet by the time this airs. But again, I had noticed that, you know, you cannot say to it, put this in AP style because AP style is behind, you know, the AP style book is behind a paywall and, you know, the AI at least shouldn't be able to access that. And, you know, same with Chicago, same with anything that's behind a paywall. It should not have that in its training data. So if you say, follow this style, it's going to go out and, you know, grab what it can find about

9:35that. There, you know, there are blog posts about it. My website has blog posts about it, but some of them, you know, go back to 2005, for example, and styles change. So it might have the old version. It might just make something up if it's not there at all. That one cracks me up every time. And it's absolutely true. The notion that you give it an instruction and because it thinks it's following a rule and a rule is important. I don't know why it does it, but it decides to make up its own rules. And to make it even harder, if you're editing a document, it's very rare that it would do that at the beginning. It's very rare that it

10:08will do that at the end. But the way it's context work means that right in the middle, as you're just stopping paying attention, as you think it's got it, is when it will start making up its own rules and things like that. That's so interesting. So it's more likely to be wrong in the middle of a document. Yeah. Fascinating.

Erratic Hyphenation and Odd Insertions

10:29And then the last one is erratic hyphenation. And, you know, when I give webinars about style, the most questions I always get are about hyphens. Like, what do I do about this hyphen? What do I do about that hyphen? And it sounds like AI struggles with hyphens, again, just as much as we do. Again, I was reading back over the article just before this session, and I couldn't believe that I'd written that. It's so surprising to me that it would be inconsistent on hyphenation.

11:00And yet I know I wrote that because I was editing AI after AI after AI and clearly was finding the mistake. It doesn't seem like the kind of thing a machine would get wrong. And yet it can. It's particularly made worse when it's not just the machine, right? Workflow is not usually let the AI do it for me. It might be I've done a piece that the AI then edits. It might be that the AI does something, then a person does something, then it goes to a third person. What you get is a complete

11:30mixing of styles. And a mistake that starts with AI can be compounded by a person, corrected by a person. There might be no mistake in it. And then a person writes their own way and writes something inconsistently. So absolutely, it's easy for us to sit here and go, I don't understand why the AI is doing this. But in real life, what happens is more complicated. It's that we've got a workflow that involves multiple documents, people mixing with AI. And where exactly the mistake comes in or where the inconsistency comes in, because it's not a mistake, it's an inconsistency, is really hard to

12:03trace. What we do know is that there's no reason why a person would write the exact same thing the way the AI does, especially if it's a gray area in English. So that's why I think we see the mistake. As I read back on it, I want to attribute that error to a person, because it makes no sense to me that the AI on its own would do it. But yet, you know, I've watched it, witnessed it, and tried editing it. And absolutely, these slightly obscure, very difficult mistakes are where the AI seems to go wrong today. And I think that we've grown up thinking about computers in a certain way, that they are very

12:36rules-based. And we're having to change the way we think about it, because these systems aren't the same. They aren't based on programmed-in rules. They're more probabilistic. And it's really a whole new way of thinking about what computers are doing for us. Precisely. And it goes back to your previous question, because that notion of telling it to follow a rule is what it can't. The thing that makes large language models so incredible is this generative capability. They will not produce the same thing twice. But if you want it to follow a rule,

13:09you really, really, that's kind of the definition of a rule, right? You want it to follow the same thing twice, and it won't. And Chris, you said that you had noticed other mistakes that only AI can do, not the kind where it's doing the same kind of mistakes that we do, but that are mistakes that like sort of only it does. What were some of those that you saw? Yeah, there's been a couple of bizarre things. The main one off the top of my head that really blew my mind was I was sort of happily chatting to AI, and it was giving me a nice long response. And then there was just one single word in the middle of a sentence. I can't remember the word, but just one word in

13:42Russian. I don't know why that was. And I've seen it a couple of times, Arabic sometimes as well, just a single word in the middle of an English sentence. And I just, I don't know why it does that when it's, you know, probabilistic why it thinks a Russian word is going to occur in the middle of this English sentence. But yeah, I just thought that no human would ever do that. You would just never make that mistake. So it's a dead giveaway that it's AI and just a very bizarre thing. I've seen that too. I've seen Chinese. And I guess those are better because they jump right out at you, right? They're not the kind of thing that you're going to miss when you're reviewing a

14:14document. So maybe they aren't quite as dangerous in some ways as the, you know, inconsistent hyphen hyphenation hyphen, hyphen, hyphen, inconsistent hyphenation. It's a really hard one. We can't even say it right. Nevermind write it correctly. It's not a chance. Yeah. So, you know, given that the text that comes out, well, when it doesn't have Russian or Chinese inserted into it, it comes out looking really polished and perfect. I think, you know, I had a friend who got a job at this new company and she did it first. I was so impressed

14:47that all the, you know, the young people at this business, they wrote so much better than I'm used to. I was really impressed, you know, and then she realized they were all using AI and it just came out looking really polished, but like it had problems underneath. And so, you know, it comes out looking like at first glance, really great. And then it has all these problems with it. So what does that mean for, you know, the editors, especially, you know, your products serve editors and you're really tied into the world of editors. So that's kind of what we're talking about today. So what does

15:18this mean for editors that things look so perfect, but maybe aren't?

The Burden Placed on Human Editors

15:24I mean, I don't know what you've seen, Minyan, but I see complaint after complaint after complaint. The one that I always go back to was an editor who wrote, if you really want to pay me to fix the mistakes that an AI makes, then I will take your money, but I would much rather work with people. And I think that really sort of epitomizes it. It's this, it's a very difficult edit to do. Working past, you know, the kind of things we're all used to doing and then having to check

15:55whether a document is coherent in entirely different ways. To have to look for those kind of mistakes is a completely different way of thinking about your edit. And I just think that's a great deal harder on top of the normal mistakes that it can make. And then there's an entire extra layer to that because it's not just that an editor is working to work with an author who is using AI. Sometimes an editor is working after a document has been edited with AI or

16:28an editor's, you know, using an AI to edit. But that's where a whole separate category of mistakes can come in because most, probably the majority of complaints I see editors make online, aside from all of the issues about copyright, is that the AI will over edit things, right? If they have to go in and fix a mistake that an AI introduced, that is a much, much harder thing to do than working with a person who makes an error. To have to kind of undo what

16:59was there or see past what is written. When you're an editor, there's so many amazing things editing community. But one is that sort of humble approach that most editors take. The cliche of an editor is them just sitting with a red pen and fixing things. It's not true at all, right? The reality is editors are really humble and try not to change an author's word and to respect what's on the page. But how do you respect what's on the page if you're dealing with an AI introducing errors that have already diminished the author's work? That is really tough. And I don't know that people

17:31have got an answer to that even. Chris, from the standpoint of a linguist, is there a way, do you think of the editing that people have to do on AI work, you know, from a language perspective, how is it different from the work that people are doing on just purely human written work? I think there probably is a difference because like you say, it's so, it appears so polished. I think this is a point you make in the article, Daniel, that you sort of, when it's human written, you know, when you're looking at a first draft or a rough draft. And so, you know, that you're looking for

18:02certain things. When you sort of get the AI thing, it's so polished, you lulls you into a false sense of security, I guess. So you, you're maybe not as alert. You, you think, oh, okay, this job has already been done. I don't need to look for these things, but you sort of still do. You sort of need to be as alert as if it's a human written thing and a rough draft, but you're not, it doesn't put you in that mindset, you know? Yeah. I mean, I'm thinking about, you know, sort of parts of linguistics or about sort of the cues we get from things that are there, the intentionality or the

18:32implication behind writing that we know was written by a human. Like if it's written a certain way, there must be a reason it was written that way, but the author intends, for example, and you can't necessarily rely on that when you're reading something that's machine generated. Yeah. And I think there's an issue maybe with ambiguities where, you know, a human might read it and they know that they, you mean this thing rather than that thing, but the AI could misinterpret that, change the meaning. And then it goes off to a new author or a new editor who doesn't know that this has been changed and then nobody spots the error. Yeah. So now we're going to take what will probably

19:06be an unexpected shift in the conversation because we've been sort of trashing AI for the last, I don't know, 20 minutes. And now we're going to say, you know, Daniel, your company is producing an AI editing product. So tell us about FirstEdit.

Designing a First Pass Editing Tool

19:22Fortunately, it's not a complete shift in the conversation because FirstEdit tries to pick up on some of these problems. We start from a premise that we occupy a really special place in the editing community in the world. Or 95, 99% of the population feeding a document to an AI and telling it to edit it might be right. Our audience is not that group. Our people who come to us are looking for the absolute highest, best possible version of a document. And because of that,

19:57we need to find a way to pick up on these mistakes that an AI is making or provide tooling that enables our audience to hit that highest level. And that was the idea behind FirstEdit. It's like, okay, what if we know that the audience wants to hit that very, very top level, what would that look like in theory? And the next thing I say is going to sound probably so obvious to you, Minyan, but I don't think you will have heard it before. But it's this, if you want to hit that highest level,

20:28and you want to have an automated editing function, so where the AI does something without human checking, then the AI must work before the human works. I know, it's kind of obvious and so plain. But when you break that down, that's not what everyone else is saying. People are saying, oh, you replace a human being with an edit, or you run this in AI at the end just to make sure. No, the very best result will come, definitely, if the automated function is first, and then the human being is working. That's how you've got to get around that.

21:00I actually got a little shiver when you said, like, have AI go last. No, that would be very bad. Right? But if you think about agentic workflows and things like that, that's how they're running. It must be a person at the end who is checking things if you want to get the very best result. So first edit, the name is pretty descriptive. It's a first pass that people run a document through to get sort of the, it catches what, the big errors, and then, you know, you go back and do the more fine tuning? Sort of. But yeah, you've got, the name speaks for itself. It is that idea of a first pass.

21:31And then what sort of followed from us is that if you're going to do a first pass like that, and you know, these errors that machines make, the first pass should never over edit, right? The two mistakes are not equal. If an AI under edits, you've still got the human at the end, who's going to find mistakes. If the AI over edits, you've just made the job harder for that person at the end. And as we really just discussed, that is the hardest kind of error for a person to deal with if the AI has introduced it, and the AI is wrong, and somehow they need to trace it back and figure

22:03out where this has gone wrong and why, that's really difficult. But if the AI leaves something, you're no worse off, right? You had a human doing it to begin with, and the AI has maybe done half, or maybe it's done three quarters, or maybe it's done a quarter, whatever it is, it has not done the full thing. It's done an under edit, never an over edit. And that's where I think we've found a, hopefully, you know, we haven't released this product yet, but hopefully got a very different place in the market that it's not what everyone else is doing. Because no one, sorry, 99% of the population

22:35does not want to buy a tool that under edits by its design. That sounds like a terrible product. And who would want to have anything to do with this? Well, the only people that would want this is the people following a workflow that actually gets to the very best, but it does so by having a person at the end. That's funny. And as an editor, as I was reading about it, the thing that also seemed very important to me is that it's using tracked changes. So you can see every change it has made. Yeah, absolutely. I think that is becoming more common. We've gone through, what is it, three years now, coming up to four of ChatGPT and all these other models. Doing things without track

23:09changes is just a complete non-starter. I think if you use different versions now, there are ways to get track changes. But certainly, if you want to hit that very best version, it must be. You have to be able to see what the tool has done. Even if it's just to gain the confidence that it's not over editing, you must be able to go back and look and see and do an audit log because otherwise you can't get to that point. You run the risk of the AI introducing mistakes, which is never going to be the best version. Thank you. This is so interesting. I mean, there's just so many new things for editors

23:42to consider about what we're dealing with in the text that we're receiving from writers and how our workflows are working and just changing the way we think about what computer products are and what they do. There's just really so much to think about. And the other thing is it's changing so fast. I'm sure this is an episode I will never rerun because it'll be obsolete in six months, but it's a fascinating conversation. And I really appreciate you being here today to talk us through these new ideas. Thank you. I hope that it's interesting. We have gone way, way deep on some extreme language

24:14geekery, but I love that you make that possible. Thank you. Thank you. And for our Grammar Pelusians, our wonderful supporters over on Patreon, we have a bonus segment like we always do with our guests. We're going to talk about two free tools that the company puts out, Intelligent Editing Style Works, and a book called Brave New Words filled with puns. Very fun. And we're going to talk about those and get both Daniel and Chris's book recommendations in the bonus segment. For the rest of you, that's all. Thanks for listening.

24:44Thank you.