How often do you use 'the'? Meet Zipf's law

0:00Grammar Girl here. I'm Mignon Fogarty. This week, we'll talk about Zipf's Law, a pattern that shapes how often you use your most common words, and why SETI researchers think it could help identify alien communication. But first, Grammarpalooza the Game Show to celebrate the 20th anniversary of the podcast is in just two days, September 24th, and you'll want to join us for the live stream at 8pm Eastern as six fabulous guests compete for Grammar

0:35Glory because we will all have fun, but some of you will win books too. Check any of my social media pages for more information, or just head over to my YouTube or Facebook page and RSVP for the event. The links are in the show notes, and I hope to see you there. Maybe it's weird for a word person, but I love spreadsheets. I love data and charts. And the other day, Pat was teasing me about it over breakfast and asked what he thought was going to be a ridiculous question. Don't you want to know how many times you use the word

1:09the in different types of documents? And I was like, well, actually, there's this fascinating thing in linguistics. And I was off to the races, and he just started cracking up. But there is a fascinating thing in linguistics about words like the, and it's called Zip's Law. And now I have to tell you about it too. Here's the deal. In many, many human languages, the most common word appears about twice as often as the second most common word, three times

1:42as often as the third most common word, and so on. Mathematicians call that being inversely proportional. Here are some real numbers so you can see how precise it is. In the brown corpus, that's this famous collection of American English texts, the word the appears about 70,000 times. That's about 7% of all the words. And the second most common word, of, appears about 36,000 times. That's almost exactly half as often as the. And the third most common word,

2:19and, shows up about 29,000 times. The pattern kind of falls apart here because and isn't showing up a third as often as the, but the words do still fall off in a regular pattern as you go quite far down the word list. And people have come up with a more complicated mathematical formula that describes the pattern that they call the Zipf-Mandelbrot law. Now, that might not seem like such a big deal, except that it is not just for English. It shows up in a huge number of languages.

2:52So, who discovered this pattern? Well, the law gets its name from a Harvard linguist named George Kingsley Zipf. And here's my favorite part. He apparently wasn't a huge fan of math. But he wasn't the first person to notice the pattern, though. A French stenographer named Jean-Baptiste Estoupe had documented it by 1916. But Zipf was the one who really looked at why it happens, which he included in his 1949 book with this great title, Human Behavior and the Principle of

3:25Least Effort. Yes, Zipf said that speakers want to be lazy. He said, we'd love to use just one word for everything because that's easy to remember. But listeners are different. They want clarity. So they want different words for different meanings so they don't get confused. So his theory was that the distribution of words we see in language comes from the balance between those two opposing forces, speakers trying to minimize effort and listeners trying to maximize understanding. Zipf died in 1950, when he was just 48 years old. But

4:03his law lives on, and it's gotten even more interesting since then. Those other languages I mentioned? Well, researchers looked at 50 languages in 2018, 50 from all over the world, Indo-European languages, Sino-Tibetan languages, Dravidian, Afro-Asiatic, and more. And they all followed the same shape, with the middle of each word list closely following Zipf's law. The rarest words dropped off faster than the law predicted, but it happened the same way

4:36in every language. Other researchers looked at other samples, including James Joyce's Ulysses, Cervantes' Don Quixote in Spanish, massive French novels, Finnish texts with complex word forms, and even constructed languages like Esperanto, and the pattern still held. So we've got this pretty enticing, universal-ish pattern, but scientists have been arguing about why it happens for 90 years. Zipf's original idea about language gravitating toward the least

5:11effort it can get away with while preserving clarity is one possibility. Another theory said that it's the result of success breeding success, technically called a preferential attachment process. The idea is that once you use a word in a document, you're actually more likely to use it again. And I won't go through them all, but another is that it's all just the result of random processes, that if you sit a monkey down at a typewriter, you can get text with a Zipfian distribution. One 2014 review looked at all the competing theories and concluded, and I'm quoting here,

5:48that, quote, no prior account straightforwardly explains all the basic facts, unquote. And now, here's where it gets really wild, though. Humpback whale songs follow the Zipf-Mandelbrot law. And it's a little more controversial, but some people also think they find it in bottlenose dolphin clicks and whistles. Researchers at the SETI Institute, yes, the Search for Extraterrestrial Intelligence, they have thought about using Zipf's law as a test for complex

6:20communication. They tested it on a pulsar signal from space and found that those signals didn't follow the pattern. So they say following Zipf's law is one thing they might look for to decide whether a new signal is alien communication or not. And here's another cool discovery. Remember how I said Zipf's law works for words? Well, a 2015 study found that it works for phrases, too, and it actually continues much longer down the rank to list for phrases than it does for words.

6:52The authors speculated that maybe phrases rather than individual words are the fundamental units for understanding Zipf's law in language. That might explain why no theory fully accounts for the word-level data. Maybe we're like researchers trying to understand a building by studying individual bricks when we should be looking at whole walls. And here's a final tantalizing tidbit. The undeciphered Voynich manuscript, whose parchment dates from the 15th century and that people have been

7:23trying to translate for hundreds of years also follows Zipf's law. Some people think that means it's a real language that we just haven't figured out yet. But other people, remembering those typing monkeys in made-up languages like Esperanto, say it doesn't actually tell us much either way. So here's where things stand. Zipf's law is one of the most robust patterns in linguistics. It appears in at least 50 languages. It works for highbrow literature, newspapers, and even whale

7:54songs. It could even help us detect alien intelligence. But after about a century of research, we still don't completely understand why it exists. Maybe it's speakers being lazy. Maybe it's our tendency to be drawn to things that are already popular. Maybe it's just the way randomness shows up in language. Or maybe it's all three, working together. But it's a fun thing to ponder over breakfast, especially for a data lover like me.

8:31Finally, I have a quick little familect from Stephanie. Good afternoon, Mignon. My name is Stephanie Bryant, and I'm in Kentucky. And our familect saying was bump medicine for hydrocortisone cream. Kids cannot say hydrocortisone cream. Thanks, Stephanie. Hydrocortisone cream is definitely a tough one for kids to say. If you want to call with your familect story a word or phrase you use with just your family, or maybe a friend-elect you use with just your friends, you can leave a message on the voicemail

9:05line or use your computer or phone to record it on SpeakPipe. And both of those are in the show notes. Grammar Girl is a Quick and Dirty Tips podcast. Thanks to Nat Hoops, Morgan Christensen, Rebecca Sebastian, Stephen Schapansky, Holly Hutchings, Dan Fireopend, and Maram El-Nagib, who has ridden all 28 subway lines in New York City. And I'm Mignon Fogarty, better known as Grammar Girl. That's all. Thanks for listening, and don't forget to like and subscribe where you listen, and join us Thursday for Grammarpalooza The Game Show.