The Judgment of Paris at 50: Wine, Wisdom, and What We Still Don’t Know

Introduction

0:00Hello, and welcome to the Harvard Data Science Review podcast. I'm Liberty Vittert Capito, the feature editor of the Harvard Data Science Review, and I'm joined by my co-host and editor-in-chief, Xiaoli Meng.

0:15This year marks the 50th anniversary of the Judgment of Paris, a blind tasting that changed the wine world forever. So in today's episode, we are talking all things wine with our guests, Orly Clark Ashenfelter, economist and wine lover at Princeton University, and Susan Lynn. In 1976, British wine seller Stephen Spurrier and Patricia Gustav Gallagher, the director at the Académie du Vin, organized the blind tasting to pit new world California wines against

0:47their old world French counterparts. Even in California, at the time that the Judgment of Paris came along, California wines were not considered top quality wines, even though many of them were. Orly says what could have been an ignored tasting was widely received and promoted because there happened to be a U.S. journalist at the event. I think it was Patricia Gallagher who contacted George Tabor. Tabor was a journalist in New Jersey, and he spoke French. So Tabor came to the event and was able to watch as the wines were unfurled.

1:24They knew that they were tasting California wines versus French wines, white Burgundy versus California Chardonnay, and Napa Cabernet versus Bordeaux Reds, which are often heavily Cabernet Sauvignon grapes. So George, bless his heart, spoke French, so he could listen over here what they were saying. And they would say, oh, this one smells just like a garbage can. It must be from California. Turns out it was Chardon Mouton Rothschild or something. It wasn't anything like what they thought it was.

George Tabor's Article

1:57And the result of that was George wrote a very short, brief piece in the Time magazine explaining what had happened and that on the red side and the white side, the dominant wine was one from California, which shook people up quite a bit. And then I think the second thing that was most important is the people in California are not immune to publicity. So everybody is aware of public relations, and the California guys really jumped on top of this. So this tiny little one little thing was then blown up by the people in California with a

2:31good PR machine behind it. And I think it took on such an importance that there were afterthoughts by many people about what had actually occurred. Ten years later, there was a replication using the same wines, actually. So I think it took on a special importance for those two reasons, really, that George was there and the fact that the California guys really knew how to do the PR.

Impact of Judgment of Paris

2:53The impact of the original judgment of Paris didn't fade with time. In multiple anniversary recreations of the tasting, Californian wines continued to outperform their French counterparts. In 2021, Susan Lynn participated in a tasting that commemorated the 45th anniversary of the 1976 event. I was pretty much a newly minted master of wine, and I was so excited and so honored to have

3:23been invited to be on this panel of tasters for this judgment of Napa on this momentous anniversary. And what was so interesting was that I think it really does show how much of an impact, not only culturally, but also on a global scale of the international wine market that the judgment of Paris has had upon our entire industry and upon people's perceptions of what wine is,

3:55what quality wine is, and what quality wine can be. I think that's one of the most important things. And I really think that it's a wonderful way to celebrate the impact of the judgment of Paris because it really, really mixed in some top white Burgundies, top red Bordeaux, and a number of Californian wines and international wines. And I think that turned out to be very, very fair. They were all fantastic.

Wine Evaluation

4:19It's actually very interesting that we call those things judgment, right? Which means that it's not something you can just determine using a formula or using some statistical, you know, methods that takes quite a bit of a human judgment. And we all know that judging the wine quality is extraordinarily difficult. Personally, I have done lots of blind tasting. And most times, we all know that most of us don't know what we're doing because it's just very hard. And, you know, when you taste the wine, seriously, as Susan, you're a master of wine, what are you

4:51actually paying attention to, what separates a disciplined evaluation from someone like me who say, oh, I like this or I don't like this? Because I know you have a lot more systematic approach. First of all, one's own subjective tastes have absolutely nothing to do with the evaluation of the wines in question. Really, it's about building one's own data set as a taster to be able to effectively have enough to draw upon to make a reasoned and sound judgment about assessing a wine.

5:25So there's two things, especially if we're the master of wine here, we're talking about the theory behind the wines. Like, what are the regions? What are the grape varieties? What are the different expressions that occur in the different regions, et cetera, et cetera? All the, you know, permutations and combinations that you can get from different grape varieties and what people are doing in different regions of the world. So, I mean, there could be quite specific, you know, grapes like mostly, you know, Mavro is, you know, going to be from Greece, but more people are growing it in different areas

5:56of the world. But then you have like Syrah or Chardonnay, where it's called an international grape variety and then it's grown all over the world and you have many, many different expressions and learning to identify that in terms of the theory behind it. What's the terroir? What's the climate? What's the, what are the cultural factors that have shaped how a wine has been developed or is developing? And then, and then the sense memory, the sense memory is so important. So tasting again and again and again in different contexts and with different people, like alone

6:27and with different people and tasting for pleasure too, you know, that has a lot to do with it. So taking in that sense memory, but then being able to, not just on a personal level, but being able to, to codify that and to map that to the data that you are creating on the back end, so to speak with the theory, right? So combining those two things is essential to being able to evaluate a wine properly. But another thing that is extremely important also that, you know, the master of wine really

6:59emphasizes in this is, uh, quality within context. It's assessing what is the quality within the context of that, of that category of wine. And, you know, you get to the point where you taste wines blind. Sometimes you don't even know the price band in some, in some, uh, uh, awards competitions, but you know, you understand enough the different elements of the wine, the organoleptic properties, the sensory characteristics to be able to determine what probably this price band is and what the

7:30audience is for. Um, and this is something that is tested very rigorously on the master of wine exam in the, in the blind tasting, um, segment, which is called the practical. And so this was, I think, very, very good training towards assessing a wine in a very fair and objective manner. I know only you and your group has done many, many tastings. In fact, you guys published some data. I think it was a 40 year data and, uh, in the journal of wine economics. Apparently your group are taking a, probably a very empirical approach.

8:03And I remember somewhere you said, you know, these data pretty much shows like very few people know what they're really doing. Well, all of the tasting notes are online. There's hundreds of them now. It's liquidasset.com. It's not S there's just liquidasset.com. I mean, you find them all and some other links to other things. It's not a commercial website, really. One of the lessons that most of us take away from this is it's good to be humble. Uh, there are many, many, uh, ways that people describe this. Harry Waugh, the very famous English wine taster from years ago was asked once if he had ever

8:37confused a Burgundy for a Bordeaux. And he said, uh, no, sir, not since lunch. Uh, in other words, he had often confused them. Another one is a friend of mine who started this group. Actually, his view is that all wines are the same. The only thing that differs is what you're eating and who you're drinking it with. That was his view. This that's extreme, but it is extremely important point. We always have food when we're tasting these wines, which is not the normal thing. When you do a giant wine tasting, uh, there are people now who think that's better.

9:10I mean, a lot of wines don't taste so good unless you have them with something to eat. And then there are so many subjective factors that at the beginning, there were people in our group who I think were very concerned that they wouldn't have the same views as whatever the great wine critic of the day was saying was their view. The slowly we evolved to the point where everyone took their own view of things and it became separate. We made mistakes occasionally. My favorite example, one person was hosting the event and made a mistake and put the same

9:43wine in the list twice. Didn't recall that he'd done this. We tasted them. They didn't get the same score. This was the kind of an eye opener for everybody in the room. So I think the first thing to remember is to be humble. The second thing is I do actually lead wine tastings occasionally. And I, and I use these, uh, Kevin's Raleigh, it had their old book windows on the wine world. He, he has the four S's. And I always think this is probably the right thing to tell everybody who's starting.

10:14You swirl, smell, sniff, and spit. Now the, the spit part, maybe we don't do it all the time, but in other words, the whole thing is that you look at it, you smell it, and then you taste it. A lot of people just jump into something without paying attention to it as you're doing it. The color I think is of some consequence too. I like Susan's point because it is true that if you, if you don't have a broad background in different kinds of wine, it's often hard to make good judgments. From our group, I would say occasionally there's enough agreement amongst the group that we can

10:48rank the wines, but often there is not. And the most disturbing part about it is we don't seem to have learned anything. So even, even when we can't agree, we can come back to the same event later and we still don't agree. This is what originally prompted me to do a reevaluation of the judgment of Paris, because I got a hold of the data with each of the judges rankings and we were able to then do a statistical analysis. So I think it's important to know, is there enough agreement amongst the judges that when

11:19you say this is, there's always going to be a first place and a last place, but is there enough agreement amongst the judges that this really makes any sense to say that? Because if there isn't enough agreement, what that really means is if you do it again, you're not going to get the same answer. So that's not really a reliable way to figure out which is the best. The final thing I'll say about this is I think the most important thing about wine judges historically, not quite as important anymore, was finding flaws. You don't see flaws in wine so much anymore, but there were, there was a time when you could

11:53easily have wines that were, you know, a white wine that would, when you open, look like it looked at the color of a copper penny. That was not the way it was supposed to be. Or I remember a red wine I bought from New York at one point, they called it hamburger red. Honest to God, it tasted like ketchup. That was not the way it was supposed to be. So, uh, flaws hopefully will keep those wines from getting into the, into the industry or sales, but that's, that's, that's historically the most important thing. I think when, when the old English guys used to go to France, they had two purposes to visit

12:27and taste wines at the, at the Bordeaux Chateau. First of all, uh, to see whether there were flaws because someone could easily have dropped their socks in the tank or something and the wine was all bad. And the second was to make sure that the barrels they picked out were the ones that they received in England, because you could, there was no guarantee back in the old, I mean, this is no longer true, but I'm quite sure back in the old days, there were many French barrels that people picked out that didn't end up back where they were supposed to be.

12:57Uh, so that was another reason. So I think that wine judges are different now, but in those days they were extremely important. Orly, I have to wonder this. And it's something I've thought about a lot is, you know, whether people are actually more statistical about the way they, they judge wine or not, you know, I mean, you, so much of your work has been to help people think, you know, sort of probabilistically about wine quality and pricing rather than just sort of voodoo, um, or totally subjective tasting. Do you think people are smarter now than they were 40 years ago or 50 years ago?

13:29Or do you think it's just, it's the same? I can judge from my group, uh, within our group, once we start telling what the scores are from each person, what the ranks are from each person, people can immediately guess whether or not there's a statistically significant difference. They don't, aren't trained as statisticians at all, but you kind of get used to the idea of how much agreement you need before you're actually going to have something of a liability. The only experience I have with another group like that is, it's been abolished now, it was

13:59started by Maynard Amarine of all people, who's extremely famous in the history of wine in California. Amarine was a PhD student who, um, got his job just at the end of prohibition, literally, and was asked to get back, get the California wine industry restarted. And he started this, a wine committee at the Bohemian Club in San Francisco. And his goal, you can imagine, this was a fancy club in San Francisco. It had all French wines when it first started after prohibition. I imagine they were drinking some things that weren't on a wine list during the period of

14:33prohibition that we don't know about. But just, just to be honest about it, I think they, the wine list was then created and it had mostly French wines. And what Amarine was very keen to do was to be able to get the best California wines on that list. One of the early lists he made is in our library here at Princeton. It's a bound leather copy, beautiful, beautiful book that was printed by a very famous press in San Francisco. He was extremely proud of this. It had mostly French wines, but you know, they would be, this is a book from 1940 and

15:06they'd be drinking 1928 Bordeaux's and things. That group had a very simple system. If you were on the committee, which I was on the committee for one year, if you're on the committee, they had wines that they were going to taste, typically they taste from a group. So this would be like Napa Cabernet, Sonoma Cabernet, or they'd stick them together. And then what they did is you only said one of three things, buy, I'm indifferent, don't buy. That was the only rating you gave. Buy, indifferent, or don't buy.

15:36And if you went to the meetings often enough, you could tell right away when there was enough agreement that the wine director was going to think about buying it. So I think, and then I think people, they begin to realize that there needs to be enough agreement statistically for there to be a reason for you to move ahead. So I think in a way, if you're in a group that does it often enough, you don't need to know anything about statistics. You kind of learn it by the fact that, you know, if they're all over the map, whether

16:10a third, a third, a third, then you know right away, you're probably not going to want to buy it. I like that idea. And I think, you know, in the same sense, it sort of extends to your other work in really sort of modeling climate and how this is working. You know, with the sort of our move into machine learning and AI, do you see a real shift in how people are thinking about this and how it's going to work or how your modeling is going to work? Do you see it as extending it, changing it? Well, the big issue in the grape growing world is climate change.

16:44And I think that there's little doubt that data analysis will be a pretty key factor in how people adapt to climate change. So one of the things that Maynard and Amory is very famous for with Winkler, it's called the Winkler Scale. It basically was their effort to figure out what kind of grape should grow in which climate. California has so many different climates that you can actually grow pretty much anything you want, depending on what you, where you are. And they made a huge number of wines, actually, over a period of 20 or 30 years to try to figure

17:21out which would be the best place to grow them. So in a way, the first step of doing this is already out there. It's a big data analysis that was done back in the 1940s. And to be blunt about it, the data collection was far better than anything we do today. But the data analysis wasn't anything like what we could do today. And a lot of that data, they made thousands of wines, it still exists. I've seen it. It's in spreadsheets. But when I say spreadsheet, I mean that old big piece of paper that Jowli and I know what

17:53those were, but probably young people never saw one. They're still there, penciled in and pieces of paper. They didn't have a calculator even, a really decent hand calculator to work with. So I think there's a lot of room for that area in particular to think about methods for adapting to climate change and other factors that are going to have to affect grape growing. Thank you, Oli. That, you know, the climate change is such a big issue that I remember when I initially started the Wine to Mind Symposium, one of the two key questions I was asked to bring

18:28people to address is the climate change. And we did quite a bit of talks on that. But the other one is really about, you know, the human behaviors, human perceptions. And for that, I want to really turn to Susan, because Susan, you're not only the master of wine, you're also a master of fine arts. I'm very curious about your experience to study for the master of wine and a study for the, you know, fine arts. You're particularly, as I understand, you play wonderful piano. And also the most fascinating thing is I understand that your master of wine thesis was actually

19:03on the inference of the music, all the perceptions about champagne. And I think that's just a fascinating topic. So can you share with our listeners, first is how it's like, how to, you know, to learn both of things, you know, what that experience it was, and most of us probably will fail on either of them, and you've succeeded on both. And second, talk a little bit about your research, and particularly how data and analysis, you know, factoring in your research. I'm sure what you did, it has some data science component to it as well.

19:36Wow, that's a lot to talk about. But at the same time, I am, I feel so very fortunate that at this point in my life, I can bring together two disciplines and two crafts that I love so much in a way that I had never anticipated. And so with the research that I did, it was interesting that Orly was talking about this, because I had, I had multiple sessions of this about the same amount of people over 71 participants, and they had no idea what they were going to getting into, they just knew they would be tasting some wine.

20:06And I had five glasses in front of them. And I had four pieces in those separate quadrants that I was mentioning plus one control for silence. Right. And each session was, each of these orders was randomized by the Williams Design Latin square, so that none of them would be repeated in the same way. And it was absolutely fascinating. I have to, I have to recall what Orly was saying, how, you know, the different taste order for the same wine that was put in accidentally twice got different scores. You know, it really, that priming factor can be such, such a big thing.

20:39But in this case, nobody knew, like, I had about, I was lucky at about 50%. 50% people who were wine experts, who worked in the wine industry, people who are master sommeliers, or, or, or master of wines, candidates, etc. And then 50%, that was pretty much what we call the social drinker, which enjoys wine, but doesn't really study it. And across the board, statistically, significantly, it was crazy. People thought that all the wines were different, and they were, in fact, the same wine. And people were like, wow, you paired the music so great with each of these pieces.

21:13But it was, you know, but there were so many, there were so many findings that just blew my mind. But I was expecting, like, maybe this to show there's no effect whatsoever. And I would have to write the paper about that, because that's important to know. But I wanted to see whether the study that I did would replicate the findings of the previous studies, and then show some insights for sparkling wine, or maybe even other insights. And for a fruit non-vintage champagne, something that is fairly, you know, light and bright and more fruited than perhaps, you know, like a vintage champagne, which would have heavier notes, etc.

21:46Significantly, there was a clear winner as to which piece was the winner. And it was the final movement of the Carnival of the Animals by Camille Sanson, which is, if you've ever watched Fantasia 2000, it is the segment with the flamingos and the yo-yo. It was really cute. Fast tempo, good articulation, and high-pitched and gentle but exciting character brought out all the qualities of this wine, sensorially speaking,

22:17zesty, fruited, light, and enjoyable, etc. But the same wine, when paired with Brahms Violin Concerto Third Movement, which is passionate and very much heavier, and just absolutely complete opposite of that piece, brought out qualities in the wine that made it seem more like a vintage champagne. Richer, more complex, you know, deeper. And then there was some other music that was like, okay, the wine is a lot less effervescent,

22:48because it was very long, smooth lines. Whether it was bold or gentle, both of them seemed to be like, you know, without that fast articulation, without that sharpness, without that faster tempo, it was less effervescent, which was so interesting. But the wine was still rated well. But the thing that actually really, really got me was the fact that the wine, when tasted without any of the music, was terrible. People did not like it.

23:18They were like, this wine is out of balance. It's bitter. It's not effervescent. They found the acid too high. It was so interesting. And I thought, I'm going to do this statistical analysis just split, you know, just individually, just for the social drinkers

Magic Wand Question

23:30and just for the wine experts, because wine experts always say, you know, I even interviewed everybody at first. They're like, I absolutely do not want any music, zero distractions, because we are trained so hard to isolate every element and to do everything very systematically and academically, right? And so people said they absolutely disliked music when tasting wines. But after doing statistical analysis for both groups individually, after having done it all together as a group, it was clear.

24:00Even the wine experts, the perception of the wine was, everything was of lower quality when it was tasted without music. And I thought that was absolutely fascinating. Susan, I think I've decided now this is my favorite master's thesis. So thank you for that. Um, I'm going to move on to our magic wand question, if that's okay. And Orly, I would love to start with you. If you could wave a magic wand and have one rigorous answer at the intersection of wine and data science, what would it be?

24:34Well, that is a magic wand question for sure.

24:43Well, you know, I have mixed feelings about that. On the one hand, it would be nice to understand when you taste a wine like, you know, for example, last night, I have a new corkscrew. I couldn't bring myself to buy a Durand. This is what you need to open old bottles. 150 bucks. So, but now they have knockoffs. So I bought a knockoff and I really needed it because last week I had opened a 1983 Ducru Bocayu in a restaurant

25:16and it was just a disaster. The cork broke in pieces. It was flying around the restaurant. There's a cartoon that symbolized it. It's a picture of a waiter standing on a table with a pile driver trying to get the cork out of the bottle. That's about the way I felt. And so I bought this thing and then last night I opened another one and it worked perfectly. Now, I thought the wine tasted so much better when I was able to open it, but I think it was exact.

25:50I can't prove this. You know, every bottle is different, but I think it was exactly the same, but it did not seem the same at all. Uh, and we didn't have with any fancy food either or just having hamburgers. The reason I have mixed feelings is because on the one hand, I would like to know what are the characteristics that made that old Ducru, which it didn't used to be anything like as good as it is now because now it's soft. I would love to know what the chemical characteristics are

26:20with the idea that maybe you could reproduce those without actually having to go through 40 years of bottle age. But I have mixed feelings about that because on the other hand, maybe we shouldn't know that. Where's the magic? Where's the mystery? I like that. Susan, what do you think? Oh, Orly, that was so good. That was so good. I'm also leaning towards, you know, the magic and mystery, but I write about this a lot with the whole music thing because we try to put so much data on everything because we want to know and we want to, you know,

26:51have some return on investment for things, but, oh, that's long lived the mystery. I think for me, this is going to sound prosaic, but I think it's, you know, people always tell me, I went to visit such and such winery and I had such a great time and the wines tasted fantastic and I bought a case and I brought them home and they don't taste so good anymore.

27:12And, you know, again, bringing this back to the concept of psychological concept. It's the doesn't travel. It's the doesn't travel critique. No, but, you know, there is, I really think there is something to do with that psychological concept of priming. You know, the environment in which, again, this dovetails with everything that I do with, you know, the presentation of music and wine. Well, don't you think Retsina is a perfect example of that? Retsina, which always tastes great in Greece. I hate that when it's in the United States. That's an extreme example.

27:43I love, I love these. Shally, what's yours? I've never get to, I never ask you the magic wand question, but now I want to know. No, actually, I would have the same answer, right? Because I want to say that, Susan, when you mentioned about, you know, buy the wine in the winery, for me, the experience has been, and I've done this multiple times, as a statistician, I can confirm there's something real here, is in the restaurant, you know, I taste something great and I will order right away on the phone and then a few days later,

28:13once I come back and I've opened up all of myself, say, why did I spend the money to buy this wine? Well, I knew why, because I was in an environment with great friends, right? Who do you drink with is incredibly important. And I think, you know, the wine is really this whole sensory experience. Yeah, and add some of your favorite music, too. I mean, that's, that'll make it even more. It's always my goal to bring music into the wine tasting experience and also bring wine to the performing arts. You know, Susan,

28:44our group has never tried to do that, have music during the event. Maybe I'll try to break that, try to find an excuse to make a change in that. Yeah, let me know if you want to help designing any of this or keep me updated. I'd be happy to help.

29:04Thank you to our guests, Orly Clark Ashenfelter and Susan Lynn. Thank you for listening to this month's episode of the Harvard Data Science Review Podcast. To stay updated with all things HDSR, you can visit our website at hdsr.mitpress.mit.eu or follow us on Twitter and Instagram at the HDSR. A special thanks to our executive producer, Rebecca McLeod, and producers Tina Tobey-Mack and Aaron Kieswetter. If you liked this episode, please leave us a review on Spotify, Apple, or wherever you get your podcasts. This has been the Harvard Data Science Review.

29:35Everything data science and data science for everyone.