S3 #47 Does sign language change your brain? Brain-to-brain with Dr. Karen Emmorey.

0:00Hi and welcome to Kaleidoscience. Here you find answers about cognition that you may or may not have asked yourself. This episode is hosted by Sophie Kühne and Elisa Palme. So sit back, relax and enjoy this week's episode. Language plays a big part in our day-to-day communication. When you hear this podcast, you hear us speaking English, while Elisa and I, for example, also speak German for a big part of our day-to-day life. Both of these languages have a big commonality, namely that they are spoken languages.

0:34Today, we will learn about different languages in a different language modality. In this episode, we will talk about sign languages. Today's guest is Professor Karen Emery from the School of Speech, Language and Hearing Sciences at San Diego State University and the Director of the Laboratory for Language and Cognitive Neuroscience. She received her doctorate in linguistics in 1987 from the University of California, Los Angeles, and her research focuses on sign language and what it can reveal about the nature of human language, cognition and the brain.

1:10Thank you so much for being here. My pleasure. Thank you for having me. And as always, we play our short get-to-know game where I will give you five sentence beginnings and ask you to finish them as spontaneously as you can. Our first sentence is, As a kid, I always wanted to be... A teacher. Did you have a specific subject in mind? Actually, when I was a kid, I wanted to be an elementary school teacher. I think I just liked my teachers, I liked school, and I wanted to be like them.

1:42Cute. Very nice.

1:45Our second sentence. If I was an emoji, I would be... A thumbs up. That's a basic one. That's good. I'm positive. I tend to be a positive person and thumbs up is kind of a good emoji for me. Yeah. Our next sentence. My favorite thing to do on a day off is... To get outside, to go for a walk or to go running, because most of my day is spent inside, working, so when there's more time off, I like to be outside. Do you have any favorite places to go, or is it just being outside?

2:19Well, I'm in California, so I do like the beach. I like to go to the beach. I live close to the beach, so I can go there often. Very nice.

2:28Our next sentence. Right now, I'm most fascinated by... Ooh, that's a hard one, because I'm fascinated by lots of things, but I would say right now, reading. I'm really fascinated by deaf readers and how they're different from hearing readers. I think we might touch on that as well. We had some questions regarding that. And our last sentence. I know it's time to call it a day when... Oh. I think just when my brain is wiped. It's time to stop.

2:59Yeah. That's a very common answer. Yeah.

3:03I think at some point, we might have to change that question, because I think it's the answer we get most often. Yeah.

3:13So, Sophie already mentioned that you are now a researcher on sign language. And you mentioned that you at the beginning wanted to... Or a kid wanted to be a teacher. How did you get from... Or how did you get to being a researcher on sign language? So, as you mentioned, my degree is in linguistics. And I didn't know anything about sign languages at all when I was a graduate student. I was working... I was interested in the relation between language and cognition and how language is comprehended.

3:44And I did a postdoc with Ursula Belugi at the Salk Institute, where the idea was to combine my expertise in psycholinguistics and her expertise in sign language. She was a pioneer in understanding the nature of sign languages. And I started working there. And I didn't know sign, but I started learning American Sign Language in the lab. There were deaf people that were working there. I learned signing from them. At that time, there were no American Sign Language classes at the university.

4:19There were some interpreter training programs at community colleges. And I took some classes there. And so, I started to really learn ASL and became really fascinated by how many interesting questions you could ask by studying sign languages. Why is language the way it is? Is it because it's spoken? Is it because it's heard? Do we see the same things in sign languages? We could really get a good view about what's universal to all human languages by studying sign language.

4:51You can ask really interesting questions about language in the brain by looking at sign language. So, I really got into studying sign languages through the science rather than knowing a deaf person or having a deaf relative. And so, I stayed on at the Salk for a number of years doing that kind of research and then started my lab at San Diego State in 2005. So, could you explain a little for the listeners who might not know anything about sign languages a little bit what sign language is?

5:23So, I'll start by debunking some common myths, which is that sign language is universal. So, I said American sign language specifically because that is the sign language that's used in the U.S. That sign language is completely different from British sign language, which is used in the U.K. Even though both countries use the same spoken language, English, the sign languages are completely different. The sign language used in Germany, Deutsche Gewehrdensprache, DGS, is quite different from American sign language.

5:56So, just like spoken languages vary across the world, the same is true for sign languages. Another interesting fact that may be surprising is that sign languages have the same type of structures that you see in spoken language. So, even at the level of phonology, which we think of as the sound patterns for spoken language, we see similar structure in sign language, where, of course, the units aren't consonants and vowels or sounds, but they're hand shapes, locations, movements that can be combined to create signs or words.

6:32We also see the same expressive power. So, sometimes people will think that sign languages are more concrete, you can't talk about abstract things, but, of course, the same expressive power you see in spoken languages, you see in sign languages. So, this is why when you're watching the TV or a show, you see an interpreter there, they're interpreting everything that the person is speaking. So, you can say anything, anything you can say in a spoken language, you can say in a sign language.

7:03Another myth is that sign languages are really just spoken languages on the hands. So, ASL is just really kind of English transferred to the hands. But, in fact, American Sign Language has a very different structure than spoken English. You can move words around more flexibly in ASL than you can in English. There are certain ways of inflecting a verb to convey information about the time something happened that you can't do in English.

7:37I can say the same thing about other sign languages. So, in the Netherlands, the word order of Dutch is subject-verb-object for sign language of the Netherlands. It's a verb-final language, so just the basic word order is different. So, this tells you that sign languages really emerge independently from the structure of the surrounding spoken language. Okay. And for signs themselves as part of the sentence, like what qualifies as a sign or how do you define a sign?

8:17So, a sign is basically defined the same way as a word. So, you can find within a sentence, you can find the lexical units, the word meaning units, just like you can for spoken language. So, you can use sign and word kind of interchangeably, and then you can create new signs in the same way you can create new words. You can create compounds, for example, by taking two signs, putting them together to create a new word, like bedroom in English.

8:48You can do the same kind of process in sign languages. So, you can create new words in the same way. One thing that's different is that sign languages tend to have more simultaneous structure. So, in spoken languages, you typically create new words by adding prefixes or adding suffixes to create a new word. So, you can have something like slow, you add ly slowly. So, it goes from, you know, an adverb to, or an adjective to an adverb.

9:21In sign languages, you can have these suffixes and prefixes too, but it's more typical to change the movement or to add a facial expression to indicate additional information so that it's not a linear string of meaning units. So, you can do something like, so you can do something like, the difference between a noun and a verb in ASL is the nature of the movement. So, if I have the sign sit where I have one hand making a single movement down, if I make it two movements, it becomes chair.

9:58So, you have one movement indicates the verb, two short movements indicates the noun. So, there isn't a suffix that you add to make something a noun, it's the change in the movement. And this tends to be a pattern you see across sign languages, is more of this simultaneous movement changes to create new words as opposed to adding suffixes or prefixes. You can also add a facial expression. So, in ASL, I can purse my lips a bit and do that at the same time as I do a manual sign like to work or to run.

10:38And when I do that, when I add that facial expression, that purse lips, it means easily without effort. So, I'm working easily or I'm running easily with that facial expression. And again, that's a simultaneous expression of the two meeting units, the ease of effort and the verb. So, you get, and that also is something that's relatively common across sign languages, is to have linguistic facial expressions that can be produced at the same time as manual verbs or other manual signs.

11:12That's really cool. That's really cool. And just to clarify this, so signs are not gestures, but how do they differ from gestures in spoken language? So, that's a great question because many people think that sign languages are just kind of pantomimes, right? But there's systematic differences between pantomimes and signs. So, one difference has to do with where in space signs can occur.

11:47So, signs are typically restricted to what's called signing space, which is basically from about the forehead to sort of the torso. Whereas pantomimes can use the whole body. So, I can pantomime an airplane by extending my arms and moving them to pretend like I'm flying, or I can indicate running by actually showing you running. Those are not possible signs. Those would be ungrammatical in any sign language.

12:17Pantomimes tend to be idiosyncratic. This is why the game charades works, because different people are good or bad at doing pantomiming information, and people do it differently. But, of course, signs are conventionalized. So, the community, this is the sign we have for run, or this is the sign we have for mother or father. It's conventionalized and can be listed in a dictionary, for example. That's not the case for pantomimes.

12:48Other differences. It also brings a really interesting question of gestures, and whether signers gesture. So, speakers, of course, gesture. Even if they think they don't, they do. There's lots of research showing that, you know, different cultures may vary in how much they gesture, but everybody gestures. And it adds communicative information. I mean, there's a whole field studying the nature of co-speech gesture, which co-occurs with speech. So, what about signs?

13:19Because, of course, signs, you are using your body already for the language, for the linguistic part. But it turns out that just like speakers who produce co-speech gestures, signers can produce what we've called co-sign gestures. So, to give you an example, there's an ASL sign that means dance that's produced with sort of two fingers that look like legs moving back and forth.

13:49And that's the sign for dance, the verb dance. Now, when telling a story, I can add a body movement. Like, I can sway my body to indicate something like a waltz, for example. So, if you're telling the Cinderella story, I can embellish that with this body gesture to show a waltzing movement along with the ASL sign for dance. So, that's a co-sign gesture. You can also gesture with your face to sort of convey different information. Speakers do that, too.

14:21And so, you have this nice parallel between the use of co-speech gestures as well. And you see in co-sign gestures, particularly in storytelling. It's true for both types of languages. That's very cool. So, I maybe have a really, really, not stupid, but basic question. Do we know how sign language evolved in comparison to spoken language? So, that's another great question.

14:50There's, what's really interesting about studying sign languages is that they're relatively new languages. So, there's a number of different research groups who are studying the emergence of a new sign language. And this is because of the different environmental and genetic factors that come into language with deaf people. So, with spoken language, you can sort of trace the history back of different spoken languages. But we really rarely see a new spoken language.

15:24Even if you discover something, some people in the Amazon jungle, for example, it's likely they've still been using that language for many, many years. You don't really know. But we can actually document the birth of a language for sign languages. So, there's a group that studies Nicaraguan sign language, which is a new language, where they were able to document when that language started to emerge. So, there was a deaf school that was set up in Managua.

15:56Before that, deaf people were isolated in their homes in the countryside. When this new school was set up, deaf people came together and started interacting with each other. Now, it was an oral school. They were teaching spoken Spanish. But they did not forbid signing. They allowed the kids to gesture with each other on the playground, on the buses. And the teachers noticed that they were using this gesture to communicate, and they wanted to understand it. So, they actually contacted some linguists, Judy Cagle and Annie Singhas, from the States to come and start studying this new language.

16:34And so, they've really been showing how, from this early period in the 1970s, early 1980s, how that language has changed over time. And you can really show who's changing the language, how is it changing. You can really understand how language emerges. So, these sign languages are created de novo from a community of deaf individuals, both children and adults. So, that's very different from some systems that have been invented for sign education.

17:10So, there's something called manually coded English or sign-supported English. You see these in other countries as well, where the signs are really based on the spoken language. And they were invented by a committee. And it turns out they often violate aspects that you see in natural sign languages. So, natural sign languages tend to use space. Space isn't part of a spoken language. It's not used in these invented systems. So, there's an important distinction between invented sign systems and naturally occurring sign languages.

17:47So, we can really, sign languages can emerge in different ways. But typically, or often, it's from the establishment of a school or a church or gathering place that deaf people come together. And then, they interact with each other and a language emerges. That's really cool. That's fascinating. And, for example, in German, there are a lot of dialects. And in English, if you compare American English with British English, it sounds kind of different, but it's still the same language.

18:23Are there also dialects in sign language? Yep. There's dialects. So, in the U.S., we can look, there's a number of sociolinguistic studies looking at language variation. So, again, changes in the phonology that is the form of signs that you might see for signers in the south or signers in the north. There's also lexical or word choice variation.

18:53So, there's a wonderful study in Philadelphia looking at Philadelphia signs. And this is trying to preserve them. One of the ways that dialects and, you know, lexical differences emerge is through these different schools that are set up in different places in the country. So, in Philadelphia, there was a school for the deaf that has been there for, you know, since the 1800s. And they developed kind of their own signs for that area for, like, months of the year, for example.

19:29They had different signs than standard ASL. And part, it's from that school. And so, you get this difference that you can trace back to the schools. This is something you see in other sign languages as well. So, in the U.K., there's a lot of dialectical variation that is related to the region. So, regions in the north may use different signs for colors, for example, than in the south. In Italy, also, because there's lots of different schools, you get different dialects that emerge.

20:02And they can be quite different across the country. So, you get dialects in both spoken languages and sign languages. That's really cool.

20:13Also, related to the dialects, in spoken language, sometimes language slips where you turn. Or change to parts of a word or of two following words happen. Or where you stumble across syllables for some reason happen. Does the same also happen in sign language? And why and how does that happen? Yeah. So, we often call that slips of the tongue, right? So, you make an error. You swap one sound for another, for example. What that tells us is that mentally, you have representations of those sounds that can then be swapped or misselected.

20:51So, you can have the same thing for signs. So, when people are signing, they make mistakes. And there are not random mistakes. So, what you can see is maybe a substitution of one handshape for another. You sort of anticipate a hand shape. And so, you produce it incorrectly in the sign too early. There's actually been a nice group that has done work on this in Germany, in German sign language. So, just to give you an example, the person wanted to say their parents, which would have been a B, sort of flat hand.

21:27And then this kind of like, you call it a Y hand shape. I'm not sure if you can get that on the radio, but a Y hand shape that the two hands touch together to mean parents or eltern. And the anticipation was, instead of a B hand shape, they produced this same Y hand shape, anticipating the hand shape for the next sign. You can get other examples where you swap the location of two signs or the movement of two signs or you anticipate that. Now, why that's important is because it tells us that signs are not stored mentally as holistic gestures that have no parts to them.

22:02They're not stored as pantomimes, right? They're stored as phonological units of hand shape, location, and movement. And just like speakers who have these phonological representations that have to be assembled during production, and you can have something go wrong in your assembly process, and then you make a slip of the tongue, make a misselection of a sound. The same thing happens for sign languages. So, as you're speaking, you have a stored representation of the hand shapes, locations, and movements.

22:35And as you assemble those into a word, you can make a mistake and select the wrong hand shape or the wrong location. So, that means that what the linguists are describing, in terms of describing, okay, I can divide signs into these different parameters or these different phonological units, is actually real in the minds of signers. And the evidence for that is that they can be misselected. They're not just stored holistically without any internal structure.

23:06That's really cool. We have one more question regarding the spelling and sign language. So, in our research, we've come across, like, that you also need to spell things from time to time. So, how does that work? So, I think you're talking about fingerspelling. Yes, fingerspelling. So, not all sign languages have a fingerspelling system. There are some sign languages that don't.

23:36But American Sign Language and a lot of the sign languages in Europe and in the UK have a fingerspelling system. What this means is that there are hand shapes that represent the letters, if it's an alphabetic system, of the surrounding spoken language. So, in American Sign Language, it's a one-handed system. So, there are specific hand shapes for A, B, C, D, for all of the letters in the alphabet.

24:07British Sign Language has a different fingerspelling system. It's a two-handed system. So, you use the two hands to create signs for the letters. And then you, for both languages, you combine those hand shapes in the same order of the letters in the word that you want to spell. So, it's a way of borrowing from the surrounding spoken language. And fingerspelling is used in lots of different environments.

24:39So, it's often used for proper names or for things where there aren't signs for those proper names. But it's used for lots of different other functions as well. Fingerspelling can be used for emphasis. So, even though there's a sign for something, I may then fingerspell it to really emphasize what I'm talking about. It can be used to link a sign to what the English translation is of that sign, if I really want to emphasize what the English word is for that.

25:11And some fingerspelled words can change their form and become signs. It's, again, a way of borrowing. So, you can have a fingerspelled form like no. So, you have the letters N and O. Over time, the form has changed. So, it doesn't quite look like the letters N and O. It looks more like a sign where the fingers just close. And that can then be part of the grammar so that I can then direct it towards you.

25:41And I can say no to you. And you can say no to me if I change the orientation. Just like verbs work like that in sign languages. So, fingerspelling is a really interesting domain. There's no, I don't think there's any parallel. In spoken language, bilingualism, there isn't anything like spelling out loud, which is a little bit like what fingerspelling is. Fingerspelling also is turning out to be really important for literacy. So, if you look at deaf kids as well as deaf adults, the kids who are really good fingerspellers also tend to be good readers.

26:17And it's a way to link print to sign representations through both fingerspelling and then also through signs, which seems to support literacy and reading in deaf kids. It does make sense because you have to kind of read each part of the word and spell it with your hands. It's like maybe a bit similar to reading out loud when you are a speaking person, because if you read out loud, you usually also or when you're good at reading out loud, you usually also are a better reader compared to a person who is really struggling with that.

26:50And the interesting thing is that Carol Padden, who is a researcher who studied fingerspelling, has this nice quote where she says that deaf kids learn fingerspelling twice. And this is because early on, before, you know, when they're little, even babies, deaf caregivers will fingerspell to their children. It's part of the language. So, even though they're little, they'll still fingerspell to them. And deaf kids who are, you know, exposed to sign in this way, they know how fingerspelling is used.

27:24So, they know it's used to introduce someone's name, for example. And they know that it's with different handshapes and they'll play with those handshapes. And so, they're learning fingerspelling as part of the sign language. But then they learn that, oh, there's a right way to fingerspell and there is a link between these handshapes and the printed words as they start learning to read. And then they have to learn to break down the fingerspelling into letters. And that is another reason why it might help deaf kids learn to read is one of the things you have to do is break words into their parts, into their letters.

28:00And with fingerspelling, you're doing the same thing. And so, that parallel can help kids realize that words need to be broken up into their letters, just like fingerspelling is broken up into handshapes. That makes sense. So, you also just touched on, like, names being spelled. And I learned that in at least German sign language, people get, like, given names, which is usually, like, a descriptive sign.

28:33Is it the same in ASL? And, like, how do other sign languages do it? Yeah, it's true in ASL. And they're called name signs. And they can be descriptors. And I think different languages and cultures vary with respect to whether they're going to give you a name sign that is something about your physical appearance, like, like, curly hair, and you do a curly hair gesture, or eyelashes, and you do an eyelash gesture for a name sign.

29:09Other times, particularly in ASL, it can be initials. So, my name sign is the letter K and the letter E. So, KE is my name sign. I have another friend, David Carina. His name sign is DC. So, that's another common way. And typically, culturally, you are given a name sign by the deaf community. You don't create your own name sign. So, it can be given to you by deaf or hearing sign parents or the community.

29:42We already touched slightly sometimes on deaf kids learning language. So, especially deaf at birth people. How do they do that? And how far is that different from hearing people learning language or speaking? So, I don't personally study language acquisition. But I can tell you that the basic milestones of language acquisition is very parallel for kids exposed to a sign language versus kids exposed to a spoken language.

30:14So, you have babbling. So, spoken language, you'll have little infants, you know, nine months old going bah, bah, bah, bah, bah, in sign. There's manual babbling where kids will be moving their hands in this rhythmic way. And it's very different than how hearing children move their hands. It's clearly a linguistic effort to produce signs. It's really cute. And then there's first signs and first words come in at about the same time.

30:50You see the same two-word or two-sign combinations. And so, it's very, very similar acquisition. That's very interesting. And you already touched upon, like, that the children then also learn to read later and that that can be difficult. I mean, it makes sense that reading is challenging because it's a different language. But are there also other reasons why it's challenging for deaf children to pick up reading?

31:28So, it's challenging in part because you can't hear the language that's encoded in print, right? So, that makes it a challenge to learn the system. And I don't do work with children, as I said, but we are studying deaf adults who are skilled readers but who don't have – who have weak phonological skills. If you give them a test about trying to match sounds to letters, for example, where the letters are different, it can be difficult for them.

32:03Like, the word kite and cat have different letters, but the same sound. The deaf readers may be more distracted by – they may think they're different because they have different letters. The key, though, is that despite these differences in their phonological knowledge, they're skilled readers. So, they can read just as well as hearing readers. And what we're trying to figure out is how do they do that when they don't have a strong phonology? And what we're finding is that they do it quite differently.

32:37They use different brain areas. And they focus more on vocabulary size is a good predictor of reading skill. Spelling, orthographic knowledge is a good predictor of reading skill. Phonology is not. So, it's not the case that those adult deaf readers who have great phonology are better readers. It's not predictive. It is for hearing people. It's for sure predictive for hearing kids and hearing adults. But it's not for deaf adults.

33:07And we're also discovering that when deaf people read, they actually have a superpower, which is they have a wider span in which they can take in information. So, when we read, we look at words, right? But we also take in information to the left and to the right of where we're looking.

33:37So, we're processing not only the word we're looking at, but we've got information coming from either side of where we're looking, the left and the right. We've recently shown that deaf readers have a larger left word span. And other work has also shown that they have a wider right word span. So, they get more information. And we think that's actually because they're signers, because we've run a group of hearing people who have deaf parents who got exposed to American Sign Language before they learned to read.

34:08They also have this wider left word span, suggesting that it's not deafness, it's actually sign language that's increasing this span. The other thing we find, maybe because of these larger spans, maybe because they're not engaging phonology, but deaf readers are more efficient in reading. They read faster than hearing people without any loss of comprehension. They understand just as well as the hearing people. They just read the sentences faster. They do that by skipping more words.

34:39So, they skip over words more often, maybe because they're getting the information in their larger span.

34:46And they don't reread as much. They don't regress and reread text as much as hearing readers do. Again, maybe because they've got more information in this wider span. So, we're trying to really understand exactly how this is happening and what they're doing. But the key is that a skilled deaf reader is doing something different than a skilled hearing reader. So, if your goal is to have skilled deaf readers, it may not necessarily be, you may not want to use the same techniques that you use for hearing readers compared to deaf readers.

35:23I think that's super interesting. It kind of makes sense for me on an intuitive level of when I'm reading, I'm also producing a voice in my head, which is reading it out loud to me. And that, of course, takes more time because this voice has to read the words I'm reading. And if you don't have this inner voice because you can't hear at all, then you don't have this time delay, I would assume. Yeah, we actually don't know that. That's our hypothesis, but we're working on trying to test that idea.

35:55This also brings me to my next question because I was always thinking of or wondering how deaf people think. Because I'm a quite verbal thinker, so I always think in conversations, sometimes in pictures, but usually in conversations.

36:10Do people think in science? Do they more think visually? Is it totally different? So that's another great question that we don't really know the answer to. And I'm quite interested in that question because I'm trying to understand how perception and production are linked in language processing. So for spoken language, when I speak, I can hear myself speak, right? So, and I use that auditory feedback. I can use that to adjust my speech because I'm hearing what I'm saying and, oh, I didn't say it quite right.

36:45I can change it. And children, of course, learn to speak using that type of information. For signers, you don't see yourself sign.

36:57So when I comprehend speech, I mean, comprehend sign, I'm watching you, your face, your hands. When I sign myself, I don't see my own face. I see a different view of my hands. And for speakers, language comprehension can use the same input for understanding another person and understanding yourself. The voices are a little bit different, but the language is the same. But people can't really understand signing from their own viewpoint.

37:29So there's a different relationship between comprehension and production. And this visual feedback is not really used in the same way that auditory feedback is. And that links to sort of what's the internal voice for a signer, right? You can easily imagine for speech. It's just I hear myself talk. I can imagine my own speech.

37:59For signers, are they imagining what it feels like for them to sign? Sort of the motor properties of signing. If you have to practice, imagine giving a talk in your head. Is it how it feels to give those, to sign or to give that talk? Or do you imagine yourself in a mirror and you see yourself from afar, how others would see you? Right now, the data is suggesting that signers probably feel their signs and the movements rather than imagine seeing themselves.

38:33But we don't have solid data on exactly what that inner signing is like. It may be a combination of both, of sort of feeling and your signs as well as sort of seeing what they might look like. I think that's really, really interesting. So you already mentioned that reading for signers is different, but like in the brain areas, but is signing in the same brain areas as speaking?

39:10So is language production similar for signers as for speakers? So the answer is pretty much yes and no. So the core language regions, typically in the left hemisphere, are engaged both for sign production and speech production, as well as sign comprehension and speech comprehension. Which is kind of interesting because the regions of the brain that are known to be involved in spoken language comprehension are near auditory regions, which kind of makes sense.

39:43But those same regions are also engaged during sign language comprehension, which is, of course, visual. And the visual area of the brain is in the back of the brain and it's in a different location. And yet that same cortex is involved in sign language comprehension. So that suggests that there's something about that region of the brain that really cares about language, regardless of the input or output modalities. There are differences, in part because of the nature of the system. So you're going to see more parietal involvement in sign production than speech production, because those are regions that are involved in directing the hands towards locations on the body.

40:22Speech doesn't involve those regions typically. So you see those differences between sign and spoken language. There's also some evidence suggesting that you may see more right hemisphere engagement for sign comprehension, which may be due to the spatial aspects of sign, particularly for narratives where you're telling stories and using a lot of signing space, a lot of movement in space. But the jury is still out on what the right hemisphere is doing for sign language comprehension.

40:57And does that also kind of change? Like, so when the language is very visual, is there like an improvement kind of in like visual cortex? So we do see differences in the anatomy of visual cortex. So one of our studies has shown a larger calcorine fissure cortex for for signers compared to speakers.

41:30There's some new ish evidence suggesting that visual cortex may actually be involved in some aspects of sign processing. Exactly what those are, we don't know, we also see differences or enhancements in certain visual spatial abilities in signers, so both hearing and deaf signers who are lifelong signers, you see differences in their mental imagery abilities, they tend to be better or faster at generating and mental images or manipulating mental images like mental rotation.

42:06We think these are related to sign language comprehension, we think these are related to sign language comprehension, we think these are linked to some of the processes that are required for sign language comprehension, and that that extends then to these non-linguistic domains. I think this also kind of nicely ties into our next question, so could that be seen, so there is a lot of debate about cognitive advantages in bilingualism. I feel like that's a big debate that comes up a lot when you start looking into linguistics and bilingualism.

42:41So there are different forms of bilingualism for people who can sign, right? There can be people who know a sign language and a spoken language, but there can also be people who know multiple sign languages.

43:03Is this bilingualism in some way different to somebody who speaks two languages? So there's a lot in that question. Sorry, I also, I went in and... So, yeah, so there's, we call it bimodal bilingual, so you know a signed and a spoken language, and we can contrast that with unimodal bilingual, so you know two spoken languages, or maybe you know two sign languages.

43:33There's much less work on people who know two sign languages, there's one paper that just came out looking at switching between two sign languages, and it turns out that that's very similar to how you switch between two spoken languages. For bimodal bilinguals, the cool thing about this group is that they don't have to switch between signing and speaking. You can switch between your two languages, but you can also do what we have called code blending, which means you can produce a sign and a word at the same time.

44:07So I can say the word cat, and I can produce the sign cat at the same time. Now, I can't do that. I can't say cat and katze in German. I have one tongue. You just can't do that. I have to switch between the two. So there's this unique ability to produce elements of your two languages at the same time. And so that can maybe have consequences for language control. It has consequences for understanding the cognitive demands of language mixing.

44:44So one thing we found is that code blending, producing a word and a sign at the same time, is not hard. And in fact, if you look at how bimodal bilinguals mix their languages, they way more prefer to code blend than to switch between signing and speaking. So if you just look at how they mix their languages, they tend to code blend. And when we've done some psycholinguistic studies to see are there any response time or error rate penalties for producing a code blend, we find that there aren't.

45:21In fact, it's sometimes easier, certainly, to comprehend a sign and a word at the same time and to produce. Even though it's a dual task, I'm doing two things. I have to retrieve two lexical items from my mental dictionaries. And yet, there's either no cost or it can help a little bit. It can help the weaker language because the stronger language, which usually for hearing bimodal bilinguals is the spoken language English. Retrieving the English helps you get the ASL as opposed to just doing the ASL by itself.

45:52So there are these benefits to code blending that are unique to that population. For hearing unimodal bilinguals switching between their languages, there's a well-known cost to switching. It's just, you know, a few milliseconds. It's not a lot, which is – and that cost is reduced if you're in natural conversations as opposed to in an experiment. But it's still there. And so one hypothesis has been that, well, maybe because code blending isn't as costly as switching,

46:25that we don't see the same advantages that have been claimed for unimodal bilinguals, where the idea is that because they mix their languages all the time, they have to control their two languages. That can benefit certain types of executive function, attention, orientation, inhibition. This is where a lot of the controversy is, is whether these effects are real. For bimodal bilinguals, we have looked at this. If you'd asked me this question last year, I would have said there's no evidence for a bimodal bilingual cognitive advantage.

47:01But there's a study that just came out that was – had lots of participants and found evidence that bimodal bilinguals also show some advantages in these cognitive control domains compared to monolingual speakers. And one reason why you might see that is we have found, in terms of the mixing literature, that even though if you're just looking at a code blend by itself, in natural language, you're switching into a code blend.

47:34You're speaking, and then you produce a code blend, and then you switch back to speaking. And it turns out that switching back to speaking – so if I'm speaking, and then I produce a word and a sign, and then I stop the sign part, and I go back to speaking – that switch where I have to turn off, I have to sort of suppress ASL to switch back to English, that has a cost. And maybe that aspect of language mixing carries over into these other cognitive domains.

48:05But again, this is a big controversy within the field, as you know. So I'm just kind of telling you what we found. But there's a lot more work that needs to be done to figure out the mechanisms for these differences that we see. And when individuals, like, code blend? So what is very hard to imagine for me is kind of the grammar they're sticking to. Because it's easy for me to imagine with, like, a noun to have both.

48:35But is there also code blending in, like, for example, word order?

48:42So that's a really important question. So one thing that's clear is you code blend elements of your two languages. So it's very hard to produce grammatically correct ASL and grammatically correct English continuously. Because they're two languages and they have different syntactic properties, right? Different syntactic rules. So to do that is almost like a parlor trick, and you have to be trained to really be able to do that.

49:13It's really difficult. So just naturally mixing your languages, we do exactly kind of what you suggested is I might produce a noun or a verb or a short phrase as a code blend, but not an entire sentence.

49:29One thing I'm just thinking about, because you said that you could, for example, say a word and sign it at the same time. Is it to some degree possible to speak a sentence and sign it at the same time? Because when they are differently structured, it must be quite difficult to get the different structures correct at the same time while producing both languages correctly. Yeah. So what we have found is that typically you, what I've said, you pick a syntax. Most often it's English, and then you code blend the ASL along.

50:03So the English structure is fine, and you just code blend ASL on occasional words. But you can have it the other way around. It's less common. It's where ASL is what we call the matrix language or the base language. So you're signing ASL, and I'll occasionally then add some English to the ASL, and it's a code blend. And so, and that typically occurs for short segments, and the English is going to sound weird because it's following the ASL.

50:38So an example, there's a phrase in ASL where you can say something like, and then what happened? And what you would sign is the sign happen, and then the sign what? And if you're code blending with that, it's going to say happen what, which is weird English, but it's following the ASL. So typically what we find is that you have sort of a matrix language that you choose, and then you code blend within that language. There is some interesting work in Italy looking at LIS, which is Italian sign language, and spoken Italian, because those languages have different word orders.

51:19And there are phrases where they're produced in a code blend, but the syntax goes in opposite directions. So you have, I have to think of an example, because I don't do this work myself, but the idea is that you have two different syntax structures that are being produced at the same time. And the linguists are arguing over how do you account for that? Is it two separate systems that produce the syntactic structure? Is it one syntactic structure that gets read out differently? How do we account for that?

51:49So there's interesting work to look at, okay, when the syntax is different for within a code blend, how do you explain that? That's really, really interesting. What I also wanted to pick back up on is that you said when individuals kind of code blend, the ASL can kind of help the English word, or like vice versa, because I'm, I'm doing this in English. My native language is German, so I'm obviously very familiar with code switching, and that this can sometimes take a moment and a bit of energy to be like, what's this in this language?

52:30And for me, it often feels like this is obviously very personal, that when I find the word in another language, it feels more like it kind of blocks the word in the language I'm looking for. And I find it very interesting that then in ASL and English, they can be kind of like linked together and actually make it easier to recover the information. Now, this is when you're talking with another bilingual, right?

53:04So it's a little bit different than what you're doing now, which is sitting on German, because I don't speak much of German. So if you switch to German or your audience doesn't speak German, that's not good. So you have to really get that English word, because you can't switch to German. If you were talking with, if the two of you were talking to each other, and you wanted to switch to German or switch to an English word, you could do that freely. So it's a little bit different than a natural code switching environment.

53:37So we talked about the fingerspelling earlier already, and the fingerspelling kind of links to, so the fingerspelling of ASL kind of links to written English. And is there kind of a writing system for ASL or any of the sign languages, or is that not really a thing?

54:07So there's no accepted writing system for ASL. And it's kind of interesting to think about why.

54:19And I think it has to do with the function of a writing system. So we certainly have systems that linguists use to try to transcribe the phonological form of signs, just like for spoken language linguists, they have the International Phonetic Alphabet, the IPA, that they can use to describe any spoken language that they come into contact with. We have similar methods for sign language that really it's just for linguists. For everyday people, there's been different attempts to try to create a writing system.

54:52The one that I think has been maybe the most successful is something called sign writing, which was actually developed here in San Diego with input from deaf people. And it's not used widely in the U.S. It's used a bit more in Europe and South America. But there isn't any accepted writing system that all deaf people use.

55:19And part of the reason, I think, is because deaf signers are bilingual. They have to learn to read the language of their community, which is difficult, right? I mean, it's difficult for hearing kids. It takes years to learn to read. So when you already have a way, if you're bilingual and you have a way of writing information, you can use that instead of having to spend years learning how to write a sign language. And in today's community, we also have video.

55:52We have video phones. We have video recordings that are much more accurate in capturing the information. So it captures the facial expression, the body movements, all of that. And so you can use that to communicate as opposed to writing. So I think it's an interesting question why there isn't a writing system that's accepted. And there may be other answers for that. But that's my opinion about why there isn't a writing system that's accepted by lots of deaf people. Okay.

56:23I imagine. So when you're a deaf child, for example, born into a hearing family with two hearing adults, and they don't know any sign language, how do children then develop sign language? Or is it common that the parents learn sign language? It's becoming more and more common for hearing parents to learn sign language, which is a really good thing because it protects children against what's called language deprivation.

56:58So when a deaf child is born into a hearing family that doesn't know sign, typically there's hopefully early intervention with either a cochlear implant or with a hearing aid. So you get hearing intervention. But it's not always clear that those interventions will allow for successful acquisition of speech. And if they're not exposed to a sign language, then they're not getting any language input early on. And there's a lot of work showing that without early language input,

57:30you have not great outcomes for both language and social and cognitive aspects of your life. So language deprivation is a real thing. Sign language can overcome that because it's completely accessible to the child. So that you may not be able to hear speech, but you can see sign. And that then allows for communication. And the other good news is there's work that's coming out showing that parents who are just starting and learning sign,

58:04just as their child is learning sign, if they have early and consistent input, even though they're not native deaf signers who are super fluent in ASL, their child is still benefiting from that. So when you look at those children who are at two years, they have about the same vocabulary as deaf children who have deaf parents. So it doesn't matter that the input doesn't have to be perfect. It just needs to be, there needs to be enough. It's not quite clear what enough is yet.

58:34And early. And then you see that these deaf-signing kids, even though they've got hearing parents who are learning sign, are still acquiring sign quite well and overcoming the risk of not having early language communication. So I think the important thing is for physicians who sometimes advocate, particularly if you're getting a cochlear implant, oh, don't sign with the child. You want them to exercise their sound or use their ears. But in fact, there's no evidence.

59:05In fact, there's clear evidence that having sign early on can help, and it certainly doesn't hurt the child. And not having sign does have dangers in terms of social, cognitive, and language development. So, and we know from spoken language bilinguals that sometimes physicians will tell parents, oh, just use one language if your child is struggling. But again, that's not the case. If you have two languages, children do just fine and they become bilingual.

59:38Bilingualism isn't bad for children. That's true for deaf children too. Yeah, I think this is a really important point that should also be emphasized with the history of sign language. It's a bit funny because for me it was always clear that when a child is born not hearing into a hearing family, that of course everyone needs to learn sign language. And it was never a thing I thought about, even as a kid who had never got in touch with deaf people. So just the thought of parents not learning it and the child being cut off from communication sounds so cruel to me.

1:00:15Yeah, I was quite shocked when I started learning German sign language a little bit. My teacher was in her 60s, I think, and she still said that she didn't have resources and people around her signing with her. So that's, yeah, it's a good thing that people now change their mind on that and that it's emphasized that this is important.

1:00:44So at the end of our episodes, we always ask our guests to kind of bring our conversation back to the point where we started, which is going to sign. So which role does cognition play in your research on the research topic in general? So in general, I'm interested in the relationship between cognition and language, that kind of interface. So evidence that knowing a sign language can affect cognition in certain ways is really quite interesting

1:01:19because it tells us that that's where language and cognition connect. So this notion of certain enhanced visual spatial abilities that you see in signers or this change in reading span that seems to be linked to sign language experience, those are all ways that cognition and language interface. The same questions about language mixing and the cognitive control that's needed for language mixing. How is that different for bimodal bilinguals compared to unimodal bilinguals? It's another example of how cognition comes into linguistic studies.

1:01:51And if somebody listened to this episode and can only remember one or two things, which ones would you like them to remember the most? That's hard because there's so many things I want people to know about sign languages. But mainly if they come away with some of the myths that I talked about dispelled, so that sign languages are not universal. You find them all over the world. They're acquired in the same ways as spoken languages.

1:02:22They use essentially the same areas of the brain for processing. We see lots of parallels between sign and spoken languages. So sign languages are not inferior languages. And thank you so much for talking to us. My pleasure. It's been fun. It was super fascinating for me. Thank you so much. This was Kaleidoscience. We hope that you enjoyed this episode and we would love to have your feedback. You can rate our podcast and give us feedback on our Instagram account. Have a great week and you'll hear from us again in two weeks.

1:02:54This episode was hosted by Sophie Kühne and Elisa Palme. Produced by Imogen Hüsing, Clara Kühne, Sophie Kühne, Sönke Lülf and Elisa Palme. The music is from Jan-Luca Schröder and the logo is from Annika Richter. Thank you.