Science Friday - How Signing Characters Help Deaf Children Learn Language

Episode Date: January 31, 2024

This radio interview is an abbreviated version of the full video interview, available with ASL interpretation on Youtube.Think back to your favorite childhood TV show—was it “Blue’s Clues”? ��Little Bear”? “Winnie the Pooh”? Animated TV shows are important for kids because they can teach them to read, draw, spell, and talk. Plus, the ways these shows tell stories and create colorful, fictitious worlds can broaden children’s knowledge and capacity to imagine.But children’s shows aren’t accessible to all deaf children, which means they could miss out on a common learning experience. Among other things, that can set kids back in learning both American Sign Language (ASL) and English language skills during their formative early childhood years.Melissa Malzkuhn is third-generation Deaf and the founder and director of the Motion Light Lab at Gallaudet University in Washington, D.C. Her lab is creating ASL-focused children’s media that is made by and for the Deaf community, using motion capture technology, avatars, animation, and signing storytellers. She talks with guest host Arielle Duhaime-Ross about ASL access in childhood, the science of learning, and how she’s creating “Here Comes Mavo!”—the first animated TV series with signing characters.Transcripts for this segment will be available the week after the show airs on sciencefriday.com. Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

In all the children's TV shows out there, there's very little, if any, deaf representation. There's no TV show or series. There's no real central core of access for deaf kids in children's media. It's Wednesday, January 31st, but as you well know by now, it is also Science Friday. I'm sci-fi producer Rasha Arredi. A team at Gallaud at University in Washington, D.C., is incorporating language research, animation, motion capture technology, and storytelling to create the world's first animated TV show with signing characters. The goal is to help deaf kids develop their literacy skills in English and American Sign Language, ASL, while being entertained by fun, colorful characters and stories.

Here's guest host Ariel Dumas. Animated TV shows are so important for kids because they can teach them to read, draw, spell, talk. and the creative ways in which shows tell stories, where they whisk kids away to some colorful imaginary world, that contributes to the learning. But shows like that aren't accessible to every child, like deaf kids and children who are blind, for example, which could, in turn, affect how these kids learn language

during those sensitive and formative early childhood years. My next guest is someone who's incorporating ASL, American Sign Language into children's media that's made for and by the deaf community. Melissa Mouskoon is the founder and director of the Motion Light Lab at Gallaudet University in Washington, D.C. Melissa is third-generation deaf and is answering my questions in her native ASL. The person you're hearing is an interpreter. Melissa, welcome to Science Friday. Thank you for having me.

I'm super happy to be here. Yeah, thanks for coming on the show. So first of all, tell me about ASL access in early childhood. Oh, that's a big question. So speaking very, very generally, all children, deaf, hearing, otherwise need language exposure to learn, to grow, to develop cognitively. Deaf children need visual language. And so for the most part, linguistically, that means a signed language.

And language acquisition, though, goes in the same way, whether it's spoken language or signed language, kids that are born into signing homes hit the same language developmental milestones that hearing kids born into speaking homes do. But now what happens when deaf kids don't have access to things on TV or media? There's plenty of good content out there. And a lot of people think that captions are the solution and that captions equal access for all. And of course, captions offer access to an extent. But we're talking about kids that aren't readers yet. Kids that are still learning language, kids that are still developing vocabulary.

At that age, captions aren't going to cut it. And what we need then is sign language representation in children's media. And there are such limited resources out there. There's no TV show or series. There's no real central core of access for deaf kids in children's media. I think that if we provide that, not only deaf kids will benefit, but hearing kids will as well. Right.

So what you're talking about is language deprivation. Yeah, it is certainly a consequence if a child doesn't get access to language or good, high-quality language or if they get such minimal access to language, that does result in language deprivation. So to back up a little bit, I don't think I shared this stat, but 95% of deaf people are born to hearing families. And that's a global statistic. So what that means is that deaf kids are born into an environment where their parents are navigating all kinds of questions about what it means to have a deaf kid, including the prospect of learning sign language.

I think that there are real gaps in the early intervention system, the systems that parents interact with when they first find out that their kids are deaf. So hearing parents will sometimes not learn sign language at all or not learn how to sign until their kid is maybe two or three. And think about what that means for, for a three-year-old to have barely had language exposure. So that results in language deprivation. And you see lifelong consequences of that.

They end up usually struggling to develop full literacy. And a lot of that can be traced back to what happens in early childhood. So that's where we're focusing our efforts. That's what we want to fix to make sure that deaf kids have the same opportunities and access to life that anyone else does. What does most childhood entertainment look like for kids who are deaf and hard of hearing? So that's another really good question. You do have some people that have put out ASL videos, videos in ASL.

So my lab, for example, we have bilingual storybook apps that are available in signed languages and written languages. And it means that kids have access to great language models, high quality. Parents can use those as resources to interact with your kids. But it's not like those storybook apps are going to substitute the full range of language exposure that a kid needs. On TV, I think a lot of deaf kids really primarily enjoy shows that are highly visual and that really are good for comedic value and entertainment value, but not educational value. So I think that what hearing kids get from shows like that are a lot more

choices. They have shows that are entertaining and educational. Parents can choose what show they want their kid to watch at different times. And with deaf kids, it's more like, well, whatever is visually appealing. And then they're filling in blanks and gaps. They're relying on behavioral information that they're seeing, the actions. But the meaning behind it, the labels for things, understanding the why isn't there. So there's a ton of guesswork on the part of deaf kids in the early years. And I don't want that to continue.

I want kids to understand what things really mean. I want them to understand the context. and really develop that knowledge base, start making those connections, because, again, it's a lot of filling in the blanks and making connections that might not always be accurate. So I don't want their early years just to be a guess and game. What about the neuroscience of it all? You know, is there a difference in how brains process language when it's seen versus heard? So it turns out that actually, and I can say this, because our lab is part of the visual language and visual learning center, which is a research center

our basic out at university. So they use various types of imaging to see what's going on in the brain and over and over they find that there actually is no difference. So the brain is actually just looking for patterns in language. The rhythms that are present in all language, regardless of the modality. So language can be spoken for hearing kids and the brain will take that auditory input and process it. And for kids that are exposed to a sign language, they'll take that visual language and process it in the same way. So it's really fascinating that the brain doesn't discriminate and the brain isn't choosing. It just wants those patterns. And so ultimately what it's about is the timing of it is getting that language exposure early. There really is a sensitive period.

And it's birth to three. So that's really, really early. But those couple years create such a foundation for the rest of our lives. And that foundation is what really allows for us to place all the rest of the building blocks that we get in life's upon. And I mean, you know this history so much better than I do as a third-generation deaf person in the U.S., but ASL was not always considered on par with other languages. And, you know, that research that you're talking about disproves that entirely. Exactly. There are so many misconceptions, misinterpretations, misunderstandings about signed languages.

I mean, even when my granddad was alive and when he was my age, Deaf people, of course, valued using sign language, and they knew that it was integral to their communication. But they grew up in a society where their language is just perceived as signs. And so even my granddad just said, we use signs. They didn't say sign language. There was definitely not any level of prestige or respect associated with the language. There was certainly a stigma to using a sign language. it wasn't seen as something that you could use to express any sort of intellect or education.

And that belief really was pervasive in the deaf community as well. It wasn't until the 1960s or 70s that linguists started to undertake research on what's now known as American Sign Language. And they realized, wait a minute, there is a grammatical structure here. And so they really flipped the whole conversation on its head to say, this is actually a language. then bilingual education, the bicultural movement began, where people believed that you had to have a firm understanding of ASL and English, and it really led to better literacy outcomes. And so we're now at a point where we certainly know better. And to add to that, there's about 300 documented sign languages in the world. Linguists are still doing this research on sign languages. Thanks for sharing that.

That really puts it into context. So you and your team started out making storybook apps. If I opened one of them, what would I see? So right, that's exactly how we got started. We designed storybook apps to help give a resource that would expose deaf kids and their families to sign language. And we wanted to help them own the reading experience. So we designed them to look like a traditional storybook. You see the words at the bottom of the screen, but then you can also hit the play button, and you'll see a narrator that's signing what is said on that page. So the kids can see the text. They can also click on individual words that will open video of just the narrator signing,

just that word. And then we also have a part that they can watch the whole entire story be told. Oh, very cool. Okay. And now you're working on a TV show. Can you tell me about it? I'd love to. It's so exciting. We started these storybook apps about 10 years ago, and our very first one, the character's name is Mavo. She's deaf. She's a curious adventure. spunky little girl. And so we figured, why not make Mavo sign instead of just featuring a storyteller in the apps? So while we were making the storybook apps, we started working on motion capture in the lab to

make 3D signing characters. We did that because we certainly saw the need to have animated characters that could sign. We knew that it would be appealing to young kids, that it would offer representation. and honestly going into animation opens up worlds of possibilities and storytelling. But we wanted to make sure that they were fluent. So we had this amazing character, Mabo. We had been working on motion capture. And we now got to a point where we can have Mabo sign.

And she looks so great animated signing that we figured we're ready to do a TV series. And she can be the central character of that series. she'll be a signer. And we're building a series called Here Comes Mavo. All right. I mean, that sounds really incredible. I can't wait to see it. So would Mavo kind of be like Dora the Explorer? You know, television in two languages, except that in this case, of course, communication happens simultaneously. That's a good question. In some ways, I'd say that you could say that they would be in the same category in that Dora is really well known as a great educational show. We intend for here comes Mavo to also be educational.

We have a curriculum as part of our series. We've worked with an educational consultant who worked on Dwar the Explorer. So every episode certainly has an educational component to it. But our goal is for Mabo to be a signer, but also to use gesture and visual communication in all of her interactions. So there won't be spoken language as a component to the show. it'll be entirely all of the communication will be visual and visual language will be centered. So what's the process of filming and making Mavo?

Do your actors have to get dressed up in a high-tech suit? You know, we're talking about motion capture, right? Exactly. Yeah. The process is, of course, that we need to have a live person create the content. So they'll act out the scenes and will use motion capture. So they'll wear the high-tech suits that you're talking about. capture them and it's in the post-production that we create the animation and we assign characters

to the data. That's incredible. That sounds like so much fun. I do want to ask you, why is it so important to use motion capture for this? You know, why go through that extra layer of animation at all? Why not just draw the characters and have them sign that way? Hand draw. Well, I don't, more hand-drawn animation or 2D animation. has a lot of limitations for signing. So we're focusing on 3D animation, partly because ASL and sign languages are 3D languages. Space is used as part of the language.

And if we were to go with 2D animation, you would miss some of the information that needs to be conveyed with dimension. So it's not to say that there's no place for 2D animation and there's no value. I think that people should always try to push the boundaries of whatever their mediums can do. But we wanted to design and build a series

where people would be signers. And especially thinking about the fact that our audience is so young, they're not already fluent in ASL. We needed to make sure that the language was as high quality as possible. We need them to be able to see the dimensionality of the characters. So to see a person signing from different angles. And 3D motion capture allows us.

to do that, to move the camera, because you can still understand ASL, if you look at someone from the side or even partially from behind. So that's part of why we're using motion capture. And then it also means that the language that you're seeing in the series is based on real language. So all of the rhythm that's part of the language, the synchrony with facial expression, is all present when you use 3D motion capture. It is, of course, a more labor-intensive process.

But I think technology changes really fast. And any long or intensive process now is going to become more automated and more seamless over time. Melissa, you've been working on this project, you know, the storybook apps and now this TV show for a really long time. What does it mean to you to create stories that deaf children will not only learn from, but also get to actually enjoy? I mean, it is sort of like my entire life's work beyond just my career. I absolutely love this work. And there's times where it's like you see a kid's face just light up because a character is funny or a kid reacts when a character does something wrong. Or you see them, I don't know, you know, like when a kid it says, oops.

Like what they're really doing is making connections. You see those light bulbs go off. You see those sparks get ignited. You see things spark their curiosity. And when I see those moments, it just means so much to me. And that's what always gives me my drive. It rejuvenates my belief in this work. And there isn't a ton of opportunity for this.

There's individual efforts. People have tried to do things on their own at home, like amateur. efforts. But there's not been a place or space that really invests in a professional production like this, where storytellers are looked for, artists are brought in.

There's been no true investment in the creation or production of original work like this. And the process itself is a celebration of our language and culture, knowledge, history, community. And I think it's not just about making cute stories or making things accessible. It's bigger than that. As a person, I just, I want deaf literature to have a place in human knowledge.

Absolutely. I think that's a really good note to end on. Melissa, thank you for joining me. It's absolutely been my pleasure to be here. Thank you again for the opportunity to have the conversation. Melissa Mouskoun is the founder and director of the Motion Light Lab at Gallaudet University in Washington, D.C. To watch my full interview with Melissa or see videos of her lab's work, go to sciencefriday.com slash lightlab. Thank you to Jennifer Vold for interpreting and to Jenna Beakum for consulting on this segment.

Lots of folks help make this show happen, including Ariel Zitch. Santiago Flores. Emma Gomez. Diana Plasker. Tomorrow, join us for a conversation with author Ed Young, about how animals experience the world through smells, magnetic senses, and more. See you then. I'm Rasha Auretti.