Episode 109: Using Machine Learning to Listen to Whales

David Gruber. Video still by Michael Rossi

Episode 109: Using Machine Learning to Listen to Whales, with David Gruber

Click on the audio player above to listen to the episode or follow BornCurious on Amazon Music, Apple, Audible, Spotify, and YouTube.

On This Episode

For millennia, humans have regarded other species with curiosity and wonder. We have tried to decode their behaviors and imagine what they are saying—but truly speaking with animals has traditionally been the stuff of stories, such as those featuring Dr. Dolittle. In this episode of BornCurious, we talk with the oceanographer David Gruber, who is spearheading Project CETI, a multidisciplinary collaboration. We learn that understanding animals is rapidly moving beyond the realm of fiction: Gruber and his colleagues are using hard science—state-of-the-art robotics and machine learning—to listen to and translate sperm whale communication.

This episode was recorded on May 12, 2023.
Released on November 30, 2023.

Guest

David Gruber is a visiting researcher at the School of Engineering and Applied Sciences at Harvard and a distinguished professor of biology at Baruch College of the City University of New York. He also founded Project CETI—the Cetacean Translation Initiative—which aims to apply technology to bring us closer to nature.

Special Acknowledgment

David Gruber and Harvard Radcliffe Institute would like to acknowledge the passing of the environmental scientist and technology expert Karen Bakker RI ’23 in August 2023. Her contributions to the field of bioacoustics, particularly through her book The Sounds of Life: How Digital Technology Is Bringing Us Closer to the Worlds of Animals and Plants (Princeton University Press, 2022), have had an enormous impact. Her book Gaia's Web: How Digital Environmentalism Can Combat Climate Change, Restore Biodiversity, Cultivate Empathy, and Regenerate the Earth (MIT Press, 2024), on which she worked during her Radcliffe fellowship, will be published in April.

Credits

Whale recordings are provided courtesy of Dominica Sperm Whale Project and Project CETI.

Ivelisse Estrada is your cohost and the editorial lead at Harvard Radcliffe Institute (HRI), where she edits Radcliffe Magazine.

Alan Catello Grazioso is the executive producer of BornCurious and the senior multimedia manager at HRI.

Jeff Hayash is a freelance sound engineer and recordist.

Marcus Knoke is a multimedia intern at HRI, a Harvard College student, and the general manager of Harvard Radio Broadcasting.

Heather Min is your cohost and the senior manager of digital strategy at HRI.

Anna Soong is the production assistant at HRI.

Transcript

[MUSIC]

Ivelisse Estrada:
Welcome back to BornCurious, a new podcast from Harvard Radcliffe Institute. I’m your cohost, Ivelisse Estrada.

Heather Min:
And I’m Heather Min. In today’s episode, Ivelisse sits down for a one-on-one with David Gruber, a visiting researcher at the School of Engineering and Applied Sciences at Harvard. David is a distinguished professor of biology at Baruch College of the City University of New York. He’s the founder of Project CETI, the Cetacean Translation Initiative, which originated during his time at Harvard Radcliffe Institute in the 2017–2018 fellowship class. He’s going to talk about his research and what the future might hold.

Ivelisse Estrada:
Hello, David. Good to see you again.

David Gruber:
Yeah.

Ivelisse Estrada:
To start, I want to ask you about when you first arrived at Radcliffe.

David Gruber:
Sure.

Ivelisse Estrada:
Why were you known as the jellyfish guy?

David Gruber:
Previous to coming to Radcliffe, I was really working on corals. Corals have a soft body relative, the jellyfish, and I was just curious about their longevity. They’re so old. They’ve survived mass extinctions. But I was also working with a roboticist here at Harvard, Rob Wood, who was using soft robotics that was inspired by jellyfish. So, I just love the idea of really doing a deep dive into jellyfish—their history, who they are, how they survive extinction events, what we could learn from them.

There’s one jellyfish known as the immortal jellyfish, which is the longest living animal that we know of that could live for eternity. It just goes back and back. It does aging in reverse. It’s unclear how many times it can do this.

So, I was known as the jellyfish guy. That’s why I came here.

Ivelisse Estrada:
Here at the Institute, we often talk about Radcliffe Moments, which is when something amazing forms from collaboration—

David Gruber:
Sure.

Ivelisse Estrada:
—and that creation sparks something magical that persists beyond the fellowship year. What would you describe as your Radcliffe Moment?

David Gruber:
My Radcliffe Moment—we’re on the third floor at Byerly, and there was just a really tight cohort up there of—we were going to lunch every day, we were convening a lot to have conversations. One of the people across the hall was Shafi Goldwasser. She came into my office several times. During my Radcliffe year, I had all this underwater camera equipment kind of piled up in my office that I brought from my university. And she was asking questions. What is this? I began sharing some of my work and played her some whale sounds. I had read a book while I was here at Radcliffe about free divers that were trying to use their free diving capacity to understand sperm whales. And so, I played her some of these sperm whale clicks.

[SPERM WHALES CLICKING]

David Gruber:
And she just thought it was really interesting, and we talked further. Then a few weeks later, she invited me to one of these Radcliffe lunches, where—it was a Radcliffe lunch, or it was a working group—where Shafi had assembled some of the best machine learning people in the world and then brought a lot of the Radcliffe fellows there that were interested in, how does artificial intelligence and machine learning—what’s happening under the hood, and how could we explore it? So, like, Martha Minow was in the room. And she was interested in, could AI, you know, read all the law books at the Harvard Law School and allow for low-cost legal advice for people who can’t afford it?

And I was there and Shafi had asked me to play some of these sounds. I just remember the room, you know, playing these sounds there, where—when you play sperm whale sounds, they’re very different than the normal humpback whale songs that we’re accustomed to. These are more like digital transmission. Almost like a jackhammer or Morse code.

[SPERM WHALES CLICKING]

David Gruber:
I just remember the fascination of everyone in the room, of, like, asking a lot of questions. Wow, how many of these sounds do you have? What’s the context? And in the room was Michael Bronstein, another fellow, and he began even further asking me questions. And we wrote a paper on this, when we did get some of the best database of sperm whale sounds. And it showed that machine learning, indeed, could play a very powerful role in understanding whale bioacoustics.

Ivelisse Estrada:
So, you started this project. It’s called Project CETI. What does it stand for, and where is it now?

David Gruber:
Well, first, just to give some context of starting from this Radcliffe lunch seminar that Shafi organized, this just seeded an idea—just an idea that we could apply machine learning to whale bioacoustics. Then, we did a proof-of-concept study. But at the same time, we threw a Radcliffe Exploratory Seminar, and this was really the key to making this project happen.

We put together maybe 15 people from different disciplines right here at Radcliffe—just two days, just discussing this idea. I remember there was a moment in this Radcliffe Exploratory Seminar where Daniela Rus, who’s the director of MIT CSAIL, kind of came together and said, wow, if we’re able to translate whale communication, we’ll also be able to translate any nonhuman communication, including aliens. I just remember kind of chuckling, and I looked around the room, and everybody was serious.

So from this point, our plan was just to apply for some grants and think of a cohort that would begin looking for funding. We applied to the TED Audacious, and that gave us catalytic funding for five years to begin this project. We decided that the best way to do this project was to actually start a nonprofit called Project CETI, the Cetacean Translation Initiative. So, we began a nonprofit, and this nonprofit is now the largest nonhuman communication initiative in history.

Ivelisse Estrada:
Wow. I have seen the website. There are a lot of people working there.

David Gruber:
Yeah. I think we’re pushing around 50 scientists now that are part of this organization.

Ivelisse Estrada:
So any plans to apply this research to talk to any other species? Or maybe talking to whales is a big enough job.

David Gruber:
You know, I guess one of the things is we don’t want to necessarily talk to the whales. They don’t need talking to. We want to listen to and translate them. We chose this whale for many different reasons. And one of the reasons is that unlike the humpback whales that sing—and sometimes, it’s only one sex that’s singing—the sperm whales use these clicks to communicate, and a variety of clicks in different fashions with tempos and beats.

[SPERM WHALES CLICKING]

David Gruber:
And this is different than—so this is more of, like, sound they use specifically for communication. That’s really interested us. Also, our lead biologist had been studying a family of about 200 female whales in Dominica for the last 15 years and had really carefully and painstakingly collected over 15,000 clicks and annotated them beautifully. So it was that data set that Michael Bronstein was able to lead us through applying some of the most advanced machine learning techniques to show its power on a carefully annotated data set.

But that opened up the question of, if this could do this on 15 years of data collecting in the tens of thousands, what could it do on millions and billions, like ChatGPT-3 and now ChatGPT-4? So partially, the goal now is to create a nonhuman database in the same size category that we have for humans, which is massive. Once we do whales, we could essentially take a lot of the different tools that we’re developing and apply this to others.

Ones that are top on the list are elephants, of course. And we’re already communicating with some of the leading elephant researchers, like Joyce Poole and Katy Payne, about can this be used for elephant communication? And what would a similar organization that—with the Cetacean Translation Initiative—would look like for elephants or for birds? And so yeah. We’re really excited to just, like, basically set up the first tool sets and bring in more people.

There is a Radcliffe fellow this year, Karen Bakker, who wrote a book, The Sounds of Life, which really beautifully summarizes this field of digital bioacoustics. She refers to the combination of digital bioacoustics with machine learning to be the new telescope, the new microscope, and having that potential where it allowed us to see outward so far and see the universe, or see inward inside of cells, that what is coming right now with applying machine learning with bioacoustics is on that level. It’s really exciting to be at the front of the wave of this new field.

Ivelisse Estrada:
I, for one, would like to know what my cat is trying to tell me.

David Gruber:
Yeah. Your cat—there are some commercial entities that are starting out. One of them is called Zoo Lingua that I think is using this more to help with dogs and cats. Of course, animals are communicating with us. That’s for sure. I think dogs and cats were—these are animals that most humans are so familiar with. So yeah, there’s clearly going to be advances in those field. We’re looking even beyond that—you know, looking at birds, looking at primates, looking at elephants. And what would it mean to us if we could understand life in a different way?

That’s one of the beauties, I think, of this project. Like, it’s just an open question. Could understanding animals on a deeper level using this technology bring us closer to nature? We’re hoping yes, but we don’t have an answer.

Ivelisse Estrada:
You’re no stranger to interdisciplinary collaborations from your past, from the research that you came with. How is this one different?

David Gruber:
This project, I feel, is really pushing my limits of interdisciplinary collaboration. In the past, I’d worked on projects with maybe one or two other disciplines. For example, we built a shark eye camera. So, we first had to study the biology of a shark’s eye, then we had to make a camera, and then we went swimming with the shark to then run some models on how the shark sees the world.

But this project is bringing together people from machine learning, people from cryptography, people from robotics, people from natural language processing. There’s linguists on the team. There’s signal processors. There’s underwater bioacoustics. There’s about eight different disciplines that are working together on this project.

Ivelisse Estrada:
What are the challenges to what you’re trying to do? Other than we don’t speak the language.

David Gruber:
No, it’s really interesting. We joke sometimes that the easiest part of this project is the whales, and the human communication among the different disciplines is one of the more difficult parts of the project. But it’s also enjoyable.

That’s something that we picked up here at Radcliffe: the ability to really listen to and hear each other and the different modalities and communication processes, even across disciplines, and just being able to listen to and understand and communicate and really respect the domain expertise. I think, as the leader of this project, that’s a skill that I practice every day, because everybody is so good at their field. And it’s about getting all of these groups to communicate efficiently and effectively so we could pursue the mission of the project, which is better listen to and then translate what whales are saying.

Ivelisse Estrada:
I remember for a really long time, at the beginning of the fellowship, the fellowship staff would do this little skit about terms that meant different things in different disciplines. And they were just very basic words, but it was—I wish I could remember any of them. But it was just, like, you can’t use these words across the disciplines. They mean totally different things to totally different people. I can imagine that that would be something—just finding a common language would have been difficult.

David Gruber:
Yeah. Yeah, there’s so many. And now I’ve become—well, especially having linguists on the team, you know, even the word “language,” it could be a charged word. And we have to be careful how we use that word. And even the word “artificial intelligence” is now—among the community, they prefer to call it “machine learning.” So just even understanding all these nuances about how the different disciplines communicate with each other is—you know, they could easily tell someone who’s not up to speed in their field by the way they use language.

Ivelisse Estrada:
And what do you hope—I think you covered this a little bit, but what do you hope the future brings for Project CETI?

David Gruber:
Project CETI—for me, the idea is really to just bring as much goodness to the world as is possible. We call it a listening project because, in essence, we’re really training our best technology to listen to another species. And I think that kind of higher-level theme of the project is really important. It’s about listening, it’s about trying to translate the voice of another species, and it’s about working together. I think the real, like, what the metric of success would be is that we just push forward our understanding, and we show that by working together, we could achieve a lot more than what any single discipline could have done.

Ivelisse Estrada:
And do you have any other—I have to ask this, because I know that you’re an oceanographer and you’re a nature guy, right? Are there any ulterior motives in terms of, you know, saving the species, or anything like that that are built into a project like this?

David Gruber:
I’m trained as a biological oceanographer, and I started from microbes. I went from bacteria-protozoa interactions, and then into corals, and then into sharks and turtles, and now into whales—you know, Moby Dick, especially. And the motive really is, I think—and as an oceanographer, especially, studying coral reefs—it was so difficult to get people to connect with corals and jellyfish.

Like, it just—people would like to hear about it, but they’re so distant from us phylogenetically, and even under the ocean, that it was really hard to get people to feel empathy for coral reefs. So, I feel with this project, there’s something about whales that really trigger this imagination center in our brain. The fact that there’s some of the biggest creatures—the blue whale is the biggest creature that’s ever existed on our planet, and they’re still swimming here, you know? And there’s still several hundred thousand of them.

And previously, through Melville and Moby Dick, we’ve vilified these creatures and murdered them. But they could be one of the most intelligent creatures in the planet. I just like the idea of bringing awareness to so many different forms of intelligence, and that it comes in many varieties.

Ivelisse Estrada:
How did your work change as you continued your time at Radcliffe?

David Gruber:
I think with many Radcliffe projects—you know, the fellowship happened, and all these amazing people that were put under this roof at Byerly Hall. So I began, starting with jellyfish, with some of the Radcliffe research assistants, and I’m going down the road and continuing. And even with Rob Wood, we went to the New England Aquarium, and we worked on jellyfish. We tested an ultra-gentle robot on the jellyfish and were able to then use advanced transcriptomics to show that, using this ultra-gentle robot that can interact with jellyfish with 1/10th of the pressure that your eyelid rests on your eyeball, it didn’t stress the jellyfish. And this was a really fun study that happened in the back of the New England Aquarium while I was here at Radcliffe.

Ivelisse Estrada:
What else did you find was very unique about Radcliffe, or the Radcliffe Fellowship Program?

David Gruber:
It was the conversations. It was the weekly lunches. It was the openness every time we’d sit with a different group of people at lunch. We got to really know each other as individuals.

And I feel for everybody that was in the Radcliffe fellowship, we were all coming from such busy, hectic lives. You could see it in our eyes, you know? Of just being inundated with work and responsibilities. And this just felt like a kind of collective deep breath, where we could think creatively and get reinspired by ideas. I think that was one of the beauties about this, just to have the time and the space after working for so many years, so incredibly focused, to then find a likewise community of individuals that was so open to sharing and exploring ideas.

Ivelisse Estrada:
So, David, tell us how Radcliffe has impacted your career.

David Gruber:
The impact it had is it really gave me the confidence to explore outward. It gave the time—the time to look at different subjects that had interested me, but not have the time. It gave the space for conversations. It really felt like going back to the days of doing my PhD, where it was just a lot of time to walk around the pond, and think, and have new ideas.

And I think that by having this, having the Radcliffe fellowship at that point in my career, it was such a precious moment that allowed different things to come together and come in. By meeting Michael and Shafi here at Radcliffe, and now we continue to work together on a daily basis, and Michael is the head of machine learning for CETI, and Shafi is the head of theory for CETI—yeah, it really just built community and changed the whole trajectory of everything I was doing.

Ivelisse Estrada:
I want to thank David just so much for joining us today to share his experiences and work with Project CETI.

[MUSIC]

Heather Min:
The BornCurious podcast is brought to you by Harvard Radcliffe Institute.

Ivelisse Estrada:
Thanks for joining us. You can find BornCurious wherever you listen to podcasts, and to learn more about Harvard Radcliffe Institute—

Heather Min:
Visit radcliffe.harvard.edu.

[MUSIC PLAYING]