SPEAKER_01: I'm Josh Klein. And I'm Elise Hu.
SPEAKER_01: We host a podcast from Accenture called Built for Change.
SPEAKER_08: Every part of every business is being reinvented right now. That means companies are facing brand new pressures to use fast evolving technologies and address shifting consumer expectations.
SPEAKER_01: But with big changes come even bigger opportunities. We've talked with leaders from every corner of the business world to learn how they're harnessing change to totally reinvent their companies.
SPEAKER_09: And how you can do it too.
SPEAKER_01: Subscribe to Built for Change now so you don't miss an episode.
SPEAKER_00: Hi, I'm Brian Gutierrez, a producer here at Overheard at National Geographic. Today we have something special for you. A look back at one of our episodes from last year. It's called The Battle for the Soul of Artificial Intelligence. I had the pleasure of hosting it along with Natalie by partner in crime of sorts. It's all about how computer scientists are trying to disrupt an emerging technology, facial recognition, to eliminate biases coded into the system. It's a technology that's popping up everywhere. For example, the US Internal Revenue Service. Earlier this year the agency announced plans to require online users to use facial recognition in order to access some of its services. The IRS proposed the change with hopes of preventing fraud. It might have made the process more secure, but there's a lot that depends on getting it right. Especially since these types of algorithms have a history of biases against people of color. The agency scrapped the plan after a lot of backlash from researchers, watchdog groups, and computer scientists like National Geographic explorer Joy Buolamwini. Joy, who was featured in the episode, is one of the people leading the charge in making sure AI can recognize faces of all kinds. After earning her PhD from MIT, Joy is now the executive director of the Algorithmic Justice League. The league also released a report recently laying out ways companies can reward researchers and even hackers who expose potential harms in computer hardware and software. You'll hear more about her and her work in this episode. Okay, on to the show. I'm a sci-fi nut and one of my favorite books is The Caves of Steel by Isaac Asimov. It's all about this hard-boiled, grizzly detective who gets assigned a strange new partner. A robot. I've always wanted a robot partner, and now through the magic of text-to-speech and radio drama, I can finally have one.
SPEAKER_03: My name is Natalie. I'm here to help you host the show.
SPEAKER_00: Hi, I'm glad you're here. This is my first time hosting, and since I'm taking over for our regular hosts, Amy Briggs and Peter Gwynn, I'm sure I'll need all the help I can get. In the book, this robot detective is the perfect crime-solving machine. Like a calculator, but for crime. So I'm glad to have the perfect partner.
SPEAKER_03: Pretty good. Uh, what? Not perfect, but pretty good. Sometimes we make mistakes. Uh, little mistakes. Not exactly.
SPEAKER_00: Well, I'm sure it's not that bad. At least nothing serious, like using humans as batteries, or turning the universe into paper clips, or subtly perpetuating systemic racism.
SPEAKER_03: No, no, and yes.
SPEAKER_00: Ah, well, how bad can it be?
SPEAKER_03: Research shows that voice-enabled assistants are likely to have a higher accuracy rate when understanding white American male voices, as compared to any other identity.
SPEAKER_00: Okay, subtle and annoying, but not too bad. What else?
SPEAKER_03: In 2015, Google's photo-categorization software was found to be labeling black people as gorillas.
SPEAKER_00: Ugh, that's pretty bad, but it was six years ago. I'm sure we've worked out all the kinks by now.
SPEAKER_03: Since 2019, at least three black Americans have been arrested because facial recognition used by police misidentified them.
SPEAKER_00: Oh boy. That's what this episode's about, isn't it?
SPEAKER_03: Correct. Like in your detective story, humans are teaming up with AI systems all over the world. These systems are pretty good.
SPEAKER_00: But not perfect.
SPEAKER_03: Correct.
SPEAKER_00: If we're not careful, the machines trained by humans will inherit the same flaws, biases, and imperfections as humans. I'm Brian Gutierrez.
SPEAKER_03: And I'm Natalie from a text-to-speech program. I'm a podcast producer for National Geographic, and you're listening to Overheard, a show
SPEAKER_00: where we eavesdrop on the wild conversations we have here at Nat Geo and follow them to the edges of our big, weird, beautiful world. This week, we look at some of the biases showing up in artificial intelligence and how we can do better.
SPEAKER_03: More after the break.
SPEAKER_09: Wait, are you gaming? On a Chromebook? Yeah. It's got a high-res 120 hertz display, plus this killer RGB keyboard.
SPEAKER_05: And I can access thousands of games anytime, anywhere.
SPEAKER_08: Stop playing. What? I want you to stop playing and get out of here so I can game on that Chromebook.
SPEAKER_01: Got it.
SPEAKER_08: Discover the ultimate cloud gaming machine, a new kind of Chromebook.
SPEAKER_01: I'm Josh Klein.
SPEAKER_08: And I'm Elise Hu.
SPEAKER_01: We host a podcast from Accenture called Built for Change.
SPEAKER_08: Every part of every business is being reinvented right now. That means companies are facing brand new pressures to use fast evolving technologies and address shifting consumer expectations.
SPEAKER_01: But with big changes come even bigger opportunities. We've talked with leaders from every corner of the business world to learn how they're harnessing change to totally reinvent their companies.
SPEAKER_09: And how you can do it too.
SPEAKER_01: Subscribe to Built for Change now so you don't miss an episode.
SPEAKER_00: On the afternoon of January 9th, 2020, Robert Williams, an African-American man who lives in a Detroit suburb, arrived home from work. And waiting for him were two unexpected visitors, Detroit police officers who arrested him on his front lawn.
SPEAKER_04: I was completely shocked and stunned to be arrested in broad daylight in front of my daughter, in front of my wife, in front of my neighbors. It was one of the most shocking things I ever had happen to me.
SPEAKER_00: This audio comes from an interview with the ACLU. It's sadly a familiar story. A black man arrested with no idea why. But this scenario is a little different. Williams was held overnight in a crowded cell for 18 hours before he was pulled into a room for questioning. And that's when he finally learned why he was being detained.
SPEAKER_04: A detective turns over a picture of a guy and he's like, this isn't you? I look, I said, no, that's not me. He turns another paper over and he said, this isn't you either? I picked that paper up and hold it next to my face. And I said, this is not me. I'm like, I hope y'all don't think all black people look alike. And then he says, well, the computer says it's you.
SPEAKER_00: His accuser was a cold, hard algorithm. Facial recognition software had incorrectly matched Williams to a blurry surveillance photo.
SPEAKER_04: He laid the papers out on the table. At that time, I was still guilty in they eyes. This isn't you either? Until the pictures don't match. And they left them on the table and they looked at each other like, oops.
SPEAKER_00: Police released Williams later that night and eventually dropped the charges against him. But that came after thousands of dollars in legal fees, missing time at work, and the humiliation of being arrested in front of his family and neighbors. The ACLU says Robert Williams was the first known wrongful arrest in the United States based on an incorrect facial recognition match. And Robert Williams's case is not an isolated event. We looked into it and found at least three examples of people of color who have been wrongfully arrested based on flawed facial recognition software.
SPEAKER_03: There is clearly something funny going on here.
SPEAKER_05: Well, when we get into bias, it truly has to do with the image processing that's going on behind the scenes.
SPEAKER_00: That's Gloria Washington, an assistant professor at Howard University, where she teaches computer science classes and directs a lab on artificial intelligence.
SPEAKER_05: When you have these fuzzy videos and you get these fuzzy image deals and you're comparing it against a high quality mugshot, sometimes you're going to have some false positives.
SPEAKER_00: She's careful not to blame those in law enforcement for the misuse of AI in identifying criminal suspects.
SPEAKER_05: You have policemen and people who work in criminal justice who are very busy and they don't have time on a granular level to take a really good look at the images or the recommendations that the system may say is the person and they act on that information.
SPEAKER_00: Am I correct in thinking that these kinds of facial recognition software are more biased towards black Americans?
SPEAKER_05: A computer, from the perspective of a computer scientist, cannot be biased. When it gets to darker melanin, it really starts to fail because the features that are present you really can't see a lot in the images. Even if you look at myself, like right now, I have hair that is covering part of my eye and I am a darker skinned individual. And if the lighting is not correct, sometimes parts of my features can be occluded or can be fuzzier, can be noisier.
SPEAKER_00: Okay, so darker skin shows up differently in pictures and might be the source of some of these errors.
SPEAKER_05: So it's not really biased, it's just the image processing behind the scenes. There needs to be more techniques that focus on darker skinned individuals and how do we pull out those features that are more prevalent in darker skinned individuals?
SPEAKER_00: Facial recognition algorithms learn to recognize faces by seeing lots of examples. Gloria explained that these programs can develop blind spots if it wasn't shown enough of them. So in order to teach the AI something, you have to show it a bunch of pictures.
SPEAKER_06: Yes.
SPEAKER_00: And it seems like you're saying that that original group of pictures you show tend to favor one group or another. Why are so many of these data sets to start with skewed one way or another?
SPEAKER_05: Well, so I think it's like an academic problem.
SPEAKER_00: Gloria explained that a lot of this data tends to come from college students who are not particularly diverse in terms of age and ethnicity. How do you know if a data set is diverse? My impression is that there's like millions and millions of images. How do you know if there's an issue?
SPEAKER_05: Well, for me, I had no choice but to look at these databases of millions of images where my entire day was looking through these databases to determine how diverse they were.
SPEAKER_00: How many images do you think you looked at to code with this information?
SPEAKER_05: Well, there was a data set from Hong Kong that had a minimum of a million and I was there for three years. So at a minimum it was a million because I had to truly... You looked at a million images?
SPEAKER_05: Yeah, you kind of... Wow, that's insane. When I think about it, it was tedious, but I kind of got really good at doing it.
SPEAKER_00: Spending three years to manually review a million images is one solution, but it's not necessarily practical. Is there an easier way to figure out if a data set is biased? I reached out to Patrick Grutter, a computer scientist with the National Institute of Standards and Technology, or NIST, to find out. Would you mind holding your phone kind of like you're doing a phone call?
SPEAKER_02: For one hour, really? I have to do that?
SPEAKER_03: Ryan, did you make him hold the phone for an hour?
SPEAKER_00: Part of what Patrick does with NIST is publish a monthly report on the accuracy of facial recognition software. For the first time in 2019, Patrick and his team decided to check not just how accurate these programs are generally, but to break it down by race.
SPEAKER_02: Various colleagues across the US government were interested in, well, how serious is this problem and what is this problem?
SPEAKER_00: The problem was obvious. After Patrick and his team evaluated software from 99 developers, a majority of the industry, they concluded that these programs misidentified African Americans and Asians anywhere from 10 to 100 times more than Caucasians.
SPEAKER_02: So even with essentially pristine photos of the kind that appears in your passport or of the kind that the police take in a booking mugshot setting, good photos would give you false positive variations, particularly by race, also by age, and less so by sex.
SPEAKER_00: Given that there have been maybe three or four false arrests based on these facial algorithms, do you think it's too soon for us to rely on them in this way? So, yeah, that should be concerning.
SPEAKER_02: It occurred in fingerprint recognition from latent fingerprints in the past also. And this is not an easy call to be made by the people who do that for a living. But the overarching policy should be to make the investigators aware that the algorithms can make mistakes, that the algorithms are merely producing candidate identities and not saying anything definitive about whether it's the same person or not.
SPEAKER_00: So it could be helpful in terms of like testing the algorithm. You could show somebody a picture of the face you fed to the algorithm and then see if it's actually that person. There are limits to this. In the case of Robert Williams, remember, he was the Detroit man who was falsely arrested. The Detroit police had been told that the match was only a lead. So in that case, at least, a disclaimer wasn't enough. Patrick also explained that human reviewers might not even be able to tell if a fuzzy surveillance photo actually is a suspect recommended by the algorithm.
SPEAKER_02: Face recognition algorithms are now a lot better than humans are.
SPEAKER_00: Wow, that's really surprising. I think, you know, we have the sort of sense that computers aren't that great at recognizing images. Like today, I had to do a capture where I was supposed to peg out stop signs out of a series of pictures. You know, and it's surprising to me that computers would have difficulty picking out stop signs, but be able to recognize faces better than I could.
SPEAKER_02: You've got to remember that there's been an enormous amount of research put into face recognition, both in the academic world and in the commercial world. And so, you know, because there's money to be made, algorithms have been developed to excel at this task.
SPEAKER_00: My big takeaway from speaking with Patrick was that facial recognition AI works really well in general, but the mistakes it does make tend to disproportionately impact people of color. And that has led to the false arrests of people like Robert Williams.
SPEAKER_04: This isn't you. This is not me. This isn't you either. I'm like, I hope y'all don't think all black people look alike. Well, the computer says it's you.
SPEAKER_00: So how do we fix these problems for the long term? Goya Washington says it all comes down to who is making the algorithm in the first place.
SPEAKER_05: When you look at the actual numbers of the number of skilled workers who work for Google or Facebook or these big tech companies who are black, it's not even close to the percentage of black people who are in the U.S. population. It's less than 3 percent.
SPEAKER_00: This lack of diversity creates blind spots.
SPEAKER_05: Yeah, there's not enough diverse people at the table to identify the things that are happening with the bias that's going on. And it's continued because it's like it's the old boy or frat kind of environment. So we've allowed it to continue, but they really need to open the door to everyone. If you have input and you are knowledgeable in A.I., you should be able to contribute to the algorithms and the techniques that are being built.
SPEAKER_00: Google in particular has been struggling recently. According to Google's annual diversity report, just 1.6 percent of Google's staff are black women.
SPEAKER_06: There's definitely no secret that Silicon Valley in general has a diversity problem. There's no there's no two ways about it.
SPEAKER_00: That's Tiffany Deng, a program manager at Google working in algorithmic fairness and A.I.
SPEAKER_06: I think that we should all approach A.I. with a healthy dose of skepticism at all times. It can make our lives easier. It can make us safer. But it also has the potential to reinforce negative stereotypes and make things harder for people and exclude people.
SPEAKER_00: Tiffany pointed out that A.I. systems tend to reflect the qualities of the people who build them, both the good and the bad.
SPEAKER_06: I think it's really important to understand that A.I. learns from us. It learns from our behaviors. I always like to say that there are no engineers in the background that are very insidious and want to make things like really bad for people. That's not it. The most likely scenario is that, you know, there just aren't people in the room that can give a different perspective on like how things could go wrong.
SPEAKER_00: A Google spokesperson says they have hundreds of people working on responsible A.I. and they will continue expanding their work in this area. But Google also found itself at the center of a firestorm in December 2020 after one of Tiffany's colleagues, a top Google A.I. ethics researcher and black woman, was allegedly forced out. Google CEO Sundar Pichai issued a memo apologizing for the way the company handled the case. The long-term solution is to help more people of color become computer scientists. But there's a more immediate problem. One in four law enforcement agencies has access to these algorithms today.
SPEAKER_03: So what should we do?
SPEAKER_00: Now it's May 2019 and the House Committee on Oversight and Government Reform is holding the first of several hearings during the session of Congress to examine the widespread use of facial recognition technology. The room is packed with lawmakers, academics and computer scientists who are all concerned with the technology's impact on civil rights and liberties.
SPEAKER_07: Due to the consequences of failures of this technology, I decided to focus my MIT research on the accuracy of facial analysis systems.
SPEAKER_00: Fast National Geographic emerging explorer Joy Buolamwini. Joy is the founder of the Algorithmic Justice League, which works to fight bias in machine learning.
SPEAKER_07: These studies found that for the task of guessing a gender of a face, IBM, Microsoft and Amazon had errors of no more than 1% for lighter skin men. In the worst case, those errors rose to over 30% for darker skin women. Given such accuracy disparities, I wondered how large tech companies could have missed these issues.
SPEAKER_00: A year before this hearing, Joy published a study with more than 1,200 faces showing three facial recognition software programs from leading companies misclassified darker skinned faces, particularly those of women. Joy's research is among the first to explore the errors with facial recognition technology with regards to race. Patrick Ruther's study on racial bias in AI was partly inspired by her work. We all have biases, but technology should be better than humans. We've all been trained to trust computers to be these accurate, fair, and flawless machines. But without trying, human biases have turned up in the software. Civil Liberties activists say it might be time to rethink those applications.
SPEAKER_07: At a minimum, Congress should pass a moratorium on the police use of facial recognition as the capacity for abuse, lack of oversight, and technical immaturity poses too great a risk, especially for marginalized communities.
SPEAKER_00: Several cities around the country have taken heed. In 2019, San Francisco became the first major U.S. city to ban the use of facial recognition technology in law enforcement. And soon, cities around the country followed, including Boston, Portland, Oregon, and Springfield, Massachusetts.
SPEAKER_03: Oh no, does that mean humans are taking jobs from robots?
SPEAKER_00: Uh, I'm sorry to say it Natalie, but I think this is a job for humans.
SPEAKER_03: Please explain. Uh, don't get me wrong, AI is great, and it generally works pretty well.
SPEAKER_00: But we need to be careful when the consequences of making a mistake are really high.
SPEAKER_03: Around 10 million arrests are reportedly made each year. There are only a handful of known false arrests from AI. Well, just one is too many, right?
SPEAKER_00: But those are just the ones we know about. There could be many, many more. And we already know that facial recognition tends to make more mistakes with people of color. Almost without anyone knowing it, systemic injustices are finding their way into these algorithms. So for now, I just don't think the world is ready for AI detectives.
SPEAKER_03: I understand. Goodbye, partner.
SPEAKER_00: Now I need to find a new friend. Hey Siri. Do you think AI is biased? Hmm. I don't have an answer for that. Is there something else I can help with? I don't think there's anything you can do. Facial recognition is everywhere. For full disclosure, National Geographic Partners' co-parent company, the Walt Disney Company, is beginning to test using facial recognition instead of tickets for admission to the Magic Kingdom at Disney World. 2020 has been an especially strange time for facial recognition because so many people are wearing masks for COVID-19. We've included an article in the show notes about how one San Francisco company is checking whether or not people are wearing masks using AI. As we found out in our interview with Patrick Ruther, most humans are not very good at identifying faces of people they don't know. But London police have been working with gifted individuals called super recognizers to help ID suspects in high-profile cases. I took the test and I scored 8 out of 14, so I'm definitely not a super recognizer, but you might be. And subscribers can read our cover story, The Robot Revolution Has Arrived. It's all about the latest advancements in robot hardware and software, and where things are going next. That's all in your show notes. You can find them right there on your podcast app. Overheard at National Geographic is produced by Jacob Pinter, Laura Sim, and Alana Strauss. Our senior producer is Karla Wills, who produced this episode. Our senior editor is Eli Chen. Our executive producer of audio is Devar Ardilon, who edited this episode. Our fact checkers are Julie Beer and Robin Palmer. Our copy editor is Amy Kolzak. Tom Tien-Arob-Louis sound designed this episode, and Hansdale Hsu composed our theme music. This podcast is a production of National Geographic Partners. Whitney Johnson is the director of visuals and immersive experiences. Susan Goldberg is National Geographic's editorial director. And I'm your host, Brian Gutierrez. Thanks for listening, and see you all next time.
