Odd Lots - This Is How Algorithms Impact Every Aspect Of Our Lives, from News to Credit Scores to Stocks

Episode Date: December 18, 2017

Algorithms. People talk about them all the time, particularly in relation to markets. But who actually designs them, and what do they do? On this week's episode of the Odd Lots podcast, we speak with Frank Pasquale, a law professor at the University of Maryland, and the author of "The Black Box Society: The Secret Algorithms That Control Money and Information." Pasquale, who has been following the growing importance of algorithms for several years explains the various ways they're shaping our life without us being aware of it.See omnystudio.com/listener for privacy information.

The news doesn't stop on the weekends. Context changes constantly. And now Bloomberg is the place to stay on top of it all. Hi, I'm David Gurra. Join us every Saturday and Sunday for the new Bloomberg this weekend. I'm Christina Rafini. We'll bring you the latest headlines, in-depth analysis, and big interviews. All the stories that hit home on your days off.

And I'm Lisa Mateo. Watch and listen to Bloomberg this weekend for thoughtful, enlightening conversations about business, lifestyle, people, and culture. On Saturday mornings, we put the past week's, events into context, examining what happened in the markets and the world. That on Sundays, we speak with journalists, columnists, and key political figures to prepare you for the week ahead. Join us as soon as you wake up and bring us with you wherever your weekend plans take you. Watch us on Bloomberg Television. Listen on Bloomberg Radio, stream the show live on the Bloomberg business app, or listen to the podcast. That's Bloomberg this weekend.

Saturdays and Sundays starting at 7 a.m. Eastern. Make us part of your weekend routine on Bloomberg television, radio, and wherever you get your podcasts. And welcome to another edition of the Odd Lots podcast. I'm Tracy Allaway. And I'm Joe Wisenthal. So Joe, you know what I was thinking. No, I don't actually. Okay. What were you thinking?

That was a pretty open-ended question in retrospect. That would have been pretty impressive if I had gotten it right. All right. What I was thinking is that we have a tendency on this show to talk a lot about quantitative trading. We talk a lot about high frequency trading, systematic trading strategies, and we don't actually talk that much about the things that underpin all those strategies. Do you know what I'm talking about? I've thought about this before. So we just sort of speak in the abstract about what quants are doing. We're like, oh, humans have no chance anymore at beating the algorithms. but they're sort of in the general discussion, and I think we're a little bit better than this here, to be honest,

but I think of the general discussion about all these things. We don't really like dive into like, you know, the math and how it really works and, you know, who comes up with all this stuff? Right, right. So you hit the nail exactly on the head. So we don't talk that much about the algorithms. And there is this perception out there that the algorithms or the maths doing all this stuff are better than human beings. And I think that's a question, you know, I don't think we actually have the answer to that. It's just an assumption that people have made. And in recent years, we have seen lots of people start to talk about weaknesses in the algorithms that

underlie a lot of these strategies. And I should just say here, when we talk about algorithms, yes, you think trading, you think finance, but algorithms kind of rule our lives in a lot of different ways now. you see a lot of the applications in business, whether it's dynamic pricing on Amazon or online advertising or content curation. Think about the Netflix movies that get shown to you all the time. Algorithms and the assumptions underpinning them are basically everywhere at the moment. And we're not really asking enough of the tough questions about them, I think. My most frequent thinking about algorithms these days is when I drive out of the city and I use the ways app.

And it tells me like different paths to beat traffic. And sometimes it seems like it gets it totally wrong. Usually it does an amazing job. And sometimes I second guess it and regret it. But sometimes it's like, oh, how did I get in this situation? So I have firsthand experience with occasionally they don't work perfectly. Right. Okay.

Everyone has an algorithm experience. Like you watch that one bad movie on Netflix and then suddenly for the rest of your life, you're getting pushed sequels. Or in my case, because I have a 20-month-old daughter, when I go on, it just thinks that all I want to watch is children shows. So those are some of the downsides of algorithms, but I think we can do better than that. We have a really good guest to talk about all the assumptions that are actually going into these things. and it is someone I've spoken to before back in my previous life over at the Financial Times. It's Frank Pascal. He's professor of law at the University of Maryland. He's also the author of an entire book on this subject called The Black Box Society. Frank, thank you so much for coming on.

Oh, thank you, Tracy and Joe. It's great to be here. So you've clearly identified some of the issues with algorithms. I think. you give it away a little bit in the title of your book, The Black Box Society. These are things that are shaping our society, but we don't actually know a lot about what's going into them. Essentially, they're a black box. What piqued your interest in this subject? You know, it's a long history. I began back in 2006 looking at search engines. And at the time, I was just so enthused about the way in which they helped you find things and how much easier they made research and finding music and movies and stuff. And my first articles on them were all about how you could expand fair use doctrine to get

people more access to them. But then I started looking at the dark side and seeing, you know, there are all these disputes about what should be highly ranked or not so highly ranked and people that had really embarrassing or untrue stories about them that got, you know, really high in the rankings and they were trying to fight that. And so then I started just writing about the search engines. And then the financial crisis happened. And I was just fascinated by that. And I found that there where a lot of parallels between tech and finance and how they used algorithms. I just want to say I'm already really excited about where this episode is going just based on that answer. Before we get to like the finance stuff, I'm just thinking about, you know,

you mentioned the early days of search engines and how it seemed like this amazing new thing ended up having Darkside. It reminds me it feels like that debate is really sort of mirrored in the discussion of social media these days and the way algorithms turn up news that reinforce our biases or reinforce our bubbles. And so it feels like probably what you were looking at back in 2005, 2006, must feel very similar when you see people talking about the kind of things that Facebook and Twitter surface. Absolutely. And it does feel like deja vu all over again. To Congress's credit, I actually was called before the House Judiciary Committee in 2008 to talk about some of this stuff.

They didn't do much then. But actually last week, I was just before another House committee. it looks like they really do get it. I mean, there's been a whole phase change, I think, and I think it is because of exactly what you're describing, Joe, this awareness that we just don't know where a lot of the ads or bots or other things that we see on Twitter or Facebook, where they're even coming from. So since we're on this topic, and I assume we'll get to some of the more finance-oriented algorithmic stuff later. But what do you think are the most insidious applications of algorithms in our sort of day-to-day non-trading financial lives nowadays.

The thing that I am most troubled by is the fact that you could have stealth health profiles of yourself out there. So we now know that there's sort of a digital doppelgonger. We all have this digital second self out there that is the aggregation of all the different profiles that are, you know, from data brokers, from the giant online companies like Facebook, Google, Twitter, etc. And what I worry about is that, you know, these things could be used to manipulate people. So, for example, there's a concern about what's called vulnerability-based marketing, where people are trying to market to the gullible others. And that's been exposed by various governmental entities.

So those, I think, are really troubling. You know, if they've got something that says, oh, this person is really upset, and when they're really upset, you know, that's the best time to target them for a really expensive purchase or something. That's worrisome. There are lots of other ones I can give that are outside the consumer side. They're more on the work side, law enforcement side, but that's a start, I think. Yeah, keep giving them. Tell us a little bit some of the other ones, too.

Sure. So, I mean, there are scores that are about whether someone is likely to be a fraud or not. And these scores, people don't even know that they exist. I mean, 99% of people don't know that they exist. Or they know that there are credit scores out there and that the three major credit bureaus are calculating those. But I did this article called The Scored Society with Danielle Citron, and we looked at the research on all these other scores that are out there, about whether people are reliable as employees,

their medication adherence score, whether they're likely to adhere to a medication regime. You know, all these sorts of things that, you know, people don't know about, and that oftentimes they're not accurate. You know, so you have a problem that, like, one guy complained, and I think this was actually in a Bloomberg article, he complained that he was on a list of diabetics,

and he's not diabetic. And the irony here is that the companies then come back and say, well, it's not really a list of the diabetics, it's a list of the diabetic concerned. And so you can't prove that we're wrong because we think you're diabetic concerned, and that's our opinion. So there are a lot of problems in terms of like there's lists out there and people are not really responsible for the lists and nobody's really looking to make sure that they're accurate. And we don't know how far are the applications of them and how they might be used to deny opportunity or to otherwise, you know, classify people in ways that are negative. Now, you mentioned credit scores just then.

And in many respects, these were sort of the first algorithm. scoring models that we saw in the consumer space. And their history kind of parallels some of the concerns that we're seeing erupt now over various types of new data-driven algorithms. Can you maybe give us a sort of potted history of credit scores and how they reflect some of the conversation that we're having now? Yeah, I think credit scores are a great place to start. And the irony is this, and we're seeing exactly the same rhetoric now, in the 60s and early

there was a big concern that the decisions made by individual loan officers, bank employees, were discriminatory. And so a lot of people said, well, you can't keep doing these discriminatory decisions. You need to have an objective metric in terms of how you decide who you give credit to, what rate you give it to, et cetera. And so that led to more demand for these sort of scoring systems that were, they were developed in the 50s, but, you know, they weren't in huge demand. But then they, over the 60s and 70s, they were in more demand. And then what ends up happening is that they go from being a relatively simple, you know, sort of set of criteria to very complex and adding in more and more data and ways of transforming the data. And they are secretive because a lot

of companies, they don't want to try to get a patent on these things because there's already prior art out there, so they want to protect them legally as trade secrets. And the concern that a lot of people have had over the past 40, 50 years is that the scoring systems are, are they incorporating data that's accurate? How are they incorporating it? Do they have disparate impacts? There are some studies that show they have disparate impacts on minority groups and others. And so there's been a lot of controversy over these scoring systems and that controversy, but if we look back to their beginnings, it's exactly the same case that's now being made for AI. There's all these people that say we need like AI-driven, algorithm-driven police department

because the police are prejudiced, and we need AI-driven hiring because HR is prejudiced. But what we found in the credit score context is that the flight to a computerized algorithmic process that has its own biases in it. And secondly, that the data in all of these systems is still data collected by humans. You don't have robots creating the credit data or the crime data or other data. And because that data can itself have all sorts of biases in it, a lot of times the algorithms are not what's really the key actor. What's key is the data and it still has the same problems that the old system had. Now, just to play devil's advocate for a second, as I think a lot of people would acknowledge, there's noise that could get into the data. It has

to get humans have to deal with it. So that introduces error. The flip side would be, okay, yes, it's kind of messy, but because we can get reasonably accurate profiles of would-be borrowers, we can lend to more people and we can lend to them at a lower rate because we can feel confident that they're not one of the fraudsters or they're not inclined to default on their debt. And so the devil's argument would be, yes, you can pinpoint all these problems, but what's unseen is greater credit availability and cheaper credit. That's a great point. And I mean, I think that this is one of those areas where we're going to have to make a lot of tough tradeoffs, And this is something that's happening in the general fairness in machine learning community.

There's now a lot of researchers in computer science and law that are working on this sort of issue. And I think that what we're going to have to try to do is maintain some of the efficiencies, but also try to get rid of, and maybe lose a little bit of efficiency at the edges, but get rid of some of the discriminatory side effects. The other thing I would say about that is that, you know, there's a really interesting book about the financial crisis, which called a call for judgment. And the author makes the argument that these systems,

they fooled us into thinking that we could calculate risk better than we actually could, right? And so the concern there was that if you, because one of the other things that happened with credit scoring is they said, well, it's not just a binary, you get credit or you don't, someone with a very low score, maybe we give them credit,

but at a very high interest rate, someone with a high score, we give them very low interest rate, And so those sorts of judgments, they can be seen one by one as being very efficient, but then they can create these larger systemic effects that it's hard to really anticipate at the beginning. To what extent are credit score is able to be manipulated by people who understand the sort of basic factors or ingredients that are going into them? Because I remember some of the research that came out of the financial crisis showed that you had bunchings of credit scores. applying for mortgages around certain cutoff points, which kind of suggested that someone was

aware of what was going on and was really, you know, keeping an eye on the FICO. That is such a great question. So, I mean, there are all of these online forums that argue that you can figure out these secret signals, you know, for example, like have four credit cards because if you have less than four, you might be seen as having too few and more than four. you're seen as having too many, those sorts of little things. And there are people that are constantly on these forums saying one thing or the other. What I have heard, though, from the empirical researchers and from a guy named Aaron Reichie at a place called Upturn,

which is a think tank that focuses on algorithmic fairness, is that ultimately this sort of extra data or very obscure data or data where it's hard to explain the exact effect of it, that it's hard to use that to manipulate your credit score or massage. your credit score into a higher. And that really what drives 80 to 90% of it is timeliness of payments. So this has led some people to say, including the group algorithm watch, I think there have been people there that have argued, look, we have to, we should just simplify this system, don't make it so secret, because 80 to 90% of the value is in very obvious things like your payment history.

But there are people out there that still are trying to work the margins. You can get the news whenever you want it with Bloomberg News Now. I'm Amy Morris. And I'm Karen Moscow here to tell you about our new on-demand news report delivered right to your podcast feed. Bloomberg News Now is a short five-minute audio report on the day's top stories. Episodes are published throughout the day with the latest information and data to keep you informed. Yes, there are other products like this from a variety of news organizations. But they usually rerun their radio newscasts throughout the day.

That's not what we do. We create customized episodes that can only be heard on Bloomberg NewsNow. And we don't wait an hour to publish breaking news. When news breaks, we'll have an episode up in your podcast feed within minutes. So you're always getting the latest stories and developments. Get the reporting and the context from Bloomberg's 3,000 journalists and analysts we're all over the world. Listen to the latest from Bloomberg News Now on Apple,

Spotify or anywhere you listen. Just on that point, you know, if we have questions about how these algorithms are working, what sort of assumptions they're making, what sort of data they're looking at, then surely the solution to this is to throw them open to some more scrutiny. And in fact, again, if you look at the credit score history, that's exactly what happened. There was such an uproar over credit scores that eventually, I mean, it took them a while, but eventually the U.S. government said you have to at least make everyone's credit score available to them. And that's why nowadays you can go online and find your credit score.

And then eventually, of course, your credit score gets hacked in a major data breach. But that's probably a slight tangent for us. I think that was very important transparency legislation. I think the problem, though, is that what we're finding now is that the companies are arbitraging around that because there is a core credit score that they will give you, but it turns out that in certain applications or certain companies want a variation on it or something that has additional data or a transformation or a bespoke type of score that you don't get. And so this is the worry that I have is that, you know,

if you don't have financial regulators and even legislators that are constantly keeping up with the newest tricks of the industry, you're going to fall behind. and the purposes of the FICRA now is almost 50 years old, its underlying purpose will be defeated. Frank, I want to go back real quickly. It's something you were talking about earlier, and you were talking about how we all have this sort of separate self, which is a collection of our characteristics and certain attributes about us.

When people talk about targeting through advertising and things like that, is it as simple as, okay, I want to target Frank. And so let me call up the information. or is it I want to target people who are vulnerable about X and Frank, you'll be on the list. Like how is, like, really like what kind of is the visibility on the individual level or is it on the attribute level? By and large, all that I've heard in terms of the empirical research on the mainstream marketing is that is on the sort of aggregate level. And so you're buying and selling, you know, very, very large sets of individual. However, you've got to have two caveats to that.

One is that the field of re-identification research that people like Latanya Sweeney and Arvin and Narayan have been doing for over a decade, that they are constantly finding new ways to re-identify people from what seem like anonymized data. There was recently some finding in Germany where someone re-identified or put back together a huge number of people's search engine queries back to their name. So this sort of thing happens all the time. The second is that, you know, if you drill down closely enough, I mean, imagine if you were like to say, I want to advertise to this person that has, you know, there are enough identifiable characteristics about people if you know something about them that maybe you can, you know, try to buy a direct profile on them. I know that that was sort of in the air when the Congress got rid of the ISP privacy rules.

All these people said we're going to crowd fund a purchase of Congress member's Internet search history or browsing history. So, and there's articles out there about, in China, apparently, you can buy these sort of dossiers. An article by Theo Rostow is up there called What Happens When an Acquaintance Bies Your Data. So I think that we are on the cusp of some pretty rapid unravelings of privacy, but as of now, it's much more done on the aggregate level. I want to segue over to the world of finance because that's one place where, of course, algorithms have kind of been in the head. for many years now. Often in a negative connotation, people complain about high frequency trading and that it's ruining the market and making it more difficult for human beings to actually outperform. What's your take on the rise of algorithmic trading and how it's changed

markets? So I wrote an article a couple years ago called Laws Acceleration of Finance, where I looked into some of the developments in HFT and dark pools and some of these other things. You know, I've had two perspectives then, one being exactly your point about, you know, some of the risks involved here, worries about systemic risk or financial stability if something like the flash crash happened again, and also worried about the way in which speed was made into this very high value through what I thought were pretty bad legal choices, that you could find other ways of breaking the tie of, say, two people's bids for a given a lot. of stock, you know, came in at the same time rather than trying to just pull out the decimal point further and further.

I will say now that it's, you know, three years later, I think I probably was too concerned about the financial stability consequences of HFT. Like, I mean, and maybe I'll be proven wrong, but it does seem that, you know, now we're almost eight years out from the flash crash and not that much seems to have happened along those lines that would be on a really high systemic level, although, you know, maybe I've just missed some stuff there. I still do keep with my earlier point, though, that I think that this is, that the emphasis on speed will eventually, I mean, has some negative side effects and that we're seeing this in terms of, you know, just the level of confusion and frustration among some traders about the, you know, how, how, who gets access to which data and at what price and stuff like that. You know, it doesn't seem like it's very economically productive use of people's efforts there.

Well, Frank, the other big way in which, and I imagine a lot of our finance listeners think about this a lot, it just questions about like, is this going to be so good that, you know, we have a handful of people who sort of program the algorithms, mathematicians, physicists, and that there's essentially no jobs for anyone else. And I think, you know, you obviously hear this anxiety in finance, but you also hear it in other areas. And you hear it with people, concerned about the future of truck drivers if self-driving cars and self-driving trucks get too good and it's just an endless discussion. In your work on this, what do you feel is, are we framing the question right? Like, how should we think about this idea of where will humans lose out to the algorithms and work? Oh, I love that question. And I have been writing a manuscript on robotics and automation for the past couple of years, and I think it is the question. I think with respect to the, in finance, one of the reasons why you see a lot of replacement of, I think traders and others, you know, with these automated systems is because they sort of made

the system too simple. You know, if you're just maximizing as to price or to, you know, get a certain return, it can be something a computer can do. And I think that when you look at other areas where there are multiple competing values, that's harder to automate. You know, when I think of like a teacher trying to decide whether he or she is going to yell at somebody who's a disruption or kick them out or try some other approach, you know, in terms of something more subtle. I think of doctors and the type of complexity of what they're recommending and even, you know, personal trainers, other people with high touch professions. So I think that my sense is that, you know, that there are probably going to be a lot of jobs left out there that whenever

you can't sort of just optimize as to a mathematical equation. And I remember this one great quote. I forget who said it, but they said that, you know, choosing how to optimize for a given value is a math problem, but choosing what to optimize for is not a math problem. And I think that'll be the bottom line in terms of the automation debate. I have a slightly conceptual, well, a very conceptual question when it comes to algorithms and finance, especially in a sort of risk management setting. Do you think algorithms are ultimately forward-looking or backward-looking? Because often they're focused on extrapolating the future, but they extrapolate that future from a set of historic data and current data. So the question of whether the past data is going to be

reflective of the future is one of the biggest problems for big data-driven AI algorithmic systems. And I think that we need to really be able to answer that and to realize that if we have systems that are just going to be based on past data without much opportunity to think about how things will change or should change, that's a big problem. The best example I can think of that is one that, you know, Kathy O'Neill gives in her book Weapons of Math Destruction, where she talks about how if you have an algorithm that is, you tell people in the HR department hire people based on who did the best in the past at this firm or who made the most money or what have you. Well, if it turns out that, you know, in the past also

there was a forms of discrimination where they were always hiring a certain type of person, you may not just be baking in assumptions about how people's qualities relate to how they perform, but also you're baking into the future prediction system, the discrimination that existed in the past. So both of those have to really be considered. So, Frank, you obviously raise a lot of disturbing questions, and it's hard to wrap our head around how you would begin to solve many of these problems. I'm thinking about, you know, recently their Facebook, going back to them, has gotten a lot of trouble because people have found that it's very easy to create ads that discriminate against races, and then Facebook rushes out to put out a fix,

and then people discover another way to do the same thing. It seems like it's sort of like this, you know, they're trying to hold back the tide, and there's almost nothing they can do with the creation that they've built. But big picture, what are some approaches to think about that could take on, you know, all the issues you raise head on? I'm really glad you brought up the Facebook example and the problem with the discrimination,

the potential for housing discrimination, with the racial affinity classifiers that advertisers are given on Facebook. And I have a couple of thoughts on it. I mean, it's funny. I am a bit of an old-fashioned or an old-timer in the sense that I think what we're finally seeing is that you can't really run

the largest media company in the world, which is what I think of Facebook and YouTube or some of the largest media companies in the world. You can't run them via robot. You can't just run them. via AI, that all the touted gains and efficiency via automation of content, automation of advertisers, preferences, and news feeds, that all of those come at great cost. And we're finally discovering the cost. It's a lot like discovering global warming. You know, I mean, the carbon, although it

happens faster, right? We had sort of a carbon-driven industrial revolution that was amazing for decades and now we're discovering, wait a second, we've got a fundamentally retool or else we're going to cook ourselves. I think it's very similar with respect to this automation of content online that we have just been amazed at how much profit these companies could make, but now all of a sudden we're saying, wow, we're going to have to fundamentally retool how they work. And fortunately, they are doing that. I mean, I've heard that YouTube is going to hire thousands of people in the wake of all the complaints about exploitative child-directed content that's been discovered over the past few months. So that's going to bite into their profit margins, no doubt, but it's also

going to make them, I think, higher quality entities. And it's going to create a precedent for how we should de-autimate lots of other fields, I think, including journalism. So, yeah. Oh, okay. Let's leave it there. Frank Pascal, a professor of law at the University of Maryland and author of the Black Fox Society, it was really great having you on. Thanks so much. Oh, thank you. Terrific questions. I really enjoyed it. Joe, I was kind of joking when I cut him off at the journalism point.

But of course, I mean, we see the applications of artificial intelligence and algorithms to our own field. We at Bloomberg have an automated news service that spits out automated news. And it usually does a pretty good job. Yeah, no, it does a good job. And I like that you cut off there because I feel like that's a whole, that's a whole separate episode. other episode, yeah. With very raw emotions from our part. That's right.

But in general, I mean, I found that conversation absolutely fascinating and it hit on so many of the big news items of our day, actually, not just in finance, of course, but also in the world of media and politics. It's interesting how many of the different things we're talking about today end up coming back to algorithms in some way. So obviously, so much discussion about. racial concerns in this country. And as Frank pointed out, you know, algorithms may play a big role in, you know, exacerbating a problem that we'd like to fix. And so whether we see that in

law enforcement or we see that on Facebook. And then, of course, other areas, markets, robotics, like it all seems to be coming back to the same topic. Or at least if we try, we can all bring it back to the same topic. Yeah. And of course, it also gets to the importance of data, which is a conversation that I think you and I have had more than once on this show. The people that hold the best data nowadays are probably the people who are going to do the best competitively. So I expect this will just harden some of the scramble that we've seen for special data, you know, data that's not available to everyone else, especially in the world of finance. Totally.

And I also think like it's pretty clear that there's going to need to be some system of redress, right? I don't think that society is going to accept it if, okay, you got put on the at-risk for diabetes list. You have no risk for diabetes whatsoever, or it's very low, but sorry, there's nothing you can do about it, right? Like, I don't think that's going to be a tolerable situation. The job of cleaning data, the job of helping people sort of have accurate data out there, getting rid of inaccurate data, seems like this mammoth task that humans are probably going to need to be employed in. Maybe a first test of this is convincing Netflix to break your account out of its, I guess, toddler TV specific suggestions. And for me, it's, you know, I watched one rom-com three years ago, and now it's suggesting an endless stream of Julia Roberts movie. No, you joke and you think about the fact that, like, you're still dealing with it, and you're like, oh, we're never going to solve any of these real problems that we have in society.

Yeah. All right. Well, on that note, this has been another edition. of the Oddlots podcast. I'm Tracy Allaway. You can follow me on Twitter at Tracy Allaway. And I'm Joe Wisenthall. You can follow me on Twitter at the stalwart. And you can follow Frank at Frank Pasquil, P-A-S-Q-U-A-L-E, and our producer, Sarah Patterson, at Sarah Pat with two teams. Thanks for listening. On April 4, 2023, around two in the morning, a man was found stabbed multiple times.

on a sidewalk in downtown San Francisco. Hey, who did this to you? What happened next turned the story into a political firestorm. Reports have identified the victim as Bob Lee, the founder of Cash App. From Bloomberg Podcasts, this is Foundering, the Killing of Bob Lee, beginning April 16.