Odd Lots - This Is What Happens When Governments Build Software

Episode Date: June 12, 2023

There's a lot of frustration about the government's ability to build things in the US. Subways. Bridges. High-speed rail. Electricity transmission. But there's another crucial area where the public sector often struggles, and that is software. We saw it with the infamous rollout of Obamacare. We see it in the UX of the Treasury Direct website. And we saw it in the way state unemployment insurance systems broke during the pandemic. So why is it so hard for the public sector to build and maintain software? On this episode we speak with Jennifer Pahlka, the founder and former executive director of Code for America and author of the new book Recoding America: Why Government Is Failing in the Digital Age and How We Can Do Better, as well as Dave Guarino, who recently left the Department of Labor after working on upgrading the unemployment insurance system. Both have a long history of working on public sector software systems and they explain why the problem is so tricky.See omnystudio.com/listener for privacy information.

Today's show is brought to you by Vanguard. To all the financial advisors listening, let's talk bonds for a minute. Capturing value and fixed income is not easy. Bond markets are massive, murky, and let's be real. Lots of firms throw a couple flashy funds your way and call it a day. But not Vanguard. At Vanguard, institutional quality isn't a tagline. It's a commitment to your clients. We're talking top grade products across the board of over 80 bond funds, actively managed by a 200-person global squad of sector specialists, analysts and traders. These folks live and breathe fixed income. So if you're looking to give your clients consistent results year in and year out, go see the record for yourself at vanguard.com slash audio. That's vanguard.com slash audio. All investing is subject to risk vanguard marketing

corporation distributor. Thanks for listening to Odd Lots. Follow the show on Amazon Music for more future episodes or just ask Alexa, play the podcast, OddLots on Amazon Music. Hello and welcome to another episode of the Odd Lots podcast. I'm Jill Wisenthall. And I'm Tracy Allaway. Tracy, you know what I felt was a really interesting thing in a recent debt ceiling episode that we did was about all the coding it would take if the Treasury were to introduce a new kind of bill? Yes, that was interesting to me too, because I think there's a perception out there that if Treasury wakes up tomorrow and decides to sell a new type of bond,

they could just do it. But actually, there are all these back-end changes that would need to be made. And as you and I know from multiple conversations at this point, it feels like government technology can be a little bit clunky sometimes. Is that a fair way of putting it? It is. I would say, you know, in defense, I mean, get into this. Yes.

You know, corporate technology can be clunky. But that point reminded me that, like, we've done software is like a fascinating thing. And we talk a lot on the show, I think, about like the sort of built economy, like the Chips Act and battery investment and semiconductor. And all these things are like, can the U.S. build things again? Like as a government, state capacity, these ideas. But in the 21st century, also like building software and tech has to be part of that question. Right. So it's one thing to announce that you're going to do this new thing.

But almost everything that we do nowadays comes with some sort of software requirement. Like, I'm sure there is a system for processing applications for chips funding and things like that. And someone has to actually build that. So there's two questions contained in the can America build things question. It's like, can it build the actual things? And then can it build the software systems that would allow it to build the things? And, you know, in the pandemic period, we got sort of smacked in the face, I would say, by this realization that, the software component is not trivial at all.

And most notably, it was like every state had some sort of problem with their unemployment insurance system. And there were just such a surge in claims, obviously, in those initial, you know, really for that first year, just on the scale that none had ever seen. And it reminded people's like, oh, yeah, 50 different state systems, much with, like, tech that was probably built by some contractor and, I don't know, the 80s or 90s and the people who knew how to run it have left. And, you know, eventually, I guess it got smoothed down, but it's sort of exposed like, okay, why did that happen? And what else is lurking out there that we don't really know how it works until it's broken? I'm sure this is going to end up being a classic episode where we mentioned COBOL quite a few times. Like, it's inevitable, right?

It's coming. You can feel it. One day we're going to do it like an actual just like, what is COBOL episode where we just like focus on that directly. But yes, we're going to talk about software and we're going to talk about it from the public sector perspective. what does it mean when the government tries to fix something? What happens when the government tries to build something? What happens when the government tries to buy something? Some of these themes, we talked about them from a private sector perspective with Patrick McKenzie earlier in the year, but more to do on this topic.

Yeah, I'm excited. I have a lot. This is going to be a really good chance to actually ask the decisions that are going into some of these things being built. I'm still floored every time I log in. to the Treasury Direct website that, you know, you have to click the little, the buttons, like the actual letters and numbers to type your password. It won't let you just type your password. How did that come to be? I have questions. That is a great question. So we have, I think literally the two perfect guests to discuss those two guests with extensive experience

answering and working on exactly these problems. We're going to be speaking with Jennifer Palka. She is the founder of Code for America. She helped found the U.S. Digital Service. She co-chaired the California Task Force on fixing its unemployment insurance system during the pandemic. And she is the author of a new book called Recoding America. And we're going to be speaking with Dave Guarino, who's a number of things. Currently an independent consultant researcher focused on this stuff. Founding engineer at Code for America helped create Get Cal Fresh, which who has allowed Californians to easily access SNAP, has recently worked on unemployment insurance modernization at the Department of Labor, also worked in California.

So two people have extensive experience on exactly these things. So Jennifer and David, thank you so much for coming on odd lots. Great. Thanks for having us. Absolutely. Yeah, great to be here. Now I want to ask, Tracy, can I steal your first question? Yeah, do it. How does it happen that a government website doesn't

let you type in something and you have to like click buttons like that like resemble a mini keyboard on a website. What happened? When you see that, do you have like in your mind, do you're like, oh, I know exactly the meeting that caused this to happen? I think Dave has been in those meetings and has come out like with his hair on fire fuming. So I will let him answer that. I mean, not. Oh, well, I don't have. Yeah, I wasn't in that specific meeting. Maybe the best answer I can give is, and there's deeper issues behind it, is that the way. The way. they government tends to build technology. The reason for that is no one wrote down specifically as a requirement up front that people

should be able to use the keypad and the numbers on their keyboard to enter the digits. And so someone's like, well, I've met the requirements. Like the requirements are that someone can do this. They can enter it. And the way they do that is they click in this somewhat, I suppose, insane way. This is going to be a conversation who are like banging our head on the desk a bunch. feel like as we listen to these answers. Anyway. Yeah, get ready. Okay, well, let me ask maybe a big picture question to start, but what is the process by which government software is developed?

Because, you know, that answer just then, it makes it sound like someone gets a mandate. We need a website, for instance, through which people can buy government bonds, and then someone else goes off and I guess they execute it along the lines of the mandate that's been set. I mean, I'll jump in on that one. I mean, first of all, it is changing. Just before we get everyone sort of bending their heads on the table, there's a lot of movement in good directions. But what has been the process is exactly that. There's sort of this mandate.

It often comes from legislation or regulation that very frequently now, like says there will be a website that will even put the URL in the legislation. And then it kicks off what's a really long process that does exactly what Dave was saying. It is first centered around requirements gathering. And that process of just requirements gathering. can easily take 10 years. Now, when it's something like healthcare.gov, you know, and they have a launch date that's three years out, they don't take 10 years. But they still gather sort of every imaginable requirement. And they kind of throw it in one giant bucket. And that's how the RFP is

created, the request for proposal. That's what vendors bid on. And it's sort of like an undifferentiated, unprioritized mess of everything everyone could think of, plus things that have sort of pulled over from previous RFPs, plus a bunch of compliance stuff around security, compliance stuff around, well, various other things Dave can maybe chime in. But what that isn't is product management, right? There's no one saying, here's the important things this should do. And as Dave said, there's never a requirement for it to just work. So then the vendor, I mean, I'll skip several steps in the middle where, you know, it takes a long time to bid it out. And then there's like a protest because one vendor thinks the other vendor shouldn't have got it. And that delays it a couple

more years. But then they go into development. And their job, as Dave said, is just like check all these boxes. And it's thousands. I mean, when we were at California, state of California, working on the unemployment insurance, they were actually about to put out a business system modernization RFP. They were actually, I think about it to award it to a vendor. And I think it had seven, six thousand seven hundred requirements. We can check that. But it's in, that's really a normal number of requirements. And so they do all these things that where you can check a box, but what they don't do is actually check that it works. So I mean, maybe I'll just tell a quick story from the book. And, you know, there was famously this application for veterans'

health care benefits at the VA that didn't work outside the building. And they didn't, couldn't see it. So basically somewhere in the specs, it had that this form needed to work on a very specific and outdated combination of Internet Explorer and Adobe Reader. And the reason no one knew that it didn't work inside the building is that's how all the computers in there were set up. But if you were outside the building and had any other possible combination of those two pieces of software, it literally wouldn't load. And so they had very, very few people applying for these benefits online and you know veterans were really really really frustrated and it took a team going out there recording a veteran who had tried to do this dozens and dozens of times bringing that

video back and showing it to the deputy secretary for them to be able to say oh okay actually there is something to be fixed here okay you know you can go ahead and make a new form but up until to them they said sorry it's fine we're looking at the requirements the requirements have been met, there's technically nothing wrong. That's amazing. It's also kind of crazy to think that people would have been looking at that and just been like, oh, well, I guess demand for veterans benefits is lower than we thought it would be, but actually it was a tech issue. I think what they said was demand for them doing it online is low.

These veterans must not have access, which is absolutely not true. Or they can't figure out the computer when, in fact, it was other people who couldn't figure out the software. They would tell them, and they told this guy Dominic, the veteran that they interviewed, they kept telling him it's user error. There's something wrong with you. Can I ask a question about that bidding process? When I hear government goes out to bidders for a website, I imagine these like big sort of faceless offices in Roslyn, Virginia, where, you know, people get like chopped salad for lunch and stuff like that. And there's like three or four of them and they sort of like rotate who wins the bid and stuff like that. How competitive, like, is it, would a typical RFP auction or bidding process be? And how hard would it be for, say, a more agile, innovative firm or someone wanting to do it differently to break into the pursuit of that

contract? That's changing a lot. So now you've got a set of vendors. I think there's about 35 of them in this thing called the Digital Services Alliance, which are smaller companies that work in a more sort of agile user-centered way. And, you know, they're just a lot smaller than the Beltway Bandit. So they're still a tiny portion of the market. But, you know, they're really viable businesses and people in government can contract out to them. In fact, I think there's a mechanism by which you can put an RFP out just to those companies. But what has happened in the past is like, and Dave can pile onto this, like if you were bidding out, say, you know, benefit system in a state, and pick your benefit. Usually they would require in the RFP that the bidder had to have done a similar

system in a different state in the same category, which means that right there, it limited to like, three vendors. And so you were never going to get any disruptors because they were boxed out in the very first step. Just to maybe jump in, that's exactly right. And if you, I mean, to some extent, it's rational. If you are thinking about this as someone who's not a technologist, you run an agency and you're thinking, we need to do this very large project, it's going to take a very long time. We really need it to go right. It is hard to think about, well, why wouldn't I look for a vendor who's done this five, ten, 15 times?

The problem is, as Jen said, well, I think there's two problems in that. One is you've now very, very drastically narrowed your vendor pool. So there's a lot less competition. And two, you also may have too big of a project. Like really, what level of this traction are you trying to work towards? And is it the case that really you need someone or a vendor who's worked on? This specific type of system in another entity that, It works exactly the same way and all of those details of the exact same program,

the exact same structure of agencies. Probably not because probably it's about more generically software. But if you really focus only on the track record there, you kind of get stuck, as Gem was saying, in this trap where there's only a small number of vendors who can meet those criteria, even if you think they're rational. And it's hard because the other thing I would say is as much as there are new vendors, and maybe Jenny can speak more of this, but it's also the case that if you have a new vendor

that wants to work in a more agile way, wants to work in a more user-centered way, but they are bidding on an RFP that is structured in a very waterfall, top-down, meet all the requirements, and that's what success looks like. It's also not going to be successful, because you're not giving them any space to say, oh, well, I know your requirement says

a way to enter phone numbers using the key, the mouse. But when we tested it with people, like people turns out really don't like using the mouse to click numbers to enter digits. They prefer to use the keyboard. Could we change that requirement? And honestly, I mean, that's an extreme example. But a lot of, and there's a lot of what Jen's book is about and what resonated with me in it is that in doing these things and building software, you learn those details that you never could have gotten right up front. And so if you make success, defined by implementing everything as we understand it before we ever get started, you almost ensure that you're not going to get what you really wanted

because you're not allowing any information after that cutoff point. Right when you might be getting to the point where in doing it, you're going to learn a lot. Today's show is brought to you by Vanguard. To all the financial advisors listening, let's talk bonds for a minute. Capturing value and fixed income is not easy. Bond markets are massive, murky, and let's be real. Lots of firms throw a couple flashy funds your way and call it a day. But not Vanguard. At Vanguard, institutional quality isn't a tagline.

It's a commitment to your clients. We're talking top-grade products across the board of over 80 bond funds, actively managed by a 200-person global squad of sector specialists, analysts, and traders. These folks live and breathe fixed income. So if you're looking to give your clients consistent results year in and year out, go see the record for yourself at vanguard.com slash audio. That's vanguard.com slash audio. All investing in subject to risk Vanguard Marketing Corporation distributor.

Eating well shouldn't be complicated, but somehow it turns into recipes, prep, cleanup, and half your Sunday gone. Factor solves all that. These are fresh, ready-to-eat meals designed by dietitians, delivered to your door, and ready in just minutes. No prep, no cleanup, no excuses. And it's not just about convenience. You're getting real food, balanced nutrition, and zero artificial stuff.

Meals that help you stay on track for all of your goals, without the grind of doing it all yourself. Grilled chicken, roasted veggies, steak plates, postables. They taste like something you get in a restaurant, but they come out of your microwave in two minutes flat. If time, cost, or effort have been holding you back from eating better, Factor just took those off the table. Right now, get 11 meals, free shipping, and free sides for life. Hurry, this offer won't last long. Go to facturemeals.ca and use code fit. That's 11 meals, free shipping, and free sides for life.

but only with the code fit at factormeals.ca. Factor, Canada's number one ready-to-eat meal delivery service. I definitely want to ask you more about that top-down waterfall nature of how a lot of these software projects get commissioned. But before I do, I'm just curious about the actual vendors because I'm sort of imagining these big faceless offices. But what types of vendors are we talking about? And I'm trying to think how to phrase this kind of diplomatically. But I can imagine a company that is putting itself out bidding for government contracts on software development. Like maybe they're not as experimental or cutting edge as a software company that's, you know, somewhere in San Francisco and it's building like new, exciting products for the corporate world.

Although we've also done an episode on how bad internal corporate tech can be. be. But it seems like it might be a little bit more, I hesitate to say boring, but sort of basic playing it safe. Well, you know, one thing that I will say, like, I hear both incredible frustration and even anger from folks that have to hire these companies. They're just like there's no alternatives. This has gone badly. It will go badly. They kind of know that. Like I had this woman in a major state, I won't say which, who, when I said, you know, your, you're probably She said, do you think we don't know that? The last seven projects have failed.

So there's this huge frustration. But there's also this real feeling that the company in San Francisco with its 40 people who make consumer software, like great, people love using that software, but they don't understand us, right? They're not going to actually understand the constraints of government, so we have to go with these big companies. and they're not wrong in the sense that the constraints of working with government are real. There's a huge compliance burden.

That 40-person company in San Francisco mostly does not want these jobs. Like there's so much that goes into getting the bid. You know, it's sort of famously said that these companies know how to get the work, not how to deliver on the work. That's not probably what a 40-person company. wants to do. But, you know, it's true that you have to deliver on stuff that is mind-bogglingly complex. Like, when we're working in unemployment insurance again during the pandemic, my colleague was talking with the claims processors, like week over week, and we're trying to, like,

dissect it and figure out what's going wrong and, like, clear this backlog. And one of these guys keep saying, well, I'm not quite sure about that answer. I'm the new guy. I'm a new guy. I'm I'm the new guy. And she finally says, how long have you been here? And he says, I've been here 17 years. The guys who really know how this works have been here 25 years or more. So think about, like, you know, going from doing some simple, cool, you know, tech app, you know, easy consumer app, to trying to build or fix or improve upon a system that is so complex that it takes 25 years to learn how to process a claim. Yeah. That's sort of, I think, what needs to be on the table as part of this agenda is not just like, can the tech be better?

But can we go back and simplify the accumulated like 90 years of policy and process that's making that so hard to make? Well, why don't we sort of back up then? And I again, I'll kind of steal Tracy's question because I had the same thought. What is it about government buying that creates these massive waterfall RFPs in which you can't. in which A, like, create these constraints that the software developer would not necessarily have with a private buyer. And B, puts government agencies in a position where they will say to you, of course, we know it's going to fail. But again, presumably, that actually probably does happen in the private sector too. But what is, what are the, to some extent, but what are the dynamics that make these things like so hard to change and so hard to, you know,

rip out, you know, come up with a new system? Well, this is a good Dave question, but I'm going to start. You know, I really spent a lot of time reflecting on this. You know, I've had sort of three years since I stepped down from Code for America where I just like sat in a room and, well, I interviewed people and thought about it and thought about my own experiences. And I think that there's a deep-seated culture in government where the policy people are the important people.

They do the important stuff. And technology, digital, is just part of implementation, which is at the not just the bottom of like a software development waterfall. It's the bottom of a big rigid hierarchy in which information and power and insights only flows from the top to the bottom. And so it's problematic in part because the people who are doing the tech are really just sort of downstream of everything else. And the power and ability and willingness to step up and say, hey, like we probably shouldn't do those 6,700 requirements. We should probably focus on these 200, get that out the door, and then, you know, add edge cases as later.

Like, that's just not, there's no permission really to say that. And until we, if you compare that with, say, I'll call it metaphysical Silicon Valley, because I don't mean actually like Silicon Valley. Like, the people who I write code started the companies. They're in power. They're at the top. That's really interesting. Like compliance like is below them in, in sort of the DC government hierarchy compliance is like way above, right? Like policy is way above. And there's not this sort of build, measure, learn cycle that Dave was referring to where, you know, people are learning from each other as they're doing the work. And that technology, you know, certainly the policy has an impact on the technology, but the building of the technology needs to have an

impact on the policy. Like that fundamental culture is something that needs to be like, looked at and called out and where it's not happening, where there's actual conversation, you know, dialogue beats directives, right? Where that's happening, we should be lifting that up and saying this is possible within government because that's like the foundation, I think, of getting away from these mega projects that fail. But Dave, you're going to disagree with me on this and I'm going to love it. I think that's, I actually do think that's right. I think a big part of it flows downstream from budgeting, meaning a lot of this, the way that, and again, everything that I think I, jensate, like, there's some degree of generalization here, and there's always

outliers and there's always exceptions. And of course, there's also a lot of people trying to change these things. That said, maybe the dominant mode is we're going to budget for a large project. We get one-time funding to spend to do that project. So we got to get everything right. We're only get this chance once every so often, we're going to replace everything, we're going to have this new system, it's going to be great. And the way that this paradigm breaks down is in what's called a design development phase where you build the system, you design what it should do, you think about that. Again, you're kind of making all these decisions up front, which I think is very counter to a lot of the, if you want to call the Silicon Valley approach. But then you're building the system.

And then you enter what's called maintenance and operation. And it's supposed to be very, very little. You're not going to change much. It's just kind of keeping the system going. And the problem is modern software, you kind of learn the most the second you've actually deployed to real users. That is the point when you are getting people saying, oh, this doesn't work, or you're getting an error, or you're getting, hey, I tried to put in this address, but I have a funky address and it won't let me do that. So all of a sudden, you have all this information. And unfortunately, in sort of what gets called a Big Bang launch project where you just put it up and you're done and you're not going to touch again, you now, almost immediately on day one, can see potentially all of these issues.

And a lot of people prepare for this in these kinds of projects. They're like, the first month is going to be really hard. You all of a sudden know all the things that were assumptions you made that you got wrong. But the model that you had that you were given is, well, we're going to do this one time. Now, contrast that with actually going live is the first step. And you're going to have funding that's continuous and kind of flat over 10, over 20, over 30 years. and you're going to have those people on staff. And you're going to continuously change this system

as you learn more about how users interact with it. As you learn more about what things you didn't get right up front. You made it possible to upload a certain type of document. You accept PDFs and TIF images, TIFFF. This is a real example. But it turns out you don't support JPEGs or the iPhone proprietary default image format. So now all your mobile users are,

saying, hey, we can't upload, or a bunch of them are saying, I can't upload documents. It says I can't submit this file type. Right now, a lot of, you know, well-intentioned folks in government get to that situation, and then they're stuck because they don't have the mandate or the funding to say, now we're going to change that. They're waiting for the next big project to replace the system and now support those new things. Whereas instead, if you have a model where you're saying, we expect to continuously change this, it's a very, very different thing. And that's a little bit, that is more how Silicon Valley operates. I mean, this is, maybe a sort of hyperbole, but Google didn't build search and then lay off 90% of their staff

and say, we're done. This is great. You know, like, look, we got the search thing. And I think that difference flowing downstream from how tech is budgeted for, where you're budgeting for a one-time project. And this comes from legislatures, you know, from Congress. A one-time project versus we want to have ongoing fundings because this is a living, breathing system, is a huge paradigm shift and has an uncomfortable aspect, which is it may seem like it's costing more money because it's potentially dollars ongoing because it's staff, not one time by. But I think a lot of the dysfunction that we see probably flows down from that root cause. Okay. So here's a big question. If we can identify the cause of a lot of this dysfunction and if we can trace it back to this waterfall structure,

or the sort of top-down mandates or maybe like theorists versus practitioners. So you have a politician who has a big idea and they want that big bang reveal of the software. They're going to expand benefits and it's going to come with a really cool website and everyone's going to be able to access them easily. And yet it leads to these problems that we've been discussing.

Why do we keep doing it? That's a hard question. Well, what is the system that is like keeping this way of doing things in place versus maybe, you know, after a few decades of developing these types of systems, why isn't someone going? Well, actually, the way we've been doing things hasn't really been working. Well, I think a lot of people have been saying that. And that's why I say things are changing. And I will offer as evidence, COVID-tests.gov, do you remember that one? Beginning of 2021, the president announced that there would be this site where you could request tests. And I think he,

He gave a date that was, I want to say, six weeks out. It launched a day early. It launched in multiple languages. It took 11 seconds for me to use. Other people say eight seconds, 15 seconds, right? And your test showed up a couple days later. Like, that sounds like it could have been really, like, okay, so it's much simpler than, say, signing up for healthcare. Or whatever.

But, like, they could have made it way more complex. They could have said, like, let's gather every possible requirement. and try to fulfill every possible requirement. But you had internal capacity, to Dave's point. There are people in-house who were saying, hey, we have been doing it a way that hasn't been working. Let's try this way. And it's great.

Like, it serves as a fantastic example, I think, within government and outside government, right? So part of it is how people in government think what they believe. And part of it is what the public thinks, right? The public couldn't imagine something that easy, but we have to start to imagine. mentioning it and holding our government accountable to doing that. So, I mean, I will say it is changing, but the belief that we aren't good at it that we couldn't ever do, like a COVID test.gov, drives politicians, pundits, vendors,

to say, government's bad at this, and so we should outsource everything. Now, of course, we're going to have vendors, and we should have vendors. But there has to be enough internal capacity and competency to outsource well and to make those decisions like, hey, let's have this thing only ask, like, name, address and like, you know, not ask for their health insurance, you know, numbers and not, like, do different numbers of tests for household. Like, that's an internal capacity question that we, is really important and we don't have enough internal capacity. for these kinds of decisions in government because we believe we're not good at it and we believe the only people who are good at out of these outside companies. Plenty of evidence that that's totally wrong. Eating well shouldn't be complicated, but somehow it turns into recipes, prep, cleanup, and half your Sunday gone. Factors solves all that. These are fresh, ready-to-eat meals designed by

dieticians, delivered to your door and ready in just minutes. No prep, no cleanup, no excuses. And it's not just about convenience. You're getting real food, balanced nutrition, and zero artificial stuff. Meals that help you stay on track for all of your goals without the grind of doing it all yourself. Grilled chicken, roasted veggies, steak plates, postables. They taste like something you get in a restaurant, but they come out of your microwave in two minutes flat. If time, cost, or effort have been holding you back from eating better, Factor just took those off the table. Right now, get 11 meals, free shipping, and free sides for life. Hurry, this offer won't last long. Go to facturemeals.ca and use code fit.

That's 11 meals, free shipping, and free size for life, but only with the code fit at factormeals.ca. Factor, Canada's number one ready-to-eat meal delivery service. Visit Don Valley North Lexus for interest rate reductions of up to 3% for lease and finance rates, resulting in lease rates as low as 0.9%. Plus delivery credits of up to $1,500, only until April 30th. Visit Don Valley Northlexis.com. I want to go back to internal capacity in a minute, but since you mentioned the COVID test website, it's sort of a good chance to pivot a little bit here.

You know, COVID specifically seemed to open people's minds a little bit about how fast we can work and how much money we can spend. And obviously, a historically fast pace of vaccine development that we've never seen prior to that. And so it sort of had this brief moment where it sort of opened people's minds. I'm not sure if it's closing again. But, you know, since both of you worked on the California unemployment insurance system and generally, like, I think almost every state on some level is seen by some debacle, like what was, I guess, the biggest eye-opening thing that going into that environment taught you or about the system? And then what was like, you know, what's the key sort of takeaway of like, okay, here's the thing that we can learn from this, that then whether it's UI and other states or more broadly, okay, this is something that you learn. that then can like have extensible lessons. So I think you're right.

COVID had sort of twin impacts on our thinking. One was, holy cow, we can actually do stuff. And the other one was, holy cow, we're really screwed. Yeah, it's a good way to put it. And there's evidence of both, which is really fair. I mean, I think for me, you know, I'd been doing this for, you know, 10 years when I got pulled into the unemployment insurance. I'll call it a rescue. We were clearing the backlog. And I felt like nothing here can scare me. I have

seen the VA. I've seen the operating of, you know, some pretty janky, you know, government agencies. But I actually really did learn a lot and was kind of shocked at how the unemployment insurance system worked. And the metaphor that came to mind for me was like everyone kept saying, well, it's a system, obviously, you can just fix it. And it isn't a system. And I think that's true of other things we think of as systems, too. Like, it is archaeological layers of technology that sort of map to archaeological layers of policy that didn't ever get designed.

It just accrued. So nobody ever goes back and says, okay, how is this going to work with this? There's such little appetite for anything that's like backward looking that would actually update and make these layers work together well, that it just isn't what people think it is. When you go look at it, you're just like, whoa, of course you can't do these things. Like this thing isn't connected in any meaningful way to this thing. And like this was built in a certain era for a certain purpose. And we just glom stuff onto it when we needed to say add internet access. Let people apply online.

I mean, the biggest thing was, like, I think we haven't grappled with the fact that when these systems were quote-unquote design, I actually would say when they were started because they really have never been designed. And a lot of them, and there's huge advantage to actually doing some design. But you went into an office and showed who you were. Like, you identified, you validated your identity by going in. And then we moved online and we never figured out how to validate people's identity. identity. And there were a number of problems in California with unemployment insurance. The fact that, you know, it took 25 years to learn how to do it meant that the any claim that couldn't be

processed automatically was a huge, huge bottleneck. And you couldn't add claims process to do that. Like, you would have to like go back in time 25 years, start new claims browsers and have them ready to come online during a downturn, right? Like that's never going to scale. And, like, that's never going to scale. But the other big problem was that they only went through the automatic sort of pipeline if they felt like they could verify your identity. But they weren't really doing any meaningful identity verification. And we, unlike most other industrialized countries, don't have a national system for that. Like, we don't have a national technology platform for that. But also, like, again, And back to this all derives from like the legal and policy framework.

We don't actually have a reasonable policy framework for identifying people and knowing who they are when we start to do a transaction. And that is causing problems everywhere. And there is interesting conversations about how to fix it. There's some stuff out there. But it's really contested. And it's not just a problem for the pandemic. It's going to continue to be a problem.

Yeah. I mean, I'll just add. And my vantage point was with some. different, right? I was coming at it from a different angle. I think the major thing that I took away, especially with so much focus on technology and so much focus on cobal mainframes, and that was, you know, that was the dialogue. Oh, we need to do a cobal like drinking game or something. Take a shot. We need to time how long it took us to get to that word. I think it's been about 30. Oh, I can't believe, you know,

and the best part is that, I mean, my next point was that's misplaced causation. And yet, I am the one who brought up cobal. Yeah, I mean, I think we, that is a, you know, for so much focus on, oh, these tech systems are falling over, they're not good. I think there's, there's, the number one lesson I learned was that, and I think this is true, but we don't tend to think about it, which is the technology systems are part of a larger sociotechnical system. What I mean by that is, as Jen mentioned, you have staff who can do stuff manually, and you have, have a system that might be able to do stuff in an automated way. One of the problems with programs that are so complex is if you have a massive spike of volume and your technology system isn't set up to just cover all those cases in an automated way, you try to throw bodies at it and you try to

hire people. But if it takes a year and a half to train someone and like train someone to a relatively basic level with some of the complexity of these programs, you're not able to hire fast enough. and unfortunately, you know, this is why I, maybe it's a sort of historical material. We were hiring really fast. We heard 5,000 people in California to help process these claims, but because they couldn't do anything, they were taking up the time of the experienced claims processors, and we calculated that every single new hire the state made slowed processing down. Oh, man.

And one of the big things we did was actually just get them to reassign staff away. I mean, some of it was just reassigned staff to other things. Like nobody was opening the mail, which was pretty critical. And yet opening the mail is something you can do if you just joined and have no background. But like, yeah, it was pretty frightening. So sorry to interrupt, Dave. I was just like, we were hiring. It's just that the hiring was having a perverse effect.

Well, and that's, I guess that's my point is that's exactly right. And I guess the number one lesson I learned was if you look at the history of a program like unemployment insurance, but specifically UI, you have this thing where once every 10 years or so, there's a major recession or in this case, there's a pandemic, which was like a 10x recession, right? Because it was overnight, so much of the economy, so many of the working force, labor force, were just out of work immediately. So all those claims came in simultaneously. But we have this cycle of every 10 years or so we have a recession, or I mean, I don't know. I'm not an economist. I couldn't tell you exactly when that is. But it happens in a recurring way.

And then everyone gets frustrated with how difficult it is to get unemployment insurance because they're overloaded with volume. And then the claims volume goes down. Recession ends. And then there's a lot. There honestly isn't a lot of focus on, okay, we're going to sort of put more money into the UI system so that we're ready for next time. But unfortunately, you can't have resilient. systems if you optimize for pure, pure, pure efficiency, like in the down years, in the years when you have very, very low volume. If you are just going to cut staff down to very, very bare-bones

skeleton crew, it comes back to this point of not only are you not able to do preparation. If you don't have the funding to have staff, you can't even make the changes that would be good. And you can't, when you get the money in a new recession, hire the people because you can't train them. And so we kind of get stuck in this cycle. So that was the number one from a systems modeling perspective, less than I took away from it. I got to add, though, you know, we did invest several billion across states during the Great Recession or after a Great Recession. So I think about half of the states technically modernized then. They did take advantage of pretty significant funds.

None of those states did any better than average in this downturn. So I agree with Dave and I would argue that taking a different approach, to quote unquote modernization, to policy and process simplification, to, you know, having goal driven modernization, like all of those things is equally important as investing during the downturn. Investing the way we have been doing is not working. There are so many different threads to pick out of there. But just on that last point, you know, when you were talking about how a lot of these government systems were not designed to be that way. I kept thinking of a lot of the software used

by the banks where, you know, you think, you think of like J.P. Morgan's software system. It wasn't designed to be J.P. Morgan's software system. It's the result of many, many acquisitions of other banks. And, you know, every time they take over this business, it gets sort of duct taped onto the rest of the system. But I'm curious, is there a moment that you have seen in your current. at which someone says this system is just like irredeamable, we're going to have to start from scratch. And like, what is the trigger point for making that decision versus reaching for the band-aids and the duct tape and just trying to make it work? I mean, and I forget if someone else has mentioned this, I listen to the Patrick McKenzie episode, maybe he mentioned this, but in general,

a good rule of thumb is never rewrite a system from scratch that's working. I, I, I, I, I, I mean, other people might disagree with that, but the reason, I don't have a good answer for sort of a moment. Lots of people have tried to do that thing where they just rebuild it from scratch. And again, it tends to go poorly because, and this is the specific thing, the old system encodes so much tacit knowledge and so many of the many, many, many little details. And some of those details might no longer be relevant. But many of them might encode some really hard learned truth about the world because at the end of the day, the software is just modeling the real world. It's just saying, okay, well, we have customers, they have addresses.

Okay, so maybe you have two address fields in your old system, and you think, oh, well, let's just streamline that down to one. But then you launch your new system, has one address, and everybody calls you up and, what are you doing? Like, the reason we have two addresses is because our billing department is here and the customer facing address is different. So I think the throw it away and start from scratch is an impulse everybody has, but it has a lot of risk as well.

And I think the better approach tends to be, can you get to a place where you can start to incrementally change your systems as you go in small ways that are safe where it doesn't take it, you know, it's not like we have to test for six weeks, any change? change before making it live. We can ship weekly. We can ship daily. We can ship multiple times a day. That tends to yield better things. There are plenty of projects out there that were we're going to build this from scratch from the bottom up. Most of the ones I'm aware of don't have great launch stories. Maybe Jen disagrees. That's my. No, they're still in that mode of like get it all in once and then we're done and we have sort of maintenance and so they're pretty bad.

But I will say two things. One, on a practical level, just because I was talking with a colleague on the way over here about this, there are great examples now of what Dave's talking about, like not waiting for this Big Bang project, but saying, we are just going to be making this a little bit better every day. And one of them is New Jersey's Department of Labor. Their unemployment insurance response during the pandemic was quite good. But they've also learned from that and said, we don't. not going to wait for some like big system to come that's going to fix all of our problems.

We have a team now that can just incrementally fix things as we go. And, you know, eventually that will involve some bigger investments as they learn X, Y, and Z. But they're just not waiting. And it's just a great, great example. But I will say, Tracy, like, I think about what you're asking about all the time. I don't think this will ever happen, but if you do get a chance to sort of zero-base it and start over, you have to zero-base the policy. Like you can't, could you do a new unemployment insurance system in California that encodes the 25 years of knowledge that those claims processors have? Well, I mean, maybe AI will help you now.

I don't want to talk about AI, but like, but like that's a problem. Like, you know, I say in the book, there's this great team that's working on this Medicare problem, and it's like there's nine different definitions of even like what a medical group is. Like, even the first question doctors are supposed to answer is like wildly complex. And the policy, the delivery team is sort of fighting with the policy team. And she's like, I get that it's complex. It has to make sense to a person.

And I like, like, fight, fight, fight, not over the tech, but out of like over, over. collapsing these nine definitions into two. They want to get to one. They can't, at least they get to two. And like this idea that it has to make sense to a person, we've like lost track of that. I mean, of course you're frustrated when you apply for unemployment insurance. Like it's wildly complex in its policy and the process that accrued from those policies. Like if you were going to start over, you can't start over with the tech. You would have to say, okay, let's figure out. Let's Look at this 90 years of accumulated policy and say, what was this actually trying to do? Can we kind of either go back and just like rationalize all the changes, right?

It's not like I say, like there's not one binder of regulations that covers UI. There's 90 years of memos that changed the previous memo that changed the previous thing that changed the previous thing. And you could at least go back and like condense that and like, condense that. figure out all of the conflicts in it. But it would make, actually, I think, a lot more sense. And I will be clear, this is never going to happen. If you just said, let's design the policy for an unemployment insurance system that makes sense in 2023. And then you could build tech for that. There's so many different threads and places we could go. There's such a fascinating conversation. I just have one more question. And it sort of relates to something, Jennifer, that you said earlier about,

like in government, the tech people are like secondary likely to like the policy walks and people designing these things. And that's a really big difference between government, public and private sector. And, you know, this internal capacity, how much does that hinder? And I think this is another thing that Patrick McKenzie brought up, but like hiring. And so, and the challenge of like, you know, public sector pay scales are not private sector pay scales. I have to imagine for a lot of these things. he talked about a little bit, you know, what you've worked on, some of these, like, sort of volunteer things that sort of bridge the private and public sector. But can you talk a little bit about that constraint of like even if like just the challenge of in those, in those roles getting like experienced talented people? I'm glad you brought that up because it's a little bit of a soapbox I like to get on because we have, you know, like politicians, like the leaders who are so frustrated with delivery in technology.

Like, they're generally, they're well-intentioned, but the sort of things they push on are kind of unhelpful. You know, they kind of increase the risk aversion of the bureaucracy by yelling them about things and not recognizing the constraints on the bureaucracy. I call this the accountability trap. But there are things they actually could do. And fixing the civil service and civil service hiring rules is really, really important. I do know what Patrick McKenzie said on your show about parents.

scales is not totally, I don't totally agree with that. The biggest problem is not the pay. It's the time to hire. So if it takes, as is really common right now in federal government, nine months to make an offer, of course you lose everybody. You've got actually, you know, when we and I started this work, there was not like a line of people out the door with great tech and design skills wanting to work government. Now there actually is, and we can't get them in the door. I mean, I know people who do wait nine months for that offer because they really want to help. And the pay is fine. Like, it's not fantastic. I think he was comparing to, like, Google, whatever, but like, you know, compared to a startup where, yes, you have the hope of getting an exit, but you're probably not going

to make, like, $500,000, right? As a programmer, you're going to make, what, half of that. I mean, We don't have to get into like exact numbers, but like the pay is in the right ballpark for like mid-level folks. It's not going to pay at the executive level. But as he mentioned, like at the executive level, if you've been at Google for 20 years, pay is not your biggest concern. I do want to push back a little bit on the notion that that is he is in government. There are some of those people. But there are far more people who just have good tech and design skills who just want to make a difference more than anything else. And we should be working on all of the constraints that are keeping that hiring process at nine months more than we're like calling up the people who, you know, manage the Department of Labor in a state and yelling at them because they had these failures.

Like, fix the environment in which they're working, give them that capacity. Then you can hold them accountable. But they can't hire people right now. You also, you need to be able to hire people and you need them to be able to be able. to use their judgment to make higher order decisions than just do we enter the phone number with the keyboard or by clicking on a virtual pad of numbers? Like, it is also the case that some of these systems are the way they are because it's just, Jen, this might be your phrase.

It's like policy vomit. It's just the rules put onto a screen, just like all the rules in statute put onto a screen. And so I think the thing I'd like to add is the whole point of product management is to arbitrate the tradeoffs between user experience, compliance goals, technical cost of change and technical tradeoffs. And if you don't give those people, if you hire them, but they don't have the ability to say, well, maybe we need to simplify this to make it make sense to a person, if instead the only response they get is, well, we need to put on the page exactly what's in statute, then, or whatever other compliance goals there are, like, oh, we need to strictly follow this process, it's procedurally what it is. If those are counter to the outcome, you need a change in government where outcomes start to be, outcomes in this area around technology, start to

sort of have a higher importance than the strict process and the strict procedures. And I do think that that's a necessary corollary to bringing smart, capable people in. Because if you bring in a superb product manager and they're just, they have no ability to sort of bring these tradeoffs up and make those calls on the ground, you're going to get the same status quo, I think. So I can't resist asking this question. I take all the points about like, here's what we need to do in order for this to change. But can I just ask, because both of you have that, practical experience of doing this what was the craziest thing that you saw during your respective tenors like what was something that just really surprised to you or shocked you

I'm back to the banging our head on the table a portion of this conversation I mean I guess the thing that comes to mind just because it really made an impression on me and it's a negative story that I want to sort of balance out with something positive but I was working in the White House and had decided to do what we convinced the veterans administration to do what we call discovery sprint on the veterans benefit management system. And it was going really slowly. There was like huge latency. And I got these two folks in to help out for just a couple weeks and just, you know,

do an analysis of what was wrong before they go like, you know, trying to figure, you can't solve the problem until you know what it is. So, like, the first thing was we show up and the guy's like, oh, I'm glad they sent the White House to, you know, verify that everything's fine now. And it turns out everything was fine because the leader had defined latency as under two minutes. Okay, so you've just defined the problem away. So people who are trying to process benefit applications, like if they clicked and it took one minute and 59 seconds for the application to respond, it was fine. No problem here. Wow.

But then I kept talking to the guy and I kept asking him questions about the system. Like, well, why is it designed this way? Why was it designed that way? And he said, I've spent my career teaching people not to have an opinion on business requirements. The IT people should have no opinions there. And I said, why? He said, well, if they ask us to build a concrete boat, we'll build a concrete boat because then it's not our fault. And I was just like, I felt like I'd been building.

got punched. Like, it was so hard to hear that. Like, there were, I think, at the time, 18 veterans a day committing suicide, most of whom couldn't get their health care benefits. And I just couldn't understand how someone could take that approach. But honestly, you know, in the 10 years since then, I've seen a lot of reasons why he felt like he could say that. And, you know, I don't want to blame him. I want to berkeley in the system. But, yeah, if you tell us to build a concrete boat, will build a concrete boat? What way is that to run a country? I have a much more technical problem, and please yell at me if this is too technically in the weeds. So at one point, in my old work on SNAP, we observed a system, one of the official systems that was user-facing,

so external users like folks trying to apply for food stamps to use it. And we noticed one day that it wasn't loading, right, and it wasn't loading in the following way. You would go to to the website, it would have buttons and everything loaded, but then when you try to click on anything, it wouldn't work. And it wouldn't work weirdly for about a minute and a half. Like, then all of a sudden you click on things. And we're like, what is going on here? This is bizarre. And we looked around. And so we did some debugging on the client side, meaning like we opened up the web browser and inspector and we said, okay, what's going on? And what we found was this website was loading, I forget exactly the name, but it was like translations.js or translations.xm.

And it was taking a minute and a half. Everything else was loading fine, but this was taking a minute and a half. And it counter to best practice. It was blocking any interaction with the page until it was fully, fully loaded. And it was loading very, very slowly. And we looked at it and it was this giant file. I was like, I don't know, it was like 50 megabytes. It was a giant file. Like, what is going on here? looked into it and it was a translation of every page on the entire system into like eight languages and we call the thing and that was like okay this is not great and there's ways to fix that like I said you can let people interact with the page until before that's fully loaded and that's fine also you definitely shouldn't put all the translations for every page in your website on

eight pages into a single file that you send to the client like that's just bad development But the thing that maddened me is we being good-hearted people who are not out to, you know, just sort of lob bombs and talk trash. We contacted the entity who ran it and we said, hey, this is happening. We're seeing it consistently. This is the issue that's going on. And they said, no, no, no, no, that's not happening. It's loaded totally fine for us. And the reason is because we diagnosed it down to one or two reasons.

One was they'd already loaded the file once, so it was cache. And so it was on their computer. So every time they went to it, it was fine. And like, oh, yeah, it works for us. The other was, and this was our better guess, is their server that was serving this was on their network and actually sending it much more quickly. And they were just like, well, it works on our network. It works totally fine.

And I don't know exactly what it was, but I think that is the reason that's so maddening and it gets back to a little bit like Jen's point about accountability is, the truth is we can't fix problems that we don't see or don't recognize. And I do think that a big part of all of this is we have some expectations that website should work, that technology should work. We also need to give more, we have to set an expectation, I think, government-wide about what stuff working looks like. Like, Jen gave an implicit one of a normal human should be able to understand this. That should be tested on every system. In the, in the e-commerce world, like, what's the number one metric you care about. It's from start to finish, how many people start a transaction and actually end up

purchasing that conversion rate. That, I mean, it would be a really useful feedback loop to measure the success rate of users trying to do a transaction across every transaction in government, right? Like people applying for food stamps, people applying for whatever. If you saw massive drop-off or massive disparities across systems, at least it would tell you, oh, there's something we need to fix here. So I think there's another layer of all this. And that's my depressing story, because eventually it got fixed and I think probably they figured out we were right like a week later,

but it took a week and it was a very... I think that's why in my mind, there's also if we can get to better accountability and give people better, on the government side, better visibility into these issues with possible routes to solving them. They want to solve them,

but right now we don't have a system that necessarily looks for those issues. And I think that would be a better form of the accountability we have relative to say, did you follow every IT checklist bullet point in policy, right? Like, if we're getting bad outcomes while following that, then maybe something else needs to change. And maybe there's a different form of accountability necessary there.

We should give a quick shout out to the President Biden's customer experience executive order that is trying to get agencies moving in that direction. There is so much to talk about and so many different subversions of this conversation. we have. But that was an amazing conversation. Jennifer Polka, Dave Gorino, thank you both so much for our coming on Oblod's sharing your time and insight and direct experience and all this stuff. It was a delight. Thanks so much. Thank you. Tracy, that was an extremely illuminating conversation I found. I mean, it's easy enough to say like, oh, government is bureaucratic and they don't pay enough, and that's why they can't build software. But they were like, there was like so much richer and more

complex than I like appreciated it. Right. The emphasis. on incentive was really interesting and sort of fits into a lot of the discussions we have here. And also Jen's point about a lot of this emanates from the policy side. Yes. Yes. Right. And, you know, it's people trying to tick various boxes that have been set in stone by policy. I got to say one thing coming out of that, though, I feel a newfound appreciation for being a journalist. And actually, you know, when you and I produce an episode or write an article, we sort of send it out in the world and then it's done. And we can move on to the next thing. We don't have to go back and refine it in response to customer feedback and various testing and things like that. If we were, if we were Wikipedia

editors, we would have to be continually editing them for years. I thought that was really great. And again, Jen's point that you just brought up, it's like this sort of like reflection, like the real solution like has to emanate from the policy level. I was actually getting drinks recently with a friend of mine who's a software engineer. And he said there's like this phenomenon. He said it's called Conway's law. Like every organization he said like ships its org structure. And then it makes like so much sense. Like the software is complicated because US politics is like complicated. 90 years of policy changes and Democrats and Republicans switching who's in power and et cetera. It's like no wonder like the ultimate like product of

that thing is going to not look like, you know, Google.com. Well, also the story of the guy who was like, it takes 25 years to really know how to file. a claim, you can imagine if you spend 25 years developing expertise in this one very specific skill set, you're not really incentivized to start like changing that anytime soon. And so the system just kind of perpetuates itself. So many different directions we can go with that conversation. So many different ways the system is self-perpetuating. Yeah. I learned a lot from that. Yep. Shall we leave it there? Let's leave it there. This has been another episode of the Oddlots podcast. I'm Tracy Allaway. You can follow me on Twitter at Tracy Allaway. And I'm

Joe Wisenthal. You can follow me on Twitter at the stalwart. Follow our guests on Twitter. Jennifer Polka. She's at Polka Dot and she is the author of the new book Recoding America. Definitely check it out. Follow Dave Guarino. He's at Dave underscore Guarino. Follow our producers, Carmen Rodriguez at Carmen Armin and Dashel Bennett at Dashbot and check out all of the Bloomberg Podcasts under the handle at podcasts. And for more Oddlots content, go to Bloomberg.com slash Oddlots, where we post transcripts. Tracy and I have a blog and a newsletter that comes out every Friday. And you can talk about all of these things 24-7 with fellow listeners on the Odd Lots Discord.

It's really fun. I hang out there a lot. Go to Discord.g.g. slash Oddlots. It's a real blast and a fun place to hang out on the internet. Thanks for listening. The news doesn't stop on the weekends. Context changes constantly.

And now Bloomberg is the place to stay on top of it all. Hi, I'm David Gurra. Join us every Saturday and Sunday for the new Bloomberg this weekend. I'm Christina Rafini. We'll bring you the latest headlines, in-depth analysis, and big interviews. All the stories that hit home on your days off. And I'm Lisa Mateo. Watch and listen to Bloomberg this weekend for thoughtful, enlightening conversations about business, lifestyle, people, and culture.

On Saturday mornings, we put the pay. past week's events into context, examining what happened in the markets and the world. That on Sundays we speak with journalists, columnists, and key political figures to prepare you for the week ahead. Join us as soon as you wake up and bring us with you wherever your weekend plans take you. Watch us on Bloomberg Television. Listen on Bloomberg Radio, stream the show live on the Bloomberg business app, or listen to the podcast. That's Bloomberg this weekend. Saturdays and Sundays starting at 7 a.m. Eastern. Make us part of your weekend routine on Bloomberg television. radio, and wherever you get your podcasts.

What separates good leaders from transformational ones? I'm Jessica Chen, and in season two of Leading By Example, we'll sit down with executives like Grace Chen of Bertie Gray to find out. It's important to understand where you spike, but also really acknowledge where you don't and find people who can fill those gaps. Listen to Leading by Example, executives making an impact on the IHeart Radio app,

Apple Podcast or wherever you get your podcasts.