PurePerformance - Perform 2020: Release Strategies and Bugs with Yoz Grahame of Launch Darkly

Episode Date: February 5, 2020

Coming to you from Dynatrace Perform in Las Vegas another episode of Pure Performance Perf Bytes. Coming to you live from Paradise, Las Vegas. We're going to invite the disembodied voice of our host. Coming to you live from... See, we actually have a viewer. So ladies and gentlemen, we have chairs this year for an audience. Yes, you can sit right here, and you can view things. Yes, congratulations.

Brian, come on down. I just want to check Yaz's audio. So, let's introduce Yaz Graham from Marsha Lively. Yaz Graham, welcome to Pure Performance PerfBytes podcast necessity. Thank you so much, Mark. It's great to be here. Welcome to Diner Trace. Have you been to Diner Trace Perform before? Never. This is my first. I'm enjoying it greatly

so far. You like it? Definitely on the food, superb. It is good food. It's remarkable. Now, did you have one of the egg slut egg sandwich things this morning? The queue was massive. It was around the block. They're really messy. Be forewarned. Okay. You will end up wearing

the egg slut home with you. I can see that. The yolks. But similarly, it's been fabulous meeting the Dynatrace team and talking especially to Andy

and other folks I'm sure audiences of this podcast already know. Very, very cool. Now, you work for an interesting company called LaunchDarkly. That's right. Which, explain to us a little bit, where did you guys get started? Where did LaunchDarkly get its going? So, it started in around 2014.

Our founders were two founders, Edith Har Harbour and John Kodamal and they were looking for something new and interesting to do and they found feature flagging was something that is conceptually incredibly simple and yet huge numbers of teams don't use it

because it's so simple it's almost insultingly simple and it's easy for people to overlook the power that it has. And once they started going down that road and pushing it, they found remarkable stickiness. Once teams start using it and

seeing the power that it gives them, more people want to use it. So, launching Darkly, meaning the idea of a launch, your feature flag dark. But there's also, there's Scanner Darkly, is the idea of a launch your feature flag dark. Exactly. But there's also, there's Scanner Darkly is the movie that comes into my brain. Oh, yeah.

It was a book from Philip K. Dick. Philip K. Dick, yes. That book kind of left me paranoid as I read it. Well, if you launch with your feature not dark, then you should be paranoid. Or you're just going to crash. Or you could be paranoid thinking, oh, my gosh, someone's going to turn it on. I forgot to turn it on. Either way, any time you launch, you're going to be paranoid. Or you could be paranoid thinking, oh my gosh, someone's going to turn it on. I forgot to turn it on. Either way, anytime you launch. Role-based access control. Now, near and dear to our hearts is

testing and specific to launch darkly. I find there's still a lot of people who presume the simplicity of a feature flag. It's so simple, I don't need to test that. But you start putting a couple hundred or maybe a thousand feature flags together, and you want to start, oh, I need to turn a bunch of them off. There's a whole elegance to once you really adopt the practices and start implementing it. Is that something you can tell us about from your perspective? Sure, and the trouble is that you can swing from not caring to over caring very easily.

Once something, you start to realize, what if this flag is on and this flag is off and this one, and you start to see this combinatorial explosion. Yeah, matrix. Yeah, the number of permutations and combinations explodes dramatically. Exactly. And our advice is don't try and test that. You test flag on, flag off.

Unit test around the individual flag one at a time. And that's the flag on? Yes. Visual jokes work wonderful. We have cameras if you didn't get that visual. I have no idea where the camera is. I'm not sure if I should be...

They're over there somewhere. Beautiful. So yeah, but then there's also the, if we didn't test this, I go to turn a flag on, and it didn't actually turn on, and maybe there's even legal or compliance issues or major functionality issues, or you have group dependencies that you're like, I turned two on, but we've never turned all three of these on

at the same time, blah, blah, blah. Do you run into a lot of that? Sometimes, and it is. Do you run into a lot of that? Sometimes. And it is something that you run into, and it's something we provide tools to make that easier to avoid the accidental slip-ups. Right. So, for example, you can link flags together to say one flag is dependent on another, right?

If this flag is off, that means this flag never evaluates. Right. But also, you want to be careful, obviously, in the same way that you want to be careful who has root access on the servers, you want to be careful who has access to flip certain flags. Right? Yeah.

My boy. All the permissioning. All your identity piece. Yeah, yeah, yeah. And some of it you may even not want humans to touch at all. So we do, and this is something, well, we're at Dynatrace, right? If you have your monitoring system, Dynatrace, noticing... No others, just us?

No, that's the only one in the market. Right, right. If Waitful, everybody else catches on, it'll be huge. But if something starts going to the bad place, being able to have rules that automatically go via the API, flip flags automatically. We call those automatic circuit breakers. Yeah.

You know. Which is a much sort of higher level of abstraction to make those adjustments versus some of the things we think about infrastructure-wise in the Dynatrace world, very low level. Oh, I'd throw some more CPU, throw another instance, maybe elasticity or, you know, hot swap memory in and out or things that are much lower in the platform where a feature flag

could be like, let's just not sacrifice whatever that initiative was and we'll pull that out and pull that out. So in this case, you could dramatically simplify an interface that's under load in order to service larger numbers of users. Absolutely. Because all those features could be using up more resources. Exactly, and it's a great thing to be able to do when, you know, larger numbers of users. Absolutely. Because all those features could be using up more resources. Exactly. And it's a great thing to be able to do when, you know,

especially your site has, you know, let's say Black Friday, which I know is a popular topic. Of course. Actually, you know what? The 20 JavaScript includes that we have on the front page, maybe we should drop a few of them right now. I've got one. The monitoring agents for all

of the beacons for marketing. There's usually like 15 or 20 of them on every single page. Pick one. It's awful. They multiply by themselves. It's like, if one's a good idea, five must be a great idea. We'll get five times the money in marketing. Or you can

turn off rum. No, don't turn off rum i know sorry dave um so question with the the feature flags now can feature flags be used in combination with canary like can you say i want to do a turn on a feature flag for a subset of our sessions yes infrastructure sessions right exactly maybe not the same exact canary, but it's not an all-on, all-off. Exactly. You can release it, roll it out, and then turn it on more. And there's things that we do to make that very easy,

depending on how you want to slice it. I mean, you could just literally say, I want 5% of users. And we literally give you a slider that you can drag along. And one thing that is useful there is that we keep track of which users fall in that 5% so that if you change that number, those 5% stay in there. We don't just randomly scramble it.

Tell us a little bit about sort of the install base currently, the customers that are using LaunchDarkly. And are they at different levels of expertise? Are there some full-fledged infrastructure? Are there some no-ops initiatives around that? It's all over the place. And we have something we find a lot of as well is that just because an organization has taken on feature flagging, we have some really big ones, we have some quite small ones.

We find other parts of that organization coming to us three years later after this customer has been a huge customer. They're going, feature flagging, I've heard of that. Should we start using this? You're in the same organization, right? But it can be very siloed. And people use it in different ways. For example, one use that fascinated me when I first saw it is basically looking after entitlements.

So if you have specific users, they say, this is a premium user, they get these features. And this is a base user, they get these features. And this is a base user, they get those features. That we can manage that as well. And that's the kind of bizarre thing that I did not think of. It's, as an engineer, I've been an engineer for 25 years, and the ideas were difficult for me to adjust to.

Especially looking at that one, just going, no, I should put an extra field in my customer record, in my database. Why am I giving that to you right the answer is because it is usually harder to do a database migration oh yeah yeah than it is to simply just throw that so taking that down to the brass tacks we travel a lot some of us are premium members for marriott and other locations. So depending upon our premier membership level, we might have different features that are enabled

based upon, hey, your gold, your platinum, your... Exactly. And you can manage that. So if they say, oh, you know, James is premium, but we don't like him, we're going to turn this feature off on him, they can do that easily via automation, depending on, like, he rage clicked.

We don't like that. I get that all the time because I'm from the south. Or, like, in Best in Show, you end up sleeping in the supply closet on a cot, right? Exactly. That's your level, status level, that's what you get. So it's really interesting then. Room service is right next door. It's just not feature flagging for new functions and features for the entire site.

And you're saying this was not, it wasn't the primary, like, it was like, oh, wait, we can use this in this way as well to, like, do customer features. Right. And especially because it means that the coders don't have to change any of the logic.

Right. None of that logic needs to be added to the code because the feature is wrapped in something that just says, evaluate this feature for this user. Should I turn it on or not? Or not. And we have, you know, you can put,

change all the rules on our side, and especially one of the most useful things is that you can change who has access to that. Because for something like this, for entitlements, it's your customer service reps who are going to really want access there. And being able to integrate that into their tools. And it's a huge amount of the power here,

especially when it comes to separating deploy from release. That's the basic. When you think about, if you have deploy and release being the same thing, it's ops who has the control. It's their timetable that something gets rolled out of. And when you separate them, then product or marketing or whoever it is gets that change. We had one of our biggest customers at Atlassian.

Oh, sure. They used us a few years back during their conference, Atlassian Summit, to roll out new branding. They had changed their branding. Oh, yeah, yeah, they did. Oh, so they revealed it during the show? They revealed it. Feature flag flipped. Everything updated.

All of a sudden, just refresh the page? Sometimes. In fact, depending on how you integrate it, you don't even need to refresh. Yeah, yeah, it'll just pick it up. That is very cool. One question I have for you, just in specifics for Dynatrace, I think about

myself as a classic infrastructure or performance person in the APM world or now part of a platform team. Feature flags are something that comes at us. How do we support them? Once they start being adopted,

at least for me, it was like, oh, that becomes my go-to answer. Like, don't put it on us. Stick it in a flag. Do you find the adoption rate changes as people start working with your solution versus kind of doing it manually the old way? Yes, generally.

In practice, I mean, as you say, it does depend on the developers and the team and the way they start rolling it out and the degree to which they get it, the degree to which they're allowed to try new things. But yes, once they start seeing that power and especially being able to... It's a great counter to the whole DevOps.

You want developers to feel the pain, right? Fundamentally, DevOps is about pushing the pain as well as the incentives. Right. And with feature flags, they get a very quick answer to pain. Yeah. You know? Yeah. So that incentive as well is huge for them.

And also maybe it's almost like it's high-touch interactive. Like I flip the flag and it's happening now, not I sent it to Brian and Brian did the package and then the DBAs did the thing and it got sent over here. And then that was a week and a half at 3 in the morning and I'm not really high touch, in touch with, you know, the change. But now they get like immediate and it worked and all of a sudden you're like, look at the revenue.

That's not going to your bonus or my bonus anymore. That's going into their bonus, which is awesome. I also had a question in terms of, obviously this is a production, but are any of your customers using this in your pre-production, your load testing, so that they can test flags turning on and off while performing all these load tests and make sure everything functions? And it's good to be able to do, I mean, especially because a fundamental part of it

is that when you're using LaunchDarkly, we separate environments. Okay, sure. So you can say, like we do this for a lot of our work, is that each developer gets their own LaunchDarkly environment. They have all the same flags that everybody else does, but they can then flip themselves

to see how in their own environment it works. So they can then flip themselves to see how, in their own environment, it works. So they can run their own little unit tests and flip the flags. So yeah, obviously that would be really important when we talk about the expanding scope, because we have the performance testing backgrounds and all, and as you move to the cloud, what you have to look at and test out changes.

So when you add feature flags in there, who wants to see everything come crashing down when you turn a feature on, right? So that's something you really need to test, even though it should seem, oh, it tested well on its own and well on, well off, but what about that switch time? Yeah. And you can do the performance. I mean, one of the scenarios I really like

with the performance testing that both works in test environment or perf environments and production is that you can run an experiment. The scenario I'm sure many people are familiar with, you have three engineers on the team who have their own very fervent opinions about what is causing the problem and how to improve Perth,

and they're radically different. And now you can say, you know, we'll try all three at the same time. Yeah. Right? And we'll link it, the custom metric, to whatever's working, and we can even expose that to production

and see what happens. Yeah, so clearly four stand-ups ago, when custom metric, to whatever's working, and we can even expose that to production and see what happens. Yeah, so clearly four stand-ups ago, when you guys said you understood each other, and everything's going to be fine, click, click, clearly you did not understand one another. Right. There was a sliding disconnect in between. That's blocked, and that's blocked, and you can't do that.

James, you look very different suddenly. Yeah, this is our colleague Leandro Melendez. Hi. Hello there. Nice to meet you again. Yes. Yeah, I was out of camera growing a mustache a little bit. Good for you.

Right. And a lot of hair. Not so much. Yeah, not so much. But, yeah, like when you mentioned those multiple changes and multiple scenarios and trying to cover too much at the same time, I often think of it like having multiple haystacks looking for the nil, and it's madness.

It is. It is. Anything you can do to very quickly either validate or disprove any hypothesis helps. Yeah, and it should be attempted a little bit like cut it in half, check one, one. Okay, so we're here, half, half. You need to go little by little. Yeah, just makes it much easier to accelerate that investigation process. You know, drill it in here and then try this without having to like recompile it, rebuild it, repush this.

Then, oh, just push it to these two nodes and then I'll do it with a load balancer. That's complex. Now you don't have, it's because that's the these two nodes, and then I'll do it with a load balancer. That's complex. Now you don't have, it's because that's the edge. You don't have access to the security of the load balancer. Now it's like this is fully secured, fully pushed code. You're just changing behavior. I really like it. I like the sort of sophistication.

What would be the most advanced thing you've seen somebody do with LaunchDarkly? Good question. I have to confess, so part of that is to confess that I have, in the five months with which I've been at LaunchDarkly, I've mainly been trying to get up to speed with the more normal, regular uses. There's...

I haven't seen that much in terms of what developers have used it, what existing customers have used it for. A combination of the things that we've discussed so far. We've seen things where, certainly because you can deploy on the client side as well, that you can have deployed software that is even part of real-time software running in hardware

and radically change how that's working right in the middle of something. Without having to dig through control panels, configurations, or an API call, or dreaded console, doing all that. Oh, yeah, so there is a super management flair to it and how you apply it. Does that also mean you could bake it into a mobile app? Sure. And enable features on the end? Absolutely.

We have Swift, Xamarin, client-side JavaScript, React Native. And the thing about them, one of the things that's really useful is that when you flip the flags, when you do a flag evaluation, you're not reaching out to us and requesting a flag eval every time. The idea is that when the app initializes, it downloads all the data it needs so it can do its own evaluation. Sort of a manifest of... Exactly.

But it holds that connection open to us. So anytime flags change, somebody flips a flag or changes a configuration, we send that change down to everybody within a couple of hundred milliseconds. And on the client side, that can trigger an event, right? Which you can handle any way you want. But you can say, for example, write re-render the page. So that's what I was saying about Atlassian. You don't even need to reload.

Yeah, and it could trigger things or pop up approvals if the flag is we've changed something and you need to now approve your ability to do this change. Like an opt-in status, you opt-out status. Yeah, things like that. I had a lot of fun. When I was first applying for the job, this is actually something I still need to tidy up.

I did an audition video screencast thing that I wanted to just try experimenting with it. And I attached, I found a music mixture, a JavaScript music mixer. And with a song that had all the separated tracks. And so I just had created feature flags for each track.

And then just, you know, while it's playing, turn various tracks on and off. Trombone? No trombone. Banjo? Did you dance during it? I did. Jiggling in my seat a bit. It was quite a bit of a track. You got hired. Done. It works.

It was the banjo. It's the banjo. Exactly. Everybody loves a good banjo solo. Yes. Electric banjo solo. Oh, yeah. Bail a fleck, heat your heart out.

I need to... Exactly. Because I think it's time in our interview, I've been dying to find out more about starving horses. Can you tell us about why horses die? Starving virtual horses die. Virtual horses die. In Second Life, did they have a glue factory?

It wouldn't surprise me. You know, the thing about Second Life, like the Internet, all of human life is there. The good and the bad. It's remote. I was at Linden Lab doing Second Life for five years. I learned a whole load of things,

especially about working with legacy code. We had some amazing bugs. The great thing about virtual world bugs, or in games or things, is that you see real world analogs, such as virtual horses,

doing things that really shouldn't be happening. But we had things like, I remember once, a developer accidentally raised the water level throughout the world by 200 meters, right? In one release. Oh, wow. There was this, and the bugs that swim in horses exactly um in this case and this is a really prime example of in terms in terms that you can't

that there's you can't test production unless you test in production right yeah um that uh second life has a built-in scripting language language called LSL that was made in a weekend and looks like it. And the thing is that it powers the economy, right? Because people build all kinds of products and sell those products and they script them, right? There is enough money going through Second Life to fully fuel. There are enough people who make their full-time living off making products and selling products in Second Life.

First of all, that just blows my mind. It's astonishing, right? So how many businesses were floated then? How many, sorry? Businesses of these people were floated when this happened. Were floated. Yes. Floated, yes.

There was definitely quite a lot. A lot of them went underwater, unfortunately. But there was... Thanks, go on. I deserve it. There was a business, for example, selling what they called breedables, virtual pets, right?

And in this case, it was quite... Breedables. You know, it implements the breedable interface. Right. And this was something that was very popular. And in this case, one of them was doing fairly realistic-looking virtual Arabian horses, right? And the business ran on the xerox model right in the sense that you buy the

horse but you also need to buy the food the food and the maintenance and everything yeah you keep buying the food what happens you put the food out and the horse is somebody had scripted a path finding algorithm okay so it would find its way around your land, wherever it was, to eat the food. Now, when you have a platform on which there's literally hundreds of millions, literally hundreds of millions of user-created scripts on your platform running your API,

you have no idea what any change is going to do, right? So, at the time, somebody made a change to the physics engine that slightly tweaked how friction worked with something walking on land. Which affects the pathfinder. Which affects the pathfinding, which means that the horses started sliding past their food, right?

Oh, my God. They couldn't get to the food to eat. Well, they could get there, but they just keep going. They were sliding through. And sometimes sliding so far past that they'd go off the skybox or sky platform. They'd go whinnying as they pirouetted into space, right? Now, this sounds hilarious because it is hilarious. There is no way in which this is not hilarious.

However, for some people, and when you realize that you are endangering, as it was, literally tens of thousands of U.S. dollars worth of virtual assets. Virtual horses. Horses, not asses. It's a different species. Exactly. There you go. One for you, Mark.

That we had QA literally up all night saving virtual horses from starving to death. God bless them. In fact, right now, if you just spend a few dollars and send us your donation, Gauze and I will save the horses. Just think of how many we could do with your help. When you went up those songs like, they may be polygons to you you but they're flesh and blood to us and it's virtual

money so spend all you want well it's the and with that one of the great lessons of learning was always

and this is something we know so much of in software that complexity builds and every change has an effect and the more

complex it is the more the changes are bizarre there was I mean certainly one of my favorite other bugs, and that doesn't really involve virtual things, but it's a wonderful, it encapsulates the history of software.

Excuse me. The Second Life client, it was a native client that ran on desktops. Right. And when it started up, it would check for updates. And the update, obviously, we hacked up our own update checker that would check a URL and look for a binary there and then download it, right? Now, one day there was no binary there.

One day it was a 404. Unfortunately, the code was not checking what the thing was to download. It downloaded the 404 page. And applied the 404 page. Well, it tried to run it. Now, in Windows, in Win32, if, you know,

there's a standard Win32 XE header. You know where this is going, don't you? However, to maintain backward compatibility with the DOS days, there was no com header, right? A.com file. Right? So, it turned out if you ran this thing, because it, again, wasn? So it turned out if you ran this thing, because it, again,

wasn't checking the values, if you ran this thing, it actually ran and would send a whole load of garbage to the LPT port, which is the printer. And sure enough, the printers are also not checking what they're getting. So in some cases, we literally actually broke some inkjet printers that were getting garbage that they could not handle and tried to execute and just bricked themselves. In the Microsoft world, you're not alone as a company that ran into that. And then, of course, they removed that extension. But it was fascinating to me.

That's beautiful. You got kind of the history of computing kind of going through that and multiple lessons of always check the return values of your system calls. That's amazing. You receive what you expect to receive and by all means never send something to a printer. Even if you mean to, just stop.

You don't really need to print it. If I'm not signing it and now I don't even need paper to sign, forget it. You don't need to print anything. And it hasn't had paper on it. And again, if you can just send $5, we can save a lot of trees. No, I'm just kidding. Thanks, guys, for being part of the campaign.

Anything I can do. Well, this is enlightening from a bug. We kind of love the bug thing. And of course, there's a lot of analysis right now on the performance issues or the massive bugs in the Iowa mobile app. I think that's going to probably be very high profile. When you see people who know nothing about computers start talking about bugs in mobile apps, then this is back to the website that shall not be named in 2013, was it? Oh.

2013, yes. Well, this is interesting because the... Healthcare dot whatever something. I forgot what that was. It was fascinating, the story, because part of my time, previously before LaunchDarkly, I spent four years in federal government.

Oh, yeah? In a group called 18F. Oh, yeah. Have you heard of this? Yeah, yeah, absolutely. So we were, and still are, teaching the government, or parts of government,

how to do agile, open source... And it was a big thing that grew from that around the healthcare, HEA, and still are, teaching the government or parts of government how to do agile, open source. And it was a big thing that grew from that around the healthcare HEA and the website and all that. Yeah, it was certainly more like USDS, the US Digital Service, that did more of the stuff on healthcare. But the stories that I think I may be distant enough to be able to tell. Sure. There's some, you know, things that, bizarre decisions that were taken by the original contractors. Yeah.

That people going in had then to clean up. Yeah, yeah. You know? And particularly stuff like, sorry, this is now going into bug stories continually. No, it's fine. But it's Jez Humble's world as well. Yeah, yeah. And Jez and I worked together at 18F.

Yeah, yeah. Which was fabulous. I learned so much from him. Yeah. Oh, yeah. Which was fabulous. I learned so much from him. Yeah. Oh, yeah. The census is coming. That's the next big one. Oh, yes.

Yeah. That's terrifying. That's not electronic, is it? Oops. Oops. Yeah. But it's always fascinating to see some of the Twitter traffic around this Iowa incident.

Right, right. And people saying things like, I could have built something better in a weekend. It's like, well, no. Of course not. Like the Second Life scripting language. It better in a weekend. It's like, well, no. Like the second life scripting language. It's just a weekend. Loadrunner, Dynatrace, I could just write that in a weekend, right? Underestimating the problem was the problem.

Grotesquely. Awesome. It keeps us all employed, right? That's right. We are sort of employed. And we're making cool new companies. It's a sad and happy situation at the same time for us. It keeps happening. Yeah.

It's sad for you. But it also puts it into the news. I remember when the first iPhone came out and both Apple's and AT&T's websites crashed.

I finally had a point of reference because I came from a performance testing role. I finally said, you see how that happened? My job is to make sure that doesn't happen.

Right. Right. Because I would explain to my family, like, I don't get it. And finally they're like, oh, okay, I get it now. So it gives you real world examples to share with your friends and family. My first job was, of course, stopping trains from running into each other. Of course. And it's really hard to find trains that run into it.

So I can't say to my family, you see all that town that doesn't exist? Yeah, I tried to stop that. It's still a kid, never really worked out. It's the Y2K problem, isn't it? The trouble is people talk about Y2K now, and you see continually there are a lot of, you know, talking heads who clearly don't understand who say, well, everybody made a big fuss, and it turned out to be nothing. Yeah, but then it came back in 2020.

Right. For like five customers. But at the same time, the reason it wasn't a big deal in 2000 is because they fixed it. Right. Exactly. A ton of people went and patched all the crap so that we didn't have to find out if something would break. The renaissance of COBOL. Y2K was just a legend.

$400 an hour. Not supposed to. It may or may not have been, but they fixed it all. It was a conspiracy theory so that they could get into our computers. Conspiracy theory. No, I don't believe that's true. Even the people I know that want to be nefarious when it comes to digital infrastructure, they look at something written like running a JCL on a computer.

No, I'm not. No, no. Even if there's great data there, no. So, you know, a big shout out to all the developers who put out code that doesn't have problems. Yes. Because they never get a big recognition. Yes. Oh, well, it worked.

Well, of course, you did your job. That's what we expected. The heroes. Yeah, that was a big thing in 18F to bring that back to that conversation was there were some really interesting precepts I remember hearing from Mike that he gave some talks about the healthcare.gov. He was kind of a lead PM. Mike, is it Michael something? I the healthcare.gov. He was kind of a lead PM. Is it Michael something?

Forget it. Sorry. Velocity Conference, I think he gave a keynote. God, his name is completely falling out of my head. That's all right. You might have worked with him. But telling those stories just about the human disruption, communication, policies, procedure, and just like, well, I didn't think to,

or I just didn't know, or nobody invited me to. It was all very human anomalies. Yeah, exactly, and it's in seeing the human costs, right? One of the bugs that I heard about was that families would go to hospitals. They weren't able to enroll online. They would go to hospitals. They would queue at hospitals for hours. They would try to enroll, and the system would basically give an error along the lines

of, sorry, not working today. Try again tomorrow. And they would try again and again, and it turned out that the problem was couldn't handle apostrophes in last names. Now, it makes you think not just about... They must not have tried that in New Jersey. Well, yeah. Or Boston.

You know. Accents, special know. Yeah, exactly. Accents, special characters. Right. But the thing that fascinates me is the various, this is multiple problems, right? This is not just that failure happening there. This is the error message getting lost along the way and somebody bundling all exceptions up into sorry not today no no no we can't tell the muggles about that we just we need to make it some generic thing

and 24 hours come back we'll have figured it out but then they weren't expecting to get a couple million errors and they're like well we need a lot more than a day to figure this out exactly and of course when i figure it out i don lot more than a day to figure this out. Exactly. And, of course, when I figure it out, I don't actually open a ticket in the system with you that has to fix the thing with the ticket for there, and then your ticket opens a ticket, and then there's three tickets to come back to me, but I put it in the wrong system.

Tiny haystacks everywhere. It's unbelievable. But now, quickly if I'm wrong, wasn't there, because you were talking about all this government side stuff, wasn't there not too long ago like a government conference where a lot of government teams were coming up and talking about their DevOps journeys and Agile?

Like some of these things are making progress. Yeah. Oh, yeah. Going back to the very beginning, not asking to give any customers or anything away, but are you working with any,

are any federal or state, any government levels looking at tools like LaunchDarkly yet? Yes. Okay, so they're getting it too. This would be a huge enabler for people to kind of come into the new century. Exactly.

There's definitely, I mean, digital transformation is a big part of it, right? Right. And the idea is that feature flagging is vital for certain operations, not just in terms of turning features on and off, but things like database migrations. If you're able to do, because you always have that thing with database migrations, we've got new database, old database, want all the traffic going there,

then want to duplicate all the traffic, then go through this multi-stage process so you don't lose anything. Now, if you're able to do that and move through those stages without ever having to redeploy, just literally just flipping a flag to take it to the next stage and the next stage and the next stage so um but the the thing about it as well i want

to say in terms of my experience in government um is that it's easy to buy that a lot of people looking to hype things up sure have talked about groups like 18f and usds and the work that and the work they are doing is fantastic yeah But at the same time, there have long been pockets of government all through state, federal, local, all levels, that are quietly but essentially competent

and working really hard to give citizens what they need. In a way, the bad rap they get is not appropriate. Exactly. And the trouble is, and I can see why they get the bad rap, especially America seems, you know, the whole, you know, I'm here for the government, I'm here to help kind of attitude. But it's, there's, and I think this is the same not just in government

but with any large organization. Honestly, I did not see that much more bureaucracy. In fact, in some cases I saw less in getting things done in government but with any large organization. Honestly, I did not see that much more bureaucracy. In fact, in some cases I saw less in getting things done in government than you would have in any multinational giant corporation. And at the same time, you have these little pockets of competence hiding out and doing something really well. And sometimes doing the innovation is not about bringing new people in. It's about finding the people already there who are doing stuff well.

Yeah, and shedding some light to it. Absolutely. You know, I've got to give a shout-out to two government agencies. First of all, Colorado has great online services. Registration, driver's license, renewals all online. Super simple, super easy. But even nowadays, like we have in new jersey when your kid is born you get a county birth certificate i didn't find out it was a county birth certificate until we

moved to denver and they were looking for the birth certificate i was like they're like this is not a valid birth certificate i'm like what do you mean it's signed by the state whatever yeah i'm like no we need the state certificate didn't know that one existed so went online you know uploaded a scan of social security card, uploaded a scan of license, put the thing in. I was able to do it online, which is amazing if you think about it. Fantastic, yes. And accepted, your thing will arrive in two weeks.

They just got to do the checks, whatever. Wow. And I was like, this was the thing where you'd be like, I have to spend a whole day. Take a day off from work. Go stand in line forever, get to the front. There's my form filled up, all that stuff.'s just like so amazing where some of these you know again some of them are just excelling phenomenally uh and i'm just just amazed by it and so happy as you were saying

uh it's a so simple quick experience but you don't have the visibility of all the people that was behind me but i know i know what they have to convert to, though, right? So I know where it came from. So I have that appreciation for where it was. And, yeah, obviously knowing a lot of people did a lot of work to make this just, like, simple. Oh, time to renew my license. Click, click, click. Do I need a new photo?

My photo is less than five years old. Nope. Go. And that was a big part of actually why I joined government in the first place. Because my second child was born and I went to I was very lucky that the Department of Social Security in California, that one of the offices was literally around the corner from from my place. But it's still we're so used to the experience of government being plastic chair and take a number. Right. And that being the day. Right. Right. And we know there's something better and everybody wants something better. Right. And we can have it something better and everybody wants something better.

And we can have it. We can actually have it. It does not have to be like this. Thanks very much for joining us. This has been fascinating. It's been an honor and a pleasure. Thank you so much for inviting me. Do you have any social creds where people can follow you? Maybe we can see some videos your audition video

I need to put the audition video properly I'm that kind of ADHD perfectionist where I'm going I love it but it's not nearly good enough let me rework it

three times over five years and eventually so I need to get over that LaunchDarkly is LaunchDarkly.com LaunchDarkly.com I am Yoz

Y-O-Z at LaunchDarkly.com I'm on Twitter as Y-O-Z Yoz and on Yoly.com. I am Yoz at Y-O-Z at LaunchDarkly.com. I'm on Twitter as Y-O-Z, Yoz. Nice to meet you. Nice. Wow.

On Yoz.com. The thing about having a name like Yoz is that you get – You don't have competition. You're wonderfully unique. Yeah. Excellent. So please do. Thank you so much for having me.

Thank you very much for joining us. Remarkably entertaining. Enjoy the rest of the conference here. I will do. You're going to come to the party on Thursday night very much for joining us. Remarkably entertaining. Enjoy the rest of the conference here. I will do. Are you going to come to the party on Thursday night? I'm hoping to, maybe on a plane, but if I can be there, I would love to. That's good. Awesome.

Great. Thank you so much. Thank you very much. Thank you. All right. See you. Yes. Thank you.