PurePerformance - Understanding the Power of Feature Flags with Heidi Waterhouse

Episode Date: June 10, 2019

Imagine a future where we deploy every code change directly into production because feature flags eliminated the need for staging. Feature flags allow us to deploy any code change, but only launch the feature to a specific set of users that we want to expose to new capabilities. Monitoring the usage and the impact enables continuous experimentation: optimizing what is not perfect yet and throw away features (technical debt) that nobody really cares about. So – what are feature flags?We got to chat with Heidi Waterhouse (@wiredferret), Developer Advocate at LaunchDarkly (https://launchdarkly.com/), who gives as a great introduction on Feature Flags, how organizations actually define a feature and why it is paramount to differentiate between Deploy and Launch. We learn how to test feature flags, what options we have to enable features for a certain group of users and how important it is to always include monitoring. IF you want to learn more about feature flags check out http://featureflags.io/. If you want to learn more about Heidi’s passion check out https://heidiwaterhouse.com/.

It's time for Pure Performance! Get your stopwatches ready, it's time's Pure Performance Podcast. I know normally you hear the intro from my dear friend Brian Wilson. But well, today I got the joy and honor to host a session without Brian. He has a lot of things to do today, but we'll be back the next time. Normally he opens with a joke. Well, I'm not that funny as he is. So I will skip the jokes today and I will go right over to our guest of honor. And today with me is Heidi Waterhouse. Heidi, I hope you are still with me on the audio line. I am. Awesome. Well, Heidi, thank you so much for

joining. I got the pleasure of meeting you, I think it was mid of April in Linz, Austria, my hometown, because you were speaking at our DEF ONE conference, right? I think that's the first time we ever met. Yes, that's right. And I have to say that was an amazing conference, and I have never gotten a fog machine and laser show as speaker intro before. I know they did a pretty good job with all the visuals. They put a lot of effort in there. So, Heidi, I mean, I got to know you and that was great. Some of the audience that are listening to this may have never heard of you.

So could you give yourself a little quick intro of who you are, what drives you, what motivates you, where you live, anything that people may want to know about you. Sure. My name is Heidi Waterhouse. I'm the Senior Developer Advocate at LaunchDarkly. We do feature flags as a service. One of the things that I'm really interested in is making it possible for more people to have control over what's getting sent to their users. I spent 20 years as a technical writer, and that gives me a lot of feelings about how it actually is to use software and to feel sort of helpless when you're trying to figure out how it works. So one of the things that drives me as a developer advocate

is making sure that all the time we're thinking about how it feels to use our software and how we could make that experience better. Cool. And then feature flagging as a service. So you said you work for LaunchDarkly. Not that this is going to be a commercial, but I think it's great that companies like LaunchDarkly exist to actually not only promote like what you do, promote the methodology, the approach of exposing features to end users in a more controlled way, but also having a product, obviously, that makes it easier. So in case people have never heard about it,

I think launchdarkly.com, or what's the website? Yeah, launchdarkly.com. Perfect. perfect now i really want to dive into the topic because uh just as you have the passion of you know helping people to figure out how to release better software understand how software is used and optimize that experience um and so do we right we are all about making sure that the software is performing well that the user experience is great. Now, feature flags. Feature flags as a service is something that you deliver. Can you tell us a little bit more about feature flagging?

What is a feature flag? For people that have never heard about it, is this a complete new concept? Or what do we need to know about feature flags? So feature flags are really a fancy if statement. But when I say fancy, it can get very fancy indeed. What we're doing is enabling people to control how their code behaves after it's been put into production. So if you think about how A-B testing works, where you try out two different variants of an interface, or if you think about how a progressive deployment works where you only release 10% of the product at a time,

or the product to 10% of the people at a time so that you can make sure that it's going to roll out cleanly without causing anyone problems. Those are both manifestations of how feature flags work. So what happens is you decide what a feature is, which is a non-trivial problem, like what size is a feature and what is a business meaningful unit to make a feature. And then you make that controllable after deployment. So you say instead of like, I'm, I've been in the industry long enough that I remember when we used to cut gold master CDs. And there wasn't like a feature, there was a product. But now that we have so much very flexible, very fast software delivery,

we have to think about like, well, what constitutes a feature and what should get controlled that way? And I think that's a really interesting problem because it's sort of the agile problem, like what is a story? But it's also what is a business useful story? Because you can write all sorts of agile stories that are not business useful.

Yeah. So are there best practices around, you know, how you come up with a feature or is this really aligned, as you just mentioned, with a story? Or is there any material out there on any approaches on how to figure out what is a feature and how do we encapsulate it?

There are several. I think probably the best book on it is called From Project to Product that just released lately. And I've read some of the preview material, but I haven't read all of it yet. But it's about what is the value that we're doing. As far as best practices for features, it's really useful if you understand that there are two feature flag models. And one of them is a permanent feature flag that you're going to keep on all the time. And that's for stuff like entitlements,

like which version of the website you're going to get because you have a membership and you're paying extra. And so that's an entitlement, or you might want it on a kill switch or a circuit breaker in your infrastructure in order to control what happens to the way your infrastructure behaves. You would leave all of those in place permanently. The other kind of feature flag is a temporary feature flag, and you would use that for experiments, and you would use that for deployment. But when you have made the decision or when you have finished the deployment, then that feature flag is going to get removed because leaving it in your code is just, it's a form of technical debt because you're leaving the potential for something to

get switched on or off that you don't actually want to have change anymore. So that's- So Andy. Go ahead. Oh, Brian, did we just sneak in? I snuck in about three and a half minutes ago. Yes. Oh my gosh. look at that. Sorry I was late. I didn't realize that I agreed to the time. I have kids I have to drop off and all that stuff. And there was some traffic and got late.

And then, of course, technical difficulties because computers are still our friends and love to give us trouble like our real friends do in life. So hi, Heidi. Nice to meet you. Hi. Nice to meet you too. Yeah, that's good. So Heidi, that's pretty cool. So permanent versus temporary feature flags. Now you said temporary feature flags will be

and have to be removed after a while because otherwise your code gets cluttered with a lot of, I guess, if statements. I mean, I think in the beginning we said, what is a feature flag and how do you technically implement it? And then you said, well, it's basically a fancy version of an if statement now does this mean that organizations have to

in their sprint cycles obviously plan for actively removing these flags or are there any let's say uh development tools any any tools that are automatically removing uh these pieces of code because i assume it's always I mean I could even see like as a macro if you go back to to languages that have macros where you can kind of block a certain piece of code with a macro and then you just say hey you know let's remove that or how does this technically work or do people go in and really remove these lines of code manually um you could use it automatically but I wouldn't suggest it because it would be so easy to get it wrong and to remove the wrong flag. So if you accidentally removed a permanent flag,

you'd be in trouble. Of course, yeah. So what we do is we have a tracker that basically notices where you inserted feature flag code. And so when you go to that flag and say, hey, where does this live in the code base? It pops it all up for you in GitHub and you can pull it out there. But not entirely automatically because we don't want other people touching our code. Like when you get right down to it, in order to allow that, you would have to let people

touch your code. And we don't really want external parties doing that if we can help it. Yeah. No, I thought more of a tool that might be integrated in your development process, in your IDE, like an IDE extension where you say, hey, we scanned the code and here's all the list of the feature flags. And based on the data we have from our feature flag as a service solution, we want to remove these five feature flags. So boom, remove it for me, or at least point me to the source code line. But I guess that's what you alluded to with the GitHub integration. Yeah, so that's cool.

So we can point you to the source code line. And because it's all API-based, you could automate that, but I don't think that's a high priority for us to automate because it's so risky. Yeah, okay. That makes sense. Hey, I wanted to, before going further, you know, in looking at some of the material ahead of time,

I just wanted to make sure the audience hears it because I think it really covers one of the main tenets of feature flagging. And I'm getting this from Heidi who sent over that link of featureflags.io. And one of the quotes on the getting started I think just really, really nails it. It's from the SRE Google quote. And I'm just going to read it because I want to set in the listeners' minds the importance of feature flags and how they can be awesome. So from this, it's by hosting code that supports new functionality in the client application before we activate the feature, we greatly reduce the risk associated with the launch. And having the sort of dormant functionality also makes aborting launches easier when adverse effects are discovered during a rollout. I just, yeah, I think to me that sums up a lot of,

there are a lot of things you can do with feature flags, but I think from putting a new feature into production and how you can manage it, that really nails it, I think. Yeah, and Google has been doing trunk-based development for a long time. So one of the interesting things about feature flags is that they are kind of a prerequisite for doing true continuous deployment. If you can't deploy broken code without anyone seeing it, you can't do continuous deployment. You have to stop until you know all your code is functional. With feature flags,

you can deploy broken code and no one will see it. So you can do continuous deployment. And when I say broken, not necessarily broken, broken, but like unready or unapproved or, you know, in some other way, not ready to be shown. Hey, then, I mean, for me, the biggest challenge then really comes down to, obviously, technically, we can easily solve it. It's, as you said, it's a fancy if statement where services like what you provide with LaunchDarkly allow you, give you an API that is then centrally keeping track of which feature flags should actually be turned on and off as code executes. But for me, the biggest challenge then really is figuring out what is a feature.

And I believe, especially in a, and correct me if I'm wrong, but especially in a more distributed world where you don't have monolithic code anymore, where you can touch a single code base to define what a feature is. Well, even in a monolithic code base, you may have different parts of the code. But in a distributed architecture, if a feature means turning on or off flags across different entities like microservices,

that becomes a real challenge. Or is this something you need to avoid in not having the same feature flag being used across different parts of the architecture? If your feature flag description, like when you're saying your feature flag out loud, you say the word and, you've violated one of the best practices. A feature flag should do one thing only. However, that said, you can chain together feature flags that say, this one will only activate if a top level flag is activated. So there's a way to do that, that says, like,

do a dependency check or do a version check to make sure that, you know that the environment is correct for this flag to turn on. So because my thought process was if I have, let's say, a backend microservice and I introduce a new capability, I don't know, a new algorithm for something, right? And then I want to expose that feature to my users on the front end. I need that feature flag on the front end, but depending on that feature to my users on the front end, I need that feature flag on the front end. But depending on that feature flag, either in the backend, then execute the new or the old algorithm. But as you, I guess, as I just in my own thought process, I guess what you would do is you would implement the backend service behind the new API version. And then the front end,

if the feature flag is turned on, says, do I call the old or the new API?. And then the front end, if the feature flag is turned on, says, do I call the old or the new API? I guess that's the better way of doing it. Yeah, that's the better way of doing it. And one of the interesting things that we've seen with our customers is that when they start using feature flags, they go for some pretty elementary, sort of the low-h hanging fruit, the obvious solutions. And as they get more comfortable with the ability to have different states of software, sort of like quantum superposition, they do some very interesting things with how their software works. And we ask, doesn't that make testing more difficult? Because

if you have this superposition of possible effects of how the software works, how do you do testing? And the answer that we've been getting is mostly that although testing is a little more complicated because they know what they're doing they can more precisely control it sort of like when you're a kid and you're doing a science experiment with ph you know that there's only this possible range and and because you know what the range the the possible range is you can do some very interesting things with it because you've like testing the pH is not going to go beyond the paper. And so you've, you've created yourself a safe space to test it. And we find that people who are using feature flags feel much safer because they're

like, okay, this is a place that is safe to test in. And if something goes wrong, I can turn it off. And I can put a bunch of analytics on it to make sure that it's behaving the way I think it's going to behave before I show it to the general public. Now, from a testing perspective, I mean, if I hear this correctly, what you said earlier, you said feature flags allow me to deploy broken code in production or let's say it that way not extremely well tested code in production for me that actually means if i do it right i can completely reduce or remove any pre-prod environments i only have prod and new code gets basically hidden behind the feature flag, which can then be tested in production

by my testers. And the testers could either be a testing team. It could be a group of people. It could be a group of end users that I'm exposing to this feature, that kind of my test group. And based on that, I decide, is the quality right or not? And then I can just flip the switch and turn it on for everybody. But if I do it right, if I get this correctly, then in the future, I don't need the traditional testing environments anymore because I just deploy the prod in a safe way and then slowly expose that new feature to a group of people and then enlarge that group.

Exactly. We have a customer who the way that they're paying for their license, at least partially, is they don't have staging servers. They're just not paying for a staging environment. They're doing all of their staging testing in production. And that's not the right fit for everyone. But I think it's a super interesting use case because they're like a software as a service company themselves. And that's a feasible way for them to work.

And our meetup in the Bay Area is actually called Test who you are, you can't replicate the full weirdness of production in a staging or test environment. You can't replicate microservice latency. You can't replicate the weirdness of user data. Like, no test suite has ever covered the fact that some people have a one letter last name appropriately. One thing that I wanted to kind of clarify here though, right, is when we're talking about, well, I'm laughing a lot when you're talking about this, because going back to my testing days and my performance testing days and all this kind of stuff, there was always the joke of, you know, production is the new QA.

And that was always laughed at. I mean, I'm going back seven, eight years ago because there was really no way to do this properly. And I want to clarify, though, that this is not talking about, and I'm saying clarify for the listeners, really, we're not talking about turning it on to the general public for testing right away. You're still going to employ things like maybe a dark deployment where it's turned on for the people with access to that section.

Maybe the testing teams themselves go into the production environment and run on a special set of knowing the proper URL. Or maybe you can then switch over to a shadow deployment and do it. So it's a very slow, controlled rollout where it goes through the ringer. It's not like, hey, let's turn that on in production and literally make the old joke true of production and our, our end users are the new QA. No, no, no. As funny as that would be, that is not how we operate. Yeah, exactly. So when Facebook rolled out the new feature where you could select your name, they spent months doing dark deploys and dark launches. And every time you logged into Facebook, they would basically do a simulated username request until they were sure their infrastructure was going to hold up to it and that they had worked all of the bugs out of the system.

So almost everyone has already participated in several forms of testing in production and not known it. Right. I wonder, Andy, too, especially even based on yesterday, the meeting we were at when you asked about who knows what a shadow deployment is, should we take like two minutes here to just kind of cover the difference between a shadow deployment is. Should we take like two minutes here to just kind of cover the difference between a shadow deployment, dark deployment, blue-green, just to make sure everyone is level set on understanding these components?

Because, you know, maybe not everybody knows. Yeah, well, I want to know what you guys are saying about shadow deployment, so... Well, I'll let you take it, Andy. Yeah, sure, and I'm not sure if I have the right definition, but this is the way I explain it. For me, a dark deployment, and I think that's also different between a dark release.

Dark deployments into an environment, for me, is you're deploying a new version next to the old one, but you only can access it if you actually know the direct access to that particular component. If it's behind the load balancer, but actually the load balancer never gives it any traffic unless you know the direct URL. That's dark.

A shadow deployment is if you're deploying a version B next to A, and then it gets shadowed traffic. That means you're hitting, let's say, a system with a load test. That means version A, which is currently a primary version, gets all the traffic. But then through traffic replication, your new version B also gets the same traffic. So you can actually compare how does A versus B behave with the same type of traffic. Then blue-green, I believe the common knowledge here is that you are deploying the new version in your infrastructure. And next to whatever blue and what green is, but let's say blue is the current version.

And then you deploy green in a separate part of the infrastructure, and then you can flip a switch over to the new version, figure out how the new system is behaving, and in case something is wrong, you can always flip the switch and switch over to the old environment. Then Canaries, I think the idea with Canaries is, again, you have version A out there, and now you have a a canary B and you are redirecting a certain part of the traffic to that B to figure out how is it behaving. And then you can decide whether you're rolling out more versions of B or more instances of B and redirecting more traffic to it. And then from a feature flag perspective, and Heidi, correct me if I'm wrong,

but this goes down to the granularity of individual requests even, where you can say, well, either you can turn on a feature on a particular node or the particular microservice instance, or even on individual requests where you can say this user, as you mentioned earlier, is a premium customer, and therefore they're seeing a certain feature, and then the code dynamically decides whether to show a certain component or not. So that's kind of my definition of it. I think one of the super useful things to be clear about is that deployment is the act of getting your code out onto the server. And launch is when you add the business value.

That's when you make it visible to someone. And keeping those separate in your head makes it a lot clearer what we're talking about. You can do deployment all day long and maybe nobody can see it. But when you launch, that's when you start exposing it to people. Yeah, that's a good point.

Yeah, I like that. Hey, I have a question. So the, how do you, coming back to testing, obviously we don't want to, I mean, there are mature enough companies that really just have, you know, a production environment, all the code that is part of that feature. So there has to be new approaches, I would almost assume, to testing whether the feature flag has correctly been implemented. Is there anything, I mean, what's out there? Do we do code coverage? Do we do, I mean, how do you test if a feature flag is correctly implemented in a pre-broad environment or in a test environment?

So you write the test to detect that. So, for instance, you could write a test that says, okay, I want to test whether dark mode is correctly implemented. So you're going to create a test group, and they're going to be the people who see dark mode. And then if dark mode meets their test story requirements, then you can go ahead and call it complete. But if it doesn't, then nobody else but test has seen it. And you can go back to the developers and say, hey, you totally forgot to make the buttons, you know, the text on the buttons light, even though you made the buttons dark, that didn't work out very well for me. Microsoft defines having something as finished, if it is in production, visible to users and returning metrics, which I think is such a beautiful definition of done, because until something is reporting back, you don't actually know that it's working and if it's working. And so adding metrics as a requirement for feature completeness seems so powerful to me. I need to write this down.

The definition of done, deployed, launched, and delivered metrics, right? Yeah. And also with metrics, I mean, this obviously falls into our, into what's the right sport terminology, into our ballpark? Yes. I don't know, because it's, that's obviously, you know, we measure a lot of things in Dynatrace. And I know you guys are, you know, any feature flag

as a service offering, you would probably, you have metrics on which feature has been executed and how often has it been executed, to whom was it exposed? And we are then additional on top of that measuring

how did it impact user experience and on all these things. But I like that. So the definition of done, deployed, launched, and it delivers metrics that actually tell me if the feature is accepted or not. I remember a story and Brian and I, we had Goranka Biedo from Facebook actually on a podcast a year or so ago, and she was talking about feature success criteria that they define.

So she said they're pushing a lot of features out there and a lot of the features never, I think she actually said 80% of the features that they push out there are never really successful. And in order to measure the successfulness, they define what success means. And their definition was a combination of how many users are actively using that feature within a certain time range after it was launched.

And if you don't hit the mark or if you don't cross the threshold, your feature, as good as you think it is, will be removed because it obviously is not successful with the user base. But just as you said, if you don't know how to measure how many people are exposed to it and how it changes the user behavior and how the adoption rate is, if you don't know that, you're flying blind and you're just starting pushing out more features and more features and we end up with the same situation we had in the past. So we build a lot of software that we think is great, but not all of it actually delivers value to our end users. And therefore, this is pretty critical. Yeah, I think it's super interesting to think about killing features off as like if there's not good uptake, you shouldn't keep it around.

Or we have a pattern that we call Albatross launching, which is the opposite of Canary launching. So say you have a client who's giving you quite a lot of money, but they don't want to upgrade, and yet you would like to upgrade everybody else and get them off IE8 or whatever. So you can feature flag that client's experience to still support that older method, but everybody else gets forcibly upgraded.

Yeah, I think that's interesting. That's the other way around. Yeah. I like that. Andy, I did want to one-up your, well, not really one-up, but add one piece to the idea of whether or not a feature stays in

or something stays in because it's not being used. Besides looking at whether or not customers are using it, I think it's equally important to be monitoring the performance of that feature because as we've seen plenty of times, sometimes customers don't use a new feature because it's just terrible. It could be the workflow is not great or it could be that the performance is not so great so that if maybe you temporarily turn it off or leave it in, but take a look at how that feature is performing and say,

well, this is actually taking a bit longer than we thought. Maybe if we can improve the performance of it, we'll get more adoption. And then you give it that shot. And then if it still doesn't go in, then maybe you pull it out. Because I think probably a lot of people think like, I just did all that work and now we're just going to kill it after one shot.

Like, you know, there's many more considerations to put into it, but I think people do need to not be scared to remove features if they're not being used, you know? And we even, again, going back to our meeting yesterday, they're like, oh, nothing ever gets removed just because no one wants to talk about doing that. Yeah.

It's not a real experiment if there aren't some failures. Yeah. Hey, Heidi, and also coming, we have to refer back to that meeting yesterday because we were spending four hours with one of our clients and we brought up some of these concepts.

And then they were, one of the arguments that they brought up is they said, well, we are not Facebook. We don't have, we don't have millions of users on our software. So we don't think this applies to us. And then I responded to them and I said, well, we are not Facebook. We don't have millions of users on our software. So we don't think this applies to us. And then I responded to them and I said, well, this is BS, right? Not BS, but I

used a politer version of this, but I basically said, everybody can do it. And you can do it with a very small user base too, right? In their case, they have all sorts of type of software and a lot of it's used in-house. And then they can define testing groups, let's say different offices, different departments, different office locations. And it's still valuable, right? Because they get feedback from even a small number of users that they expose the feature to. Right. And statistical significance is super easy to do if you're Google and Facebook. And it's harder to do if you're a mom and pop shop, but that doesn't mean you're not getting information.

We have a relatively small client who does, I wouldn't say bespoke software, but like semi-customized software. They have an automatic thing where if your net promoter score, if your happiness with the software is very high, you get all the new features first. Like they just put the high net promoter score people into a group where they get the new features first. And because of the demographic of their particular customers, that's very valuable to their customers because they feel, you know, hip and in advance of what's happening. But their user base is in the thousands, not the millions. And I think, too, with companies where your employees might use your own software, either based on a company IP or if there's a company flag on the account, if they log in, you can even at you know, not in a UAT style, but just based on your company users, turn on flags for them.

Oh, absolutely. And that's just a way if you're not that big Facebook again, you still have a base of people. Hopefully your employees are using your website or your software or something, but, you know, not in all cases, but there's always a way, you know? Yeah. And we use feature flags all the time internally, like not just for our customers, but for ourselves so that, you know, we can see QA and support have a whole extra set of feature flags so they can

see more of what a user is selecting than even development, because it's not relevant to development to be able to answer those questions. MARTIN SPLITTINGER- Hey, Heidi, I have one technical question. I know when we were sitting down in Lint after the F1, I wanted to know more about how technically feature flagging as a service actually works.

Because I know when I want to feature flag my code, I basically have an if statement that calls, let's say, your API saying, if I am on, then do this. Right. In a high volume, in a high transaction volume environment, I would assume that not every time this if statement is called, it makes a call back to your SaaS environment asking, am I on or off? Right. Because this would obviously then mean that your system has to be extremely highly scalable, which I'm sure it is,

but it gets hit with a lot of requests. So can you quickly explain to me how you guys are actually implementing the feature flag API and your client library? How often things are pushed or how this works so that my software actually knows whether a feature flag is turned on and off? Sure. So the way LaunchDarkly does it is it is a lot of client-side evaluation. There's an evaluation engine and a rule set that exists. And when I say client-side, sometimes it's your server side,

but we think of it as the client side, whatever is closest to delivering your product. And so very seldom does it actually need to call back. It instantiates, it gets its rule set and its engine, and then it's just going to operate at the same level as your software until it gets an update. And our architecture is that instead of pulling back to a database to figure out what its settings are, there's a listener that's listening for a server sent event. And when you make a change to the feature flag rule set, we send out an event through a CDN.

It's about 200 milliseconds to distribute it worldwide, and that changes the rule evaluation set. Without having to restart or refresh anything, it just changes the rule evaluation for the next request cool so that means uh if i launch my microservice let's say that way during startup it reaches out to you guys and say hey what's my current configuration and then additionally it registers basically a quote-unquote a webhook yes and then it operates and when you you i may make a change and I say, hey, I am the whatever dev lead and I want to play around with the feature flag settings and I make a change through your UI or your API, I assume I can also control it through an API. Yes.

Then you are broadcasting that change and say, hey, we have a change set for the feature flag. Cool. Yep. And it just pushes down and then the rule set changes. And it's very, very lightweight, both from our end and your end, because we're evaluating 200 billion flags. And our numbers just keep going up. And we're like, if we had designed it any differently our cost of

goods and services would be catastrophic and i i know you showed me the product a little bit and again i don't want to talk too much necessarily about how you do it but in general feature flags have the ability the way you define your rule set, you can say a certain geographic location, a certain browser type. So you can actually be more specific on who is seeing this feature and who not. And I'm sure you can also specify some request context. So I really want to have it for, let's say, Brian, when he logs in. And then the context Brian obviously needs to be captured in my app, but it will then be evaluated as part of the rules engine. Is this correct?

Right. And Brian can belong to any number of groups and the rule set can be additive. So Brian could be a part of the QA team and also, you know, part of the geographical location or the, we have a lot of people who are doing demographics. So, you know, people over 20. And so you can do an additive rule set that says if you're in this group and or, or if you're not in this group, we want to like, LinkedIn has two modes. There's being pro mode and nagging you about going to pro mode. That's absolutely a feature flag where they have two groups of, you know, pro mode and not pro mode. And also, I'm sure they have a QA mode where they're like, what does it look like in pro mode? And what does it look like when you're getting the request to upgrade? Yeah. And I just want to say that since you're using me as the group example, I would never be a member of any group that would have me as a member.

Yeah. Hey, last question there. So this context information about the user, obviously, that's something that the app has to provide. Making calls to your and then forward it to the feature flag API so that then the API can run it through the rules engine? Sometimes. Sometimes we manage to do it so that you do not have to. We have settings for PII. We don't want to send Brian's social security

number across the wire. So we can say, on the client side without leaving your zone of protection, please give us a proxy for, is he in the group where the social security number ends in four? Okay. So that means the proxy will be another callback where you basically call, how does this work?

All right. We're getting into the part where I would hand you off to a sales engineer. But basically what's happening is at the client side, on your side, it would say,

hey, that's information I'm not supposed to send, but I can send data on whether or not they're in a group because I know the rule for the group. Oh yeah, of course. Yeah, I get it. Makes sense. Yep. Because there's a lot of information we do not want to send across the wire, even though it would be super useful to evaluate people by it.

And we have some financial clients who have been quite clear about this. There's a lot of things that would be easier to do if we could just use all the information. Right, but we are GDPR SOC 2 and ISO 2701 compliant, which means that we have somebody who is almost full-time making sure

we don't make stupid mistakes like that. Cool. Heidi, is there anything else people should know about Feature Flags? I think we gave a lot of information already. Anything we missed? I think that it's really interesting to, like, featureflags.io, we run it, but we're trying to keep it relatively neutral to go look at all the different patterns, to talk about kill switches and circuit breakers and canaries and champagne branches. And there's a whole bunch of specialized ways that we talk about kill switches and circuit breakers and canaries and champagne branches.

And there's a whole bunch of specialized ways that we talk about it that I think is kind of fun to go look at. So there's a glossary there of different things. And then if people have others, they tell me about them, and I add them in. Pretty cool. Brian, anything else from your side,

or ready for the summary? Come on, do it. Let's summon the summary later, Andy. I'll keep it short because I think there's a lot of information out there on the web as well, like FeatureFlex.io. Heidi, you have also links that we will add to the podcast proceedings, your websites, your blogs, so people can read up more. But I think what I like about FeatureFlex, if you do it right, if you do it really, really right in the future, that means we only deploy into production because the act of deployment doesn't mean launching. That means we can deploy code that may has not yet been approved for 100% rollout, but yet we can deploy it.

And then we have the control through FeatureFlex on when we actually launch these features to a smaller group, a larger group, the whole user base. I think that's the great thing. Also, what I learned is that we really need to think about the difference between technically deploying code versus launching it, which means making it accessible to the end users. And very important, we should never do anything without having the ability to monitor and measure it. Therefore, the definition of done is a feature is deployed, it is launched, and it actually delivers metrics. And based on that, we can decide what to do next

with it, roll it out, turn it back off, or improve it, or whatever needs to be done. Heidi, thank you so much for all the insights. Hopefully, people that have not been aware of FeatureFlex are now aware of FeatureFlex, how powerful it is. That technically, it's just a fancy statement. But obviously, there's a lot of stuff happening in the back end and that leads up to actually implementing it but uh i'm also really excited uh heidi that we are we talked about captain our open source framework for continuous

deployment and automate operations this is also a feature flex will play a big role and hopefully we can follow through with building some integrations with Captain and your product that's going to be interesting. And yeah, that's it from my side. I learned a lot. Great, Andy. So one thing I wanted to bring, I'm glad you mentioned that you talked to Captain

about Captain because I missed that part of it. I think this conversation and the fact that there are services around feature flags now really speaks to the maturity of where we've come since the move off of waterfall, the move into, you know, continuous delivery pipelines and microservices. If you think back to, you know, first it was just containers and then we needed an orchestration layers and then we needed managements on top of the orchestration layers. and now people have pipelines and as you mentioned with with captain the pipeline management is becoming very onerous so now we're doing putting a control plane on top

of the pipeline and as you start using feature flags you know the maturity level of them and the many use cases as as heidi you mentioned earlier, they can become very, very difficult to handle and manage by just trying to keep track of them all. So there is, it's, I'm just, I guess, in awe seeing in the last few years, how quickly this all changes in the maturity level of how fast these ramp up that, you know, something like feature flags, the adoption and usage is now requiring some sort of a management level on top of them because it's successful really you know that's a real mark of success of using feature flags because humans can only do so much and you know managing like oh here's my excel spreadsheet of my you

know feature flags which was on or off or something right so that's that's really cool and it's really awesome what you all are doing so uh thank you for being on yeah thank you for the time it was lots of fun all right and if anybody has any questions comments or topics they'd like to hear please uh contact us at pure underscore performance on twitter or you can send an old-fashioned email at pureperformance dinatrace.com um heidi we're gonna put up those links uh do you have any social media or anything you'd like to put out there? No, I put it in there. No, I mean, you could just say it on there. Oh, okay. You can find me at WiredFerret on Twitter.

In 2007, we didn't know this was going to be a thing. That's perfect. I love it. All right. Well, thank you all very much all right thank you