PurePerformance - Encore - Understanding the Power of Feature Flags with Heidi Waterhouse

Episode Date: August 24, 2020

Imagine a future where we deploy every code change directly into production because feature flags eliminated the need for staging. Feature flags allow us to deploy any code change, but only launch the feature to a specific set of users that we want to expose to new capabilities. Monitoring the usage and the impact enables continuous experimentation: optimizing what is not perfect yet and throw away features (technical debt) that nobody really cares about. So – what are feature flags?We got to chat with Heidi Waterhouse (@wiredferret), Developer Advocate at LaunchDarkly (https://launchdarkly.com/), who gives as a great introduction on Feature Flags, how organizations actually define a feature and why it is paramount to differentiate between Deploy and Launch. We learn how to test feature flags, what options we have to enable features for a certain group of users and how important it is to always include monitoring. IF you want to learn more about feature flags check out http://featureflags.io/. If you want to learn more about Heidi’s passion check out https://heidiwaterhouse.com/.

Hello everybody out there in Pure Performance land. Due to summer vacations and many other factors, Andy and I did not have a new show for you this week. Never fear though, because we've already scheduled new recordings and we'll have them up as soon as possible. In the meantime, we hope you enjoy this encore presentation of Understanding the Power of Feature Flags with Heidi Waterhouse, originally aired June 10th, 2019. It's time for Pure Performance.

Get your stopwatches ready. It's time for Pure Performance with Andy Grabner and Brian Wilson. Welcome everyone to today's Pure Performance podcast. I know normally you hear the intro from my dear friend Brian Wilson. But well, today I got the joy and honor to host a session without Brian. He has a lot of things to do today, but we'll be back the next time. Normally, he opens with a joke.

Well, I'm not that funny as he is, so I will skip the jokes today, and I will go right over to our guest of honor. And today with me is Heidi Waterhouse. Heidi, I hope you are still with me on the audio line. I am. Awesome. Well, Heidi, thank you so much for joining. I got the pleasure of meeting you. I think it was mid of April in Linz, Austria, my hometown, because you were speaking at our DEF One conference. I think that's the first time we ever met. Yes, that's right. And I have to say that was an amazing conference and I have never

gotten a fog machine and laser show as speaker intro before. Yeah, I know they did a pretty good job with all the visuals. They put a lot of effort in there. So Heidi, I mean, I got to know you and that was great. Some of the audience that are listening to this may have never heard of you. So could you give yourself a little quick intro of who you are, what drives you, what motivates you, where you live, anything that people may want to know about you? Sure. My name is Heidi Waterhouse. I'm the Senior Developer Advocate at LaunchDarkly. We do feature flags as a service. One of the things that I'm really interested in is making it possible for more people to have control over what's getting sent to their users. I spent 20 years as a technical writer, and that gives me a lot of

feelings about how it actually is to use software and to feel sort of helpless when you're trying to figure out how it works. So one of the things that drives me as a developer advocate is making sure that all the time we're thinking about how it feels to use our software and how we could make that experience better. FRANCESC CAMPOY FLORES- Cool. And then feature flagging as a service. So you said you work for LaunchDarkly.

Not that this is going to be a commercial, but I think it's great that companies like LaunchDarkly exist to actually not only promote what you do, promote the methodology, the approach of exposing features to end users in a more controlled way, but also having a product, obviously, that makes it easier. So in case people have never heard about it, I think launchdarkly.com, or what's the website, I think launchdarkly.com or what's the website?

Yeah, launchdarkly.com. Perfect. Now, I really want to dive into the topic because just as you have the passion of, you know, helping people to figure out how to release better software, understand how software is used and optimize that experience. And so do we, right? We are all about making sure that the software is performing well, that the user experience is great. Now, feature flags.

Feature flags as a service is something that you deliver. Can you tell us a little bit more about feature flagging? What is a feature flag for people that have never heard about it? Is this a complete new concept or what do we need to know about feature flags?

So feature flags are really a fancy if statement. But when I say fancy, it can get very fancy indeed. What we're doing is enabling people to control how their code behaves after it's been put into production. So if you think about how A-B testing works, where you try out two different variants of an interface, or if you think about how a progressive deployment works, where you only release 10% of the product at a time or the product to 10% of the people at a time, so that you can make sure that it's going to roll out cleanly without causing anyone problems. Those are both manifestations of how feature flags work. So what happens is you decide what a feature is, which is a non-trivial problem, like what size is a feature and what is a business meaningful unit to make a feature.

And then you make that controllable after deployment. So you say instead of like, I've been in the industry long enough that I remember when we used to cut gold master CDs. And there wasn't like a well, what constitutes a feature and what should get controlled that way? And I think that's a really interesting problem because it's sort of the Agile problem, like what is a story? But it's also what is a business useful story, because you can write all sorts of agile stories that are not business useful. Yeah. So are there best practices around how you come up with a feature, or is this really

aligned, as you just mentioned, with a story? Or is there any material out there on any approaches on how to figure out what is a feature and how do we encapsulate it? There are several. I think probably the best book on it is called From Project to Product that just released lately. And I've read some of the preview material, but I haven't read all of it yet. But it's about what is the value that we're doing. As far as best practices for features,

it's really useful if you understand that there are two feature flag models. And one of them is a permanent feature flag that you're going to keep on all the time. And that's for stuff like entitlements, like which version of the website you're going to get because you have a membership and you're paying extra. And so that's an entitlement, or you might want it on a kill switch or a circuit breaker in your infrastructure in order to control what happens to the way your

infrastructure behaves. You would leave all of those in place permanently. The other kind of feature flag is a temporary feature flag, and you would use that for experiments, and you would use that for deployment. But when you have made the decision or when you have finished the deployment, then that feature flag is going to get removed because leaving it in your code is just, it's a form of technical debt because you're,

you're leaving the potential for something to get switched on or off that you don't actually want to have change anymore. So that's so handy. Go ahead. Brian, did we just, did you just sneak in? I snuck in about three and a half minutes ago. Yes. Oh my gosh. Look at that. Sorry I was late.

I didn't realize that I agreed to the time. I have kids I have to drop off and all that stuff, and there was some traffic and got late. And then, of course, technical difficulties because computers are still our friends and love to give us trouble like our real friends do in life. So hi, Heidi. Nice to meet you. Nice to meet you, too. Yeah, that's good. So heidi uh that's that's pretty cool so permanent versus temporary feature flags now you said temporary feature flags will be and have to be removed after a while

because otherwise your code gets cluttered with a lot of i guess if statements i mean right it's in the in the beginning we said what is a feature flag and how do you technically implement it? And then you said, well, it's basically a fancy version of an if statement. Now, does this mean that organizations have to, in their sprint cycles, obviously plan for actively removing these flags? Or are there any, let's say, development tools, any tools that are automatically removing these pieces of code? Because I assume it's always, I mean, I could even see like as a macro,

if you go back to languages that have macros where you can kind of block a certain piece of code with a macro and then you just say, hey, you know, let's remove that. Or how does this technically work? Or do people go in and really remove these lines of code manually? You could use it automatically, but I wouldn't suggest it

because it would be so easy to get it wrong and to remove the wrong flag. So if you accidentally removed a permanent flag, you'd be in trouble. Of course, yeah. So what we do is we have a tracker that basically notices where you inserted feature flag code. And so when you go to that flag and say, hey, where does this live in the code base?

It pops it all up for you in GitHub and you can pull it out there. But not entirely automatically because we don't want other people touching our code. Like when you get right down to it, in order to allow that, you would have to let people touch your code. And we don't really want external parties doing that if we can help it. Yeah, no, I thought more of a tool that might be integrated in your development process,

in your IDE, where like an IDE extension, where you say, hey, here's all the, yeah, we scanned the code and here's all the list of the feature flags. Based on the data we have from our feature flag as a service solution, we want to remove these five feature flags. Boom, remove it for me,

or at least point me to the source code line. But I guess that's what you alluded to with the GitHub integration, so that's cool. Yeah. We can point you to the source code line. Because it's all API- based, you could automate that. But I don't think that's a high priority for us to automate because it's so risky. Yeah. Okay. That makes sense. Hey, I wanted to, before going further, you know, in looking at some of the material ahead of time, I something I just want to make sure the audience hears it because I think it really covers one of the main tenets of

feature flagging. And I'm getting this from Heidi sent over that link of featureflags.io. And one of the quotes on the getting started, I think just really, really nails it. It's from the SRE Google quote. And I'm just going to read it because I want to set in the listener's minds the importance of feature flags and how they can be awesome. So from this, it's by hosting code that supports new functionality in the client application before we activate the feature, we greatly reduce the risk associated with the launch and having the sort of dormant functionality also makes aborting launches easier when adverse effects are discovered during a rollout. I just, yeah, I think to me that sums up a lot of, there are a lot of things you can do with feature flags, but I think from a putting a new feature into production and how

you can manage it, that really, really nails it, I think. Yeah. And Google has been doing trunk-based development for a long time. So one of the interesting things about feature flags is that they are kind of a prerequisite for doing true continuous deployment. If you can't deploy broken code without anyone seeing it, you can't do continuous deployment. You have to stop until you know all your code is functional. With feature flags, you can deploy broken code and no one will see it. So you can do continuous deployment. And when I say broken, not necessarily broken, broken, but like unready or unapproved or, you know, in some other way, not ready to be shown. Hey, then, I mean, for me,

the biggest challenge then really comes down to, obviously, technically, we can easily solve it. As you said, it's a fancy if statement where services like what you provide with LaunchDarkly allow you, give you an API that is then centrally keeping track of which feature flex should actually be turned on and off as code executes. But for me, the biggest challenge then really is figuring out what is a feature. And I believe, especially in a, and correct me if I'm wrong, but especially in a more distributed world where you don't have monolithic code anymore, where you can touch a

single code base to define what a feature is. Well, even in a monolithic code anymore, where you can touch a single code base to define what a feature is. Well, even in a monolithic code base, you may have different parts of the code. But in a distributed architecture, if a feature means turning on or off flags across different entities like microservices, that becomes a real challenge. Or is this something you need to avoid and not having the same feature flag being used across different parts of the architecture? If your feature flag description, like when you're saying your feature flag out loud, you say the word and, you've violated one of the best practices. A feature flag should do one thing only. However, that said, you can chain together feature flags that say, this one will only activate if a top-level flag is activated.

So there's a way to do that that says, like, do a dependency check or do a version check to make sure that the environment is correct for this flag to turn on. So because my thought process was if I have, let's say, a backend microservice and I introduce a new capability, I don't know, a new algorithm for something, right? And then I want to expose that feature to my users on the front end. I need that feature flag on the front end, but depending on that feature flag,

either in the backend, then execute the new or the old algorithm. But as you, I guess, as I just in my own thought process, I guess what you would do is you would implement the backend service behind the new API version. And then the front end, if the feature flag is turned on, says, do I call the old or the new API? I guess that's the

better way of doing it. Yeah. Yeah. That's the better way of doing it. And one of the interesting things that we've seen with our customers is that when they start using feature flags, they go for some pretty elementary, you know, sort of the low hanging fruit, the obvious solutions. And as they get more comfortable with the ability to have different states of software, sort of like quantum superposition, they do some very interesting things with how their software works. And we ask, doesn't that make testing more difficult? Because if you have this superposition of possible effects of how the software works, how do you do testing? And the answer that we've been getting is mostly that although testing is a little more complicated because they know what they're doing, they can more precisely control it. Sort of like when you're a kid and you're doing a science experiment with pH, you know

that there's only this possible range. And because you know what the range, the possible range is, you can do some very interesting things with it because you've like testing the pH is not going to go beyond the paper. And so you've created yourself a safe space to test it. And we find that people who are using feature flags feel much safer because they're like, okay, this is a place that is safe to test in. And if something goes wrong, I can turn it off and I can put a bunch of analytics on it to make sure that it's behaving the way I think it's going to behave before I show it to the general public. Now, from a testing perspective, I mean, if I hear this correctly, what you said earlier, you said feature flags allow me to deploy broken code in production.

Or let's say it that way, not extremely well-tested code in production. For me, that actually means if I do it right, I can completely reduce or remove any pre-prod environments. prod environments i only have prod and new code gets basically hidden behind the feature flag which can then be tested in production by my testers and the testers could either be a testing team it could be a group of people it could be a group of end users that i'm exposing to this feature that kind of my my test group and based on that, I decide, is the quality right or not? And then I can just flip the switch and turn it on for everybody. But if I do it right, if I get this correctly, then in the future, I don't need the traditional testing environments anymore because I just deploy the prod in a safe way and then slowly expose that new feature to a group of people and then enlarge that exactly we have a customer who the way that they're paying for their license at least partially is

they don't have staging servers they're just not paying for a staging environment they're doing all of their staging testing in production and that's not the right fit for everyone but i think it's a super interesting use case because they're like a software as a service company themselves and that's a feasible way for them to work. And our meetup in the Bay Area is actually called Test in Production because that's something we're so excited about. And the thing that I keep telling people is that no matter who you are, you can't replicate the full weirdness of production in a staging or test environment. You can't replicate microservice latency.

You can't replicate the weirdness of user data. Like no test suite has ever covered the fact that some people have a one letter last name appropriately. One thing that I wanted to kind of clarify here, though, right, is when we're talking about, well, I'm laughing a lot when you're talking about this, because going back to my testing days and my performance testing days and all this kind of stuff, there was always the joke of, you know, production is the new QA. Right. And that was always laughed at.

I mean, I'm going back, you know, seven, eight years ago because there was really no way to do this properly. And I want to clarify, though, that this is not talking about and, you know, I'm saying clarify for the listeners, really. We're not talking about turning it on to the general public for testing right away. You're still going to employ things. You're still going to employ things like maybe a dark deployment where it's turned on for the, you know, the people with access to that section, maybe the testing teams themselves go into the production environment and run on a special set of, you know, knowing the proper URL, or maybe you can then switch over to a shadow deployment and do it. So it's still, it's very slow, controlled rollout where it goes through the ringer.

It's not like, hey, let's turn it on in production and literally make the old joke true of production and our end users are the new QA team. Yeah, no, no, no. As funny as that would be, that is not how we operate. Yeah, exactly. So when Facebook rolled out the new feature where you could select your name, they spent months doing dark deploys and dark launches. And every time you logged into Facebook, they would basically do a simulated username request until they were sure their infrastructure was going to hold up to it and that they had worked all of the bugs out of the system. So almost everyone has already participated in several forms of testing in production and not known it.

Great. I wonder, Andy, too, especially even based on yesterday, the meeting we were at, when you asked about who knows what a shadow deployment is. Should we take like two minutes here to just kind of cover the difference between a shadow deployment, dark deployment, blue-green, just to make sure everyone is level set on understanding these components? Because, you know, maybe not everybody knows. Yeah, well, I want to know what you guys are saying about shadow deployment. Well, I'll let you take it, Andy. Yeah, sure. And I'm not sure if I have the right definition, but this let you take it, Andy. Yeah, sure.

And I'm not sure if I have the right definition, but this is the way I explain it. For me, a dark deployment, and I think that's also different between a dark release. Dark deployments into an environment, for me, is you're deploying a new version next to the old one, but you only can access it if you actually know the direct access to that particular component. If it's behind the load balancer, but actually the load balancer never gives it any traffic unless you know the direct URL. That's dark. A shadow deployment is if you're deploying a version B next to A,

and then it gets shadowed traffic. That means you're hitting, let's say, a system with a load test. That means version A, which is currently a primary version, gets all the traffic. But then through traffic replication, your new version B also gets the same traffic, so you can actually compare how does A versus B behave with the same type of traffic. Then blue-green, I believe the common knowledge here is that you are deploying the new version in your infrastructure.

And next to like whatever blue and what green is, but let's say blue is the current version. And then you deploy green in a separate part of the infrastructure. and next to like whatever blue and what green is, but let's say blue is the current version, and then deploy green in a separate part of the infrastructure, and then you can flip a switch over to the new version, figure out how the new system is behaving, and in case something is wrong, you can always flip the switch and switch over to the old environment. Then Canaries, I think the idea with Canaries is, again, you have version A out there, and now you have a Canary B, and you are redirecting a certain part of the traffic

to that B to figure out how is it behaving. And then you can decide whether you're rolling out more versions of B or more instances of B and redirecting more traffic to it. And then from a feature flag perspective, and Heidi, correct me if I'm wrong, but this goes down to the granularity of individual requests even where you can say, well, either you can turn on a feature on a particular node or a particular microservice instance or even on individual requests where you can say,

this user, as you mentioned earlier, is a premium customer and therefore they're seeing a certain feature. And then the code dynamically decides whether to show a certain component or not. So that's kind of my definition of it. I think one of the super useful things to be clear about is that deployment is the act of getting your code out onto the server. And launch is when you add the business value. That's when you make it visible to someone. And keeping those separate in your head makes it a lot clearer what we're talking about. You can do deployment all day long and maybe nobody can see it. But when you launch, that's when you start exposing it to people. Yeah.

That's a good point. Yeah. I like that. Hey, I have a question. So how do you, coming back to testing, obviously, we don't want to, I mean, there are mature enough companies that really just have a production environment. But you still always want to test things around,

did we actually encapsulate all the source code, all the code that is part of that feature? So there has to be new approaches, I would almost assume, to testing whether the feature flag has correctly been implemented. Is there anything I mean, what's out there? Do we do code coverage? Do we do I mean mean how do you test if a feature flag is correctly implemented in a pre-prod environment or in a test environment you write the test to detect that so uh for instance you could write a test that says okay

i want to test whether i whether dark mode is correctly implemented. So you're going to create a test group, and they're going to be the people who see dark mode. And then if dark mode meets their test story requirements, then you can go ahead and call it complete. But if it doesn't, then nobody else but test has seen it. And you can go back to the developers and say, hey, you totally forgot to make the buttons, you know, the text on the buttons light, even though you made the buttons dark, that didn't work out very well for me. Microsoft defines having something as finished,

if it is in production, visible to users and returning metrics, which I think is such a beautiful definition of done because until something is reporting back, you don't actually know that it's working and if it's working. And so adding metrics as a requirement for feature completeness seems so powerful to me. I need to write this down. The definition of done, deployed done deployed launched and delivered metrics right

yeah and also with metrics I mean this obviously falls into our into what's the right sport terminology into our ballpark I don't know because

it's that's obviously you know we measure a lot of things and I know you guys are you know any feature flag as a service offering That's obviously, we measure a lot of things in Dynatrace. And I know you guys are, any feature flag as a service offering, you would probably, you have metrics on which feature has been executed and how often has it been executed, to whom was it exposed.

And we are then additional on top of that measuring how did it impact user experience and on all these things. But I like that so the definition of done deployed launched and it delivers metrics that actually tell me if the feature is is accepted or not i remember a story and uh brian and i we had goranka bieto from facebook actually on a podcast a year or so ago and she was talking about feature success criter criteria that they define. So she said they're pushing a lot of features out there. And a lot of the features never, I think she actually said 80% of the features that they push out there are never really successful.

And in order to measure the successfulness, they define what success means. And their definition was a combination of how many users are actively using that feature within a certain time range after it was launched. And if you don't hit the mark, or if you don't cross the threshold, your feature, as good as you think it is, you know, will be removed because it obviously is not successful with the user base. But just as you said, if you don't know how to measure how many people are exposed to it and how it changes the user behavior and how the adoption rate is, if you don't know that, you're flying blind and you're just starting pushing out more features and more features. And we end up with the same situation we had in the past. So we build a

lot of software that we think is great, but not all of it actually delivers value to our end users. And therefore, this is pretty critical. Yeah, I think it's super interesting to think about killing features off as if there's not good uptake, you shouldn't keep it around. Or we have a pattern that we call Albatross launching, which is the opposite of Canary launching. So say you have a client who's giving you quite a lot of money, but they don't want to upgrade. And yet you would like to upgrade everybody else and get them off, you know, IE8 or whatever.

So you can feature flag that client's experience to still support that older method but everybody else gets forcibly upgraded yeah i think that's all that's that's interesting that's the other way around yeah i like that you know andy i did want to one-up your well not really one-up but add one piece to the idea of um whether or not a feature stays in or something stays in because it's not being used. Besides looking at whether or not customers are using it, I think it's equally important to be monitoring the performance of that feature because as we've seen plenty of times,

sometimes customers don't use a new feature because it's just terrible. It could be the workflow is not great or it could be that the performance is not so great so that if maybe you temporarily turn it off or leave it in but take a look at how that feature is performing and say well this is actually taking a bit longer than we thought maybe if we can improve the performance of it we'll get more adoption and then you give it that shot and then if it still doesn't go in then maybe you pull it out because i think probably a lot of people think like I just did all that work and now we're just going to kill it after one shot.

Like, you know, there's many more considerations to put into it. But I think people do need to not be scared to remove features if they're not being used. And we even, again, going back to our meeting yesterday, they're like, oh, nothing ever gets removed just because no one wants to talk about doing that. Yeah. It's not a real experiment if there aren't some failures. Yeah. Hey, Heidi, and also coming, we have to refer back to that meeting yesterday because we were spending four hours with one of our clients and we brought up some of these concepts. And then they were, one of the arguments that they brought up is they said, well, we are not Facebook.

We don't have, we don't have millions of users on our software. So we don't think this applies to us. And then I responded to them and I said, well, this is BS, right? Not BS, but I used a politer version of this. But I basically said, everybody can do it. And you can do it with a very small user base too, right? In their case, they have all sorts of type of software and a lot of it's used in-house.

And then they can define testing groups, let's say different offices, different departments, different office locations. And it's still valuable, right? Because they get feedback from even a small number of users that they expose the feature to.

Right. And statistical significance is super easy to do if you're Google and Facebook. And it's harder to do if you're a mom and pop shop, but that doesn't mean you're not getting information. We have a relatively small client who does, I wouldn't say bespoke software, but like semi-customized software. And one of the things that they do is they have an automatic thing where if your net promoter score, if your happiness with the software is very high, you get all the new features first. Like they just put the high net promoter score people into a group where they get the new

features first. And because of the demographic of their particular customers, that's very valuable to their customers because they feel, you know, hip and in advance of what's customers, that's very valuable to their customers because they feel hip and in advance of what's happening. But their user base is in the thousands, not the millions. And I think too with companies where your employees might use your own software, either based on a company IP or if there's a company flag on the account,

if they log in, right, you can even at least just, you know, not in a UAT style, but just based on your company users, turn on flags for them. Oh, absolutely. And that's just a way, if you're not that big Facebook again, you still have a base of people. Hopefully your employees are using your website or your software or something, but you know, not in all cases, but there's always a way, you know? Yeah. And we use feature flags all the time internally, like not just for our customers, but for ourselves so that, you know, we can see QA and support have a whole extra set of feature flags so they can see more of what a user is selecting than even development, because it's not relevant to development to be able to answer those questions.

Hey, Heidi, I have one technical question. I know when we were sitting down in Linz after the F1, I wanted to know more about how technically feature flagging as a service actually works, because I know when I want to feature flag my code a service actually works. Because I know when I want to feature flag my code, I basically have an if statement that calls, let's say, your API saying, if I am on, then do this. In a high volume, in a high transaction volume environment, I would assume that not every time this if statement is called, it makes a call back to your SaaS environment asking, am I on or off? Right. Because this would obviously then mean that your system has to be extremely highly scalable, which I'm sure it is, but it gets hit with a lot of requests. So can you quickly explain to me how you guys are actually implementing

the feature flag API and your client library, how often things are pushed or how this works so that my software actually knows whether a feature flag is turned on and off? Sure. So the way LaunchDarkly does it is it is a lot of client-side evaluation. There's an evaluation engine and a rule set that exists. And when I say client-side, sometimes it's your server side, but we think of it as the client side, whatever is closest to delivering your product. And so very seldom does it actually need to call back. It instantiates, it gets its rule set and its engine,

and then it's just going to operate at the same level as your software until it gets an update. And our architecture is that instead of pulling back to a database to figure out what its settings are, there's a listener that's listening for a server sent event. And when you make a change to the feature flag rule set, we send out an event through a CDN. So it's about 200 milliseconds to distribute it worldwide. And that changes the rule evaluation set. So without having to restart or refresh anything, it just changes the rule evaluation for the next request. Cool. So that means if I launch my microservice, let's say that way, during startup, it reaches out to you guys and say,

hey, what's my current configuration? And then additionally, it registers basically, quote-unquote, a webhook. Yes. And then it operates. And when I make a change and I say, hey, I am the whatever dev lead and I want to play around with the feature flag settings and I make a change through your UI or your API,

I assume I can also control it through an API. Yes. Then you are broadcasting that change and say, hey, we have a change set for the feature flag cool yep and the it just pushes down and then the rule set changes and it's very very lightweight both from our end and your end because we're evaluating 200 billion flags and you know like our numbers just keep going up and we're like if we had designed it any differently our cost of goods and services would be catastrophic

and i i know you showed me the product a little bit and again i don't want to talk too much necessarily about how you do it but in general um feature flags have the ability, the way you define your rule set, you can say a certain geographic location, a certain browser type. So you can actually be more specific on who is seeing this feature and who not. And I'm sure you can also specify some request context. So I really want to have it for, let's say, Brian when he logs in. And then the context Brian obviously needs to be captured in my app, but it will then be evaluated as part of the rules engine. Is this correct? Right. And Brian can belong to any number of groups and the rule set can be additive. part of the QA team and also part of the

geographical location or the we have a lot of people who are doing demographics so people over 20 and so you can do an additive rule set that says

if you're in this group and or if you're not in this group we want to like uh linkedin has two modes there's being pro mode and nagging you about going to pro mode that's absolutely a feature flag where they have two groups of you know pro mode and not pro mode and also i'm sure they have a qa mode where they're like what does it look know, pro mode and not pro mode. And also, I'm sure they have a QA mode where they're like, what does it look like in pro mode? And what does it look like when you're getting the request to upgrade?

Yeah. Yeah, and I just want to say that since you were using me as the group example, I would never be a member of any group that would have me as a member. Sick. Yeah. Hey, last question on that.

So this context information about the user, obviously that's something that the app has to provide. Making calls to your, let's say, client library saying, hey, this is Brian. He's not part of any group because he's special. So this context information is something that has to be captured by the app itself and then forwarded to the feature flag API so that then the API can run it through the rules engine. Sometimes.

Sometimes we manage to do it so that you do not have to. We have settings for PII. Like, we don't want to send Brian's social security number across the wire. We can say on the client side without leaving your zone of protection, please give us a proxy for

is he in the group where the social security number ends in four. That means the proxy will be another callback where you basically call, how does this work? All right. We're getting into the part where I would hand you off to a sales engineer. But basically what's happening is at the client side, on your side, it would say, hey, that's information I'm not supposed to send, but I can send data on whether or not they're in a group because I know the rule for the group. Oh, yeah, of course. Yeah, I get it. Makes sense.

Because there's a lot of information we do not want to send across the wire, even though it would be super useful to evaluate people by it. And we have some financial clients who have been quite clear about this. There's a lot of things that would be easier to do if we could just use all the information. Right. But we are GDPR SOC 2 and ISO 2701 compliant, which means that we have somebody who is almost full time making sure we don't make stupid mistakes like that. Cool. Heidi, is there anything else people should know about Feature Flags?

I think we gave a lot of information already. Anything we missed? I think that it's really interesting to, like, FeatureFlags.io, we run it, but we're trying to keep it relatively neutral, to go look at all the different patterns, to talk about kill switches and circuit breakers and canaries and champagne branches. And there's a whole bunch of specialized ways that we talk about it that I think

is kind of fun to go look at. So there's a glossary there of different things. And then if people have others, they tell me about them, and I add them in. Pretty cool. Brian, anything else from your side or ready for the summary? Come on, do it. Let's summon the summary later, Andy.

I'll keep it short because I think there's a lot of information out there on the web as well, like FeatureFlex.io. Heidi, you have also links that we will add to the podcast proceedings, your websites, your blogs, so people can read up more. But I think what I like about FeatureFlex, if we do it right, if we do it really, really right in the future, that means we only deploy into production because the act of deployment doesn't mean launching. That means we can deploy code that may has not yet been, you know, approved for 100% rollout, but yet we can deploy it. And then we have the control through feature flags on when

we actually launch these features to a smaller group, a larger group, the whole user base. I think that's the great thing. Also, what I learned is that we really need to think about the difference between technically deploying code versus launching it, which means making it accessible to the end users. And very important, we should never do anything without having the ability to monitor and measure it. Therefore, the definition of done is a feature is deployed, it is launched, and it actually delivers metrics. And based on that, we can decide what to do next with it, roll it out, turn it back off or, you know, you know, improve it or whatever needs to be done. Heidi,

thank you so much for, for all the insights. Hopefully people that have not been aware of feature flags are now aware of feature flags, how powerful it is, that technically, it's just a fancy statement. But obviously, there's a lot of stuff happening in the back end. And that leads up to actually implementing it. But I'm also really excited, Heidi, that we talked about Kaptn, our open source framework for continuous deployment and automated operations.

This is also where Feature Flex will play a big role. And hopefully, we can follow through with building some integrations with Captain and your product that's going to be interesting. And yeah, that's it from my side. I learned a lot. Great, Andy. So one thing I wanted to bring,

I'm glad you mentioned that you talked to Captain about Captain because I missed that part of it. I think this conversation and the fact that there are services around feature flags now really speaks to the maturity of where we've come since the move off of Waterfall, the move into continuous delivery pipelines and microservices. If you think back to, first it was just containers, and then we needed orchest orchestration layers and then we needed managements on top of the orchestration later layers. And now people have pipelines. And as you mentioned with, with captain, the pipeline management is becoming very onerous. So now we're doing, putting a control plane on top of the pipeline. And as you start using feature flags, you know, the maturity level of them and the many use cases, as Heidi, you mentioned earlier,

they can become very, very difficult to handle and manage by just trying to keep track of them all. So there is, it's, I'm just, I guess, in awe seeing in the last few years, how quickly this all changes in the maturity level of how fast these ramp up that, you know, something like feature flags, the adoption and usage is now requiring some sort of a management level on top of them because it's successful, really, you know, that's a real mark of success of using feature flags because humans can only do so much. And, you know, managing like, oh, here's my Excel spreadsheet of my, you know, feature

flags, which was on or off or something, right? So that's really cool. And it's really awesome what you all are doing. So thank you for being on. Yeah. Thank you for the time. It was lots of fun. All right. And if anybody has any questions, comments, or topics they'd like to hear, please contact us at pure underscore performance on Twitter, or you can send an old fashioned email at pure performance at dynatrace.com. Heidi, we're going to put up those links. Do you have any social media or anything you'd like to put out there? No, I put it in there.

No, I mean, you could just say it on here. Oh, okay. You can find me at wiredferret on Twitter. In 2007, we didn't know this was going to be a thing. That's perfect. I love it. All right. Well, thank you all very much. All right. Thank you. Thank you. Bye.