PurePerformance - How AI is shaping Flag-Driven Development with Todd Baert

Episode Date: September 28, 2026

Feature flagging is almost as old as software engineering – but only since the increased adoption of AIDLC (AI- Delivery Life Cycle) it has really taking off like AI has taken off as the default way to generate code!In this episode we welcome back Todd Baert, Maintainer of OpenFeature, and learn about the many different use cases of Feature Flagging. Todd explains how feature flagging evolved from simple variables that pivot code execution towards dynamic evaluations that are supported by modern frameworks like OpenFeature and feature flag management systems that are based on that cloud native standard.Tune in and learn what the biggest competitors of OpenFeature are, why feature flagging requires less thinking during development than you think and how it mitigates the risk of AI generated code having negative impact on your systems resiliency!LinksTodd’s LinkedIn: https://www.linkedin.com/in/toddbaert/OpenFeature: https://openfeature.dev/Introducing OpenFeature Podcast: https://www.spreaker.com/episode/introducing-openfeature-stepping-into-the-footsteps-of-opentelemetry-with-mike-beemer-and-todd-baert--50043067

It's time for pure performance. Get your stopwatches ready. It's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson. And as always I have with me, my fantastic, wonderful flag-waving co-host, Andy Grabner. Hello, Andy Grabner. How are you doing today?

Oh, you know what? Is this podcast, is this episode airing in October? It is. So that means you might be Andy Candy Grabner now. All right. Let me think. Let me think. Actually, no, I think it's the last episode in September.

Oh, okay, okay. Yeah. That's his Halloween name for grabbing candy. Yeah. No, no candy for me in September. Yeah, all good. I got three flags with me. Why do I have three flags?

Because today's topic magically happens to be about flags, not the flags that we wave in front of in our yard. or when there's a holiday, but the flags we put into our software. But the question is, why do we put flakes? Boom, boom. The question is, why don't we put flakes in our software? Brian, do you know?

So that we know what country, the lines, the software belongs to? Maybe we not. So we can control, so we can control different features and turn things on and off. Ah, look at that. Maybe we should ask somebody

that really knows this stuff. I think that's a great idea. It's a great idea. We are happy to have back on the Pure Performance Podcast, Todd. Todd, you look like you have aged. At least your hair just this away. Yeah, I've definitely aged since the last time I was on here. And my hair is probably aged three or four times as rapidly.

But your hair is at least much thicker than mine. because I age too, but my hair doesn't turn gray because there's just some little hair left that could turn gray. Yeah, I probably have more hair since the last time I was here, but it's definitely gone gray or no question. Hey, Todd, back in 2022, which is four years ago, we had a session that was called Introducing Open Feature, stepping into the footsteps of Open Telemetry with you and Mike. And we also had another session on. feature flagging with Ben, Ben Aromage from Flexsmith about a year and a half ago. So this is not the first time we talk about feature flagging and open feature,

but I'm sure it's the first time that some of our listeners hear about feature flagging and open feature. And so if you could do me the favor and maybe just give us a quick pitch on what is feature flagging in software engineering? Yeah. Yeah, the people you mentioned actually are hugely involved in the space and in the project, the Open Feature Project. To answer your question, basically feature flags are putting a pivot point more or less in your code so you can control how it behaves remotely and dynamically. And this is a practice that probably any software engineers familiar with SREs, maybe even product managers, they're going to be familiar with the idea as well. And like the basic kind of iteration of it is just, yeah, like let's put a pivot point in the code, maybe control it with some kind of environment variable.

And depending on how that's set, the code is going to behave differently. That's like the most naive version. I think everybody's done something like that. If you've done any software engineering in the real world, you have to have that kind of dynamism. but things can get much more interesting, right? So we can have flags of different types. Where things get a little bit more mature is where not only can you change the behavior of your software, but you can change it at runtime.

So an environment variable, it's read once at startup and the application for the rest of its lifetime behaves one way or another. But being able to say without deploying the software, without changing a Kubernetes config map or something like that, change the behavior. That's very powerful because that allows you to, you know, open circuit breakers, cut off access to a vulnerable segment of code, perhaps. So you can see how that is immediately more valuable than something as naive as an environment variable.

And then the next level is something we like to call dynamic evaluation. So not just dynamically changing flags, but using contextual information to control that pivot. So things like, well, you were talking about flags, flags of countries earlier, things like geographic and jurisdictional requirements. Maybe I want to turn something on for everybody, except for people in a particular legal jurisdiction,

because there's some ongoing GDPR compliance thing or something like that. So you can easily create targeting rules that are outsized, side of your application that could specify things like this, say like allow users who are from Ireland only, or allow users who are of this age group.

So anything you know about the user, about the request, whether it's technical or personal, you can use that as a basis for a flagging decision in a mature feature flagging platform. And that's really where a lot of the power comes from for devs and for SREs and for

PMS. Yeah. So Todd, if somebody that listens in it might not be familiar with software engineering, obviously this still means actually a little bit of, I guess, more heads-up thought work and implementation work because these features and these changes of behavior, obviously, don't just magically happen because you put in a feature flag. That means you obviously need to build that feature, but you kind of hide it behind the flag and then you can dynamically, as you said, turn it on and off. I just wanted to obviously, you know, make sure that people,

also understand. And for me, it feels almost like, from an engineering perspective, you need to put more thought in on how you put these features into your code, but maybe at the time of deploying the code that contains this new code, that it's still not enabled or released. So I'm just thinking about the whole mechanism of software delivery. Obviously, that has completely changed, right? Because you're deploying something that is not really,

least, meaning it's behind the flag, and therefore it's not enabled yet. Yes, I would say it allows you a lot more free range, but I think at development time, it actually means you have to think a lot less. And I'll tell you why that is. So when I, this is standard now in Google, in Cloudflare, for example, is another one. And we're starting to do the same thing at Dimetrace, where every feature just gets a flag. What I like to say to people is the developer is deferring and delegating the decision of if that flag is going to be on or off to later. I don't think about that when I'm writing. I don't think about who's going to see this when.

I don't think about the rules that govern this release at all when I write it. I just put my feature behind a flag. And then later, an SRE, a PM, whatever, or maybe me is going to, before we release it, say, this is going to be on for so-and-so. maybe we're going to do a scheduled rollout, maybe we're just going to turn it on. The only decision I make when I write the flag is what the default is. So I choose a default, and then I'm delegating and deferring that decision to later. That's really the beauty of the whole thing is you don't have to bake in at development time, any kind of jurisdictional rules or, you know, like I said, like age-gating rules or subscription rules, even.

All of that can be deferred. Cool. I never thought about this this way. So is this kind of like, if I think back 20, 30 years ago, when we talked about test-driven development, you basically wrote the test first and then the code. In this case, you always start with the flag. So it's flag-driven development. So every feature-driven development,

feature-flag-driven development, that means every feature always has to be baked into a feature that can be turned on or off with a flag. I mean, this makes a lot of sense. And I really, what you also just said is really, cool because the developer who builds a feature does not need to make the decision when or for whom this feature will be available because that's decoupled on the rule set that you have behind the scenes where you say hey you know we've developed this but it will only be needed during their

black Friday holiday campaign or something like that right and I can and maybe only for certain states in the US or maybe for certain countries yeah yeah absolutely. Like, being somebody who has written enterprise software for a long time, it adds a lot of cognitive complexity to have to be thinking about releasing as you're actually writing code. And so I think if you just go into this flag-driven development paradigm as you're calling it, which I think is a great name, then you basically just remove that from your mental model. And instead you just focus on writing the code, just put the flags in at the right pivot points, which is a really generally a pretty easy decision to make.

And then document it. And then like, yeah, you say like here's all the flags. Here's my manifest of flags for this feature. We can control them in a complex way or make it really simple and just turn them all on. Whatever it is, it's as the business and release dictates later down the road, make those decisions. And I also, I had a conversation last night. So we had a little achievement celebration here in the Dandreys office. and I talk with one of our colleagues,

and he said it took him a long, long time to understand really the power of feature flagging, feature flag management and open feature. Because he said back in the earlier days, at his career at Dinotrace, we had feature flags, but they were always very granular. Sorry, not granular, but they're very broad. So you turned on a feature for a complete cluster

or for a complete tenant. You had no option to say for that particular user that opted in to do a certain thing or that user that had a certain capability already turned on and then the other feature also made sense. And he said now that you can do this, that fine grain, that dynamic, I think you talk about evaluations where a feature flag can be evaluated based on the conditions we have not only about the software that runs, but also about the end user, then this gives you so much more flexibility. Yeah, absolutely. Yeah, I think one of the things that really empowers that

kind of delayed and robust decision making you're talking about. And this is where you do need to put a little bit of forethought in it, but it's a one-time cost, is what ambient data do you want to feed into the evaluations? So you make this kind of decision usually once or twice. And I mean, as you mature your application, you might add more data there. But basically, like, into the rules engine, you feed all of this contextual data. So request specific data, data about where the server is running, the geof-the-the-geor of users that serves, that kind of thing.

You feed that in. But as long as you have that information, you can make any of those decisions later. Cool. So that means to sum it up, if the three of us would have a great idea. We have built a software company. We have already some features.

I mean, we have five new features ideas. We can build these five new features. We put them all behind the flag because that's what we do by default. And then we can individually decide or have somebody else decide for us. when is the right time to make these features available. We even have the power to say, let's not turn it on for everyone because we want to first want to experiment. We want to give it to maybe some users that are friendlier to us, that want to give us feedback and don't just run away if something breaks because this is also something. New features can also break experiences.

So we want to contain the blast radius. Yeah, or our most excited users who are always excited to try, latest thing and like they don't mind if it if it maybe goes wrong like yeah that's that's a real use case that we see a lot of we have enthusiastic data trace customers right yeah so so some of it sounds like um there's some semantics just in terms of clearing up the the concept right because i guess some confusion coming because there's also things like blue green testing and all that right i would think blue green testing is a methodology that can be controlled with feature flex might be a good way to say that. Would you say that's a good?

It could. I'd say that's true. I get this question a lot from SREs. So I'm going to, I'm going to use this opportunity to like give my perhaps controversial take. I think bluegreen solves a different problem. So blue green most of the time when I hear people talk about it, I think it's it's trying to do something different. So blue green is in my experience, usually making an assertion that an implicit assertion that two binaries behave in the same way, when we don't maybe know that. So that's one reason, one of the probably the main reasons that people use blue-green is like, I did a big refactor and I think this should behave the same, but I'm not entirely confident. So what I'm going to do is deploy this new thing

and I'm going to slowly shift traffic from blue to green. And if everything looks good, that's fine. But that's a fundamentally different thing than saying, I want to target a specific user demographic, which, as in the example you just gave, has nothing to do with a new binary. We're not talking about anything about a different binary or different running code. We're talking about known ways we know our code can behave differently

and then controlling those. So I think that's probably the thing I would say is fundamentally different between blue-green and feature flagging. But yes, to your point, you could use feature flags to do blue-green and people have and do all the time. Wouldn't you use blue-green though for the deployment part and not the release part? So coming back to the example earlier, we have existing software that is running, and now we have a new software bundle, but that includes five new features.

They're all by default turned off because they're behind a feature flag. We still need to get this code auto running, and we assume we have put everything behind a feature flag, and everything will be just working as before if they're turned off, but we don't know. So this is why you can also do a blue-green deployment, then slowly shift traffic to validate. Everything still runs stable. Hopefully it does, because then you have code running where you can turn on these feature flags.

Exactly. And 100%. That's exactly how I see it. So the implicit assertion in that release, right, is what you said. We assume these things are going to behave different, or going to behave the same. So that's where blue-green is really the most valuable,

but it's fundamentally different from feature flags. Or at least maybe I would say blue-green is like a subset of all the things you can do with feature flags. Hey, Todd, you mentioned that obviously the practice of feature flagging is nothing new, right? I mean, the idea behind feature flagging is nothing new, right? I remember my early days, I had environment variables that I put on the process, and then when I needed to change something, you know, stop the process, changing that environment variable, launch it again.

Obviously, this is the, it's very static, but it basically got the same thing. And I think, as you mentioned earlier, some people implemented it with environment variables. Some people may have implemented their own version of feature flex and controlling them by storing things in the database and reading it from a database. And now my question is open feature.

We've introduced open feature four years ago in the podcast. We unfortunately forgot in the very beginning of this podcast to give you more time to talk about your role in feature flagging. So let me give you the opportunity now to talk about open feature, your role in open feature, and why open feature is really, was really needed and why it's taking off and how it's currently, what's the status of it right now. Sure, yeah. So I think the project is about, it might be approaching about five years old at this point.

But it basically is kind of, yeah, and it was started by kind of a consortium of different interested parties. A couple of people from Dinah Trace, myself and Mike Beamer, some folks from eBay, some folks from Launch Darkly, who's a, I would say, like, the incumbent vendor in the space, yeah, and a few others. And so we basically saw that, like you're saying, there was this great thing somebody from Lash Darkly said early in the process. They were like, listen, our biggest competition is homegrown solutions. And that's a really interesting thing to note.

Like, that's an interesting problem space. And so the idea was like, let's try and develop a standard to make it easier for people to use their homegrown solution. and also to migrate to vendors if they want to. Because vendors see the SDKs. So these are the things that you put in your code to actually do these feature flag evaluations. They mostly see them as commodities. They need to maintain a whole bunch of them for a whole bunch of different languages.

That's a huge development burden. And it requires a lot of people who really know the ergonomics of the language that they're using. So why not kind of do some of the stuff that O-Tel has done and kind of just say, like, let's develop these as a community of people who use feature flags, and then control the feature flags, which is where all these vendors, you know, where they have their specific strengths and things that they're good at, the backends. Like, let's just front those backends with a common SDK. So that was like the start of open feature.

And it exploded. So everybody really got on board with that idea quite rapidly. We have a huge buy-in. I mean, and now there's open-feature native offerings on even some hyperscalers, like GCP has one. And basically every vendor in the space is fully open-feature compatible. Some are even open-feature native, so they don't have their own SDKs at all. They just rely on you to use open-feature, which is how our new Dinotris offering works as well. Got to drop that in there.

Yeah, so that's basically how the project started. And then it kind of got even bigger. So we have a whole bunch of really cool integrations now. Native integrations with Kubernetes. We have an operator so you can control your flags as custom resources. We have native integrations with open telemetry. We've contributed to the open telemetry standards for future flags. Tons of vendors have what's called OFREP compatibility.

So you can actually use our line. protocol to evaluate your flags, the line protocol and Rest API defined by Open Feature. And what a lot of that means, really, is you essentially get for free all of this collective momentum. You know, if one vendor wants to write an integration with Snowflake, so you can see all of your flag evaluations there, that vendor can do that. And then 10 other vendors have to do the same thing. If you're using Open feature, the integration is written once and then every vendor benefits. So you can see how this benefits vendors.

Obviously benefits end users because they have full portability. They're not locked into anything anymore. So it's really just like a win-win. I think that's how I would summarize the whole idea of the project. Yeah. And I think back when we had the podcast four years ago, we also compared it with the open telemetry moment or kind of like a parallel where open telemetry was standardizing the way you instrument applications, making it easier for the

end user in case they want to switch and opening up an opportunity for vendors to focus on what to do with the data and not having to implement each of them, their own agent and maintaining it, but basically putting this into a standard. Yeah, it's very similar. And we've seen to take off. So we're now the biggest SDK over. vendors in terms of downloads. So Open Future is now the dominant SDK out there in the real world, which I'm quite happy to report. Cool. What are the numbers? What do we talk about? The last time I checked, the CNCF has like an aggregate download count. So it's 25 million downloads a month across all of our different SDKs. JavaScript is, you mean, that's

publicly available on MPM. We have about six or seven million downloads of our JavaScript core bits. So that's for server and web workloads, which is, yeah, like I said, in the last couple of years really taken off. I think about, if I can recall correctly, about a year ago, we had about 10% of both of those numbers. So impressive. Hey, and because you mentioned JavaScript, that means this is heavily used on the front end.

more so than on the back end or what do you see? Is there a parody or? I say overall it's 50-50. Yeah, I'd say it's 50-50. What I would say is generally users bias in one direction. So some users do a lot of the stuff on the server for various reasons. It means that you have to expose less to your users. If you're really secretive about your features,

it's much easier to do that on the server. And then some people really, don't care, they can, they don't mind kind of exposing that there is some of these flags in their code to their end users, because obviously if you're doing evaluation on the client, there is some communication happening that people can see that's showing that flags are being evaluated. So some people favor that. It really depends on your specific business requirements. One last question on Open Feature and the adoption, because I've seen this with Open Telemetry, that vendors of SDKs, vendors of runtimes,

SaaS vendors, they basically pre-instrument everything with Open Telemetry so that if you're using their service, if you're building on top of their runtime, then you also get the benefits of good instrumentation. Have you seen this with Open Feature as well? So are there maybe frameworks and libraries and runtimes that I would use as a software engineer

to build my software on top that already come with open feature support, meaning I can change certain behaviors dynamically through open feature? Rucel has some built-in open feature support, for instance, yeah. Yeah, and GCP, like I was mentioning earlier, they have some, it's a bit higher level,

but yeah, they have some built-in support. You would still need to install the SDK yourself, but yeah, it's something like what you're saying. Cool. And I've said one more question, one more use case question. Do you see this more for features? being used? Like really, let's make a feature available, or do you also see this for configuration

changes? One classical example would be the size of a thread pool, connection pool sizes, buffer sizes. Do you see this use case as well where you can then dynamically adjust it? Definitely, definitely. The challenge with those is like you need to build your applications such that they can actually reinitialize those things, right? So there's some, There's some challenges associated with doing that, but absolutely 100% we see that all the time. Cool. So now my next question is,

and this is really why I was so excited to talk about this with you today, we talked about, first of all, we gave an intro about what is feature flagging, hopefully everybody understands this now, why feature flags into the power it brings us. Then we talked about open feature, how it's been taken off and everybody's adopting it. the next thing is when I think about rapidly developing new features behind feature flags in

26 I don't think that most of these features are written by humans but they're most likely written by an AI that is obviously directed by humans but how is feature flagging what's the role of feature flagging in in an AI software development lifecycle what do you see uh I mean, to put it in a nutshell, huge, I would say it's been one of the reasons why the project has become so popular so quickly even in the last year. One is because AIs love standards. They love things that are well specified and can be easily reasoned about and are super well documented, and that's basically how the whole Open feature SDK works. So it's very easy and consistent to use across languages and implementations.

And we kind of take all these different vendors and put them into one API. But I think the bigger point that you're asking is like fundamentally people have a different trust relationship with code and the code that they're running, the code that they're writing, that they're modifying in the last few years that they have in the past. And I think that makes feature flags even more important. So perhaps there's a risk that there could be, you know, PRs have gotten bigger. I think everybody, at least everybody I talk to, that's something they're saying.

PRs have gotten bigger. And maybe to put a really fine point on it, people are writing code faster than it's being reviewed, at least with high quality. And I think that's just an industry chat. challenge. So we're all trying to get better reviewing. We're trying to use AI to help us review. But this all comes down to safety and trust, right? And so if you can know confidently that, well, there's a lot of new code here and it's been reviewed, but I'm still nervous that it was written a lot by a bot. I can at least put it behind a feature flag and control it. I can control its rollout. I can control the blast radius. I can turn it off and revert it. I can change the behavior. I can keep the old implement. even and just switch it on or off. And then if something does go wrong in production,

I don't have to wait for a whole other deployment cycle or like maintain a release branch. And I mean, that's a big part of this too, is like how this simplifies development so you can consider it trunk-based, right? Because the whole point of release branches is I have this thing in parallel over here and I don't want it to be in the main line yet. So I have to keep this separate code branch running. Well, if you can just keep the feature in Maine but off, it solves that. So I think really what feature flags allow us to do from a development perspective is mitigate some of the trust and speed challenges that we're seeing in the new world that is AI-assisted and agentic development.

So I would say that. I would say that's probably the biggest, most important thing. Do you have skills available for the different coding agents that people can show the developers can just add to their projects that basically instruct coding agents have for every new feature that you build, make sure it's behind the feature flag? We do have some skills available. Yeah, and that's certainly something you can leverage.

We also have skills for like migrating to Open Feature if you're using another SDK. But honestly, like one of the things that's that's so good about Open Feature is it's ubiquity, like I was saying. So you can just say like, hey, agent, add Open Feature and AsinFlex this and it will know how to do it. It's it's so common and ubiquitous at this point that even like even some older models are well familiar with it. Now, for me, my point was, I understand I can ask the AI to edit behind the feature flag,

but I don't want to ask the AI. The AI should be smart enough to know in modern software engineering, every new feature by default is behind the feature flag because we're doing flag-driven development. So that's why I wonder, I wonder if there are best practices around this, so that I have no way around it. Yeah, we do, we do have some skills in that direction. or something that you can add to your agent's MD to get this stuff going. And it can do things like, yeah, evaluate what new context attributes who might need.

Like I was saying before, like you want to make sure you have the right ambient context so that you can make those decisions right. Those are the things that might be a little bit trickier than just putting the flag in the right place. But yeah, absolutely. Agents, agent harnesses can benefit really from like those kinds of skills. Honestly, I think it's something that we need to get better at publishing standards for because we are kind of like thought leadership in the space. So I think it's something we're going to keep revisiting as we support people who are using AI

to develop feature flight code for sure. Do you also see, because you earlier mentioned, coming back to our company, We have five new features. We release them behind a feature flag, and then we make decisions based on external factors, a certain time, a certain legal requirement. But it's basically the humans making the decisions when to turn a feature flag on,

when to turn it off again, because maybe we see bad behavior or maybe that marketing campaign is over and we sold all the products and then we turn it off. Do you see that this is completely automated in the future with AI, that AI controls, also the rollout decision? And then also maybe they're making a decision when it's time to pull it back.

Definitely. I mean, completely, really that's, I think, more of a policy issue. But is it technically possible and is it desirable in a lot of situations? I think absolutely. And we're definitely seeing that happening. And this is part of the reason why I think Dinotrace is even interested in feature flagging, as well as a lot of other telemetry vendors. there's a, yeah, like, we're seeing a lot of telemetry vendors in the space actually come out with open feature native feature flagging because I think that closes the gap that you're kind of talking about, right?

So it's one thing to say a human can turn the feature on and off, but if you have rich telemetry data about the feature flag evaluation, so is this particular variant? And when I say variant, I mean like this particular outcome of a flag evaluation, right? So it could be on or off or it could be one of seven different values, whatever it is. Is this particular variant associated with increased error rates? Is it associated with increased rage clicks if it's a UI and user frustration? These are all things that if you have rich telemetry, you can feed back into an AI, an AI ops agent, for instance, and you can have them govern the rollout based on real data. And I think this is huge.

This is another place where we have a lot of opportunity to instill confidence in people. So I see, I would love to see a world where basically no flag is just, maybe few flags are just turned on in a quote unquote dumb way by just turning the flag, but saying like, okay, I think we're ready to turn this on. Let's turn it on, but let's have it monitored. And so every feature becomes basically a rollout that's governed by some intelligent telemetry and some observability AI, actually making sure that the rollout's going smoothly. I imagine from getting on board with that point of view, we'll probably see more best practices coming out as there's more adoption, right? but a starting point would be find some safe flags that you can put in the hands of the AI, make sure everything looks good, make sure it's operating okay, not flipping flags at random, right?

But the idea would be you can control which ones it has access to, I would hope, right? And then as you get more comfortable and get the policy set, then start rolling that out more and more. Has there been any sort of best practices for this written up yet, or is it more an exploratory phase or experimental phase at this point with giving governance to the AI? I'd say it's experimental. I mean, I'm sure there's people who are doing it, but it's experimental. I mean, humans driving a similar sort of process is something that's decently well explored. But yeah, having it being like fully agentic is really a different thing because when humans have to manage this process, the cost to doing it is very different, right?

So you have to have somebody who's actually making these intelligent decisions, looking at a dashboard, looking at reporting, and so you can't do it all the time because there's an inherent human cost. So I think, yeah, cost and time, that whole decision changes when we have AI models that can kind of do this for us. I imagine if you do this agentically, you're going to have the ability to add in flags for a lot more things that you wouldn't have added in previously

because the effort to make a decision for a human to turn those on or off might not be worth it. But if they're smaller maintenance things or this or that and if the AI is going to handle it, that's going to give you a lot more flexibility on really fine-tuning your application throughout the course of the day or time periods or when things are occurring. So it's really a great prospect, right? Because there's all these things you might not have thought of because I'm not going to, I don't want to deal with that manually. Yeah, it's shifted the cost equation, right?

So it's shifted the metric or the heuristic that you run in your head about whether or not you want to have a flag here and then whether or not you want to manage it yourself, whether or not you want to do some kind of roll it versus just turn it on or do an experiment with it. Like all of those decisions are now impacted by the fact that we have these like pretty intelligent models that are really, really good at gathering lots of data and then making decisions. Todd, one more question on the lifecycle of software with a feature or a flag development style. That means for me, this sounds like over the course of time I will have more and more and more features and feature flags piling up. Is there a consensus now on what happens with those features that will stay?

Will they then just be removed, meaning these control blocks around it will be removed or the code will just be there? But the feature flag management solution basically doesn't allow you to change it anymore. Is there any? So my question, basic, is code cleanup? Or is it you just prevent to turn it off? Yeah, I think the question to that is kind of a both-and, because I think what most really mature users have found is that there's kind of a taxonomy of flags.

There's different types of flags. So there's flags that are inherently temporary. And so these are like release flags. Like I have this new feature. I know that this is a net new feature and has a whole bunch of new parts. and this flag exists just to ease that delivery or make it safer and to maybe experiment with it, do some data, whatever.

Those flags, most people refer to them as like release flags, and they're inherently usually temporary. And then there's like ops flags, and that's kind of what you were talking about with configuration. These tend to live forever. As long as that feature is there, there's no reason to clean them up. You may want to add new variance to them, change the numbers around whatever.

But they're perpetual. There's also like security flags, securely related like circuit breakers or, you know, things that cut off a whole swath of functionality because it has a security issue or because it has a performance issue associated with it, those are perpetual. And then there's flags based on experiments,

kind of similar to releases. They're usually inherently temporary. But they might not actually necessarily relate to a new feature. It could be relating to even an old feature that you want to experiment on whether or not you want to change it or if people use it or whatever. And they're also inherently temporary. So I think it really depends on the type of flag, like the species of flag, I guess, that

you have. But the tech debt is real, right? Especially with the release flags, which are very common. These can build up in your code base and not. Not only do they build up with your codebase, but they build up in whatever management tool you're using, whether you wrote one or you're using a vendor. And this is, again, where telemetry data is a game changer because you can see if a flag is being evaluated. And not only can you see if a flag is being evaluated, because maybe a flag is being evaluated. But is it always evaluating the exact same way?

And has it always evaluated the exact same way for the last year? Well, it's probably doesn't need to be there anymore, right? essentially have like a branch in your code that's dead. So if you have telemetry feedback here, you can feed that back into your management plane and say like, hey, this flag is probably stale. Like you might want to get rid of this. It hasn't evaluated differently for anybody in this amount of time. Or maybe, hey, it has, but it only evaluated differently for this guy named Andy who

was like, you know, manually changed it for himself because he was like messing with something. Who knows? But that's the kind of thing that you get when you have the rich telemetry data for sure. So I think that's another thing that's important. One more question on the AI topic, but not on the code generation, but more on controlling maybe models through flex. Is this also a use case? So if I'm, for instance, building an AI application and for whatever use case,

maybe a jet bot on a travel system, and I may have the option to use different models. Maybe I can only use certain models in certain regions, depending on where the user comes in from. Is this also a use case you that you've seen, that basically feature flags are used to then control which model is used behind the prompt? Of course, yeah. And so jurisdictional concerns are one thing.

We definitely see that a lot. But I would say that it's, there's a lot of, so fundamentally, right, LLMs and models in general that we're embedding into our code. So like you pointed out earlier, I think this is a distinction really worth noting. When we talk about AI, we can talk about AI generated code and how that involves feature flags,

but you're talking about AI embedded in our workloads themselves, right? And so that's a fundamentally different thing. So I think they're equally important and feature flags are equally applicable to both of them. But fundamentally, right, what is a model? It's something that is non-deterministic, right? We're used to as software engineers dealing with determinism. We can say, like, I can trace this code.

If I know all the inputs, I can tell you the outputs. Well, that breaks down fundamentally with large language models. Models are very complex. They have billions of hyperparameters. We're beyond our ability to fully reckon with them in the same way that we would with an algorithm, right? So being able to control those with feature flags is critical. Things like model temperature, things like guardrails, things like the model and use itself.

These are all things that you're going to want to be able to dynamically control. And guess what? Here's the other thing. You can't really understand how a model is going to behave until you see it in production. You can lie to yourself and say that you will in your staging environment, but you won't. because user inputs are way more diverse in production. It has a different level of scale in production. You're never really going to be able to know the behavior of any given model outside of that scenario.

And so this is where inherently a lot of people end up actually testing in production when they don't intend to. So my prescription is test in production intentionally with feature flags because you're going to end up doing it accidentally otherwise. And why not have all these controls around your models? I know you just mentioned things like temperature and guardrails and everything. This is terminology. Folks, if you're not familiar, the way you can basically configure how an LLM should react to your prompt.

And obviously, this is then great when you can easily, dynamically change these things and don't have to roll out a new release to just change one of these flags. Todd, we are getting towards the end of the podcast and I know there's probably a lot of stuff to talk about but this one is going to air. This podcast is going to air end of September which means it's only six weeks out to KubeCon in Salt Lake City. I assume Open Feature will have a big presence there as well.

You will be there. Yeah, we always have a booth in the Project Pavilion and I'll be there. It's one of my favorite things to do is hang out of the project pavilion and talk to users, honestly. Are there sessions as well that you are giving or that you know, I don't know, maybe adopters are talking about? Yeah, there is. There's a session by an adopter from CVS Health, which is a big kind of health care and pharmaceutical vendor in the U.S. if you're familiar with it.

They've been an adopter early of not just open feature, but also are kind of reference backend implementation that's cloud native called Flag B, which is really fun. So they have a talk. I have a talk. I think there's one or two more, and there's also like a maintainer overview.

There usually is one of those as well. So I can try and get you all the links for those. Yeah, it'll be cool because, you know, If we happen to be in Salt Lake City, that's a great way to meet Todd and the team and adopters and talk to them. And, yeah, you hinted to it earlier. Brian and I typically don't want to use the podcast to promote the work that we do in our daily life. We all work for Dinah Trace. But you hinted that there are some feature flag management also coming from Dinah Trace.

Do you want to maybe just slightly pitch it in the... Yeah, I mean, it's a fully open feature native like you might expect. And obviously it takes advantage of a lot of the existing rich telemetry data and observability data that Danetrius has. So it kind of loops those both in together on one pane of glass for you, has all the nice features you'd expect. And I mean, I'm personally really hoping to reach a lot of big vendors with this who maybe don't realize that this is something that they need, but I think they do. And a lot of this technology that also comes in here into the data trace is part of the,

or came in through the acquisition from DevCycle, if I remember correctly. Yeah. Yeah. We're also in almost at the ground floor with open feature as well. Yeah. Yeah. Cool. Did we miss any critical topic? Do you want to do anything we, you want to make sure that people hear about? I really wanted to talk about the AI stuff, but I think we did a good job covering it.

So I can't think of anything to critical. I mean, I could talk about this stuff for days. Like, one of the things I'd like to talk about is like some of the PII privacy security implications of feature flags. But I think it might be a bit too in the weeds for a last minute add-on. Well, maybe an episode too. Yeah, possibly. Cool. All right.

Brian, what do you say? Learn something new about feature play? Yeah, I mean, it's definitely come a long way since its beginnings, right? And I do recall having conversations in the past about the open feature thing, and it's amazing to see how, well, it's been adopted because it seemed just like a no-brainer to, like, you know, it's standardized. But I think the really key thing is how much of a world using feature flags opens within your software, your applications, so that it's key to being flexible, like everything. It's just like it almost seems like it's an imperative for people to use feature flags in their software so they can handle any kind of situations that come up and move on the fly.

and it's great to see that it's so solid now and there's so many ways to, even with the AI interaction, it's going to be exciting to see what people start doing, especially once you take that, as you were saying, the cost budget out of it and you can start thinking about building anything into it. So great stuff.

That's it. All right. So thank you, everybody, for listening. I hope you learn something today, and we will be back with you all in the next episode. Bye-bye. Yeah, thanks a lot, guys.