Software Huddle - Architecting for SaaS with Bill Tarr

Episode Date: July 16, 2024

This week on the show, we talk with Bill Tarr, Principal Solutions Architect at AWS SaaS Factory. He's a super thoughtful guy, expert in SaaS architecture and architectural patterns. We talk about tenancy, infrastructure decisions, SaaS gotchas, security, permissioning, and even dip into technical challenges of building Gen AI into SaaS. Timestamps 00:01:29 Background 00:07:26 Common Challenges 00:11:46 Infrastructure Choices 00:14:29 Missteps 00:18:57 Control Plane & Application Plane 00:25:52 Permissioning in a Multi-tenant setup 00:32:54 Gen AI & SaaS 00:35:07 Amazon Bedrock 00:49:27 Quickfire questions 01:01:17 Security In SaaS

I've been thinking a lot about security in SaaS lately. There's been some really high-profile news stories about ransomware attacks on SaaS. They've really caught my attention. And if you're in the business of SaaS, they should have caught your attention as well. Can you explain, starting with the control plane and the application plane, what are each of those responsible for? And if I'm essentially architecting for SaaS, how do I think about each of those different components? Let me work backwards to that

question because I like the way you phrased that last part of the question. When I think about these different planes of an application, I like to start from the profiles who are accessing those applications. If you could invest in one company that's not the company you work for, who would it be? Oh, Skyflow. Good answer. Hey folks, Sean here. And this week on the show, I talk with Bill Tower,

Principal Solutions Architect at AWS SaaS Factory. I've known Bill for a few years. He's a super thoughtful guy, expert in SaaS architecture and architectural patterns. We talk tenancy, infrastructure decisions, SaaS gotchas, security, permissioning, and even dip into technical challenges of building Gen AI into SaaS.

I hope you enjoy it. And if you have any questions or suggestions for the show, please reach out to myself or Alex on social media. And with that, let's get you over to my interview with Bill. Bill, welcome to the show. Thanks, Sean. It's good to be here, man. Yeah, I'm glad we were able to do this. So I wanted to start off with a little bit of your background. So I know you're a principal solutions architect at AWS SaaS Factory.

So what does exactly that entail? Like, what's your sort of day to day? Just can you kind of shape the picture of that for people who are listening? Well, I'll tell you what I did today. I mean, every day seems to be its own day. And it happens to fall all over the map. My day started today with a really awesome call with a new Greenfield customer, who's looking at, you know, they've got software that they already built, you know, and it's been around for 20 years. They've been operating it, they've been basically handing it out to all of their customers and they're looking to create a SaaS model, to create some recurring revenue from that.

In the past, they mostly just relied on the services that they gave alongside that software. But there's value in the software itself. They've developed 20 years of experience on how to develop workflows and manage the software in their industries. And they're looking to build a SaaS product. So I started off my day with an awesome conversation with a customer,

had some really interesting internal conversations about some of the thought leadership we want to build out, some of the challenges we're facing in terms of how we better provide services to SaaS providers. And then another great SaaS conversation I had in the middle of the day, which is often the case. I'll often talk to one or two customers a day. Usually it's just for an hour in this new virtual world. We used to do workshops for days at a time. This new virtual world, we've all been conditioned to focus for exactly one hour on a topic and then move off to the next topic. So a couple of those a day, that's pretty typical. Some internal conversations, a couple of blogs I'm trying to push out, a couple of customers we've worked with

in the last year that I was really happy with and we're trying to get some things published together with them. That's kind of how today and now I'm here. So eight hours into the day, now I'm talking to you. And this is one of my favorite parts of the day, by the way, is getting to, you're part of the SaaS community like me, getting to talk to people in the SaaS community and keep this conversation going and keeping SaaS fresh in people's mind and how we do SaaS better. Yeah. So I guess like going back to that first conversation that you were talking about, that pattern essentially of maybe someone's, what we would consider maybe like a legacy company had software around for a long time. Maybe they want to modernize it. They want to explore new

revenue streams or something like that. Is that a common pattern you're starting to see where people are looking at their software solutions and seeing like, hey, can we adapt this to more of a SaaS modernized SaaS model that we deploy on the cloud or something like that? Yeah, very much so. I think I've been with SaaS Factor about four and a half years. And when I first joined, the very first customer I had was a modernization story, but it was a product that they delivered on-prem. And this was sort of the typical migration to SaaS. Say, we've got this product, we're deploying it to customers' accounts.

We'd like to turn around and operate this as SaaS. Still having that conversation today, of course. That hasn't changed. There's tons of on-prem software out there still. But this conversation about, well, we've got software that we've been building over the years, but we're not really selling it. We're not really, we've just got it. It's an internal piece of software, which is some of the conversations we have.

It's software that we give away for free. These different conversations are definitely evolving. And we're seeing more and more companies really looking at their portfolio of software and saying, what do we have here? You know, is there a chance to generate revenue? Is there a chance to improve the state of the industry that we're in? Because probably the best part of that conversation, these are customers are customer obsessed, right? They were talking about their customers and how they're able to save millions of dollars for their customers with the software that they've built. So they've already delivered to one internal customer. They managed to cut the time to do the projects. And in this industry, there is a little specific there in telco.

So the industry that they, you know, these are months or year long projects, and they were able to cut the time to deliver down to a quarter or a third of the time. So they're able to help their customers. And this is how they're viewing it, right? If we can deliver this as a SaaS model, we can reach more customers and help them and save them more money. And in turn, we should be able to monetize this as well. And this sort of introspection on all of the software suite and that everything should be moving toward a SaaS model is just increasingly finding more and more of the niches that we missed along the way, especially, and I mean, the telco part of this is

important. These highly regulated industries were some of the last ones to move, but they have some of the highest impact. Healthcare, telco, these things touch our day-to-day lives for all of us, right? And all of us have a phone in our pocket that relies on this crazy network of infrastructure out there that didn't even exist a few years ago, right? It's amazing stuff. And the fact that these, you know, a lot of these industries were very slow to move because they're naturally cautious. These conversations are starting to happen and we're seeing really cool software start to emerge out of those spaces. Yeah. And I think like a lot of companies that have been around sufficiently long enough are probably sitting on a mountain of like internal tools that they built to solve,

you know, specific problems. But there's probably, if that solved, you know, your problem for whatever your challenges were, there might be a market for that. Like, I always felt like, like, you know, Google spends a ton of sort of calories on trying to do just moonshot projects and super innovative stuff and find these new, you know, $100 billion revenue streams. It's like, they just kind of look internally at some of the stuff that we've built over the last 20 years and turn those, like actually productize those. They probably have, you know, billions of dollars of new line revenue that they're just like not even really paying attention to. And there's been a ton of course, companies that spun out of that those, you know, internal types of projects to create those types of revenue streams. So it makes a lot of sense that companies

are kind of looking at these internal projects and seeing, is there an opportunity to make this bigger and offer it as a solution to people outside of their own employees, essentially. Now, when people are going through this journey to build SaaS, whether that's modernizing something that exists or even doing that journey sort of net new, what are some of the sort of common technical hurdles and challenges that they tend to run into in your experience? It's an interesting question. Increasingly, I think about this question in terms of stages, because, you know, a lot of times when you're trying to guide companies, you try to provide the sort of middle ground, like, oh, here's the path we expect you to take. But if you think about

SaaS, SaaS ranges all the way from the earliest greenfield born in the cloud products, where they're going through, say, Y Combinator right now. And they've got this great idea. They've got an application. But that's what they have. A couple of really smart guys, smart people with applications that they want to deliver to their customers. And, of course, SaaS is the delivery model for the world now for software. But they really want to focus on this application. So the challenges they might be facing are how do we build this application

efficiently so we don't create a ton of technical debt later so that we can handle the customers as they're coming to us, you know, identity. You know, if we get a big enterprise customer coming to us, they're not going to want to create all of the identity stores that they've created of their 10,000 users in your SaaS application. So being able to have an SSO federation story with your customer's identity store, that can be a pitfall of an early stage customer. Security and isolation, that's a huge one. And that's a huge one across the board, but especially early, making sure that you've got a strategy. If you're going to have on board 10 new customers, how are you going to keep their data isolated from other customers? And early on, when your SaaS journey, if you haven't built SaaS before, it might seem obvious, but actually having policies in place that prevent traffic from accidentally crossing

over those boundaries and exposing tenant data, right? These are the early stage pitfalls that we often talk to. On the other hand, you could talk to an enterprise customer, right? And it might be very different in terms of the profiles. You know, their challenges might have to do with growth. And, you know, we've got tens of thousands of customers and we built this SaaS product, but we're not able to scale. It takes us too long to onboard our customers. It takes us too long to onboard our customers. It takes us too, you know, we're spending too much time operating the product

and not enough time innovating. So those challenges tend to morph over time. And when you're talking to SaaS customers, you really have to understand where they are in this journey, right? Are they kind of still searching for product market fit? At which point, you know, hey, it's focus on the application. Make sure you don't shoot yourself in the foot with security, don't leave the door back door open and let people in

to take over the application, make sure isolations in place, you know, bring in people, if you don't know how to do those things and lock those down. As you grow into that growth curve, it's usually the challenges of, hey, we can't grow fast enough, right? We built something, but we don't have the automation built in, we don't have the frictionless onboarding story built in. And then as you kind of get into that optimization phase, it's, hey, we've lost agility. Like we're not really innovating at the pace we were. We're getting a little bit lost in the details of operating this thing instead of actually innovating on behalf of our customers. How do we get back to innovating? And then it's a

lot of it's about automation, about analytics, about being able to really observe what you're doing and understand the state of your customers at any given time. And what the health is of your SaaS solution without spending lots of human resources doing that, those analytics, have the application start to work for you and that control to, to give you a single viewpoint into everything that's happening in your application. Yeah. I mean, there's a lot, I feel like, um, and this is true of any, you know, customer like, um, journey with a product or with a business is you each stage, you're like earning the right to new problems. You know, in the early days,, your problem is probably not scale because you don't have any customers. So your problem is, I don't have customers, or I don't have a product or something like that.

So you earn the right along the way, and that's going to change the nature of even the technical challenges that you're trying to address. So from the very beginning, though, if I'm of starting out, I have an idea of what I want to build. Does the fact that I want to build for SaaS, like, how does that impact the kind of infrastructure choices that I'm making and the types of tooling that I need to deploy? Yeah. Again, I mean, it could depend a little bit on stage, but I would probably emphasize as I'm thinking about SaaS, it's a modernization play, right? It's not as simple as, hey, I'm going to build a VM, right? In the simplest sense of the word, I'm going to build a VM and deploy my application. You certainly can. It's not going to scale very well. You know,

the things we're trying to get out of building SaaS, like cost optimization, like operational efficiency, really align well with modern application development, right? Whether that be serverless, right? Whether that be serverless, right? A lot of great SaaS applications have now completely shifted over to serverless, right? They're running on Lambdas, they're running on Fargate, if you're on AWS, of course, for these services. They're using containers. Kubernetes, of course, is in, I'd say, more than 50% of the SaaS applications out there for good reason, right? And the ability to scale, to grow, to optimally use resources, all of those things configure into both your cost and your

operational footprint. So those choices you make on infrastructure kind of play across the board, like that's mostly the compute side. And then you get down to the data side and the same kind of stories evolve, right? How am I doing multi-tenancy at a database level? Am I sharing databases, database instances? Am I creating multiple schemas in a database? Not even sharing the same tables across multiple tenants. And those choices, they're all two-way doors too, right? So even if you make these decisions like, oh, we're going to do it this way, we're going to have an instance per tenant, over time, you're still even continuing to evolve those infrastructure choices. So like where SaaS really differentiates itself, if you build an application

and you deploy it out to people, that's probably the application. It's going to evolve over time, but you picked your architecture. It's very hard to diverge from that if you've delivered software remotely out to people and they're running it on their desktops. SaaS, you can continue to evolve it, introduce new pricing tiers and say, here's a whole other application stack we've come up with that runs the same code, but is much more cost efficient. And that's going to allow us to approach an entirely different market than we were before, where we're say we were only focusing on enterprise customers, or vice versa, we had a completely shared product. And now we've got a whale of a customer, a giant bank coming to us saying they want their own stack. Now we figured out how to deploy our application as a complete silo stack for that one customer. So these architectural choices are pretty diverse, right? And this is even just I'm just mostly just talking about a regular full

stack development, there's, you know, data pipelines and all those different versions of this as well. Are there, you know, choices that you've seen, maybe classify them as, you know, missteps or oversights that people make early on that end up being things that they have to sort of correct for later? Whether that is either either decisions from an infrastructure standpoint that impact scale challenges later, or even cost where maybe they're just like, well, we don't really have a lot of customers right now, so we don't need to over-optimize things.

But maybe that leads to a situation where they're overpaying for cloud resources down the road because as they scale, they haven't done things in a certain level of efficiency. Yeah, very much so. I actually did a reInvent talk last year on pitfalls, on SaaS pitfalls. And we tried to enumerate some of those. It's hard.

It's hard to enumerate all of these because there's so many ways. You can accidentally paint yourself into a corner if you're not careful with architecture. And SaaS simply amplifies that. If you build a bad architecture for a single application, you've developed some technical debt, you've got to get out of it. If you've deployed it out to 10,000 people and they're all using it, now you have to go and correct that problem across multiple customers, across their users, potentially got to retrain. There's a lot of different aspects that can affect you. Now, specific to what has affected the customers I've seen, I think as that growth phase has happened, customers who've made isolation choices often have a hard time getting out of them.

Right. So I talked to a customer that was growing really rapidly, incredibly successful, building a headless CMS on AWS. And I love their product. It was a cool product. It was a great market that they were in. But they developed a silo architecture, right? They were, you know, spinning up a new AWS account for every customer that they onboarded.

It was successful. They were able to keep doing it, but it was getting operationally more and more difficult. It was hard to keep spinning up these AWS accounts. Their SREs were jumping around between accounts trying to correct things. They didn't have enough automation built in. So you can imagine that's a pretty typical story. And of course, the cost of that is impactful, right? This is another example of this is one of like another example of this that we spanned up for that pitfalls talk

was I had another customer that did something similar, except only at the database level. They were spinning up an open search instance for each one of their customers. And it was getting very expensive. And they just didn't stop and, you know, just stop doing it and say, hey, let's revisit this decision and think of there's a pooled story behind this.

Once we had that conversation, they made that change and were able to operate with just a few open search instances very efficiently. So it often is along those lines, these isolation choices and the architectures. Right. So it's like, how do you sort of optimize around like what can be shared across you know tenants versus we're gonna like the extreme is we're gonna create a separate aws account for every single customer and then we have no sharing which might be necessary for certain types of industry certain types of customers from like a security perspective but majority customers are probably fine if you're doing some level of shared resources across certain aspects of the AWS account. Yeah, I think if you scratch the

surface of that a little bit, there are almost always things that are more sensitive than others. If you just look at the compliance profile of specific services, what's the data you're storing? Can you actually isolate that data in some way? Can you say, you know, this service really needs to be siloed. But these other ones over here, my catalog of products that I'm selling, that's not PII. Like that's not something that I'm concerned about someone else finding. I can share that data a lot more easily. And then even if you are going to make these isolation choices in general, not breaking down your workloads and saying one by one, like what is the isolation profile of this workload versus this workload, and potentially saving quite a bit in terms of the overall cost and operational profile of those applications. SaaS applications, there's often a combination of different components and patterns. We often talk about these

two halves of a control plane and an application plane. I want to start there, breaking down these core components and patterns. Can you explain, starting with the control plane and the application plane, what are each of those responsible for? And if I'm essentially architecting for SaaS, like how do I think about each of those different components? Let me work backwards to that question

because I like the way you phrased that last part of the question. When I think about these different planes of an application, I like to start from the profiles who are accessing those applications. Because in a lot of ways, the control plane has a completely different audience than your application plane. Your control plane is very much back office software, right? This is what you as an operator

of your SaaS solution are going to use to operate your solution. And that's just like every other control plane of everything else ever built. There's a control plane to EKS, right, for our container service on AWS. That control plane is where people can go in and tweak the settings that will deploy how we're running all of our EKS clusters. SaaS is no different. You know, it's what are we operating? What are the different deployments of that that are out there? How are we deploying them? What's the state of the individual tenants we have in there? So tenant management is one of those first items we think about. How are we billing the customers for all of those? What's the state of billing? What's the telemetry that

we've collected about billing for each of these application planes? What's the user management? Who are the users who are using these individual applications? How are we managing those? Metrics and analytics can be a control plane topic, depending how you've implemented your solution. And configuration as well. Like the configuration is one of my favorite topics in terms of SaaS, right? Like how do you actually manage the configuration of your applications, of the, you know, even the entitlements of what are, what these individual applications and their tenants have access to could be a control plane concept as well. And sometimes these control plane concepts are implemented in your code. Sometimes they're

third-party services as well that make up part of your control plane. But, but the easiest way to think of it, because you might be accessing different things, is what are those users? Are they your SaaS admin users, as I would sort of describe your product owners and the people who may actually manage your SaaS applications? Or are they your tenants themselves? And if your tenants are accessing it, I think the best definitions I've heard is that's not your control plane. Your control plane is for you and your your company. If you've developed some other application, other plane that perhaps your tenant admins access, maybe that's a different plane. Maybe that's a management plane of some kind. But keeping those two audiences completely different is a really useful exercise for thinking about the responsibilities of these different planes. Right. And then your other part about the application plane and what's its responsibilities. Well, for the most part, it runs your stack, right?

Like when you deploy a stack out there, it may have some responsibility for maintaining the state of that stack, right? So it might be, your control plane might just send a command, like an event bridge event saying, hey, go deploy the stack. And it goes and it pulls in a repository

of configuration with some infrastructure as code and kind of deploys itself. It bootstraps itself, if you will, right? So that might be one of its responsibilities. It also is responsible for emitting metrics, right? Like we really want to nail down that every application we deploy should be to some centralized place, whether that's a third-party product or whether that's a, you know, centralized analytics platform that we built ourselves, we need to be emitting tenant-aware metrics, you know, whether that's traces, logs, or metrics and analytics themselves. And we also need to be emitting billing telemetry as well. And a third one, cost telemetry, right? We need to be able to understand, especially if we're doing anything shared, what it costs to use that software and understand the cost profile supporting individual tenants in the application plane. And in terms of like from a data perspective, if I'm in like I have my application plane and my control plane, and then I'm deploying this for

specific customer. And then that customer has sort of like proprietary information that's relevant to the application. Is that typically going to live within the application plane? Yes, yes and no. So it can kind of, so there's a fine line between the application plane and the control plane. I would say the data itself could be in the application plane. I have had some customers who even break out yet another plane and say there's a data plane. And that could be an interesting way to think about what you're talking about, because let's go back to sort of that isolation conversation, right? So I might have an individual AWS account. It might have multiple tenants in it. Maybe I have a VPC per

tenant. So I've actually isolated my tenants into individual VPCs. In that model, I might actually have a separate VPC where all of their data lives. So I might have a completely shared data model where there's a single RDS database. All of my tenants have a schema inside of it in a Postgres database, for example and they're all sharing that one vpc so so there might be different paradigms for isolation even within the application plane and the control plane so like again this could get a little bit more intricate across the different implementation hopefully that answers your question like i want to i think i got what you're talking about there but but. Yeah, well, I guess like, how does, what factors into that decision

of like where I essentially put that data? Yeah, I mean, a lot of it. So we're going to get into a different topic here, I think at some point, which is also, you may not be able to bring the data into your SaaS application. You and I have had this conversation before. It's an important one for SaaS providers.

If you own the data, if this is your data, right, then that decision can be very simple or it could be very complex, depending, again, on the cost and operational profile of what you're trying to deliver. Now, it's a whole other ballgame if we're talking about our customers' data. If we're a SaaS provider that handles PII or IP of our customers, well, now we're in a very different situation. Our customers might demand data doesn't leave their account, right? So we might actually be forced to make hard choices about where we deploy our software, or we might have to be able to offer them some

concessions about what the data is and how we handle it. So, I mean, you work for Skyflow, you know the vault model very well. We might tell them, hey, just don't send us your PII. We want you to keep your PII and strip that out before you send us the data. We might be, you know, they might bring their own keys, their own KMS keys, right? So a customer provided key, so we can't actually access their data. We're handling their data, we're processing their data, but we don't have any visibility into it. That's another model we see.

But the data almost always makes this more complicated, right? Especially if it's customer data and especially if there's some sensitivity to it. Okay, absolutely. And then in terms of like in a multi-tenant setup, how does like permissioning work within like the AWS account? Like, how do I make sure that, you know, you know, one of my customers essentially isn't able to access another customer's like information that they're storing for the particular application? Defense in depth, I think is probably my best answer to this. So like, there isn't a single one layer that I would talk about. This starts with identity in SaaS. Almost all cases, I want to be able to identify users as they come into my application and their tenancy,

right? So the simplest examples are using a third-party IDP, whether that's Cognito, our own service, whether that's Auth0. I want to make sure that as a tenant comes into my application, as a tenant's user comes into my application, I can tell who they are, what their roles and responsibilities are within that tenant and their tenancy. And I want that token, that SaaS token to persist through my application. So this is the front door of my application. And right from the very beginning, I want to have this locked down where nobody's coming into my system without having the SaaS token available to me. That's the first piece of this. The second sort

of layer that we put around this is since I've got this tenant identity flowing through my system, any part of my application that accesses any resources, whether that's a database, an S3 bucket, I want to be able to use that tenancy to create isolation policies that determine what they can and can't access. This could be a runtime decision, where as they're coming through, I'm looking at the tenant and saying, okay, this guy can reach this database.

It could be a little bit more structured if we have silo, right? It might be like, hey, this database can reach this, this compute can reach this database by policy. So that's also possible. But those runtime decisions are very powerful and also open the door to more sophisticated permissioning rule-based systems. Like Amazon verified permissions. of a rule-based policy system that can allow us to not just say like you, this tenant has access to this, but create sophisticated rules around what individual users

and tenants and roles can access that give us a much more granular logic around this as well. So if I had something like, let's say individual RDS instance for each customer, but I have a multi-tenant set up for some other parts of essentially my application stack. I'm going to use some likely identity provider to identify the user. I get an off, you know, basically a token that represents who that person is. And then I'm also going to have some representation of tenant and I'm passing that essentially through. And then when it comes to actually querying the RDS instance, can I essentially have a policy enforced there where it's not going to

allow me to even possibly like query the RDS instance if I'm not essentially allowed to access that tenant? Exactly. Right. And it might depend on the data partitioning or isolation choices you've made for the database. So the example you gave, each one of your tenants has their own instance of the database. So there could be a username and password that we're storing in Secrets Manager, for example. We can use that tenant identity to retrieve that identity. Now that username and password only has access to that instance of the database. It might be a little bit different if you're doing a schema per tenant, or we might want to lock down the data in different ways,

or even in the shared isolation stories, like Postgres has row level security. So if you've actually got a shared story, where your tenants have multiple tenants have data in the same database in the same tables, we can use the policies that are built into Postgres and use those as isolation policies, pass that tenant in and create a session that says, hey, Bill's coming in from AWS. He can only, even if he makes a query and leaves his where clause off, he can only get AWS's data back in that query. So it seems like, you know, kind of going back to some of the stuff that we were talking about earlier, where if you're starting on this journey of SaaS, kind of knowing upfront that you want potentially like a multi-tenant setup and

putting the right things in place for isolation to work, for deployment to work and enforce these policies consistently across essentially all the different services. You have to put a lot of like sort of thought into that to make sure that as you're continuing to build, you're taking those policies into account. Yeah. And a little bit of upfront planning can save you a lot of technical debt later. I told you I had two customer conversations today. The second one was a really interesting one. They'd already decided, you know, this is a pre-product market fit product. They're not a startup, but they have an existing product. They're looking to deliver it as SaaS.

So the conversation was like, hey, you know, we just need to find our first 15 or 20 customers. So we're just going to create instances of this, individual VPCs. We're going to deploy our software. We're not going to build a control plane right now. We don't know if we need one. You know, we're not really sure if this product is going to take off. You know, we need one. We're not really sure if this product is going to take off. We think it's great, but we need to find out what this looks like from an isolation perspective later. But the conversation kind of shifted as we were talking. I was like, you know,

you can make some of these choices now. You could put a tenant discriminator into each one of your tables as you're building this out. Later on, if you need to create a shared tenancy model within your database, well, the discriminator's there. You use the same schema, if you need to create a shared tenancy model within your database, well, the discriminator's there. You use the same schema, whether you're doing silo or pooled, and those choices can proliferate across your whole application. And just because you deploy a thing in one way, it should never be a one-way door. You should never think, great, we've solved the isolation problem. We're never going to have to revisit this. You are going to revisit this if you're successful. You're going to want to move up market and reach enterprise customers with very specific requirements around

compliance. Or are you going to want to move down market where customers have more limited choices in terms of affordability and what they can afford to pay for your application? Yeah, right. Like you don't have to, even if you're, you know, that you're going to go on this journey of multi-tendency, if you're successful, you don't have to build everything from, you know, to basically boil the ocean from day one. But there are choices that you can make early on, that's going to make that journey or easier, if you end up being successful and going down that path. That and it's one of the harder conversations, because you don't want to take like, we always say an MBS, MBS, minimum viable service is what you want to build out of the gate. At the same

time, like, even when you're building, you know, if you're building a bicycle, you know, you might only need two wheels and a handle for now, eventually, you're going to want to, you know, think about what it means to go faster, and maybe you're going to want an electric bike. And did you build in a space for that battery to fit? If you didn't, you're going to have a much harder journey trying to turn your bike into an electric bike, Whereas you've just put a little forethought and say, I might want a battery in here later. I'm going to leave the space open. Well, then that journey is a lot easier, right? It's same type of thing with SaaS. And then as more and more, you know, companies essentially shift to essentially incorporate generative AI technologies, I'm sure there's going to be a

whole new generation of like SaaS that has Gen AI built into the core of the product. Like how does some of these concepts around like multi-tendency start to apply to, you know, incorporating, you know, LLMs and other

generative AI technology into the SaaS stack? Yeah, it's an important conversation. I've heard plenty of people who are a little exhausted about talking about Gen EI right now because there's just sort of a fire hose of Gen EI coming at you.

And I get it. I get like, you know, this is changing a lot of people's lives. They're having to think about this thing that isn't having a lot of real effects on their lives, but they can see the potential. And it's the same for SaaS. Right now, there's already potential to offer multi-tenant

services with SaaS. Like you could take an LLM, you can use, you know, you can start to do queries against it in a general sense. But if you're going to build multi-tenancy in, you have to plan for multi-tenancy in your model. And that could be the RAG model. It could be starting to fine tune your LLMs, whatever that looks like. You want to be able to plan for that up front because it might be drastically different in terms of the impact for your costs. Building your own model and training your own model is expensive. Right. And it's a complicated process that can take a long time and cost a lot in terms of computing costs. So if you can take an existing large language model and put RAG on top of it, and we'll talk a little bit, I think, about how AWS and how we think about that process. But if you can do that, you can start to add value to your customers right now just with what you already have in place. Maybe with just fine tuning the

model, which which can still take some training and time, maybe with with with rag, which can you can pretty much do almost right out of the gate with a little bit of planning, using Amazon bedrock and getting that up and running. So so you know, if people are already familiar with these topics, and are already researching them, you just have to have a plan for the multi tendency this and let's talk through what that kind of looks like as well. Yeah. So how does it work with Bedrock? Is Bedrock designed with this notion that people are going to be using it in this model like upfront? I think to be fair, almost no Amazon services out of the gate think too much about multi-tenancy. It's often, let's get this working, let's get working for the most common use cases, and then let's sort of

build out the multi-tenancy tenant story for our customers as we go. And it was the same with Bedrock. I think as it launched, it was like, okay, now what's the multi-tenancy story? Where is it? And they were like, it's common. We've got this. And then knowledge bases came out. So knowledge bases came out with Bedrock, I believe maybe November of last year, sometime right before reInvent. And we were still trying to get our heads around what this meant as we were going into reInvent. Todd Golding delivered a great session on Gen AI and SaaS and started to talk about how these knowledge bases could be a tool for multi-tenancy, right? And at least as of right now, and again, evolving story, like ask me in a week what the story is of knowledge bases and multi-tenancy, I bet my answer is different. Right now, the answer is it's pretty much a knowledge

base per tenant. But of course, we can still have multi-tenancy in the underlying vector databases under there. So if you're using open search serverless, which is the default vector database for Bedrock, if you're just using Bedrock out of the box, you can do multi-tenancy and open search again with tenant discriminators, and we can have a shared database instance. So we can do shared lot of stuff, but still it's going to be a knowledge base per tenant. But we can do multi-tenancy and it can allow us to unlock some very specific stories. So if there are different tenants are in different domains, if we're an e-commerce store and they're selling different types of products, we can use retrieval augmented generation to augment their query, to augment their queries and say, hey, you know, this guy's interested in baseball. This guy's interested in hardware. This guy's interested in, you know, in boats,

right? And add those to their queries. And as they're going to our catalogs and asking questions, we can give them much more specific answers. And we'll, you know, we're still creating a some version of tenant isolation within this, right? So we still have tenant isolation built in because the knowledge bases are still specific to our individual customers. So as we're adding data about those customers, if there's any sensitivity to it, we're still protecting ourselves from an isolation perspective in Bedrock. Yeah, maybe we could work through an example of what we're talking about. So I want to paint a picture of how this sort of tenant augmented experience would work. So if I have like a business, let's say it's similar to something like Shopify. So I have different, basically different shop owners running

independent shops on a single SaaS platform or Shopify. Then when a customer comes along and they want to put an LM prompt into Bob's store, then Bob's store needs to know about the inventory of Bob's store. But if I go to Sally's store, then Sally's store needs to know about the inventory of Sally's store so I get essentially this augmented experience. So how would that work in the case of what using something like bedrock? Yeah, yeah. And again, yeah, that that is where knowledge bases, I think pop up from for for us on our radar, right? So these knowledge bases will enable us to create

a knowledge base for each one of those stores, right? And to feed the data in for those individual stores, feed their preferences in, and also us to augment the prompts that they're coming through. Now, the case that you're talking about might actually lead us further down the road and talk about fine-tuning the model itself, right? So we might get to a point where we actually want to break out of just the retrieval augmented generation. That takes us back to SageMaker though.

So this is one of the reasons we start with retrieval augmented generation and augmenting the prompts rather than fine tunings. Fine tuning is more work, right? You're gonna actually have to go at this point, go back into SageMaker, build out your fine tuned models. And then you could actually take those fine tuned models

and feed them back into Bedrock if you wanted to. So it is possible to do it through Bedrock, but you're still going to have to step back, take the time to build out those models with better information so that you can get to the specificity that you're talking about, right? So if it's not just, well, I just want to add in the prompt that this particular tenant is interested in this topic, if it really is like, oh, we need specific data for each of these, and we need more inputs that are specific to these specific use cases, then we probably will need to go fine-tune our model on top of retrieval augmented generation. And they can work together well just by taking that fine-tuned model and feeding it back into Bedrock. What about in the case where we start to see more people incorporate agents?

So we could have an agent that's actually carrying out a series of activities on my behalf. Maybe they're actually executing Lambda functions behind the scenes that I've written to perform some sort of action on behalf of the customer. How does this multi-tenant story start to work when we start to incorporate this even more complex sort of Gen AI-powered workflow? Yeah, and of course, Bedrock has its own version of an agent

as of, like, maybe March or April. I mean, again, this is all breaking news. Everything, the cool thing about Gen AI is, again, you turn around tomorrow and I will be telling a different story because they're going to have introduced yet more innovation into this space. But the agents for Bedrock, they call this sort of workflow-ish engine that you're talking about a chain of thoughts, right? So, so basically it allows you to do this type of execution where you can, you can set up a series of steps to augment our prompts, right? So basically, it allows you to do this type of execution where you can set up a series of steps to augment our prompts, right? So instead of just saying like, oh, just go query the database and pull back the prompts, it could do exactly what you're thinking. We can train it to

say, okay, if you get a question about insurance, we should expect a policy number to be in there. Don't have a policy number? We can go back in and ask the questions of the user. Once we get a policy number, we can go reach out to any other systems that might be able to feed us more information about what we know about this query, right? We can reach out to a third-party system. If we're in the auto insurance industry, what do we know about the car? Can we go reach out and pull back information about this car? Can we reach out to our own internal system and see what the current state of this? Can we understand what information we actually need at this stage of the process to move the insurance application forward? And we can enhance our process with all of these, return those back, and then we can also still feed a

prompt into the LLM with all of this additional information as well. How do you think stuff, you know, things like agents, you know, it's probably too early to talk about like real examples of these within SaaS. But as we start to, you know, a year from now, like how do you how do you think this is actually going to change the way that like SaaS gets built or the types of applications that we can build with sas i mean yeah it's always a little difficult to guess too far ahead because i think some of that innovation that's why i selected a year and now five years right right i mean a year from now like five years from now who knows where they'll be making their own decisions about what the chain of thoughts is and telling us how that should work but i mean a year

from now yeah and it'll be two be two agents recording this podcast, having a conversation with each other. Right. Well, what would Bill Tarr say in this situation? All right, Sean Falconer would answer this question, right? It's an interesting problem in that we're only starting to see how SaaS companies are reacting to these things. Now, I think if we think about where SaaS companies spend a lot of their time, it kind of divides into a couple areas. We've mostly been focused on customer functionality, where I think there's going to be a lot of changes. What we haven't really talked about yet are how SaaS companies themselves operate. And I think there's another multi-tenant story here

in terms of Gen AI. And I think this is where this sort of chain of thoughts could start to be really powerful. Because if you think about operating a SaaS solution, there's a number of signals that we're looking for at any given time. If we have a tenant on board onto our solution, what's their onboarding situation look like? You know, did they successfully start to use features we expected them to use? No? Well, why didn't they? Did something happen along the process?

Is there a follow-up step that we should be taking that helps them to, points them to some training material? Yeah, like it looks like you guys haven't gone over and used this module yet. A lot of people in your space love this module. But these are, you know, if humans are making these decisions, it's time consuming and error prone, right?

The person who is supposed to be looking at these things was out for a week. The customer didn't get that email. They just gave up and walked away. And they're never going to use our SaaS solution again because they just didn't see the value. So having Gen AI actually improve the operational profile of what we're doing, analyzing our onboarding solution, saying, OK, the customer's success will have been onboarded. OK, now we're monitoring their activity. And as a SaaS solution, we're always culling all of this data all the time. Actually getting to the intelligence behind the data, we have analytics running all of the time.

And people are gathering human intelligence, business intelligence out of this, which is great. It's important. But Gen EI could be a lot more efficient going over this vast amount of data, looking at the individual tenants, collecting all of those experiences holistically and saying, here's our averages. And here's the customers that are outside of those averages. What's going on? Making intelligent decisions based on the experiences we've had with our tenants. So being able to train our models about how our SaaS solution itself should be operating might even be more influential than the individual features we can offer to our, as important as those are, developing multi-tenant features for our customers, I actually think SaaS is such an intricate model to operate. And successfully operating a SaaS product is such a very informed sort of, you know, CPO has tons of organizational knowledge and historical knowledge.

If we can implement that into Gen AI and start to make those decisions, I think a year from now, we might be seeing a lot more in terms of SaaS operations in terms of this conversation than even customer-facing features. Right. Yeah. I mean, even going back to our conversation at the beginning around cost optimization and scale, if you could just sort of put the agent on it for monitoring those systems, they might be able to essentially build a model to figure out where you can actually optimize your costs or optimize for scale that might take a, you know, human expert, like a long time to kind of figure out. And they're always sort of proactively looking for opportunities to, you know, cut costs or, you know, cut unused CPU cycles or something like that.

Yeah, I mean, it's, it's inevitable, right? Human processes fail. That person was on vacation, the developer left the machine running all weekend. We've all heard these scenarios in our development careers, right? Like if things went out of control, nobody noticed for a week, because we weren't looking at it. Gen AI doesn't sleep. We can turn it loose on the data and it can be watching the data continually and reacting to it, taking real actions. This is kind of where this next jump happens, right? It's easy enough to say it's gathering intelligence,

it's building out insights, it's answering questions. It can also take actions, right? The AI ML can easily say, hey, it looks like we've scaled up our cluster too far, you know, our usage is down to 60%. We can scale down an EC2 and turn that off for a while and see what the operational profiles look like at that level. And if it starts to get worse, it can turn it back up, right? And start to make those intelligent decisions. You know, tenant behavior has changed over the last week since our last release. Well, what is it?

You know, can we see that there's errors? Can we react to those errors and tell the developers? Or even, hey, an amazing, you know, it's able to do code. It's able to, you know, can it tell us where the code problem is and suggest a fix, right? And then we can roll that out. Yeah. It'd be amazing. Yeah, I mean, absolutely.

Right? That'd be a lot of fun. Well, it's going to transform any kind of like monitoring tools that exist today. They're all going to become these like sort of adaptive systems. Exactly. Exactly that. Right. And that's the future of SaaS. And it's happening. But I think there's so much more potential in that space right now. Yeah, it was super early. I mean, I think that we've, you know, unfortunately, in some ways, like Gen AI has become sort of synonymous with like chatbot experience,

but it's going to be so much more than just a chatbot. I mean, there's really, not to disparage the chatbots, but they're amazing what we can do with them. But there's a lot more that we can do with this technology beyond just that type of experience. And I think that's sort of where, you know, we look forward a year from now, I think we'll start to see transformation. Yeah, it would be, you know, if we looked at spacecraft and said it was an amazing thing to

drive Tang as a product, like then then in missed space travel, like that's kind of where we are right now. We're all focused on chatbots, which is the Tang of Gen AI, there's going to be so many more interesting developments, so many more impactful things that are going to affect people's lives, livelihoods, and, and make things more efficient. But those are going to happen a little bit more slowly. And they're going to, a lot of those are going to happen behind the scenes. Like SaaS operations isn't going to be something grandma can go in and, and, and, you know, type into her chatbot and figure out like a clever answer that came back. This is going to be a training on lots of intelligent information over time and injecting

human intelligence into this process in terms of what we've learned to get Gen-AI to a point where it can make those intelligent decisions. It's not as simple as going to the internet and gathering up some information and coming back with a clever answer, right? This takes real time and human intelligence to make that process work. Yeah, absolutely. So I have some quick fire questions to get into before we start to wrap things up. So the first one here is if you could master one skill that you don't have right now, what would it be? I actually heard a really interesting talk by Kelsey Hightower not that long ago, who suggested that the architect's job is less about architecting decision than building consensus. I loved the insight. Honestly, you know, we don't always make the perfect solutions, but getting people to agree and go in the same direction and build something is an amazing skill. I wouldn't say I don't have that skill at all.

I'd like to think I could at least influence some decisions. But I think if there's a skill I'd like to work at more, it's building consensus and getting people moving in a direction, especially as we gather insights about the right things to do. Working for an organization like Amazon, it's an amazing opportunity. There's so many talented people around you. If you have the ability to develop consensus, that would be like my superpower. If I could pick a superpower I think I don't have, that would probably be it. Awesome. What wastes the most time in your day? Just general bureaucracy, I think. And not that it's, I don't want to blame the bureaucracy itself. I think working for a large company, I come from a startup background, you're working

for a startup. The agility of a startup is addictive, like the ability to make quick decisions and changes. The responsibilities of working within a larger organization are much greater, right? I'm working on a blog right now with a customer I worked with, and it took months of work to get the blog processed. And I spent a lot of time, you know, filling out forms, doing communications, sending out marketing releases, like lots of different steps. And I work with a lot of people in the SaaS community, like Ron Eisenberg, and Ron Eisenberg, writes a blog a day. He's out there just cranking out awesome material. And I'm so jealous. I was like, man, this blog has taken, and I'm working on a blog with him as well. And it's taken us

months to go through all of the wheels of the things to get to a very high quality product. And we do, we get to a very high quality product. really raises the bar but sometimes it's painful getting there sometimes the i and i would say that's a lot of my time is trying to to raise the bar on all the stuff we're working with and not just simply experimenting and pushing stuff out so i'd say that's that's probably i wouldn't call it a waste of time but it is the biggest time sink that i have of where a lot of my time disappears to. Yeah, absolutely. If you could invest in one company, that's not the company you work for, who would it be? Oh, Skyflow. Oh man, there's so many. Good answer. I mean, I love startups and the funny, I usually can't invest into startups. I meet so many startups that I was like, oh man, I'm looking into the

future a year from now and you are gonna be so successful. There's so much potential. I'm wearing also another startup. This is Little Horse. It's another startup that I'm a big fan of. I meet so many startups that I was just like, oh man, I wish I was on the ground floor of what you guys are doing. I don't wanna name too many names

because I could get myself in trouble trying to sort of positioning this. But I just, working with startups in trouble trying to sort of positioning this. But I just working with startups in AWS, it's just so amazing. Like I saw Pinecone as a product before most people had heard of Pinecone. I was like, oh man, like this is going to blow up like with vector databases and Gen AI. But you know, you get to watch those things grow and we get to sort of help them and foster them. It's almost as good as investing, right? I'm not making as much doing it, but it's still amazing. Like we're, we're investing into these companies and in a different way. So it's pretty awesome.

Yeah. Yeah. I mean, I think that also you're like, uh, if you're, you're working for a larger company, there's always sort of, uh, you know, a certain romanticism about like, Oh, like, you know, I want to be at the startup thing. And then sometimes when you're in a startup and it's, you're really grinding and things are chaotic, you're like, Oh, well, you know, maybe life would be easier if I went to something where I had a little bit more support and a little bit more process.

I mean, we can all tell stories about our startup days when we spent a month working on something and they said, nah, never mind. Let's just stop doing that now. You're like, why? So, yeah, I absolutely agree. Grass is always greener. What tool or technology can you not live without?

So many things. I mean, I'm sure almost everyone would say Slack early on. Right. So my day to day life revolves around using Slack as a communication tool. I'm so connected to people in various places and communities. I belong to like 20 different, you know, SaaS or Slack channels and workspaces, constantly communicating through that. But there's a whole pile of tools, like all the tools like Riverside here that we're using today, all of these communication tools that like Zoom and Chime that allow us to talk to one another and create this human connection that typing into

Slack doesn't necessarily. Oh, Slack also has a tool there now too as well. But like just all of these communication tools, honestly, that's so much of it is I, I work at home for a lot of, you know, for the most part, 70, 80% of the time I'm in the office sitting in a room by myself, another large percent of the time creating human connections. I think to me, that's, that's the most important part. I could live without writing software. I could live without email. Um, that's communication too, but like, I mean, I could live without writing software. I could live without email. That's communication too. But I mean, I could live without a lot of other stuff.

In terms of software development, that's a whole nother conversation. But this human connectivity to me is so important to what we do. Yeah, absolutely. I mean, I think it's been so transformative both in the last few years as companies have gone more and more remote

but you know even in my own life as well where i i my parents live in another country for me uh my sister lives in another country we're spread out all over the world and across time zones if we didn't have these ways to connect asynchronously or even synchronously from remote world it would feel very isolating and disconnected, but I don't feel that way because we have these types of tools. So I totally understand that. Yeah, yeah, 100%.

My parents live 3000 miles away too. And, you know, they're a video call away at any time. It's kind of amazing, right? Yeah, absolutely. Which person influenced you the most in your career? You know, I had this conversation recently and the question was different. What was the sort of the most in your career? You know, I had this conversation recently. And the question was different. They said, what was the sort of the most important moment in your career. And sometimes

the person who influences you the most isn't the person who's had the most impact on you, but rather the person who informed the single decision that like really changed your trajectory. And I kind of, I narrowed it down to a lunch that I once had with an old coworker who I'd worked with maybe a decade before. She and I knew each other pretty well, but we had kept in contact, but we weren't particularly close. We hadn't had lunch in a while.

And I was working for a large travel insurance company that we were struggling to get to the cloud. The technology was a little dated and she was working for a startup that was about 10 people at the time. And she talked me into leaving the safe, stable environment that I was in and joining this born in the cloud startup and just jumping in and starting to build. That was probably the most influential decision point of my career. Could have stayed at that company another decade and continued to build important software that was important to that company. Would have never landed in the place I am today if it wasn't for that lunch and that decision.

So I would say she had the greatest impact on my trajectory with maybe the least amount of effort. Like she was able to, it took her an hour of talking me off of the cliff I was on and to dive into the ocean and see what happened. And that changed my entire career, I would say. Yeah, I feel like sometimes, you know, when you can have these moments with somebody where it could be a decision that you're kind of already maybe know is the right decision for you, but someone says it in a way to you that makes it very like crystal clear that that is decision that you have to make.

And that's kind of all you need is like someone to kind of just like give you that little nudge, push you over the edge a little bit. Yeah. Yeah, I would. I would. And I would say that's it. I've been thinking a lot about that. I actually haven't reached out and told her that I've said this on a couple of podcasts. Maybe I should tell her I'm talking about her on these podcasts, too. Yeah. So last one, five years from now, will there be more people writing code or less? I, you know, interesting, I would say less people writing code, more people handling code. I think Gen AI is going to have a huge role in writing code. And I think the human element of being able to decipher what that is and add the human intelligence to it is, I think, I mean, five years is a big horizon. I think that that's still going to be an important role is taking all of the code.

Just like, hey, a developer today is taking a lot of stuff that was on Stack Overflow and copying it into their own code. But they have to understand what it does and they have to understand what the impact is on the greater system. I think there's going to be more of that from Gen AI. Like, hey, Gen AI, write the code for me. It's going to go look at Stack Overflow and it's going to figure out what the code is that you need. It's going to inject it, but there's going to be a lot more of like, okay, well, I understand the flow of what this is doing. I can, I can produce more, but I have to understand how these pieces actually work. Maybe, you know, more, more test writing to sort of, you know, start with like a test driven development approach and say, okay, you know, Jenny, I make the code that, you know, performs

the tests. Maybe that's a, I've actually heard a couple really, you know, people who, like Jason Wadsworth, who's pretty influential on my thinking recently talk about this, like, you know, we're asking JNI to write the test for the code we wrote, but isn't that backwards? Let's write the tests and then ask JNI to write the code that actually implements the business decisions we're implying with our tests. Maybe that's a better direction. And he and Alan Helton had a really good conversation around that. So I think that might be more of the human role than just writing code itself. Yeah, I feel like in some ways, we get a little bit too sort of, like worked up on where the code is coming from. I always say that, you know, the reason companies pay engineers is to solve problems. And that's not going away anytime soon regardless of

what you know maybe is doing the actual code generation and we've had different levels of code generation tools for a very long time it's just this is like a better you know version of what was maybe there before or going and searching stack overflow and copying pasting and then you know you still need to understand what you're doing. And you still need to understand how to ask the right questions in order to get the thing that you want. Yeah. And eventually, maybe Jen, I does that too. But I think that's a much harder jump to make, right? Like those type of intuitive, like, I understand how the business wants to work. I understand what the intent is of what we're building. And I'm going to create something that does that intent, not something that follows the letter of the law and says, this is how this shall work.

That's not how developers work. We think about what the intent is. We try to think ahead of what those outcomes will be and plan for things that aren't even really in the business requirements to keep things flexible. So it's an important role. I don't think that's going away anytime soon, like you said. Yeah, absolutely. And then as we start to wrap up, Bill, is there anything else you'd like to share? Yeah, I mean, I want to talk about something that's been on my mind a lot.

And I don't want this to turn into another hour conversation, even though it could. But I've been thinking a lot about security and SaaS lately. There's been some really high-profile news stories about ransomware attacks on SaaS. They've really caught my attention. And if you're in the business of SaaS, they should have caught your attention as well. I mean, we're already spending a lot of time thinking about data, data security, data privacy, data sovereignty. All of that's incredibly important. But all of it is predicated on a base

of security. I know it's unnecessarily with the way it rolls off the tongue, but we say at Amazon security is job zero, right? Before you do everything else, think about security. And SaaS companies especially have to be doing this. we're seeing entire industries grind to a halt based on the SaaS software that they were using get taken over by ransomware. You know, where's the blast radius limitations built into this? Where's the resiliency stories? You know, how could we have prevented this in the first place? How did they, and, you know, some things aren't preventable. There are black hats out there who are smarter than us sometimes. And sometimes things are going to go wrong. What's the story when that happens? If they can take over a single account,

does that cause our whole application to stop? What are the other stories we have in place that prevent that? Cell-based architecture is one example, you know, deploying tenants into different accounts so that we have some resiliency if if we for whatever reason, lose an account. But just, you know, baseline security, you know, what are we doing for our root credentials, credentials? How are we rotating out any sort of security credentials that we have? What are we doing when we off board employees, all these very basic questions are probably exactly the root cause for some of these large problems that we're seeing in terms of ransomware, in terms of these environments being blown up and not being able to recover.

So it's been really on my mind for the last week. I was kind of curious. I wanted to hear what you thought about this and if you've been thinking about this too. Yeah, I mean, I think it's in a lot of ways, if you're in technology right now, it's something that's like difficult to avoid because over the last month or so, there's been a tremendous amount of high profile data breaches

and ransomware attacks that are happening. And 2024 is really turning into the year of the data breach. I mean, there's more PII records compromised this year. And we're like, you know, just starting the seventh month in here than there has been in any other year previously. And 2023 was the biggest year prior to that. So like, it's not a problem that's going away. And when we think about SaaS, a lot of these tools become like, our business, like whatever our business operations are, or even like what our product is depends a lot on these different products. And if there is security flaws within those products,

or there someone's able to exploit them in some way, it could cripple our business as well as all businesses that use that. I mean, imagine if for example, something like Salesforce got compromised and like so many businesses rely on Salesforce or something that is like part of your core infrastructure could have tremendous impact to your ability to generate revenue, sell customers. And it might be something that's completely out of your control because of, you know, like a flaw in the system of the SaaS product that you purchased. Yeah. And it's, it's a complicated problem. You know, that I think competition's important in these areas too. It's, it's, you know, when we see SaaS companies get too big,

we often see young startups come up and compete with them, keep the pressure on them to, to keep building and being, being aggressive about making their products really ideal. So I'm kind of hopeful that, you know, we'll continue to see startups keep the groundswell of better practices continuing to develop and that that will keep, you know, larger SaaS providers continuing to innovate and doing a better job at their business. In the meantime, I mean, if you are using large SaaS products, you should also be asking them the questions, right? Like if you're a customer of these large SaaS providers, ask the right questions.

Like what are you doing to secure our data? What are you doing to secure your own environments that we're going to rely on and keep them resilient? And they should have those stories. If they're working with AWS, they should have well-architected reviews, foundational tactical reviews that go through all of those things.

Again, there's nothing's perfect. There's still gonna be that scenario where the black hat somehow gets in, but we need to have stories behind stories. If that happens, what do we do? Like what's our fallback strategy? What's the fallback strategy? What's the fallback strategy if the fallback strategy falls apart? Layers of defense, right? Yeah. I mean, a lot of security

comes down to layers of defense. It's like the onion strategy, essentially. And you can think through, if someone's determined enough, they can probably get into any system. But when they penetrate that first layer, what do they have access to? Does that compromise a customer's information, all customers' information, all these types of things? And you need to be thinking through this. And are people that are using our software internally over permission in some fashion? Does the customer support person really need access to every customer record? Or do they only need access to the six or so that are in their queue? Like you have to think through these types of things to help reduce essentially the blast radius of a potential attack.

Yeah. And again, I mean, your work for Skyflow, this data, the idea of a data vault and keeping some of that data segregated is an important part of this, right? Even if you are hacked, what data do they have access to? Have we protected that data as some other line of defense, right? Is our PII exposed the same way as our catalog of products is? Because that's really not where I want to be sitting if I'm running a SaaS product. So like thinking really deliberately and intentionally about where our data lives, what the compliance story is for

each of these layers, and then the security and the resiliency stories that we've built on top of those. It kind of blows my mind that we're talking about all of these data leaks, talking about ransomware. Again, in 2024, when we know a lot of best practices that chances are probably would have stopped a lot of these situations, and of course not all, but a lot of best practices that chances are probably would have stopped a lot of these situations. And of course, not all, but a lot of these situations just come down to unfortunate mistakes, human mistakes. Maybe we need to point Gen AI at our security processes to let it figure those out as well.

I don't know. Well, I mean, and then you throw Gen AI in the mix and it might help solve some problems, but it also introduces a whole host of new problems and challenges as well. And that whole market is moving so quickly. Like, are people really spending time thinking about those challenges from a privacy security perspective versus like pushing out applications that they hopefully can do like a land grab on the market or something like that? So it's a challenging issue. I think there are baseline things that everybody can do like a land grab on the market or something like that. So it's a challenging issue. I think there are baseline things that everybody can do. And a lot of it comes down to being intentional about it.

And if you're building SaaS or you're buying SaaS products, like really thinking about these things and making it core to what you do. Yeah. Yeah. Ask the questions, right? Ask the. Yeah. Yeah. Ask the questions, right? Ask the questions of your teams, ask the questions of other SaaS teams, make sure that they're thinking about these things, especially if it's sensitive data, if it's your customer's data, be, you know, even more intentional about it, but be intentional about your own environments, be intentional about what you're deploying, how you're deploying it. If you're doing Gen AI,

like whose data are you using? Like, you know, are you actually just using your customers data? Have you asked them? Have you asked the question whether you can use that your customers data? If you have, are you protecting it? Right? Are you protecting it the same way you think about other PII that's living within your application? I answer so many great questions around security. This is I mean, it's really been on my mind for the last couple of weeks. So thanks for talking a bit through me a little bit. And anyone listening, yeah.

I mean, if you've got thoughts on this, on security, on protecting your data, on protecting your environments, I think it's an important conversation. I'd love to hear from you if you wanted to talk more about this topic, for sure. Awesome. Well, Bill, I think that's a great place to leave it.

I think this was, we touched on a lot of stuff, covered a lot of ground, but thanks so much for being here and sure. Awesome. Well, Bill, I think that's a great place to leave it. I think this was, we touched on a lot of stuff, covered a lot of ground, but thanks so much for being here and cheers. Awesome. Hey, Sean, thanks for having me, man. It's a great conversation. Always good talking to you. Thanks.