Screaming in the Cloud - Episode 60: Managing Secrets for Kubernetes with Kamus with Omer Levi Hevroni

Episode Date: May 15, 2019

About Omer Levi HevroniOmer has been coding since 4th grade when his dad taught him BASIC, and he got hooked. From that point, he learned to code in many programming languages (today his favorite is C#). Today he’s working at Soluto by Asurion, and coding is a huge part of his day job.His passion for AppSec started by accident when he was offered the role of security champion. The AppSec journey was (and still is) fascinated, and taught him a lot. OWASP helped him a lot during this journey; This is why he decided to become a paying member and also leading OWASP Glue.Omer’s current job is DevSecOps – helping the entire team to produce more secure software. Besides his job, he’s also giving a lot of talks all over the world, and heavy OSS contributor – mainly to Kamus, a secret encryption solution for Kubernetes platform.When he’s not working – he’s enjoying the company of his two beloved kids and his wife.Linkshttps://twitter.com/omerlhhttps://www.asurion.com/about/smb/who-we-are/https://github.com/Soluto/kamushttps://omerlh.info

Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode of Screaming in the Cloud is sponsored by O'Reilly's Velocity 2019 conference. To get ahead today, your organization needs to be cloud native. The 2019 Velocity program in San Jose from June 10th to 13th is going to cover a lot of topics we've already covered on previous episodes of this show, ranging from Kubernetes and site reliability engineering over to observability and performance. The idea here is to help you stay

on top of the rapidly changing landscape of this zany world called cloud. It's a great place to learn new skills, approaches, and of course, technologies. But what's also great about almost any conference is going to be the hallway track. Catch up with people who are solving interesting problems, trade stories, learn from them, and ideally learn a little bit more than you knew going into it. There are going to be some great guests, including at least a few people

who've been previously on this podcast, including Liz Fong-Jones and several more. Listeners to this podcast can get 20% off of most passes with the code CLOUD20. That's C-L-O-U-D-2-0 during registration. To sign up, go to velocityconf.com slash cloud. That's velocityconf.com slash cloud. Thank you to Velocity for sponsoring this podcast. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Omar Levy-Havroni, who's a DevSecOps engineer at Saluto by Asherian.

Welcome to the show, Omar. Hello, Corey, and thank you for having me. It's a great honor to be on this podcast. I'm a huge fan. Thanks. It's super kind of you to say that. So in addition to the day job, or I guess in a component with the day job, you wound up releasing an open source tool called Camus, which I'm almost certainly mispronouncing.

There's some people going to say it's called Camus. Some will say Camus. It's K-A-M-U-S, which out of the gate is already a better name than almost anything Amazon has ever given to anything. What does it do? Let's start with the name because I guess I don't know how much of the people hearing us right now are speaking fluent Hebrew. Camus in Hebrew means a secret. And we call it Camus because it's a tool that helps managing secrets on Kubernetes.

We all love secrets and we all want to keep them confident. And Camus came to solve exactly this problem. So this is like the very high level overview. And it's different from all the other tools because Camus lets you encrypt a secret to a specific application. So only this application can decrypt it. And this is a very unique approach to secret management. That makes a lot of sense.

Take me back to the beginning here. There are a number of different ways to address secrets management of varying degrees of validity. Some make an awful lot of sense. Some are less sensible. It ranges from using GPG, which is heavy and difficult to wind up encrypting secrets that then applications wind up retrieving. There's just embedding it in your source code and hoping that no one ever finds it.

There's putting it in insecure S3 buckets and just acting shocked when that winds up getting discovered. And 50,000 other horrible things you can do. But then there's good stuff. There's things like AWS Systems Manager Parameter Store. There's Systems Manager in the AWS side. I have to assume Azure and GCP and the rest have something vaguely similar. What does this do differently? Yes, you pretty much covered all the different options.

And I think when thinking about what secret solution to choose, you need to think about the culture of the company you are working at. For example, the way we work at Soluto is a way we like to call super devs, which means developers do everything. They do the fun work of writing the code, but they also do the hard work of deploying it and monitoring it.

And when you have developers who do everything, you need to choose a solution that, on one hand, they can use easily, and on the other hand, they can use it easily but still be secure. So, for example, if we go with GPG, like you mentioned, you need to find a robust solution that will let all the developers have access to the encryption keys and keep it secure and all that, and it gets complex pretty fast.

And this is why I'm saying that the culture, the way you work, the way you work is a very important factor when thinking about which solution to choose. So pretty much you can think about different approaches. And the approaches I like the most are the GitOps, which is, I think, one of the hype buzzwords existing today. Maybe

GitOps and Bitcoin and blockchain. GitOps is the idea of managing everything via code. So you have code describing your cloud architecture and everything else. And when you do things with GitOps, you get a lot of

benefit of Git, like audit and review and all that good things. And this is why I like also to manage secrets via Git. So solutions like AWS SSM or Azure Keyword, which is pretty much the equivalent of Azure to SSM, are a really good choice, but they don't have any GitOps support. So if you go on this path, you need to solve also authentication, authorization, and a lot of other things that you don't want to mess with. So I think this is pretty much the reason not to go this path.

There are a number of different things that I will look at whenever I see something that purports to handle security and discount them almost out of hand. The easy, low-hanging fruit one among those is rolling your own crypto. One of the nice things I like about Camus is that it uses whatever KMS version is being used by the underlining provider. To wit, that's Azure Key Vault KMS, there's Google Cloud KMS, and AWS KMS for the big three. You're not generally rolling your own crypto solution here with the singular there's Google Cloud KMS and AWS KMS for the big three. You're not generally rolling your own crypto solution here with the singular exception of running your own keys

for just local development, which you very clearly call out in the documentation that that is not what's something to be done other than when you're in a hurry for testing purposes. So first, can I just congratulate you on not doing the most ridiculous thing imaginable? Yes, at least we tried. Actually, this is a pretty interesting aspect. The whole idea of Camus was to say, hey, I have no idea how to do encryption good.

I do security and I do AppSec for two and a half years, but still I'm saying out loud, I don't have any idea about key photography and I don't want to do these things alone. So this is why we prefer to defer all the hard cryptography work to the cloud. And the interesting part is that in this area, GCP was the best experience. In GCP, you have no control on the cryptography. You just tell KMS, hey, and keep this and keep this, and you have no control on the cryptography. You just tell KMS, hey, encrypt this and encrypt this, and you have zero control about what happened behind the scene. In my opinion, it's the best experience because there are less chances you will do mistakes.

For example, in AWS, you need to write the envelope encryption yourself. So the master key is encrypted in AWS, but you are doing the data encryption in your code. And I guess we might have a mistake there. We tried really hard to do it good, but, you know, mistakes happen. And this is a very interesting aspect of looking on all the three clouds and looking which one gives the easiest developer experience. One thing that I found fascinating about Camus

compared to an awful lot of other projects that have come out of the open source world is they start off being built for AWS, and that's it. Then eventually they add a second cloud provider, begrudgingly, and then it sort of always feels like a secondary thing. This came out of Azure first, and then I think, correct me if I'm wrong, then it was GCP second and AWS third? Yes, we started with Azure because at Soluto we work mainly with Azure. It's the cloud we use the most.

We started with the cloud we are using because we first wanted to make sure that commerce is usable and it's working as we expected. And then when we understood we were going to release it to Open Source, it was clear that we can't support only Azure if we want people to use it. Because unfortunately, not everyone loves Azure as we love. So yes, then we added GCP because someone asked for it,

but then never used it, which was pretty disappointing. That sort of at times feels a bit like the GCP story, but that feels a little mean past a certain point. Yeah, it was really disappointing because I did it really fast and hoped he would use it, and then he vanished. So if you're hearing us, I'm still waiting for your review of the GCP implementation. And then two months

ago, we added AWS, which, as I said, was the hardest, I think, to implement. And now we support all the three major cloud providers, and we look forward to add support for other chemists, so HashiCorp, Vault, Encryption as a Service, or any

other HSM, Encryption thingy you have that you want to add support to. It should be really easy. It's usually around a couple of hours to add support for it, and you're done. Something else that I found fascinating about this is that unlike virtually everything modern that is written in Go

or everything designed that's written in Rust but never actually built because no one wants to stop talking about Rust long enough to write anything in it, you wound up writing Camus in.NET. Yes. Why was that? Yes, well, the other reason, historically, I started developing in.NET. Yes. Why was that? Yes, well, there are two reasons.

Historically, I started developing in.NET, and I sticked with it, and it's still the language that's the most easy to me to write in. But.NET is actually a pretty awesome language, and a lot of people hate it because it belongs to Microsoft. But since.NET Core and the open source of.NET, it's become really, really easy to use and a really handy tool. It's still a bit heavy compared to other tools.

A simple project has a lot of files and all that. So compared to one Golang file, it's a lot more heavy. But you have a lot of things that are harder to do in other languages. And this is why I still like it. It was a pretty hard choice because choosing a language that not a lot of people will use means that you will have less contribution.

I already have some people saying I would maybe consider contributing a PR, but I'm not sure how to write things in.NET. So there are some downsides to choosing this language. But this was what was easy for us at that time. And maybe in the future we will consider a write it in golang or something else so we can have more people be able to

contribute to commerce realistically i find that languages make sense picking a particular language makes sense when there's a compelling reason to do so and you just named one of them where having an ecosystem of people in the space who are able to contribute to a project, that's a viable strategy. But I have zero tolerance these days, or frankly, most of my career for language bigotry where, oh, this is an amazing tool

that does the thing I want it to do, but it's written in Perl, so nevermind, it must be crappy. Or, oh, it's written in the wrong version of Python. Or, oh, Ruby is for children. We're not going to go and use that at all. And none of those statements are ever true, and none of those statements are ever helpful.

It feels almost like it's a form of gatekeeping, where the entire purpose is to make people feel bad about what language they've chosen to write things in. In fact, we want to go by sheer number of expensive things that will break written in it. Java is pretty much the clear winner. Between that and still the things that run our entire world, written in COBOL,

I don't think that there's a compelling argument to be made that only the cool modern languages are the ones that matter. I'm a big fan of seeing things that aren't written in the usual suspects, and this is absolutely one of them. I've taken a bit of a look through the code base. It seems clean. It seems well-written. I understand what it's doing, despite not being converse in a.NET myself, which tells me that either the language is not that hard to learn, or you've gone significantly out of your way to make this approachable to people who aren't just you? I think this is a fair point you touched. For me, it's still very hard to read Golang code. And I'm working with Kubernetes for more than a year, so I had to read a lot of Golang code.

I think it's the language, but I cannot say that for sure because I don't have enough experience with Golang. But again, it's just only my experience. Golang is still very hard for me to understand and write and read. And this is places where I think language do matter. If you have a language that you know well and you know all the tiny points of what to do and what not to do, it's very hard not to use that language and use something else because you will probably produce a copier product. So there are places when languages matter and it's more what you control and not what is popular.

The problem that I've always had with popularity-driven development is that it leads almost to groupthink. We're starting to see this not just with languages but with other things as well where if you're not using AWS, you're clearly doing it wrong. Well, I don't know that that's necessarily true. Or if you're building any monolith that, no, microservices are the way and the light of the future. Or, oh, if you're not going with serverless, you're completely missing the entire point of the modern revolution. And I don't like any of those things.

They smack far too heavily of orthodoxy, and it's getting away from solving business problems and into the realm of religion. And that doesn't seem to be serving anyone particularly well in a technical or a business context. Well, I'm a religious Orthodox Jewish, so I'm pretty fine with being Orthodox and going with religion.

There's validity to that. The question though is, is at some point it blurs the line between things that we have to take on faith. And I think that religion is inherently going to be one of those versus things that lead to definable business outcomes

and doing something just because the thought leader on the stage said it was the way to do things, that's always been a problem for me. Because our choices are built by constraints in a technical context. And the fact that we don't wind up making decisions based upon other people's constraints, or even our own constraints, leads to disaster. If someone gets on stage, like when they did back in, I want to say 2014 ish and talks about how Docker is amazing and terrific and solves everyone's problems. Well, it didn't. What about stateful workloads? What about networking? What about orchestration of these things? What about

monitoring? How do you emit statistics? How do you troubleshoot when the thing that's running was turned off 20 minutes before you knew that there was an error? And there was a giant list of problems that have largely been solved today. But at that point, the whatever your problem is, Docker containers are going to solve it was sort of what everyone was saying. throw everything away that I've built previously and build everything again in this new context, this new environment. I could see doing that, but I didn't see a compelling business case for it. And now here we are six years later almost, and we're talking extensively about Kubernetes in some cases being the answer to everything, the problem that I may possibly have. I don't see it. There are absolutely use cases for which Kubernetes is a valuable path forward and a great use case, but certainly not all of them. And the level of complexity

still feels slightly higher than most companies are probably going to want to dive into unless they are themselves tech companies at heart. Is that something that you're seeing? Do you agree, disagree? Don't feel you have to agree with me just because I'm the one with my hand on the mute switch.

No, I pretty much agree. I think it's a level of maturity as a software developer or a security developer or as any other, I don't know how to name it, but the hard part is not always answering for what is good

but for what is not good. It hit me first when I heard a really good interview question when you need to answer when microservices are not a good choice. And there are pretty good answers to these questions, but I heard this question and it got me

thinking. It wasn't my job interview, but I heard it in another. I read it somewhere, I don't remember when. And I started thinking. It was a pretty interesting thing to think about. What situations are not good cases to use microservices but monolith? And as you say, it was very interesting because it was going against the dogma and starting to say, hey, microservices are not always that good and it has a lot of overhead.

And I think it's a fair point. Actually, I like the example of Kind. Kind is an open source that lets you run a Kubernetes cluster using Docker, and it's really good in the CI. And what I liked about Kind documentation website is that they have a whole list of alternatives, projects that are similar to Kind, but do other things. And you can look on all the alternatives and choose between the alternatives and Kind, the one that best works for your use case. And I wanted to do it also on Camus' documentation website,

and I forgot I need to do it. But I think it's really important to say what you are not good for and what use case you don't want to use, Camus or Kind or Kubernetes or whatever. I absolutely love the idea of an interview question where there's this thing that you're good at, that you're passionate about. Great. Tell me about its shortcomings. Tell me about when it's not a good idea,

because that tells you a lot about a candidate, just as far as can they see both sides of an issue? Are they so blinded by zealotry for this thing that they're in love with that they can't see another perspective? And if you wind up making aotry for this thing that they're in love with that they can't see another perspective? And if you wind up making a technical decision down the road while they're working with you, are you suddenly going to wind up having a screaming meltdown because that is not acceptable? Finding ways to tolerate and deal with disagreement is incredibly important for almost every environment I've ever been in. So I just want to say I love the idea of asking that kind of question. Yeah, we can do it on Kamos.

Kamos, as I said in the start, it's not good for everyone. If you already have Vort and Vort works for you and you have a culture when you have a specific team who will manage the secrets, it might be easier to continue using SSM or something that is not GitOps and it might work better for your solution. Another caveat of Camus is that Camus do one-way encryption. Once you encrypt the secret, there is no admin account that lets you decrypt it back. And it's not work for everyone. For us, it was good enough, but it has downside.

Sometimes you need this value dec value dedicated and Camus by design don't let you do that. It's important to understand the idea behind the design of Camus and understand if this idea fit your workflow or not because it's not always fit. Pay attention, listener.

The reason to look into Camus is because you heard on this podcast, yeah, that's a good idea. Deploying it blindly because the super confident sounding people on the podcast said you should, that's a bad reason. It all comes down to making sure whatever you're using works for your use case. Understand what it is. Understand how it breaks. Understand how when it breaks, because everything breaks, that's what computers do, will impact your production environment.

It all comes down to understanding constraints, understanding trade-offs. Yeah, this is part of the reason we decided to release with Camus a full-blown threat model of all the different threats and controls we think about. So you as the user can go and read the threat model and say, hey, this is an unacceptable threat by us, and I don't go into deploy comms and have this threat available. I'm not feeling good with it. And you can look on the controls and understand the different things

we decided to do in order to mitigate them. And you might not like some of the controls, or you might even think about threats or controls that we didn't think about. So it's all part of the idea of giving our users all the information, it's all there, just to hear them. And with all that being said, go use Camus, it's really amazing. So don't listen to other things I say before. Exactly. Adoption is where it all comes from.

Sooner or later, we'll put a foundation around it and get people to start supporting it with giant corporate piles of contributions. Changing gears a little bit, you introduced yourself at the beginning as a DevSecOps engineer. And I tend to come from an old school world where I started my career as a grumpy Unix systems administrator. We also call that a Unix systems administrator because they're all grumpy.

There's no second kind. And over time, I watched the world evolve. We started calling the same thing I was already doing DevOps. In many shops, they started calling it SRE or site reliability engineering. And it feels to me, and this is probably a slightly naive, that the role hasn't particularly changed in 15 years. Sure, the tools we use are different. But if you say, well, back when you were a sysadmin, you didn't have to write code. I didn't. I was doing an awful lot of stuff in Perl. And if you

start working on Unix systems administration, you generally in that era won't get very far without at least a working knowledge of C. And it turns into, it seems everything old is new again. So then we called it DevOps, which people argue vehemently is a title, is not a title, but at least we're debating rather than doing work. We don't want to do work. We want to argue with people. And now we're seeing DevSecOps as adding additional words to that construct, which on the one hand, I like the idea that security has to be baked in from the beginning is a fundamental tenant of safe systems design. And I'm curious, though, how did you wind up coming by that job title? When did you start using it?

It's complicated, like all the good things. The funny part is that today I'm part of the DevOps team at Saluto. We talk a lot with our manager about changing it to SRE, and I ask him if I change my title from DevOps to SRE, whether it will increase my salary. And he refused to answer this question. So today I'm not SRE, but if it will increase my salary, I will. Oh, it absolutely will to an extent.

I've been to a number of DevOps days where they've done job title polls. And going from sysadmin to DevOps or SRE was something on the order of a salary boost of at least a third taken in the aggregate. So it's weird, but the things we call ourselves change how people view us and more importantly, how people pay us. So I think I will go with DevSecOps SRE. I think it will be good enough. Oh, I like that.

FinOps in there too somewhere. I think someone's trying to make that happen somewhere. Yeah. And the last funny part is that I don't have any official degree. So I am a bit ashamed to call myself engineer. to make that happen somewhere. Yeah. And the last funny part is that I don't have any official degree, so I am a bit ashamed to call myself an engineer. But, you know, it's a different discussion

about the importance of doing a bachelor degree in computer science and all that. I don't know about that necessarily. I have an eighth grade education myself. I've always toyed with the idea of going and getting

a computer science degree at some point in my career. And there are aspects it would be helpful for, but from my perspective, it seems for some folks, it's more of a credential. In my 20s, it was rough not having the degree. I had to talk my way around it a lot. Now that I'm in my 30s, I can bring this up in public that, hee hee, I don't even technically have a high school diploma. And it's a punchline. It's not a career limiting move. By and large, careers do get easier with the right credentials. But I don't think that it's not a career-limiting move. By and large, careers do get easier with the right credentials.

But I don't think that it's ever an absolute, at least not as we tend to think of it as. It just means we may have to be a little bit more creative. Yeah, my parents are disappointed and hope that at some point I will get the sense of it and go and do an official degree. But it's just too hard.

I never managed to get over all the hard math courses so I don't think it would happen at any point. I'm in the same boat. My parents still think I'm an accountant which, yeah, we're going to go with that. It's a lot easier to explain. Anyway, the job title is complex.

I started at Seroto as a developer and then someone offered me to be part of the security champion. You heard about the idea of security champion? I have not. Tell me more. Okay. So it's relatively new. It's not that new, but it's coming from the idea that the security team, especially in

companies like Asurion, are usually very, very isolated, located somewhere in the United States and they cannot control all the developers all around the globe. So usually the solution to that is either hire a lot more security people who will be located in each site, which is very hard, or taking developers and allowing them, allowing and converting them into security, taking them to the downside, to the dark side, taking them to the dark side. So this is what I did. I did a SANS course of one week. So I have a certificate that I'm an officially pen tester and I can use this title if I want. And since then, I started to do all things related to security,

AppSec and all the fun stuff of DevSecOps, which going from security testing to running things in the pipeline. The things I like the most, my job in our DevOps team, which I think fundamentally different from sysadmin, is what I started with. We have a lot of developers and our job as the DevOps team, and especially my job as the security person, is to help them and guide them. We don't have any control on what they do. We cannot tell them what is wrong and what is right. We can't just help them and provide them with best practices. And if they come and say, hey, I want to use this cool new technology,

we are the bad guys who come and say, hey, what about monitoring? How we can monitor that? How we can deploy that? Is it secure? and all that question? And these things, I think, are fundamentally different from sysadmins because sysadmins usually were not responsible to do this work. And this is why I like to call myself

DevSecOps engineer. I thought I'm not that happy with this title, but I think it's better than security champion, which was the title I used before. I think that that may wind up being for the best, all things considered. We're about out of time, but if people are interested in what you have to say,

where can they find you to learn more? So the best place is my Twitter account. I try not to tweet too much in Hebrew, but I do it also. It's by language, and I try not to post. So you can follow me there. That's O-M-E-R-L-H.

We'll put the link to that in the show notes. Yeah, and I also have a personal blog, which I try to post new things once in a while. You can also ping me there or ping me on Kubernetes or CNCF Slack. I'm on each one of them. And of course, GitHub and all the other things, you know, where everyone

are. And of course, LinkedIn. Yeah, I think some people are also there. Sounds good. Thank you so much for taking the time to speak with me today. Thank you for having me. It was a really big honor. And as I said in the start, I'm a huge fan of you. It was pretty nervous to be on this podcast after the last episode with Jess. Yeah, you're talking, there's always a bigger fish. I've got to say, I was nervous as heck talking with Jess. She's incredible.

And everyone I talked to is incredible. But she's been one of those people I've been following for a long time. And it turns out everyone has someone that they look up to. Oh, I could never talk to is incredible, but she's been one of those people I've been following for a long time. And it turns out everyone has someone that they look up to, oh, I could never talk to them. I'd never be going to be at their tier. There's always a bigger fish. And something I'm learning is that reaching out and talking to people

and asking for help, everyone worth admiring is a human being who's willing to have a conversation. Don't ever hold yourself back from that. Omer Levy-Havroni, DevSecOps Engineer at Saluto by Asurion. I'm Corey Quinn. This is Screaming in the Cloud. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at Screaminginthecloud.com or wherever fine snark is sold. This has been a HumblePod production.

Stay humble.