PurePerformance - 065 Running a successful internal bug bounty program with Pascal Schulz

Episode Date: July 2, 2018

Security is on everyone’s mind. One way to strengthen security of your software and increase the awareness of your engineers is running a Security Hackathon – or a “Bug Bounty Program”. We invited Pascal Schulz ( https://www.linkedin.com/in/pascalschulz/ ), Security Engineer at Dynatrace, to the show to give us more background on HACK.DT – a security hackathon he and his team ran earlier this year within the Dynatrace Engineering Labs. For additional details check out his blog Running a successful internal bug bounty program and ping him on twitter (@PascalSec) in case you have further questions.https://www.dynatrace.com/news/blog/running-a-successful-internal-bug-bounty-program/

It's time for Pure Performance. Get your stopwatches ready. It's time of Pure Performance. My name is Bizarro Brian Wilson and with me is Andy Grabner. Hello Andy. Hey, what happened to your voice? I don't know. You finally grew up, you got into what's this called when you finally... I started taking testosterone pills exactly not to laugh at anyone who's doing it but you know yeah well uh andy

you're you're in uh you're in nola aren't you nola yeah new orleans louisiana yeah i just arrived here last night um here for a conference uh nutanix next conference they invited me to talk about guess what devops really i can't imagine they invited me to talk about guess what, DevOps. Really? I can't imagine them asking you to talk about it. I was in New Orleans for Mardi Gras this year. I keep on saying that was the first and last time I'll ever do that.

Very interesting. Very intense. Very crowded. Glad I did it. But I don't think I'll... I think there's still maybe some people lingering around here i think they have the jazz festival going on right now that's another big time yeah yeah and this morning i i walked downstairs and had a call from the hotel lobby very early morning and then the guy came in and he sat down next to me on the couch and then fell

asleep and then uh the security guy tried to wake him up and get him out, but then he just refused. And eventually they escorted him out a little while later. So there's a lot of interesting people here. Yes, it is a very interesting place. Definitely worth checking out sometime. Hey, talking about interesting people. Yes. Yeah, we have an interesting guest today.

Who? Who's our interesting guest? Who would this be? Well, I want to introduce Pascal Schulz. Pascal, are you there with us? Yep, I'm here. Awesome. Hey, Pascal, thanks for making it on the show.

And typically the way we do this, right, you may want to quickly introduce yourself, and then I want to give a little background on why we invited you and what we want to talk about. But maybe we want to get started. Who you are, a little background, and then we jump right into the topic. Sure. So, well, thanks for having me. I appreciate the invitation.

So my name is Pascal, and I'm working in the Lintz office for Dynatrace, and I am a penetration tester. And I don't know if everybody in the show knows what that is, but that basically means I improve Dynatrace's product security by running a lot of tests and see if I can find any vulnerabilities. Cool. Basically, that's it. Cool. Well, I have a question before we go into that topic.

And, I mean, we know each other, but just a little bit because I have not lived in Linz for a while. But I will relocate soon. So we'll see each other more. But I am really intrigued now by your accent because you don't have a traditional Austrian English accent. Where did you pick up English and that accent that you have? I don't know. Like, to be honest, I think I have like dozens of different accents over the past couple of years. And people gave me South American.

And then I heard like, oh, no, not South American. I got South African and I got Australian. And then I got North Island. And I learned a couple of others. I don't know. I like to travel. And before I started to work for Dynatrace, I traveled Southeast Asia for like seven months. So that's, I guess, where I picked up most of the X that I'm having right now. And I'm not going to say that Asians speak like that.

But obviously, you travel with a lot of backpackers. Yeah. So I don't know. That's probably where it's coming from. So Pascal, penetration testing. yeah so i don't know that's that's probably where it's coming from so but uh so pascal uh penetration testing so that means obviously we at dynatrace we take security serious because we have dynatrace out there monitoring systems that obviously uh you know that host applications that that that are secure and by general and then our customers want to make sure obviously that we

don't introduce any problem now you are the reason why we brought you on the show is not just because of your job or what you're doing on a day-to-day basis, but the program that you introduced. And I'm actually just looking at the blog post here. So if people are interested in learning more about what you did after the show, if you go to blog.dynatrix.com and searching for running a successful internal bug bounty program. So that's what intrigued me because the last time, I think it was one of the last times when I was in Linz, I saw a poster in the office. And it had your picture on it. And it reminded me of one of my favorite shows out there, Mr. Robot. And without further ado, I just want to let you talk about what is this all about, what is this HackDT program all about, how did it come about the idea, and how did it work for you?

Right, so I actually love the poster. You guys on the show, you've got to check it out. So the way it started was I kind of started to work with Dynatrace a year ago and my job was to build the whole penetration testing unit and at this point I found an old wiki page talking about Intel and Buckbunny programs and we had one in the year of 2014 and I was saying you know what like we got to host another season of this and like come up with a new Buck Bunny program.

And that was the point we sat down with our like design department that we have in Linz as well. And we said we got to need some kind of theme for that. We got to print the poster and it gotta be awesome and i don't know like after a couple of of rounds we came up with the idea of um kind of um using elliot anderson out of mr robot um as a theme and they came up with the idea to use me as as elliot and i don't know i ended up um getting photographed and i had like a ton of makeup in my face to look super tired and like on drugs and whatever. Like people out there watch the show know how Elliot looks like. And that's how I ended up in all our offices.

I like how you had to specify that they had to put makeup on you to make you look that way. And that's not how you normally look. So appreciate that. That's my regular day to make you look that way and that's not how you normally look so appreciate that's my regular day-to-day self that look yeah you love me but i have a lot of people coming up to me saying um what did you do like the night before the photo shoot yeah we're going out again like you're usually doing i'm like come on guys it's called makeup special effects right that's pretty cool i think the is the poster is, if people are interested, the poster is up there on the blog as well, right? Hack TT. You said it's season one.

And so what was the – I mean I understand now how it started. I didn't know actually that we had like years ago a little program like this before, and then you kind of reengaged. So how did it work? And did you get any inspiration maybe from somebody else as well on what the rules are? How does it work? So basically, I think to start off, what a Buck Bunny program is, I got interested in those programs a buck bunny program is. I got interested in those programs a couple years ago and what

those are is big companies like the Googles and the Facebooks out there, they don't want to just hire like an external party to test the systems. You want to hire like an army of people who are into the field and that's why they set out a program and

they tell you like, you're allowed to hack our systems, but only under these circumstances. And well, that's basically what they do. And then it's called an external bug bounty program. And if you respect the rules, you can hack the systems and send them an email or send them like a message on any kind of portal which is out there telling them that you found a vulnerability and this is what we've been doing as well and we actually have an external program running as well on a platform called hacker one but this time we wanted to have it internally and that's there was the point where i just said you know what like we have confluence uh what if i just set it up there and and ask all our developers to take part in this and that's that's

how it started awesome pretty cool and uh and i think there's a there was a reward like the bounty itself uh i think you explained it a little bit on the website or on the blog i mean what kind of what the the monetary motivation is for can you give us a little rundown on how you came up with these numbers and and uh sure kind of what you paid out in the end so obviously like if if you're a an excellent penetration test and you wanna uh and like a big company is asking you to check the systems you want to get some kind of reward and in a lot of cases this is just kudos or uh i don't know some some like fancy swag but what people need and love the most is money

so it was obvious to us that we gotta have like a pool of a couple thousand dollars as well and we came up with the idea of spending ten thousand dollars that's how it all started and it's it's internal people working on this right and they obviously already have their jobs and they're earning money for working at donatrace but we said you know what like if you search for a security bug within Donna Tracy you get an extra payment on top of your your monthly salary That's pretty cool and did this Organizational wise or time wise did you? Did the people do it like in their spare time or did you set aside?

Let's say I don't a couple of hours a week or how did that work like the logistics for it that would be interesting for me so um the ground um the basic rule was you guys so we work in sprints at dynatrace right and obviously like the teams have a team lead and they tell their employees what they have to achieve during that sprint. The one and only rule was you can work whenever you want on finding security bugs, but do not neglect what you have to do during the sprint. So if you're done with all your tasks after like five days and the sprint is still ongoing, then you get to go to search for those vulnerabilities even during your regular work time. And of course, they can search for some of the vulnerabilities if they're including security

in their sprint testing anyway, right? That's correct. I mean, those are people who know our product the best. So it was the easiest to actually search for security vulnerabilities because they are dealing with those applications that we are having on a daily basis. And was everyone invited, meaning every engineer? Or did you have a certain, let's say, basic, I don't know, minimum job requirements or anybody within the engineering team could participate?

Well, if it was just up to me, I was saying let's let everybody participate. Who cares? If they are finding anything, they just should be able to report it, right? But in the end, we ended up having just our research

and development teams participating because if you take a close look at it, I mean, those are actually the only ones dealing with our product on a daily basis. So it doesn't make too much sense to include like salespeople or like human resources and all those guys. So we just had R&D on board.

Cool. And I see one of the, I think it was one of the screenshots or the pictures that is on the blog as well. When you basically did a kickoff session, right? I think it seems like it the screenshots or pictures that is on the blog as well. You basically did a kickoff session, right? I think it seems like it's next to the cafeteria in the Linz office where you kicked off. It is in Linz, but did you also have the remote people in Gdansk and in the U.S. invited as well?

Yeah, so we shared this whole talk, obviously, with our remote office as well. And we had a lot of people from Poland and the U.S. as well reporting issues. So it definitely made sense to include them as well. Pretty cool. And I assume, I know in the blog post you said you couldn't reveal all the numbers because obviously we don't want to say what type of bugs we found. But I would assume overall a big success of this program. It was. We had a huge number of people contributing, and that was all that mattered.

And if you think about it, it's not just about finding stuff. It's also about raising awareness and actually give our developers some kind of motivation to dig into the topic of computer security and security in general. And I think there was even way more than just the people actually contributed who tried to find any bugs. And obviously that raised their awareness by a lot. So that was one big goal that we wanted to achieve. So it means it – go ahead, Brian. I was going to ask, from all the things that were found, was this the sort of thing where a lot of those tests that were used to, or the functions, whatever they did to find the flaws, were those then turned into parts of standard testing within any sprint? Or was there a carry-through from what was learned into part of the release cycle with that? So this is another task we're focusing on right now, which is security automation.

And what we are doing at this point, so we're in the aftermath, basically, of our BuckBunny program. We're taking those findings that our developers found and write regression tests. So we have them, not now, but soon in in our um pipeline which is going to be awesome because that means we won't have those vulnerabilities again yeah we need to we need to get you on another podcast and and explain continuous security validation or whatever you call that or i'm sure you will write some blogs about it because I think that's very interesting. And you see that all over the world, like in all the big companies, they are going down that road.

Like you cannot test everything manually anymore. It's all about automation. I mean, you guys know that, I guess. Yeah. So we have that insecurity as well. So what I love a lot, so you said, I mean, if I'm an engineer, right, and I've been working on my features, and obviously I'm very good with what I'm writing there and the technology I'm using, I really love the fact that you are giving developers a chance to kind of exit their regular realm and learn something new. And as you said, the complete security awareness

is also a major uplift for the whole R&D organization. And it's not that everybody becomes an expert, but if you are at least familiar with the basic security concepts, and if you looked for them, and if you learned there are actually certain problems in the product that you have just now detected and then every line of code you're going to write from now on, you keep these things in mind that you just learned and found. I mean that's amazing. That's an immediate uplift of everyone.

Did you – Pascal, did you offer some type of initial training kind of – or was this part of your initial session where you said, here are some tips and tricks, so here are, here's where I would start, you know, with digging for security holes? Did you give them some basic education? Not too much, to be honest, because we all have busy schedules right um i gave um a talk more like a workshop in detroit a couple of months ago on on like using penetration testing tools but that was just for a small group um what we did was we released the the findings people found obviously back to people searching as well. So it kind of created a situation where they had the chance to look at the findings up to a specific point,

and they learned out of those findings, so out of those tickets. And because we still had some money to give away, they kind of ran through the description of the vulnerability, learned what it is, learned how to fix it or what it actually does to the system. And they started to search for the very same vulnerability as well. So that was a big effect that we saw on our developers that they basically learned while doing, while searching for the vulnerabilities, which is absolutely great. Can I take this in a cynical direction? So I don't know if this is the result of my upbringing and the poor choices and friends I had when I was little,

but I'm wondering, do you think there's any risk in a bug bounty program like this, especially when security is involved, of someone saying, oh, hey, they just announced a new one. I'm going to bake in a flaw and either find the flaw that I baked in or tell my buddy. This way it doesn't look like I did it. Tell my buddy about the flaw so that we can get the money. I mean, I know it's a really cynical thing. And the real important thing is finding all these problems. But, you know, in other security hacking things security hacking things, does this ever come up?

Is this ever a risk or something to even consider? Yeah, it is. I mean, we were thinking of that. So we had a couple of rules. And another one was if it's your own code, it's not eligible for a reward, right? Yeah. So first of all, you cannot check.

But I can tell Andy about it, and he can make the money, and then we can split it. That's true. I don't know. I'm just going off the deep end. A major rule was do not trick the bug bunny program. And I actually put it into the program guidelines this way. I was just saying do not trick the bug bunny program and i actually put it in into the the program guidelines this way i was just saying do not trick it because we're all colleagues at the end of the day why would you why would you do it and i mean it's just for a couple of hundred dollars if you really go

down the road i mean something's wrong with you if you go down that road. It is. And you can do so much worse within an organization, right? You don't have to, I don't know, build in a bug and get like $200 out of it. You can do much worse. But I need to get a new game. I don't know. But I think, Brian, it's great that you brought up kind of the risks. Anything else, Pascal, where you said, well, if we would do it again, we would have certain things that we would do different. Anything that

you learned to improve it, to make it better? Yeah. So we learned relatively fast that we kind of have to keep the bugs to us first. So if there is any sensitive information in it, which we don't want to have to be out there for everybody, and everybody speaking of just our colleagues, of course, not the world, we realized that we want to have the chance to go to finding by ourselves. And by ourselves, I mean the board, which was me and three other people out of the security team who looked at the findings first and if we found that there was any sensitive information in it we kept it to ourselves if it was all fine we released it and all the other dafs basically

had the chance to look at it as well. That was one improvement that we found. What else? That's a good question. It took a lot of time. Yeah. I got to tell you, if you get a lot of findings in and you want to fully understand what our developers were reporting

and give them an appropriate reward, it takes a lot of time. And that's basically a key learning for ourselves for the next time that we're doing this, that we have to keep it simple, make it simple in any way to, I don't know, evaluate the finding a little bit faster and come up with a reward faster.

Because at this point, it took us a couple of weeks. Yeah. Yeah. Go on, Eddie. Yeah, no, I'm just saying, so the lessons learned, and you're kind of implying it, there will be another round. There will be another season, as you call it, for the program with these improvements. There will. So first when we started off, we said, you know what, let's do this every quarter from now on and it's not

going to be much work and then we realized it is and so we we kind of came to the conclusion that we want to do it at least once a year and i think this is where we are at right now so we had it starting in in january so we probably have the next round next year in January again, which gives us a lot of time to, well, first of all, let our developers remediate the actual findings and to improve the program. And, well, I don't know, to deal with our other tasks as well. I mean, we're not just running a bug bounty program. Obviously, we have other tasks as well within the team. So I guess once a year's good to have it like that

yeah exactly they have all their own regular stuff to do and this is not a security only team so that would kind of and if you had a security only team they'd probably have a singular mindset that you wouldn't get the variety of the developer pool who might look at things differently.

So I think it's definitely good that you're continuing to use the general pool. Because going back, I remember a way long time ago, this was way before security was serious. I was working at a popular weight loss website, and I think there was some new search function, and I was working in QA at the time, and the developer was like, go on, I dare you to break it. You can't break it. It's bulletproof.

So I took and copied and pasted a SQL query in it and broke it. And I'm not even a hacker, you know, but he completely didn't think of it from that point of view. So the only point I'm making there is the variety of not just security people, obviously, having it really be the whole development team who might have a lot of different ways of looking at it. It's key to these kind of operations, I think. Yeah, right. Hey, so, Pascal, I'm just browsing through the blog post. There's a table where you are kind of listing the monetary reward. And you are using a – it's called a CVSS, the Common Vulnerability Scoring System.

Scoring system. Yeah, scoring system, yeah. So it seems there is something out there that is kind of its first org. Yeah, cool. So any other resources out there if people are interested in running these type of hack days or bounty programs? Any other pages, links, blogs that people should look into besides obviously what you did and what you wrote? Yeah, there's a lot. So if I would start a bug bunny program within my company, what I would do first for all the listeners out there, I would go to a page called HackerOne, or there's another one called BugCrowd. So it's hackerone.com or bugcrowd.com. And I would say those two are the industry leaders for running BackBunny

programs. And they have a portal where you can basically run your BackBunny programs for external testers, but also for internal testers. But if you go to the pages, what you see is all the external programs offered. And you have like a big set of rules that you can go through and you can compare them. So that's what I did as well. I took the program off of the Googles and Facebooks and I was like, okay, I probably should restrict that. Ah, right, I probably have to do it like that, like this.

And you also see the reward table. And I didn't really see a system behind the rewards. It's like the Googles, they pay 20 times as much as the smaller companies, obviously, because they want to have more people searching for books. So I came up with my own one. And what you mentioned, the CVSS is just a well-known scoring system within the security community. And it basically asks you a couple of questions. It has a calculator and it asks you stuff like, how much confidentiality do you use with that security

vulnerability? Or how much availability do you use and so you click through the calculator and you get out a number which has a range between zero and 10 um and that's what i what i took i just said you know what like we have to evaluate those findings anyway so what if we just give it a cvs score and then give it some kind of what I named the multiplication factor. And this gives me the reward for the finding. And that ended up being pretty awesome, I guess, for us. It was sometimes a little difficult to calculate the score, but that's another story for itself. But maybe just a little example, if I get a score of like 7.3, which is a high finding, so you want to remediate that as soon as possible. We gave it a multiplication factor of 75, that makes it 547. And we just took this as the amount in dollars.

So $547 for the person who found it. That's pretty cool. All right, that's cool. So that means if anybody got inspired by what you did within Dynatrace, there's a lot of material out there. I can just, again, reiterate a great place to start is the blog post on blog.dynatrix.com. It's called Running a Successful Internal Bug Bounty Program. And really much looking forward to hearing from you again about Season 2.

And then also, very much, Pascal, we need to have you back once you have integrated the security checks into the pipeline. Because what Brian and I have been talking about a lot in our podcast is talking about continuous performance validation. And in this case, what you are doing is basically continuous security validation. And I believe as security, as we all know, it becomes more and more important and has always been important. But it seems we get too many examples these days about breaches. We want to make sure that we also cover these types of topics. And a program like this is just a phenomenal way to get not only a company thinking about it, but addressing the problem and, more importantly, leveling up everyone that works for that engineering team, make them aware of security, give them a reward,

but also strengthen their skills when they write the next line of code. I love it. That's really cool. Yeah, and you know, Andy, I'm going to think out loud here just for a quick second. I wonder if there are,

it might be a stretch, but I wonder if there are any kind of performance metrics that would indicate or if there are any kind of performance characteristics that are tied to security flaws or certain types, you know, if there's something to do with the database queries or numbers of executions or whatever.

I just can't think of how there would be, but it would be interesting to start looking at. You just mentioned it, right? So database queries. If you would expect a simple query to be inserted in a specific application, and then an attacker is coming up, inserting this massive vulnerable payload, it probably takes a little bit longer. So that would be an indicator. Okay, interesting. Yeah, and also I believe, I mean, it depends on a little bit longer. So that would be an indicator. Okay, interesting.

Yeah, and also I believe, I mean, it depends on what you want to do as a hacker, right? If you want to really provide malicious data and feed that into the system, or if you want to get data out that you should not see, or maybe attack the system and bring it to its knees. I'm sure there's all sorts of different

security vulnerabilities here. Then the last example, bringing a system to its knees, I'm sure there's all sorts of different security vulnerabilities here, then the last example, bringing a system to the knees, we can easily look at things like change memory behavior of a certain feature. With a certain combination of parameters, we see a different memory or CPU behavior, and we could flag those. So that would be interesting. That's like you would need some kind of artificial intelligence engine

to be analyzing what's running normal and then... Oh, my God. Maybe we should... Isn't there a product out there that starts with Dynatrace and ends with Trace? I think so. Shameless plug. There's lots of potential, that's for sure. Dynatrace has huge potential with the capabilities that we already

have. We have massive potential to integrate security checks as well. Actually, that's actually a question that, I mean, this is an interesting segue. Pascal, I know that the Dynamitrace product has been obviously influenced by our engineering teams

because we are using Dynamitrace internally. I know that, for instance, Log Analytics has been heavily influenced. The reason why we have, I think, Log Analytics, one of the reasons is because we needed it internally. Now, do you work with the engineering team that works on Dynatrace and actually discuss some of

these security use cases and how we can bake it into a product? Is this something that you're working on? Well, yes and no. So like, personally speaking, I'm not. But you just mentioned log analytics, for example. We just had a topic this week where we kind of sat down with the team and we said, you know what, like, there is a couple of interesting logs from a security perspective. Can we introduce them to Dynatrace as well?

Can we monitor them as well? And I think it was a pretty successful meeting, and we're probably going to see them soon in Dynatrace. Ah, cool. So that means Dynatrace is looking for certain log files that are written by certain components, but then maybe also alerting on certain patterns in logs, like a DDoS attack or whatever it is. So you sat down and said, hey, Damatrix, let's make sure we are automatically monitoring these log messages and in case there is a certain sequence of log messages,

it's a good indicator that there's a security breach. Is this in the direction that you're heading? That is absolutely what I meant and what's working, but we just don't have it at this point. But I'm pretty sure it's coming. And like I said, I don't have the specifics right now because that's a colleague of mine who had that call. But yeah, we were going that way.

All right, that's for sure. That's really cool. Cool. All right, Brian, what do you think should we wrap it up yeah that's some of the summerator let's do it so pascal thank you so much again for for joining us and i mean what i took away from from this session is that not only is a program like that great for the software itself we are kind of testing and finding more vulnerabilities. But I believe the better aspect from an engineering perspective, from the individual is that you

actually get more aware about security, about building and writing better code. And I think this is what I love. I would encourage everyone out there that is listening, again, read Pascal's blog. In the blog, he's also linking to HackerOne and also the other blogs that he mentioned earlier. So if you want to do a bounty program like this, there's a lot of examples out there and a lot of material. And Pascal, looking very much forward to Season 2 and all the other work that you're doing. And you're always welcome back to the podcast because I believe this is just another great aspect of our industry that we need to cover more in our podcast. So thank you so much. Yes, thank you. Well, I want to thank you guys for having me and looking forward to being on a show again,

that's for sure. Also, for the listeners out there, if you have any more questions, I guess I have my LinkedIn account linked and my Twitter account. So I'm also up for answering any questions our listeners are having. Yeah, that's what I can offer. And what's your Twitter account? We'll put that on the site, but also if you could just verbally say it out loud. It's PascalSec, so for security, but just a preview. Okay, awesome. And we'll put that on the link. I think this is

awesome too because as we know, security is all the rage these days. If you're going into obviously any kind of IT career, security is a wide open door at this point I think. And running something like a hackathon, especially internally like this, is going to be more and more important because what you see in the news all the time is not only the hacks that are occurring, but the public hackathons. if you're doing this all internally on your own and finding all these problems before other people

get a chance, you're way ahead of the game, not only for the security and safety of your customers, but just your whole organization and people. So I think it's a really, really awesome that we've started down this journey. And I think it's going to open a lot of cool things, even, even within our product about this. So I think it's awesome that you did this and I'm really, really happy to be hearing about it. And as Andy said, we'll hopefully want to start covering this topic a bit more. I think there's anything else from anybody or we're all set here.

No, I'm good. Just I think we're not sure when is, when is this episode airing? It's going to be probably be so two weeks after June 18th. Okay. So it's going to be be two weeks after June 18th. Okay, so it's going to be late June or early July. So by then, I want to wish everyone a lovely summer

if you're obviously on the northern hemisphere. If you're on the southern hemisphere, then enjoy the winter. And you'll be in Austria. Actually, by early July, I will just be flying down to the southern hemisphere. I will be doing performs in APEC. So bringing the unbreakable pipeline to Australia, Singapore, and Malaysia.

There you go, traveling some more. Yeah. All right, if you have any questions or comments or you want to be a guest on the show, please contact us either on Twitter at Pure underscore DT or email us at pure performance at dynatrace.com. And if you like performance podcasts, also make sure to check out our friends at PerfBytes and the new PerfBytes show called Ask PerfBytes hosted by yours truly. Thank you for listening, everybody. Thank you. Bye.