The Changelog: Software Development, Open Source - Attack of the Canaries! (Interview)

Episode Date: September 13, 2023

This week we're joined by Haroon Meer from Thinkst — the makers of Canary and Canary Tokens. Haroon walks us through a network getting compromised, what it takes to deploy a Canary on your network, how they maintain low false-positive numbers, their thoughts and principles on building their business (major wisdom shared!), and how a Canary helps surface network attacks in real time.

This week on The Change Law, we're talking about network security and deploying canaries. Today, we're joined by our new friend Harun Mir from Thinkst, the makers of canary and canary tokens. On today's show, Harun walks us through a network getting attacked, what it takes to deploy a canary on your network, how they maintain low false positive numbers, his thoughts and principles on building their business, some major wisdom dropped here, and how a canary helps surface network attacks in real time. A big thank you to our friends and our partners at Fastly and Fly. This podcast got you fast because Fastly is fast all around the world.

Check them out at Fastly.com. And our good friends at Fly help us geolocate our app and our database all over the world with no ops. And they'll do it for you too. Check them out at Fly.io. what's up friends i'm here with vj rajji ceo and founder of statsig where they help thousands of companies from startups to fortune 500s to ship faster and smarter with a unified platform for feature flags experimentation and analytics so vj what's the inception story of statsig why did you build this yeah so statsig started about two and a half years ago. And before that, I was at Facebook for 10 years where I saw firsthand the set of tools that people or engineers inside Facebook

had access to. And this breadth and depth of the tools that actually led to the formation of the canonical engineering culture that Facebook is famous for. And that also got me thinking about how do you distill all of that and bring it out to everyone if every company wants to build that kind of an engineering culture of building and shipping things really fast, using data to make data-informed decisions, and then also informed to what do you need to go invest in next. And all of that was fascinating, was really, really powerful. So, so much so that I decided to quit Facebook and start this company.

Yeah, so in the last two and a half years, we've been building those tools that are helping engineers today to build and ship new features and then roll them out. And as they're rolling it out, also understand the impact of those features. Does it have bugs?

Does it impact your customers in the way that you expected it? Or are there some side effects, unintended side effects? And knowing those things help you make your product better. It's somewhat common now to hear this train of thought where an engineer developer was at one of the big companies, Facebook, Google, Airbnb, you name it. And they get used to certain tooling on the inside. They get used to certain workflows, certain developer culture, certain ways of doing things, tooling, of course. And then they leave and they miss everything they had while at that company. And they go and they start their own company like you did. What are your thoughts on that? What are your thoughts on that kind of tech being on the inside of the big companies and those of us out here,

not in those companies without that tooling? In order to get the same level of sophistication of tools that companies like Facebook, Google, Airbnb, and Uber have, you need to invest quite a bit. You need to take some of your best engineers and then go have them go build tools like this. And not every company has the luxury to go do that, right? Because it's a pretty large investment. And so the fact that the sophistication of those tools inside these companies have advanced so much and that's like left behind most of the other companies and the tooling that they get access to, that's exactly the opportunity that I was like, okay, well, we need to bring those sophistication outside so everybody can be benefiting from these.

Okay. The next step is to go to statsig.com slash change while they're offering our fans free white glove onboarding, including migration support. In addition to 5 million free events per month that's massive test drive statsick today at statsick.com slash changelog that's s-t-a-t-s-i-g.com slash changelog the link is in the show notes All right. We are here with Harun Mir from Thinkst. Hey, thanks for coming on the show, Harun. Thanks for coming on the show. And it starts to...

Thanks for having me. Sorry about that. I had to do it. We're happy to have you. Owen Valentine. Shout out to Owen. Longtime listener. He likes to put episode requests in.

And he said, hey, talk to you. And I said, okay. I take orders around here. And I checked you out, and I thought this is pretty cool. Security products coming out of South Africa, Canary tokens, lots to discuss. Where should we start? I know you have strong feelings on bootstrapping versus VC funding. I know you have strong feelings on InfoSec, the industry.

You probably have a cool perspective coming from where you're coming from. What's most interesting to you? So I think it's pretty open. I think we should probably start with, I don't know, with what we do. So Canary and Canary Tokens is probably a reasonable place to start. Yeah, let's hear it. Okay, so effectively what we do is we build products to let people know when they've been compromised. Like the opening logic is, so in our previous lives, most of our

early team worked as pen testers. So breaking into networks. And one of the terrible secrets is that for years and years, like we'd break into networks all around the world and nobody knows, like not until you hand in the report. And this hasn't changed much. So it happens when you're doing pen testing. It happens when real attacks happen. And so companies just find out they've been compromised months after. And so our whole pitch is to try to fix that. And so canaries are an old concept, which are honeypots. And honeypots have gotten a bad name historically because mostly they were used by the research community. So people would put up honeypots and say, look at how many attacks were from Russia. Look at how many attacks were from China, which is pretty useless for real world activities. And what we do is we say, if you had honeypots on your network and they were really low effort to deploy, they end up giving you a really high signal.

So you find out at two in the morning that Bob from accounting just tried to log into this machine that really shouldn't be there. And what it works on is just the logic that the people on your network or in your infrastructure know your infrastructure, but attackers who land there need to situate themselves. And so typically those attackers run around like bulls in China shops when they get there. And so they touch, they explore things and they touch stuff. And inevitably they touch these canaries, giving you a really high signal that badness is happening. And that's all we do with canary. We make it super easy to deploy them so that people actually do it. And then we focus really hard on not generating extra noise. So customers with like hundreds of canaries will get four alerts a year

so that when they get an alert, they know they need to react to it. And so Canary was the first product we built. And then we built Canary tokens, which are the same concept, but much smaller tripwires. And for the broad applicability for your audience, will give you, for example, an AWS credential file. You put it on your CFO's laptop. And when anyone uses that credential file, that API token, you'll get a message saying, listen, the AWS creds that were only on your CFO's machine, somebody just used it to log in.

So again, you get a really high quality signal that tells you someone was on your CFO's machine, somebody just used it to log in. So again, you get a really high quality signal that tells you someone was on your CFO's machine. And so Canary tokens are bunches of little tricks like that, that are really hard for attackers to resist, but give you a really high quality signal that something is going wrong. And for Canary tokens, we give them away free. And so literally millions of people have used them or use them to figure out when they've been preached. It's a really simple concept. I mean, this is like read receipts on things that you don't want someone to read, you know? Exactly right. And a part of, so I've done this talk in a bunch of places where I point out that really simple things done well are in really short supply. And it's a whole other soapbox of mine where I feel like people building products are incentivized without anyone being mustache twirlingly evil, like the world kind of sets up so that every time you speak to them, you ask, well, what's new in the product? Well, are you now doing this? And so nobody's ever

incentivized to do something and just focus on doing that thing super well. Because what they've got to do is keep showing new features so that you think that it's worth it. And we fight the urge really hard. We try really hard as a company to make sure that we always did simple, always low noise, always our CTO is just as reliable as a brick. People need to be able to build on it, no complications, and know that it works. And that's our pitch. These canaries act as like standalone machines on a network. Give me an example of the footprint. How does this work? Yeah, exactly right. So version one, we shipped these little hardware devices, which in 2016, when we started, nobody was starting a

company saying, let me ship hardware. Now it's cool. Yeah. Yeah. It worked out pretty cool. Now it's kind of cool. Part of the reason we did it was because we really wanted it to be easy for security teams to deploy. And security teams still have a problem with, hey, can I spin up in the DMZ? Can I get a virtual machine? And this way we were saying, take this box, plug it in, and you're good to go. And typically, each canary imitates exactly one host. And so when you plug it in, you say, listen, I want you to be a Cisco router. And from that moment on, that device, like its MAC address is a Cisco, the services are Cisco. If you NMAP it and get its TCP IP stack, it'll respond like a Cisco. And the work that we put in is that if you then

would literally just two clicks say, I want you to be a Windows server instead, that device reboots and now it's a Windows server on your network. And now it runs a Windows file share or Windows RDP and you can enroll it in Active Directory. And the whole point is that it shouldn't be hard for you. You should be able to go make this a Windows box, put it in my AD, enable RDP and a file share. Or you say, hey, I want you to be an IBM mainframe, expose TN3270 and LDAP. And you drop them and you forget about them. We've got customers who never looked at their canaries for seven years. And what you're looking for is in year four, when people break into their network and are logging on, you get this one message that says, listen, somebody found

this Windows share. Somebody went into the directory called exec salaries and somebody copied all these files. You've got a problem. And that's the whole pitch. What about the stack trace after that? I got more questions, but is there any sort of like, this seems like error monitoring basically for an application, but instead it's sniffing out attackers, finding hosts on your network. Yeah, it's literally what we're looking for is, and the way we pitch is, we want to give you one clear signal that you've got a problem now. And we'll give you the details related to that attack. But we end our mission there. We don't then do forensics beyond that or all of that stuff. Part of our pitch is like from old Unix, do one thing, do it well, play well with others,

give your output so that you'll work with others. And our major push has always been make this quick and painless. So we obsess about that to ridiculous degrees. It's got to be quick, got to be painless. So the repository on Thinkist, not Thinkist. On your org on GitHub is OpenCanary. Is this the software that's running on your hardware? It's a version of it. So what happens is we built Canary

and beyond OpenCanary, so OpenCanary shares part of its internals. But if you downloaded OpenCanary, you'd have to install it. You then need to make sure you've got monitoring on it and reporting on it. And if you subscribe, if you buy our Canary, you'd have to install it. You then need to make sure you've got monitoring on it and reporting on it. And if you subscribe, if you buy our Canary service, you get these devices, but the devices report into your console on AWS. And so from your console in AWS, you get to click on a Canary and say, I'm tired of you being a Cisco. I now want you to be a Synology NAS.

Your device reboots and now it's a Synology NAS. And then I mentioned earlier that version one was these hardware devices. Since then, we've got VM options. So Hyper-V, VMware, or cloud options, GCP, Azure, AWS. And the Canary tokens, which we spoke about, if you're a customer, you get your own private Canary token server. So literally you can mint a jillion of those tokens free all through your environment. And yeah, it becomes the lowest priced high fidelity alerting you can get. My lens is sometimes the home lab lens.

Right. And so I'm thinking of this from the home lab perspective, because you mentioned a Synology, which is awesome because that's like mostly, I mean, it's not only in the home lab. There's a lot of small businesses, offices using that, for example. Totally makes sense. And part of the joy with Canary, which in some ways we got a little bit lucky with because we thought it was a good idea, obviously, which is why we started building it. But something that we didn't really count until we saw it in action is that with lots of the stuff, that's why attackers are on your network. And so people sometimes go like, well, what happens when attackers get smarter or attackers wise up? And the simple thing is like as pen testers, like this is what you do. Like you go further

on the network by finding one more open file share, grabbing one more config file that had a password. And you can't just not do that. Like that's what you do. Yeah, exactly. And if you take the Canary token example, if I get to your CFO's machine, and if you tell me these folks are running Canary tokens, and I now see an AWS API key on that machine, am I going to not try it? Potentially, that's access to your cloud. And so one of the things we're super proud of is like, we do zero outbound sales, right? All our sales so far has just been internal. And we collect a whole bunch of tweets on a site called canary.love, which is people saying nice things about us. And one of the interesting things is that you'll see lots of those comments from pen testers and

red teamers saying, well, now when I find stuff on an engagement, I don't know whether I can use it or not because maybe it's a Canary token or now I'm scared to try this when I find it because, hey, maybe. And so it's interesting because it changes that calculus a little between attackers and defenders. Yeah. I have a couple of questions, both on the implementation side, but also while we're talking pen testing, because I've done some pen testing as well. And you're absolutely right. Like you basically are feeling around in the dark,

you know, and you're just looking to discover, you're trying to shine light on new areas of the network. And so of course you're going to like touch and feel and like, that's exactly what you're after. But my question is, and maybe you can't divulge, but is there a way to like fingerprint a canary without touching it somehow? Like if I was super smart, what would I do? Yeah, it's a great question. And it's been our thing from day one. Like when we proposed this, one of the early anti-takes would be, well, you're starting an arms race because now people will start trying to fingerprint you. And I have a bunch of answers for it. One of them is like, at least get into the arms race. Like right now, you're just getting your tail kicked as a defender. Yeah, true.

Okay. Get into the arms race. But two, we can, as far as possible, try to identify you trying to identify us. So for example, the early Nmap is a really good example. When Nmap does an OS scan on you, it has a very observable fingerprint. And so a canary will be able to tell you, hey, you're not just being port scanned right now. This person is Nmap OS scanning you. And depending on how you configure your network, you might say, no, we should never be NmapOS scanned. Let's react to this. And so it for sure starts this arms race where clever people will try to figure out ways to do it.

But we end up being on a better wicket there because all I have to do is say, hey, what you're doing to me is not usual because you shouldn't be talking to me at all. And so it dramatically changes that calculus because now attackers have to be careful of everything. One of the folks who work for us says like our entry-level package should just be stickers saying I run Canary because in all that just says I'm running it, you just put attackers in a horrible position like like you said if you spend all of that time feeling around in the dark now you're just terrified that anything you touch is gonna make you exactly yeah it's a bomb essentially right well the the trip wire it could be the tripwires you to use your terminology it's like playing minesweeper you know but with no

information you're like well yeah exactly Yeah, exactly right. For sure. So on the implementation side, the software side of what Canary does, is it masquerading as these different OSs and services, or is it actually like rebooting the VMs, or how does it actually do it? No, we masquerade. So there have been people who've tried deception products who boot full-blown operating systems. Right. And we've got, like for many things, we've got strong opinions on that sort of stuff.

One of the things we feel very strongly about, for years and years as pen testers, we'd own networks because of their security devices. So they'd buy stuff, they'd implement it, it'll be dual-homed. You take it on this network and you'll hop across to that network. And so we spent crazy amounts of time making sure that you will never attack a canary and be in a better position than you were in before you attack the canary. And so we're not running vulnerable versions of Windows. We're faking out the TCP IP stack. We're running network services that we've written in memory-safe languages on jailed file systems. And again, we're not Oracle. We're

not saying we're unbreakable. What we are saying is we're really conscious to minimize splash damage or blast radius. So nobody's going to be better off for attacking us. And one of the reasons, one of the ways we're able to do that is by emulating stuff. And then the question is, can we emulate enough of it to pull off the con? Because effectively you're trying to con a user. And again, there's two things to that. The one is we believe we can. So for some of them, like, we'll talk good Redis right till we've caught you, or we'll talk RDP till you authenticate and tell us who you think you are. But on the other hand, I think it's one of those things we make a mistake with when we judge security products sometimes, which is where people question like, well, I can think of ways to defeat this in the lab. Like, what if I did this and then did a timing attack? And I could tell that you were responding to me slower than a real machine

would respond to me. And while we're doing that posturing, like Snowden is mapping to every share that he can and stealing every PPT that he can in your organization. And so like if Snowden was getting an SMB share or a Samba share or a black box version of a Windows share, he didn't care. He was just grabbing files. And this still happens on networks everywhere. Like attackers want what they want and almost don't care what's underneath it. Another super interesting thing with that, we spend crazy amounts of time making sure that our con is complete, right? And the user never has to know this, but it should completely con an attacker. But I've been on pin tests where it's the middle of a Windows network and suddenly there's a scorebox. And I'm like,

what is the scorebox doing there? That doesn't stop me from browsing its file system. Like more than anything, I browse that file system. And so I think people overestimate how deterred an attacker would be if something looks odd. In actual fact, that's par for the course. Like everyone has a Frankenstein box that they've forgotten about that actually has the keys to the kingdom

because they've forgotten about it. Yeah. And so it just turns out to work in our favor. Yeah. Just really hard to ignore that one weird looking box that you think, well, maybe this is my way to the next stop, you know? Exactly. It's like an unsolved mystery, basically, you know?

What's inside the box? Especially when you've been not making progress for a while, you know, and you're like, oh, wait a second, you know, what's this? It's a super interesting insight and one we didn't have when we started off. So we spent all this energy making sure we could totally imitate stuff. And at some point we realized, hey, we used to give classes on pen testing at Black Hat for several years. And one of the things, like we've got slides where we tell people, like if you land on a Windows network and you see lots of Windows servers and one lone Red Hat box, go for the Red Hat box.

Like these people know how to configure their Windows network, but they had to put up this Red Hat box for their telephony or for their NAS. And you own that box because these Windows people don't know. And then when we made Canaries, our first instinct was we've got to make stuff that really blends in. Well, actually, we were teaching attackers, go for the stuff that doesn't blend in. And so, again, totally by accident, it puts us in the fortunate position where we say, listen, just deploy your canaries. Like, don't think too hard about it. Just deploy it. You almost can't deploy it wrong because if it blends into your environment, that's cool.

And if it sticks out, that's cool. Just deploy them. Empirically, it works. What's a typical ratio, like canaries to real? That's a super good question. So initially we thought it would need to be high. In fact, we had it as an open research question for how many needed to be deployed. And in truth, the number can be shockingly small. And in part, that's because attackers are bouncing around networks for months. Like until ransomware

started giving attackers incentive to disclose themselves early as they ransomed your network, like the average used to be more than a year. So attackers would sit on networks for more than a year. And during that time, what they're doing is trying to find your crown jewels. And so if you have 10 canaries on a large network, but in your DMZ or next to key servers, that's what the attackers are trying to do. They're trying to get there. And so part of the way we pitch the service is take five. Like take five to make sure that this is not vaporware.

And again, we've got kind of a strange sell approach. It's almost a hyper version of the PLG that everyone looks for. I mentioned we've got no outbound sales team. And so pitches, you go to our website, you see the price and you say, try five of them. And nobody tries to upsell you ever. And typically what happens is within a year, you have a pen test and those pen testers get rumbled by your canaries or you catch something. And then when it's time to renew, typically someone says, you know, actually,

we'd like to put these in all of our remote offices, or actually we just made an acquisition and we don't have time to go down there. Can't we just send four canaries down there? We're like, cool, we'll just ship them to that address and people can grow their flock. Literally, we've got customers now paying us hundreds of thousands who we've never met and we really don't have the sales team to sell to. And they've just upsold themselves every year. That's interesting. I would expect the sales process to be somewhat difficult because your payoff moment is like when they get hacked, you know, and it could go maybe, I mean, I guess if you have an annual

pen test, maybe that's what triggers it. But if you don't, then you could go years without ever providing value, quote unquote, right? Visible value. Yeah, it's so it's a good insight. And it's another place where I think we got accidentally lucky. And by that, I mean,

I think if people were not doing pen tests, far more people would have been questioning the value of it. And we didn't have a plan. If you asked us on day one, we hyper-optimized for being silent unless it's a real attack. And so how would people know that there's value? And typically what happens for us is, so there's two things. The one is, this sounds super corny, but I'll say it anyway. We make sure

that the installation is delightful. And so our initial pitch is, listen, 7.5K, you get five canaries and the hosted console. Just try it. Like, how bad can it be? Like 7.5K, try this. And then we got super lucky because like Slack used us and said something publicly and Airbnb used us and said something publicly. And so then our message was, hey, seven and a half K and Slack says we cool. Like how bad can we be? And then we've got to make sure that your first experiences with us are delightful because we've got to convince you. And so we put a lot of effort into removing all the suck from that experience. And then we've basically got a year to earn our keep. And typically within that year, we'll catch pen testers or we'll catch real attackers or we'll catch some network misconfig that you never saw coming. So you've got canaries

in this zone where nothing should happen because it's sealed off, except someone made a firewall change and now traffic is hitting it. And we have almost a constant refrain, and it's a little bit unfair, but I'll totally take it, where customers will say, like, we spend so many million on our security products, but when we had that pen test, Canaries were the only thing that caught them. And it's like, yes, because that's what we optimize to do. But the look is just great for us because the customer's like, well, we just paid you 30K and you're the only thing that caught those attackers. And it's why we've been able to consistently grow and keep our customers.

I'm here with Lazar Nikolov, developer advocate at Sentry. Okay, let's talk about your live streams. You're constantly building something and live streaming it. Give us a peek behind the curtain. What do you do in your live streams? What constantly building something and live streaming it give us a peek behind the curtain what do you do in your live streams what can we expect if you watch yeah so at Sentry I that's even how I started I started to build a mobile application or tracking expenses in four different frameworks because I wanted to you know explore basically the DX of the four most popular frameworks SwiftUI Jetpack Compose, React Native, and Flutter. Then I moved on during October, of course, we did the Oktoberfest where we tried to fix a bug

in the React Native SDK. That was really cool. And what else? For example, right now I'm streaming on, and I'm usually streaming on YouTube. I started building a really cool project that I want to call the Errorpedia. So it's like a Wikipedia of errors. So my idea was to just build a website that enumerates famous frameworks, like used frameworks, and what errors can be encountered within those frameworks with a little bit of explanation why they happen and also how to fix them. I had an interesting choice of technology so like Astra for the whole website because of its ability to embed react components or view components or solid svelte components and these are frameworks that I want to cover the errors from so like the whole website the whole doc site would be just Astra and Markdown but when the

interactive example needs to happen I'm just going to export that from a package in the monorepo. So that was interesting. And I started building that and it put me into this mode of thinking about errors. And I was like, okay, we can do these errors and we can do these errors. I started to compile a list of errors that we can document. And I started thinking about, you know, what about errors that don't necessarily break the page? I mean, they break the page, but they don't produce errors in the console, right? There could be responsiveness errors, like mobile or tablets, something like that. Something gets pushed off the screen.

There's like an overflow hidden on the body. You can't really access that, you know? So it breaks the flow, the operation flow for the user, but it doesn't produce anything in the logs, right? Maybe there's, maybe we're talking about, I don't know, Firefox or Safari CSS issues. Cause we know, especially Safari, when you're building something for Safari, those who do front end, things usually break, but they don't produce an error. So I was thinking about that and I was like, okay, we have all the tools in Zendry. So, so yeah, that's what I'm doing right now. I'm streaming, building that widget that lets you, you know, start the recording and send it to Sendry. Okay. If you want to follow along with Lazar, you can follow him

at youtube.com slash Nikolov Lazar. We'll put the link in the show notes, but it is youtube.com slash N I K O L O V L A Z A-v-l-a-z-a-r. Lots of cool live streams, lots of cool videos. Check them out. Again, the link is in the show notes. Also, check out Sentry. You know the drill. Sentry.io slash changelawpod.

That's S-E-N-T-R-Y dot I-O slash changelawpod. And make sure you use our code change law. They get you a hundred dollars off century. That's basically the team plan for free for three months. And this is in addition to their completely free developer plan. Once again, century.io slash change law pod and use the code changelog. I think the other thing that could kill you, which it sounds like you're hyper-optimized around, is false positives. Like, that would destroy your value as well, because we've all deployed, you know, Nagios, for instance, which is not a security product, but a network monitoring product.

We've all been nagged to death by false positives and, like, throw it out the window, you know? Yeah. And so we take it, like, in the the window, you know? Yeah. And, and so we take it like, like in the company, we take it so seriously. Like we've got blog posts going way back where we'd blog about features that we've removed. So for example, on the, when a canary acts like the Cisco and you can say, enable fake SSH, enable fake telnet, enable fake Finger. We used to say enable SNMP because like everyone's got SNMP. And the number of things on your network that just randomly talk SNMP would set those things off consistently. And we could explain to people, hey, please don't

enable SNMP, but most people would, and then get that false positive. And so we disable it and remove it and say, no, from now on, you can't do this thing incorrectly. And yes, the company takes it super seriously. If like, like we say, we promise you, we're not going to be the noisy thing stealing your, your staff's time. And if we are, then we're breaking our promises and you should not renew. And so, yeah, we all react pretty quickly. It might be a naive question to ask this, but how are these attackers getting into the network? Where's the holes at? Is it social engineering? Is it bad hardware? What is it? It's, again, a super neat question. The short version of the answer is we don't have to care. And again, that's one of the benefits

of Canary is the assumption that they'll get in with whatever the attack of today is. So, they social engineered Bob and they're using his machine versus you up against Mossad and they actually in the firmware of your Yeti microphone, they popped out on your network. No way. The point is, it doesn't matter to us because now they're there and they want to do stuff. And so traditionally, security tools have tried to preempt all of these attacks. And there's always the next attack, right? They're coming in via this, they're coming in via phishing, they're coming in via a new thing. But once they're there, there's a core set of things they have to do. They have to look around for stuff. They have to grab stuff. And so while we, like,

we often say, like, we're the stupidest product on the floor at RSA. Like, we do what we say on the tin and we work. But it's that simplicity that people can then rely on. It's uncanny how genius this is, really, because what you've said, essentially, is like, to repeat your words, is you don't care how you get in the network. But there's a particular set of things that every attacker does and you bank on that happening and you watch for it and you masquerade as necessary in the network to attract essentially. And then I'm sure you log, right? Once that happens, you get that authentication, who are they trying to be, et cetera. Exactly. At that point, we'll push out, your console will get an alert. We'll push an alert to Slack, to Teams, to your SIEM, to your SMS, to your email, however you want to do that. But our pitch is one alert when it matters. You should know that that stuff's happening and you should get that clean message. But yeah, we were lucky with lots of that stuff. I think we started off thinking this

is a good idea. And as we worked on it more and more, some of those things kind of fell in our lap. But so far, it's worked really well for us. And at this point, empirically, it just works. Like other than canary.love, we get emails at least a few times a week from customers going, yep, just caught out pen testers. Or we get pings from pen testers saying, this stuff makes me sad. This stuff makes me sad. So just timing wise, because literally last week, there's an Australian podcast called Risky Business.

And the co-host is a pen tester for years and he, not in a sponsored slot, gave this whole talk about, yeah, this stuff would catch him because this is what he does and that's just how it works. So yeah, we think it's good. We see a future where everyone should be running at least some canaries on their network. Gotcha. What about since it's so set and forget it, or it can be almost forgotten until it's necessary to be remembered. You mentioned all these different ways you can alert out. Is there ever a time whenever those credentials have changed over time because of the set and forget it that your alert actually goes unheard? So when we started, like most engineers, we were like, I ended up being the non-engineer and we were just a team full of engineering.

And one of the things we realized is exactly that problem. What if somebody buys you and never installs you? What if somebody buys you and got the message saying, hey, someone just logged into SSH and logged into RDP and you never checked that message. And so now we've got a tiny customer success team who literally do not try to upsell you, but exist almost as a mini SOC. And they'll pick up an alert like that. They'll reach out to you and say, hey, this looks serious.

Do you know that this thing happened? Is someone aware of it? Are you picking up on it? And then we'll build tools on the back end as we grow to make sure that even though that team is tiny with three people, they can manage thousands of customers. But over time, we've spoken to customers who've told us, so when you buy Canaries by default, it'll email you and send you a text message. And we've got CTOs who tell us, I still get text messages from Canary because you don't spam us ever. So I think the trick there is to keep that promise that says, when we send you an alert, it probably matters. And if we can show that that's true, then people don't farm us off six levels down. What's the footprint that you can masquerade as then? Is it a pretty large

footprint and how do you keep up with masquerading well? Yeah. So when we started, when we shipped in version one, we just did three, we call them personalities. Cisco personality, actually it was a Switch, Linux box, and a Windows box. And version one just had that, and it was already useful. And today we've got dozens and dozens of them. So you can say I'm a JBoss server, I'm a Windows 2016 box, I'm a Windows XP box, I'm a MacOS machine, way down to saying like I'm SCADA equipment, I'm a Siemens PLC. And if you say you are Siemens PLC, like you can talk good Modbus. Like if somebody thought they were talking Modbus to you, you'd respond in Modbus. And part of our team, that's what we'll do.

Like we'll say, okay, we should build a Sophos server. We should build a SolarWinds admin panel. And we build those. And we have some customers like a large retailer who'll buy us and say, listen, we'd like you to look like our point of sale system so that we could do this. And we'll build those personalities for them. At this point, it's pretty easy for us because we've got this archive almost of machine parts. And when you deploy a personality, we really stress, like I think I've said it dozens of times on this call already, we really stress that you should be able to say, make this a DiskStation NAS and step away. And it does everything.

Creates the file share, creates good names for you. We use a little bit of ChatGPT with our last install where you can say, I'm in aerospace and it'll create aerospace-y files in an aerospace-y folders for you. You can say you in finance and it'll create aerospace-y files in an aerospace-y folders for you. You can say you in finance and it'll create that. And so the default should just work. But if you want to mess with it, you can say I want to run an Nginx web server, but change the header to this and upload my own certs. And actually on port 1234, I want to run my own TCP service.

When someone connects, say hello. And if they say hello back, log it. So our watchwords there have been that the default should be trivial and anything else should be possible. So people can even customize their own personalities if they wanted to. Sounds like you guys have thought it all. Certainly you didn't think of it all at the start, but I'm curious about the start because you have this perspective of the world with bootstrapping VC funding, like how to actually do this. And it sounds like you had a consultancy kind of help you bootstrap the product. Can you tell us kind of the story of how the product came together and how much effort was there up front before you started making these

amazing inbound sales? So part of it or a big part of it was informed by our previous gig, right, where we were pen testers. And so we had a really good pen testing business, like from 2002 to 2010, like I think we spoke at almost every Black Hat there was. And so again, small South African company, but we got to spread our wings internationally by doing research that could get shared like at Black Hat and DEF CON. And that also gave us a good amount of exposure. So when I left that car, so we sold that company in 2007. And more than anything, I wanted to build a company that was not tied to headcount again, because pen testing is great, but you've just based on how many hours of pen testing you can sell. And so I wanted a product company, but didn't know what the product

would be. And so the plan was that I'd speak to a few customers and build a product for them that I could then resell to other people. And we tried a few products before Canary, tried to use this to fish your company type product, which now has become a cottage industry. There's tons of people doing that type of business. And then we tried another product that didn't particularly take off. And then Canary happened almost by accident because I was trying to help a company, a really big media organization that was being hacked left, right, and center. And when I visited them, we told them, hey, you should take all the old machines that are lying around, get your intern to just put

honeypot software on it and drop these widely. It'll be good experience for the intern and you'll get insight into where your real fires are. And the next time I visited them, I said, hey, how's that thing going? Like, are we getting insights? And they hadn't gotten around to doing it. And the next time I visited them, they hadn't gotten around to doing it. And so we said, there's something here. Like, we should make this so that it's easy enough that even those people would actually do it. There's actually an interesting story with that because we drew up the specs and we started building it and I pinged, I think it was 12 of our previous customers.

So people who used us for pen testing and trusted us. We pinged 12 of them and said, listen, if we built this honeypot and made it quick to deploy, would you buy it? And from the 12 we pinged, 10 of them said, no, we can do our own honeypots. We won't buy this. And it's one of those interesting things that in retrospect sound heroic. But I thought most of them were wrong.

Because from experience, almost everyone intellectually knows honeypots are a good idea. But almost nobody uses them. Because life just happens and you don't do it. So when someone says, would you pay for this? You go, we can do that. Why do we need to? And so we bought version one anyway and there's pictures of it but like the hardware that we wrapped it in was super janky because we 3d printed the boxes and and we made 12 of them and we sent them out to

these customers some are really good names like like Unicorns are currently in the Valley. And then all of them came back and said, for 5K, we'd buy that. And from those 12, eight bought, most of them are still customers. And then what we were really lucky about is we got to grow the company and the product as sales group. And I fully admit that that stuff needs super fortuitous timing. But like the early customers who bought version one, like it had a lot of rough edges and it was still useful. And they tolerated those rough edges while we got better. And it allowed us to hire more people, get better. Like today, we've got people working for us

who are way smarter than us. And so it allows us to start tackling hairier problems that we didn't have the bandwidth to tackle initially. But I think there's an important lesson that lots of founders get wrong. And that's that you almost need to earn the right to work on the nicer problems. Like initially, you've got to work on some problems that seem pretty mundane,

but you've got to get it across the line for the customer. And if you solve those, and if they buy you, you get to solve other more interesting problems. And so far, we've managed to keep that balance right. And it's worked well. It's interesting that you had that experience in most startup or indie hacker threads that I read about people trying to do lean startup kind of things where you're asking people, would you buy this? Or you're setting up a fake page that they would sign up for. The signal is usually the opposite. Like they would say we would buy it. And especially if they know you, they'll say they'll buy it because they want to support you.

But then you go build the thing and then it finds out, actually, no. When it comes time to swipe the credit card, they won't buy it. And yours is like the opposite. They said no, but then they bought. It's interesting. Yeah. So I'll tell you that that still becomes a problem because after a little while, like in year one, we did a few thousands of sales, but I was horribly terrified that people were only buying because they liked me.

We were pen testers for a long time and researchers for a long time and we had a good reputation. And initially our price tag was 5K and almost anyone, they can find 5k. And so I was really worried in year one that people were just buying because they liked me or liked us and that the product wouldn't stand the test. Like would they renew? And I think one of the things that served us really well and continues to serve us is an almost never ending paranoia about like, are we doing enough to justify that people are actually paying us? And it might just be because we were so surprised that anyone would pay us. But like to a person in the company, we still react with our hair on fire when we drop a ball. It's like they paying us all this money and we

did that, like that just can't be right. And yeah, I think it creates the right type of panic. I think I know lots of people hate it because they remember with fondness a time when they used to buy their software outright, but we charge every year, right? So every year people pay us the same amount. And I think in some ways that creates a really strong positive incentive for the vendor to keep doing their job. Because if we don't show value, then people don't renew. And so we don't get to sit on our laurels because next year we just won't make that money. And so it kind of forces us to make sure we're still keeping our promises and still adding value. Yeah. It must be really hard to resist the urge to add big new features when you have that annual

contract, right? Like the adding value, usually you want to add some value, not just continue to produce the same amount of value. It's interesting. Early on, it was harder. So we're adding stuff all the time, right? Like I think like most software or startup tales, there's, I think what's called the genius of the end, as opposed to the tyranny of the all. Like you've still got to be adding stuff because there's more value to be gotten. But early on, you certainly, in your early days, if you have a strong opinion on not throwing the kitchen sink at the product,

you could be confused as just being lazy, right? Like you just haven't, you're not feature complete. Like don't tell me you're minimalist. Like you just don't got stuff. And so early on, there were like five or six funded companies that started in the same space as we were. And I was worried about them, right? Like all of them raised $30 million. And typically what most people like that do is they just pile on named

features like we support this and we support that and we support xml and we support like pick a standard and they make sure that they've got that uh logo on their site and we'd get people then saying like hey do you cover this like do you follow this taxonomy? And we were like, no, we don't think that's useful and here's why. And interestingly today, almost all those folks have exited. They either, they managed to raise another round, they went to 60 million and then they either pivoted out or they folded. And as time went on and as more people start taking us seriously, you get a little more credibility to be able to say like, I know this thing's popular, but we don't think that's the right way to do it. If you want to do it, like there's other things you can do, but here's why

we do what we do. And I think you try to do the right thing and you sweat all those details and sometimes you'll get it wrong and then you've got to figure it out and put it into the product. But mostly we've been pretty good with those calls. And I'll tell you a stupid thing that we blogged about a little while back. But at some point, like most companies, we went for a visual refresh. So we wanted to update our front-end JS and we went for okay. When we built version one, like our graphic skills were terrible and now we better, let's make V2 pretty. And we worked on it forever. And we trialed it with our first customers and they liked it. And literally, the week before we released, so we use FreshBooks or we were using FreshBooks internally.

FreshBooks mailed us to say they've gone through a front end change. And our reaction was, damn it. Because like, I don't want to learn FreshBooks new front end. Like I want it to be the same as it was because like FreshBooks is not my life. I just want them to do stuff. And we had this discussion internally that said like, are we doing that to everyone who's just been using us for three years? Because almost of a type of vanity, like we want this new thing. They just want to forget about us. And so we scrapped that whole thing. We still did a look change, but we made sure it was super close to the last thing. We didn't break away from usage patterns that people had.

We gave people a way to slowly go through it. So we try to be thoughtful about that sort of stuff like to add new things but not gratuitously yeah adam he's speaking directly to you here i'm over there nodding because uh yeah we use fresh books and similar like i've been using fresh books for pretty much ever right just forever so long that like it's like version one interface for me right and when they told us that i i punted so long to the point where they're like, you have to move to this new thing. Because like, we're just done maintaining the old thing. And I'm like, I don't even, I fought with them on the phone, basically. I'm like.

So it's so interesting, right? Because like, in that story, you see exactly that thing. Like, everyone knows this experience of using something and really not wanting it to upgrade. But everyone thinks that products should keep upgrading to stay fresh. When like realistically, most users are like, listen, I don't want that. I want you to just work. And there is a sweet spot where you can add functionality and add stuff if you're mindful of i'm giving you new potential tools without changing the way you do stuff and they keep making it more expensive too like by doing these things like there's just so many things in fresh books that we don't even use and i'm like yeah you're the best at this one thing we really need and everybody else pretty much sucks in comparison

which is the good thing about fresh books but everything else they offer i'm like i don't want it nor do we need it and so we have to pay way more than I think we ever should have to for what FreshBooks gives us. And I love their software. I'm happy to pay for good software. I'm not being cheap by any means, but we're like, it's more expensive than necessary because they keep layering on these features. I am so super with you. And I'll tell you again, just because this plays perfectly into our thing. So we've been running Canary now for eight years and we've never increased our prices ever. A big part of that comes from the same thing.

Like we picked a fair price when we started and the company is profitable and we're doing well and we don't have to send a price increase all the time. And we'll often have people say, like those Canary tokens that we give out, like literally millions of people use them. And you almost never talk to a VC or finance person, I'll say, but there's so much value there.

Like, why are you leaving that money on the table? As if leaving money on the table is a horrible thing to do. Like for us, it's like, well, we're doing really well with our other product. And this stuff gets to help people who don't pay us. And we get goodwill and we get people become aware of us. If we're recruiting, I get to say to a young student, you can go write bank interfaces for First National Bank, or you can work on us on tokens, which just got 3 million users in December. Like it's immediately attractive to them. And so we get all these benefits, but we don't have to extract every dollar from every customer.

And there's an amount of user hostility that we've come to tolerate from lots of our products. And we just don't think that has to be. And again, we're not complete hippies. I want canaries everywhere. I think they're useful. I want to beat all the other VC-backed companies because I think our products are better. I just don't think it has to be done at users' expense. That's good thoughts. That's wise. I mean, most people would think that you should, as you've said, extract every dollar from the customer, not because you're greedy, but because that's what capitalism does. It's what a business does. Businesses are meant to

make money. So why would you leave money on the table when it can be made and used and invested to build out your business and do more things? But that's kind of like, that's FreshBooks-ing it. I mean, but you have that luxury, right? Because you don't have anybody to answer to, do you? So I think that's certainly a part of it. I think when you've taken investments and particularly VC investments, there's a growth rate. And what's interesting is like we've shown good growth. Like we get VCs

pinging us all the time because we track really well as a VC backed company. We just haven't done it with VC money. And again, I think what's interesting is when we started, we made lots of these choices because that's the sort of company we'd like to interact with. But today, they also just make good business sense. Like when COVID hit, I was terrified because we'd see headlines of our customers laying off huge numbers of their security staff. And I was worried like sooner or later, that's got to cut into our sales, right? Like how can they, they just laid off 60% of their staff and they'd renew us at full at all their canaries again. And many of them told us, they're like, hey, listen, like you folks are so affordable. We're not throwing this out.

Like if we've got one security person, he's managing the canaries that are there. And so in part, us not being crazy expensive and making sure we always add value meant that when people were doing cuts, we just weren't the first thing that got cut. It just worked for us. And then to crazy extremes, like again, during COVID, we had a handful of customers ping us and say, listen, like we're on the verge of going out of business.

We love you guys, but we can't keep this. And for lots of them, we said, okay, we'll stick around with you. Let's chat again in a year. And most of them who survived came back a year later and said, hey, we back. We'll pick the subscription up. It's all good. And for the most part, they're customers for life now. They're like, that was great.

And again, for us, it's not crazy altruism. It just makes good business sense. Like those people really unkind. They don't make kind decisions. They don't have grace and forgiveness and scenarios like that where just treat people with kindness. Sometimes that doesn't go very far, though, because you might give somebody kindness and then you get abused. And I get that. I totally get both sides of the equation. But Jared and I are the same way we operate this business, Change All Media.

We're so kind with folks and we're so forgiving and we love the relational aspect of every brand we get to work with and the ones that aren't in that relational aspect just don't stick around long because it's just not how our dna is operated it's too transactional yeah it's just too transactional for how we operate as a business and we're here for the long term in the trenches to help not just our brands we work with, but the people listening to the show right now. You know, we vet everything like that and we care. Right. And sometimes we get a short end of the stick because of that. But more often than not, it works out, you know.

I think so. I think in the fullness of time, that's how, and look again, like one of the things I often say is the thing that we've been most lucky with, and we've been lucky with lots and lots of things, but probably the biggest is from my previous company to this one, we did things our way and the market rewarded us. And I know lots of really good people who've done the right things and the market kicked them in the teeth. And so they end up learning, it doesn't matter what you do, like the market's going to kick you in the teeth. And we've just been lucky because we were like, at my previous company, it was like, if we work really hard on this research,

we'll get to talk at Black Hat. And we did. And if we talk at Black Hat consistently, we'll get to talk at black hat and we did and if we talk at black hat consistently we'll become international trainers and we did and and with uh things it was like if we truly add value like people will appreciate it we've dropped balls right like early on i remember with our early deployments we'd have canaries deployed in the wild and canaries were dying. Like this was in year two and we were using SD cards for disk storage and it turned out that our SD cards had a fault in them. So like 200 canaries in the wild died. Like at that point, it's like the worst thing ever. Like people trusted us, they bought. And suddenly, there's no disc on them. And at that point, we just worked like hell.

We got new units out. We made sure that would never happen again. And we said to those customers, this is what happened. This is how we're making sure it'll never happen again. Thanks for trusting us. And they did. And we got past that. And yeah, I think there's room for kindness.

And if you're lucky, and it sounds like you folks have, you get to build an org where the org then holds you to that. So that's what people who join us now sign up for. They want to work in an environment like that. For engineers, what's really important to us is the craft of what we build. We want to build stuff we're proud of. We want to build stuff that customers really like. And so instead of building a company that's trying to grab every dollar and optimize for everything we can grab, we optimize for, can we really nail this problem? Like, can we do this thing so nicely that everyone goes that smart? And so then we start to attract those sorts of people. And hopefully that becomes your flywheel and you just get more and more of those people. And so

far it's working well for us. On the dead canary front, a couple of thoughts. The first one is, it seems like your move away from hardware and towards software makes that less of an issue. So interestingly, hardware canaries still sell really well. People like a device. Yeah. And there are lots of places where the device still just makes sense. So the one example that I mentioned earlier, people doing an acquisition and they just say, look, we're not going to get to taking in that network for another six months. But today we can just have you ship five canaries there. Just ship the hardware. Someone there will plug them in and they'll work.

So for that, to the other part of that question, we certainly had to learn lots of stuff along the way. So supply chain stuff that we hadn't gotten a hold of, shipping hardware. Version one of the hardware was truly ugly, like even past the 3D printed one. And there's this really good, Bunny Huam was this crazy hardware genius. He had a blog post at some point that said, like a message to all startups, you are not Apple. And the thing is, like when you're making hardware, everyone knows how pretty Apple devices are. I know how pretty my iPhone is. I don't want to ship something that looks junk. But you haven't earned the right to make those beautiful devices yet.

Like we had to sell our first few hundred of these ugly things and we had to make sure that it was functional enough, that it was still useful enough to add value. And today I love Alcuner, like they're beautiful, they're well-designed. We just changed the boxes that they ship in. And again, we've spent crazy amounts of time making sure that they're a lovely experience for people opening them. But again, I think, yeah, it's a tough line of having to earn that right as you go, if you bootstrap. I think if you raise a bunch of money, then you can aim at lots of that stuff on day one. But I think that brings a whole class of problems for people too. When you go to the generated quote, though, it says five beautiful thinkists. Gosh. Thinks.

I'm so sorry. No worries. It's just stuck in my brain. I'm the fool here, okay? I'm going to admit that. These five beautiful devices, you're saying, based on your question, Jared, they're not a hardware company? Or you are hardware?

You're just hardware because you have to be. Well, he was just saying that they have software canaries now that they can deploy, which I think would be a lot easier to deploy in terms of just logistics. Well, the simplicity really is like you ship them the device. Let me assume how you would deploy this thing. You plug it in, literally, into the wall to power it up, and then

you put an Ethernet cable into it. Hopefully that goes back to the switch somewhere. It DHCPs back to the primary. It gets an IP address, and you have a console that manages it that's it oh yep yeah in fact for just for the geeky listeners i'll tell you a little more so when you plug it in so it's cryptographically paired with your console up in aws so your console is tied to that one or your canary is tied to that one, or your Kinect is tied to that. And when it boots, all communication actually happens over DNS. And so if you take it and plug it into some network, as long as it can resolve DNS on that network, so not even port 53 going outbound, like it can talk

to your internal DNS server, it will get a message out to the console via DNS saying, I'm now awake. Do you want to give me a new profile? And then on the console, you can say, yes, I want you to be a Cisco router. It'll get that message when an alert happens. And again, we've built this whole communication channel on top of encrypted DNS, which is something that most users never think about. But the reason we've done that is if someone's plugging these canaries in on a complex network, we don't now want them to have to open holes in networks and firewalls so that these things can communicate. You plug it in on your network. If it can talk DNS, it just works. How do you get all that done via DNS?

It's pretty cool. And there's a funnier story to it. In about 2007, one of the talks that we did at Black Hat was on a tool we built that allowed you to steal information via SQL injection. So SQL injection attacks, I'm sure most of your listeners have heard of. And so we built this tool where as long as you could get SQL injection going, this tool would allow you to pull data through easily. And it could do it just via SQL injection attack that just had timing attacks or DNS attacks or all of that stuff. And so when we built version one of Canary, the first network that we took it to do a test on, in fact, the network where I'd asked the intern to build honeypots and they never did, I went there to tell them, hey, try this. And you see the problem. It's, hey, this is not going to

get out. They're going to have to talk to networking to allow this to get out. And so I went back and said, now what we're going to do is take our DNS channel from that research talk that we did. And we're going to make Canaries communicate with the console via that DNS channel. And so we've hyper-optimized that to the point where, remember our promises, you buy these and forget about them. And we put out new hardware versions almost four times a year. And so if you buy Canary in year one, and you've just got it sitting in some basement somewhere,

today it's running Canary Current, and it's pulling those updates just via DNS. And you never have to think about it. You just get an email saying your Canary can now do these things also. It's fine if you leave it, it's still running what you had it running, but you've now got the capability to do these other things. And all of that is just via DNS and customers never have to think about it. I don't know the DNS protocol very well. I know it's UDP, so it's not, it's stateless, but like, how is it just like you can open up a DNS and you can just like send stuff over port 53 or whatever it is.

So the easiest or shortest way, and obviously it needs a whole bunch of optimization, but in the easiest version, you think about it as you are a canary. And we'd now tell you, okay, you should go get your update. And you'd send a request going, hey, update.myhash.canary.tools. And I'd then respond to you with, okay, the answer is hash.hash.hash.hash. Ask me again. And you'd go, okay, the answer is hash dot hash dot hash dot hash, ask me again. And you'd go, okay, ask me again. I'd go hash dot hash dot hash dot hash. Okay.

And on your end, you'd assemble all of that and say, okay, it's our thing. So it sounds really slow. So interestingly, for a canary to give an alert, it's really tiny, right? You get to push that alert, you get that. But if you were doing an update of the sort that we would do four times a year, you'd basically get a message saying, hey, your canary is updating. If you looked at it, it would run for about a day and then it would be updated. And like even there, I think the benefit of being practitioners, and again, I think of really caring comes in. So with version one or the first few versions, we'd have like everyone else, your canary is now 3.2.x or 2.9. And at some point we're like, listen, users don't care. Like if you were a user, you're either up to date or you're not. And so that's what our version numbers according to customers now is. Like your canary

either says it's up to date or it's not. And if it's not, you hit a little button and it'll request an update and it'll come down. But other than that, customers shouldn't have to care. Like I don't care what version of Chrome I'm running. I just want to know that I'm not running something old. That's really cool. So all communications from the Canary go over DNS. Over encrypted DNS. Go to your console that way.

Yep. And at this point, like, we've been doing it eight years. So it's like, it absolutely works. Like, we've hit every edge case. We've fixed it. We've pushed binary updates multiple times to thousands of devices. Empirically, like it just works. This is a changangelog Newsbreak.

Large language models are multiplying like wet gremlins. The question is, which one or ones should you use? Vince Lewitt wondered the same, so he built a thing. Quote, I had the idea of writing a script that asks prompts testing basic reasoning, instruction following, and creativity on around 60 models that I could get my hands on through inference APIs. The script stored all the answers in a SQLite database and here are the raw results. On the linked website, you can view all the answers from a specific model or all the models

answering a specific prompt. Here is one such prompt. Argue for and against the use of Kubernetes in the style of a haiku. My takeaway is that most models don't understand the rules of a haiku. Example, CodeLlama says, Kubernetes is a tool, a tool to manage your containers, but is it worth it? A profound poem, but a haiku it is not. Don't quit your day job. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our

free companion email with even more developer news worth your attention. Once again, that's changelog.com slash news. tell me about your hardware then so like you got to care about hardware at some point because it does look good the one i see it looks nice now so version one 3d printed and later version one not 3d printed what's it like now are you do you care i suppose deeply about the hardware yeah we do so like over time initially with version one almost all our design stuff was done by me and i'm a technical hacker like like i'm not the best design person you'd get. And so I used to do all early stuff in OmniGraph or work with an external designer. And now we've got a great designer on the team from Canada and he lives and breathes the stuff.

And so he's doing pretty stuff all the time. We've almost got to hold him back just with like, no, we're not going to spend time on this. We're going to spend time on that. And again, for me, that falls into one of those categories of earning the right to do cooler stuff as time goes on. What is the hardware? Is it, I mean, it seems Raspberry Pi-esque, like at least from a footprint. What's the actual hardware built on? Yeah. And so in there, we've got a tiny little daughter board that we manufacture here in South Africa. You can swap out with any number of small factor machines inside. But again, and it's something Jared said that was interesting. If we do a Hyper-V version,

a VMware version or hardware version, we charge exactly the same for all of them. And so our pitch is that's not something customers should ever have to care about. So with most of these, they'd be running the equivalent of Pi 4s with a small daughterboard in there that we have that drives that little button that you see that drives some of our other stuff. But again, fundamentally, it's pretty simple. So is it built on the Raspberry Pi 4 then or is it something? Yeah, the current versions are. Okay. So a daughterboard on the Raspberry Pi 4. Yep. Yep. Powered via plug into the wall, not PoE, right? Yep. Yep. Not PoE.

And then all you've got is a barrel port plug, it seems, based on pictures, and then a single LAN port. Yep. Exactly right. Okay. A reset button. So, yeah. So, the little LED that you see is actually also its button.

Okay. So, you can boot and hold on that button, which would put it into configuration mode. Right. And fundamentally, we want that to be dead simple. So there's nothing you can do on it that's wrong. You can hold on that button to reboot it. And the way we run the service is if your hardware device, if you run over one with a truck, you mail support and we'll just send you another one.

Like the point is that you should never have to think about it. And so you just get another device and it just magically shows up. So when you were alluding to the hardware supply chain challenges from before, obviously like the rest of the world, Raspberry Pis in the last couple of years have been like in high demand. So it's interesting. It's like the SD card issue was one that took us by surprise, more even than Pis in demand. Like we didn't realize the difference.

So essentially what happened is SanDisk had a speed wobble at some point, which I guess normal people don't have to care about. And so SD cards were in short supply. And we went out and bought a whole bunch of SD cards from wherever we could to show up our supply. And it turns out there's just tons and tons of fake SD cards on the market. So they're in SanDisk packages, but just poor quality SD cards. And when we realized that we did a bunch of testing, because you can get SD cards for $20 ranging up to $200. And we were like, well, okay, if the $200 one is going to stop us having this problem, let's find out and we can plan around it.

It turns out you just need a legit good quality SD cards, but you can track them. You can put in quality control to make sure the batches you buy aren't going to fail after 300 reads or 300 writes. But again, that's the sort of stuff we had to figure out as we went. So all of your hardware is powered by then an SD card, not the optional- NVRAM. Yeah. Yeah. Exactly.

And so what you end up doing a lot, and again, it's something that you don't know early on, is you start building fail-safes that you can in software. So wherever you can for that stuff, you'll start adding watchdogs, you'll start adding more robustness. And because we've got a communication channel between the client and the server all the time, we can start having the client say, hey, send me my config again, I'm in trouble, that sort of stuff. So yeah, you end up building robustness in software. So are you building your stack on top of the Raspberry Pi OS or is it a different image?

Give me from the hardware up, what do you do? Yeah, so we've got to have our own custom kernel because we're doing packet mangling. So we need to be able to fake out that our operating system is actually Cisco iOS. And so we have our own hardened image that goes on there that we'll customize, that we'll maintain, and we'll maintain that internally. And fundamentally, we then run a master service that runs all of the fake services that the service claims to run. So you have a hardened base to make sure that we don't get caught out that way.

And then we have a system that fakes out the rest of the services fundamentally. And then you've got to have a component that's communicating with the console. You piece those together. And then the console becomes its own software, right? Because that's got to handle alerts and integrations and all of that stuff. But those become the two big pieces of it. So now that I know more about your hardware and your software, let me suggest an attack.

Sure. Let me hypothetical with you. And you tell me how your system would react. The attack is an inside job. I work within. I know that we run Canary. I know where they're all at. Right.

And I either go and plug them or I decide to pull out your disks. And then I submit my attack because now I know that the guards are not there and I can go and I'm part of the security team. Or maybe I know the security team. My friend, I'm in finance and my friend is in security and he has a loud mouth and we drink a lot together. Whatever it might be. I now know how to locate the canaries, either dismantle them by either pulling out the SD card, because maybe it's accessible, maybe it isn't via the hardware. Maybe it's inside the actual shell and I got to unscrew it

or unplug it. So I take down all the canaries. What happens? So if a canary is down by default for eight minutes, but it's configurable, it'll reach out to you and tell you, hey, listen, I've just been turned off and that wasn't part of your plan. And so in some instances, you'll be like, okay, that's because that section has just powered down. We know it. In fact, someone tweeted yesterday that it's the best quality indicator of when your network is down because you'll get

an SMS saying Canary 52 is now down. But the simple thing there is a Canary going offline is a surprise. And so you'll get an alert telling you this Canary that should have been up isn't reporting in anymore. You should go figure out why. But if you know where all the Canaries are, you're just not going to touch them. Yeah. Like they're canaries. You're going to, inside jobs, I mean, it's like having physical access. But I'm in finance in this scenario.

I'm in finance. Okay. I don't know what they're configured as. I'm just saying, I'm not, let's try to throw a. I'll tell you two interesting versions of that. The one is, and I feel strongly on this, is one of the original sins of the security industry is them promising too much and trying to be too much. And sometimes people need to be able to say,

yeah, we don't do that. Like, yeah, we wouldn't catch that. Like, if you know where all the canaries are and you don't do that. Like, yeah, we wouldn't catch that. Like, if you know where all the canaries are and you don't touch them, like, you know where all the tripwires are. Like, that stuff's not going to catch you. And I think people should be okay with saying that. And our pitch to try to mitigate against that is that we want to make things that are easy enough to deploy that a person can deploy it without letting the whole company know, hey, here's what I'm doing. We're doing this Canary rollout.

Like literally, go plug it in. Forget about it. It's in that corner. It doesn't need huge shenanigans. And Canary tokens add trickiness just because they could be anywhere and they could be... We've given a few talks on Canary tokens because some of them are really dependent on how tricky the security team wants to be. So some of them are obvious, like that AWS API key that I mentioned. But we've got another one, for example, that's a wire guard, a legit wire guard endpoint. And so you take your CEO's phone and you add a wire guard tunnel on his phone

that says secret exec network 123, and you forget about it. And what you're waiting for is when he gets his phone compromised, when he's going through customs into China, when that phone gets grabbed, it's the sort of thing that an attacker who you're interested in looks at it and goes, I see, I'll use this endpoint. I'll check what this is. And our pitch is, if we can make those things easy enough to do, then security teams can do them.

And so if you take, like I know lots of vendors use it, but if you take people having their SolarWinds moment where attackers have compromised a build server deep in a network, and the only time they find out about it is after the attackers use the Bolt server to build new software that's been deployed to all of their customers. That sort of attacker who finds AWS credentials on that machine has to try to use them

because maybe that's SolarWinds cloud environment. Or if they find a VPN endpoint on that machine, they've got to see what's at the other end of it, which means in week one, you get notified that a machine nobody should be touching is doing strange things instead of waiting till you read about it on CNN. And mostly that's our pitch is do this now, forget about it, it'll be good for you.

So as you look at building, maybe not more features on Canary, but new products or services, you know, one that makes sense, I think, as a follow-up is like mitigation, right? So now we know there's a problem. Well, our good friends at Thinkst let us know, maybe they can help us fix it. Now what? Yeah, exactly. Now what? I'm sure that's crossed your mind. You're nodding on your head, so you've thought of this. As a service, it's something that we've stayed away from largely. And there are people who roll incident response, right? Like anyone gets in trouble in the world and they call in Mandiant or they call in a bunch of folks like that.

And part of our pitch has been, again, we want to do one thing really well and we'll partner with those folks. So we have bunches of MSSPs, managed service providers, who will take Canary, deploy them at their customers, and have those alerts go to them. And so what they're getting is they're already trying to manage all of these customers. They deploy those Canaries at those customers. If something happens on those networks, they get the alerts. They then react to those customers. And for us, it's a good deal.

Like there's at least a few MSSPs in the US who have Canaries deployed at every one of their customers. And for us, that's perfect. We'll keep making this thing that works well for you. You keep offering that service and everyone's better off for it. Keep it simple, keep it focused. Have you ever had the bug?

I'm sure you've had lots of people walk up to you with large checks. Have you ever thought, maybe we should do this one? Maybe we should take some funding and do something bigger. That's a super good question and super insightful.

We have. So we have conversations with lots of VCs who ping us periodically. And in 2019, one of those VCs, like probably one of the best named VCs in the world pinged to say like, hey, would you do breakfast with this named partner? And I was like, of course, I'll do breakfast with that named partner. Like, are you kidding? And it was great. Like they say, don't meet your heroes, but he is every bit as amazing as every talk of his that I heard. And I came back to South Africa and they phoned me and said, hey, would I come up and meet the other named partner?

And I did. And they did this thing that said, hey, here's why you should take money from us. And we flirted a little. And my take was, listen, I've got money in the bank. It's our own money. And I'm really worried that this is how focused product companies lose their focus, like with this stuff.

And they said, look, like we won't take a board seat. We'll give you all this money. We'll tell people why you're great. And we were super tempted. And again, because it's super flattering, right? Like I've read about those dudes forever and they think I'm cool and they think our company is cool and they throw barbecues that Obama attends. Like that stuff is flattering as hell. And we flirted with them for about a year and

decided not to. And I'm still good friends with them. I am officially a scout for them, which means I can invest some of their money in small startups. But we figured we didn't need it. And yeah, I think we're better off for not having taken them. I'll still listen to everything they say and read everything they write. I just don't think that the business needed them. And yeah, at this point, I think VC, almost as a segue, I think that the VC model isn't super well suited to building good security companies. I think there are some companies that, I think if you're trying to build the next social media powerhouse, you should raise VC. And it works well for VCs because they'll give money to a bunch of people. And as they see which one makes it through, they can give more money to that one, and then the winner will take it all. But I think in security, there's a side problem that makes that harder. And I think that the VC

model kind of muddies the water. And I wish more founders knew that it wasn't a law of physics, that you absolutely had to do it the VC way. Well, it makes me think back to years and years ago when 37signals was just starting to take off. And famously, Jason Fried and David Hanmeier Hansen took investment from Jeff Bezos. And it was more like, it sounds like what you were being offered. Like it wasn't like board seat control, blah, blah, blah. It was more like, here's some money. We would like to be a part of this. And their stance back then was we took some money off the table or something like this. Like we, we didn't need to, they didn't need to either according to them. And so that was probably, and that seemed like, I don't know the history of that. Did they buy that back from

him or is it just the case? But it seems like you could have done that, you know, taken, had your FU money and then just continued along your way. So it's interesting for multiple reasons. One is like we based a lot of our stuff, including lots of our company thinking on the 37signals books early on, like opinionated software, all of that stuff, lots of it was informed by early 37 Signal Thinking. And look, for us, it sounds like a terrible thing to say, but we make good money now. For the first few years, we've got a few million in the bank and it's not by an island money, but we keep growing, we're doing well. We can pay dividends at this point. We pay the company good bonuses based on that sort of stuff. And so again, I think that lots of people have a pretty static view on that path to generating wealth. And it's largely because VCs were the ones talking about how to build companies. And so I think lots of the literature out there was on doing things the raise a seed round,

raise your next round, keep doing that way. And again, I wouldn't begrudge it because I think that's perfectly fine. But the biggest problem I have with that stuff is that it's super distracting and almost runs in a completely different direction from founders focusing on products. And I think like ages ago, Paul Graham had this essay where he spoke about the top idea in your mind. And you'll see how often these days, founders who are on that raise money, VC route, hamster wheel, that's the top idea in their mind. It's how do I raise the next round? How do I talk to analysts so I look good so I raise the next round? How do I talk to VCs? Which means almost by definition, the top idea in their mind is not their product.

And yeah, I think we are all poorer for that. And I'm surprised that it's so acceptable. And I know it's terrible because every founder in the world secretly thinks he's Steve Jobs. super appreciate about Apple today is that we get to see a multi-trillion dollar company where they care deeply about the product. And like one of the jokes, as Doug Song used to mention, like find CEOs of companies who can demo their product. In the security world, it's shockingly rare. Like when you had Symantec and McAfee what matters. And so the people in the company are not then optimizing around building the best product they can. They're optimizing around acquisitions, mergers, capital allocation, sales stuff. And I think there just needs to be more for focus on the product. What you're describing is being grounded, right? Like if you can demo the product, you're kind of grounded in what you're producing.

You're grounded in the value that your employees create, that the things you do are delivering to the market to create that value and to receive cash value back from that value being executed and delivered it's a grounding in in your purpose your company's purpose not chasing the money to a degree or schmoozing with you know networks and whatnot to get more capital just for capital's sake yeah and again i'm pretty convinced it's also the path to winning like like i'm'm pretty convinced. And again, we've been lucky so far. But like, I think the market rewards that stuff. Like, you end up making a good product and the market hopefully rewards that stuff. As a really silly aside, like, we just did the Black Hat conference where we had a booth. And one of the things I was talking to someone about, which was super interesting, is when we do a booth,

like we've got this really long blog post out on doing booths and why we think it's actually good for people. Like young me hates it, but booths are super good for us. Like we do a booth at RSA, we do a booth at Black Hat, we get to meet all our customers. They come by and chat to us. People we've never seen before come and say nice things about our products. Other people hear them.

But at Black Hat this year, something that occurred to me is we have this booth and I'm there for the full two days. And Marco, who's our CTO, is there. And Bradley, who's one of our other founders. So literally all of our original founders are there, plus some of our engineers. And so for two days, people are rolling up to us, talking to us, hey, I've been using you for six years. Hey, I did this. But that's surprisingly rare on the showroom floor. Because on the showroom floor, what lots of people have done is they've paid a whole bunch of young interns or a whole bunch of college students to say, scan as many badges as you can. You scan the badge, you then get to spam all these people

trying to sell stuff. And again, it's horribly mixed incentives. Like for us, the thing is, we get to meet our customers, we get to do demos with new people who might be interested in the product. And it's so counterintuitive because if you talk to any VC, one of the playbooks that they'll tell you is the truth is not in your building, like go out and meet customers. If I told you, you want a showroom floor and you're going to meet 20,000 of your customers in two days, why wouldn't that advice mean that every CEO, CTO and chief product officer is the person on that floor? You're going to meet 20,000 of your customers or potential customers. You can talk to them about the product. But it's just not done.

Because fundamentally, what the execs are doing is they're sitting in a suite somewhere trying to arrange their next raise or trying to talk to analysts or trying to talk to the media. And again, the state of the products in our industry are a reflection of that. Like mostly we build terrible products because people just don't care enough about them. Do you think that that's unique to InfoSec? You mentioned that you think it's particularly a problem in InfoSec, but it seems like that would be more broad sweeping perhaps. It's a great question.

I think in other verticals, and I can't speak for all of them, but I think in some verticals, the vertical itself keeps you honest. I think if you have five competing social media companies, the ones that suck are going to fall away and the ones that people use get traction. And the thing that InfoSec has that's unique there is it's really hard for most customers to tell the difference between good products and bad products. And instead, what they then use as a proxy for judgment is funding. So companies say, we funded by big name X. And customers then say, well, you must be okay because you just got funded. And you'll see it. If you track the industry, you'll see how many of the press releases are just new funding round. Here's what we did. We just secured a new funding

round. It's like, tell us you've got new customers. Tell us you've solved a problem. Don't tell us that the people who gave you money before gave you money again. And mainly what happens with that is that becomes a proxy for quality. Customers then buy it. Investors then say, well, you've got all these customers. I should invest in you for another round. And what it does is it means that bad products last longer than they should, which is also not great for VCs because it now takes them longer to figure out that they've backed a product that isn't sustainable. And that's why I think that stuff is bad.

I think focusing on the product is a quicker route to honesty, because unless you make something people I could work at one of these places or whatever. And the vibe I got in general was like lots of snake oil here, like lots of just like sales going on, but not much substance. And I didn't really like that feel. And so I kind of left the community, so to speak, and went into web development. But it's largely still that. And there are a few companies now,

you're starting to get more practitioner-led companies. And I think one of the big things that certainly we are a beneficiary of is that, I'm guessing, 15 years ago, like, even if you made a great product, you couldn't sell it. Like, you still needed the traditional coin-operated sales team that went out and strippers and stakes and all of that stuff. But today, like with Slack and GitHub and Box and the empowerment that engineers have, like you don't need that stuff. Literally, we've like we cleared 16 million in ARR without an external sales team because people will try you and engineers will try you again and then they'll pull you into the org. And so I think there's never been a better time for developers, for engineers who've been through the idea maze to build their products and give it a shot like it's possible now like it's as

good a time as any to throw your hat in the ring well i think that's a great point to end on adam do you have anything else you wanted to ask arun before we let him go one more question just uh waiting for the plus plus so oh we're saving it for our oh yeah our plus plus people who and these are our insiders changelog plus plus our paid supporters so we'll we'll sit we'll save that for the post show for now we'll just say man thanks for sitting down with us thanks for sharing what y'all are up to your design decisions your extreme focus and your willingness to turn down large bags of money because you already have enough bags of money and you're doing just fine and you're staying product focused. That requires discipline and that's pretty cool, pretty unique out there.

Thanks for having me. Happy to hear about it. It was awesome. Thank you for coming. Thanks, folks. This show is exactly why you subscribe to this podcast. Where else are you going to get such depth and variety? I mean, network security, canaries in the coal mine, literal hardware canaries. You can deploy canary tokens.

Such cool tech, such cool people. And you didn't get to see us, but Jared and I were head nodding quite a bit during this conversation because Haru just laid down some wisdom, some good principles. And the way that they run things is just admirable, just very admirable. We were head nodding. We loved it. And we hope you love it, too. Coming up next week, we have, of course, Monday news. Next Wednesday, we are talking about open source matters, literally open source matters and open source matters with Steven O'Grady from Red Monk. And coming up next Friday on

Change Logging Friends, we are joined by Nick Nisi talking about browsers. And we get deep into many rabbit holes, some related to browsers and some not related to browsers. But that is friends. Come back next week and we'll have that for you. If you're not a Plus Plus subscriber, the best time to do so is right now. changelog.com slash plus plus. Get that awesome extra content we just teased at the end of the show. And we love you because, hey, that's direct support to us.

No ads for you. And you're getting closer to the show. And we love you because, hey, that's direct support to us. No ads for you. And you're getting closer to the metal, the cool, cool change all metal. Well, that's it. This show's done. We will see you next week. Game on.