The Changelog: Software Development, Open Source - The insider perspective on the event-stream compromise (Interview)

Episode Date: December 5, 2018

Adam and Jerod talk with Dominic Tarr, creator of event-stream, the IO library that made recent news as the latest malicious package in the npm registry. event-stream was turned malware, designed to target a very specific development environment and harvest account details and private keys from Bitcoin accounts. They talk through Dominic’s backstory as a prolific contributor to open source, his stance on this package, his work in open source, the sequence of events around the hack, how we can and should handle maintainer-ship of open source infrastructure over the full life-cycle of the code’s usefulness, and what some best practices are for moving forward from this kind of attack.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at rollbar.com and we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by our friends at Rollbar. Check them out at rollbar.com slash changelog. Move fast and fix things like we do here at Changelog. Catch your errors before your users do with Rollbar.com slash changelog. Move fast and fix things like we do here at changelog. Catch your errors before your users do with rollbar. If you're not using rollbar yet or you haven't tried it yet, they have a special offer for you.

Go to rollbar.com slash changelog. Sign up and integrate rollbar to get $100 to donate to open source projects via Open Collective. Once again, rollbar.com slash changelog. Welcome back. You're listening to the Changelog, a podcast featuring the hackers, the leaders, and the innovators of software development. I'm Adam Stachowiak, Editor-in-Chief here at Changelog. Today, Jared and I are talking to Dominic Tarr, creator of EventStream, the I.O. library that

made recent news as the latest malicious package in the NPM registry. EventStream was turned malware designed to target a very specific development environment and harvest account details and private keys from bitcoin accounts we talked through dominic's backstory as a prolific contributor to open source his stance on this package his work in open source the sequence of events around this hack how he can and should handle maintainership of open source infrastructure over the full life cycle of the code's usefulness, and what some best practices are for moving forward from this kind of attack.

So we're here in the wake and aftermath of a event stream malware incident that happened to the open source community, to the NPM community, to the JavaScript community, and to Dominic himself, the maintainer of the repository and NPM package. Former maintainer. Thank you for correcting me. Please correct us as we go here, as there's lots of details and we're definitely going to probably slip up on some of them. But, uh, for those, uh, who weren't on the internets, um, on, on and around November 26th, 2018, um, we will recall some of the events that happened to catch

everybody up and get us all on the same page. So there is a, uh, repository and package called event stream, which was created by Dominic Tarr and maintained up until recently by him, which had an issue opened on November 20th of this year by a fella goes by Falling Snow. I think his name is Ayrton Sparling, probably mispronounced, but the best I can do there. And he was wondering what had happened to this repo and why specific access was given to a GitHub user by the name of write9ctrl. He had found some issues there and was asking about it. This person injected some malware into the NPM package, a specific version of EventStream, which was used by many folks. And that was very problematic. Now, this caused all sorts of confusion, conversation, some yelling, some nice things.

It caused a lot of discussion on Monday and some actions. Thankfully, NPM acted pretty quickly and the package was removed the morning of November 26th. The initial incident was about a week earlier, but didn't really hit too much of the zeitgeist until Monday. Dominic's here with us. Dominic made a statement on November 26th and we're here to talk about that and talk about the aftermath, what causes these problems, really open source the community and the culture writ large because there's huge

implications and lots of fun stuff. So that's my summary. Adam, Dominic, please hop in and help me out there or fix any things I had wrong and we can get going. I woke up that morning to a friend telling me that that issue had been posted all over the internet

and I might want to block it or something. And I was actually quite excited because I thought that that issue brought to attention quite important issues for open source, which I made a... See, everyone was very excited about it, so I made a statement about it. But I think I'd like to explain how everything fell into place.

I wrote this... The event stream module, I wrote seven years ago, and at that time um like node.js like node.js is now um like hugely popular like there was right now there's like hundreds of thousands of modules in the in the um on npm um back then tens of it was like tens of thousands i've been involved with node since there was, I remember when there were a thousand packages in NPM and that being celebrated as a big

thing. And so EventStream was of that era and it was like I had I went on to write a huge number of streaming, of stream related modules and then and EventStream was actually the very first one that I wrote and

I wrote it and then after about I think it was 11 months of stream experience I realised that EventStream was kind of the wrong basis and I wrote a thing called Through which became the basis for all of my streams stuff after that and so even by that point, so that was like 6 years ago I had basically moved on from um event stream and wasn't really interested in uh like i wasn't using it as like my first go-to um thing for writing streams anymore and then like another maybe like a year or two after that the like node core team decided they were going to fix all the problems with streams and create

Streams 2. And I hadn't managed to participate in any of these discussions on what was going to go in Streams 2 because it was all at this Node conference in California and I wasn't there. And when I saw what they wanted to add, I was this is like horribly like bloated and ugly but it was also backwards compatible which like made it like twice as bad and so I started like I tried to like I tried some old protesting and they were just like oh we're ready to start you know we already decided that this is how we're going to do it and that sort of spurred me to be like well if you were going to really make a really minimal efficient stream thing that wasn't backwards compatible with the current streams, what would it look like?

And I started experimenting and with me and some friends and stuff, I came up with pull stream and pull stream is like really minimal. You just have two functions. One function is like just a normal node async function that you call it repeatedly, like one at a time. And the other, so that's like, you have a readable function and then you have a reader function, which is a function that the readable was passed to.

And I've got detailed blog posts about both the history of Node.js streams and Pullstream. So you can go on dominictar.com. But the point is like Pullstream was like, I decided this was actually so much better. It solved several of the like problems that Node streams had like error propagation.

So like if an error occurs somewhere in the stream, it like cleans up and aborts the whole stream and you end up getting the, you know, you know that the stream it like cleans up and aborts the whole stream and you end up getting the you know you know that the stream ended in error and it was like for just moving like data about like you did with event stream it was just like much more minimal and lightweight and efficient and benchmarks that it was like faster and stuff like this even though i hadn't really tried to optimize it i had just written the list code so i had like fully moved on by that point. I was like, this thing is great. And I've really tried to promote full streams. Some people will cotton it on and there's

a pretty good community of people that use it. But anyway, by the time that was also several years ago. So I had completely moved on from event stream twice. Yeah, not only had you stopped working on it and maintaining it, but you had replaced it with things that you consider much better quality,

the way to go. In your mind, this particular package was ancient history, right? Yeah, and it was like at that point, it wasn't really popular yet. I think it became popular when Gulp used it. So Gulp was, I mean, I never used Gulp.

It's like a build tool. It's kind of like Make or something like that. It's for building all of your projects and stuff like that. And I just felt that JavaScript didn't really need one of those. But anyway, Gulp happened and stuff like that. And, you know, I just felt that JavaScript didn't really need one of those. But anyway, Gulp happened and got pretty big and it used event stream and the first version used event stream and it's like example documentation and stuff. So that's when it actually became more popular. But that was after I had already moved on to Pollstream's family enough.

So you moved on twice, then it became popular and you're still moved on from this project like yeah from a from how you use it it's usefulness to you not so much just the project but how it's usefulness to you like how you use it to develop applications or using your tools and tool chain but grunt made it popular grunt gulp all those i'm not sure if grunt may have used it or not, but they were all in. No, you said Gulp. I said Grunt. Yeah, it's Gulp.

Gulp. They all run together to me. Grunt, Gulp, all in the same era of like this events as streams to build things era prior to Webpack becoming more and more popular and things changing so the the real interesting thing about this situation is that the the compromise or the the injection of this this code didn't come from hacking github's permissions or npm js.com it was really a a third party who came to you on a project that, like you said, you moved on from twice. This is the right nine control user on GitHub, which is no longer a user on GitHub, of course, a malicious actor. And it must have been acting like a normal person and obtained credentials to this repository basically by asking for them and then used that access to basically

he or she added, according to FallingSnow, FlatMapStream, which has an injection in it. Very, very briefly added it, published a new version, and then took it out in order to kind of cover their tracks and leave the actual installed version, of course, on as many computers as possible.

Right. And so that's really the bigger picture of this is like, this was an interesting, unfortunate situation, but one that is like very difficult for you to see coming, right? So tell us about this, you know, maintainer that you added. And I know you're kind of working your way there with this story of how

event stream is like outside of your own mind and use. Um, tell us about that decision. Was it long time ago? Was it recently? Was it, uh, well,

there was like, um, start of September. It was like a few months. It was only a few months ago. Okay. And so I hadn't, like, my maintenance of event stream was basically like

ignoring it. I hadn't made any feature changes in like five years. Like basically, I just, if anyone made a feature change, I'd tell them to publish it as a module. Let me throw out a couple of stats about Dominic, because I doubt he would say these things himself in order to boast but um if you if you're thinking about dominic tars make me perhaps like

event stream was you know one of his a couple packages written how could you ignore this thing that you toiled over dominic dominic has at the time of this call 62828 source repos on GitHub. That means those are non-forks. He actually created those repositories. And 422 packages published on NPM under his name. In November alone, Dominic, according to GitHub's activity, you've done 257 commits on 33 repositories in November. So I would call that prolific in your open source work and products.

So EventStream at this point was an old package that you wrote, a thing that you moved on from twice. And it's not like it's, you've got the six packages that you care for. It's like you have hundreds of packages. And so, like you said, the maintenance on that

was you were basically ignoring it. Yeah, and like none of my friends used it um either like it was basically just things that like like so i would occasionally get like you know issues or support emails because of it and i was like these are all just like annoying and but it was just like i wish like i wish this would go away and then someone like emailed me and was like hey uh this this was useful to me do you want to you know can i i could spend a few hours a week like maintaining it and i was like hell yes like i jumped at the time like please you know thank you so much like you've actually like you're the first person who's actually offered to help so i gave it to them um so and then this is like a critical thing that really like ended up exasperating the um the whole situation is that then like in between then and now

uh maybe a month later i was like i had the you know i had a whole bunch of other old modules that i was like no longer really interested in um that i didn't use and um occasionally people would post issues and make pull requests and stuff like this. And it was always like a huge bummer for someone to like earnestly come looking for like help or want to merge, you know, or make some change or something like this. And then, you know, this would require me to like probably think about the implications

of their change or something like this for like at least 20 minutes or something. And I no longer just like like i moved on from all these things and it was just kind of like a huge bummer to like tell them like look i'm i just can't be bothered like i'm just not interested in working on this like and it was like not the like the 20 minutes or something it was just the like letting some stranger like down that hard um that was like that was actually i realized that that was what was getting to me and then because i have hundreds

of modules i had to like i thought i wanted to like disown in bulk so i've actually um written like more like 700 and something modules and like in between handing off EventStream and now, I like disowned like 340 of them. Oh, wow. Yeah, so I wanted to do it in a way that I would no longer have access to them. So I created an email and then used a random password that I forgot

and then used that to sign up for another NPM account called NoPerson'sModules and then forgot the to sign up for another npm account called uh no person's modules and then i forgot the password to that as well so i can't log into these things and then i transferred like hundreds of modules to that account which now no one controls i told i emailed uh npm support and told them i was doing this as well for like uh like based mental health reasons drastic measures yeah but i thought it was quite uh cathartic to just be like these like no longer my responsibility and the things that i still retain control of are the things that i actually use and like directly or indirectly mostly maybe there's a few things i can clean up

as well but and then there were a couple of cases where someone i had added someone else as a publisher as well and so if someone else was already a publisher, I just removed myself from those modules. And EventStream fell into this category where I had already, someone else seemed to be an active maintainer. And at that point, they seemed to be legitimately maintaining EventStream. So I was just like, okay, that's under control. And I'm just like, you know, so I no longer had access to it. And by the time they report the attack,'m just like, you know, so I no longer had access to it. And by the time they report the attack,

I was like, sorry, I actually can't help you. Like, I don't have, I don't have access to this module anymore. Was this recently then, this cathartic moment for you?

Because you said that control, what was it? What's the person right now in control? Right now. Reached out a few months ago and asked you for commit bits so that they can help you maintain. You gave it to them. So was this cathartic

moment you're talking about where you did this deliberate... Yeah, it was something like a couple of weeks after that or something like that. Okay. Yeah, basically it was a coincidence. There was basically a coincidence in timing. I think if I hadn't have done that, then when someone reported this thing, I would have been like, okay, well, we'll just unpublish that version, remove this weird thing and stuff like that. I probably never would have made the news because it would have been dealt with in that issue. Right. But because I had forcibly, I had locked myself out of it.

That's interesting. Yeah. You really couldn't go in and see it go. This was like a perfect storm. And let me just say that it was a pretty sophisticated social engineering attack because that's really what it is, right? Like we talk about hacking into computers and whatnot. And we've, we've, we realize over time, go all the way back to Kevin Mitnick, like the best crackers are the ones that just ask for permit, just ask for your password or act like they're somebody else and get a password.

Most people were very forward with information. And this was a situation where this person picked really a prime repository where it hasn't been active. Probably the issues were building up. You can tell when somebody's ignoring one of their repositories just by doing a little bit of research and then not only that but so deployed right which you can probably see via like npm download stats you know on builds and stuff how many downloads of this particular package in the last month and then know, really weaseled their way in. So it's, it's so interesting that, you know, why we found out about this. First of all, I think falling snow

is like a saint. I love people like, so thanks falling snow for, you know, doing this. I don't know what you were doing necessarily poking around and finding this thing, but shedding light on these things. But then also the fact the fact dominic that you had removed your own access from npm and so there was there's really nothing you could do like this was going to have to be brought into the light and so here we have it yeah but i think one thing important to note is that the attack that right nine control was like trying to pull off um was actually a very targeted attack like it was only it only actually affected a specific bitcoin wallet that understand hadn't actually been like fully released or something yet so it it didn't actually end up doing any massive damage it ended up failing

completely like the damage was all um basically people's time got wasted because they had to like remove the code. I wonder if they were targeting a specific person that they knew used EventStream or something of course that's all just. Oh yeah it was this it's quite obvious who they were targeting because

the malicious payload was encrypted and the key to the encryption was. The wallet., it was the name of the wallet thing. It was like environment variable that is set when it's run inside an app. And so when they found that there was some suspicious

encrypted material in the minified file, but they had to go through every single module that depended on event stream until they found the one that actually decrypted the thing. So it was like they had to just try thousands of things until they discovered that it was this Bitcoin wallet. Of course, by the time that they had noticed

that there was some encrypted thing inside the minified content and not inside the regular content, then obviously there was something, something was up and they didn't know exactly what was the thing. Although now in hindsight, you know, next time this happens, like some Bitcoin wallet is probably a pretty obvious guess. That's interesting. I didn't realize that they had figured out that much. I think I also want to point out just the highlights of the community with people like FallingSnow, but also the others, like you mentioned, so many people digging into the code and really the reverse engineers come out and do all their Sherlock Holmes work to figure these things out. It's really quite amazing how fast and successful they are at tracing the trail here.

Also, what's up with the right nine control? Don't they know that Bitcoin is crashing right now? Like, come on, what's the deal? The value is way down. I think they started working on this hack before it crashed as well. If we could just get all cryptos to go back to zero, then maybe we'd have less problems like these out in the world. Yeah, the analysis of a supply chain attack from Hayden Parker

laid some of this out in terms of the backdoor would only be activated if the code was included in BitPay's open source wallet called copay or any forks that did not modify their project's description so there were like you know like dominica saying it's very targeted but i think one thing that i'm seeing here that's kind of interesting and correct me if i'm wrong is like you know github and npm are not a one-to-one you know so it seemed like like this person was able to deploy this sort of undetected in a way because they had done some merge on GitHub and then undone their work and did a deploy or something like that to NPM. And so NPM and GitHub are out of sync, which I'm sure is very common to be a case.

But I'm wondering if that's not a line for security vulnerability whenever something that is on GitHub in source code does not somehow match, you know, as a code repository, you know, place doesn't match NPM, the final build built module. You know, what do you guys think about that? The misnomer there, the anomaly, the fact that they're not connected or they're not the same. There is a number of ways that this particular attack could have been systematically prevented or made a lot more difficult to pull off or a lot easier to identify. And one of those is you check that there's a deterministic, that the build product is deterministic

from the source product. So the minified version of the code, like if someone else minifies the same code, it should produce the same result. And if the unminified thing didn't include the encrypted stuff, then the output shouldn't include it. So a simple thing that you could do

to prevent this kind of thing is, at least that would have detected this thing. Let's say there's this tool that you install your dependency tree, then you run this thing and it goes through all of the dependency tree, clones all of the repos, builds them all, and then checks that what you've installed is actually exactly the same down to a byte as what was built. And if anything is different at all,

then you flag the, you know, you'll be like, that's something, there's something non-technistic there. Even from a maintainer's level, like a maintainer isn't going to catch this unless they have the right kind of tooling because no maintainer, I'm not going to say, well, maybe not no maintainer, but not many maintainers are going to confirm that what is on NPM is what I was on GitHub. And especially if it's, you know, minified or whatever, like who's going to take that measure to ensure that, hey, I'm going to give commit bet access to this person. And I'm going to assume that they're not malicious because the previous commits, whatever, like they're just not going to do that level of like vetting we need tooling in place to sort of as you said systematically catch this kind of exploit because github did not match npm in this case yeah and like this this has been um this issue of deterministic build which is sort of hasn't been on the radar

i think until now um of like the javascript community but has is something that so like debian has spent the last couple of years moving to like every debian package which is mostly like compiled c c++ etc uh is like fully deterministic now so it means if two people build the same like compile the same program then the built output will be exactly the same so So if you have multiple trusted, independent people building something, and one of them is different, then you know that there's something up, and possibly it's a Trojan inside of the compiler. Because this was an attack described in the early 90s

by the creators of Unix, the Trusting Trust attack, which is pretty famous, but also generally no one had actually used it to do anything malicious that anyone was aware of, although there had been a benign Trojan inside

of GCC for some years, where it was just like something that when you compiled it, it went into the, it would insert itself into the compiled output and never appeared in the source code and then when you use that to compile the next version of GCC it inserted

itself and didn't do anything bad it was just I guess it was probably just a funny prank. Perhaps a good moment Adam to once again cross promote an old episode we did reproducible builds and secure software with Chris Lamb we talked to Chris Lamb all about his work with reproducible builds and how that's going into the Debian and Ubuntu distributions back in February of 2017.

So a couple of years ago. But man, that was a prescient show because it's come up so much lately as these are definitely things that people are starting to realize are super important because we're having a fallout from not having that as a feature of our package management tools. So go back and listen to that. We'll put that in the show notes if that is of brought to you by Linode, our cloud server of choice. It's so easy to get started. Head to linode.com slash changelog. Pick a plan, pick a distro, and pick a location, and in minutes, deploy your leno cloud server they have drool worthy hardware native ssd cloud storage 40 gigabit network intel e5 processors simple easy control panel 99.9 uptime

guarantee we are never down 24 7 customer support 10 data centers three regions anywhere in the world they got you covered head to leno.com slash changelog to get $20 in hosting credit. That's four months free. Once again, leno.com slash changelog. so we're talking about ways that these kinds of attacks can be prevented down the road systematic things we can do maybe we can't do them maybe certain platforms have to do them certainly maintainers have things that we can do, but you mentioned the deterministic builds or reproducible builds, but you also have some other things that could be features of our systems that would help from this kind of attack happening.

Do you want to elaborate on those, Dominic? Yeah, well, so firstly, I'm not... Like, reproducible builds might have made it easier to... Detect. Like, reproducible builds would have flagged this, but he might have gone to... Like, if they had, like, checked in the malicious code and said, like, maybe no one would have noticed.

Like, once it was apparent that it was, like, that there was some encrypted code that was being run, it was like, something is definitely very suspicious. But if it had been unencrypted, that might have actually been like more effective like we don't really know um it could it could have been overkill to encrypt it and so that would have actually gotten past a reproducible build because the bad would have been just hiding in plain sight and the other thing is that the attack depended on the event

stream code doing several things that well rather the flat map code doing several things that really had no business doing so to like successfully steal people's Bitcoin keys then send them back to the attacker it needs to do network IO and event stream itself like didn't need to do that like that is completely outside the stated

purpose of event stream the same with like accessing the crypto module and that sort of stuff. So if there was a specific list of what permissions on a module basis that you could request from... Yeah, that event that a module depended on.

So event stream was like, it doesn't do any IO, it doesn't do any networking or file access, it just glues other things together. Then compromising a vent stream wouldn't have been useful for this attack. You would have to compromise something else that had access. And then

of course it also monkey patched the constructor of something that then it got the keys from and you had a good sandboxing thing. It wouldn't have been able to do that. I'd been aware of this stuff being developed for some years. There's this thing called

e-rights, which is quite old. This guy, Mark Miller, was also the chief architect on the Xanadu project. Are you familiar with Xanadu? Oh, this is really getting into the... This is something you should definitely know about. Maybe you should do

a whole podcast on it. So this guy guy ted nelson basically had this idea for the worldwide web but better and it had like versioning built in and you had this thing called transclusion where like a link like just embedded other documents and stuff but but the thing is he had this idea in the 1960s and then spent like several decades trying to develop it and by the time that the web came along the first release of the web from tim berners-lee cited xanadu and was like i wish xanadu was ready but but given that it isn't here's a like a crappy version of the idea that i cobbled together and the sort of project xanadu was like a far far more ambitious idea that you know it actually inspired a lot people, but failed to deliver any usable software.

The history of it is quite amazing because it did have quite a big impact in terms of ideas, but it didn't successfully deliver anything. But it still was instrumental like actually creating the web but it was just actually the creators of Xanadu was like the web is actually like a really disappointing like crappy version of what we were trying to do. I found Project Xanadu on Wikipedia reading a little bit along and I agree this would make a great

show to do separately but what's interesting is that there was a working deliverable called Open Xanadu that was released in 2014 it was called Open becauseanadu that was released in 2014. It was called Open because you can see all the parts, so not necessarily open source, but just open to see.

On the site, the creators claim that Tim Berners-Lee stole their idea and that the World Wide Web is a bizarre structure created by arbitrary initiatives of varied people and has a terrible programming language. And the web is a complex maze. I'm not sure if this was mark miller doing this or somebody that was inspired by mark miller's work on xanadu but it's super interesting and this is just wikipedia by the way so take it for what's worth so mark miller was the chief architect but the xanadu project was started by um ted nelson i actually you know had the privilege of meeting mark miller a few months ago as well. And I had been aware of his work in e-writes since then, which was basically the idea was he was trying to build a programming language that was optimized for security auditing. You could definitely say that this part can only access these things.

And unless something has been passed into it, it can't interfere with that other thing in any way at all and there's this old website that explains this and it looks like all the really excellent ideas that for the mainstream computing which is like basically being completely ignorant of any of security like it's security is just like a huge pain in the butt for most people they had like envisioned like this is how we could solve all of these problems and be working on it just for decades now. But interestingly, in the meantime, they had actually infiltrated Google and had managed to add several features to JavaScript that enabled JavaScript to have all the pieces of the puzzle to create this in JavaScript. Who's they? You said they infiltrated Google. You're talking about Mark Miller? Yeah, Mark Miller. Infiltrated, perhaps is the wrong word, but he worked at Google. Sure.

And I understand it was, who's the other JavaScript guy? Douglas? Crockford. Crockford. Yeah, yeah, yeah. Crockford had been like, you know, JavaScript is like, nearly has all of the features you need to make this secure thing. So this produced, I think this is my understanding, strict mode and object.freeze. And so now left Google is working at this thing, Agoric,

which has produced now a thing, secure ECMAScript SES. And there's a bunch of like versions that are more or less constrained or something like this, but it gives you a pure JavaScript way like you don't need like a special platform.

So you don't change the background like this already works in like a web browser and you could block off some code so you couldn't have to completely use a like user provided code that then runs in a context that you can be confident it's not touching other things so it can't it can't do things like prototypes and i can't it can't do anything it can't use things that haven't been passed to it directly you know many people will point out that getting sandboxing right is like extremely difficult.

And that's like absolutely true. But luckily, these people have been like literally spent their careers working on this. So who would be the people that would task with working on something like this? Are we talking about browser vendors,

Node? Are we talking about package managers? Like where would the sandboxing and the application, these principles go? Adding it to an actual deployed application that was originally created without this stuff in mind, so currently

written code and applications that are running with NPM and stuff, someone would have to decide, I guess, what APIs things have access to and stuff like that. It would be easier to start fresh. Well, I think it could actually be added in user space. You might have to go through and be like,

this thing can have these permissions or not. You mentioned earlier that it didn't need I.O. access. So that would be an easy one. So if ever this module ever requested or used I.O., then something is very odd about its behavior because its described behavior says that it shouldn't use certain APIs or certain feature sets, essentially. Yeah.

And for the most part, unless they do something really weird and dynamic, probably 95% of modules would be an easy call to see what they should be able to or shouldn't be able to access. But you can just weed out dependencies that like do weird stuff. So as the attacker though, let's say I have what right nine control I had, I have access to the source code

as well as the deployment mechanism. Even in this world where there is this sandbox, could I not simply, I mean, I would have to provide the white list as the author of event stream. So would it not just add IO to my list of things that I require and then deploy? Maybe you could have at that point, some sort of like this permissions have changed.

Do you want to allow this to do that? Yeah. What a world. Yeah. I think it would be a good improvement by Android phones, et cetera, have this kind of permission system. And yeah, I think most people just like click OK. Exactly. We're pushing the attack vector up the stack to the end user

who's more likely to not even know what the heck it's talking about and say yes. Yeah, but on the other hand, when you install a module, if you made the call then and it was actually controlled, because you end up things with like BitPay didn't actually install EventStream. They installed something that installed something that installed EventStream. Right. So when you install something, you say like, oh, that should never do such and such. Then it's sort of basically a question of like, who do you trust? And perhaps if you were something like a high value target, such as a Bitcoin wallet, then you would just go through the entire tree and be like, what should this be able to access?

What should this be able to access? Which might take a while if you have hundreds of modules, but it would definitely give you peace of mind and be an appropriate action. And this kind of attack would really be even worth bothering with. You said something there too, like to trust, right? So we're, we talked through sort of like systematic ways to prevent this, which seem to have varying degrees of user experience degradation and, or as you said, Jared, just pushing the, you know, the attack vector up or down the stack to different places that, you know, may have ill effects like clicking okay, or just bothering the developer at some point or maybe even the user at some point with things that they're just not concerned with. The other is sort of like what you said there, Dominic, around trust. And even in your readme, the statement on event stream compromise,

you mentioned two strong solutions to this problem, one being paying the maintainers, but the additive to that was only depending upon modules that you know are definitely maintained. I'd like to kind of get your idea of what definitely maintained means to you because your version of that and my version of that and Jared's version of that may or may not be the same. And then point two you made was when you depend on something,

you should take part in maintaining it. So maybe we can break down those levels of solutions where it's like rather than changing how the software only makes sense reminding people to trust their dependency tree but how do you do that yeah well i mean part of the a big part of the problem here from the social perspective is that the tooling basically assumed I was responsible for like this module and had like full control over the decisions made about it. So when you delegated to someone who had installed something that installed event stream, you didn't have control over who made the decisions about event stream. So I was able to like, like it was basically like

I was able to just transfer the right of access to event stream and i like didn't i didn't want to have the right to control event stream because i had no interest and um like i had no skin in the game you abandoned it basically yeah you wanted to be you want to be out it's sort of this really weird thing that only happens in software like there's no other like part of like it's hard really hard to make an analogy there's some other part of life where like a hobbyist ends up maintaining some kind of critical infrastructure while they don't want to yeah it's like insane what's funny is that is that maybe i can break this down a little bit is that you you know to the world own this thing you personally have moved on from the concept, the paradigm, and the idea.

Meanwhile, the rest of the world found it to be useful years later and made it wildly useful in many different ways. It's been included in several different packages and very popular organizations. But meanwhile, the original creator, who's as described earlier as a prolific open source developer, has since moved on not only from the project or desire to move on. And in some cases with some some angst because you had some pain with telling people that you were were not going to be involved anymore. And you mentioned just sort of the the the mental tear on yourself there. But the fact that you've just moved on from it, but the world hasn't, yet you're still involved or at least somehow in the blame zone. Yeah, and I think it's also important to mention

that Modular itself hadn't changed at all in any significant way over that entire time. It became popular for what it originally was back when I thought it was a good idea. I hadn't really been doing anything except for just ignoring it and reluctantly occasionally responding to something or merging. It was just a pain in the... How do you deal with that then?

If this is not a thorn in your side, you've moved on from it. The world still feels it's popular. Two million downloads per week, according to Tidelift. Used by open source, large open source projects like Angular, Mocha, Electron, and others. Other commercial code bases from organizations like BBC News and Microsoft. So, like, clearly somebody had some value in this thing, but you want to move on. How do we collectively as open source look at the scenario and say, how can we allow maintainers to move on in ways that a project that is widely used or at least widely useful to many people or organizations? How do we let them move on in a way that keeps the code base secure

and doesn't allow something like this to happen? Like, do we hand this off to support organizations like Ruby Together might be, you know, in this case, it's a JavaScript NPM module. So maybe Ruby Together isn't the right one, but that example there where you have sort of organizations that are intended to be sort of this catch-all that are trusted or could be trusted or have some sort of vested interest in the future of an ecosystem. Yeah, like basically when something is new and exciting, like the thing is like most of this, I think most of the code on NPM

was created by like, I mean, there's probably some big things that people are working on for their job, but the vast majority of them are like small things that people have created in their own time or to fix their own little problem. And some of those things, such as event stream, became hugely dependent on. And the kind of like thought process

and like skills and interest and motivation for creating those in the first place is very different from the thing that is required to maintain them in the long term. So once something has depended, it's like people are using something, then you don't really want to change it. It's better because if you break anything, loads of people are going to be upset with you. Everyone's going to have a bad time.

It's better just not to change anything at all. Just keep it completely stable. And that's not really very fun. When you've created it the first time, it was like a new idea that you were exploring and something worked really well. It's like, it's fun for a while, but by the time it's like popular,

like maintaining it as a job, and it's not like a job that you necessarily signed up for, like it would be much better if it was maintained by someone who saw it as their job. So what you're saying is if somehow over the 700-ish modules you've created in your career, I think you mentioned roughly 100 or so you decided to abandon a couple months ago or a couple weeks ago.

300. Okay. Thanks for correcting my math there. What you're saying, if I understand you correctly, is like if there was a way for you to be paid a fee that makes sense a fair fee to continue to be a security measure a maintainer even if that's necessary to improve most of these are mature and stable and don't need to be changed much you would stay on board as as a an on the hook person to provide maintenance and security sure yeah i mean there's only like a i mean there were 300 modules but only a small handful of them were really popular and this i think this is the

other thing too is the most popular ones um are completely boring like the things that like i'm personally most proud of and most interesting you know the most interesting problems are not really very popular at all the things that have like millions of downloads, like my most downloaded module, which I actually still use is RC, which is a configuration loader. Like it just loads configuration file. And, you know, it's kind of like pop music. It's like to have the broadest appeal,

it has to be like completely bland and uninteresting, such that like the broadest spectrum of people can relate to it. Like everyone has to load a configuration file. General purpose use. Yeah, there's no exciting way to do it, though. I agree with you completely. I think that's why Nadia Ekbal's report about open source funding and sustainability was so well-named, Roads and Bridges,

because we're really talking about infrastructure. And the phenomenon, I guess, that we're actually seeing here is people accidentally create infrastructure. They're not trying to. They're solving problems they have. But it turns out those problems are general use and easy to pull in and easy to deploy. And over time, it becomes infrastructure. And the fact is that nobody wants to work on a road or nobody wants to work on a bridge,

right? Maybe building that bridge was fun and interesting problem for engineering, but it's just maintenance. And so we accidentally create these roads and bridges and they become public infrastructure or not public, but you know what I mean? And then it's like, oh, I'm supposed to just work on this road for you for free. And so that's where the real rubber hits the road is like, we accidentally got here, right? You accidentally became the maintainer of a thing that you wrote seven years ago and people are still using today and you have zero interest in.

And so that's where I think that's why many of us turn to, okay, now I'm ready for a financial compensation to take care of this because i also wouldn't maintain a road for free well to that effect too you don't have a lot of people coming by this perfectly fine bridge saying you know what i think i'm going to build a different bridge right next to it slightly better because i want to like rc works or in this case event stream works why would you come by and rebuild as you mentioned

that's why it becomes accidental infrastructure yeah I mean the metaphor falls apart because we can clone and fork you know these things

like you can't clone and fork a bridge but to a certain degree it fits right anything to respond to that do you agree with that analysis is that how you feel

well yeah but the thing I'd like to stress is that when i talk about like paying the developers like that's actually not my first choice like i like hobbyist style programming i don't necessarily want that i would i would kind of rather people who like depend on that code for their job take over maintaining such code my point is just that the incentives of who's going to take responsibility for this should be who ends up actually feeling the impact. The problem here was that someone who wasn't interested in having that responsibility ended up with the responsibility. If people who actually needed that thing ended up being the ones maintaining that, and if I didn't even...

The other thing is you can give it away. You can hand things off to someone who will maintain it, but you're still responsible for figuring out who that was. And that's what I thought I was doing. And to speak a little bit to the, in general, this system does work. I've seen many cases where somebody writes a thing, an open source thing, and then other businesses or enterprises come to rely on it and the original maintainers do not use it anymore. And then the businesses or the enterprises or whoever currently has a stake in it, they take over maintainership. And this is something that happens.

So it's not like that. And I'm glad that I agree with you. I think that's the best way for it to work. In lieu of that, like when that doesn't happen happen pay the maintainer if you don't want to maintain it but somebody has to and i i do i do see it working so it's not like it's utterly broken it's just that there are times where things fall through the cracks and then we have issues yeah a lot of people who were like upset about this but like why couldn't you just like like depreciated it or just stopped maintaining it

and not giving it away? And the funny thing is that actually was a decision that I was coming to and I actually did that for hundreds of modules shortly after. But coming to that decision when I had hundreds of modules that I didn't want, it wasn't like a one at the time thing. It was like a bulk decision that involved like writing scripts to disown like all of these modules.

Like I had was, it was like, like the tool, the current tools don't provide like a good way of like doing this in bulk. Like for example, so I had moved all of these modules off of my NPM, but I hadn't archived them on GitHub yet.

And I just simply hadn't got around to it. Like, I felt that removing myself from the NPM was sufficiently cathartic that I was like, oh, I'll just come back and do the GitHub thing later. If I had done that before it was reported, then they wouldn't have been able to post the issue on that thing. I don't know. Something else. What if it had to happen? This episode is brought to you by our friends at GoCD. GoCD is an open source continuous delivery server built by ThoughtWorks.

Check them out at GoCD.org or on GitHub at GitHub.com slash GoCD is an open source continuous delivery server built by ThoughtWorks. Check them out at GoCD.org or on GitHub at GitHub.com slash GoCD. GoCD provides continuous delivery out of the box with its built-in pipelines, advanced traceability, and value stream visualization. With GoCD, you can easily model, orchestrate, and visualize complex workflows from end to end with no problem. They support Kubernetes and modern infrastructure with elastic on-demand agents and cloud deployments. To learn more about GoCD, visit gocd.org slash changelog. It's free to use, and they have professional support

and enterprise add-ons available from ThoughtWorks. Once again, gocd.org slash changelog. And by our friends at Red Hat who produced the podcast Command Line Heroes. Today we're featuring a segment from Season 2, Episode 7, titled At Your Serverless. But now, of course, all over the United States of America and all over the world, the Internet is revolutionizing our lives. It's 1998. Google just hired its first employee, and Vice President Al Gore is talking to the press. This technology is still in its infancy. When President Bill Clinton and I

came into the White House, there were only 50 sites. And look at it now. I got a bouquet of virtual flowers on my birthday. Okay, I can sense your eyebrow arching already. Why am I playing you some bit of 20-year-old internet history? It's because I want to remind you that the basics of the internet are still the same. Today, developers talk a lot about going serverless, which sounds like Al Gore's client server internet just got trashed. And if we're not careful, we can abstract away so much infrastructure that we forget there are still servers out there doing their server thing. All right. Learn more, listen and subscribe to this awesome podcast at redhat.com slash heroes. Once again, redhat.com slash heroes. So we've talked about some technical things we can do. Some of us as a community, we talked about some, you know,

better practices in terms of like consumers of these things is either hopping and help maintain, throw some cash at the problem, which is always nice, but hard to convince upper management to do. As maintainers, Dominic, it sounds like you hit on some of the things, which is like, what is a good practice for abandoning a project? And you hit on, like you said, a few weeks after you gave access here, which, you know, in a bit of serendipity, we've allowed the community to have this conversation and really

to talk about these things. And so we can all together learn and realize what we should be doing about our projects that we're maintaining. One of the suggestions in our community Slack, I think it was Dan McLean brought it up in terms of like how it should work or what would be a good way of going about it is basically he says,

I'm really starting to think that the model should be, if you don't want to maintain it anymore, update the readme and let it be forked. Forks have to establish their own reputation. I would hate to have my name used maliciously at the same time. I don't feel that I need to keep maintaining something. And then he says, look at CanCanCan, which was a good example of a situation where there was CanCan. This is in the Ruby community. CanCan was maintained by Ryan Bates, created maintained by Ryan Bates and abandoned by maintained by Ryan Bates, and abandoned by him when he took his long hiatus from the internet.

And then the community came around and created CanCanCan, established their own reputation by improving that, maintaining that, and eventually people switched over. So that seems to make sense to me in terms of the passing of the torch. That seems like the hardest part because uh in the in a situation where you're like i'm just done but there's people who still depend upon it how do we like actually pass a torch in a way that makes sense without you know throwing the baby out with the bath water so to speak yeah i mean in heinz obviously um the solution is um something like that and uh in this particular case it was just really a matter of timing that meant that

that hadn't happened. Like if right now it had come to me like a month later, then I might have been like, sorry, I abandoned that module. I can't give it to you. Which makes sense on GitHub side, but from on NPM side, in terms of just the mechanics of that, if there's still hundreds of thousands of dependents out there with event stream in their package.json. How would they know it's abandoned? How would they be able to come back and see okay, here's a new one. It's event stream dash some namespace of a new person and then maybe eventually switch over. Are those things that are unsolved problems at this point? There is a deprecation

option, but I've never really felt like deprecating something. The code worked. It was fine. I hadn't even needed, like nothing really needed to change. So it didn't really feel right to me to be like depreciate because it was just leaving it as it is. It's like

fine. So it still worked. I hadn't really changed it. It didn't need improvements. Well, let me throw out a hypothetical then. So like it's, it's still worked. Like I hadn't really changed it. It didn't need improvements. Well, let's, let me throw out a hypothetical then. So let's say that a vent stream is out there on NPM and its current version,

and it's just fine. And then some sort of dependency of event stream has an issue and NPM security team detects it and goes out and says, okay, everybody who depends on this version of event stream you're going to need to bump your version up and so event stream needs to be patched basically yeah well that's when that's that's the case right that's when someone needs to fork it there's other issues around that does like uh being able to adopt eventually an abandoned

package name too so assuming event stream is a super cool name it's abandoned there's a way to flag it as abandoned after a while someone like that's the other i guess smaller less concerning issue is that because this has been talked about when kick left pad and all that what if i want to name my thing kick yeah then what do i do yeah i guess you have to sue somebody um you know you might want to take over thing kick. Yeah. Then what do I do? I guess you have to sue somebody. You know, you might want to take over the abandoned name or something like that. But that's what I was thinking. I was like, if you could just allow somebody.

But that means claiming and owning a name on NPM becomes far more valuable. Because if you claim and own it originally, even if it's not used. I mean, obviously NPM could do whatever they want to to circumvent those rules as well but they they kind of inch back into some sort of policing model of the community and what is and is not allowed so they become more and more vulnerable to attacks of their choosing by just basically how they choose to to run things but if you were able to in this case uh if there was a way for dominic to say to npm hey i want to abandon this let me attach an abandoned flag this is what you're saying jerry where the dependency tree now knows that so as i uh you know ever use this package or dependency then i'm somehow made

aware like hey this this dependence you have in your tree has been flagged as abandoned on npm there may be something you want to look at here and or look for a fork or and or create your own this dependence you have in your tree has been flagged as abandoned on NPM. There may be something you want to look at here and or look for a fork or create your own fork and begin a new line of trust. And that what you're saying is that if that's the case, then this abandoned version

is cemented in stone and frozen forever. So let's say I had been just be like, okay, I've abandoned event stream. Then like if that somehow prompted the users of event stream and some users to like update then that would actually be a prime time for like the attacker to come along and be like

oh I'm attending a new fork of event stream it's you know I'll respond to things I like use it that sort of stuff then people like you know opt into that like it's you could pull off the same kind of thing there. Don't you think, though, if that's the case, though, that that would be seen as like a, hey,

the same model comes back into human choices, which is to part one of your statement in your readme was basically like, hey, trust, make sure you trust the maintainers, right? So that would go back into rule one for you, at least based on your two strong solutions is like a human should trust this new fork and there should be reasons to, and that's brand new opt-in, a brand new line of trust and a brand new line of scrutinization versus this inherited one. Yeah. But continuing to use the abandoned code might actually be better unless an update is really needed the funny thing is like compared to you

know what last year the the wanna cry worm that was a a hack that only affected people that hadn't updated their code this one was one that only affected people who had updated their code. Well, you're screwed either way then, I guess. Update or don't update. You're in the wrong. We're all in trouble. That's right. So this leads to a follow-up. Just as an end user of dependencies, like as a developer then,

I've always looked at dependencies with two extremes. On the one extreme, you have dependency hell, where it's just like pull it all in all right? Like I don't write my own code. I'm just gluing my dependencies together to cobble something together. And the other side, we have like the pristine not invented here syndrome where it's like, I know every single line of code in this thing. And I've written every single line of code. And those are kind of the two extreme angles you can take at dependencies. And I've always, I mean, I've always said it's a balance, it's a trade-off, you have to make these decisions with as much

information as possible. But the older I get, the more I'm starting to err on the side of not-invented-here syndrome, because it seems like the trade-offs to having more dependencies is worse than the trade-offs of having to write some more code myself. What are your

thoughts on that? Well, I have been accused more than a few times of not inventing here. But at the same time, there's a lot of things that I am satisfied with someone else's solution and have used that instead. What are some of the heuristics for you? What are the things that make you satisfied? Is it personal relationship with the maintainer? Or at least maybe a reputation of quality or is it your own inspection of their code?

What makes you more comfortable than less comfortable? I mean, for things like I didn't create JavaScript or implement my own JavaScript. There's things like that problem is too big to handle when he has a good thing that does it. One is if it's a small thing that I could do, I would sort of end up looking for a whole bunch of options or something. I would evaluate them in terms of like, is this sufficiently compatible with my opinions? Would I do it the same way?

Does this make me really mad or something like that? There's some stuff that would just be like a pain in the butt to create myself like time zones well time zones are easy what are you talking about just kidding it's like the bane of every programmer's existence is time and time zones that's right yeah and it's especially worse if you live in new zealand because you're constantly um you're in edge case people yeah um people in other time zones and the northern hemisphere hemisphere one is fun too because you go in and out of daylight savings at different times so between like say you know california and new zealand there's like four different possible differences where there's like you can both be in daylight savings for a bit and then

you're both out of it and then one of you's in and the other one's out. So it's like, and so you change like how many hours you are apart like multiple times throughout the year. Well, I don't even think I really understand daylight savings time.

There's times I think I understand it and then there's like, this is, then there's times I'm like, no, what you thought was daylight savings time is not.

So I just, I just don't even know. I just know I'm where I'm at and I just, this is my time right now. What's your time right right now it's one of the worst ideas oh man and there's certain small precincts you know geographic areas which will not abide by it right like states or cities or countries it's like how do you do you even understand time and the earth's rotation and the things that actually calculate that and the things that that have been established scientifically ages ago to make time time and us be in sync with what we think time

really is it yeah it goes really deep and time zones are political and political things change with new registrations and so there's yeah it's a it's a complete mess but well one thing we haven't asked yet and we we we did mention at the top dominic that it was a it's a complete mess but well one thing we haven't asked yet and we we we did mention at the top dominic that it was a lot of conversation this has been a a the big topic of the week around these parts and so i think that's a benefit i think it's a good thing i'm curious about you personally it sounds like you've you've taken it all pretty well but no doubt there were some people that were mad or were criticizing what you did why couldn't he have just done this why did he do that curious what your overall feelings are with regard to the way the community has received this the way it's gone you know have you gotten a lot of backlash do you feel attacked

do you feel loved how do you feel i received many like personal messages of support from like other um like old friends friends and other open source developers and sometimes random strangers who had admired my work and stuff like this. So I really felt quite boosted overall from that. And that certainly gave me the confidence to use this to draw attention to what I feel is the plight of the open source developer and like what i like the systematic ways that something like this can be like approached and addressed and i think the way that i've kind of like been very determined to like shirk responsibility for actually like adding the thing like actually

helped a lot there because it it made all the people who were like thought i should you know apologize or something it made them like way more upset you were very nonchalant about it all yeah which to me was refreshing because I could tell that you were kind of cool calm and collected about it and just like this situation you know the conversation around you know this being a fun project for you like this was just a thing that was fun it was no longer fun and that's what you said in your statement is if it's not fun anymore you get literally nothing from maintaining a popular package. That's just the cold, hard facts. You can't be or act guilty if you're not guilty, right?

If you don't have a guilty conscience in the scheme of this, then you're not going to run around acting guilty or feeling guilty. Yeah. Yeah. And I also knew that we're kind of like weird to, you know, people who like enjoy programming for fun. But there's still quite a few of us out there and like many of them and lots of them my friends. So I felt like I was, you know, I was speaking for these people. As as Brett Cannon said in that recent episode, Jared, I forget which number it is.

Help me out if you can. But, but you know open source maintainers owe nothing i believe that's somewhat a direct quote correct me if i'm wrong we can pull up the transcripts and confirm that but just basically like you know they they've maintainers of projects haven't signed on for a we'll maintain this we'll be responsible for everything in this forever it's in their good interest in their heartfelt interest to create an open source in the first place and it's your choice to use it so they owe you nothing the other thing i want to say is like on one level like creating open source is like fun like challenging interesting technical problems but also the social side of it for the most part you're just like solving problems with

friends and helping each other out and no one is like the boss of anyone so if you want to like get things done you can persuade people and i sometimes people get wound up in like the strong emotions because like things that people feel strongly about but on the whole it's a very um rewarding kind of like mode of interaction and I wish that like more of life gave you the the ability to like affect change on things that affect you like you get at open source like I think that's a big part of like why I've continued to do it for like the best part of the decade absolutely and Dominic we just want to say thank you for all the work that you have done no doubt you brought lots of value to lots of people,

countless people around the world that you've never even met. And the beauty of open source, all these modules on NPM, the work that you're doing in the JavaScript community. Hey, we definitely want to get you back to talk about Scuttlebutt and the interesting stuff there. Talk about hacking with your friends. This seems like a very cool kind of offline social networking thing happening, which is very much in the spirit

of what we like to cover on the changelog. So definitely wanted to have you back, but we're happy to have you. We're glad you joined us, especially on short notice to talk about this situation. Any last words from you with regard to event stream

or what's happened or anything else you'd like to say to the open source community before we let you go? Yeah, I think despite all this, I think open source is like a great idea go? Yeah, I think despite all this, I think open source is like a great idea and we need more of it

and more sharing, not less. If we let things like this make us too suspicious of each other to share and collaborate, then the terrorists win and that would be worse than being hacked occasionally.

I think that's a perfect note to end on right there. Yeah. Dominic, thanks so much for joining us. You're welcome. All right, that's a perfect note to end on right there. Yeah. Dominic, thanks so much for joining us. You're welcome. All right, that's it for this week's episode of The Change Log. Thanks for tuning in.

If you enjoy the show, if you got any value from it, do us a favor. Go into iTunes or Apple Podcasts, rate the show, review it. It helps us get ranked up inside those indexes so more people find the show. If you're using Overcast, go ahead and favorite it. And of course, tweet a link to a friend or share it wherever you might want to. Huge thanks to our sponsors, Rollbar, Linode, GoCD, and Red Hat's Command Line Heroes. Also, thanks to Fastly, our bandwidth partner. Head to Fastly.com to learn more.

And we're able to move fast and fix things here at changelog because of rollbar check them out at rollbar.com and we're hosted on leno cloud servers we trust leno because they're fast they keep it simple check them out at leno.com slash changelog today's show is hosted by myself adam stukowiak and Jared Santo. Editing, mixing, and mastering was done by Tim Smith. And the music is done by the ever awesome Breakmaster Cylinder. If you want to hear more episodes like this, subscribe to our master feed at changelog.com slash master. Or go into your podcast app and search for Changelog Master. You'll find it.

Subscribe, get all of our shows as well as some extras that only hit the master feed once again thanks for tuning in we'll see you next week