The Changelog: Software Development, Open Source - Who in the world is Jia Tan? (News)

Episode Date: April 1, 2024

The big story right now is the recently uncovered backdoor in _liblzma_ (aka _XZ_) – a relatively obscure compression library that happens to be a dependency of OpenSSH. This incident is noteworthy for so many reasons: the exploit itself, how it was deployed, how it was found, what it says about our industry & how the community reacted. Let's dig in!

What up nerds, I'm Jared and this is Changelog News for the week of Monday, April 1st, 2024. Yes, it's April Fool's Day, the worst day of the year to be an internet denizen. But don't worry, Changelog News is 100% prank-free, AI-free, and gluten-free too. The big story right now is the recently uncovered backdoor in LibLZMA, aka XZ, a relatively obscure compression library that happens to be a dependency of OpenSSH. This incident is noteworthy for so many reasons. The exploit itself, how it was deployed, how it was found, what it says about our industry, and how the community reacted. Today's episode is entirely

dedicated to this story, looking at it from all those angles. So let's get straight into the news. The Discovery. Let's start our story the same way most folks did on Friday when Microsoft researcher Andres Frund posted an email to Debian's OSS security list containing this bombshell. Quote, after observing a few odd symptoms around liblzma, part of the XZ package, on Debian SID installations over the last weeks, logins with SSH taking a lot of CPU, valgrind errors, etc., I figured out the answer. The upstream XZ repository and the XZ tarballs have been backdoored. At first, I thought this was a compromise of Debian's package,

but it turns out to be upstream. End quote. Andres goes on to explain his findings in detail. The mind-blowing thing is that he decided to shave this particular yak because he was doing some micro-benchmarking and needed the system to be super low load, which made him realize that SSHD was using a lot of CPU. Go read all the work he put in to find the backdoor and then consider how specific his situation had to be in order to even notice it. Thankfully, he found the backdoor relatively early in its rollout. Quote, due to the working of the injected code, it is likely the backdoor can only work on GLIB-C

based systems. Luckily, XZ 5.6.0 and 5.6.1 have not yet widely been integrated by Linux distros, and where they have, mostly in pre-release versions. The code. The exploit itself is super interesting as well. I'm not ashamed to say most of it's over my head, but recurring changelog guest Filippo Valsorda does a great job explaining the nitty gritty details. Follow the link if you're interested in all of the particulars, but this statement sums it up well. Quote, this might be the best executed supply chain attack we've seen described in the open, and it's a nightmare scenario. Malicious, competent, authorized upstream in a widely used library. Looks like this got caught by chance. Wonder how long it would have taken

otherwise. End quote. So the attacker is competent, malicious, and has authorized write access to a widely used library. How did that happen? The maintainer. The linked page is maintained by Lassie Collin, the solo maintainer of XZ. Can you guess where all this is headed? Lassie says, quote, XZ Utils 5.6.0 and 5.6.1 release tarballs contain a backdoor. These tarballs were created and signed

by Gia Tan. Tarballs created by Gia Tan were signed by him. Any tarballs signed by me were created by me. GitHub accounts of both me and Gia Tan are suspended. End quote. Gia Tan? Who's that? The plot thickens. It's now time for Sponsored News. AI-powered autofix debugs and fixes your code in minutes. Ben Pevin, writing for Sentry, says, quote, Sentry knows a lot about the inner workings of an application's code base. So we got to thinking, how can we use this rich dataset to make debugging with Sentry even faster? Many generative AI tools like GitHub Copilot improve developer productivity in their dev environment,

though few have the contextual data Sentry has to help fix errors in production. Our new AI-enabled autofix feature understands what your users are doing when an error occurs, analyzes the error, generates a fix, and even opens a pull request for your review. It's like having a junior developer ready to help on demand. End quote. Pretty cool stuff. Give it a try. Oh, and don't forget to use code changelog when you sign up for Sentry to get 100 bucks off their team plan. Thanks again to Sentry for sponsoring ChangeLog News. The attacker. Evan Bowes looked up the public history of GitHub user GiaTan, which goes all the way back to 2021. Tan starts slowly, then ramps up as he gains trust alongside a few other accounts that appear to be

sock puppets. Evan tried to use the other public information to identify who Gia Tan really is, but a potential LinkedIn match seems unlikely. Quote, I have received a few emails alerting me to a LinkedIn of somebody named Gia Tan 2. Their bio boasts of large-scale vulnerability management. They claim to live in California. Is this our man? The commits on Gia T75's GitHub are set to plus 800,

which would not indicate presence in California. UTC minus 800 would be California. Most of the commits were made between UTC 12 to 17, which is awfully early for California. In my opinion, there is no sufficient evidence that the LinkedIn being discussed is our man. End quote. Analysis of the name has also been performed,

but when you include the middle name, Chong, that was found in one Git log, it seems unlikely that it's a real name. Evans says, quote, it's most likely our actor simply mashed plausible-sounding Chinese names together. End quote. As of the time of this recording,

it's unknown who Jiatan really is. The pattern. Rob Menching lays out the process of the attack and focuses in on step zero. Quote, original maintainer burns out and only the attacker offers to help. So attacker inherits trust built up by the original maintainer. End quote. Somebody found an email thread that captured the individual messages sent when step zero was taking place, and Rob goes through and picks out salient messages to paint a picture for us. Quote, First, we start with a reasonable request asked reasonably.

The question forces the maintainer to address his quote failings. I use quote failings in quotes here because A, the maintainer doesn't actually owe anything here, so he hasn't actually failed, and B, I know exactly how this feels. It feels terrible to let down your quote community. The question asked was, is XZ for Java still maintained? I asked a question here a week ago and have not heard back. The maintainer acknowledges he's quote behind and is struggling to keep up. This is a cry in pain. This is a cry for help.

Help will not be coming in this thread, end quote. The question quoted doesn't originate from Giatan. Instead, its author eventually points to Giatan as a good person to, quote, have a bigger role in the future. How many of us find ourselves in positions similar to Lassie? I've spoken with so many maintainers who would love to pass their project on to someone capable and interested, but it's darn near impossible. Rob closes with this.

It takes skill and knowledge to write software. And while many skills and some knowledge will transfer, working on a new software project inevitably requires developing new skills and more knowledge. Software developers are not fungible cogs that you can swap in and out at will. The email thread ends with the complaining consumers offering no help while continuing to make demands. Only the attacker is left. The debate. In the wake of this event, many voices have called out the unhealthy relationship between unpaid maintainers and companies that benefit from their work. And don't get me wrong, yes, that absolutely is a problem. But Substack writer LeCamtuff wrote up a different take that

I haven't heard previously that I absolutely believe plays a part. They say, quote, The real issue with a lot of small foundational OSS libraries is just that there isn't enough to do. They were written decades ago by a single person, and beyond bug fixes, they are not really supposed to change that much. You don't do major facelifts of Zlib or Giflib every year. Even if you wave some cash around, it's hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn't all that Unfortunately, sometimes that person with a pulse and some modicum of skill is a highly competent, malicious actor. Quote, More fundamentally, the XZ backdoor isn't a technical problem, and it probably can't be solved with technology alone. To a large extent, it's a counterintelligence challenge,

squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft. That's the news for now, but scan this episode's companion newsletter, link in the show notes, for more commentary on the XZbackdoor0 and 13 other interesting links that are completely unrelated. Have a great week, leave us a 5-star review if you dig it, and I'll talk to you again real soon.