The Changelog: Software Development, Open Source - Why you shouldn't use AI to write your tests (News)

Episode Date: May 28, 2024

Swizec's article on not using AI to writes tests, LlamaFs is a self-organizing file system with Llama 3, a Pew Research analysis confirmed that the internet is full of broken links, Sam Rose built a spectacular interactive study of queueing strategies & Jordan Cutler shares a real-life experience of him writing clear/readable code... and it backfiring.

What up, nerds? I'm Jared, and this is Changelog News for the week of Monday, May 27th, 2024, but delivered to you on Tuesday, the 29th, because Memorial Day here in the States. Hello to all of our new listeners who learned about this program on the Free Code Camp podcast. Sit back, relax, and enjoy the change-loggy goodness. For those who missed it, I spent almost two hours chatting with my old friend Quincy Larson on Free Code Camp about too many topics to list here, but I will play you a soundbite of one of my favorite moments that begins with Quincy saying they don't do clips of

their podcast and ends with me telling him he should make a clip. You know, with podcasting, it's completely different. It's like almost like this dark matter of the content world because it's so impenetrable. I mean, Spotify has done a lot to like create like automatic little clips and share it and stuff like that. And we don't generally do that. We don't like cut out like the highlights of the video and do that. I know you all do that. But I like listening to the conversation in its entirety.

Take it all in. It's like when I read a book, I don't go to the bookstore and get like cliff notes. I go and read the actual book, you know? I want to hear everything that this author had to say about the subject matter, right? And so I do think something's lost when you compress. Like there's no such thing as lossless compression with human communication.

I like that. That's a good quote. Okay, cool. Put that on a clip, Quincy. You should clip that. Put that out there. All right. I agree with you.

We clip for two reasons. There's a taste for you. Like I said, two hours, some good conversation in there. I'll put a link in the newsletter for those interested. Okay, let's get into this week's news. Why you shouldn't use AI to write your tests. This well-reasoned piece by Swizik feels like a punch in the gut. I thought writing tests for us was the perfect TDM-reducing role of LLMs.

He starts by listing out why people write tests, like to find related changes, to follow the Beyonce rule, which is, if you like it, then you should have put a test on it, to avoid talking, to trick programmers into thinking, and to catch regressions. After a brief survey of all those reasons, he asks, quote, do you think deriving tests from your implementation is valuable? The answer he wants you to come to is no. But if we do, what instead? Swizik recommends writing higher level tests.

An integration or end-to-end test higher up the stack would be way more useful. When a user clicks this button and enters a phone number, does the search work? That test won't need to change until the feature changes. You could swap out the whole implementation underneath and keep the same test. Well, that's a relief. We can once again have AI write our tests, just not the tests we're probably having to write for us currently. This reminds me of a conclusion I made back in my heavy test-driven development days. In my experience, writing the right tests was the hard part. Once you figure that out, the design of the implementation was pretty much baked, making it trivial to write the

implementation code. So maybe we're thinking about this all wrong. Maybe we should write the tests and have the AI write the implementation. Someone try that and report back. A self-organizing file system with Lama 3. I feel like we are starting to understand what LLMs are good at. Expanding text, contracting text, organizing text, etc. With that in mind, I imagine Llama does quite well at the otherwise mundane job of organizing the files on your file system. Quote, LlamaFS is a self-organizing file manager. It automatically renames and organizes your files based on their contents and well-known conventions. For example, time. It supports many kinds of files and even images through Moondream and audio through Whisper. End quote. Maybe point this at your downloads folder and see what happens.

The internet is full of broken links. A recent Pew Research analysis confirmed what many of us very online people already know. Some 38% of links from 2013 led nowhere as of October 2023. So that's news, but probably not new. Here's something else from the linked article that was completely new to me, the dead internet theory, which quote, posits that the web has been colonized by bots that are pumping out the majority of content that we encounter online. Indeed, some commentators now argue that we've moved into the zombie internet era, where once human-operated entities are now automated content farms

publishing AI-generated images. End quote. I think this is a good time to mention a page which hasn't changed since 1998. It's called Cool URIs Don't Change. Give that a read if you like. I'll link it up in the newsletter, but the title pretty much says it all. It's now time for sponsored news.

Monitor your home lab with Chronitor. You've probably heard my co-host Adam raving about how he uses Chronitor in his home lab. It turns out he's such a fan of the service that he went out and got them as a sponsor. Here's what Adam loves most about Cronitor. There's an easy CLI for Linux. It captures the status, metrics, and output from every job. You can name and organize each Cron job. It works with every kind of background job.

And it makes monitoring Cron jobs too easy. His words. Monitor your Homelab or your side projects or work stuff at cronender.io slash Homelab. For 10 bucks a month, you can monitor 30 cron jobs and websites, retain a year's worth of data, and more. Or use Cronender for free with their hacker plan and get five monitors. Check it out. Once again, that's chronitor.io slash homelab. An interactive study of queuing strategies. This is a spectacularly executed explainer on queuing strategies by Sam Rose for Encore. Quote, in this post, we are going to explore queuing in the

context of HTTP requests. We'll start simply and gradually introduce more complex queues. When you're finished with this post, you will know, one, why queues are useful, two, three different types of queues, three, how these three queues compare to each other, and four, an extra queuing strategy you can apply to any type of queue to make sure you don't drop priority requests. End quote. This is a must-read, nay, must-experience post. The interactive bits are expertly crafted to get the point across, while also invoking fun by enticing you to overload the queue being described. Bravo, Sam. Check it out by following the link in your chapter data and newsletter.

Clear, readable code is probably the hardest code to write. Jordan Cutler took the old axiom and put it to the test by taking a real-world pile of functioning spaghetti code and splitting it into 30 plus diffs that he described as a beautiful landing of the data enrichment module with easy to read, clear code. The end result, quote, while I was proud of it, there was suddenly a problem when I talked to my manager about it. The end result, quote, while I was proud of it, there was suddenly a problem when I talked to my manager about it. The manager says, quote, while I understand how complex this was,

when it comes to performance reviews, this code looks trivial. It looks too easy, too simple. I would recommend writing an implementation doc of this module just so we can demonstrate that this was actually quite complex. End manager quote. I was shocked. This wasn't some startup. This was one of the biggest companies in the world known for their engineering culture. I now understood why big tech seemingly

had so many docs. Half of the docs I wrote didn't need to be written, except they did, because I wanted to get raises and be promoted. End quote. I guess that's the trouble with clear, readable code. It's super hard to write, but it looks super easy to write. What are you going to do? That is the news for now, but we have some great episodes coming up this week. Scott Guthrie from Microsoft Build on Wednesday and Louis Villa from Tidelift

on Friday. Have a great week. Leave us a five-star review if you dig our work, and I'll talk to you again real soon.