The Changelog: Software Development, Open Source - Nushell for the GitHub era (Interview)

Episode Date: September 27, 2019

Jonathan Turner, Andrés Robalino, and Yehuda Katz joined the show to talk about Nushell, or just Nu for short. It’s a modern shell for the GitHub era. It’s written in Rust, and it has the backing of some of the greatest minds in open source. We talk through what it is, how it works and cool things you can do with it, why Rust, ideas for the future, and ways for the community to get involved and contribute.

Bandwidth for ChangeLog is provided by Fastly. Learn more at Fastly.com. We move fast and fix things here at ChangeLog because of Rollbar. Check them out at Rollbar.com. And we're hosted on Linode cloud servers. Head to Linode.com slash ChangeLog. This episode is brought to you by Linode, our cloud server of choice. It's so easy to get started with Linode.

Servers start at just five bucks a month for your big ideas. Head to Linode.com slash changelog. Choose your flavor of Linux that works for you. Then pick a location that's right for you. London, Tokyo, Dallas, and many other places in the world. They've got you covered. Go from having that amazing shower idea to a hosted website in just minutes. Start small.

Expand as your idea blossoms into a huge hit. And we trust Linode because they keep it fast. They keep it simple. Check them out at leno.com slash changelog. Welcome back, everyone. This is the Changelog, a podcast featuring the hackers, the leaders, and the innovators of software development. I'm Adam Stachowiak, Editor-in-Chief here at Changelog. On today's show, we're talking with Jonathan Turner, Yehuda Katz and Andreas Roblino about NewShell.

It's a modern show for the GitHub era. It's written in Rust and it has the backing of some of the greatest minds in open source. We talk through what it is, how it works and cool things you could do with it. Why Rust, ideas they have for the future and ways for the community to get involved and contribute. So today we're here to talk about a new shell. It's called New Shell, which is fitting for a new shell. I suppose as it gets older, maybe that name needs to be updated. But we have a whole bunch of people here.

The whole team, it seems like, the whole core contributors, Andres, Yehuda, and Jonathan. Guys, thanks so much for coming on the ChangeLog. Thank you for having us. Thanks for having us. Thank you for having us. So, big endeavor. It's time for a shell.

I like how you cast a modern shell for the GitHub era. Whose idea was this, and how did it get started? Jonathan, you want to take that? Yeah, I can give that a go. So I think a couple of things happened all at the same time. So Yehuda and I kind of talk pretty regularly about different ideas of things that we want to try. And he was going on a PowerShell kick and showing off, hey, look at what you can do in PowerShell

and then the next day he would show something else and the next day he would show something else. Like, wow, there's a whole untapped arena of ideas around how we can make shells better. And I started working on a little shell project, he started working on a different one and we said, well, let's just merge this together. And that was kind of the genesis of this new shell project, was taking

some of these ideas, but we're both Rust programmers and kind of seeing it in the Rust direction or a functional programming direction. Yeah, and I would also say that PowerShell is cool. I really enjoyed it. I wrote a bunch of blog posts about it, actually. What was a little underwhelming and disappointing about PowerShell was that it just felt like it was written in 2001

by somebody who was kind of getting their feet wet with programming language design. So the example that I usually use here is it's dynamically scoped. Emacs Lisp is the only other thing that's dynamically scoped. And you might not think it matters, and I don't want to spend a huge amount of time explaining why it matters but like there's a reason that no programming

languages are dynamically scoped it just makes callbacks and abstractions very painful and it started to become apparent to me as I was playing with it in general it just doesn't feel super fun like the errors are just a big blob of red text I remember like one of the first things I said, Jonathan, this is cool, and he said, that's cool. And then he typed something in the shell, and both of us use Windows, by the way, so he just went and typed something in the shell

and it spewed out this big block of nonsense red text. And it was just like, this just doesn't feel, given that it's 15 years old, it's surprising how many things don't feel that modern. Well, from a person who uses Bash every single day, all day, it doesn't feel modern at all. But I also don't have that twinge of I need something new. I think you just get comfortable.

And I don't even know that I need a thing like New Shell until I start reading some of your examples. And I'm like, oh, I do want this in my life. I need this thing, yeah. But a huge undertaking. I was going to ask why Rust, but it sounds like because you're both into Rust first and foremost.

That's sometimes a reason. I think that's the main explanation, but I think one thing that I really want to give Rust a lot of credit for is that Rust is both low-level and fast, but also very portable. So that means that on balance, if I want to get a crate, which is the name for package in Rust,

for the thing that PS would have to be implemented in, on balance, the best-of-breed Rust crate supports Windows and OS X and Linux and maybe FreeBSD, whatever. So I think it might not have literally been possible for us to quickly prototype this five years ago because they're just doing, like we could have done it in Mac or something. And there's like tons of projects like that.

But the fact that this, like we use Windows and it kind of just worked on Linux, it just kind of just worked on OS X, that's a good reason to use Rust. You're going pretty deep on PowerShell, Yehuda. You had a lot of great blog posts about that. Yeah. I mean, I think among other things, it's good if you're going to...

So I didn't know right away that I was going to write an alternative. But if you're going to write an alternative, it's probably good for you to know exactly what is not good enough about the thing that you're alternating. Among other things, I just think that Windows is hiding a lot of really useful things. Like the theory of Windows is like everything is OO, more or less. The theory of Linux is everything is a file, more or less. People like to make fun of the registry, but it's probably not bad that the console API is an object with methods on it, instead of a bunch of 1960s escape characters. It's probably nice that console.go up is a thing.

So Andres, are you also on Windows? Are you also into Rust? Or how did you get involved with these two? The story is interesting. I've always assumed that when Yehuda says something, he's doing something. He has a plan. So in this case, it's due to him that I learned Rust maybe pre-stable 2013,

but not really got that deep like both Jonathan and Yehuda. They went all over the place with the language, even becoming part of core teams there. So I was working on a project with Yehuda a couple months ago, and I noticed the PowerShell posts. And then again, I assumed he's into something, right? So we're just talking about the other projects here and there,

and I started noticing and asking about some other repos. And then he's like, I'm working on errors. Okay, let me see. And that's when I saw a different shell. And I just, I knew, I assumed. So my reaction was like, it wasn't even a decision. Like it was already a decision. Like you just see it and you believe in it, right?

So if we talk about films, I'd say like, you know, how do I join the Nebuchadnezzar now? One thing Andres is not giving himself enough credit for here is I first met Andres, I had a project called Tokaido, which was basically just an attempt to make a better OS X Ruby installer. It kind of morphed over time, but more or less that project is why statically installable Ruby exists on OS X now. That was more or less the consequence of that.

It was a Kickstarter project. And I had gotten to a point where I kind of got everything working, but I just really needed help. And Andres showed up and did really a massive amount of work and got the project over the finish line and continued to maintain it. And that was like, the project of like, why is Ruby not statically installing itself super well on OS X

is not like an easy project. It was a hard project and Andres did really good work and that's how I met him. Oh, thank you, sir. That's how we met online actually yeah it's funny because it's it's funny it's it's very it's interesting it's interesting because during that time that's how i was learning more about uh yehuda so to speak he was doing he got into rust one of the first who had a software written rust in production he told me about the

language he had other ideas and I regret at that time, not getting into that much because he mentioned to me something about Ruby.rs. And that was six years ago. And now you have another project called Helix, which is that. Like an idea became reality. And it's basically a way for you to Ruby DSL-ish write extensions that run with Rust, interoperate with Ruby code. So it's for optimizing hotspots in a given application in an easy way. So we work on that project, and we officially met in person at RailsCop last year.

So that would be maybe four years later. That's true. Sounds like me and Jared. That's right. How long did we work together before I came down to Houston? Probably years. Years?

Well, two or three, maybe. Okay, at least a couple. Two or three. How are the internets? How can they connect us? That's right. And it's very funny how that even happens. And I appreciate the invitation

from Tilda, Yehuda's company. Lee gave me a pass to get to the conference. And the reason of all that was because of Jonah, Yehuda's son, he accidentally dialed the number. He does a lot of that.

He does a lot of that. Yeah. I guess it's not really a butt though, is it? Yeah, like toddler dial. I told him I'm a computer scientist and he told me he's a tablet scientist. So I see the number. I'm like, hello? Because I recognized Lee in the picture.

It was from her phone, right? So I said, you know, I was thinking about I was going to be in the Pittsburgh area. So it'd be nice if we could have lunch or perhaps meet because I have to be some other states around those states and that's how it happened and we literally

met in person. That was fascinating. Quite a story. And now this, I just saw the source code. Again, I didn't have to think. I assume,

usually Yehuda's design ideas are good. There's some great decisions. Apply really well in practice. And I saw it. I mean, why do I have to think about not working something together? Because I was told, you know, let's work on something. But I saw it, and I don't think I even need to find arguments or any reason not to get involved.

So I just went and never stopped. And that's how it started working for the second time. That's awesome. You were all in. Yeah, I'm sorry. So, yeah. So right now we're answering the GitHub era question. Right, Jonathan? Right.

Well, yeah. A couple of people were giving us a little bit of guff for the GitHub era. They're like, what does the GitHub era even mean? I think it seems pretty... To the rest of us, it's like, haven't we been in the GitHub era for at least the last five years? Yeah, and somehow we have no new shells in that entire time.

Right, I think that's where it really comes in. It makes sense there. It's not so much that it's pompous to say, it's the fact that there is nothing new that's sort of come out to embrace this ecosystem being there and the accessibility of things like that. So it makes sense to embrace this ecosystem being there and the accessibility of things like that. So it makes sense to me to say it like that.

Yeah. I always feel like if people want to complain about stuff like that slogan, they're probably, like, I have better things to focus on than whether it's really the GitHub era or whatever. And I'm happy for those people to spin their wheel. What's a runner up to that then? If that was the tag people to like spin their wheel what's a runner-up to that then if you if that was the tagline y'all chose what's a runner-up I think we considered stuff like playground in a shell and stuff like that which I still like as like secondary secondary taglines well one thing about it is that it's ambitious and this is something that I think marks a lot of the stuff you do Yehuda is like very ambitious projects and I would just say that one of the things you do, Yehuda, is very ambitious projects.

And I would just say that one of the things that we feature, I mean, Adam and I are impressed by ambition and people are like, you know what, I'm going to take on this thing, which is a big endeavor and potentially difficult endeavor, but has a lot of wins at the other side of the road. One, I guess a surefire way to get on the change log is start an ambitious project and then tell us about it. We'll just invite you on. And so kind of what you did, you guess a surefire way to get on the change log is start an ambitious project and then tell us about it. We'll just invite you on. Kind of what you did, you started a project, you told everybody about it, and we saw it. We're like, all right, come on out. One thing that I think is great about Jonathan

and why I like working with him is there's a lot of people with ambitious ideas. You'd be surprised how many people come out of the woodwork to be like, here's a document I wrote that explained what a better shell was. And there's a lot of people who start doing them sort of in a repo somewhere. But it takes a lot of discipline to actually focus

on the right small MVP. And Jonathan is really the best person I know to be like, no, that's a real problem, but we're not going to solve it right now. Yeah, that leads directly into what I was going to ask him. I'm glad you said that, because the question for you, no, that's a real problem, but we're not going to solve it right now. Yeah, that leads directly into what I was going to ask him. I'm glad you said that, because the question for you, Jonathan, is, okay, you have this idea, you know you want to do some cool things

similar to the way PowerShell is, but fix these problems, and you know you've got this ecosystem of existing shells and the Unix philosophy. It's shortcomings. Where do you start then? It's like, okay, let's do this. And you have your editor open, and you're like, okay, I'm going to write a line of code. What do you do? How do you get started? I mean, I think that's a great question. So when we were starting to hack on it initially,

I think a lot of the early time was spent just me and Yehuda throwing ideas around. What if it did this? What if it did that? And like Andres said, Yehuda has tons of good ideas. And his design ideas come from really a fundamental place. So they have a lot of good gravity to them. And so if he's like, oh, we should go this way, there's a good reason. And you should follow that conversation. And so we just did one conversation after another. And I also kind of putting on my PM hat. So I was the program manager for the TypeScript team for years, you know, starting at its

first public release and then growing it through the use of TypeScript and Angular. And, you know, the discipline of growing a small project and then into a much larger one, I think you kind of get used to working with, okay, what is the MVP for this release? Like, what are the things we actually want to hit? That's basically where it kind of came down to it is we saw a new shell, and we saw all these ideas, kind of a garden of ideas, and we could kind of cut through that garden. All right, this is clearly enough to show off the value of what a new shell would work like. Let's work with new file formats, or let's work with existing file formats.

Let's build the pipeline. Let's build it in Rust in a way that is really composable and has plugins and that kind of thing. So we could kind of carve out some of those main features and that's enough to really, once you could see that, it's like I tell you part of a story and there's enough detail that you can fill in the rest in your mind. I think that's some of the philosophy of an MVP.

I think a good example of where it's not, there's a version of what an MVP is that you read in blog posts or self-help books where it's just about paring down. I think there's a good example in how the evaluator works in New Shell where I actually wrote a real programming language interpreter, but we didn't write scripts. There's no scripts in NewShell. There's no script file.

There's no parser that parses more than one expression at a time. But the way that it is implemented is like a real evaluator, which means that things like LS pipe, where CPU is bigger than 10, is actually turning where CPU is bigger than 10 into a block that really looks like $it.cpu is bigger than 10. It's really evaluating. It really has a real scope object, right? Now, that's not the right way to do a programming language implementation

if you're trying to make a fast programming language, but it is a real programming language implementation, and what that has meant is, number one, we didn't waste our time with scripts, which are going to be important, but are just like not what 99% of anyone is doing in a shell, period. But second of all, as we've added more features,

there was sort of a natural way to add them. Because it's like, oh, you want to add a binary expression? I know how to implement a binary expression in an interpreter. Like, you do that there. As opposed to maybe like a regex for $it or something. Could have been a different, or some other, like, we'll just write a posix shell and we'll just treat each thing as a bare word and we'll do some thing in every command, right? That would have been a valid, like, self-help book MVP. But it would have meant that when we got a whole bunch of interested contributors who were trying to add stuff, they would not have succeeded.

So I think there's a balance that I think I care a lot about and is kind of quintessential to how I think about how to build software that is not just about purely just paring it down, but is also about being pretty willing to pare down. You only do the things that are the highest leverage that are the most essential. Have you guys heard of, maybe this is a good one for the listeners too,

there's a thing called the MVP pyramid. Maybe we can have a link or something. Link in the show notes. Yeah, so an MVP pyramid is that a lot of times if you think of a pyramid and you have layers of a pyramid going up as it goes higher, it gets more narrow and whatever. But each one could be some core feature.

And oftentimes when people do MVPs, they kind of do a couple core features and then stop. They don't ever build the pyramid, if you will. And a better way of thinking about an MVP is actually to slice more laterally across all the features. So you get a taste of each of the core things that you expect to flesh out

by the time it hits 1.0 or whatever. I got a pyramid up here. I'll read it so I can kind of maybe describe it visually to the listeners. So you have a pyramid at the bottom. It's got user segments, user needs above that, functional, reliable, usable, enjoyable. And what you're saying, Jonathan, is if you just take a slice down the left-hand side of that and you get a little bit of each of those rather than just a couple and not really a full-featured application or product. And for people to know, the whole point of an MVP is to sell a vision.

It's to sell a story. And a story is fundamentally about, the original MVP pyramid used the word emotional for the top one, and I think I'm partial to that. To tell a good story, it has to be emotional. It has to hit you somewhere. If you just make something that works, then it's not telling any story.

And you need to tell a story to get contributors, and users. So how long y'all been toiling away at this MVP? I think four months sounds about right. I have a self-imposed rule that I don't... That you don't follow? No, I try to. You have to

say the truth. The original commit is the day of your birthday. May 10th. We talked about this. May 10th, 19... My birthday is 82. I'm 37. So it was a long thing. I'm actually interested to see

I would be interested to plot the velocity, but I have a self-imposed rule that I don't, first of all, I only do work in public, so I generally don't, there's a lot of reasons people don't do work in public, which include things like people will see it and they'll annoy me and whatever. And my point of view, maybe it's like from Ember,

is you have to like actually do a lot of things to get anyone to care about what you're doing, no matter what. And if you do something in public, it doesn't matter. Like New Shell and Deno were both like public projects, and no one cared despite the pedigree until they were announced. And second of all, if somebody actually is interested, like Andres, it's like good that I didn't have to like figure out how to give him access, right? So if somebody somebody's super interested in very early stage work, sounds great. And additionally, it makes it really low friction to share it with friends, people who are generally interested. I could open up somebody's GitHub.com on their computer and show them.

And I think all that friction is annoying in the beginning. And Secret doesn't buy anything, so that's rule number one. Rule number two is I try to ship, I try to make an announcement within three months. It sounds like June, July, August, September, like I think we hit that target. And the basic idea is not that you have to announce something that is ready, like ready to use. You have to announce something that's like ready for more contributors. And I think it like actually is reasonable that it takes a few months of, you know, the MVP pyramid is like, there's also a version of that in tech, like you have in the tech, you have to kind of lay out the principles in code, right? So this is how pipelines work. So if somebody comes to add another feature that builds on pipelines, if the pipeline

feature isn't laid out, then all the contributors coming into the project are going to make it really hard to advance the core primitives, right? So I kind of like, I want to build stuff. I want to build the core primitives quickly. But again, Jonathan is great at this. I've got to be really ruthless about what that really means, because I am not smart enough to do everything myself. I desperately, there are definitely people who do. And I am just desperately, desperately in need of other people to help me do the most basic of things so i think getting like i i have a basically a deadline of approximately three months we talked jonathan and i talked about this in other projects we kind of spiked out in the past and i think that works

like if you think about it almost nobody does the thing i'm saying and i'm maybe like if you the objections that you might imagine in your head to why you shouldn't do it probably apply more to me than most people, and yet it works for me. I think this is a good recommendation. It keeps you honest, keeps you shipping, it avoids you making secret things that are only in your head for a year. It kind of reminds me, what was the article called? How Not to Die? Is that right?

I find it ironic. So Paul Graham, it's called How Not to Die. Jonathan, you can tell the story and I can say why I find it ironic separately. Okay. I mean, I'll try to do a decent job. So the How Not to Die is Paul Graham giving advice to startups and how to continue going, how to not die as a startup. And this applies to software projects and open source too. And one of the main,

to me, one of the main takeaways from that article is this idea of smoke signals. Like if you're looking across the hill and there's a smoke signal, you know, someone's still in the cabin and they're still, you know, set in the fireplace on fire. But at some point, if there's no smoke signal, you go and check on them and knock on the door, right? And that's true of open source projects too. If they're not releasing new releases or blog updates or tweets or some signal of life, then it's really easy to assume that that project's dead and all the momentum behind it just starts to peel away. Right, and I think a couple things.

So he said a counterintuitive thing, which is probably true enough, which is, therefore, if you can convince yourself to keep working, you are not dying. Even though it seems, it seems like you think you're alive or dead based on like funding or resources or time or something. But like, in reality, if you could just figure out how to keep working, you're not dead yet. And then the other thing that I, that, so this essay is like my, like one of the most influential essays I've ever read. And the other thing is that he just says, you have to prioritize it.

Like you can only prioritize like one or two things. And it's weird to prioritize staying alive as a thing, but, and it like, nobody wants to. But like Tilda, the company I started with Leia, also has a priority of like, don't die. And that like supersedes, like adding features to the product or whatever, right? Like staying alive matters a lot because if you stay alive for enough time, it's good.

The thing I find ironic about it is that that advice was meant to be counterintuitive compared to like what other VCs are telling you, right? It's like most VCs give you some money. You last for like 18 months. Instead of telling you don't die, they tell you like triple, quadruple down, like keep taking more and more and more risk. And like maybe you'll get lucky. And if you're not, it's not our problem. Like you'll get acquired or whatever. And the point of the essay was to like, kind of tell you to focus on survival. And what I find ironic is I really believe that Paul Graham believes it.

But ultimately, Y Combininator is not particularly different here like companies that get funded by y combinator have the same 18 month or 24 month boom and bust cycle as anybody else and i think that's like too bad right i think it would be good there's a vc called indie.vc by the person who actually was the vc of the last company that i uh was part of that boomed and bust. And his thesis is basically like, I want to invest in sustainability. And I just really wish, like that story, that essay suggests to you that Y Combinator should

be that, but it isn't. So I think Jonathan has done an excellent job in terms of the whole thing, really, especially the management of the project itself. To be frank, also, if you haven't checked, the MVP has been sliced. I'm so used to slicing right now the word. Why? So there are MVPs number two and number three.

There is theoretically a roadmap, right? And this is good because if you see the issues, anything that shows up following Jonathan's practices, file it, just file it. Or metadata it and merge issues and referencing the single issue as the Huda started doing and tagging that in tags, first issue tags, good preparations for people that might be interested in contributing. And it's important I say this because since the announcement, it's like out of

nowhere, we get issues all over the place filed. At least that's what I think we want to fix the issues as soon as we can. So if you think about it, it's easy to track for the next MVP. The target is there. So Jonathan has been doing well. I enjoy working with him because clearly we know what needs to get done. There's a sort of tension between what I sometimes shorthand 8D, the articulate the vision of like, okay, we have a roadmap, we have a plan. Like we really want people to join the project who more or less share the vision that we have. If somebody like, let's say someone is like, oh, NewShell seems cool. I like your UI, your output, but I really wish it was based on tech streams. Like they should definitely not

join the project, right? And so I think there's a sense in which you have to really clearly articulate the vision. On the other hand, what people are excited and enthusiastic about doing when they show up and try to contribute, of course, has a big impact on what actually gets done. And therefore, you want to shape the roadmap based on that. So there's sort of like a push and pull between like laying out as much of the vision as you can to like attract people who really share the ideas and then being willing to be pretty flexible as the, you know, if someone shows up and they're like, the only thing they care about is making it work on FreeBSD, like FreeBSD is going to be in the roadmap, right? Because that person is doing a lot of work on it, unless it's like making

everything else really hard, right? So I think there's a sort of, it's again, not a self-help book, right? There's a push and pull between letting people drive and not articulating. And Jared, just like you asked about the GitHub era thing, the most interesting pull request for the moment is a suggestion of changing the motto while we work. That's a very GitHub era. That is a GitHub era issue.

I feel like we closed it. I think MVPs, releasing them, having worked sufficiently gives a message. It's perfect for getting feedback for any type of project, but this one is more important, and Jonathan has done incredible work by writing a plugin system, for instance, defining an API. That's very important.

That lets you give the message for new contributors to see how things work the book as well everything everything everything everything we should probably get into some of the technical stuff but um i actually think so jonathan did the work and was like i have to admit when i i was definitely not like yes let's make plugins was definitely not the top priority that i had when i was working on it at the time the thing that's cool cool about plugins for anybody interested listening is that plugins, basically almost any command can be implemented as a plugin. A plugin is just a binary. It's like a exe or elf executable, whatever. It's like a binary that runs. There's a JSON RPC protocol that we communicate with. And so if you want to implement

like LS, it happens to not be implemented as a plugin, but LS could be implemented as a plugin. And you could either use the Rust, like if you're excited about Rust, and again, I think that's the easiest portability story, you could just like write a plugin, you could just copy it from any of the many plugins that already exist. But if you want to write it in like Python or Ruby or something, there's not like great CalPAS right now, but there's nothing stopping you from doing it because it's an executable that's on your path or in some place and communicates over JSON RPC. So basically the idea is that instead of making every single person who wants to work on a new command have to contend with a big code base

that's the internals, we can basically make commands be something that anybody can go write. Jonathan, what's a good example of a command that somebody wanted to write and just came and did? That's a good question. We've got different sizes of things. So we've got really tiny plugins. So for example, if you open a JSON file and there's a version number and you want to just

increment the version, then you could just write that as a plugin. And now it lights up for everybody. So because we haven't really talked about this part, but it's a really important part of the design is that the way that New Shell works in a nutshell, I'll try not to use the pun. Too late. Yeah. So the way that it works is that we have a set of things that produce values. So I can load from files. I can load from a URL.

I can, you know, I can load from a URL, I can run LS or PS, and these generate tables of data, basically. And a table is like our core fundamental value type. So everything produces tables, all the filters work on the tables. And then what comes out is some way of viewing or saving the data. And so if I add the ability to increment to a plugin or the ability to view a different kind of table data, that kind of lights up for everybody because everyone's using the same value type.

And to be concrete, that means that you can open a cargo.toml or a package.json or basically any type that we type that we support can now, you can now increment the version or look at the version, like in the same way. This episode is brought to you by Get Prime. Get Prime helps software teams accelerate their velocity and release products faster by turning historical Get data into easy-to-understand insights and reports. Because past performance predicts future performance, Get Prime can examine your Get data to identify bottlenecks, compare sprints and releases over time, and enable data-driven discussions about engineering and product development. Shift faster because you know more, not because you're rushing. Get started at getprime.com slash changelog. That's G-I-T-P-R-I-M-E dot com slash changelog. Again, getprime.com slash changelog. I think we should hover right here and just talk about cool stuff you can do real quick,

because that's going to get people... Talk about getting concrete. What's some cool stuff you can do with News Shell that I can't do in Bash today or require scripting? Usually the demo that I give is, first of all, if you just do ls, you get back a table. In some sense in your mind, you're like, oh, I already can do that. And then you can do ls where size is bigger than 5kb.

So that's a thing you can type. And now you're thinking, oh, that's, I kind of can, like if I do a lot of man page searching, can figure out how to type that in LS. And then you're like, okay, well, what about PS? So you're like PS, you get back a table of data. You can say PS pipe where CPU is bigger than zero. That gives you all the processes

that have any CPU activity, right? And then you say, okay, I'm going to look at, I'm going to say open cargo.toml, and I'm going to get the dependencies and say all the ones where the version is bigger than something, right? And then you say, okay, I'm going to look at, I'm going to say open cargo.toml, and I'm going to get the dependencies and say all the ones where the version is bigger than something, right? And basically, any one of those individual things is not new in the sense that there was probably some cryptic set of things you could type or some other thing that you could possibly have used, JQ people say, once you figure out how to get it massaged into the right shape. But once you learn where CPU is bigger than zero, where size is bigger than 5K, now everything

works like that. The world is basically a bunch of Excel spreadsheets. And I think that is the capability. Absolutely. And so the where, those are the filters, Jonathan, you were talking about. And so when you do LSPipe or PSPipe, you are sending about. When you do ls pipe or ps pipe, you are sending the where command, I assume it's a command, but whatever it is,

you're sending this data. It's not text like we're used to. It knows about this tabular data and so it can filter it. It's a stream of rows. You look at it, it looks like a table, but it's's a stream of rows. So basically, you look at it, it looks like a table. But it's actually a stream of rows. So that way, I think this is really important if you say there's a command that's first five.

So you can say, give me the first five of something. If you were required to get the whole thing and you said first five, then there might actually, people have talked about things like, oh, there's Azure events. I would love to have a command that basically gave me a stream of Azure events, let's say. And that would be a table, but if you had to get

the whole thing before you could say first five, then you would never be able to. It's like the head command. So it's a stream of data that the command does whatever. There's some that are first that work on the whole stream, and then there's things like where, which operates on individual rows one at a time. What about commands that don't have this kind of data,

they just have the old school style? Can you push it into the right format somehow? I think Jonathan can say more, but basically the way to think about it is there's a lines command, which basically takes a big bunch of text and converts it into a table. There's one column in the table called value, and then there's just like a bunch of lines. And then there's more commands like parse,

again, Jonathan knows, which basically allow you to take the command, take the individual rows and convert them into a real column or format. So the key thing is that whatever input you have for a given command internally, it has the same structure no matter what. The same abstraction.

So internally, you always have in the code a view of a table. And you can decide how to access things in it from the code perspective. And you can either process the stream per record or as a single thing the whole

thing so anything works based on that so for instance the where command ls gives you the it's represented the output as something with structure a table and the where command what it does is that takes uh per object conditionally tests it against the value, and gives you the result back. Depending if it wants to accumulate the whole thing, gives you one single value, or returns just another string, which is always the same representation internally with the same structure.

And for plugins, I'd say, because, for instance, Jonathan added fetch for fetching URLs no matter what format if it's supported it will create the table for you if you fetch from URL a JSON a TUML file, a YAML file XML file it doesn't matter it will just show you a nice table

just like LS does just like PS does and you could just wear in it or get fields in it but if fields in it. But if you think about it, fetch, it's HTTP get requests. He also wrote post for doing post requests. So those are two different commands. Then you start realizing that you can make a single command

that does the whole enchilada like curl. Now, if we think about it since the project is maturing, how about if we write a plugin instead that has all these Swiss Army verbs, call it HTTP, for instance. That would be a good use case for a plugin, for example. Jonathan, would you like to chime in?

I pasted something in the group chat that the listeners can't see. We'll put it in the show notes. Maybe one of Adam or Jared can describe it, or people can look at it in the show notes. But it basically shows it's like a very small one-liner to go from blob of text to structured data. Go ahead, Jonathan.

So I think as far as being able to work with the outside world, we were talking about that MVP pyramid before, and each of the core features. We need to touch on the MVP pyramid aspect of it. So one aspect would be being able to work with the outside world. If we can't work with the outside world, we're a walled garden. And that's cute. But that's just a demo. That's not really a product. So being able to slurp in data from, all right, I'm going to run Git and whatever Git comes back with, I need to be able to work with it. So like Yehuda was saying, there's a set of verbs that are intended to be used to pull data in from the outside world and convert it into a

table for you. And that's something that we can allow you to make aliases for or functions for in the future to kind of quickly say, all right, run Git, structure the data that's coming from these streams of strings into a table, and then use it going out. And of course, the same is true on the output as well. So we can go from a table, we can pull data out of a table and then call into external commands using the stream of text that was sitting in that table. And from a user perspective, what this effectively amounts to is that when you say open cargo.toml or open package.json,

for the most common situations that people have heard of or use, and that includes any files, a lot of common cases, you just get a table. You don't have to figure it out. And I think what we kind of expect long term is that the format system itself will be like a plugin system. So for example, the.gitconfig is in a particular format or.gitkeep is in a particular format.

And we would expect some of those things to be built in, but a lot more of them to just exist in the world for you to install. One thing that I thought was really cool, and it seems obvious once I read it, is in terms of user adoption, you have this MVP set of commands that you all have implemented,

and then plugins and all that stuff will be built out over time. But if you do a command that isn't supported by New Shell, it just shells out directly to, it has a fallback. So on Windows it's CMDD or on Unix it's Bash and then it just passes standard out, standard error, all this stuff and then sucks it right back up

and it's kind of. So Vim works, right? Yeah, exactly. So that's the thing is like on adoptions, like these are all cool things, but when I fall outside of that, the stuff you support, I don't want to just give up my current workflows. Yeah, and Jonathan was like aggressive about this, actually.

I think a really important principle is that their escape valves have full coverage. By the way, Vim is an example. Maybe listeners might be thinking, I don't really use Vim. But it's actually pretty common for people's Git commits to open up Vim and workflows like that. And if Git commit doesn't open up the editor that you're used to, that's really a fatal problem. One of the last things I did before the MVP was add an escape file where if you put a caret

before any command, it forces it to be external. So maybe we have ls, maybe you want the real ls. So caret ls does that. And Jonathan has been really solid about making sure escape files were full. And you got the feedback immediately from countries in Europe saying, it's almost impossible

to write from... Which one is the carrot? Yeah, the carrot is not... Somebody should make a list of ASCII characters that are in the intersection of all keyboards. Let me see if I can find it. That was interesting. So this is something I haven't heard.

So certain keys are just not on these other keyboards or they're hard to reach? No, there's literally not there. Some European keyboards don't have, yeah, some of the keys. What's wrong with these keyboards? Get all the keys on there. They need extra characters. They need other, yeah.

I think the bottom line is like, there's a sort of intuition that people have about which characters are permitted. I honestly have skepticism that Carrot is a bannable character. I think Carrot actually shows up in a lot of random things. I use a program, I don't remember what it's called. It's called WizKey on Windows where you double hit control and then it gives you all the characters. And I suspect if you use a keyboard like that,

you will end up with a solution like that for some characters. I can't be sure. I think I'm always pretty open to hearing that something is very painful for people and we should consider some alternative. Would you all say that this table UI is New Shell's secret sauce, for lack of better terms? It's sort of

core underpinning of what makes it what it is. I think that that definitely gives it a lot of the power. Some of the other cool things about it, we were talking about the plugins, so you can just create a plugin in Python.

Ultimately, that's the idea. And that gives you all the access to the same, you know, tables that we were talking about before. So I think there's a couple of pieces that work together to give you that secret sauce. What I'm trying to get at is, how did you get there? So if this is such a great way to display and kind of fine tune data, what's returned to the prompt and stuff like that. How did you, like, what did you have to go through to get to that realization?

How many other things are on the cutting floor? So it was a realization from PowerShell, first of all. So PowerShell sort of does it. One of the things that was annoying to me about PowerShell that, like, motivated me to work on New Shell in the first place is that I think PowerShell just doesn't bubble down that hard on any of these things and so it's like oh well we don't really have array literals we have a syntax that effectively is an array literal oh you want a nested array of arrays like good luck there's like many stack

overflow questions about it and the same thing is true about tables there's like tables and lists and like depending on what ends up happening you it's very easy to fall into a situation where you expected a table and didn't get one in PowerShell. And so I really like tables. And I think Jonathan convinced me early on that we should really double down on it. But I also want to point, there's another reason. And I think for me, this is more fundamental, which is, I just think people understand tables. I think spreadsheets are maybe the universal programming paradigm in all software. Everybody has used Excel.

Yeah, Excel changed the world. One thing I'm hoping people get from NewShell is a realization that a lot of the vocabulary, the way that people think about data modeling could often be turned into a table-based paradigm. Not because it's perfect, but because it's so universal. I was thinking about it this way too. So I did programming, my gosh,

I've been working on programming languages for 14 years or so. And studying what makes a programming language popular? Like, what do you do to make it rise in the ranks? And if you look at the programming language ranks, you always see things like, you know, Java or JavaScript at the top. But no one ever really measures the real one, which is Excel, because it's an order of magnitude much larger than any of the other languages. It's like you're saying, it's so ubiquitous, it's so common, you don't even think about, hey, wait, that is a data language that we've been using for however many decades at this

point. Honestly, one of the mind-blowing things the other day is we need object literals at some point. We don't have them right now. It's actually been somewhat of a revelation, like how much of a shell is a few small number of things. But anyway, we realized that row is actually the same meaning as object, and table is the same meaning as array of objects. Actually Tommel has some of the same, it's not totally doubled down, but Tommel calls

dictionaries tables as a vocabulary matter. I think thinking like row, a row literal is the same thing as an object literal. I'm trying to avoid saying, a lot of this was known to us intuitively early on, but seeing it being crisply true, rigorously true, is a revelation. So Andres is over there. He's pasting into our chat just cool stuff that happens. Curl is not on new, for instance, so it shells out. You get the output from that, and you process that string

from JSON that builds a structure for your table, and the get command basically gets the field from the structured table. So if you want to think about it at a higher level than mapping, you know that JSON, it's an object or many object, and get would be like, you could put a path in there. You could say, if you have a root object called user on a JSON, you could say get user dot property dot property, and it basically translates as an access to the table for a given row. If you have a root object called user on a JSON, you could say get user.property.property,

and it basically translates as an access to the table for a given row. Yeah, so we'll also include that in the show notes, but I'll just read it. So the command here is curlapi.github.com, so you do that. And curl, that's not a new shell curl. That's just the curl curl, right? That's the standard curl, yes. You can pipe that into from JSON,

which this is one of your built-in commands. Which is a command that processes as a filter. It works as a filter. Two types of commands. Filters, and what's the other kind of command? Like an execution command? There's a filter? I think sync. We call them syncs.

I don't know if that's the long-term name. But the last part, the first part is like a source. The last part is a sync. Everything works in sync. Gotcha. And then the last one is get, and you pass it the string current underscore user underscore URL, which gets that out of the JSON that's returned from the API.

So that get command, that's a generic thing, though. That's not just for JSON. That's for anything that is in this format. That's right. For any given table. And let's assume if the api.github.com returned you a tumble file instead of our yaml file or xml file there are equivalent commands inside new for the moment called from dash the given format and what the command it does is it takes the output from the from what's being piped into it and

gives you back the structure. So that just like LS gives you, LS gives you back a structure. And then you can manipulate the data and that's it. You can even do get something, where something from the very beginning from the pipeline. And I think it's really important that these commands from JSON exist, but I also want to just say if somebody is actually doing an HTTP request, the open command, which is like the general purpose

thing that opens files or whatever, it does effectively the same thing, but it also looks at the content type of the response and automatically processes it unless you type dash dash raw. So if you're actually working with HTTP APIs, like open will save you a little typing and not make you have to know what exactly it is.

I think it's called fetch. Yeah. Yeah. Yeah. We just Jonathan, that's not fair though. Cause we, we just did that like yesterday or something. It might bring confusion. So we left open for one thing and we left fetch for, you know, we didn't even thought about the get command that it might get, do the actual get request of something because there's the post command that Jonathan did

that does a post HTTP request. There's another like somewhat important technical thing we didn't talk about yet, which is basically because streams are so like, so I don't have a particular horse in the race, unlike the paradigm wars, but streams like really lend themselves to functional style. And what that means is that there's like in the current system, and I'm going to like work really hard to make this true as long as I humanly can. There's no mutation in the system at all, but there's like mutation in the real world, right? So let's say you're trying to open Cargo Toml

and bump the version number. It wouldn't do anybody any good to be like, well, you can produce a new Toml, but like, sorry, there's no mutation. Like that's not a very useful thing. So the way that we see the world is you can open something, you get some data, then you can like functionally change it.

So the inc command, for example, increment, will let you say inc package JSON, or inc version dash dash minor or something. And then you can pipe that into a save command. So basically anything that's not in the system, so files or even like the configuration. The configuration, of course, has to be mutable. It wouldn't do anybody any good

if the configuration was immutable, right? That doesn't even make logical sense, right? So there are some root kinds of state that are like files, configuration, and I'm going to get to HTTP in a second, that are mutable. And so the idea is that the system itself is built around like ink doesn't,

ink gives you a new copy back. Right. But eventually you just want to save something so you can pipe to save, which actually does produce a pretty nice one liner. It's like open cargo.toml pipe to ink version dash dash minor pipe to save cargo.toml. Like that's pretty short. But one thing that's like pretty nice about all that is that if you don't pipe to cargo.toml,

you can just now you have it right now you have the output. So maybe you wanted to post it, or maybe like to a web request, or maybe you wanted to put it in the registry in Windows, you can also avoid specifying the file where you want to save after at the end when you save because Jonathan added metadata support and remembers the the origin. In this particular case with the plugin inc command, if you do that save on it without passing in the file that you want to save, it remembers from the beginning of the command that you opened that file and it'll save to that file.

Which is another thing, so if you use new, you'll probably notice there's really awesome error messages. I'm not going to say they're perfect, but compared to PowerShell or Bash, the error messages are usable. They're like a programming language. They're pretty good, but one thing is better. It's not a little better. It's not a little better. It just reminds me a lot of Yehuda's wonderful PowerShell posts, blog posts. So I think our errors are good.

One of the ways that we get really good errors is that, so I'm somehow a parser person now. I just do a lot of parser work. The parser basically maintains the spans, the original source locations of every piece of syntax. And then something that most languages don't do is that every value that gets evaluated keeps around the original source code. So if what you're, let's say you're, let's say you open CargoToml and then you try to like increment the whole table, right? That's, that's not legal. You can't increment a table. So you want to give

an error that's like, Hey, why are you trying to increment a table? That's not legal. And what we do is we say, okay, you can't increment a table. Like the original value came from and then it underlines open cargo.tom. Right? And so first of all, that's like pretty cool. And because we're not worried about being

the fastest possible thing, that's like a really nice thing. But second of all, that's the same metadata that allows save with no arguments to work, right? Basically every value that you ever see in the system, to the extent possible has a place where it came from. You know, I'd love to Jonathan tell the enter story. Let's hear it. Ah, yes. So we, um, one of the things we haven't really talked about is the importance for things to be fun yeah like we have these ideas we have these wild-brained ideas and it's okay to to chase some of them down because you can't you can't create a really interesting product by

saying all right i'm putting my pm hat on right and you're gonna check this box and you're gonna check this box that's boring no one to check this box. That's boring. No one wants to do that. I'll kill you. And that's exactly. You'll kind of burn out. So the idea that you have this little shell, let's make this fun for us too.

And I was laying on the couch one day, and I had this fever dream of an idea, which is what if you could enter into things that are not a directory? So I kind of wired this up as a demo, and I showed Yehuda. I said, look, I can now enter into a Toml file, enter into a JSON file. And what it does is that, you know, when you think about a shell, you can CD and LS around in a file system, right? You do that all the time. But why not be able to CD and LS inside of the structure of other files? So you can kind of jump into the JSON and like CD up into one of its fields and the LS and then see what that structure is in that field. So I wired this together and I showed Yehuda and he's like, wow, that's super cool.

And I'm not sure if it's useful, but it's cool. I left it there for a while and Andres kind of joined and was playing with it and said, this is cool. This is like one of those things that when you see it, you keep thinking about it. Like is one of those things that when you see it, you keep thinking about it. Why would you do that? That's weird. That's neat. Yeah, the same way as I saw the project,

I'm like, yeah, I mean, same with Enter. And I think you were about to... I even wrote a test for it, and it was right when Jonathan removed the feature. Yeah, you were about to remove it. I'm like, what? It's gone now. I'm looking forward to trying this. So the enter command, you get in, you get out. And we were trying to button everything out for the release

because it felt like this is cool, but it doesn't really mesh with the philosophy of, oh, it's a table. It's like values flowing through. We thought we had to be nice and regimented with our MVP." And Andres kept tugging at it and saying, no, this is cool. This is cool. Don't take it out. And he finally on chat one day said, why don't you just have enter, enter other directories, and you can think of each directory that you enter. So not cd, but enter.

So if you enter a directory, it just adds it to a ring buffer of other directories. So now we can have all kinds of different shell types, if you will. Some are just a directory that you're in, and you can hop in and out of that directory. So you might use screen, or you might use tabs in your terminal emulator, whatever to kind of simulate that. But in one shell instance, we can have you jump in and out of directories or in and out of a file. Or potentially in the future, we're working on, Andres is working on this help shell now, where you can jump into the help system and CD around and LS and ask questions, basically.

And you use the same muscle memory that you have for file systems, but you're working in other types of structures. And PowerShell actually has this feature, it's called a provider. It's just that the way to LS is get child item. Sorry, the way to LS in Windows is LS, but on macOS and Linux, it's get child item, which is the way to LS and Windows is LS. But on Mac OS and Linux, it's get

child item, which is a very long thing to type. I think there's actually two interesting, like two separate things that happened at once here that we conflated here for expedience, but I want to unconflate. So there is the like, like, what is LS? LS is really just saying list the current context. And like, why is file special special and the way that we solve that problem was with the there's a trait called shell which has stuff like cd and ls and mv and stuff like that right so in powershell parlance that would be called a ps provider but really all it is is like the basic functionality that you need to move around inside of a tree, right? And so that's one thing, problem that we were trying to solve.

The other problem we were trying to solve and the thing that like, that Jonathan was super excited about that I was like, I don't know what the point is. Sorry, I do that a lot to Jonathan and I'm sorry. Was like, if you make a second shell, like it seems like you want to go back and forth between them or like something. And it doesn't seem like the thing that I wasn't sure what was useful was like,

if all it is, is push the like push the impopty or whatever, if I don't like the ergonomics of push the impopty, right? So I don't actually want that. And what Andres figured out was, we can just like have another command called shells, and it could just give a list of them. And that's maybe not the perfect answer, but it makes it really useful. So if you're inside of the Rust new project, and you enter CargoToml, now you type shells, you see the new project and the CargoToml. You could move around the site CargoToml, and you could switch back. Maybe you're like, oh, in order for me to know the version, I need to cat the version file.

So you can go back into the main file shell, do that, and then come back to being inside of Cargo Toml. And I think just the simple, like that's a good simplistic, but very high leverage answer. The answer is just have a shells command. And as Jonathan was saying, now that we have the concept of a general purpose shell,

like the help system is just another way of navigating around. It's like you're basically looking at the metadata lens on the command tree. This episode is brought to you by TeamCity. TeamCity is a continuous integration and delivery server developed by JetBrains that helps software teams release their software faster, get fast feedback on every commit, quickly investigate build failures, and so much more. In this segment, I asked build engineer Oleg Garevich from Wargaming, who's been using TeamCity for seven years, about what he loves about TeamCity.

So I love how easy it is to manage TeamCity on a daily basis. I don't have to hack any mysterious XML to configure it or make changes, though there is an ability to do that. I choose not to. I do most of my work through the UI. I also like the fact that I can customize a lot of its behavior, either through the UI or through custom programs that I wrote or through plugins with their open API.

I don't think I could do my job without the support that TeamCity development team provides. And I use that support at least weekly, whether it's for new features that I'm interested in or for bugs that we find. They're very collaborative. And honestly, over the past 10 years, they've made my job so much easier.

I really owe them. All right. To get started with TeamCity, head to teamcity.com slash changelog to learn more. The professional version of TeamCity is free, even for commercial use. For large orgs, you'll want to check out TeamCity Enterprise. And right now, there is a 50% discount for our listeners on TeamCity Enterprise. And as a bonus,

if you want a personal intro to our friends at TeamCity so they can help you with your CICD path, email me, adam at changelog.com. Head to teamcity.com slash changelog to learn more and give it a try shells and shells so could this potentially i love how it started off as just a fever dream and ended up being like kind of a core thing that might make this better in terms of, could it replace a screen or a Tmux? Like if you had an interface into switching back and forth.

So another of the ideas was that that's how things emerged is that you could also enter other sessions remotely and do something like Tmute, so to speak. things emerge is that you could also enter other sessions remotely and do something like team meetings so to speak. This is required for structure for it. You would have just under the list

of your Shell sessions that you have in there. It can be any type. In the MVP for the next one, the next version we have a help system, a basic help system. I was thinking about it because, to be frank, on other shells, Bash, et cetera, I won't say PowerShell, but let's just say other shells, usually the help commands are just, I don't know,

we're all used to the pain for other shells. You get lost easily sometimes. You just get used to it and accept it. Accept reality. But I was thinking, because we still need to... This is something fairly new. We need to figure out also if users are going to use the features. This was what Jonathan was not convinced enough for the answer command. That's why he removed it, the original feature.

Like, okay, this is so cool. You can inspect a file, a TAML or JSON, but will people actually use this, right? It's interesting, but people might not use it. So what if of the all given shell sessions, there's also a help session, a help shell session, where you can explore the command and you can easily change

with the shell commands because it marks you what current shell you're at. So instead of when you need help on some command, instead of using the usual help command that you keep working, you have at your disposal

just another shell in there under the shell ring, and you can just cycle with next and previous, just a single command, and whenever you need help, it's easy, simple. I actually realized there's some pretty fundamental thing we didn't say yet, which is that this is a PowerShell thing but not a batch thing. Every command has a signature.

So command signature has its types and whatever and the reason i prioritized that in the first place was just to reduce the amount of boilerplate for errors right so if you say that something is supposed to be a path like it's not a path if you give me a number instead of a path that's not a path and then i give you a good error right i think we'll eventually add completion like we'll basically make completions sensitive to the types so if something is supposed to be a path it will types. So if something is supposed to be a path, it will support path completion. If something is supposed to be, you know, some other date,

it will do date completion. However, that same signature, which is now basically mandatory, like you could say that the signature, like in any programming language is like a list of anything, but that's pointless, right? That same signature could now be used in the, like as a column in the help table so basically that's cool

I just had this idea when Andres was talking I'm just going to put it here for the record we'll go implement it after this podcast we'll file the issue okay so check it out so if we give you help in a structured table-y way that you can work with which is our plan, give you help in a structured table-y way that you can work with, which is our plan. You type help and a command name

and we give you a structured table that walks you through it. We wanted to put examples in there. We could have each example as a discrete thing that you could pull out and then just run with another extension of the same command, which I think would be kind of cool. You could even make it have a templated example where you could just give it the variables to fill in.

Yeah, they're just coming up with new features on the spot here. I like this. That sounds awesome, let's do that. I think the answer is for basic kinds of Tmux usage, yes. The thing that I'm not totally sure how to deal with right now is as we discussed before, we care a lot about the Vim case working. And that means that we just wire up the standard out.

That's what we do. Basically what that means is, let's say you make a new shell, and inside the shell you do something like the equivalent of a tail command. That's going to keep printing output. You kind of do need a way to keep that running in the background and come back to it, which involves something pseudo-terminal or whatever. I think I don't find it hard to believe that we could make that work in some way. I just think that the thing that Tmux is very good at, that the shell system as currently implemented is not good at,

is allowing you to have persistent output that is not in your face. So like the Unix shell background system, backgrounds task, and then it turns the output in your face. That's not good. That's not what you want. I also say one more thing about this enter thing, which is that there's a feature that I still want

that is in the same space but not in conflict, which is the ability to, let's say you're inside of a Rails project. It's actually pretty common to just want to run like Rails S in the background, or like you're in Express, you might want to run like npm start in the background and then just like foreground your normal work. And I really want a way to like put a file in your repo that just says, like when I I start, like, make a background shell for for npm start, and then give me the foreground shell. And then we would use the shell system

to let you go back and forth between them. And I think like that feature to me is like a big, like for all the things that I personally do is like a big workflow enhancement. And I really want it. And that's, that's honestly why the enter thing in the first place was like, that's not quite also kind of related is I really want the ability, like, I think we're going to do this, but I want directories to be able to have files that give custom environment variables and path. So right now, like the path and environment variables are like a global thing. And then people might have like.env plus some extra hacks to make it work. I kind of want the ability to say like, here is the path for this directory.

Here is the environment variables for this directory. And then like new shell just makes it be. Like sort of like, it used to be that you, like before NPM and bundler, it used to be like whatever happened to be in your gem directory or your package directory was accessible to you. And then it was like, no, that's stupid. That should be something that's checked into your repository and is actually application specific.

And I really want the shell to have the same structure. You don't have to put everything in the universe in your path. Maybe this directory has a special thing. I think that's usually true, basically. And I think we should do it. Okay. Yeah, like someone in the Discord was saying about configurations,

saving, opening, fetching, changing, because we also have a config command. And I think this aligns well. If we could leverage this and do something with those configurations, it would be very nice if we could work with configurations that way for the shell. It's interesting because we're so used to how things have been done since the 80s, right? It's shells.

So like, for instance, when implementing CP or MB, like we questioners ourselves, so should we have a recursive flag or not? Why do we need capital minus capital R? What does anybody want to do when they're copying a directory? Come on. Very basic stuff, like MKDIR. Okay, so do we implement the minus P equivalent such that if you put multiple in a single argument,

you specify, say, three deep level directories? Okay, if you specify minus P, it creates the three directories. Just do it. Just do it all the time. Yeah. Who doesn't want that? So it's a completely change of mind at all levels at all the levels of implementation everything it's been a fun experience that is fun rethink things from the bottom up

yeah exactly so we're deep in the weeds and i'm loving the technical talk here we want to talk about some of the community stuff you guys have going on because we're running a little bit short on time and you have a lot more in this project beyond just the design philosophy of the community stuff you guys have going on because we're running a little bit short on time and you have a lot more in this project beyond just the design philosophy of the technical side. You have the community side, you have a book going on, you have this amazing multilingual scenario going on,

you have cross-platform, Windows, et cetera. So tell us about kind of the higher level metagame that's going on with Nisho. I'll mostly talk about community because that's the part that's like the most near and dear to my heart. I'm personally, and I think everyone here, is a very strong believer in just open communities,

people doing work together that are not structured inside of any company where everybody is part of a, like everybody is part of the project by virtue of their individual skills, not by virtue of their company membership. So there are basically two ways to run a project. You can run a project where the project is hosted at a company. And if you are an employee of the company, you intrinsically have higher status than a person who is not there. And you can tell when this happens, right? People talk about the community as an outside thing, or you find

out about things as part of corporate announcements or long after they were designed, right? And if you're a regular person just trying to help, you very quickly discover that that is a bad use of your time because you can't actually be in the conversations unless you become a partner of the other company. And like, as a person who works in a very small company, I am never going to be a partner of Facebook. That's not a sensible thing for me to try to do. So I'm not here to reject the idea that there could be Facebook-owned or Google-owned

or even Mozilla-owned open source where being an employee of that company gives you higher status. But I will say that the alternative is under-discussed, despite the fact that it is popular. So Rails, Postgres, Rust, Ember are just some relatively large projects that don't have a corporate host whose status matters, where your status matters. And I think there's a lot of things that come out of doing that.

Let's say you stop working at the company anymore. I'm going to use React just not to make any specific claims, just as an example, because it's been around for a while. If you're a Facebook employee and you stop working at Facebook, your participation as a member of the React core team significantly reduces. Everybody knows that.

And I just think that's lame. As a person who wants to contribute to open source, I actually have been in that situation a bunch where I was working at a company on an open source project and left. And I've always wanted to structure open source projects I work on so that your involvement in a company is a matter of sponsorship. Your company is helping you work on the project. But just because you leave that company

has no real impact on your intrinsic status. And so that was a whole bunch of stuff I said, but that's basically the philosophy, the community philosophy is like, I am not special because I work at Tilda. It's not a surprise that none of us really, I guess Jonathan works at Microsoft. Is that a secret? I don't know.

Is this true? It's true. Do you work at Microsoft, Jonathan? I do. Confirmed. There you go. But that doesn't matter. That's not an important thing.

And I think it might matter in the sense that Jonathan has to get approval if he wants to contribute on work time. But basically what I'm trying to say here is I believe very strongly in building a community that is a welcoming community that accepts people of any different skill level. We've had already contributors who come in just to do, just is really the wrong word here, to help us set up our CI environment. And that person does something that I do not know how to do and is an extremely high value, high status contributor to the project. And so I believe in that and I think all that is just much easier when you just say, you bring yourself to the picture. You don't bring your, like your company doesn't tell you anything about your status.

Right. So you're saying new is its own entity. It's not under a corporate banner. It's not represented by a corporation if you anything about your status. Right. So you're saying new is its own entity. It's not under a corporate banner. It's not represented by a corporation. If you're involved in it, it's because you've been sponsored by wherever you work or the like. Yes, there is no corporate management chain that has any control over new.

And I think if you just think about that heuristic, if you think about other projects that you think of as being this company's project, there is a project manager and a manager and a director If you think about other projects that you think of as being this company's project, there is a project manager and a manager and a director and a VP and all the way up to the CEO that is the management chain that decides what that project actually does. And that is the thing I'm saying we do not do. What do you say to that, Jonathan?

I would add, too, that there is another piece of that philosophy that works together, which is even though like the three of us are quote unquote core team, if people come and they have ideas, just because we as the core team have an idea does not make that somehow way higher value. So it's kind of like the openness that we have with each other should be with everybody.

And then the ideas that come, it's important to kind of keep it on track. You don't want to go like way off the rails, but there's a lot of energy. If they want to do CI, if they want to, we were talking about, we worked on a book. I wrote a book for New and then Andres translated it to Spanish. And then someone came along and said,

here, here's the Japanese version. It's like, wow. And in Chinese, someone just said Japanese version. It's like, wow. In Chinese, someone just said if we could translate in Chinese, Jonathan. Yeah, I know. It's awesome. That's great. I mean, I used to have a heuristic that I used in my head, which is like, how does it feel to be a person who was like pretty active but not on the core team?

And the answer should be like pretty close. Like the rules that we hold ourselves to involve like uh putting our ideas out there in public getting some feedback getting general agreement or like at least approval from the wider community that this is a good idea um i i also just want to say that this sounds like this is not any of our first rodeo and and to the extent that this is the first time anybody is hearing anything like this or feels like it's a pipe dream, I feel like Ember and Rust are both very successful. You don't have to believe that Ember is a successful project. I'm not taking on that fight right now. It's a different show.

It has longevity and a large community of people using this model. And Rust is also an even bigger community with a little newer that use this model. And I think things like Postgres and Rails are other examples. So I just think that people underappreciate. It's just considered a relative. Everybody thinks that, well, the money has to come from somewhere. So that means we have to give control of our project to the management structure of the company that's funding it. And I just think that's wrong.

Let's dive into an example, only because Jonathan, I know your work, and Andreas, I don't know where you work. I'm only using Jonathan for this example because of that. So what does Microsoft say in your involvement in this project? So do they literally sponsor you to do this? How are you able to put so much product management, project management, PM status into this? It seems like you've definitely been quite invested. You wrote a book, which got translated.

We just talked about that. How do you do that? This has actually been kind of a nights and weekends project for me. When I had the idea, I got hooked on it. Like Andres was saying earlier, I'm really excited about, oh, let's just build it. And I'll just invest my energy into it. And then of course, like months later, I'm still working on it. So there's, there's definitely some of that where all that energy was kind of just channeled and it was just the hobby energy. We put out the public announcement. So I wrote up the blog post and put it on Hacker News

and the thing shot up to the top of the page. And I'm like, what's happening? It's got like 200 votes. I should probably tell someone. At like over a thousand votes, I was emailing my director saying, I apologize for not telling you ahead of time that I'm working on this thing on the side. I hope it doesn't conflict with anything. And they've been super supportive over there,

actually, which I just want to give a quick shout out that I work on the Azure SDK team. So there's no conflict. I'm not on the PowerShell team working on a PowerShell competitor, right? It's nothing like that. My director looked at it and said, hey, this is totally cool. If this is your way of giving back in open source, your way of using your creative energy, keep going and maybe in the future Microsoft will jump on and maybe write their own plugins as part of it.

That sounds good. I think people just largely underappreciate. My view of how permissive open source works is that it's just a bunch of engineers I think people just largely underappreciate the, my view of how permissive open source works is that it's just like a bunch of engineers circumventing management to work together. Now maybe that's not the right thing to say right after Jonathan just said he got permission. Your word's not his.

I actually don't mean like your direct manager. I think a lot of times, if you want to become a partner with, I keep wanting to say React because I know it, but there's a lot of things. If you want to become a partner with React, the right way to make that work

is through BizDev. There's an open source way of doing it, and especially projects like React, which are kind of hybrid models, make it a little easier to do the normal open source model. But there are other projects, like if you want to contribute to any open source part of like Google or AMP or like Google App Engine,

those projects, you just have to actually go up your management chain a bunch of times and get them to go to a person who's pretty high up the management chain and get it working. And that's like, as you go up the management chain, people have less and less of a clue of why you would care, right? So the thing that I think works really well about permissive open source is that it lets me talk to Jonathan and Andres and be like, hey, like, let's work on this project together. And then like all the people in our Discord and all the people submitting pull requests

are not having to get approval and create biz dev arrangements. And I think that whole story basically means that there's this, it's like a loose confederation of not just engineers, right? It's like engineers, people know how to do DevOps, people know how to do design,

build websites, whatever, infrastructure, right? It's like engineers, people know how to do DevOps, people know how to do design, build websites, whatever, infrastructure, right? It's a loose confederation of those people. So it's not about what skills you have. It's about the fact that you did that without having to ask your CEO for permission. And it's not because your CEO doesn't care. It's because everybody who runs a plausible company in tech kind of agrees that it's okay to contribute to open source. So there's like this general agreement that open source is an acceptable way for people to collaborate outside of the management structure. And I think that's

just good. We should leverage that more. It was hard to get. Multilingual, got a book, got a community. Where does community happen? Discord is the main chat location. Oh, you should join. We're having so much fun there. I believe it. Y'all sound like you have a good time. What's the...

In the general sense, like I've been sharing Latin American food, people laugh. It's fun. Newshell.sh is the very bad website. Sorry, Jonathan. Bad on purpose. Sorry, because you made a bad website or sorry because you just is the very bad website. Sorry, Jonathan. Bad on purpose.

Sorry because you made a bad website or sorry because you just called it a bad website? No, it's like bad on purpose, but then I... It's bad. No, it's bad. It's real bad. But we kind of want to say, look, this thing is 0.2.0, that we have all these cool ideas, we made our first release.

Please do not take this and think, oh, it's got everything that you possibly want out of a shell. So we're trying to message that. If you click on GitHub, then you will see a readme. In the readme, there's a link to the Discord and some other places. Discord is really the main place where very rapid conversation is happening. We're also doing a pretty decent amount of volume on the GitHub issue tracker. And I think so far that's going okay. And I think if somebody is, basically, I guess what I would say is if someone is enthusiastic about like contributing, definitely hop into Discord, we can figure out how, likely figure out what makes sense for you.

If somebody is just like wanting to tell us about an idea that they have in a very async way, also feel free to file an issue. We've had like a dozen people who said, here's just a, I've been wondering when the next show is going to happen. I've been thinking about that for a long time. Given that it happens,

here's my idea. And that's like, we are totally open. I can't promise that those issues will stay open forever because like triage is important, but I think we read all of them and I think have conversations conversations in there i guess we haven't asked about the status i assume the book is the best way to get started if you want to play with the thing but i mean are we in

like play with it but don't really use it stage are we and go ahead and give it a shot maybe you can use it as your replacement shell what's the status there i think we're kind of transitioning into so we use it so i use it like kind of daily. I think most of us dog food on it. And it's at the point now where it's stable enough, things generally work well enough that you can just use it as a shell. But I think it's just recently kind of crossed into that. So don't have this assumption that, you know, I was just watching Discord earlier before this podcast. People were like, well, can you do command and end, like ampersand, ampersand, command?

No, it's not there yet. We still have a ways to go to get all the things that you would expect. And we go crash a lot. I think the shell still crashes a lot. It's not actually crashing. It's a rust thing. It's panicking. It panics instead of crashes. The shell ab crashes a lot. It's not actually crashing. It's a rust thing. It's panicking. It panics instead of crashes.

The shell aborts a lot. A lot is actually an overstatement. It is not hard to cause the shell to abort, which is largely okay if you ran it from Cargo Run or just ran the executable and then used it. But if you try to make it your main shell and open your terminal, have it be open, then it will close your tab a lot.

So MVP2, which was the, we have to revisit anything we thought before we released because of how many inbound contributors we have. But the idea for the second MVP release was gonna be, it makes sense to make it your shell, which involves doing something generic to every panic and just in general reducing the amount of panics, right? Like we don't need a panic for a bug, right? A bug. If we have a thing

that, you know, there's a common idiom in Rust where it's like, oh, I really don't expect this to ever be possible from a programming perspective, so I'll just unwrap here, right? And that, it's fine, but if you were wrong, then you were going to crash the shell. It seems okay to have a I added an error type that's like something weird happened, and it seems fine to turn that into a regular error.

I love the spin that you guys just put on this. Our software doesn't crash, it panics, and then aborts. I'm going to start using that. My stuff never crashes. The reason Jonathan said that is that usually when people say crash in the Rust space, they're talking about a segfault, and segfaults are security vulnerabilities, and Rust makes some

promises about security. And so when Rust makes those promises, it talks about never crashing, by which it means it never has the segfault that produces a security vulnerability. So anyway, that's the reason. It's that, but it's also a lot of what we wrote, we just wrote, kind of knock out code as quick as possible. So we take a lot of shortcuts, and those shortcuts lead to, well, okay, the condition wasn't met, panic, fail, and then just dropped straight back to the original terminal.

So there's a lot of that that we just need polish. And I actually think this is a great thing about Rust, by the way. Rust is mega rigorous. Rust makes you think about a lot of things, but it helps that we were able to ship an MVP of a new shell with a bunch of police holders. Not that many, really, but I don't think people appreciate that enough about Rust, that there's enough escape valves to prototype things.

So if y'all had a magic wand and this show's over, it gets released, world loves it, whatever, they're learning more about a new show, what would you want to happen? What do you want people to do because you put this out there? What's the next step for adoption? That's the success of open source, not to die, right? To be adopted. What is that? So a couple of things that I would love to see, we talked about plugins a little bit. I would love to see the beginning of an ecosystem of plugins, of people experimenting with it, kind of stretching the limits of what's possible so that the design that we end up with at 1.0

is fully fleshed out and can do anything that you'd want to do in a plugin. And that's across languages. So the design makes sense. Oh, I can write it in Python, I can write in JavaScript. So that's certainly one thing. One thing that we talked about earlier was different communities, language communities jumping on. So we have, you know, a Japanese translation, and maybe in the near future, a mainland Chinese translation. So having communities of people pick it up and run with it from early days, I think is amazing because that's something that we want to facilitate. We want to make the community open,

not just for the English speaking world, but create other communities and other languages. And along those same lines, I think for me, the definite, like, why did I care in the first place is I really want more people to not feel scared of the shell and it's very easy for us to forget that like i when i first started software the leia the ceo of tilde uh is in the middle is taking a flat iron course now and like she happens to have been using the shell for years because she like makes maintains websites with

github but like one of the first things that they tell you is like don't be afraid this very scary thing is not actually that scary and like every single new programmer has to go through that don't worry this is not so scary it actually is scary but it's like you know you'll you'll you'll deal with it it's worth it and i it. One thing that I'm really, I would consider this a big success, and it's my goal, is a lot of people who know Excel don't feel scared. They feel like it's normal.

That's true. Actually, I know somebody who runs a very successful HVAC system, large-scale system install and servicing company here in Texas. And he runs his business off of spreadsheets. He has a very fast way to very accurately give a proposal within a day or even a couple hours, he says. And it's because he wrote this Excel spreadsheet that he wants to one day turn into actual software that could then be a service for other HVAC providers

throughout the world, for example. So it's just so wild how that program is so accessible. And he's not a programmer, but I'm like, dude, you're a programmer. You know, it's like you made this thing that powers your business, that software. One thing I think about a lot is whenever I go to the airport or go to Target,

the person behind the counter is using a 1980s era terminal. The other day, someone was typing in my email address and then I was like, at gmail.com. And they were like, that's F4. They had made F4 be at gmail.com. I don't actually think that the terminal itself is that scary.

There's like a huge amount of like relatively entry-level workers, knowledge workers in the country, or even like shop workers who are using a terminal. It's just that the way the terminals work are so baroque, I don't mean broken, but like baroque that it's just that the way the terminals work are so baroque, I don't mean broken, but baroque, that it's just not worth it. I'm not going to tell a random person to use a terminal,

because then I'm signing up for teaching them all the weird, why does Bash have an arithmetic parsing mode? That is not a thing I want to explain. Not a thing I want to explain. I think people underestimate, people think it's the terminal environment. I don't think that's true. I think it's the terminals that we have, the shells that we have.

What's this Easter egg you have, Jonathan? Yeah, I was thinking a fun one for the end of the show for people that made it all the way to the end through our philosophy and our technical. All right, how about this? So one of the things that has been a going philosophy for me in New Shell is that we should be able to work with any kind of file. Now that's future looking, right? I cannot make importers for every single file type. But one of the kinds of files that I wanted to do

was to open binary files. So two things that you should definitely try if you try out new shell is opening an image. And I'll also say this one, if you open a file for an entertainment system of a certain era, certain 8-bit era, and there's a file for this kind of entertainment system, you should definitely try opening that. You've got me. You got me hooked. I gotta go download this thing now. So listeners, you have homework. Go do that and get your Easter egg. Fellas, thank you so much for sharing this. I mean, one, nights and weekends, and even creating an environment where it's open for everyone to contribute. And it's not based on their work status or who they work for.

And it doesn't change if they leave that workplace, whatever. That's awesome. We got links in the show notes for all these things. It's awesome talking to you guys. Thank you. All right. Thanks. Thanks for having us.

Thank you again. All right. Thank you for tuning into this episode of the change log. Hey, guess what? We have discussions on every single episode now. So head to changelog.com to discuss this episode. And if you want to help us grow this show,

reach more listeners, and influence more developers, do us a favor and give us a rating or review in iTunes or Apple Podcasts. If you use Overcast, give us a star. If you tweet, tweet a link. If you make lists of your favorite podcasts, include us in it.

Also, thanks to Fastly, our bandwidth partner, Rollbar, our monitoring service, and Linode, our cloud server of choice. This episode is hosted by myself, Adam Stachowiak, and Jared Santo. And our music is done by Breakmaster Cylinder. If you want to hear more episodes like this, subscribe to our master feed at changelog.com slash master, or go into your podcast app and search for changelog master. You'll find it. Thank you for tuning in this week. We'll see you again soon. Bye.