The Changelog: Software Development, Open Source - Gradually gradually typing Elixir (Friends)

Episode Date: January 27, 2024

Our old friend José Valim & his team have been hard at work adding gradual typing to Elixir. They're only 1-3% of the way there, but a lot of progress has been made. So, we invited him back on the show for a deep-dive on why, how & when Elixir will be gradually typed.

Welcome to Changelog and Friends, a weekly talk show about the knife and the cheese. Today we're joined by Elixir creator, Jose Valim. Thanks to our partners at Fly.io. Launch your app near your users. Learn more at Fly.io. Okay, let's talk. Oh, you've been on six of our episodes. Six. This is the episode seven. We should do a most frequent guest thing. When did you start the show? So the Change Log started in 2009, officially. Oh, okay. But it had some fits and starts.

I started in 2013, and I think you've been on. Since then, yeah. It was not before then. It wasn't before then. Your first appearance was 2016. And was about Elixir, most likely. Elixir and the future of Phoenix. I think we actually had,

well, it was just you. Oh, was it? So now we're just doing the show. Welcome. It's Change Dog and Friends. So our story with Elixir started with Chris McCord, not with you.

Did you know that? How unacceptable. All right. So you call Chris and he comes on the show. Yeah, that's pretty much what happened so phoenix dropped and i'm a web developer so i'm interested and i was a long time ruby and rails i knew who you were and i used devise and stuff and so it wasn't like i was unaware of yourself we just hadn't met and when phoenix made a splash i was like we should

do a show on phoenix and adam said yes and so we invited chris mcc like, we should do a show on Phoenix. And Adam said yes. And so we invited Chris McCord on, and we did a show about Phoenix. And he licks her from his perspective. And I was sold enough to give it a try because he was very convincing. And I ended up liking it quite a bit. Did a quick, not a demo app, but like a very small single endpoint. I think it was like a Slack inviter kind of thing where it'd do the whole Slack invite thing for us when somebody joined a channel. I can't remember what it was,

but just like a really quick one file Phoenix app that I could deploy and it worked. And I was like, this is cool. So that got us going on Phoenix. And then we invited you on for the deep dive on Elixir. And that was in February of 2016. And then a few months later we had you and chris on together that's probably the one you're thinking about oh yeah that's the first one i remember yeah

ecto 2 and phoenix presence that was in june of 2016 so it was literally four months after the first one so we just we're kind of hooked there and now this is your seventh time on the show what's interesting though about his very first appearance with us the elixir and the future of phoenix episode episode 194 70 000 listens and that's what we tracked which is like not inclusive of spotify which i guess since it's that old maybe it doesn't have a lot in the external ecosystems we're not pulling into our internal stats so that's 70 000 internal stats listens that's a pretty big show you know what like speaking about numbers i think the tweet that we are going to talk about today on the show about

types i think it was my most popular tweet ever it was like ever yeah i mean i don't keep track of everything but i can't remember sure last time i checked it was 160 000 views i don't think i've ever crossed that why do you think that happened i mean i get i don't know i like it's elixir and types, right? So I knew it was, maybe we should talk about what the tweet is. Sure. Yeah. So the tweet. Spill the tweet or the X or whatever it is. You want to read it?

You want to read the tweet? Because this is something we do now. We have people read their tweets. I only have Twitter on my phone. Oh, we'll wait. And I keep it locked. Oh, okay. Under trickies or something like that.

If you have it, but otherwise i can quote it from memory yeah i don't have it you please do you'll do better than me so it's basically announcing that elixir is now officially a gradually typed language and it had a screenshot from some very small snippet of elixir code where we were finding a type violation because you had a variable that was declared to be an integer in some place and then declared to be a binary or something like that, all inside the same binary pattern. They're like, oh, this thing cannot possibly be an integer

and a binary at the same time. And I kind of knew that it was going to like it was probably going to like a lot of people are going to like it and it was going to go out right so what i did is that i of course i did a tweet with like the main headline and then i added a bunch i started a small thread and I added more context. Right. And then later I realized that, so people, they were asking me questions. I was like, hey, that is in the thread.

And they're like, oh, it's because I don't have a Twitter account. And now Twitter doesn't show you the whole thread with all the context. And the context is important because if somebody is expecting Elixir to be like fully, gradually type right now, it's not. It's not. Yeah. We've begun to work and we should probably talk about that. But it's just the beginning. Yeah. You do say officially though.

So that may be the misnomer on the assumption because the context and the tweets that they can't read because they're on Twitter X, is hidden behind you saying, I'll read it for you. It says, tomorrow marks 13 years since the first commit to Elixir repo, and today we celebrate by announcing that Elixir is officially a gradually typed language. And then the screenshot. And that's got 177,000.8 views on it, and almost 600 retweets, reposts, 2.2K likes slash hearts, and almost 200 bookmarks on that too. So it's a little bit fake newsy, isn't it, Jose?

Like it's a little bit like it's... No, it is official, right? It's because, so let's go a little bit into it, right? So what does it mean, right? What does it mean that Elixir is officially like gradually typed, right? So before that, before that tweet, let's say, Elixir was a dynamically typed language.

And one of the ways that you can think of a dynamically typed language is that it's a statically typed language between quotes, right? Where all the functions, all the inputs are dynamic, and then it returns dynamic, right? So everything that you receive is dynamic. Everything returns dynamic.

And dynamic means I don't know what is the type at compile time. I know at runtime. So if I try to do, I try to add a number to a list, which is not possible in most programming languages. In Elixir, you are going to get an error at runtime. But if you had a static type system, you could potentially catch that error at compilation time. So Elixir was a dynamically typed language. And being a gradually typed language means that you have the dynamic,

but at the same time you have the dynamic, you can also have precise type information. So what the tweet was saying is that, well, instead of Elixir now seeing all the variables, all of your inputs as dynamic, now we are doing inference on your patterns and guards, which are very common in Elixir code. So in your pattern, you can say, oh, I'm going to match on the list. You can use a guard to say, oh, this thing is a number or an integer. We are getting that information to say, oh, so now I know that this variable is no longer dynamic and it's actually an integer and it's a list. And now that we are mixing this dynamic world with this little bit of type information, right? Now it's officially gradually

type and that's correct, but it's more like it's officially gradually type, but we are at 1% of the journey. And that's what the following tweets they meant to say. So if you think about everything you can do in a pattern and in a guard, in Elixir, you can check if it's an integer, if it's a function, you can pattern match on maps, on tuples, on lists. We only understand one construct right

now, which is the binary matching and the binary construction. So of all the things in the language, we understand one of them. But getting to this one is super, super hard, right? It's like literally two years of research have passed. So getting to this one was like super, super hard. Now getting to the next 10 is not going to be as hard.

But yeah, we are officially gradually typed, but it's like we are 1% gradually typed, if that makes sense. It does. That's why you had to add those additional context thread tweets. That people can't read and the ones who can won't read. Right, exactly. But that is how you generate more and more buzz as people continue to engage with you on the thread.

So that explains some of the excitement. What's the adage, too, of work in public? That's what this is, right? This is working in public. This is showcasing something that you've been working on for a couple of years, not research-wise, and saying, hey, like it may change, but we have declared that we are going to have three milestones in this, right? So what are our milestones? Well, actually, so we did a bunch of research already, and then we can go into this. We have published articles. I have given talks. We have done a bunch of research. And by we, I mean Giuseppe Castagna,

which is the senior researcher, and Guillaume Duboc, the PhD student who have been working on this. There is a paper with our names and my name is the last one, deservedly, because I did the smallest part of the work there, right? So when I say we, I mean, not me, pretty much them. So there was this, so we did all this research and our idea for the implementation, which is what we have started, is that we're going to have three milestones. And the first milestone is that we are going to look

at all your patterns, all the guards, get information about them, and use that information inside the function, inside the program to catch obvious typing errors, right? So the idea here is that, so I like to say, even before I started thinking about typing, that Elixir is an assertive language because usually, even if we don't have types,

we actually kind of know what we are working with. We know if it's a struct, we know it's a list. That information is there in the program. So the idea is what if we can leverage this information that is there in the program and use that to find bugs. And we are going to do that without changing the language, like the language surface for the developer. And then we can use that to find bugs. And we are going to do that without changing the language, like the language surface for the developer. And then we can use that to give it a try on the type system, make sure that it's going to be fast, make sure we're going to have good error messages. So our first milestone is meant to be like, you can't disagree with it. It's kind of like, it should be a win-win situation for everybody. If you're worried about type systems or you don't like about type systems, like, hey,

we are going to find bugs for free. You don't have to change your code. You don't have to write types. You're just going to find bugs for free. If you like type system, it's one step to what you may perceive as the right direction to go to. So that should be like a win-win situation. Should please everybody.

If it does not please everybody, it's probably because we did something wrong and we may be in the wrong direction. So that's like the first milestone and we did 1% of it, right? And then the second milestone is to actually start changing the language to introduce some idea of types, mainly around like data structure. So when you define a struct in Elixir, you may want to say, oh, I have a user struct where the name is going to be a string,

the age is going to be an integer, and you can have more precise information around data. And then the third milestone would be, well, we are now 100% gradually typed, and you can also provide your own type signatures if you want to right so it's it's like we have been researching for two years we were able to move out of research and there are still like those very large milestones um for us to go through so one of

the things i tell people a lot like i'm not asking and anything from anybody except patience right like let's be patient we are being very careful about this. It's like we are thinking each step after the other, right? We are not trying to rush it, right? So we'll see where I have to go. And the usual disclaimer I give people is that these can still fail. It doesn't mean that, you know,

Elixir is going to be like 100% gradually timed. It may be, oh, something did not work out. We have, we will give up. Maybe we can only do milestone one and we are not happy with milestone two and three. And then that's what we'll have at the end. what's up friends this episode of team talking friends is brought to you by our friends at and i'm here with lee robinson aka lee Rob, VP of Product over at Vercel. So Lee, let's talk about Vercel's front-end cloud.

I know Vercel supports over 35 frameworks, but the question I often hear is, is the happy path on Vercel Next.js? Are you optimizing the best experience, the sugar on top, so to speak, for those using Next.js? Yeah, it's a great question because, of course, we're biased because we make Next.js that we really like Next.js. code in the frameworks that you prefer and still get to take advantage of all of the cloud infrastructure primitives and the developer experience platform that Vercel provides in that same model. So tangibly this framework defined infrastructure concept means that I write

code in my framework native way and Vercel is going to do the heavy lifting of transforming that into managed infrastructure. So in Next.js or in Nuxt or in SvelteKit or Remix or all of these frameworks that we support through framework-defined infrastructure, I'll write my code in their framework API. So I want to do server-side rendering. Great. I'll do it through their conventions.

When I deploy to Vercel, our platform is going to transform that code to work with our managed infrastructure that's very scalable and also flexible for what developers need. Okay. So help me break down what you mean when you say the front-end cloud. I get Vercel, front-end, front-end frameworks, all that good stuff, but break that down for me. What exactly is the front-end cloud? We believe that there should be tools designed and built by and for front-end developers. We want to have tools that help enable you tools designed and built by and for front-end developers.

We want to have tools that help enable you not only from the framework and the code that you're writing with React or Vue or Spelt, all the way up to the platform that allows you to deploy your code, not only in US East, but around the world as well, too. And I think historically, a lot of tools for developers have been built back-end first, have been built for the back-end developer. And if you think about AWS, right, it's a great back-end cloud. It has a suite of tools that you can pull from and really craft and build anything that you want. But when you're building the front end of your application, there's a lot of things that are undifferentiated. It's like, I don't really want to have to stitch together all of these different pieces and set up my pipelines and set up all of this front end infrastructure. I just want to build a really good product. And really what the front end cloud means to me is focusing more on the product experience and focusing more on the

user experience and just letting a platform like Purcell's's front-end cloud take care of all of that infrastructure and tooling for you. Very cool. I love it. With zero configuration for over 35 frameworks, Vercel's front-end cloud makes it easy for any team to deploy their apps. Today, you can get started with a 14-day free trial of Vercel Pro or get a customized enterprise demo from their team, visit vercel.com slash changelogpod to get started. That's V-E-R-C-E-L dot com slash changelogpod. So can we go back even further and just answer why, why this effort? Why do you think this is a good direction for Elixir? So I think that's a very good question.

I gave a whole talk at Elixir Conference Europe when we went in 2022, perhaps, on this topic. And what I said, and usually when I go give a conference talk I usually give Elixir updates when I'm at Elixir events like hey here are the things that we have been working on or I usually talk about a project that I'm involved like a live book or stuff with machine learning and then I started the talk saying like oh these are the things that you know like I'm excited about in the future of Elixir, like machine learning and stuff. But before I talk about those things, I need to address what I call the elephant in the room. Because if you go to an Elixir event, or I think if you do a poll in a forum or

something like that, if you ask what is missing in Elixir, most people, they're going to say it's types. Types are the thing that like a type system is the thing that is missing in Elixir, right? So if a lot of people feel that's the thing that is missing, I have to understand that. I have to understand why they feel it's missing. Why do they think they are going to gain? And one of the things that I say in this talk, I don't remember precisely the arguments I use at the time, so I may butcher my own presentation.

But one of the things that I mentioned at the talk is that usually discussion about types, they tend to get so polarized very quickly. And it's very common for you to hear things that I don't believe to be true, if you try to look at it. So I remember seeing like people saying, well, if I was using a type system

in this particular language, it would have caught like 80% of my bugs. And then I'm like, I don't think that's, like, I feel like the, what the type system can do for you is like in that particular scenario was being overvalued, right? I don't think it's 80% of the bugs. And then when you go look at it, when there are some studies trying to find a correlation

with that, they can't find like something that precise or that clear that says, yes, a type system is going to lead to a program with fewer bugs, right? I don't think we have found like the universal proof of that. And I even think like this discussion as a whole can be harmful, right? I don't think we have found like the universal proof of that. And I even think like this discussion as a whole can be harmful, right? Because you probably heard people saying like, hey, well, if you have types, you don't have to write documentation. And I'm like, I don't agree with that. Like the, you know, get like the logical or operation. The types in some ways which would be Boolean, Boolean to Boolean, right? That doesn't tell me how that thing behaves, right? And then some people say, oh, if you have types, you can write fewer tests.

And I also think that's very harmful. I like to say, well, if you have types, you can write fewer bad tests because I don't think types would replace 99% of the tests that I actually write and I find important, right? Like, for example, going back to the same logical or, the types are not going to find bugs in the implementation of logical or if you make it always return false, right?

So I get this whole presentation, which was kind of like preparing people, like why we may need a type system in Elixir, but the presentation was much like, well, types are not all this thing that you think they are. So very good salesman here, right? Yes.

But the thing that I want to try to get to is like to kind of put everybody on the same page, right? Elixir is also a dynamic, it runs on top of the Erlang virtual machine, which is dynamic. So you're not going to necessarily get like large performance improvements out of that as well. So what do we tend to gain from adding types, right?

And what I mentioned in the talk, which I'll try to get through audio right now, is that it's all about contracts. It's all about making sure that, well, when I'm calling this function and that function changes over time, what is going to break? So it's defining contracts between different parts of your code base and helping you find bugs, violations to those contracts sooner rather than later.

And that's kind of like what we are to gain for Elixir in particular from having a type system. It's this guarantee where the different parts of my code, you know, as they evolve, as they change for time, they are still in accordance with each other. And there's something else that people, they mention a lot. They think, well, will Elixir help us like improve the developer tooling as well? Like having a type system improve the developer tooling as well? Like having a type system with improved developer tooling? And usually the example people give about that is about TypeScript, right?

And the funny thing is that it will help for sure, but everything that we need... So for example, if you take JavaScript, the issue why you need a type system to have a way better tooling is because JavaScript is completely dynamic and it's also polymorphic. If you have an object, you don't know what that thing is. All the time you're passing around, you have to call something in that object and you don't know what that thing is. So you have no idea what you're going to complete.

But Elixir is not like that. We're not calling things on our variables. We are usually calling things in modules that are well-defined. So in theory, we need to do work. We need to improve our developer experience, continue improving it. But in theory, we have the same where we have the, in Portuguese, where we have the knife and the cheese. we have like the knife and the cheese, like we have everything ready to improve the experience.

Everything is there. So in theory, we have that for the developer experience and the types always help improve. If you can easily see what this thing expects, you get that quick feedback cycle. It definitely improves, but you know, it's not going to be like an essential jump like you have in TypeScript where the type system is effectively necessary.

Can you say the Portuguese version of that for us? In Portuguese? I want to hear the Portuguese version. Yeah. What's the knife and the cheese in Portuguese? Yeah. Você tá com a faca e o queijo. Was that French?

Yeah. Close. Yeah. It may have come from the french actually it may not you know be native portuguese yeah i love that saying there's something very cultural about that right the knife and the cheese is cheese a common uh is it part of like most meals in portugal or i mean why is that an idiom there yeah so actually i don't know if that's also another good question i don't know if they use it in portugal or if it's just a brazilian thing so uh people that's the problem with portuguese it's spoken in many places yes queso is probably pretty stapled right like

cheese alone in any form melted or not is a staple in in almost all cultures yeah i suppose that is true yeah well i mean it's mean, who doesn't like cheese? It originated in the Latin cultures, but everybody was like, you know what? That's so good. We got to take that, you know? Right. I'm appropriating that for my likes because queso is required. I want to start using that.

We got the knife and the cheese, you know? Yeah. Like, so you have everything ready, everything in hand, kind of. That's what it means. If you're like a Brazilian, it would be, I would think it's more like you have the rice and the beans, you know? Because like that's the base of our meal it's like a Brazilian, it would be, I would think it's more like you have the rice and the beans, you know? Because like that's the base of our meal or something like that. Right.

That's probably how I would say it. But yeah, for some reason it's the knife and the cheese. Adam would say we got the barbecue and the home lab. The home lab. Yeah, man. All right. That's my staples. It's home lab and barbecue.

Everything you need. Yeah, I like that. Okay. We have the mic and the earphone. There you go. Yeah, man. All right. That's my staples. It's Homelab and barbecue. Everything you need. Yeah, I like that. Okay. We have the mic and the earphone. There you go. That's right. We could just keep doing this. Yeah, this is the show now.

Thank you. The blank and the blank. Fell in the blank. Okay. So you had the knife and the cheese. You're not going to have as big of wins as you would have in JavaScript because of just the nature

of the difference of the languages. It's not that we won't have such as a bigger wins, it's that we already have... You're ahead already. Yeah, that would be one way of saying it. Yeah, no, that makes sense to me. Absolutely. And you're starting with the guard clauses and the pattern matching. Is that where you're kind of like staking the claim?

Because there you're already kind of doing. So in Elixir, you can define multiple functions or methods and they can have the, depending on the actual parameters coming in, you can have multiple versions of the same function and it's checking all kinds of stuff. So you can say, is it a struct?

You can say, does it have these particular fields? You can do crazy amounts of pattern matching there. And then it will call the appropriate function that matches the pattern. So you're kind of doing type analysis there in a sense. And then you can have the actual guard clauses, which you guys provide a bunch, like is underscore map or something like that, is list.

And you can write your own as well, can't you? To a certain extent. You can as a composition of existing ones. Okay. But you can't write arbitrary code there. Trust me, I've tried. Yeah. And I would like a feature like that because you know me, Mr. Dynamic, I would just like

to be able to write my own guard clauses right there in line. Has anybody requested that feature, Jose? I don't think it will ever be approved because we can kind of see the reasons why. Because for example, if that was possible, then the work that we are doing right now of doing type inference between patterns and guards, we would not be able to do it.

And also Erlang wants to study that to get information. So we're talking about the guards and stuff, but Erlang already used that information as well to get information. So we're talking about like the guards and stuff, but Erlang already used that information as well to do optimization. So for example, instead of like boxing integers, if it knows like it can keep them unboxed because the type was already checked or to avoid some type checks, it already does that. So, and it may, Erlang may also rearrange the order of the clauses if it thinks it's more optimized and at the moment you allow people to just write whatever they want then you throw all that out like people are going to write code that does an http request on their guard yes i am

yes i'm going to yeah yeah it makes sense it's trade-off, and I understand why it will never be the case, but I will say that oftentimes I wish it were the case because I'm like, this would be much simpler if I could just do that, but I understand that that's a bridge too far. You can do your own, like you said, but you're composing existing available guards that Elixir and Erlang already know about, right? Yeah. That's 80, right? Yeah.

That's 80% solution. Yeah. So we are, so exactly as I said, and we started with the patterns and guards because we want to be able to start using the type system without changing the language, right? And that was very important for us. Like we want to be able to evaluate everything and give things a try without imposing any changes on users. Some of the stuff's going to roll out in 1.17, the upcoming release, right? Yeah.

So the 1% work that I've done, that we've done, sorry, it's already in main. So it's merged. You can give it a try. And there is one other thing that I want to get for 1.17, which is if you pattern match on a struct, and then you try to... So if you pattern match on the struct, and inside the struct declaration,

you pattern match on a field that does not exist, that raises a compilation error. But I want to do it so that if you pattern match on a struct, and then somewhere else you do like struct.field and that field does not exist, I want that to emit a warning. Previous Elixir versions already did it, but it was a hacked thing that we've built and this would be done properly back by a type system.

So that's the only other thing I want to have for 1.17. So baby by 1.17. So, you know, maybe by 1.17, which will be May, we will have like two or 3% of everything. But I think we'll be able to go quite further. I wouldn't be surprised if we get close to doing like at least all the patterns and all the guards, because here's the thing. So just so we have an idea how much work we have ahead of us because so for every construct in the language, I don't mean like every function call. A function call is one construct, but for every construct in the language,

a case, a count, receiving messages, everything like accessing a map field, for all those constructs that we have in the language, we need to go and implement the typing rules for them. And we need to do that for all the patterns, all the guards, right? And then what also we need to do is that for each data type in the language, we need to devise like the proper data representation from the point of view of the type system. So like atoms for the type systems, they are represented in one way,

integers in some other way, lists in some other way. And each of those have their own internal representation as well. So one of the things that the tweet did not go, actually, I believe I went in the thread, I talked about this, is that today we have no granularity over the type. So in the current implementation is that we have atoms and in the current implementation, the atom OK, we cannot make a distinction between the atom OK and the atom error. Those are all atoms. We don't make a distinction between them. We don't make a distinction between a tuple of

two elements and a tuple with three elements. Those are all tuples. So you also have to work on that as well, right? And sometimes when, and this is like really hard work. So we stayed like two years doing research because for example, we would look at some data structures in Elixir, like maps, and we're like, wait, we don't have the proper theory to properly type Elixir maps. So Beppe and Guillaume, they actually had to go and develop new theory, like improve mathematically that the theory is correct, just so we can type check Elixir maps, right? So that's the work that we have ahead of us. We have to implement that now. And yeah, and that's all part of that initial milestone.

Yeah. So in May, when we inevitably upgrade our Phoenix app to 1.17 without any sort of changes at all to accommodate this, what would happen to us? Nothing? Would we get some better or our compiler would just be smarter, basically? Yeah. So the hope is that we are going to be able to find more bugs in more occasions.

So if you have like very obvious mistakes. So those are mistakes that I'm thinking like you would be able to find in a code review, for example, very easily. Like you're reading somebody, imagine somebody committed some code and they did not run the test or anything. They just changed some code. You'd be able to say,

oh, I don't think this is going to work because you said that this thing is a list and you're trying to add that to an integer, right? So like very obvious ones. But then even within asking user to write the types we'll be able to to ramp that up considerably and start finding bugs between a guard that you defined in this other module right and then you change that module and this module change so that thing that i talked about contracts and we want to observe and find out when contracts change, we will be able to do that even if you don't write any type signature at all.

And I think this is going to be really valuable. And if we can do it really fast, then I think it's going to be fantastic. Have you used, I know you have like a Phoenix app, right? Yeah. Do you use Dialyzer in the Phoenix app or no? No. Okay. If I i do i don't know that i'm doing it would that be set up by phoenix for me no what dialyzer yeah so dialyzer is um

love the smile he gave you no yeah yeah so so dialyzer is a discrepancy analyzer so elixir has like type signatures you can add type signatures to things. And then Dialyzer is going to look at those type signatures. And it's going to also find obvious bugs in your program. Costis, who implemented Dialyzer, he said Dialyzer is never wrong. Because Dialyzer is only meant to be able to find bugs that it can prove with 100% certainty that that is going to be a bug, right? Because the other thing about type system is that not a lot of people realize is that type systems, they restrict the kind of code that you can write. Like in many programming languages, you can write a code and you're like, this makes total

sense to me. But the language is like, well, the type system cannot understand it or you cannot prove that code to the type system so nope you can't compile it and if you build a language statically type since the beginning you can make you you make the language design the syntax the semantic mirror the the type system right and you build everything together side by side so those like sharp edges, they do not look sharp. Everything makes sense. But if you have an existing language like Elixir or Erlang, adding a type system is much harder. The approach that Alizer chose probably two decades ago at this point was like, well, it's very similar to what I was saying about Elixir. We want to look at the

code. We want to understand the code. And if we find a bug, we are going to let you know. But they don't call themselves a type system, right? And what is new in our research is that we are kind of implementing something like Dialyzer, something that is going to tell you, hey, I for sure have a bug in here, but that's within a type system, right? So it's not a separate thing, which means that if we get the foundation that we're working on and we continue evolving it, we are going to have a type system that people can

use. So I don't even remember what was the question anymore. You were asking me a question. You were asking me a question. Yeah. You asked the question. And I turned it around. Well, I didn't know what Dialyzer was. Yes. You were telling me about it. Yeah, I was talking about the question. You asked the question. And I turned it around. Well, I didn't know what Dialyzer was. Yes.

Yes. You were telling me about it. Yeah. So a lot of what I'm saying, I, so yeah, I went for Circle. So a lot of us, if you use Dialyzer before, a lot of the things that I have been saying, they feel similar to Dialyzer. But Dialyzer, like the error messages in Dialyzer, they're not good. They're very confusing.

I think Dave Thomas, he gave the best quote about dialyzer, they're not good. They're very confusing. I think Dave Thomas, he gave the best quote about dialyzer. He said that using dialyzer is like talking to a monk. You know the monk is correct, but you can't understand what it means. So that's like, hey, this code cannot

fail. And then you're like, oh, sorry, this code cannot work. And then you look at it, this code cannot work. And then you spend like two hours figuring out, right? So we have that dialyzer. It can do some of this work, but the error messages, they are not clear. It also runs separately, right? So it's like, it's a separate workflow. So I was just to say like, if we had a dialyzer that ran all the time, right? As you compiled, it was fast and it had good error messages. It would already be like a huge, a huge boom, like to the user experience. And we are starting there, you know, that's what we are, we are trying.

In a way, I think it would be like, because I'm saying like, oh, we can have a type system in the future. But I wonder, could we just do like the first milestone that i'm saying like we try to guess everything and then people are so happy that we don't have to go the whole way they're like stop right like don't change it like i don't know uh but we'll see what happens yeah well yeah i think that would be a big boon i mean especially with especially with things like Dialyzer, where if you're a casual user of Elixir, like myself, over the course of six or seven years, I didn't even know. It rang a bell, but I didn't know what that bell meant.

So having it built in, first-class citizen of Elixir, and just there doing its thing with good error measures and all that stuff, I'm sure will be very helpful. Yeah, and you asked, is that something that Phoenix generates by default? Dialyzer has been part of Erlang way before I started programming Erlang. And the reason why we didn't add it

is because we don't think the trade-offs are worth it there. It's really a power user too. And if you want to opt into that, do your research, right? Like search your inner peace so you can talk to the monk, you know? And yeah. Well, I've used Credo.

So there's an analyzer tool, which I think is mostly a linter and a thing that's going to tell you this is not the Elixir way kind of thing. And that was fine, but also I guess got annoying for me. a thing that's going to tell you this is not the Elixir way kind of thing. And that was fine, but also I guess got annoying for me as a solo dev because I guess for me the reason why I've always been dynamic languages

first and foremost is I just like my freedom. And if a tool doesn't provide major value but it just annoys, like I'm sure Dialyzer, I would turn it off probably very quickly because it's telling me something's wrong, but it's not helping me know why it's wrong and everything looks right to me. I don't want it. I don't want a tool that just gets in my way and tells me that my stuff sucks, but not why.

And so I've traditionally ignored a lot of that kind of stuff because I just want the freedom to write the code that I need to write to get the job done. But if the tool can provide value and help and not be just a hindrance, you know, just something I'm fighting against, then I'm here for it. So hopefully you guys strike that balance, especially with the 3% solution. If you're listening, you may remember the early days of the internet where open networks like HTTP and SMTP led to an explosion of websites and online communities. Building a fan site and connecting over shared passions led so many of us to careers in software. Back then, it seemed like anything was possible because the internet of the 90s was built to democratize information, not consolidate it with a handful

of big tech companies. Read, Write, Own, Building the Next Era of the Internet is a new book from startup investor Chris Dixon that explores how network architecture plays out in our online lives and the decisions that took us from open networks governed by communities of developers to massive social networks run by internet giants. ReadWriteOwn is a playbook for reclaiming control and for reimagining applications so users can own, co-create, and even profit from the platforms they use every day. From AI that compensates creators to protocols that reward open source contributions, this is our chance to build the internet we want, not the one we inherited.

Order your copy of Read Write Own today or go to readwriteown.com to learn more. Enjoy the draw. There's a story I like to tell. Okay. I like stories. Yeah. About how all this started. So, like, there is also the thing, like, similar to the machine learning work, the machine learning of work.

It's like, you know, I haven't, like, it has been a while since I have been hearing about types, right? Or, you know, that I've been like familiar with machine learning for quite some time. But like the reason why we started was because like certain events happened with machine learning was me meeting Sean and Jacko and we're able to form a band. And I actually tried to implement a type system

in 2017, 2018 and failed, like failed really quickly. And then Eric from the Elixir team, he also started implementing one. And that's like, so Elixir sometimes can find like type violations because we have like a very simple type system. But we very quickly, we ran into roadblocks. Like, oh, how are we going to solve this problem?

Because like, we don't have the proper background. We're reading papers. And I think both of us, we have a master. We have not gone further than a master. I don't even have a master in computer science. So we were trying to figure out this and we were basically stuck. And then somebody sent me a paper they wrote.

It was their master thesis, or they wrote for their master thesis in Uruguay about implementing a gradual type system for Elixir. And so they sent me an email like, hey, we wrote this paper. You may enjoy it, right? And I was like, oh, this is very exciting. Like, would you

be interested in working on this? And they're like, nope, we are out of the university. Probably never to be back, right? And I was like, oh, we are out of the university. Probably never to be back, right? And I was like, oh, okay, that's a shame. But I went to the citations in the paper, in the article, and then there was a lot of citations from Giuseppe Castagna, which is a researcher, very known,

has a lot of publications in the world of type systems. So he had a bunch of papers, and I start reading those papers. And one of the papers that I found was exactly about the problem that Eric and I were trying to solve. And the paper was that, oh, when you're implementing a type system, some languages have to solve this problem. And then, you know, I started reading the paper, like section two, the introduction. I was like, oh my God, god like this is exactly what i need this is the answer i'm going to solve my problems and then like section three well we can express this problem as the following like mathematical uh understanding of this and this and that and then it's like but the mathematical understanding and then section four now we are going to prove that our mathematical

understanding of the problem is correct and there's like a bunch of proofs like three four pages of proofs and then it never comes back it never comes back to tell me like here's how you would actually implement this right it's like oh and then i sent an email to giuseppe saying hey giuseppe like i read this paper i work on elixir but i have like i don't have the proper background right and and i basically told him this story like that his paper is basically i cannot understand it and and then he said oh i actually wrote like this larger paper for implementers for people who want to implement a type system and then he shared that with me and then we started talking if there

was a way we could like collaborate and work together and that eventually led. And then we started talking if there was a way we could collaborate and work together. And that eventually led to, we partnered with some companies. So a lot of companies have sponsored the type system work as it happened. Fresha, Superbase, Remote, Dashbit, Starfish. So many companies have been sponsoring and then we were able to make it happen, right? And the thing is that it's also about, like you have a lot of programming language, so the story is over. But the other thing related to this

is that there are a lot of programming languages, but we don't tend to think about this, but there are also a lot of type systems. Like those type systems, they are effectively different in how like they are implemented in their mathematical foundation behind them. So how do you find, right?

Like, you know, if you have a bunch of languages, how are you going to find what is the proper type system for our programming language, right? And the thing is that the approaches that we're trying before, they were like ultimately incompatible. And if the type system is incompatible,

it means that it won't be able to type certain features of Elixir. So you either have to say, hey, if you're using a type system, you cannot write the Elixir that you write every day, right? Or you do not have a type system. And

I wanted to avoid that, right? If we go into this place where there is like, maybe it's like an uncanny valley of gradual typing, which is one of the reasons why I say the project can fail. So what we did,

like one of the other reasons we had like those three years of research is to going through a bunch of Elixir constructs and saying, hey, does this map well to this type system? And this type system is called a set theoretic type system. And it looks fancy.

But basically what it means is that everything is implemented in terms of sets, in terms of unions, intersections, like things that you learn early when you are studying, right? So that's also something that makes me excited because if the type system is implemented in terms of unions and intersections there is also a hope that is going to be more accessible right and it's funny like all this started because you know two master students in Uruguay decided like to write the thesis but not only that they they decided to send me an email, right? If they didn't send me an email, maybe I would never have heard about it.

Yeah. So Giuseppe ended up helping you implement or how did the thing move forward from, you know, you were stuck, you mailed him back and said, hey, this doesn't give me what I need. He had a bigger paper. Did he continue to work on it with you or was it kind of a handoff of that larger paper and it was all. Yeah. So we got a PhD student. So he is like a, he's the poor soul with two bosses. So yeah. So, uh, Guillaume, so, you know, he's doing, he did a prototype. He's the one who is like effectively doing the work.

I believe of like, oh, I get this Elixir construct. It maps to this, maps to this. And then if he has a question, he brings to BEPA and BEPA is going to be like, oh, maybe you can use this theory. Or I just want to be clear, I am kind of assuming that's how they work. But and then BEPA was like, oh, actually, this is a problem that we have not solved yet we have to work hard or maybe we have to write new papers and what i really like about working with them is that so there's potentially it's like i like to ask a lot of questions because the way i learn things is about like kind of like trying to find the boundaries that's like how i i learn things

like i try to i try to find like to as much as possible what I can do with that particular abstraction, what I cannot do. So I can try to situate where and how I'm going to use it. And what I like about working with them is because this process of asking questions when work with them is a collaborative process where it's not like, no, you can do it.

No, you cannot do it. But it was like, huh, maybe we can do that in a different way that has not been tried before, you know? And that's very interesting because it's like, it's a learning experience for everybody. I definitely learned a lot, but the process of asking questions helps us approach problems in different ways. And one of the things that probably doesn't make sense, we can have a link to enter to talk about in the show, but we wrote an article for the Elixir website called Strong Arrows, a new approach to gradual typing. And usually gradually typed programming languages, they have to deal with some trade-offs in their implementation, which is like, or you can do like TypeScript. So TypeScript, for example, sometimes the type system is going to say, hey, this thing

is an integer or a number, but at runtime, the thing comes out at a boolean, which should not be possible, right? So, or you make the type system behave like that, or you make the type system work a lot with dynamic, which is not very useful because one of the reasons you want to work with the type system is to not work with dynamic types all the time, or you have to introduce like runtime costs. So this was one of the cases where we are not happy with any of the solutions out there. And we're able to develop a new approach to how gradually typed languages can be implemented that as far as I know, nobody has tried that before. And that's the exciting thing. And also the scary thing in that we are trying something that I think

like nobody has tried. And it's like, you know, maybe it means that we're going to, to find something really good that really suits the language. But it may also mean that we're just going to make a bunch of mistakes because it's kind of like a, maybe somebody actually tried that, but they didn't write about it because it ultimately failed. And you don't often hear about things that fail. So you're 13 years in to this Elixir project of yours. You've had some diversions. Maybe they're not diversions, but you've had time working on NX. You have Livebook.

You've done some other stuff that hasn't been right on Elixir proper, but now you're back on the Elixir programming language work. You seem as excited as ever. I mean, you seem as excited as ever i mean you seem very enthusiastic but i wonder like do you ever see an end to this road with you and elixir do you think like do you ever feel like i should do something else with my life or is this your life project now what's your your take on it yeah so um do you remember which year COVID started? Was it 2020?

Yeah, 2020. So I've always kind of tried to see myself or position myself as replaceable. That's my ultimate goal. One of the reasons is because, yes, I'm not going to be here forever, obviously. But also, I think in this sense, it's kind of like a sense of like, you know, kind of like it's fewer responsibilities and that's a very comfortable place to be at, right? It's like, well, if I'm replaceable, it means like fewer people depend on me. And that's very comfortable because I don't like people depending on me. So I think, so a lot of that has happened in Elixir design as a language.

So Elixir doesn't change a lot, right? Like as a programming language, it changes. It has been quite stable. I've said at this point, probably five years ago, that Elixir is kind of done and the type system, it's potentially the only thing that is going to change that. And I think it's going to be, it's going to have ripples, right? It's not like, oh, we have a type system, right? It's going to, I'm sure it, like, if we go all the way in, it's going to change some idioms that we have in the language. We may say, oh, that's actually, you know, maybe because we can type it or also because after we start typing it, we figure out that that's a bad idea. I like to draw like comparisons with the code formatter. When we added the code formatter to Elixir, sometimes people would come with code and say, look, the code formatter makes this piece of code ugly, right? And then when you went to look at the code,

it was because it had like a function call inside another function call, inside another function call, with something happening inside that function call. So like highly nested code. And then I'm like, look, that code, I'm sorry to inform you, that code was ugly, right? It's just that you try to squeeze everything together and maybe try to fit one line.

And the formatter is just telling you, it's like, yo, this is not going to fly, right? You have to change. That should be an official response from that, from the formatter. It's like, yo, this is not going to fly. That would make programming more fun, right? With love, Jose.

Yeah, with love, for sure. Yeah, and I think the type system is definitely going to have some of that. We are going to learn things similar to the format. It's not the format is perfect. The format has limitations and you're going to have to. And we adjusted as a community to that.

And the type system is going to be more because it's not only about opinion and style. It's going to be about actual software correctness. So yeah, so there is like this whole journey happening right now, It's going to be about actual software correctness. So yeah, so there is like this whole journey happening right now, which is going to take quite a while, right? And we started the whole machine learning stuff, but the machine learning stuff, like my idea for things is like, well, I'm involved in machine learning, but I hope eventually I'm going to find a way to

hand it off and have that whole thing continue without my input. It's just like, you know, it's like, I mean, my kids, they are five and nine, but, you know, like I'm eventually hoping that all my projects go to university, if you know what I mean. So that's how I see myself. And the funny story, the reason I asked for the year is that the year before COVID or maybe even the year that we had COVID is that I was really at a place where I was like, am I done? Am I close to done? Am I close to really making this 20% of my time? Am I close

to making this 50% of my time? And I was really considering that, right? And then when that happened, when I was thinking about that, it was when I got the email from the university in Uruguay that led me to Types, right? And it was when I met Sean and we started the machine learning work. So it was really funny that when I was thinking like, hey, am I close to done? Lifecom was like, no, boom, just slap a chair on my back. It's like, you have more 10 years of work ahead of you.

Yeah, good timing. Good timing indeed. So how you've been working on the type system, you know, with the team for a couple of years now, how far do you think you are from it being like, we can just call it, we have a type system. Are we talking another year, another five years? Like just roughly, what are you thinking? It's more like five because we don't call it, we have a type system. Are we talking another year, another five years? Like just roughly, what are you thinking?

It's more like five because we don't want to, we are being very intentional. That's the word I want to use earlier, intentional, being very intentional. We are being slow on purpose. I wanted our paper to be called a gradual, gradual type system, but it was rejected.

Oh, that's a good name. Right? Because we are like gradually introducing everything, the gradual type system, but it was rejected. Oh, that's a good name. Right? Because we are like gradually introducing everything, the gradual type system and so on. And because of that, I think it's going to take some time until we do everything. I also like to say that it's kind of like, I use this analogy a lot of like the boiling frog. I don't know if this is true, but in theory, if you put the frog in the pan

and then if it's hot, it jumps out. But if you start warming the temperature, and I think that's kind of how we need to approach this. If we just add the type system all at once, it's going to be a big shock to the community. But if we can do that in steps, we'll all be happy like boiling frogs.

So I think it's more like five years. Not that, oh, we are, right? It's like, not that we're going to have it in five years, but like the whole process being delicate as we are being is going to take about that time. And I talk about research, for example, and we had to research a couple of things, there were some things that we've identified that we can improve over the status quo. So there is one thing that we did not do, for example, that for us is out of the scope of the type system, which is, so in Elixir, we have processes

and you can send messages between those processes. I said since the beginning, I am not interested in that because very few people, they are sending messages directly. Everybody's using existing abstraction like gen server or tasks. So I said, I'm not honestly very interested on that part. So there are things that we didn't do. And maybe in 10 years, maybe somebody will have like cracked that problem in a very interesting way. But also there are things like we have behaviors today, like a gem server, and we can kind of add types to that thing, but it's loosely defined. So we are studying

bringing a post-doctorate to think about the problem of how to type behaviors, right? And this is going to be, you know, like somebody, a post-doctorate researching the theory on that for maybe a year and then figuring out a proper implementation. So by the time in five years where we are like, oh, Elixir is in a good place to type system now. Somebody say, hey, I like this brand new theory that is going to make behaviors in Elixir much better than they are right now and then need to consider introducing it. So I was used to say very naively some time ago that maybe I would retire when I'm 40. And I don't mean like retire, like I won't, I will stop coding,

but it's like coding does not need to be like the main driving factor of my life. Like I won't be like waking up for coding anymore, let's say professionally. I could try to find something else to do. But yeah, I don't think it's going to happen. I feel like I should probably revisit that to 50, you know, like, but I'm also like, man, like 50, it's 12, 13 years ahead. And I'm like, maybe 45 is going to cut.

But I think like 45 is not going to cut. So somewhere there. Speaking of decades and how you think about things, we had a great conversation about a hundred year programming language called HAIR. You may have heard of it. And you mentioned maybe in a decade, this researcher, this post-doctorate can research this. And in five years, or you just sort of laid out as if, you know,

knowing or thinking that Elixir will be here in a decade and be relevant in a decade. How do you or how does the community that helps you support this language and those folks who are sort of thinking, okay, let's get a post-doctor to do research, et cetera, how do you all think about the length of time that Elixir will be relevant and useful? How do you think about that?

Oh, I think it's a... Yeah, I'm going to say something that... I think it's not... I think the answer to that question is not technologically related at all, actually. Okay. I think unless we have a big shift in technology where it's like quantum computers are viable

and then everything that we use is kind of gone and we are starting fresh. So assuming that there is no, well, maybe AI is a big change that changes everything and then the good programming languages are going to be the ones that are good for AI. Maybe somehow that happens, but

assuming that there are no major changes to the industry or if there's a major change, we can survive to them. I think the relevance of a programming language is mostly building a community, work with the community, and marketing more than technology. So we use the technology, the Erlang virtual machine being great as the motivation motivation the things that attracts and excites people but after i think you do that initial jump it's all about the community being self-sustained continue being excited about what they do what what the build how they interact with each other than any technological aspect if if that makes sense.

There could be some truth to what you said with perhaps languages, quote-unquote, winning if they are better suited for code generation. In fact, I think Natalie Pistunovic on GoTime has a talk about that where she thinks that Go will eventually eclipse Python in usage because it's so well suited for code generation. Just the aspects of Go make it very good for AI-generated code.

That's not bad code. And so I don't know. That's obviously just an idea. That makes me think about tooling right now. As an Elixirist, I feel like the state-of-the-art GPTs are not very good at Elixir. I find them better at JavaScript. Those are the two languages that I write the most. And I'm curious if you have used them, if you agree with that, if you wonder. Because it's obviously a very successful language, like 99% more successful than,

or more successful than like 99% of languages ever invented. But it's still a niche language in the scope of the industry today. And so much so that when I tell people I write Elixir, some people have to ask me what that is. And I'm happy to answer it, but it's somewhat obscure. And I feel like the amount of blog posts and Stack Overflow answers and all the stuff

that the GPTs slurp up must be less than JavaScript and TypeScript because it's not very good at Elixir yet, in my experience. Yeah, that's the thing. So I think we are very early on the AI side of things to be able to say what is a good language for AI. Because I think right now a lot of the things are dictated by the corpus size, but it's not yet clear if when we are coding in the future, if Elixir has not passed the threshold of our good corpus size.

It may just have been diluted. We don't know that. And I want to go on another tangent that I come back to it later. So I think it's still unclear that's the case. So for example, I've heard last week somebody telling me that Alpho GPT-4 is good for JavaScript. They were having a really bad time using that for things like React and Next because things have evolved through time.

And for complex problems, it would frequently give mixed usage. Like, oh, like a pattern that it was using this React version, but not on this one. So it's like, you know, like maybe if you had things that were like fine-tuned per React version or for Elixir, we could have like a different experience.

But I also think that I feel like everybody kind of like, I heard people say like functional programming languages, they are going to be really good for AI because there is less state, there is less assumption being passed around, which makes sense. But I think ultimately we really don't know. It may mean that other people of people say typing is also good

because then you can go into this loop with the AI where it generates the program. And they're like, yo, this doesn't type check. So if you can do more static analyzes, right? But maybe there is a chance, maybe the best AI language is not necessarily textual, right? It's less based on the textual mode. Just that the textual mode and the LLMs

is what is the hot thing right now. So I think it's still a very open problem and I wouldn't try to guess. Sure, we're pretty early on in that. Yeah, and I think maybe other people, they can have a pretty good idea, but I'm also new in the machine learning space to understand what is like, how things are volatile,

because now we are all riding on this architecture, right? On the transformers architecture that everything's like building on top. But what if there is a new architecture, right? So we're talking about, oh, there are changes in computing, right? But it can also have like a major change inside AI. So if there is a new architecture right so we're talking about oh there are changes in computing right but it can also have like a major change inside ai so if there is a new architecture that changes everything like then maybe this conversation goes like in a completely different way well i agree with that i

think probably it's too early to tell and obviously people will try to decide what makes a good programming language for like you said for transformers and it may not be true for the next particular platform shift if there is one. With the state of the art right now as like Elixir community, have you considered from like a marketing or like a community perspective, like having a fine-tuned or, I don't know, ragged,

like Elixir, all-knowing Elixir chatbot that would be like awesome in Elixir because we made sure that it is by doing certain things that we can do. Yeah, that's a great question. And I have thought about it. I have built some prototypes about the best way to go about that. And it's, there's another funny story here, which is that I took some holidays early December. And then before the holidays, there was like the OpenAI, they had an event where they announced

like the GPTs kind of things. And I've been also thinking about AI in the context of Livebook. And then before I went to holidays, I told people like, hey, I've been thinking about this. I've been thinking about what OpenAI announced. And I probably think like the new APIs

they provided for completions we should build something on top of that because even if in the future OpenAI is not the winner or people don't want to use a closed source model you have like a hundred different solutions which adds OpenAI APIs to other models so it's like their API

whatever we think about it, their API is kind of going to be the winner, you know? So it makes sense to build against them. So I was like, let's build something against open API. And then I go on holidays and I come back from holidays

and apparently the CEO was fired and then they come back and then it was like, like in the two weeks I was on holidays, I know they came back. It's like, what happened? And then I was like, so I was ready. I was on holidays, I know they came back. It's like, what happened? And then I was like, so I was ready.

I was ready to go. Right. And then it put me back like, okay, take a deep breath. Relax. Right. It's like,

maybe not quite, but it's something like, so something that I want to do, I don't remember who did it. I think it was Coe. It was one of the companies that started with Coe. What they did is that they generated embeddings for all of Wikipedia. So if you want to do

something, you can just get the embedding and search the embedding. It's something that I have been thinking a lot about. What if we could generate embeddings for all the documentation that we have in Hackstocks? So people can very easily start building their own indexes, their own regs and this kind of stuff. Right. So we started exploring these. We already did some groundwork. We had some students at the University of Krakow studying different embedding models

so we could at work. So it's something that we have been thinking a lot about, like maybe having an official Elixir GPT, but it's still not sure if it's going to be something useful. We need to find the line between useful and gimmicky, and I think that's still very hard for a lot of the

AI stuff. But maybe gimmick is fine, that's how people get excited, but we need to be aware of where we will be investing our time. And maybe this year we are going to have something going in this direction. We also have Robert from, we have a machine learning channel in the Early Incosystem Foundation.

I know he fine-tuned some code completion models for Elixir. And we plan to evaluate those and give those a try as well. So I think we can do exciting things. But yeah, I think, I feel it's still very hard to draw the line between what is going to be useful and what is going to be gimmicky.

And that makes me a little bit uncertain of how I'm going to prioritize because there are a bunch of projects we can work on and improve the ecosystem. So is that the best use of our time? And this brings me back to the thing that I said I want to talk about, about the corpus size,

which is that I feel that starting a new programming language is actually becoming harder because... So there are a lot of things that got easier. Like, for example, everybody who started a new programming language, they are going to use LLVM, right? So a huge part of our compiler stack that got easier for sure. But the requirements for you to have a programming language that people are going to consider useful, they have just over the last decade, potentially a bit more, it feels like it has kind of ramped up.

So for example, before we were like, oh, you want to have a new programming language, you write a new programming language, you would share it on the mailing list, you would put on GitHub, people would give it a try. And I think, for example, one of the things that Go was very influential was making tooling first class. So today, tooling has to be first class. There is no discussion about that. And I think, you know, despite of everything that happened around that led to Go adoption,

I think having a good tooling since the beginning definitely played a factor, right? So yeah, you have to create a language now and the tooling. And most likely you need a package manager, right? And I like to use, again, Go as an example, because Go thought that they could, you know, not have a package manager, right? And if there is a language that could potentially not have one, is one that is backed by a big corporation, they're like, well, it does happen that we need a package manager, right? So you also need a package manager, right? But the other things that I like to say is that with the language server protocol,

you would think that that would help programming languages and that does help programming languages, but it kind of make editor integration table stakes. So now everybody needs to have editor integration. If you don't have a good editor integration, right? Because it's like, because it made it more accessible. Now everybody has to use it. Before, when I started with Elixir,

nobody really cared much about editor integration or about the language supporting it because a lot of programming, you had a bunch of editors and all those editors, they had kind of spotty integration with the language, right? So it was not like, people were not like, you need to have a very good one right but now you need to have a very good one you need to

have people and time invested on having a very good language server integration and ide and this kind of stuff right and now with ai right if it really becomes like a playing factor, a big playing factor, you need to have a big corporate size to train the AI. But how are you going to do that if you're just starting out, right? So it's kind of like it has become harder, but at that point you could work at it. But if AI becomes such a big player on developer tooling, like how can a new programming language go like cross that chasm, right? Like that cost, that gap, right? And I don't have an answer for that.

I think you said like Elixir is still kind of niche, but I think we're lucky because we have enough, right? We have enough for like to be there, to have a corpus size that is at least there, right? We have enough for like to be there, to have a corpus size that is at least there, right? And that's one of the things that worries me about, you know, well, if AI is not going to be decentralized, right? Then it means that me 10 years ago, right? If AI is not decentralized myself 10 years ago, maybe I wouldn't be able to start a new programming language nowadays, right? If AI is not decentralized, myself, 10 years ago, maybe I wouldn't be able

to start a new programming language nowadays, right? Because I can't customize the AI. So I think it's very important for us to have open source AI tools because it means that if AI becomes a big factor and you are a small programming language, there is at least an option for you to make a dent, for you to participate compared to everything being closed and hidden somewhere. I know how it works. I know how we get there. Wow. I suppose

you assume a little bit that developers stop solving problems because artificial intelligence is part of the game. I agree that if we rely on them more and more, that assumes that all programming languages have been, all problems have been solved in current LLMs and that status quo is long-term status quo.

And I think that's just not going to be true. I think that developers will eventually have problems with whatever can be done with what is status quo and what is supported by AI. That if we as developers stop solving problems, then new programming language cannot come around. Because hair, for example, is not, you know, it's a hundred year programming language. We talked about that, you know, a little bit earlier.

And we talked about on this podcast before. But that is, that's an example of where someone in particular sees a different problem and does not rely fully on artificial intelligence to generate their code. Right. So if developers at large and I mean like 99 percent or more rely on artificial intelligence to do all their work, then that would be a problem. But there's still that one percent and that 1% can still be influential. And there is still a path forward into the future LLMs and the training and the embeddings, as you mentioned. I think if for some reason the artificial intelligence locks us out, maybe then we have nothing new. We only have what's existing. Yeah, I think that's a very good take.

And I wonder if we get there, like, what are the dynamics, right? Like, does the 1% need to be, like, whatever we continue evolving, does it need to be really good for people to just find, oh, I am dropping my AI tools to learn that 1%? Because it's such a big boom that is worth like dropping some of the conveniences, right? So I think that's kind of like the, one of the main points, which kind of like, it's not really AI specific in the sense that it has always been the trade off and adopting new technology. And I think that's kind of why I talked about the editor tooling, right?

The fact that now everybody should have editor tooling. That's no longer a differentiating factor between programming languages, right? The fact that now everybody should have editor tooling, that's no longer a differentiating factor between programming languages, right? Because it's kind of table stakes. So yeah, I think that that's a very good point. Like if we continue being able to innovate and continue like finding things that are worth moving to and exploring, then yeah, it's just like, does it need to be a much bigger gap right now? And if it is, what is this type of investment and time you need to put into that? Sure. Well, this is how disruption happens though,

is not something comes by which is marginally better at one thing and also satisfies all the table stakes, what happens is something new comes out that is radically different than everything else that's come before it. And it's 10x or 100x better at one particular thing that everybody cares about. So for instance, it's 100 times faster than every other programming language, but it has zero AI and tooling integration.

And people are like, I don't care because I want the 100x improvement. Or it's 20 times more productive. You can write three lines of code and end up with Facebook.com or whatever. You're always a default to Facebook.com. Well, it's a huge application. I understand why.

It has so many features and everybody knows about it. But I was going to use Twitter. Then I was like, no, it's just been losing features all year. It's like way more productive in the way that it works. And so therefore, all the other things don't matter. And I'll put up with not having any of them because I want this one thing. And then slowly over time, everybody laughs at it. Oh, you don't even have VS Code integration?

Like no one's ever going to use that. And this is how it always happens. We just don't know what that thing is or what it will look like and maybe it won't. Maybe it'll just be incremental improvements and those are going to be harder to stake a claim because there's so much surface area

you have to cover like you explained, Jose. I also agree that what you said before about Elixir with community, that it won't be a technological thing that keeps elixir being relevant for years it's community it's you know non-change non-major paradigm changes in the tech stack and a couple other things and i think that's true too this is why i think essentially what we're saying is that tool makers need to remain toolmakers, not just AI implementers. That whatever the thing gives us, we must just assume and take and use. That we must always keep thinking for ourselves, is there a better way?

I think the future will be, I think, largely human-made and maybe augmented by AI at some point. Maybe fully augmented if we ever get there. I'm sure we will because that's how progress happens. But for the foreseeable future, at least in my lifetime, I think humans will have a vested interest in innovation and assisted by some other thing like an AI. And I think if we have tiny communities or indie conferences that people can still get to, or slacks to be a part of, or just groups to be a part of where human to human interaction and agreeance on challenges and solutions and research and all the things is still a thing i think we'll continue to have innovation that really maps to where we need to go and what we need to do and it won't be stopped or encumbered

by does ai support this or does AI not support this I think we'll we'll force whatever's necessary to have human advancement yeah I like that idea a lot and I think it kind of so Jaro was saying well you know you need like 10x improvement to an x improvement but a lot of people end up migrating to Elixir and I don't think Elixir is a 10x improvement over anything right like maybe between our choices of programming languages there is nothing really that is 10x right so so maybe for some reason i am overvaluing the tooling you know the tooling that is there only to support us but not do all the work you see what i mean mean? It's like... Yeah, it's on the margins. I mean, obviously I say 10x or 100x doesn't necessarily have to be those factors.

But if we took something like Rust, for instance, like if you're a C programmer and you've been writing C programs and here comes Rust and it's offering security guarantees that you'll never have with C. Rust now is established and has all the tooling and stuff, but imagine it didn't, but it still offered that. You might say, yeah, I'm going to do Rust because it's so much better at this thing that matters that the small stuff doesn't matter quite as much.

That's just one for instance. And which you can't really quantify properly, right? So the 10x, the 100x doesn't even matter at this point because you really can't quantify it. And in Rust case, I mean, there are other trade-offs there, but for the bugs, like going back to type systems, right? It's like it is provable that it is avoiding those kind of bugs, right? So, yeah.

And the reason why I say 10x is because when we do talk about the innovator's dilemma and disruption theory, it usually is 10 or more. But what it is is it's 10x less expensive. Usually there's one thing that makes it so much cheaper. And with programming languages, we don't talk about that because everything's free and open source at this point. Maybe adoption and labor and your investment time matters but most disruption comes because something is radically different and it's worse than the incumbent but it's 10 or more times less expensive and so you'll put up with worse because you're paying so much less and

that's just i'm just taking that whole dichotomy and applying it to programming languages which doesn't exactly fit either. Food for thought. But I don't know, why did I switch to Elixir? I think I was a Rubyist and I appreciate the freedom and dynamism of Ruby and I saw that in Elixir and I liked the speed of Elixir compared to Ruby.

Maybe it was 10x faster, I can't recall. But it certainly had, especially in web server performance, very impressive numbers with very similar looking code. If we take it at face value, once you get into it, you realize it's not all that similar at all, but has some shared pedigree with a language that I love, but was going to be a lot faster for me in production. And so that was why I switched, plus Chris McCord convinced me.

Yeah, I guess I'm contradicting myself all over this episode because... A little bit. Because, yes, there was about comparisons to Ruby, there are several cases about people migrating. And usually, it's usually a 10x on machine capacity, right? Like, so people are like, oh, we went from... Cores, wasn't it cores? It's all about cores. It was, usually machines.

It's like, oh, we were deploying like 80 machines and now it's four. And by the way, it's four because we are over-provisioning because we are worried that we went too low. That's usually how this story goes. So no, you're actually right. There were, in particular to Ruby, there are 10X. The other thing that I'm thinking right now,

so I'm enjoying a lot of this conversation. The thing that I'm thinking right now is because I was asking, like, why I'm so worried about tooling, right? Like, because I was saying like, oh, you know, like now the tooling is so hard, it's hard for our programming language to restart and maybe it's not that it's hard for the programming language restart but maybe it's for the programming language to cross a certain step throughout its evolution you see what i mean like maybe elixir the reason why i think so much

about tooling is because of course I hear people talking about tooling and maybe it's at a point in its adoption that in order for that to make the next leap, tooling plays a crucial role. And that's why I am indirectly putting so much attention on tooling and thinking about AI and think about which cyber protocol

and thinking about types because that may be part of the journey there. I don't know. It's definitely food for thought. I thought I had this figured out, but I clearly don't. Well, sorry about that. Well, I mean, you could draw lines to civilizations as well, you know, at different stages of civilization. Where you have, you know, the things that we care about and are concerned about, for instance, in the United States now, it's not basic sanitation, right? Like we're at the tooling phase of country, right? Like where you are, you know, obviously in some cases now we're regressing,

we do have sanitation problems, but you know, we've gotten there. Like when Elixir first started coming out, like you were probably worried about like, do I have the data structures right? And like, Hey, why can't I enumerate over this list the way I want to, you know, like those are core features of a language. And you're established and you're, you're there. And now you're worried about things that are higher up the stack, like tooling. And, you know, playing that game is how you make it better now. And so at different phases, different things matter. And that's probably why you're so concerned with tooling because that's where Elixir is. I mean, it's a great language. It's stable,

it's fast, it's productive. I mean, it's all the things that you set out to build, in my opinion. And so, you know, congratulations on the success, not just yourself, but the whole community. And now it's like, okay, how do we take it better from here? Well, apparently we gradually, gradually add types to it and that's what that's where you're at i uh i actually did some math i couldn't help it because i was thinking like you know when you think about directors or particular actors you think about like tom cruise for example i think top gun was like his all-time mavericks of course the latest one was his top grossing movie of all time for himself.

It trumped every other, and sorry, Jared, I used that word, trump. It has exceeded all past records for Tom Cruise as an actor in a movie. And so I was like, well, what has Jose done in terms of listenership on the Change Law Podcast network? Because you've been on Practically I, talking about different things. You've been on this show. You've been on this show. You've been on Now Friends. You've had Chris along with you to sort of talk through some things.

So collectively, you're responsible for at least 210,000 listens on our podcast. 210,000. Yeah. Across your one, two, three, four, five, six shows six shows now the seventh one that you're on right now may take you into the 250 this could be your top gun maverick right here it's gonna be your maverick right here well yeah just you know the title elixir is officially typed is officially language that's right officially in quotes we're to get that virality going. I do have one more, and maybe you answered this,

but I like a more direct answer if you haven't. I think you alluded to some things, or the challenges at least. So to Jared's point earlier, AI, as it's known today, GPT, ChatGPT, and others are not that good with assisting with Elixir programming. And so I guess the question is, what does it take to make it good? You mentioned embeddings earlier, you mentioned documentation being more readily available. What does it take from a, I guess, a leader in the Elixir world to enable LLMs to be better? Like what role do you play in that journey for them to better consume the documentation and

better know how to programming an elixir to help folks like Jared and myself or our team or others to to really become better and more proficient elixir versus just like anytime Jared you know asks chat GPT for assistance it's just like no it's not good so just quit you know so I think if I got the question right, I think we did our work correctly in the sense that at least from the language point of view, in the sense that like documentation was always first class. So documentation is very easy to access. So if what you want to do is to like configure an LLM, it's actually very easy to access

that programmatically, send that, extract information. And we talked about one of the things that you also have to do is try to get understanding from the source code. So you can find, oh, this code is using those modules, is importing those things. And those are things that you can do relatively easily in Elixir. We can most likely improve that. So I feel like we have the knife and the cheese.

It's just a matter of somebody going and cutting the cheese. I feel like the foundation is there in terms of having this information structured, but somebody needs to feed it somewhere. But again, we can go back, like maybe it's a corporate size, like maybe ChatGPT like indexed HexPM already. Not sure, right? Maybe it has done that. I don't know.

I don't know if I can send a letter to somebody, hey, please index my website. Or maybe it's a matter of, so one of the things is that RedMonk, they have, they release twice a year, like kind of a graph plotting GitHub against Stack Overflow. And I think they're having like the most popular languages according to GitHub and Stack Overflow. And then there's like a, you know, a linear thing in the middle. And it's very funny because Elixir is high on the GitHub side, but quite low on the Stack Overflow side. And one of the reasons for that is because we have always had the Elixir forum.

So that may be one of the things. Where's the knowledge? Where's the back and forth from the community? Yeah, the knowledge is in the forum. Is that thing being indexed? Because we know the Stack Overflow is, right? Yeah, right. And so, and ironically, that's one of the reasons

I think I may be misquoting that RedMonk, they are considering removing Stack Overflow from their plots because it's prone, like I think it has been losing relevance in the last years, right? But, you know, maybe in the effort of trying to have a closer community where everybody can engage with each other, where I'm active in the forum, and I'll probably not have

this patience if I was dealing with Stack Overflow, right? We created our community, a special place, but it's not known. So, yeah, so I think it's still like too many unknowns, but I think at the core, we unwillingly did a good job because we were worried about documentation being accessible, documentation being first class. So we did that and that can be, and we promote people to write documentation, lots of documentation, right?

So there is a lot there and yeah and maybe the rag is going to be the thing that is going to be enough that's one of the hopes right going back to we want everybody to be able to use this if rag is good enough then a lot of people would be able to augment their ecosystems without depending on on open air or whatever but we are still evaluating when you when we talked about sort of the the long-term future of Elixir, artificial intelligence, and that sort of larger topic of how long will be relevant and can AI generate it well, that whole conversation,

this makes me think of this necessity to not have a black box that is whatever AI is. Because just like you said, who do I send a letter to, to index my stuff so that my very relevant language today remains relevant tomorrow because tomorrow says AI will continue to be more and more relevant to developers in their journey to develop, right? So who do we send the letter to? How do we know? Well, currently the status quo of AI is for the most part, a black box, obviously open source LLMs and indexes have become more and more pushed because of this challenge. But I think this illustrates and highlights really the long-term challenge, because even you can't say for sure

why what wasn't indexed was indexed for the Elixir corpus, whether that's the forums, whether that's the documentation through hex documentation or whatever. It's unclear to someone like you how to enable chat GPT or the likes to better support Elixir assistance for developers using those things to use as tooling. And that's just not cool because long-term, we need to have inroads into those places so that we can be part of the future if AI is predicting how we'll get to the future. Yeah, and I think, yeah, it's still early.

I think we're going to improve a lot. I was listening to a podcast today where Sam Altman, he was saying like they improved ChatGPT 3 about 40 in the orders of magnitude in terms of size, performance and things like that since they started. I think 10 times for ChatGPT 3.5. And I think open source is going to catch up, I think. And I think that's the hope. But yeah, it's also like we go back to this when we are thinking about Livebook because what I want is for open source to win, right?

But when I'm building a feature for Livebook, right? I need to build the best feature for the users, right? And when I can use ChatGPT 4, right? And I can immediately see the results and they're reallyPT 4, right? And I can immediately see the results and they're really, really good, right? I can use other tools off the shelf. They're not as good, right? So we are a small company.

We are doing open source. So my options, if I have to choose for my users, is going to be ChatGPT 4 because it gives me the best result for the least amount of effort, right? I just, it's there. And this is like, so we were back in like about my indecision about investing in this

stuff is that because I want open source, right? I want things to be open source. But right now, the quickest return of investment is GPT. And then I am in this contradiction space, right? But yeah, and it's's just I think it's just patience we have to be patient and you know I think probably in one year and the whole thing is like it's crazy to think about is that this thing has been happening for a year only right it appears that this thing has been out for so long but it's a year and I think like if I'm back

on the show in a year, we may potentially be having a very different conversation. So yeah, we'll see. Do you have any fear about this? Like even as you respond to that, you sort of had some, I wouldn't say like trepidation in your voice, but you sort of had some uncertainty. Do you have any like fear and uncertainty and doubt, the FUD that people sort of pass around? Do you have any fear about this? No, not really in the sense that I consider myself like very lucky, very fortunate or whatever, or blessed, whatever you want to say it. I think maybe it's a, I'm not being overconfident here, but more like thankful that I think whatever happens to me,

it's going to be fine. I truly believe that what's going to make Elixir survive is the community more, you know, than whatever technological changes. Unless there's something very drastic. I talked to my father about this, about investments, right? So like when Bitcoin was in the crazy, right? And then my father is like, oh, have you heard like about this thing

that if you put your money there, like people got this huge return and then i always told him father if we got to know about it it's because it's too late you know it's like or or if something happens right it's like oh father like if something happens is because like if something goes this bad it's because it's going to be bad for everybody so like don't try to fight it. Right. So again, like, unless there's a very major change, I think I will be fine. Right. So I'm not worried about me in the sense, I always think more about, it's more about ideals, you know, again, like I like to say, well, me 10 years ago, that's where my trepidation is, if things go like closed source,

you know, and those things, they happen by, we don't see the results. Like I think another polemic topic about this, it's like, hey, I use Chrome. As soon as Chrome came out, today I don't use Chrome anymore, but as soon as Chrome came out, I immediately swapped to Chrome, right? And if I had known that this would lead to a point where, you know, Google is in this position where it has a lot of control over the browser, over the web, right? And over how we use the internet, like 10 years ago, I would probably not have used Chrome if I could have seen it, right? So I think that's where my trepidation comes from of like things being closed source like the developer experience so today another example today like elixir was the first programming language that github had like the new navigation code navigation things that

were provided by the community right so there were some programming languages and there still are where they have very good navigation and exploration on github ui and the path for that to get that feature to get that behavior was and i'm very welcome that i'm very thankful that the github team they you know they discuss with us and allow us to do that but that's closed source, right? And GitHub plays a major role over, you know, how developers use, right? So it all comes back to this idea of like, if you want to provide a good experience for your users, how much of that is behind something closed source that you have no control and you are depending on somebody, you know, paying attention to you, like, or you having a contact or, you know, me having a name

because I was very active in the Rails community that GitHub uses like 10 years ago, right? Those are the things that, but I, like, I feel lucky, you know, but it worries me, right? Like how much is being closed, right? How much is going to be out of our control?

And then the trepidation i guess is like what does that matter for the small jose out there right that's what's to start building his thing today and they won't be able to well you killed the vibe there jose oh thank you that's me at parties you know not invited just kidding alright well let's should we try to close on an up note on a high note on an upper

what's that wow had no idea I think we should end it right there Adam don't you think? We ended on a high note. This cheese and knife tactic there. I love it. Do you want higher?

I don't think it will be good for the listeners. I think that was plenty high enough for me. Adam, were you satisfied with that? That was a high note, literally. Yes. And I dig it. Yeah. Yeah.

Trying to go put ourselves back in the high note. We're going back to the discussion. It's all about open source. And I think if we can continue investing, if we can continue batting on open source and open source models, as Adam said, I think you had a very optimistic perspective there about why we are still investing in tooling. We may still be ahead.

This may be a horrible idea. Okay, this may be a horrible idea. Okay. This may be a horrible idea, but I started because my arms, they are, they are hurt. I started, uh, I went back, I go back to reading more and I started, uh, reading doom the book, right? And at the very beginning, it has a, it has a quote, which may make the mood even worse. Okay. Let me give it a try. Let's try it. So I've marked it because it was, so it says once men turned their thinking over to machines in the hope that this would set them free, but that only permitted other men with machines to enslave them. And how old is Dune? It's probably like 70 years old or something like that, right?

And I think, you know, maybe it's some nice food for thought. Maybe it's some horrible food for thought. But when I read that, I was like, huh, that made me think, right? You know, especially with everything that is happening with ai right now yeah it's on point i liked it i think maybe we should just end it right there just let that be the final word yeah you know i dig it i dig it no need for further doom and gloom we've already got it jose it's always fun always on your high notes and your low notes. Air quotes. I'm looking forward to May for Elixir 1.17 for your 1% to 3% solution deployed out there into my code.

I'm looking to make my code a little bit better. So thanks for all the hard work you've been doing on the language and all the things, man. Thank you. I am a very big proponent of under promise over deliver. So I'm hoping it will be way past 3%, but that's what I'm going to promise. The 3%. And yeah, I hope people will try it out.

And yeah, as Adam said, we'll continue improving our tooling and that's what we can do for sure. Very cool. All right. It's been awesome. Very cool. All right. It's been awesome. Bye, friends. All right. All right.

That is our show. Thanks for hanging with us. Next week on The Change Log, news on Monday, best-selling author and recovering data scientist, Joe Reis, on Wednesday, and on Friday, I'm not

sure. But Adam and I will be at that conference in Austin, Texas, the first part of the week. So I'm positive we'll have some fun stuff to share with you by the time we get back. If you're headed to that conference too, connect with us in the hallway track, on the main stage at 4 p.m. on Monday, and at game night later that evening. Seriously, do it. Thanks once again to our partners at fly.io, the home of changelog.com, to the beatfreakin' residents, Breakmaster

Cylinder, and to every one of you who help make Changelog's community the absolute best the internet has to offer. That is all for now, but let's talk again next week.