The Changelog: Software Development, Open Source - The first cloud native programming language (Interview)

Episode Date: September 5, 2018

Jerod talked with Paul Fremantle, the CTO and Co-Founder of WSO2, about their new programming language, Ballerina — a cloud-native language which aims to make it easier to write microservices that integrate APIs. They talked about the creation of the language and how it was inspired by so many technologies, cloud native features like built-in container support, serverless-friendly, observability, and how it works with, or without, a service mesh — just to name a few.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at rollbar.com and we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by Indeed and I had a really interesting conversation with Darren Nix, the group manager at Indeed Assessments. And Darren is running a remote first team that operates like a startup inside Indeed. And you know Indeed, it's a huge company, lots of resources, solving big problems, lots of big data. And Darren's team is hiring. Take a listen. Darren, tell me about the big picture problem you're solving at indeed assessments what our team does is we build

tools so job seekers can show off their knowledge skills and abilities when they're trying to get a job way better than a resume can and that lets employers find great hires a lot quicker too and makes the process better for everybody so you're running a remote first team looking to hire pretty aggressively java engineers front end to react engineers ruby on rails engineers ux designers You're running a remote first team, looking to hire pretty aggressively Java engineers, front end or React engineers, Ruby on Rails engineers, UX designers, business intelligence, and you operate Indeed Assessments like a startup that lives inside Indeed. Tell me more. Because we're basically a startup within Indeed, we get to hire folks all around the country, even if they're not in Austin or San Francisco or Seattle.

And that means we can hire really great engineers who want to be able to work from their home city, work on really big problems, but solve those problems in a startup-y way. You know, we host our code on GitHub or Rails and Redis. We use Postgres and React and we're push on green. So we deploy six times a day. So I've seen charts that say like, hey, we deployed 13 times this week. And I'm like, haha, we deployed like 78 times because we like to go fast.

And so what we're doing here at Indeed is finding ways to be able to continue to be startup-y, but solve really big problems and help hundreds of millions of people get jobs. So if helping out your fellow engineers get jobs sounds like an exciting problem and you like working on startup-y tools at a really big scale, send us a note, reach out. I actually interview every single person who comes to join our team. So I'll be meeting with you and I look

forward to hearing from you. So if you're looking to join a remote first team working on really big problems that will literally impact hundreds of millions of people head to indeed.job slash changelog to learn more and take that first step all right welcome back everybody this is the changelog i'm your host adam stachowiak editor in chief of changelog jerry went solo on today's show, talking to Paul Fremantle, the CTO and co-founder of WSO2, about their new programming language, Ballerina, a cloud-native language which aims to make it easier to write microservices that integrate APIs. They talked about the work being done at WSO2 and how they're selling a subscription to an open-source product, as well as consulting services, similar to the Red Hat model.

They cover the creation of the language and how they were inspired by so many technologies, cloud native features like built in container support, being serverless friendly, observability, how it works with or without a service mesh, just to name a few, and whether or not Ballerina will replace Java over the next decade. So Paul, you're CTO and co-founder of WSO2, formerly with IBM, a lecturer at University of Oxford. You got a lot going on, but you're here today to talk to me about Ballerina, which is a new programming language that makes it easy to write microservices that integrate API. So first of all, welcome to the ChangeLog.

Thank you very much, Jared. Nice to meet you. Nice to meet you too. Well, let's get to know you a little bit, Paul, and your background and what brings you to be the CTO and co-founder of WSO2. So I guess my real, you know, I grew up a complete geek. I had a ZX80 when I was, as soon as it came out when I was 12, I badgered my parents. My birthday is just near Christmas and I got them to buy me the computer for a joint birthday, Christmas and all my savings put together. Now, do you get double presents around Christmas time because of that?

Or do you get lack of presents? I would always try and kind of wangle a big present. I would say, look, I'm going to join them together. Give me the best you can. I like that angle. That seemed to have worked. Yeah. Cause otherwise you just end up with like a good one for your birthday. And then three weeks later, they've got no ideas and they give you a useless one. So, so there we go. So, uh, so I've, I kind of grew up as a massive geek and programming. I guess I've programmed in 20 different languages. And I ended up joining IBM kind of in the 90s, working on internet stuff and security and doing a bit

of firewall, a bit of web design, all sorts of kind of webby things. And then I joined a group just when sort of dynamic web applications were first coming out and people needed software. And we created some of the early software to do that using Java around servlets. Long before it was called WebSuite, it was called servlet express and then i got into building sort of helping people build web systems and then in the just at the end of the 90s beginning of 2000 i came across xml and i suddenly started getting into distributed computing and stuff, and integration and how you connect different systems together. And that led into something called service-oriented architecture.

And then, so in 2005, Ian and another guy at IBM were really into into soa and and web services and and distributed systems and we were building those at ibm and we realized that actually there was an opportunity to to do something in open source and to to set up a company so that's we set up wso2 back in august 2005 so we are as of about two weeks ago, officially teenagers. The company's 13 years old. Congratulations.

That's kind of fun. I don't know what a teenage company is like. Does it sulk in its bedroom? I don't know. And then we went from doing XML-based integration to REST and gRPC and event streams and all sorts of fun stuff. And I guess that's where I am today. So you've been in this game for a very long time.

Just to give some context, that 05, 06 era, I was coming out of university. So I feel like I've been doing this for a while, but you got me dwarfed. Curious what keeps you interested? What makes you wake up in the morning and think technology, you know, thinking about programming languages and service meshes and microservices? Like what's the kind of stuff

that makes you want to go to work still? So I think what's really exciting is that pretty much everything is becoming a network programmable endpoint. So, you know, you can connect to so many different systems now through APIs, so many different SaaS systems. You can build completely massive, powerful applications

just by connecting other stuff together. And that kind of ability to program the world kind of really excites me. I'm also quite big into iot i i my i did a i took some time out of wso2 to to finish up my doctorate and i was really focused on iot privacy and security and and how to build systems where you you can manage your own it devices and so this kind of explosion of endpoints, this explosion of things,

and then to me, what's exciting then is how easy can you make it to then program all that? How easy can you make it to interact and program and do these powerful distributed things? And that's, you know, I think that's really kind of fun. Let's talk about WSO2 a little bit before we get into Ballerina. You said you're teenagers now.

What is the company? What do you offer? Is it services? Are you contract development? What's the kind of stuff that WSO2 does? I think that will help us understand Ballerina a little more when we start to dive into it. It definitely will. Because although it's quite different from what we've done before, it's evolved definitely from our background.

So WSO2 is a completely open source company. So we're a software development company. We build software products that customers use. And we have around 500 customers in all sorts of walks of life. We have customers like eBay, Fidelity, StubHub. If you're in Europe, then we help the tax offices in the UK and France and Switzerland, which you may or may not like. We transport for London, runs all the underground and buses in London, and we help them build a system that manages all the roadworks and does real-time alerts to people about traffic throughout London.

So it's fundamentally software products, but we do help those customers as well. So we have some consulting, but most of our revenue comes from selling a subscription to an open source product so it's very much like red hat okay and the products are really we have four main products the first three go together really clearly so that the most popular product is our api management product so this is used by people like StubHub, Bank of New York, Mellon, Proximus, a whole bunch of customers. And they effectively offer remote network accessible APIs to their partners, to other developers who can then build systems that connect through those

apis and we manage the security we help build the portal for the developer to find out about it we do things like policy management and analytics on those apis gotcha and then sorry go ahead well i think that does speak a little bit to to of the circumstances in which Ballerina grows out of. Because it sounds like, I mean, a lot of people who are already programming languages, of course, they may have capitalistic interests in mind. But of course, they're probably solving problems that they have. And so is that the case with Ballerina? It was something that you guys needed. And so you decided, let's build it or give us a little bit of the background into that so the the next thing that we also do is we help

people actually integrate those systems both internal systems external systems apis legacy software databases so the way we've traditionally done that is with something called an enterprise service bus. And that's a piece of software that you configure with a graphical interface and an XML configuration language. And it helps link together your Salesforce, SAP, Oracle databases, RESTful APIs, and XML services, and so forth. And that's the sort of second of our products, and it's the oldest of our products, really. And it's a highly successful product. It's used by probably more than 300 out of those 500 customers but it's it's definitely a challenge to developers because it although it's very it's got a lot of capabilities lots of power

it doesn't really fit into the kind of edit build deploy, deploy, test cycle that developers like to do. So that kind of rapid agile development cycle doesn't really fit with this kind of enterprise service bus type software. Because why? It's a set of reasons. So firstly, they're quite big products,

so they typically have a slow startup time. So our product is kind of one of the quicker ones, but it still takes about a minute to boot it up. It doesn't fit that well into containers and Docker. I mean, we've done a lot of work and we've actually just released some improvements on that front. So we are still, as well as doing ballerina we're still kind of tweaking and improving the esb product but that's a challenge the fact that

it has a graphical interface uh to design flows which is backed by an xml syntax you know certain people love graphical interfaces kind of drag and drop, build my flow. But fundamentally, when you're a developer, you're just most productive in Visual Studio Code, IntelliJ, Atom, VI, whatever your favorite editor is. That's what makes you productive. And editing XML is not a nice experience. You want a language, you don't want an XML. And just because the XML has a graphical syntax, if you're a developer, doesn't really fix it. So there's that.

And also just, you know, there's sort of things like type safety. So, you know, one of the reasons developers have become really kind of rock stars in the last 10 years is because they're so productive and they do all these amazing things. And one of the reasons they do all these amazing things is because you have type checkers, you have all the compilation tool chain and the unit test tool chain that helps you build a continuous integration pipeline. And then you're really rapid. You know, if someone says, I want to change this, you can change it, and you can be sure that it's robust because you've got all the tests, you've got all the type safety, all the compilation checks. So those things are really valuable, and they don't really exist in the world of an ESP.

Let me tell you a few things that stuck out to me about Ballerina as we get into the language a little bit, and maybe the genesis story of it, how long you've been working on it, and what it is, and that kind of stuff. There are a few things, and you mentioned them a little bit. So you mentioned that the graphical interface of the XML configurator things aren't that maybe impressive or interesting to developers. One thing that's very cool about Ballerina is you have a textual and graphical syntaxes, so it's kind of the best of both worlds. I thought that was very interesting.

This idea that maybe this is the first or one of the first maybe cloud-native programming languages, a lot of the stuff when you talk about your ESBs and the system or one of the first maybe cloud native programming languages but a lot of the stuff when you talk about your esbs and the system that many of your customers are still using i would assume the reason why it's not like container friendly and all these things is because well it just predates you know these technologies or the at least the proliferation of these technologies and so sometimes just starting fresh even though it's a tons of work to start fresh, and maybe it's not even a complete fresh start, but we'll get into that is allows for things that just, you know, are incredibly cum is an incredibly ambitious project. And that just impresses me. One thing I read just today, you guys have an article on the News Stack and your CEO and I assume co-founder says that he envisions Ballerina replacing Java over the next decade. That's an incredibly ambitious thing to say and believe. And so

these are a few things that caught my eye about Ballerina. But tell us about the start of this new programming language. You mentioned why it was needed, but even inside WSO2, was there arguments? Was there a big decision? Was there a meeting that had to take place? Or were you coding this on the side? How did it get started? So I think all your points are really valid and they're really interesting. And I'll come back to the replacing Java. You know, so Sanjeeva, my co-founder, is much more bullish than I am. I'm very English and reserved and I would never say that.

Well, I liked that level of ambition. Even if it's misplaced, I appreciate that he's thinking that big. No, no, I think there's a difference between expectations and ambitions. So the ambition to do that is one thing. The expectation is another. So in other words, you know, having the kind of vision to say, actually, we want to create something that could replace this, that has that capability, that has that power, that has that ability is I think really, I think you need that when you're starting a new language, you have to have a big vision. You know, you have to have that. Yeah. Otherwise, why go through it all, right?

Exactly. On the other hand, to say, you know, to say I expect it, that's a different thing. Okay, fair enough. I'm more pessimistic than that. But, you know, but I kind of, you know, just for a second, let's divert from all the questions you've asked and just talk about, you know,

Java, for example, is used in lots of scenarios, but there are increasingly, you know, it's, it's really ended up as a server side language and on the server side, increasingly people are writing applications that are not just standalone applications that talk to a database. They're, they're all talking to SAS endpoints, to other APIs, they're writing microservices. So if you fast forward five or 10 years, you can say, well, what is Java going to be used for mainly? Pretty much all new programs, I think, will be writing sort of glue logic between network services. And so in that sense, if you see where the world is moving and you look at that point,

and you say, well, if we can do that better and we can create a language that really targets that, then maybe it's not going to replace all of Java, but maybe that particular use case you can see that that this language could this new language has there's there's an opportunity let's say for a language to be the one that is used to do that fair enough so so back to you know how did this start well this started by you know honestly we we we wrote this esB product I talked about. It was in production 10 years ago.

And over the last 10 years, we've probably had about 40 meetings where we tried to say, look, we did this. How can we do it better how can we make a better esb configuration model how can we improve it how can we write the kind of esb version two the next big one and in all those meetings we never really succeeded because that model, that idea was to still have this configuration language. And every time you have a configuration language, you kind of end up with problems and this lack of agility. And so, you know, about three years ago, Sanjeeva, actually, the CEO and co-founder, he just said, look, why don't we just write a language? And that was kind of like a massive moment of truth.

And the other thing that I think really influenced it, another really interesting aspect of this is you talked about the graphical syntax. So the graphical syntax of most ESBPs is a sort of a flow. It's like a pipeline with occasional divergence. But when we actually sat down to build systems with customers, the pictures we ended up drawing the most, the ones we always drew, the mental images we have of these kind of distributed systems is always a sequence diagram where you have, you know, you have the columns, these, these lines

coming down, which represent different parts of the system, different entities. And then you draw the lines between them that in the capture, the interactions between them. And the the other real inspiration here was to say you know what if we used what if we use the sequence diagram as the kind of inspiration for a programming language and that's that's kind of a a weird thought but it sort of says that the you know it has real implications for the concurrency model for the for the how you manage workers and and independent parties how you think about services and endpoints so that that really kind of gave ballerina a kind of a an amazing foundation to say well actually we have a we have a model for the language. It's actually

a sequence diagram. And that's something you just mentioned is that any ballerina program, you can say, show me what the sequence diagram is, and it automatically, we can draw that sequence diagram of the interactions that are happening between different parties. So that does sound very cool and innovative. Is there prior art or is there is there anybody else that was doing this? One of the things I appreciated about maybe it was a blog post or maybe it was on the Ballerina website, which all these things are linked up in the notes for those curious, was all your influences. And I love when people who are creating new technologies aren't doing it in a bubble. And you're java go c c++ russ haskell all

these you know all these languages which you kind of have to have a breadth of knowledge in order to write a new language but so i'm curious if that particular aspect which sounds foundational and very differentiating of having these diagrams built right in or as a foundational part of the system has anybody else done that before? Is this a first? The only thing, and it's a sort of an oblique aspect, is that there's this website called Web Sequence Diagrams. And it's not a programming language,

but it's for drawing sequence diagrams. And there's actually some Eclipse plugins and other tools that do the same thing. There's something called Plant UML that does the same thing, but in open source. And what this website's really nice is you don't draw the sequence diagram you don't you know what people have done for for 20 years is pull up you know vizio or powerpoint or some kind of drawing coral draw or something and physically you know lay out the lines and everything right with web sequence diagrams you write a text you program you type a little definition of the sequence pattern between the different parties and it draws the diagram for you and we just would we sat in meetings doing this and

and i think input you know somehow behind the scenes this may have influenced the idea that maybe you could write a programming language on a sequence diagram. It's not the same thing, but I think there's a bit of a leap from one to the other. Well, a massive leap. I'm not trying to say it's not a big leap, but maybe that inspiration came a bit from that. This episode is brought to you by DigitalOcean. They now have CPU optimized droplets with dedicated hyper threads from best in class Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one-click machine learning and AI application image.

This gives you immediate access to Python 3, R, Jupyter Notebook, TensorFlow, Scikit, and PyTorch. Use our special link to get a $100 credit for DigitalOcean and try it today for free at the do.co.changelog. Once again, do. just make a programming language. What happens after that? Because to me, that's where my brain just explodes. When you decide we're going to do this, like, where do you go from there? What's the first step?

So, you know, we have an unusual company, which is that we are now around 500 people. We've taken multiple rounds of venture capital funding, but none for a couple of years now. We're cash flow positive. So we have very happy customers who give us repeat business and we're growing pretty strongly. So as a result we have you know we have some spare capacity so you know what what what WSA2 did was just to stick a you know a couple of smart people off on the side prototyping going through little iterations what does this language look like and you know that's that's a

luxury that i guess not many companies have you know but it was it was a nice luxury for us to have which was that was kind of small enough to have a very clear focus but big enough to have the ability to to do this kind of work on the side. Effectively like an R&D budget would be the equivalent. Yeah, I mean, obviously we have a kind of more normal R&D budget, which is about improving our existing products and investing in those. And we have a small research team that writes a few papers and does some sort of longer term research but

this was a sort of yeah it's part of the r&d budget but a kind of little side like a skunk works but a bigger play like a you know trying to hit a home run yeah exactly so were you one of these people i mean cto probably not probably got other things to do so i wasn't one of these people and I regret that. I regret that. And, but, you know, even sort of being a side player on this, it's been really fun. You know, I don't take a lot of credit for this. I'm just the mouthpiece that gets to talk about it really. But it's still an exciting initiative and, and it's, it's kind of really exciting for me because,

you know, I always, you know, I, I love programming. I, I, you know,

I, I really, I'm not a great, to be honest, I shouldn't tell anyone that. Why not? I'm not a great programmer, but I love programming. I just,

I just adore it. And I even like, you know, I said how annoying that XML syntax and the ESP stuff is. I like hacking on that. I like hacking on XML stuff. You mentioned earlier that I teach some courses at Oxford University. One of the courses I teach is on service-oriented architecture and of course as part of that we do all sorts of things we use Spring Boot and Jaxx RS and all kinds of Java and Node.js and Python HTTP stuff and and we you know we would do an ESP example and show people what that's like. That's part of the course.

And the last couple of times I've run it, I've replaced the old ESB example with a ballerina example. And what used to take a sort of a lab exercise that used to take people about two hours and lots of struggling and kind of like, I don't get this. You know, they just with ballerina they just finish it off in between 30 and 40 minutes and they get it instantly and they love it and and that kind of you know that kind of joy of programming

but doing kind of complex distributed stuff in a really easy way is, is quite liberating. It's quite, it's quite kind of enlightening. Well, I, in my mind, I stopped at, you're not a great programmer. I was curious, you just stopped me dead in my tracks. I'm thinking what makes a great programmer and why aren't you one? And then I heard you say you like hacking on XML files. And I thought, okay, maybe, maybe he's not such a great programmer just so i just love creating stuff jared i love creating stuff and i you know anytime i can create something that does something yeah then that gives me a buzz it just it's the best yeah and if i have to use an xml file or i have to pipe together some Unix commands into a kind of ugly thing, or I've written Perl before, I'm not proud of it, but I've done it.

Anything that actually makes something work and does something, to me, is an achievement. It's a thing. So what makes me not a good programmer is that it's that buzz I get from making it work the first time. Oh, you don't want to make it better. You don't want to make it resilient. I don't want to fix all the bugs

and handle all the error cases and document it. And I want to move on. So once it works, I'm happy. Here's the secret is nobody wants to do that work. But somebody has to. The great programmers, they realize if I don't do this, it's only going to work right now. It's not going to work later.

Do you have kids, Jared? I do. Excellent. I once wrote a blog years ago about, it's probably still somewhere on the web but about how before my first child was born i was really focused on the birth getting this getting this kid and the birth is kind of difficult and tricky and whatever and but it was fine you know she was born she was fine she's perfectly healthy baby and then we got her home and then i realized that the that was the wrong thing to focus on. It was the maintenance of Anna.

Not the launch of Anna 1.0, but the ongoing maintenance of the project that was the challenge. Yeah. It's the same way with marriage. So many people focus on their wedding. And that's the first thing you do, right? So, of course, it's a big focus. But's the same way with marriage. You know, so many people focus on their wedding. And at first, I mean, that's the first thing you do, right? So, of course, it's a big focus. But so much emphasis on the wedding and the planning and the cost and where it's going to be and who you're going to invite. And it has to be perfect.

And this and that and the other thing. And then they don't realize that the wedding is not the marriage, right? The wedding is merely the first day of the marriage. It's day one, exactly. The marriage comes after the wedding. And it's way harder and way more important. But yeah, we tend to focus on beginnings, that's for sure. Okay, so let's loop back in somehow.

There's no good segue back to where we were, but we're talking about ballerina. And we're talking about, we talked about its influences a little bit. We talked about its genesis. You had R&D budget. You had this even like skunkworks budget a couple people working on it figure out things right like what's it gonna look like of course we have this great idea of the graphical uh diagram based kind of underpinnings but what are the language influences what does it look like what came out of it let's talk about the language its features and give it look like? What came out of it? Let's talk about the language, its features, and give me the nitty-gritty of what Ballerina is all about.

It looks sort of similar to Java, JavaScript, because it has curly braces and that sort of thing going on. So it's not too dissimilar to C Sharp and so forth. The type system is a little bit more like a functional type system. So the type system has what's known as a union type system. So I can say string or XML or error response equals. And, you know, in a lot of languages, you would have to create a wrapping object

in order to do that. But in this, you can just say, this could be a string, it could be a union. I mean, it could be an XML or it could be an error. And actually, that's very reminiscent of what you actually get over the network so so sometimes you might be talking to a old xml service but there's some proxy server or gateway in the way and when the proxy server sees a failure then it sends back a text message

when the network has a failure then you get a local error and when it all works you get an xml so so those kind of union types are there and and it's a strongly typed language so although it has it has a type inference so you can say var response equals and and the compiler will work out that it could be a string an xml or an error but um it does validate all the types the other thing that's really interesting is it has um is that null is not a part of like things like strings and ints cannot be nulls so there is a null but you have to say it's a string or a null. So it explicitly makes you handle nulls separately from normal values, and you have to deal with them. So it's deliberately forcing you to firstly say,

well, this could be a null or it couldn't. And if it can be a null, then deal with it. So in a way, that's encouraging you to say, well, if it's, you know, most of the time where things are nulls is usually not true. It's an error or it's a value. So in other words, it's just as easy to deal with an error as it is to deal with a null. So there's no point in using a null, if you see what I mean. So that's a really nice aspect. So, and the concurrency model is based on communicating workers.

This has some similarity to Go and the Go routines. They're based on CSP, a is a kind of mathematical concept and in go the channels have names so in other words you have named channels in ballerina the the name is implicit so you have a channel between two workers and the name between that worker that is really implicitly based on the two workers so but it's similar to that you have these these sort of channels between workers for communication so those are some of the things that that look like other languages and it has imports and it has a main function and um or but then there's some things that don't look at anything like other languages. So firstly, a lot of, a lot of programs don't have a main, they aren't really a, you know,

I start up, I do something and then I exit. A lot of programs these days are providing a service that just sits there and waits and then gets called over the network by GRPC or Kafka or something else or HTTP, of course. Right. So a service is a first class citizen in Ballerina. So a program can have a main or a service. It can just say, I'm not a main, I have a service. And that's what I do is I wait for people to call me.

And does that service imply like a network service, like sitting on some sort of port, like a TCP service? It does imply some kind of network service. So that's a kind of implicit. And so that's talking to, you've got some kind of endpoint there. An endpoint is another thing that's a first-class citizen of the language so whenever you talk either when you whenever you listen for other people to call you or whether you talk to others uh there's an endpoint defined and and that's a a key part of the language because here here's the here's how that's interesting which is that there's a syntactic difference

in the language between when you call an endpoint and when you call a local object. So when you call a local object, you just, like in most languages, you go a.method, you know, I go, you know, myV know uh you know my value dot set temperature but when you call a remote endpoint there's a little arrow you say i want to send a message to this http client endpoint this http server over there and i go a arrow post and so that's there's two things about this one is it's kind of implying you know

it's giving the programmer a visual difference between distributed local calls and i think that's you know there's a lot been written about the fallacies of distributed programming but it comes down to this that you know distributed programming is not the same as local programming. And so seeing that right there in front of you, hey, I'm making a remote call at this point. I have to now be aware of network errors. I have to be aware of load balancing, circuit breaking, discovery, all the kind of things that happen when you do distributed computing is a pretty important thing.

The other thing is that under the covers, we do everything in an async way. So one of the challenges of building these kind of network systems is, you know, what do I do while I'm waiting for this web server across the internet to respond? Do I block a thread? Do I sit waiting or do i do it in an async non-blocking way and everything every time you use that arrow to call a remote endpoint

we under the covers are doing non-blocking io but you don't have to worry about that in the language. Nice. So if you're writing Node.js, you use callbacks. And I know there's a lot of great libraries in Node.js to make that better. But the fundamental model is quite difficult for developers to kind of keep track of all these callbacks and so forth. So I know this from firsthand because i wrote my a lot of the code for my for my phd thesis in in node and and literally a few months later i went back and looked at it and i was like oh my god what was i doing i mean it all works fine but i have no idea how sure so so this is this kind of takes care of that in In other languages, the default is to be blocking. And then if you want to do non-blocking async IO,

you have to go find all kinds of clever libraries and packages and learn a new programming model. In Ballerina, it just happens under the covers because we kind of designed this to build network services and talk to network services out of the box. That's really cool. I want to go back to the arrow versus the dot in terms of the, what would you call it, the remote procedure call versus a local procedure call. It seems like as a developer, I don't want to know the difference. Like, you know, there's this idea

of the uniform access pattern. So like, you're kind of violating that. I understand the reasons why it's like the distributed world is, is worse than we hope it would be. And we just want you to be aware of this, but isn't it, I guess maybe in a perfect world or in a better world, wouldn't I not have to know? Because like you said, you're going async in the background for me anyways,

and I don't have to program in an asynchronous fashion. So that seems like a it'd be nice if there's language that just papered over that and yeah if there are networking problems you know it has to deal with it but i don't have to deal with it is that is that just a pipe dream or so so we were discussing how long we've been doing this yeah uh we've been talking about making you know distributed computing seem transparent you know for 20 years i i remember going to a to the launch of a system called dc distributed computing environment i still have a mug at home beautiful mug that i got given at the launch of dc and and this was you, this was maybe, this was before I joined IBM.

So this must have been 1993 or 92. And they were trying to do that then. And we haven't succeeded yet. So we kind of have come to the conclusion that you just can't, you can't paper over it. And it's better to just say look okay this is local this is distributed and be aware of it now we can make it life as easy as possible for you yeah and we've done a lot of things in the language to help deal with those sort of problems but you know i don't think you can solve that i don't think you can say that

that these two things are identical and you have uniform access. Well, I'll just defer to your experience. No, no, I never say never, but certainly it seems to be more effective to kind of admit that this is a remote part of the call and this is what we're doing. I think that's fair i

think when it comes to readability going back to code and reading it i think saying you know i think in that case explicit is better than implicit and so saying yeah it's an arrow i know for sure this is going over the network versus i don't know let's go check the method that you're calling and see if it's a service or see if it's a local thing. So, yeah. And so go back to that sequence diagram. You know, you have these different parties communicating and you have this arrow that says, hey, I'm talking across boundaries between you. So it's very reminiscent. I mean, I know we draw the sequence diagram from it, but it's also very reminiscent in

the text as well of what's going on here. Very cool. That makes a lot of sense. All right, let's move moving forward. What else you got in terms of unique bits or design decisions? Something that makes ballerina stand out from the crowd? So I think that those core things are really important. The other thing that we've, the other things we've done that are kind of nice, some of them are unique, some of them are similar. So if you look at things like C Sharp, they have this thing called LINQ, which is Language Integrated Query. And it's basically, instead of using SQL, you sort of jump into, you have a real program syntax that is reminiscent of SQL and allows you to code queries. So we have the concept of a table in the language.

So effectively, you say I have a record structure and a table is a set of rows of that record structure and you can query that table. And that table can be backed by a real SQL database or could just be in memory. So that's just like C-sharp's stuff, but it's very nice. I'm not saying it's not great. It's really cool, and it makes you very productive. But then something that we've taken and done that I don't think anyone else has done in that way is that we have one of the things you often do in in distributed systems is you start looking at events and you start trying to process those events and and if you're just processing one event at a time then that's kind of a service

but you start thinking well actually I want to know what's happening to these events over time. So a classic example is I'm maybe trying to spot someone trying to break into my system and or trying to trying to build a fake app or something. And what I see is I see multiple logins from the same IP address happening within a certain period of time. And so now I'm not just looking at individual events. I'm kind of saying, well, what happened in the last 10 seconds? Did I see that? So we've built that concept of a stream right into the language. And the language integrated query allows you to query across time as well across those events so you can you can really

quickly and simply say you know tell me you know if if i see the same ip address sending a login request in the next you know over 10 minutes then we're gonna we're gonna um send an alert out or disable that IP address. So that's one aspect that I think is really cool. Another one that I think is really important that I haven't really come across is that we're trying to really build security right into the language in a way that I haven't seen before. So, you know, there's two aspects of security in distributed systems that are kind of difficult. One is identity. So most people nowadays have moved to kind of try and use token-based systems for identity across distributed systems.

So things like OAuth tokens are very common Google Facebook GitHub everyone uses them all the time but typically that kind of identity model and the idea that I might say well okay I've got a request coming in it's got a identity attached to it it's something that's handled through some, some libraries or whatever. So we've built that right into the concept. So, so the concept of identity of callers and, and we're actually building the identity of the service itself into the language.

That's, that's some research we're doing right now around a standard called spiffy. And so we're, we're kind of trying to build the identity of the the service itself into the language but then the second thing that's a big problem in distributed systems is basically spoofing and tainted data and injection attacks so that's a another big challenge and we've built that kind of concept right into the language as well. So when you receive data over a network socket into Ballerina, we automatically realize that that is potentially tainted. And the taint analysis is part of the compiler and part of the compiler checks.

So effectively, you as a developer, your code won't compile if there's potentially a SQL injection attack here. Because unless you've actually cleaned that data and validated it's not tainted, we won't let you use it somewhere where it's dangerous. I'm just over here looking at your try the language with Ballerina by example on the philosophy page.

Because one thing that I think of with a new language, I think you're demonstrating, I mean, you're just demonstrating how much you guys have already accomplished um and you always think well it's brand new or it's a few years old surely there's a lot of missing pieces or there's there's things that i cannot do and i'm sure there are things that are missing but i look at the the list of things that are accomplishable and just looking at your security section like you mentioned the, the taint checking, a secured service with JWT, basic auth, OAuth 2, you got Swagger stuff, gRPC. I mean, it goes on and on. Testing built right in. What's not there yet? What's glaringly obvious? What are you trying to get to? Or what am I missing here here so before i answer what's

not there i want to make a really interesting point which is that effectively you know we we spent kind of 10 years building middleware and we have these products and they're kind of big products and and you know we we always were proud that our products were a fraction of the size of Oracle's and IBM's equivalents. You know, so what we would ship in a, you know, 200 meg download or 300 meg download would be what they would ship on a DVD. Because it was so big, you wouldn't want to download it. And effectively, you know, we've kind of built pretty much everything our products do into a language so effectively we've taken what was you know four products you know four big products each you know a few hundred meg of download and and a lot of complexity and so forth

and put it into just just a language so it's it's kind of mind-blowing in a way and and one way of looking at this is to say well you know this is a this is a way of sort of saying well you don't need middleware you know middleware is a sort of set of these servers that do stuff you just need the right language to do things so i guess what i'm saying is it's remarkably productive what you can do once you have a compiler and once you've built up a certain level of maturity of the language so we you know and with the right vision it sort of became quite quick and effective to do a lot of these things you're talking about so so what's missing you know i i think i i don't think there's anything really

really major missing i mean i you know there are certainly things that are in our to-do list and we haven't declared the language 10 yet so so it's still morphing right the actual run time's very stable if you want to build something today, we're pretty confident that you can build a production system with it today. What we're not confident is that if you build a production system, that you won't find it won't compile on the next revision because we're still tweaking the language a little bit. So I think we want to get to the point where we think this is really a robust, resilient foundation for the future.

And we're not quite there yet. Every time you make a nice tweak and you tidy it up a bit, you think, oh, well, hey, what if we did that there? So the union type system that I was talking about earlier, a really good example of this is we have json um built an xml built in now you can just think of json as being a union of different things json is either a string or a int or a array uh or a or another json and so you know we kind of have put in this union type system we're like oh we could rebuild our json our JSON support just as a union type of other stuff.

Yeah. So there's definitely things we're doing there. You know, I don't think there's anything really major that's missing. You know, maybe it's because it's late at night and I've forgotten some massive things on our to-do list that we should be doing. I should say we're recording this across time zones. So it's early in one time zone and late in another. But yeah, no, I don't think there's anything really major missing, but there certainly is a lot of kind of tidying up. I guess the one thing I would say that is not there yet is the sort of standard library. So, for example, we haven't got the ability to do complex sorts of strings.

We don't have, you know, we have lists and, sorry, we have arrays and records and maps and tuples, but we don't have list objects. So some of the sort of things you'd expect to be in a standard library aren't quite there yet. Now, that's not really the core of the language, but obviously they do need to be added. Yeah. So getting the standard library really up to the sort of spec of, you know, the C standard library or the Java standard library is going to still take a little bit of time. So I have some pretty awesome news to share. We are now partnered with Algolia.

If you've ever searched Hacker News, Teespring, Medium, Twitch, or even Product Hunt, then you've experienced the results of Algolia's search API. And as we expand our content, we knew that one day we'd have to either roll our own search solution on top of Postgres, or we could partner up with Algolia. And I'm happy to report that phase one of our search is now powered by Algolia. We're able to fine tune our indexing, gain insights from search patterns and analytics. We can create custom query rules to influence ranking behavior, as well as improve our search experience by adding synonyms and alternative corrections to queries. Sure, we could build search ourselves, but that would mean we would be busy doing that instead of shipping shows like you're listening to right now. Huge thanks to our friends at Algolia for working with us.

Check the show notes for a link to get started for free or learn more by heading to algolia.com. And by GoCD. GoCD is an open source continuous delivery server built by ThoughtWorks. Check them out at gocd.org or on GitHub at github.com slash gocd. GoCD provides continuous delivery out of the box

with its built-in pipelines, advanced traceability, and value stream visualization. With GoCD, you can easily model, orchestrate, and visualize complex workflows from end to end with no problem. They support Kubernetes and modern infrastructure with Elastic on-demand agents and cloud deployments. To learn more about GoCD, visit gocd.org slash changelog. It's free to use, and they have professional support and enterprise add-ons available from ThoughtWorks. Once again, gocd.org slash changelog. It's free to use and they have professional support and enterprise add-ons available from ThoughtWorks. Once again, go cd.org

slash changelog. So as you mentioned, Ballerina, not quite a 1.0, but still very feature complete, very useful, downloadable now, available for Linux, macOS, and Windows. Tell us what it's like using Ballerina, building something from scratch. Take us maybe from like, I'm hitting the download button on ballerina.io to I've written a simple service and I've somehow deployed this into a cloud or into a production environment. Yeah. Okay. So the download comes in really four major flavors. So there's, as you said, Linux, Mac, Windows, and there's a zip file. Now, there is one thing that you asked me just before the break, what's missing. So at the moment, we've built Ballerina in a kind of interesting way. So we've built a bytecode specifically for Ballerina

and a bytecode interpreter, the ballerina virtual machine that interprets that bytecode so that's the same way c sharp is written um same way java's written a whole bunch of languages now that bytecode interpreter is built on top of the the jvm so at the moment you need to have have Java running in order to run it. So you don't see the Java. The Java is not visible, and you can't call Java libraries. So this is not like Scala, where it's kind of a mixture between Java and a different language. This is really a clean separation because it's got a different type system, different threading model, different worker model,

different concurrency, and so forth. So one of our big goals is to completely rewrite that Ballerina virtual machine on top of LLVM. So at that point, this will generate native code today. It generates what we call a ball X file, which is a bike code, the ballerina bike code. Gotcha.

In the future, the, the LLVM will help you generate native code for your particular Linux, Mac OS system. So, so just explaining that when you download this, the download includes the JVM. We embed the JVM into that Linux, Mac, and Windows distribution.

There's also a zip file, which is a bit smaller. If you already have a JVM, you can just download it and run it on top of your existing JVM so you download it and you start writing code and there's a simple command line ballerina build you create a our extension is a dot bal file and you then build that, that creates the Balex, and then you do a ballerina run and point to the Balex, and that runs it. So it's kind of a bit like you have the Java compiler and the Java runtime, or you have a Go compiler.

It's very similar to most programming languages. Now, I would recommend, if you are downloading this and trying it out, that you also download a plugin for your favorite IDE. So we have IntelliJ and Visual Studio Code are probably the two best plugins, the most feature-complete. I personally use Visual Studio Code. So that gives you all the kind of tab completion that's going on.

So, of course, that does help, and it has syntax checking within it. So if you've made a syntax error or there's going to be a compilation error, then it will tell you in your code editor by underlining it. So, you know, from my point of view, if you were going to try and get started with this, I would download the Ballerina distro and I would install the Visual Studio IntelliJ plugin. Now, those are both in the respective repositories. So if you start editing a.bal file with Visual Studio Code, it will say, hey, we have something in the marketplace to help you and you can go and install it. If you start up IntelliJ and go to settings, there's a plugin and then there's a plugin repository in the internet.

You can find it there. So you don't have to download anything for that. So let's say I've done all that and I'm loving this new little ballerina service that I've written and I want to make it a production deployed thing. Excellent. So let's suppose you want to deploy it into Kubernetes.

Okay. So that's going to mean you're going to need to build an image a docker file and you're going to have to create deployment yamls and service yamls and all sorts of stuff to say this is how it deploys this is the end points and the ports it listens on here's the config files it needs and so forth. So in Ballerina, what we've done so far for Docker and for Kubernetes, and we're looking at other orchestration systems as well, but let's say Kubernetes, so let's say you've written an HTTP service. So it's sitting there.

It's going to listen for requests on a network port. So you have, as I said before, and it's hard to visualize this because we're just talking, but you have an endpoint definition that says I'm listening on port 8080. And I have a service definition that says what happens when a request comes into that port. So you can actually use some annotations in the language so in your dot file file just you have this one file file you have an annotation in there and you say i want this to be a kubernetes deployment and so you give it an image name you give it a service name and if you need a config map to pass some config file to that,

you put that in there. Now, when you type ballerina build, it doesn't just build the bytecode. It actually creates those deployment yamls for you. It creates the Docker file, and it will even push it to registry as part of that build process. Nice. And so now, and it just says at the end of the compilation step, it says if you type kubectl apply minus F and it gives you the directory, that will deploy this into your Kubernetes.

And bingo, you try it out and it does it. So, you know, one of the kind of aspects of this is we've really tried to make it, you know, the concept of we've clearly defined what maps into a container, how do you build that container, how do you deploy that container right into the language. The last thing I'd like to chat about before i let you go paul is the community side of ballerina so this is a fully open source apache 2.0 licensed thing you've recently had ballerina conference the first one i believe and um yeah you seem to be really trying hard to build you know not just a a programming language that happens to be open source but it appears to be you know one of those fully open source things where it embraces the ecosystem can you tell us about ballerinas community you know where it is what you're trying to build there and kind of the

the angle that you guys take at you know not just wso2 building ballerina but a group of people building it so this is kind of you know really key to us we've we've obviously put some effort you know some significant effort into to creating a language but we are very very keen that this takes on a life of its own and develops that community and and so we've you, as you say, it's all in GitHub. We've got more than a thousand stars already. And, you know, hundreds of people forking it. We've got a very active Slack channel. So we've released, you know, there's a, I would say, you know, I would say it's a small group getting into this. But there's certainly a real excitement about it that's built

since we launched this in May. And we had the Ballerina Conference, as you say, in San Francisco and online. We had, I think, 150 to 200 people come in over the web to participate and a similar number in person in the room so that was really exciting and you know as well as as well as really wanting this contribution to the core language we've we've done an awful lot of things to try and create contribution around the language as well so let me just talk about some of those so firstly one of the things we did from day one with Ballerina was to build a package management solution.

So like NPM or PIP or Maven, the package management is built right into the language and there's a website, central.ballerina.io, where you can go and... So when you build a package in Ballerina, you can just type Ballerina Push. You sign in with your GitHub or Google ID, and you get a name in Ballerina Central. And so you can start creating connectors to different systems, new libraries and so forth. And that's a really key part of our aim is to kind of create a whole environment

and an ecosystem of how do you connect to the world. So, you know, going back to how we started this, the idea that you can, that everything is becoming integration, that everything, every program anyone writes is now going to talk to remote systems. The logical conclusion of that is that there's going to be billions and trillions of endpoints. There's going to be hundreds of thousands of millions of different types of endpoints.

And the only way to solve that problem, to let you program all those endpoints easily is to crowdsource, is to have the community, is to have an ecosystem of people saying, Hey, I, I learned how to, you know, talk to, uh, Twilio from, from Ballerina. I'm going to push that in the repo and anyone else can find it and use it so that's a really big aspect another really interesting thing we've done is you know i i explained about this kubernetes and docker annotations that that build those artifacts as part of compile time those are actually done in a completely open way so anyone can actually extend the compilation using annotations and can write a plugin that will actually change

the way the compilation happens or create new artifacts out of compilation. So that's really cool. So I guess what I'm trying to say is we're trying to make it open, not just the core language, but the way that

you can extend the language, the way the language can fit into this ecosystem of APIs and cloud systems. It's really vital to us that anyone can help extend it as well. Yeah, just checking Ballerina Central as you talk here. It looks like there's connectors for Twitter, Gmail, Twilio, GitHub. A lot of these are WSO2 packages, but there's also a lot of community packages, Amazon EC2, S3. These are all third party packages. So people are definitely getting involved.

What's the best way for the listeners out there who are interested in Ballerina? Obviously, we have all the links in the show notes. So Ballerina.io, the downloads page, you can just go read about it. But specifically on the community side, like what's your call to action on a community? Is it join the Slack channel? Is it, you know, come on GitHub issues and participate there? What's the best inroad for people to get involved in Ballerina? I would say, you know, the first point is just to join the Slack channel.

You know, that's a, there's a sort of, because there's channels there for users, as well as people who want to get into the Slack channel. You know, that's a, that's a, there's a sort of, because there's channels there for users as well as people who want to get into the language design. I think the second one is that if you, you know, if you raise an issue in GitHub, you know, that's always really valuable, you know, for us to improve the product, improve the language, that's always brilliant.

And then if you're really into this language and you want to participate in the discussions around the language itself, then there's a ballerina dev group on Google Groups, which is where all that discussion happens, all the design happens in the open. The way we do it is we basically create google docs open google docs and we use them to document the specs and then we have the discussions around those google docs in the ballerina dev google group so

you know that those three things are sort of like, you know, incremental stages of involvement. So number one, sort of start chatting to us on Slack, ask questions. Number two, get involved in raising issues in GitHub. And number three is actually join the core development channel and start discussing the languages on there. Very good. All links in the notes for those interested. Paul, thanks so much for joining us. Hey, any last words before I let you go? I know it's getting late there. Anything you'd like to say as we close the show?

No, I think this has been a really useful discussion. It's been really engaging. And we certainly will be making announcements. We're going to do more ballerina days. We did the one in San Francisco. We did one in Asia that actually was so oversubscribed, we had to run a second day to repeat it. So we're going to be doing more of those. So keep an eye out on Ballerina.io. And the other thing i guess i'd

like to say is that we do run regular webinars uh from the ballerina team often with other partners talking about how we fit into build systems like code fresh into observability like jaeger and prometheus into waltz and Honeycomb, into all sorts of stuff. So come along and sign up to our mailing list and you'll get notifications of all those. Very good. Well, this has been a joy, Paul.

Thanks so much for joining us. Thank you, Jared. It's been a great pleasure. All right. Thanks for tuning in to this week's episode of The Change Log. If you enjoy this show, do us a favor. Go on iTunes and give us a rating. Tell a friend.

If you tweet, tweet a link to the show. If you make lists of your favorite podcasts, include us in it. And, of course, thank you to our sponsors, Indeed, Digital Ocean, Algolia, and GoCD. Also, thanks to Fastly, our bandwidth partner. Head to Fastly.com to learn more. And we catch our errors here at Changelog before our users do because of Rollbar. Check them out at rollbar.com slash changelog. We're hosted on Linode Cloud Servers.

Check them out at linode.com slash changelog. This show is hosted by Jared Santo. Editing is by Tim Smith. Music is by Breakmaster Cylinder. And you can find more shows just like this at changewell.com slash podcasts. And if you didn't know, we have a master feed for all of our podcasts. We ship six awesome shows each week, and you can get them all by subscribing to Master. Head to changewell.com slash master.

Get them all, and I'll see you next week