CoRecursive: Coding Stories - Tech Talk: Erlang And Distributed Systems with Steven Proctor

Episode Date: May 2, 2018

Tech Talks are in-depth technical discussions. Today's interview is with Steven Proctor, the host of the functional geekery podcast.  We talk about distributed programming in general and specifically how erlang supports distributed computing.  We also talk about things he's learned about functional programming and applying FP principles to various non FP contexts.   Contact Proctor: Functional Geekery Podcast @stevenproctor @fngeekery

Let's do the object-oriented idea, which is a good idea, and we're just going to turn it up to 11. Hey, in today's interview we talk about distributed systems and functional programming, specifically a lot about Erlang, and also we touch a little bit on my favorite topic, which is building your own mechanical keyboard. Proctor is the host of the Functional Geekery podcast and also runs an Erlang meetup. Proctor, welcome to the podcast. Thanks for having me. So I'm a longtime fan of your podcast and I have some questions for you.

So what is functional programming? What is functional programming? I'm not really sure. That's one of those things that I've heard a bunch of other people try and say their definitions. And I've heard the quote, I think it's, you ask a thousand people what functional programming is, you'll get at least a thousand and one different answers. So I think at this point, the basis for what appeals to me, at least, of functional programming is limiting side effects. And by limiting those side effects, you drive towards what people commonly think of as functional programming solutions. So it comes down to limiting, in my view, where the side effects in your system happen. So you try to limit the proportion of your code base that does

side effects to the minimum part that you can. So what's the advantage of that? So for me, that gets back down to having worked on an app at one point for 10 years.

There were a lot of innocuous decisions or decisions that seemed innocuous at the time that five, seven, 10 years later was like, oh, this is real pain. And so I started feeling these pain points. And it was trying to figure out what these things are. And some of that came from the test driven design side and making contracts and kind of having the hint of pure functions. It's easier to test if data coming in or data coming out is dependent on data coming in and getting rid of mocks. So if you have to mock it, you realize it's harder. And these ideas were kind of setting that foundation that when I saw functional programming, I was like, okay, I see where this, like this thing's going even further.

So single responsibility principle, single responsibility principle and object oriented says it has one reason to change. Well, if you take that to the extreme at a certain point, that's just an interface with one method on it. And that method is really almost a function because you don't change the data underneath it. You just change the behavior. Or you have interface segregation principle and object-oriented side, which is things that don't change together get separated out.

And so if you take that to the extreme, well, every interface is just one function. Can I push something? I have something that's pushable. I can push something onto a stack. But I can push something onto a list, or I can push something onto this. And so the interface is really pushable,

which is just a push method. Things like mappable, or I have things that are stop-poppable, or I have things that are enumerate, where I just iterate over the next, and I can just call next, and that becomes like a stream. So some of these ideas were kind of setting that thing, and I saw functional programming, and it realized that, yeah, some of these things are functions. And it kind of realized that if I want to limit these changes, I can start doing this.

And that's kind of where the appeal for functional programming came for me was all these principles that get touted in object-oriented or procedural, they're the same principles in functional programming. They're just looked at it from a different side and almost taken to the extreme of, oh, because data is one reason to change and behavior is another reason to change. So maybe your data and your behavior doesn't necessarily belong together. And so it was a lot of all these things that kind of pushed me down that track and seeing what are those things that go and what are the common principles, whether or not it's from functional or from OO.

And so there was a lot of these ideas that formed that I think functional programming, to me, is just a lot of these ideas about how do we control what goes on in our system? And by pushing to these bounds, you wind up in a more functional programming oriented mindset. So you're saying functional programming is sort of best practices from the OO

world, you know, taken to a, to a larger, taken to a more extreme stance so do you find is oo and and fp like in conflict with each other or or do they sit well together i think they sit well because there are points at which you need to actually modify stuff and i think in that case of modifying stuff, the object oriented side

can fit together. And some of this object oriented side is more of the Alan Kay's small talk vision of message passing between things. So this is where people who are familiar with Erlang say Erlang is actually both functional programming and it's object-oriented and it's probably one of the more object-oriented languages besides small talk that you actually find maybe Eiffel's up there as well but I think there's a lot of the sense of they live well together in a certain extent now again I haven't gotten into monads and free monads and a lot of these other side effects capturing concepts. So I'm not quite sure how well that fits in.

But I've heard other people, Michael Feathers, I believe, Gary Bernhardt, I believe, and a couple of other people talk about functional core, imperative shell. And some of this stuff is Alistair Coburn's hexagonal architecture. The outside of your system, you push everything that does side effects as far out of your system as you can. And you keep the pure stuff and you keep the center, you keep the core domain stuff as pure as you can. So I think there is a place where both of these fit in. Because eventually, as a lot of people tout that if you didn't have side effects, you wouldn't be

doing anything because even at the purest level, running this computation is going to eventually make your fan run hot. So there is some side effects there on our current architecture, it's controlling those and pushing them to the outer bounds of the system. So in a lot of cases, for anybody who's doing object oriented programming, people talk about checking your data at the perimeter and assuming once it gets inside your system, all the data is good. So do all your null checks at the very edge of your system. Assume once you're in your domain logic, you'd never have any nulls. If you take that idea and you just treat that with side effects as well,

it's put all the side effects at the outer boundaries of your system. And then anything that needs to happen already has its data. And then you can just do data transformations and modifications through the rest of your system, and it starts being easy. And then you build immutability on top of that

as another way of limiting your side effects and the way you interact with things and change things. Then you know that, okay, if this thing is this, it's always that. And in other languages like Erlang and Haskell and certain other languages, you only have binding. So you can only set it once in that scope too. So you know if X is Y, X is always Y for that scope and it's never going to change and all of a sudden X becomes Z.

So that's the view that I found. And sadly, I don't get to do functional programming day to day. As a lot of people would say, oh, you're working in a Haskell, you're working in an Erlang or Elm or PureScript or Scala or pick your closure or pick your functional language of your choice. A lot of this is still, okay, I'm in JavaScript, or I'm in Ruby, or I'm in Bash programming, or I'm in whatever it is that requires me to do it.

How can I essentially bring the most sanity to what I'm doing? So that's a good question. So you're saying day-to-day you're working in Bash and Ruby and JavaScript. So how do these principles influence what you do in those languages? Some of that is, like all good answers, it depends. If there's some certain areas, I try and think of data transformations first. That becomes easier in certain languages. Potentially, JavaScript might be a little bit easier to pull that off if you fold in underscore or Lodash,

or potentially you can sneak in some Ramda because people understand underscore, and you can kind of say well here's underscore like improved and evolved so pretty much everything you know about underscore here's a few things and you can use it pretty much the same but if you need to you can do this other stuff uh if you're doing ruby sometimes you can find off the place that okay well what if we copied this or what if we just took this object and it returned a new object and didn't really modify this, or at least what if we think of a transformation,

even if we are modifying it because of the constraints render, but think of this as a series of map and reduce and folds in any other operations. And we can kind of just chain these things and think of it as that pipeline operator that certain functional programming languages have, where you can say, here's the series of transformations instead of, okay, well, it's a series of a bunch of these other calls.

And when you say, to make sure I understand this, when you say a data transformation, you're saying immutability is like a functional programming principle. And so in Ruby or wherever I am, rather than changing the value of a user, my function can create a new user with that value changed. Is that what you mean by transformation? Sometimes, if I can get away with that. Sometimes in Ruby, sometimes in JavaScript or these other languages,

to do that, A becomes hard. It becomes hard A because that's not how that language is necessarily shaped. So sometimes there's things where you're like, okay, let's just take the hit, we can take this hit. Other times it's, well, you're on an existing code base, you can't all of a sudden just go make this object immutable. That's going to break too many other things. So at least with data transformation, maybe we can outline it as a series of steps that this data goes through. So this data, this thing happens, and then this thing happens, and then this thing happens. So yes, I may be modifying it in place,

which is kind of cringeworthy, but at least I've outlined the discrete steps of how this data gets transformed and thinking about it in a pipeline or composition of these things together. Even if it is mutating the state, I can at least start to ship that mindset of it's a series of steps that we're going to do. This is we're thinking in an algorithm. So let's actually think in the steps and outline those steps that the data goes through, even if it is being transformed on the same objects. Yeah, because there is a cost to immutability, especially depending on the language, right, of actually making copies of things. And it's not just that cost, it's the cost of the introduction to teammates, potentially.

If you're in a team where a bunch of people may or may not be familiar with the functional programming ideas, you start introducing immutability everywhere. That might be a harder push as opposed to just say, okay, well, let's think about how we're changing the data.

And then maybe once we get that, we can show that, well, do we need to change the data or can we create a copy? And it's that slow evolution of some of these ideas. So it's a culture war within your workplace where you have to convince them that you're not crazy, A, if people aren't bought in to begin with, just to essentially make it a sink or swim idea in something that is an existing context becomes something you want to watch out for because then you don't want to start that culture war, right? If you're taking a Ruby and saying rewrite it in Haskell, then you don't want to start that culture war, right? If you're going into, if you're taking a Ruby and saying, rewrite it in Haskell, then you're thinking in Haskell. If you're saying Ruby and let's think in terms of more functional styles in a more object-oriented language, then it may just be,

how do we start to think these things? Because even then, these are still potential object-oriented things. We think of workflows. We think of pipelines. We think of potentially state machines. Even if that state machine or that pipeline is modifying the same data, can you take these ideas and introduce them so it's not one of those things that's completely unfamiliar and you lose the battle, you lose the war before the first battle is even finished in the fact of just trying to get exposure to these things.

Right. So there's a thing, there's the idea that's more of the pragmatist that says, yes, I love functional programming. Yes. I wish I could write every code functionally. Even I'm bad at that in non-functional languages that don't keep me strict. So how can I, knowing my limitations, how can I make this easier for someone instead of making them swallow everything at the same time and slowly expose these ideas and build upon ideas little by little and exposure by exposure and saying, okay, what if we did it this way? How do we think about this? Instead of redoing and reintroducing

someone and making them feel like they don't know anything about coding or challenging and saying, you don't know anything about coding. Here's how you do this. And this is the much better way, even though what you're doing has worked for you and you've been successful with it in your career. I'm going to completely invalidate it. It's more about how do we approach these things? How do we make this more familiar instead of, and therefore easier to grasp onto because that familiarity is there instead of saying, no, now for something completely different. And I expect you to learn this and cater to my will, even if the language is not even wanting to cater to my will.

Yeah, I can understand that battle can be hard to fight. But a pragmatic approach, I think, always makes sense. You mentioned Erlang. So for anyone who's not familiar, what is Erlang? So Erlang was the solution to the software crisis that Ericsson was identifying in the 80s. They had Joe Armstrong, Robert Verding, and Mike Williams as part of this lab. Their goal was to, people are expensive, software is expensive to write. How do we do this? We need to solve this problem of making our telephone switches and doing the software for this and making it reasonable to write. Can you go off and solve this problem?

Because we need constraints of high availability because we get fined if the phone lines are down. You don't want the phone lines cutting out when you have to make a 911 call. You don't want phone lines to go down because my call with you is ruining someone else's call because our call goes bad. So they had a bunch of these constraints. And so Joe, Mike, and Robert went off and started looking at a bunch of different solutions. And essentially Erlang was a language that they evolved in what sounds like a very agile way of sitting with the people who are actually going to be writing the software and building a tool for them that solved their problems. And so Erlang became, out of necessity of the problems they were solving, it became functional because they needed to limit their side effects. It became functional because they couldn't have shared state. So if things are running on five switches, if one switch goes down, you can't share that state.

It's got to be isolated. So you kind of start being functional in that way. It became message passing because of the way that they were dealing with concurrency. And these things can happen in whatever order they can happen. And a lot of these constraints kind of led them to reinventing an active model, as they say. They didn't know they were doing an active model with message passing. They didn't realize they had functional programming experience, but a lot of that functional programming stuff became implicit because of the problem they were trying to solve. If we take down one computer, we take down one switch in the network, we can't crash the whole network. We need to be able to move things around. And so Erlang became this functional message passing

system that was designed for high throughput, high availability, had things to schedule. So it was near real-time requirements, not absolute real-time, but very near real-time. So they couldn't starve any piece of work for too long. There were all these other constraints that made them wind up with Erlang as it is. And it became a very interesting language and it made me, one of the things that I appreciated about Erlang was it made me rethink how some of these systems work. There's a lot of distributed systems stuff that gets tied into the Erlang ecosystem. Like they were doing distributed cloud programming.

I mean, it wasn't called that, but it was definitely distributed programming way before everybody else. Yeah, because all these switches were everywhere. And one of the things that Erlang did was, because it was isolated, because they had the code across these switches, if a switch crashed, they needed to be able to recover from that crash. And some of that is their way of supervisors and management of applications, of what they call applications, which are like micro apps, with supervisors and workers.

And then because of the distributed sense, and if this thing happens, we don't know which switch it's going to be on. And so they had an idea of location transparency. So you have this process identifier and that is encoded where this is running, which node this is running on. But for all intents and purposes, you as the client don't really know and can't really be involved about which node this is running, other than a few constraints about you can't manage, you can't supervise something that's on another node. But if you're just sending a message to something, you don't care if that's your local machine, you don't care if that's a different machine. So all these things that they were doing was essentially distributed to cloud in a sense

because if we're going to put this on however many switches, we don't know at any given point which switch is going to be running which piece of code. And it could be running on a bunch of different stuff. It could be running on one place and we don't know. And we, at a certain point, don't care. And so, yeah, in the case, it was essentially a managed cloud in a way. These switches were servers that were managed by this.

But when it came down to running it, which piece of code is running on which server? Well, who knows? Maybe this code detects something else went down and we start up another instance on this thing because we can't communicate that. So we've got a process that just spins up a new application here and starts managing that lifecycle here. And we can broadcast that, hey, we've got this service running. And so there was some stuff around that are very much the problems that we're hitting today with my understanding of Kubernetes and some of these Docker orchestration levels and microservices and all this other stuff

about making what's running where invisible and not really caring about it and having these small things that if they crash how do we monitor that crash how do we restart it back up okay good we got it did we start it up in the same place maybe maybe not i never thought of this kubernetes connection but yeah i guess for anyone who's not familiar like my understanding may be wrong, so just jump in there. But like the message passing actor system in Erlang is like each function has a mailbox, right? Which is basically a queue. So messages can sit in front of it and pile up and it runs within a cluster. So it could be running

on one machine or another, right? And I guess, yeah, I can see what you're saying. This is very similar to sort of a microservices architecture, except a lot more lightweight, right? Because you talk about today, like, you know, I have a Docker container with a JVM instance running in it. And that's being distributed by Kubernetes. And there's a whole bunch of other things. But in Erlang, this is just actually a function, right?

Like it's a much smaller unit of distribution. Yeah. And there's a little bit of a distinction, just to be clear, is between functions and processes. So processes run functions. So you can actually call a function without having to go across that mailbox boundary. And depending on how you define it, I know I've done early days where it's like, okay, well, everything is concurrent, so let's spawn up a whole bunch of these things and every module is now its own process.

Well, sometimes that gets you a bottleneck if you've got a hot code path. Now you're going through a mailbox when that could just be, I can invoke it as a function or I can start it up as a process. hot code path, now you're going through a mailbox when that could just be like, I can invoke it as a function, or I can start it up as a process. And that was one of the questions that I kind of was talking to with Martin Logan on one of the early podcasts is, how do you define that boundary of what's a function? What's just a function in a module? And what's a process that you spawn up to handle something? And so that's kind of the same in the microservices world of

at what point do I start up a service and invoke a service to do something versus at what point do I just call a function? And even Lambda on AWS and other function as a service things are more than just a function, right? Those are still super small services because if you need to calculate a min and max value, you're not spinning up a Lambda to invoke max to give two numbers, right?

Yeah, totally. So there are certain things you start to think about. This is a functional unit but maybe there's still functions here but when i need to do side effects and i need to go to outside stuff that's when i start to figure out where those boundaries when it gets elevated from a function to a process and and the confusion i think where i've seen people get confused on the process and function and i think where i started was to start a process, you just essentially give it a function. So if you just have one function that runs, that can be a process.

You can start up a process that just invokes this function. And if it just does something and immediately returns, that process spawns up quickly, does its work, and dies. And sometimes when you're first learning this, I know I fell into it a couple of other people I know have fallen into it, was like, process everything. And that is one of those things that you have to figure

out where does it make sense to have a process that's the equivalent of a max function or a string length function? Or do I just call that as a function with a module namespace? So how do you decide? How do you decide where to draw these lines? One of the good ideas was after I asked Martin Logan that question on it

when I was still trying to process it myself, his answer was, what are the concurrent operations? So that was one of the things that informed it he was gave an example i think it was lambda dam up in chicago a number of years ago that was talking about this and he outlined a vending machine and he said here's the parts that are concurrent like if this thing goes down what parts of the vending machine could still operate? What parts of the things are the concurrent and isolated factors? So some of that is what's concurrent, what can be run whenever and in whatever order. Some of that is back to the supervision.

The supervisor has restart strategy so it can monitor the worker processes and some of those can be other supervisors, but it can know when that process terminates. And some of that boundary is if this thing terminates, what goes along with it? What has to,

if I have to terminate and restart this, what kind of failures cascade and how do they cascade well i think one of his examples with the vending machine was if my change if my money part the part that takes in the coins takes in the coins, takes in the dollars, and dispenses change and accumulates change fails, does that die? When that dies, when that air is out, does that kill the cooling system? That keeps the drinks cold. Well, if the refrigeration of the cans and the vending machine die when the coins become inoperable or there's a jam or it's out of money and it can't take money,

there's a resiliency boundary there, so maybe those are different processes. Isn't it true that the cooling system doesn't even interact with the change system? Yeah, and so therefore, if the change system. Yeah, and so therefore, if the change system dies, why do you take down the cooling system, right? So those become two different domains in your system that become obvious process boundaries. Just because things interact doesn't mean they have to be in the same process,

I'm assuming. Correct. And some of that's the concurrent. So the first thing was, what are the concurrent pieces of the system, right? So the cooling thing can operate independently of the change. So once you start to realize that, you start to see, oh, yeah, maybe it is taking a step back. Do I really want to kill my cooling when my change goes or vice versa? Okay, there's a boundary of some processes. The other part is some of that state management and which processes

hold which parts of state. So once you get the dividing line of some of those concurrent activities and failure modes and which things are actually related to each other you say okay well do i have an individual process for each rack of drinks potentially in that vending machine maybe i have something there and if i'm empty in one, if that one spinner or release or whatever dispenser is broken for that spot in the vending machine, and that has its state of its own sodas or its own beers, whatever you're stocking in that vending machine. So if A3 breaks, does that break A2? Does that break A6? So that one can break. So, but that has its own state. It has its own level of how full that thing is. So the whole row of A may be the same drink because that's a very popular drink.

But if one of those slots in that row breaks, do I want to have everything else break, right? So if that state gets messed up, if that thing's empty, can I not dispense any other things? So there's a little bit of isolation there in the fact that it's got its own state. Its state mirrors other states. I think one of the important things about the data is that

because because each process is like single threaded um it's a good it's a good um it's a good way to prevent like concurrency issues with data right if you're like this very small process this very small instance of a whatever of a thing that's running is the only thing that can touch this state um and it has it has a message queue so so like work it needs to do can pile up but only ever one you know mutation will happen at a time yeah absolutely and in that process that process single threaded, but you extract the concurrency out. And that gets one of those things back to the object oriented if there is concurrency well the concurrency is different things being concurrent not i have three different things modifying this same state at the same time yeah the way i think about the uh like the actor you know model of concurrency is like

as an actor like as a person right it's like the the concurrency model is like you have like a guy or like a gal like a process that's sitting there and only it can can manage this data that it's in charge of so like there can be as many things in parallel telling it to make various updates but they just get put in its mailbox and it processes them one at a time. That's the way that they kind of cut that knot of concurrent data. Yeah. And that was one of the metaphors, I think, Alan Katie said, along with cells, was each cell is isolated, but if you take it out and it's an actor, yeah, I send you a message.

If I'm going to, I might send you an email and someone else might send you an email and someone else might send you an email and you're only ever going to work one of those emails at a time. We don't communicate. We're all locked in rooms. And all we have is that little drop box of an inbox and an outbox that we can communicate through.

So whatever I do, you don't know. Yeah. And I mean, that's quite functional, right? All your only inputs are this mailbox and your only outputs are also the mailbox. And it ties back to your single responsibility principle, I guess, right? Because you're like, this process is in charge of the A1 row of this vending machine and that's it. Yeah, exactly. And my responsibility is either to supervise things that are doing jobs or do a job. So what does the supervisor do? I know you've mentioned supervisor a couple of times, but what's the role? So a supervisor essentially is a process that monitors other processes.

So it has children processes, and it can either have workers or it can have other supervisors. So you can get this nice nested hierarchy of supervision. You can think of it like the company you're in, right? Hopefully, hopefully it's the same company and you're not reporting to multiple supervisors, but you've got a supervisor at the top level. So maybe you've got the CEO, CEO has its own set of supervisors. That's the CIO, the CTO, the COO, all these other whatever levels that that organization has. They have their supervisors,

but they also have people who don't supervise and do stuff. So they may have administrative assistants. They may have some other people who do research for them, gathering data, go to meetings for them, but they don't report. They may have consultants as in contractors, but people that go through and say, okay, you're my sounding board. So I'm going to bounce ideas off of you. You're a domain expert. So you'm going to bounce ideas off of you. You're a domain expert, so you've got this thing. So you might have this whole hierarchy of an organization,

and you have a hierarchy of an organization in your code. You have a web server, which handles multiple web requests. So every time you get a web request coming in, in a lot of applications, you have something that says, how many web requests can I have at a time? Okay, this web request goes off and does some work. It may have its own set of things to do. And so all these processes are isolated and you get down to the things that are actually

doing the work. And so when this thing does a work, it works, it does its job. Supervisor, you can have long-lived workers, you can have just dynamic workers. But when a supervisor detects that the worker's not working, so the supervisor thinks it's died, it's not responding to any messages, it gets a message that this process is terminated, it can take that and it can act on that and say, okay, well, this process has died.

What do I need to do? Can I restart that? How do I restart that? And how many restarts can I do in a given period of time? So Supervisor has different, they call supervision strategies and thresholds. So if this thing keeps crashing and crashing and crashing, well, maybe it's time for me to just die myself and let someone else higher handle this error.

So it's part of an error propagation strategy too. Dave, Joe, and Mike and Robert have joked about the, have you tried turning it off and on again? That's applied to processes via the supervision strategy. If you've used Word or Excel or Outlook and something's acting up, sometimes you might have multiple Word documents and this thing's acting up. Okay, well, let me close that one Word document, reopen it, restart it, reopen it, see if it's still acting up. Okay, well, let me close that one Word document, reopen it, restart it, reopen it, see if it's still acting up. If that's acting up, maybe I close out all my instances of all my Word documents that I have open because I have 10 different Word documents that I'm working on now.

Okay, well, does that work? Okay, cool. Now I'm back and I'm in a known good state. If not, okay, well, now I close every Office program because there's some shared library code in Office and maybe something in some sort of shared state higher got affected. So maybe I close all of Office down, close my PowerPoints, close Outlook, close Excel, and reopen it and restart Word. Okay, cool. Now that's working. If that didn't, well, maybe it's time to go nuclear and restart the whole computer, right? So you've got the supervision strategy, which kind of is levels of escalation of, can I handle this error?

Yes. Okay, let me try. Okay, cool. Or if I keep getting repeated failures, it's like, I can't handle this. Let's escalate it to the supervisor above me. It definitely mirrors the way that debugging works. I think that also, isn't there an Erlang phrase about to fail fast?

Or I don't know, if you don't know what you're doing, kill yourself? I'm not sure. They talk about fail fast, let it crash. Okay. So some of that is, again, back to the separation of concerns and do one thing and do one thing well.

If your code crashes, there are some things that you might know about, like, okay, that file's not there. I expect it to be there. So if I'm writing, I just touch that file and reopen it. But if there's things that like, I don't know how to handle, don't litter your stuff with the try catch and error handling clauses and if checks and all this stuff, essentially code for the happy path. And then if you, there's something you can't do, your code is not responsible.

Your process, your code is not responsible for handling that. Let's light that up. Let the parent, let the supervisor know how to try and handle these failures. Because then you isolate your error handling from your work. In a lot of cases, you get some of these languages where, or some of this code where you've got all these guard conditions and you're like, what part is the error handling and error checking and making sure that the contract's good? And what part is the actual work? And I know I've been in places where I've even wrote a code that was probably 75% error handling and 25% work. And maybe that's a good ratio for that piece of code.

But now it's like, okay, I can actually understand what this thing's trying to do. And I can look at the error handling as a separate concern that says, okay, here's the cases of when it crashes. How do I handle that? And so I'm going to let it crash and I'm going to restart from a clean state because maybe I'm assuming that something got corrupted again. The joke of have you tried turning it on and off again goes to your processes. So do you still have the 75% of the code that's error handling?

It's just it's in a different place? Or is it not needed anymore? You still have some of it. It gets moved to a different place. So then it gets condensed because now you're handling a set of classes of errors instead of every potential different permutation of it. And then the other side is maybe it just goes away in some cases because it gets reduced because you're like, I don't really care about that class of error. Let me just retry this process.

Let me turn it off and turn it back on again. I got into a bad state. I don't now i'd like at a certain point in some of those things it's like well if i've got my state and it's not necessarily a pure function because i've got an internal state let me just retry this and just restart from a clean clean known state that's well and maybe something because these processes hold their own state maybe some message came in that eventually caused it to corrupt its own state and get into a bad state so you just say

instead of me trying to figure out all those error conditions of how a state can get into a bad state we just say eh whatever like kill it and restart it with a fresh state so it can be simplifying that's good one thing um about erlang and i guess maybe about the actor model in general yeah like it doesn't seem

to work terribly well with with types like i mean erlang itself is is a dynamically typed language to my understanding and then like there's other implementations of actors. I know ACCA on the JVM, but also it tends to be untyped as well just because these messages coming in to these mailboxes can be anything, I guess. Erlang's a weird mix because it's typed. It's just got like,

I think it's got eight or 10 types that it can be. And then you just build out more complex types in a sense that everything's either a tuple, a list, or a hash kind of thing, or an atom, which is just something that refers to itself. So the types are there. The way it's more dynamically leaning, at least for Erlang, and some of this is, I don't think it's impossible to do because they have Dialyzer, which you can give type specs and document the stuff,

and it's pulled outside. Some of the catch is you could be running these systems and running these nodes, and the messages might change and evolve. So the message becomes a little bit more dynamic, because you could be dealing with different versions of the code running, because this thing got upgraded. And this, this thing got upgraded and this this node got upgraded this node didn't this process is still running version one of the api where this version is running node three of the api and it tries to send a node one to node three message so you might

have a little bit more dynamic dynamic side there with that because you don't necessarily know which code is running and there's upgrade paths that it can go through so you can do some matching and pattern matching on this stuff but my understanding was it was more about the dynamic and long-lived code because one thing we didn't touch on earlier is it's got hot code upgrades, which means you can keep your app running, and it can hold two versions of your code in memory at the same time. And once it crosses a fully qualified function call, it will upgrade to the new version of the code. And so sometimes if you say this function takes this type of A,

and now it's type of A prime, well, some of that could be due to how do you compile that when you can be running multiple versions of your code at the same time and how do you manage those types? And so I think some of those were just kind of... It's a constraint. A little bit of more... It's a constraint of...

Yeah, it's a little bit of a constraint and pragmatic solution that says, sure, I can compile all this. And again, it's like, if you think of microservices, right? I can determine a contract, but that contract is an HTTP request

or a message in a message queue or something else. If I'm going to communicate across these boundaries, I may not know who's calling me. So I just have to essentially match on these things, handle those messages I can, because it could be any one of these messages. And how do I do a type system like that

when the types can evolve and the code can evolve and I can have three different versions of my services running at the same time kind of thing and which services are going to hit so how does it work with the hot code reloading like okay if i update this this uh receiver of this of this uh queue of this message mailbox. That's the word I'm looking for. So I update it, but now it's expecting

a new field. But then the old messages are still in the mailbox. So do I explicitly handle that case? So each process has its own state and its own message. still in the mailbox. So do I explicitly handle that case? So each process has its own state and its own message. So if you get different versions of the messages in, you have to be responsible to

pattern match on those different versions of messages you get in. But you also have versions of your state. So maybe I have a user. And instead of having an email address associated to that user, and this process manages the state for a user, the state now went from a record or a map of name to string and email to string to,

and that's version one. In version two, it might be name of string and emails instead of email, which is a list of strings. So now you have to say, now you have to say this state that this process holds onto while it's running, how do I upgrade that internal state of what this user represents? And so there's a code change callback that gets invoked when it says, oh, I see you've got a new piece of code here. We're loading a new piece of code. We're going to take that state. I'm going to give you that old state. Here's the version and go from version one to version two. And then you code and say, okay, well, from version one to version two, I have email that needs to go to emails, and I put that single email in a list.

And now I have other things I can do by appending in my new code. So there's some management around that as well that allows me to essentially do my database migration on the fly because my database is just the state that the process holds and not an outside database it has its own private variable that's its internal state that is its database non-persistent held in memory that it holds onto, but how do I do migrations of that data, that state between versions of the code while keeping that code running?

Yeah, the migrations are per actor. That makes sense. I think you helped me to understand that. I have kind of a general question for you. So how do you learn? So it seems to me you've used a lot of languages in your time. You've played around with a lot of new technology. So how did you learn about Erlang and other things?

What's your learning process? I was reading a bunch of technical books, trying to find things just across a bunch of different topics. It eventually got me into picking up SICP, picking up Clojure. Clojure evolved into hearing some podcasts about.NET Rocks with Brian Hunter, talking about Erlang, hearing some other people talk about Face Messenger and Erlang and RabbitMQ and Erlang and a couple of these things that were kind of getting popular at the time. Somewhere around the 2010 area, I started putting on my radar more,

moved into a job where they had some Erlang as well, so worked to pick that up, started putting it on the user group. So the user group became one of those tools for forcing me to learn because now I have to explain it to other people. So essentially the, I don't have to outrun the bear. I just have to outrun you becomes, I have to, you ask a question. I can answer, I don't know at this meeting, but now it's something on my list to understand well enough to explain the answer back at the next meeting in the early days. So essentially that the best place to teach from is the person who just learned it. So putting myself in that route and then the podcast, I find these languages and I hear

something else and that sparks an interest. It's like, well, okay, a lot of these podcast episodes that I do become curiosity of like, okay, let me ask questions. Let me see what kind of high level concepts I can start to put together and get a very high level, but very incomplete picture and see how some of these things fit together and what are the common patterns a across languages what are the common patterns across paradigms as well as i mentioned where o seems to be a lot of ideas and functional programming seem to be just let's do the functional programming idea which is a good idea or let's do the functional programming idea, which is a good idea, or let's do the object oriented idea, which is a good idea. And we're just going to like turn it up to 11.

And then what's common between Haskell? What's common between Erlang? What's common between Elm or Clojure and the dynamic versus static languages and the strongly typed languages? And what are these things? And like, if you listen to me talking to edwin brady about interest is like i've just heard about this i still really don't

understand this help me explain and eventually i get enough people where i can start to ask more and more intelligent questions but i also go in knowing that well if, if I'm asking this, I'm sure I'm not the only one. So maybe someone else who doesn't know this can be able to piggyback off my learning. I interviewed Edwin Brady, too. And I have the interest book and I've been working my way through it. And I found him to be like super modest. Like I was kind of blown away by all the things that this language presented. And he was kind of like, oh, yeah, this is what it does. A lot of these ideas just came from other places.

And I'm just polishing them up and presenting them for programmers use. He was so modest. It was a bit surprising, I guess. I'm used to people kind of wanting to push their technology. So speaking of technology and your learnings what um what are you excited about right now with all your interviews and your um reading what technologies have uh are on your radar interest is still one i want to get deeper in for understanding dependent types. Haskell's still a bit on my radar and try and take some peeks in, try and understand some of the concepts better for

that enforced purity and say, okay, well, I know I get impure. The LISPs are interesting because of all the metaprogramming stuff and treating that code as data and macros and how do you actually design a real DSL and essentially design from the outside in and say, well, this is what I want this to be. Let me write this and then I'll figure out how to transform this either via macros into something that's actually usable.

Then you get things like Reason, which is appealing from a different perspective of how do I take this? This is OCaml made friendly to JavaScript, so it's not quite OCaml syntax, but maybe I could do OCaml and using their compiler of BuckleScript, compile OCaml down to JavaScript.

But even shorter is if this is designed by the intent that we want to make this, bring all these ideas from functional programming and OCaml and make them more easily accessible to the JavaScript community and make it more familiar instead of being looked at and say, what, like, this feels like a far stretch, because now I've got to learn a syntax that's completely different, along with completely different ideas. OCaml seems appealing and or reason seems appealing in the fact that we're taking this and will this be coming up soon enough that it might actually be reasonable that where people talk about CoffeeScript or TypeScript now, probably TypeScript more than CoffeeScript, but CoffeeScript a number of years ago,

or any of these babbles and taking some of these other JavaScript, even if it's future JavaScript and providing these functionalities of, there's reason close to being something that could be more mainstream, more introduced into a workplace environment with people who aren't necessarily the functional mindset converts, but be able to start introducing this and say, look, here's the way to do this. You've seen CoffeeScript, you've seen TypeScript, you've seen some of these other things. How much of this stuff, it's not that far of a leap, and now we can start taking advantage of some of these ideas. Elm's interesting for the React stuff.

I think JavaScript is really an interesting space now, isn't it? I mean, it's just become the assembly language for a bunch of other languages. Yeah, and again, ClojureScript is appealing on a personal level. Elm and PeerScript are appealing on a personal level. I think Reason might have a chance of being one of those nice introductions to getting more functional JavaScript without feeling like the burden of

having to learn a new syntax with Elm or Peerscript or going to full-on ClojureScript with parentheses everywhere. And again, I like Elm. I've actually pointed a couple of coworkers to Elm when we were doing a React project using Redux. I said, play with this, look through the Elm architecture. That's an interesting idea that these other, that have been fed back in, back and forth between these things.

So that's, some of that stuff is looking at what are the concepts and what's appealing. That means I can bring that back easier. So in the Elm perspective is how can I understand Elm and the Elm architecture and play with that a little bit and then bring it back. And now when we start doing React and Redux, I've got a head start because I've

understood the Elm architecture, which is what Redux was kind of cribbed off of and exchanged ideas from. So that's kind of the perspective I take in a lot of this stuff of learning is what are those principles that can come out and what can I learn from the principles that I can then apply even if I don't get to do functional programming directly in this thing. I understand those principles and practices that can help make some of the stuff I'm doing more sane than if I were to just go off and not even think about some of this stuff. I can understand that.

And it's interesting to see these concepts move around from language to language or framework to framework. Totally unrelated question. I was kind of stalking you looking at your blog, and I saw that. Are you into like custom keyboards? Is that true? Yeah. Got into mechanical keyboards a number of years ago and then have gotten in via another co-worker. I got hooked on mechanical keyboards. He reinforced the ability to do custom keyboards. Very cool. I have a Plank keyboard here, and I have my customized firmware.

I think it's pretty cool stuff. I actually have a couple commits in the QMK keyboard firmware. Yeah, my coworker got me hooked on some of this stuff. So I found Steve Walsh's a modern Space Max keyboard post a couple years ago and I started following his advice so I kind of software modded on my laptops by like

I use Vim and we'll get the holy wars going a little bit but because Vim uses the escape key and I never use the caps lock key I turn caps lock into control and escape. And then that software was Carabiner. It stopped working with High Sierra.

So based off my coworker who created essentially mini dongles, I went off that route and started doing mini USB adapters so I could actually put the firmware as essentially a key logger slash translator from what's the normal keyboard key into what I want. And it kind of just went down the rabbit hole and did a couple of adapters for a code keyboard that I had, a DOS keyboard that I had, and then found an old Apple M0110 from 1984 or 1985 that takes the phone jack connector and actually created an adapter that's a phone jack connector on one side, an RJ22, and USB on the other side.

And so I was able to... I started going down that within the past year. Actually, within the past couple months, I just finished in ErgoDocs and have been starting to love that. Kind of went down a little bit of the hardware hacking route as well.

So that's one of the other things that sound interesting as far as upcoming topics. There's some spikes about functional languages running on hardware yeah yeah the one i'm i've been interested in is because uh like the qmk firmware that my keyboard runs and i think ergo docs can run it as well uh i have an old ergo doc somewhere like uh it it appears to be mainly composed out of like c uh macros like it's it's quite uh i don't know it seems hard to understand to me so i want

to see like a rust version uh because you know rust is supposed to compile down to you know very small runtime and bc equivalent so i'd love to see like some of these more modern languages taking a stab at keyword keyboard firmware i think it'll be awesome yeah i've seen some again a closure with a macro system or some of the haskell with the types to know that yeah when you did this you didn't screw up this thing some of that stuff and being able to put that on some of these smaller arduino's or teensies or other microcontroller boards is something i'm anxiously looking forward to it well i think we went over but uh thanks so much for your time, Proctor.

It's been great talking to you. Thank you for your time. Thanks for having me.