CoRecursive: Coding Stories - Tech Talk: Throw Away the Irrelevant with John A De Goes

Episode Date: March 21, 2018

Tech Talks are in-depth technical discussions. Today's interview is with John A De Goes. We talk about performance problems with monad transformer on the jvm, various flavours of IO monads and reasoning about polymorphic type signatures.  On the lighter side of things, we discuss how to write technical articles well, flame wars and Zee vs Zed pronunciation.     Show Notes: John's Website and Twitter Descriptive Variable Names: A Code Smell Data Structures Are Antithetical to Functional Programming A Modern Architecture for FP

Hey, welcome to Code Recursive, where we share the stories and the people behind the code. I'm Adam Gordon-Bell. This is an FP interview. It's about functional programming, and it's an interview with John Degose. I think I originally stumbled upon his blog trying to answer some questions about Scala Z, and he had a lot of super interesting articles, so I invited him on. And I think it's a very entertaining interview. Turns out John is somewhat of a divisive figure in the Scala community, but I think it's a very entertaining interview. Turns out John is somewhat of a

divisive figure in the Scala community. But I think this interview is very interesting. So I hope you'll enjoy it. John A. Degose is the CTO of Slam Data. John, welcome to the show. I found you had an interesting take on functional programming, and that might be fun to kind of explore that. So what is wrong with using descriptive variable names? Right. Descriptive variable names are a code smell, I argue in one of my posts. Yeah. And the reason I say that is I just say it in that way. It's clickbaity. I say it in that way to grab people's attention. What I actually mean by that is that oftentimes when your variable names are very descriptive, that is, you can say this is like a Java bean proxy factory or something else like that.

You know exactly what it is. And when you know what it is, it necessarily implies that it's not a polymorphic type. So it's a so-called monomorphic type. The type is fixed. It's not a variable. And what that means is your code is specialized to a specific type. And there are lots of drawbacks to writing tons and tons of code with monomorphic types. And one of the principal ones is just that once you write a type signature for a function that's entirely monomorphic, the number of ways you can implement

that function and have the compiler type check it goes way, way, way up. So in polymorphic code, classic example is for all A to A, there's only one way to implement that function whereas in monomorphic code if we have something like string to string then you can imagine that there's an infinite number of ways to implement that function so in general i always encourage developers to write as polymorphic code as humanly possible get rid rid of all the types. Prevent yourself from knowing anything about what structures you're operating on. And if you do that, if you do it pervasively, and then if you use principled type classes to inject the minimum amount of structure necessary for you to actually do your task, then the number of ways you can implement these functions will go

way, way, way, way down. And hopefully, in an ideal case, you'll find there's only a few sensible ways to implement the function, and one of them is the one that you want. And then the other thing that you do is not only do you make it harder to introduce bugs by cutting down on the size of the implementation space, but you also make the code much more reusable because now it's able to solve not just one problem, perhaps, but potentially a whole family of problems with totally different types, totally different data structures. So that's why I say if your code is so monomorphic that you can give everything a richly descriptive variable name then most likely you're

missing some opportunities there for abstraction and for reuse and for polymorphism that that are gonna make your code a lot safer and cut down on the odds of defects and make it easier to maintain and easier to reuse over time so do you think is that still true if reuse doesn't happen? So I, you know, I abstract, I pull out some portion of code and because the logic in it actually doesn't refer to anything except for, you know, kind of the structure, I can make it fully polymorphic. It's just like changing the shape of something or returning it um but it's only used in one place is it still

useful to to have it polymorphic in that case i would say you can't necessarily say it's always useful because i can imagine situations in which it's not useful and just adds a boiler blade and doesn't introduce any additional safety. But I find in most cases, if you can make something polymorphic, then in a language like Haskell or PureScript or any other language with full type inference, it often does make a lot of sense to do that because even if you don't achieve reuse, even if you still use that function in one place and in one way with specific concrete data types, you're still making it harder for that function to go wrong. Because it knows less about the types, it means that first off, your initial implementation is more likely to be correct. But that's relatively trivial savings. Like you can write something once and you can make sure it's right and then it's good to go. And the cost of that is constant relative

to the size of the function. The real savings is as that function is maintained over time, if it's highly polymorphic, that means implementation space is constrained, which means all the other developers who go in there and muck with that function, they're going to have a harder time screwing it up and introducing defects over time. functional programming that has long-term effects on minimizing the cost of maintenance by making it just harder in general for different parts of the code base to go wrong. Also, I think, I don't know if this is true for everyone, but I think that for me and a lot of other people I know who do a lot of functional programming, we find that over time it becomes easier and easier to reason about polymorphic code um we gain better intuition for how we're going to satisfy the type signature what kind of things we need in order to be able to satisfy and and that's a matter of like, it's an interval or whatever it might be that you're replacing with a monoid or a group or whatever you're replacing with

it's suddenly something you you don't need to know about you don't need to keep it in your head and all the different things that you can do with that you don't even need to consider them you can't consider them because the types don't even allow you to consider them. And I find that that way of sort of using polymorphism to get rid of stuff that we don't need to keep in our brains right now actually has a tremendously simplifying effect on the way I reason about the code and construct it. So do you feel like this is a muscle? So this is a muscle that needs to be developed when you move to a you know a system like a programming language with a with a more expressive type system i do feel that because i know in the early days me looking at haskell code that was highly highly polymorphic i'm like what is going on here i have no idea and and the variables are all like a and b and C and D, and they don't mean anything.

It's sort of a way of thinking about your code that I think has to be learned. And it only has to be learned because a lot of the programming languages that we're familiar with and that we grew up coding in are monomorphic. Or at least almost monomorphic. Like Java has generics, but they're very constrained in what you can express. And many, many other programming languages don't even have that. You're stuck in C or a language without types or anything like that. Then you look at code and the variable names

and their types give you a big clue as to what's going on. And the concreteness of it makes it easier to understand. And that's a way of thinking about software that I think we learn. And after we learn, we forget the fact that we learned it. And then when we're exposed to this Haskell style in which everything is highly polymorphic in many cases, our brains are like, what?

There's not enough details for me to understand what's happening. And then eventually, after you work at it for a while, you're like, oh, there's no irrelevant details that I have to consider when I'm trying to prove to myself this is correct, because I've thrown all those details away. They don't need to clutter my brain. So it's a totally different way of looking at the problem that I think has to, at least it did for me. And it has had to be learned for many others who are like, yeah, John, you know, this type signature stuff doesn't make sense. I want concrete types here. I want to give these big, long, descriptive names. And then I tell them, stick with it and concentrate on reading code and trying to become familiar with this style. And then six months later or 12 months later, they're like, you were right. I don't want to go back to writing code the old way. So I think it is something that it's like muscle memory. memory it has to be learned you have to train yourself over time i feel like there's some there's some switch where you go from when you're calling a

function looking at the names of the parameters as as a description of it as it may very well do in in java or c to looking at the the types of the parameters as kind of explaining what it does. Yeah, that's right. Yeah, the names are so important in so much code. But then when we move over into functional programming that's highly polymorphic, the names become less important and the structure, the underlying structure, becomes much more important. The structure of the types or their lack of structure in some

cases and you mentioned in there about having the proper i forget what you said the proper type class instances or something could you explain or yeah so oftentimes we're writing a function and we would like to make it fully polymorphic. So for all A, B, C, D, E, F, G, etc. But usually, that does not permit many implementations. Usually, it doesn't allow us to solve an arbitrary problem inside our code base. So in many cases, having a type totally unbounded, unconstrained is insufficient. It's not enough structure. Our code can't truly be polymorphic in all types. We're going to require a little more structure.

And so how we reintroduce just enough structure to solve the problem, just enough details that allow us to write a problem, is we construct principled type classes. And a principled type class can take that type out there that's arbitrary and it can give it some capabilities, the minimum amount of capabilities necessary to solve a problem. And by principled, I mean that ideally the behavior of this type class is given by algebraic laws that we can reason about all possible instances of the type class by using these laws and when we have a type class that does have these algebraic laws we say that the type class is principled and that's the ideal type of

type class to add additional structure to these polymorphic types in your implementation of the function. So instead of saying we're going to work with all A's, we're just going to work with A's that have the structure of a semigroup or the structure of a monoid or the structure a ring, or one of these other algebraic structures. And we're going to be able to use the laws like associativity and so forth that these things have in order to be able to create an implementation that satisfies our requirements. So polymorphism is how we throw details out of our implementation space to constrain the space of implementations and make our code much more reusable and easier to get correct. And then type classes,

principled type classes, are how we add structure back in, just enough structure to be able to solve the problem at hand. No more, because any more structure than we actually need to solve the problem is going to end up creating, enlarging that space of implementations, creating more ways for the code to go wrong now and in the case of interfaces. Because they always used to say one of the best practices for object-oriented design

is to require the smallest interface that you possibly can to solve the problem. And then also when you're building types, provide the maximal interfaces possible to make that type reasonable in the most different circumstances. So I think that this is not new. It's expressed a bit differently in functional programming, but the overall concept of minimizing

the amount of information that you, minimizing the things that you can do with the inputs to a function is a good idea and has been recognized even in object-oriented programming for probably decades at least a couple decades so i saw a post you had something along the lines of maybe this was another clickbaity article but data structures are bad. And I feel like this ties in a little bit. Could you explain that concept? Yeah, it does tie in.

And again, I have to apologize for clickbait. Hey, no, controversy is good. You should see, I mean, I post on my blog and most of the people who read that blog are functional programmers. So they know, they get it, I mean, I post on my blog, and most of the people who read that blog are functional programmers. So they know, they get it, you know, they like it. But I posted that on LinkedIn, and you would not believe the response I got. Just dozens of comments, and most of them are like, what the heck is this guy talking about? I think there's a problem, too, where people often only read titles.

They do, and you can tell who read the article by whether or not they agree with you if they disagree and think it's crazy then they only read the title they stopped there anyway the data structures are uh antithetical to functional programming i think is the blog post you're referring to and it's a very similar idea. And the idea is as follows. A lot of the goodness in functional programming comes from deferring decisions. And an example of that is a polymorphic function defers the types until much higher in our application. And this deferring also not only constrains implementation, but also enables reuse. So deferring, and that's not the only thing we defer.

We defer effects until the end of the world. In Haskell, we defer evaluation until the last possible minute when a piece of information is required. So there's pervasive deferment. It doesn't matter where you look in functional programming. All the good things about it are deferring the destruction of information because that just makes systems a lot easier to predict and reason about. And I think one of the areas where programming languages have not done as good a job enabling us to defer decisions is data structures.

Because many times you'll look at a function and it will produce a concrete data structure, even though it doesn't necessarily need to do that. And an example is, say we have this function that returns either an error message or some HTTP response. Now, all well and good, but the reality is most likely our larger program is embedded in a more powerful monad than either. It could be IO, or it could be some free structure, or it could be a monad stack with IO at the bottom. Most likely, we're not operating at the level of the either monad for big chunks of our program. So why do we, down here in the little pieces, why do we return an either when we don't actually need to return an either? We just need to return something that has the capability to represent success and failure, but may be able to represent lots of other cases as well as success and failure, or may have more fine-grained notions of success and failure. And I think that one of the good things about functional programming that could,

in theory, be enabled by a sufficiently adept programming language is enable us to defer the decision of what actual concrete type data structure to use in a huge number of cases. It's not easy to do that. It's actually extremely difficult to do that in today's programming languages like Haskell and PeerScript and so forth. But I think you could design a programming language in which it's very easy and natural to say, I need to be able to construct something like this, or I need to be able to deconstruct it in these particular ways and maybe some other set of ways that I don't have to handle. And if we did have such a programming language like that, then we would be able to be much more polymorphic in the types of data structures that we work with,

and we would have to do less sort of mind-numbing conversion from one thing to another, from an either to however we're going to get it into the type that represents our monad stack. And I think that when expressed like that, it's maybe not a controversial idea, but it suggests that people take a hard look at when they're writing code and they're using a concrete data structure structure do they actually need to require that or can they can they give up some of those requirements and return something that's that's more polymorphic and i think the answer is in some cases you can do that you can do that through type classes and other types of things in today's programming language but it's not very easy. So let's try to put an example on that.

So I'm thinking, actually, here's an example from your article. So you have a function that takes a list of employee and then another employee, and then it returns an either, which is either an error message or that employee's manager. So what's the problem here? So most likely this function will be embedded into a context that represents failures differently than an either. So, for example, it could be the IO monad, or it could be some monad error of some error T of state T of something of something of something, and so on and so forth.

And if we want to be able to use this function and have those different types of errors lined up, then we're going to have to take the output of this function, which is neither, and we're going to have to manually convert that data structure into a data structure that's bigger, that can represent everything it can represent, but potentially lots of other things that it can't. And there's no problem with this. It's just boilerplate. And not only is it boilerplate, but in some sense, it's like premature specialization. We decided that we were going to commit to the either data type when,

in fact, all we needed was any type that had a failure and a success constructor, but it could have had a thousand other constructors as well. We honestly, in this example, we didn't care if it had more constructors. We just necessarily wanted a success and failure constructor type. And you see this kind of mind-numbing, boring plumbing a lot when you're using, well, just in any case where you have smaller data structures and you're lifting them up you're constantly lifting them up to be in

bigger data structures whether that's monads or um uh all sorts of other things like if you if you write programs using the free monad style you will encounter this a lot with just co-products and so forth just lots of mindless boilerplate lifting that in theory doesn't have to exist it it is maybe in today's programming languages it's actually probably easier just to pay the cost of that boilerplate because getting away from it is even more painful in many cases but in my opinion that's just an argument for a better programming language. It's not an argument why we should always specialize our return types, even in cases where we don't need to.

So in this specific case, I think existing programming languages can handle it, right? We're saying instead of returning an either, we return some type class like either-ish or something that encapsulates this concept? Right. You can have some either-ish type class that has a success and failure constructor. And then any type which is bigger than that can, of course, provide an instance for that. So it's actually relatively easy, I think, to do that in today's programming language. Now, is it warranted? I'm not sure. It depends on how much pain you're in. But I've seen code bases where there's a lot of lifting smaller data structures

into bigger data structures. And it's not necessary if you make your code sufficiently polymorphic. And also, as a side benefit, you actually increase performance by deferring the data structure until the point where it's final in your application, actually end up increasing performance. How does it increase performance? Because less conversions,

less manual conversions. So for example, if either is converted to an either T of something with an error inside there and something else, and then later on that's converted to IO, I mean, there's two conversions in that case, but in theory, there could be five or six conversions. I've seen code bases where there's 12 conversions from the sort of fine-grained individual function all the way up to the top level of the application and all those conversions they're allocating memory on the heap and they're moving bits around and so forth they don't necessarily need to happen so i could see a

case where maybe the performance goes the other way like i'm thinking i have some function and instead of a list it takes it takes like list like or something right it takes a type class that encapsulates these kind of list like options but then um maybe list isn't a good example i'm thinking of what i'm thinking of is a function wherein it does something like add elements onto the end of it but then the concrete instance that we pass in is actually very expensive to add elements onto the end of it. But then the concrete instance that we pass in is actually very expensive to add elements onto the end of it. Yeah, so I'm actually a big fan. I think that Haskell and some other programming languages have made some mistakes here

because they've not baked into the definition of a type class required performance characteristics. And in many cases, that proves to be fatal. You're trying to do too much abstraction. You shouldn't be abstracting over random access over type. You shouldn't be abstracting random access over types that don't efficiently permit O of 1 random lookups. And in many cases, that's being done.

And so you have type class functions or functions implemented in terms of type class functions, which have pathological performance with some data types and wonderful data performance or wonderful performance with other data types. And in my opinion, that does not make sense. It's one of the things that we need to clean up. We need to go in and say, no, this type class, we're not going to,

we're going to provide performance guarantees on these different methods. And I know testing that is something different. And obviously there are some tools out there that you can verify that something's like big O of n or big O of 1 and so forth. They're not very good, but we could push them further. We could make them as good as QuickCheck or Hedgehog or these other tools out there. We could make them really good if we put some effort into it. And then we would be able to say, okay, I'm going to use this function and I'm going to have a guarantee from the laws of the type class, if you will, that it's going to perform at such a

level based on the size of my collection. I think that's something that we absolutely have to do because too much time, I've just seen too much time be wasted tracking down performance problems that exist solely because an instance of something was created that could satisfy the laws, but only at horrendous performance. And that type of thing is not principled programming, in my opinion. So we would incorporate the performance characteristics in the type class laws somehow. Exactly.

And right now, type class laws are by and large just comments. So it's not like we can't add more comments and develop additional machinery to pull that stuff out of there and verify at runtime in your test suite. It's possible, and we should be doing it. It seems a little bit like circling back around. Like we have something, like I have something that takes in an array, which is great because I have, you know, random access.

And then I'm going to make actually a type class that's like array-like, I guess, to say this is a list type thing that I can, you know, randomly access. And aren't I kind of circling back to being a data structure again? So I don't think so, because, for example, in the case of array, you can actually use an array, but you could use a skip list or you could use a vector or you could use lots of other things. And depending on how that thing is produced, it may exist in one concrete form or another. For example, it may exist as a skip list because it was cheaper to build that than it was to build an actual array. And so by making your code polymorphic over all things which are array-like and permit all of one random access, you're actually facilitating its reuse in situations that you did not intend. Whereas if you hard code that to be an array, then suddenly someone has to actually build an array all at once through pre-allocation or whatever technique they're going to use. And that may actually not be all that

efficient. And the question is, well, why did you need to know it was an array? Why did you need more than random access? Arrays permit all kinds of things. If all you needed was random access, then ideally your function should be able to work with anything that supplies random access. then that's just going to open up so many possibilities for reuse you can't even imagine them right now as well as prevent you from doing silly things like mutating arrays and so forth that you didn't need to do in implementing that function because you locked down that interface to the tightest possible bound that you needed in order to implement the function it's kind of circling back to what you were saying earlier that like the o concept of programming to interfaces yeah yeah it is it's very similar to that so type class laws are not enforced in any way do you do you, like, should that be improved, or do you have thoughts?

Yeah, I wrote a blog post on this topic, I think, years ago, but it's still online, and basically I argue that type classes are not the best that we can do. In fact, we could probably do a whole lot better, and one of the things that we, or one of the ways we can improve them, I think, is by allowing some sort of first-class means of specifying laws inside of type class. I mean, now they're written as comments, and depending on what programming language you use and what tools you use,

those comments can sometimes be extracted and used in tests. But it's not part of the language spec. It's just more convention. And I think that there are very real benefits that could be achieved by moving that into the language itself and saying, okay, here's a first-class facility for defining laws for my type class. And there are two common types of laws that are used right now, algebraic laws, but also operational laws, and both of them are useful.

And then there's, if you want to consider it, there's performance laws. Like what are the laws of this have to be? Big O of n, big o of n squared etc and a language that provided first class support for that would would make the concept of principal type class something formal and something well first class so how could it how do you envision it checking things like is this like property-based testing or how is it enforced? Yeah, so it could be enforced.

I don't know if you could do it automatically in all cases, but it's possible that you could do it in many cases because basically if you bake this machinery into your compiler, then in theory, it can have a notion of a generator and it can have generators for all the default types and generators for the polymorphic ones expressed in terms of the quick check style machinery into the language and require things in a much reduced set of circumstances. I'm not exactly sure what that would look like, but I feel like there's a lot of stuff that you could get for free. And in fact, like if you look at a lot of the arbitrary instances out there, they're just sheer boilerplate. That's why you can automatically derive some of these things, because they're just relatively straightforward. So I think that if you were to bake laws into type classes, you would have a lot of this machinery as well baked into the compiler's notion of being able to create arbitrary instances of some type. And you would require less assistance

from the developer. And then you can run these things in the same way that we run quick check properties today. You could actually run it as part of compilation if you wanted, perhaps. Or maybe a special special mode like verify my laws actually hold yeah i imagine some dark future where my my scala compile times like it's like okay verifying this uh this it's kind of a you know not a baked in concept but approximated and has some differences from maybe the Haskell type class model I'm a huge fan of type classes in Scala. I think that, um, I think it totally changes the way you write software in Scala. And it's, I think, I think objectively superior to the object oriented style

of doing things because with the object oriented style, you need control of the inheritance hierarchy of all the types that you want to be able to support these operations. And with type classes, you don't. So you can define a type class called monoid for types you have no control over because they're not bundled into its inheritance hierarchy. And so I think for that reason, type classes are objectively superior to use of traits and object-oriented style interfaces in order to model similar behavior. And that's why I think you see even type classes in the standard library. A standard library can't avoid them.

Things like ordered and whatnot, they were invented. They were invented because, yeah, they didn't have control over all these different types, and they wanted to be able to make some of the code much more generic than would be possible if you required everyone to implement a certain trait in order to provide this functionality. So I'm a big fan of it. I do think that the standard way of encoding type classes in Scala, it does have some drawbacks.

Scala Z8 is experimenting with a different alternative way of encoding the same thing that should move some of the drawbacks at the cost of maybe a little bit more boilerplate. So we'll have to see. I think that the current default way of encoding type classes is good enough to be considered an asset and will hopefully improve with time and will eliminate some of the ambiguous implicit error messages that come when type classes have complex dependencies on other type classes. So would you use a type class even if you controlled the uh underlying object rather than a trait uh yes i would absolutely do that and there are very few circumstances in which i would not in fact i can't even think of any that

i would not all right well i can think of one but aside from that one, I can't actually see any reason at all to use a standard object-oriented style trait instead of a type class, just because it leads to, first off, cleaner will basically limit the potential for subtyping to screw with type inference. Because if you have a sub-trait that has a bunch of methods and your data types are going to extend that, then that's going to affect type inference in very subtle ways. Most of these ways you don't actually want. You want to get rid of that. You want to make things as non-object oriented as possible in order to help with the type inferencer. And if you represent that functionality as a separate type class, then it opens up the possibility of doing that.

And yeah, it may be a tad more boilerplate, but it's the sort of write one spoiler plate, you do it one time, and then it's done, and you don't have to think about it after that. So the cost is constant for a given number of data types. So it's, in my opinion, acceptable cost for a long-term, long-lived project. So you mentioned the IOMonad and Haskell.

So do you think that people should be writing Scala or other languages that don't enforce purity kind of in this Haskell style of using the IOMonad to kind of mark up things that may have side effects? So I absolutely think that they should be. Now, the reason I say that is as follows. First off, if you mix imperative style with functional style, then your brain has to switch modes. Because in one mode, you can reason about the meaning of your program using

equational reasoning, A equals B. So anytime I see an A, I know what it means by substituting the B. And that is pervasively true in a functional code base. But if you mix imperative style with functional style, then suddenly that becomes true in some places and not true in other times. And so you have to have an extremely high level of diligence and awareness about which section of the code you're in, because you have to change how you think about its correctness, because the imperative style is A, then B, then C, then D, and so forth, making all these mutable changes. And so you track it like simulating a machine in your mind and then the functional style is much more mathematical and you understand

its meaning through substitution and those totally different styles your brain has to switch modes and so i it i find it very difficult to do that um and easy to get wrong because things that hold in one case will not hold in the other. And so like the substitutions and transformations you can make when you're sort of mindlessly refactoring, you have to be super careful when you switch modes because the invariants don't apply. And so just having one uniform reasoning principle that can apply to your entire code base, you never have to switch modes. You can just stay in the functional mindset all the time. It's tremendously beneficial.

But then you get other benefits as well. You get the benefits of knowing what a function does by looking at its type signature. And it's hard to overstate how just life transforming that is. Because once you get used to it, then you stop drilling into functions. And that totally changes your life. Without that capability, without the ability to know what a function does by looking at its type signature, then you have to drill into all the functions it calls. And then after you do that, you have to drill into all the functions they call and so forth until you get to the very edges of the graph. And that takes a lot of time. And that's why it's very hard to go in

and modify someone else's code. Because in order to understand what it does, you need to sort of trace out all these branches until you get to the very end but in a functional program you can look at its type and you can have a very good understanding of what it what it can do for example if a function if a if a pure function returns a bool then you know for a fact it's not going to be doing any console input output any socket input output you can memoize the result that function like that tells you so much whereas if an imperative program a function returns a bool who knows you know you know java's inet address uh constructor actually does network calls to resolve the uh the uh domain in in a canonical

fashion it's it's preposterous like that can happen anywhere it could happen in two string or in hash code or all these sorts of weird random effects that take you tons of time to track down and debug you didn't expect them because you didn't you were lazy and actually didn't go and inspect every function that was called by the function that you called and um that's just it's a common source of bugs and it's a source of complexity. And when you have a purely functional program in front of you, you don't have to do that anymore. So you have the uniform reasoning principle that you can apply to the whole code base. You don't have to dig into things because the types tell you what these functions do. And then you end up, I think, as a result of these properties,

developing code that's much more modular and much easier to understand. And of course, much easier to test because everything is purely functional. All your functions are total, they are deterministic, they don't have side effects, and so forth. And it's just much easier to test and reason about that type of code. And then I would also say, like, you get all these benefits, but you get other benefits in terms of expressiveness. So, and the reason why you get additional expressiveness is because functional programming reifies all the things. It makes them real first-class values. So, like effects, like printing a line to the console, that becomes a value.

And so you can take that value and you can put it in data structures like other values. It basically takes programs and it turns them into, so you can reuse all the machinery you know and love from value manipulation in order to build your functional program. And that is a tremendous increase in expressiveness. And if you look at code using future versus code using a purely functional IOMO net like Scala Z8, you'll see a tremendous difference

in just in terms of how many lines of code are required to express the same idea. It goes way down in purely functional code because all the things are first class. There are no effects there, just values. So you can manipulate values and chain them together and use combinators and do all these wonderful, amazing, magical things that you simply can't do

when all your things are actually effectful and they're mutating state and interacting with the real world. So it seems to me what you're saying in a way is that the benefits of using IO everywhere where you're doing side effects is that the things that aren't in IO, you know, are pure. However, like if you have a function that returns IO int, it's hard to reason about that, right? Like all you know is when this is executed, it will return to me an integer. That's right. So it could do any possible thing

at all, which is a reason to make the code polymorphic and to use type classes that say, I need monad socket IO here. So I know it's going to be doing some sort of socket operation, or I need monad random. I can work with any M that supports monad random. And then if it returns M of int, I know that it uses, it potentially uses the random effect, but doesn't use anything else. So I think, yes, you still run into that problem, but the mere fact that it returns IO int

is still tremendously useful because that should raise a big alarm in your head saying, okay, this function could do anything at all. And every time I call it with different parameters, it might return me a different IO int. So I can't rely on it to be deterministic in that same way. So it still communicates a lot of value. And how we take it to the next step is we use type classes

to throw away all the effects that we didn't need to constrain lock that function down and communicate more accurate information to the developer who ideally doesn't have to dig into the implementation to find out that it called math.random a single time so have you have you used this approach where you kind of split io up into a whole bunch of different capabilities or effects or something? Yeah, that's right. So we use that to some degree and actually quite extensively in Slendit as codebase. And then it's the style that I prefer to use when I'm doing my own personal projects as well.

Interesting. There's a whole bunch of potential IO monads in Scala, I guess. There is Scala Z, there is a future, there is a monics task. Is this good or bad? Well, it's like JavaScript having a thousand different frameworks. The bad part is it's it's like javascript having a thousand different frameworks the bad part is it's sort of painful and um it creates lots of confusion and lots of incompatible code bases incompatible libraries and so forth but the good thing is it's parallel exploration of the landscape of possibilities. And I think as painful as it might be, standardizing prematurely on one type too soon

means you didn't explore the space of all possible solutions. And I think that that, in the end, it hurts the ecosystem. It's like standard libraries with programming languages. Once you ship something in the standard library, no matter how crappy it is, it's going to be there forever and all the libraries are going to use it, which is a reason to not have a standard library or a reason to very carefully add things into there because it's very, very hard to change it. It's very hard to improve it over time in response to new learnings. And I think the situation we have in Scala is we have lots of old monads or monad-like things,

like futures, not strictly speaking a monad, but it's kind of monad-like. It has the method names anyway. And then like those made so many mistakes, just so many terrible, terrible mistakes that have screwed people up and introduced bugs and made systems slow and so forth. But we've learned from mistakes. And we've created other data types that don't have the same set of drawbacks. And then we've learned from those.

Like, we're actually probably three generations in. Yeah, we're probably a good three generations in, you know, future, the future era. And then the task era, the ScalaZ task era, basically. And then the sort of post-task era in which we have things like Monix task and ScalaZ 8.0. And we've learned a lot in each of those generations. And even though the existence of all these data types is confusing and leads incompatible libraries and so forth, ultimately, the community is going to benefit. Because like the very latest cutting-edge generation, it's not only way faster than anything that's ever come before, but it also supports incredibly rich functionality that you just can't get access in the older data type.

So it's, and no doubt, we're not at the end of innovation yet. You know, there's probably going to be a fourth generation. There's a limit to how far you can push these things in the JVM, but there's probably going to be additional evolution, at least incremental, on top of the third generation effect monads that we have.

And that will result in, ultimately, it'll result in more people coming to functional programming for the benefits that these effect monads have, which I think is a good thing for the community in the long run, as painful as it might be in the short term so do you think that it would be good at some point to you know to not have Scala Z and cats or or to is it good to have these two competing functional standard lives or something yeah I think that's an interesting question and I think that prior to Scala Z8, there was really no reason for both of them to exist because Katz replicated and used some of the code from Scala Z7. He used the same type class encoding that used a lot of the same structures. It was a subset of Scala Z7, but still architecturally very, very similar. And there was no compelling reason to use one versus the other. But I think with the difference between cats and Scala Z8 is going to be significant enough that it's possible they end up living alongside each other for a good long while. CATS is aiming to be much smaller, much more minimalist,

and much more sort of more of the same in terms of the Scala Z7 style hierarchies and type class encoding and so forth. And it's also aiming for long-term compatibility. So CATS is at 1.0, which means that by and large, it's not going to change, which is going to make it attractive for certain class of companies who need, you know, a few minimal abstractions. They need a monad and so on and so forth. And they want to be able to stick with that for, you know, two or three or five years, maybe never change. Maybe you never have to upgrade. And Scala Z8 is quite different. It's a rethinking of everything that you can possibly imagine about

functional programming in Scala. For example, monad transformers, in my opinion, do not scale on the JVM. There's no way that you can make efficient monad transformer. It's a mistake, a huge mistake. And all these tutorials telling people to use monad transformers in Scala, they're setting companies up for failure. Because if they have a deeply nested stack, five monad transformers, I promise you that machine will be overloaded doing the simplest of things the massive heap churn and massive megamorphism and all these things

we're setting people up for for failure and and as a result i think what we have to do as we strive to innovate in libraries or library versions, as the case may be, that are free to break backwards compatibility. We have to figure out what it means to do functional programming in Scala because it doesn't look like functional programming in Haskell. And just copying more of the same stuff is just going to result in more people rejecting functional programming because they try it and then they find out this stuff doesn't actually work. It doesn't actually solve the problems in a way that the business needs them to be solved. They can't satisfy performance requirements or memory requirements or this or

that. And we need to get away from that sort of, you know, mindset that functional programming in Scala should be exactly the same as Haskell. And there should be no deviation because that's just not true. In an ideal world, that would be true. In reality, it's not. We need to figure out what FP looks like in Scala in a way that solves the problems that enterprises actually have and also gives the functional programmers the things that they need in order to reason about the software. And to date, no library, whether it's, you know, cats or scholars at seven or whatever it might be, has done that pervasively, especially in the area of effect composition. So we need, we need more innovation in my opinion. And cats is going to probably stick on its current path,

which is going to make it useful for shops that are doing light FP and don't need a whole lot. And then I believe that I see Scouts at eight sort of as being the choice for the shops doing pure FP, who also don't want to compromise on performance or any of the other things that are important to business yeah i feel like you know for any given uh problem like there's a certain correct like having one implementation is probably not enough and having four is probably too many like i'm thinking in terms of an open source project right If you have four, you're distributing the people who are working on things too thinly. And if there's only one, experimentation is sort of lost. But yet Haskell, all the type classopedia stuff

is baked right into the standard library, and they seem to do just fine. So I don't know what that says. Yeah, well, every once in a while, they go through these, you know, flamers, like the monofoldable proposal, and like, should tuple be a functor? And like, they go through all these different things.

And they do have to occasionally break backwards compatibility. And of course, their motto is avoid success at all costs. So maybe that makes sense. But yeah, there's still a lot of legacy cruft in Haskell for that reason, because there was a standard library, you know, a more or less official standard library. And as a result, like in Scala, we have better type class hierarchies than Haskell, more clean. clean and like with uh the scholars at eight i o monad has a vastly cleaner semantic for error handling and interruption than the haskell model so like even though people like to trash talk including myself from time to time we actually do have the capability to innovate faster. And partially it's because there is no default thing. It's just, it's wide open and there's nothing that's standard. So people are free to innovate it at their own pace.

And that results in many more improvements over smaller timescales than we see in Haskell. I have a small question about pronunciation. It's funny because, so I'm Canadian. And so I would say the last letter of the alphabet Z, but I figured, you know, Scala Z because it was, you know, predominantly American, but I hear you saying Z as well. Yeah, you know, I've gone back and forth on it. I think I originally said Scala Z, but then I was hanging around with too many people at conferences who were saying Scala Z. So I ended up adopting Scala Z. I don't think it matters a whole lot. You mentioned that, you know, mana transformers don't scale. How so?

Well, so you have the following problems. First off, monad transformer is basically a stack of data structures. It's a data structure inside a data structure inside a data structure inside a data structure, inside a data structure, inside your outer effect monad, IO, or task, or whatever it might be. And every time you do any operation, for example, the sequencing operation via flat map, you have to construct this entire chain stack of nested data structures. And so you put tremendous pressure on the heap, not just from the data structures and all the nesting, but also from all the things that are allocated on the heap that you didn't even know. For example, when most people write a

function in Scala, they don't know that actually transforms into object allocation and so forth. And then, so not only do you have all this heat pressure from all these closures and all these data structures that are nested inside one another, but you have a problem that's probably just as bad in many ways, and that's megamorphism. Megamorphism is where the JVM goes to call a method, and it doesn't know what concrete data type is associated with that method. It's going through the interface. So because it doesn't know, it ends up running a lot more slowly because it has to do a virtual dispatch. And the problem with monad transformers is that every layer of your monad stack adds an additional level of indirection. So not only did you have one layer of megamorphism up there at the top, which is sort of inevitable, or it's not inevitable. If you're using a concrete type, then you don't even have to pay that. For example, if you call IO.flatmap, then the JVM is going to know which flat map you're talking about.

It's the one on IO because you have that concrete data type. But on the other hand, if it doesn't have that piece of information, then you're going to end up in your implementation. When you call flat map at your monad, that's going to end up delegating to other flat maps, which will delegate to other flat maps and so forth. And monad transformers are polymorphic in the type of monad that they wrap. So what that means is you have megamorphic calls to other megamorphic calls, other megamorphic calls, and so forth for as deep as your monad transformer stack is. And so if you think using an IOMonad or TaskMonad or future can be slow try using a a stack of five monad transformers on top of io and then we'll talk about slow and and not just slow by the way but massive heap churn so just massive pressure on the garbage collector just increased latencies you're going to see spikes You're just increased latencies. You're going to see spikes. Performance is just terrible. And you can actually get away with this, surprisingly, for some applications. Like if you have some sort of business process that only has to do 10 requests per second or

something like that, you're going to be fine. You can use these deeply nested Monad Transformer stacks, which has convinced some people who just do these type of applications that this is a good idea, but it's not a good idea because anytime you try to do a typical business application where you need to handle hundreds or thousands of requests per second, then you're not going to be able to achieve those performance numbers using your deeply nested stack of monad transformers. It's just everything is going to be terrible latency, response rates, everything is going to be terrible about that memory utilization. And this gives functional programming a bad reputation, deservedly,

because we've blindly copied things from Haskell and expected it to work on the JVM, and that's never going to happen. You have to change what it means to do functional programming. You can still do purely functional programming in Scala, and you can make it performant, but you cannot do it the same way that you do it in Haskell. And if you try, you're just setting companies up for failure. So I should just be like double flat mapping everywhere or however deep my stack is?

What's the solution? Well, the solution is no stack. No stack at all. Ideally, your program is using type classes to represent the effects that it requires. And for example, instead of using error T, you should use monad error.

Instead of using state T, you should use monad state and so forth. And all these type classes, they don't say anything about what the data type is that has an instance. So what you're free to do is when it comes time to supply a concrete data type at the top of your program, you're free to supply any monad that has the required instances. So what you can do is you can create a cost-free new type, you know, extending AnyVal for like my IO or whatever it ends up being that supplies

instances for all these different things, monad error, monad state, and so forth. And all it's going to do is express all these operations in terms of the IOMO. So you end up going from, you know, potentially a monad transformer stacked with 10 levels to something that is merely one layer of indirection away from my IO, which is secretly underneath the covers and to the JVM. It's in fact the IO monad, which means it goes from 10 levels of megamorphism and nested data structures to one monomorphic level where there's no additional nesting. So you get the performance of raw IO, but yet you were able to use monad error and monad state and all the other things required by your application. And this just dramatically transforms your performance. It makes functional programming actually practical on the JVM. And it handles, I'd say, 80% of the problems that monad transformers are used for.

And to handle the remaining 20 you need localized effects you need the ability to eliminate effects locally inside your application and you you actually can't use this technique for that at least not directly but there are there are ways that you can work around that issue as well so it seemed i thought makes a lot of sense, assuming I have a single stack, but if I, which often you do, I guess, but if I have a different combinations of transformers all over the place, I need like a concrete instance of each one, right? Yeah, for every, if you want to do a strict translation of monad transformer program into one without it then uh yeah what you need to do is for every concrete fully realized stack

that you had you need to create a new type for that which is a wrapper around io with its own custom instances for all the different type classes that were previously supported by that monad transformer stack. So many applications just use one. I mean, there are generally when you see different monad transformer stacks being used, it's because they need localized effects. They need to add a state T to something and then run it and get rid of the state T and then expose the underlying monad. And like I said before, you use a slightly different technique to handle those cases, but it is possible. And I've never seen any application that actually needed a monad transformer stack. In some cases, you'll need the ability to do localized effects

so you can escape from them. But I don't think there are any applications that actually do need a Monad transformer stack. Now I'm worried about my performance. I'm going to have to do some checks. Well, like I said, for some business process type applications, 10 requests per second, the memory requirements don't even matter. And performance is not an issue. And it's good enough.

But for a lot of applications out there, it's just Monad Transformers just totally destroys any hope at achieving performance that is even within two orders of magnitude of the imperative equivalent that we're talking just dramatic reduction in performance one thing i wanted to uh just ask you about is is your writing style i think that you're a very good communicator so you have good good headings that kind kind of maybe are a little bit controversial. And then you dive into details with code. What's your secret of communicating about technology effectively? The most important thing when you're trying to teach someone something is the ability to see the world from their perspective. So you have to know your audience, but then you have to be able to understand what they know.

And the other thing that I think, I mean, obviously you can structure things, you can work on the structure of things. So make sure there's a logical progression, make sure that there's flow. I think flow actually is one of the most underrated aspects of a good presentation or a good blog post. There should be some sort of arc and you should be moving people along that arc. But then beyond that, I think you can do

a little to make things interesting and relevant because when you write something, you're asking for people's time and you don't want to waste their time. You want to give them a reason to stick around. So you want to make things as interesting as possible. And sometimes that's a clickbait article, clickbait article title, just to make people think a little bit like state something in a different way to make them think, you know, that sounded like it was wrong. But after I think about it, that actually is kind of true.

Like give them little puzzles that can have a reward because their reward for spending their time reading your stuff is hopefully going to be some realization that gives them a bigger picture or that teaches them some concept they didn't understand or that cultivates a skill in them, like reading a polymorphic type signature, whatever it is. And that's the payoff when they have that aha moment, that's the payoff for them. That's the thing that keeps them coming back for more, hopefully. That was the episode. I hope you liked it. You know what?

And if you did like it, can you do me a huge favor and just tell somebody else to try out the show? Maybe you could message somebody and say, hey, I think you would like Co-Recursive. It's a good podcast. Until next time. Thank you so much for listening.