The Changelog: Software Development, Open Source - Rails as a day job, Diesel on the side (Interview)

Episode Date: November 4, 2017

Sean Griffin joins the show to talk about doing Rails full-time, his love of Rust. and his project Diesel - a safe, extensible ORM and query builder for Rust. We discuss Sean’s path to working full-...time on Rails, what he works on specifically, why Rust, why Diesel, and how much of Diesel’s design and featureset is a product of his experience with ActiveRecord and Rails.

Transcript
Discussion (0)
Starting point is 00:00:00 Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by Linode, our cloud server of choice. Everything we do here at Changelog is hosted on Linode servers. Pick a plan, pick a distro, and pick a location. And in seconds, deploy your virtual server,
Starting point is 00:00:25 drill-worthy hardware, SSD cloud storage, 40 gigabit network, Intel E5 processors, simple, easy control panel, nine data centers, three regions, anywhere in the world they've got you covered. Head to lyndo.com slash changelog and get $20 in hosting credit. Welcome back, everyone. This is the Changelog, a podcast featuring the hackers, leaders, and innovators of open source. I'm Adam Stachowiak, editor-in-chief of Changelog. On today's show, we're talking with Sean Griffin about doing Rails full-time, his love of Rust, and his project, Diesel,
Starting point is 00:01:13 a safe, extensible ORM and query builder for Rust. We talked about Sean's path of working on Rails full-time, what he works on specifically, why Rust, why Diesel, and how much of Diesel's design and feature set is a product of his experience with ActiveRecord and Rails. All right, I'll jump right into it. Added helper types for inner join and left outer join. Diesel debug query has been added as a replacement for debug SQL. Is this your changelog?
Starting point is 00:01:41 Added support. Yeah, is this not where I come on and dramatically read Diesel's changelog? That was pretty good. That was good, yeah. Keep going. I'm way unprepared if that's not what this is about. I was entertained for a split second. Literally the changelog. Love it.
Starting point is 00:01:58 Sorry, I had to get one dad joke in early on. I came up with that one like two days ago and I've been waiting for it. I like it. We should leave that in. I don't know why. This is up with that one like two days ago, and I've been waiting for it. I like it. We should leave that in it. I don't know why. This is tape, man.
Starting point is 00:02:08 Let's go. All right, let's go. Well, now that we know what kind of guy you are, Sean, you'll fit right in here on the show. But we have two subject lines. I believe they're intertwined, as most things are, and that's the subject of you full-time on Rails and your work there on ActiveRecord. And then secondly, what you're up to with Diesel, which is an ORM and query builder for Rust and the interrelation between those two things.
Starting point is 00:02:36 But let's start off learning a little bit more about you and your path to what we often call living the dream, which is working on open source full-time. And in a way that seems like a pretty awesome way. So tell us about your work with Rails, how you got involved, and how you ended up full-time at Shopify. Sure. So let's see. So I got started with Rails, gosh, like four years ago now, I think. I had done a commit here or there, but nothing particularly major.
Starting point is 00:03:07 But I'd been on some projects for a while that all sort of had these weird, similar needs in terms of modifying attribute accessors on ActiveRecord objects. And then at, I think it was RubyConf like four or five years ago, Ernie Miller gave a talk
Starting point is 00:03:24 where he was talking about how Rails is missing a lot of APIs that are kind of one level down in abstraction. And so all this culminated in me wanting to build out what is now today known as the Attributes API. And it just so happened that Aaron Patterson was, I lived in Denver at the time, and he was in town for a local meetup to give a talk. So I went out with him for drinks afterwards and was like, Hey, I have this idea for this API.
Starting point is 00:03:51 I got really, it sounds polite, but I was actually really drunk and kind of shouting at him about this API. Whatever works. But anyway, so he was like, yeah, sure. Whatever. Do it. I think, I think he was partially just trying to get me to stop asking him about this idea that I had, since the majority of people who do that are never going to follow through. Anyway, that resulted in a pull request that was way too large to be reviewed, and then a series of smaller pull requests over time. And ultimately, I vastly underestimated the amount of work that implementing that was going to require.
Starting point is 00:04:28 And when it was all said and done, I had rewritten a significant chunk of active record for like non association and non ARL related things. Um, at which I was like, well, crap, I guess I'm maintaining this code for the rest of my life.
Starting point is 00:04:44 Um, so that I merged then. Oh yeah. well, crap. I guess I'm maintaining this code for the rest of my life. So I assume that got merged then. Oh yeah, yeah, definitely. What was that like to feel like you're maintaining it for the rest of your life? Are you being facetious or are you being serious? I mean, both. I didn't really go, well, I'm maintaining this for the rest of my life. But yeah, it ultimately was. Part of why I ended up sticking around was just because
Starting point is 00:05:05 I felt a responsibility to be around to fix the bugs in this code that I had submitted and kind of make sure that it continued to evolve in the way I had envisioned. So just real quick for clarity for those who are not in the Ruby on Rails scene,
Starting point is 00:05:24 ActiveRecord is the ORM or the database library that ships with Ruby on Rails and is a significant portion of the Rails code base and probably a significant portion of the complexity and a lot of the bugs. And so rewriting any small subset of ActiveRecord itself is not a small undertaking. And probably, Sean, you know better than anybody at this point, as maintainer of ActiveRecord, while thousands, if not tens or hundreds of thousands of people use ActiveRecord,
Starting point is 00:05:55 there's probably a group of people that you could count on one or two hands that are intimately familiar with the way that it works. Is that fair? Yeah, I think so. And to give some actual numbers, I don't know what it is in the code base today, but around Rails 4.2,
Starting point is 00:06:11 so not including action cable and not including active storage. Active Record accounted for about 80% of our code base. And back then, I would say it was about 80% of our code base. And back then, I would say it was about 80% of our issues as well. Certainly, I would guess the percentage of the code base has gone down just because we've added two additional libraries.
Starting point is 00:06:36 But probably still similar, probably like 70%, I would guess. Although interestingly, I've been noticing that it is starting to become a smaller percentage of our opened issues, which is good. Yeah, you can probably see that as a win since you've been dedicating a lot of your time there. So you open up this massive PR, this idea that you were challenging Aaron Patterson, who was then very involved with Rails. I'm not sure of his current status. In fact, he was full-time on Rails, I believe working for AT&T Interactive
Starting point is 00:07:08 or somebody was paying him full-time to work on Rails back then. He now works at GitHub and I'm assuming is still very much involved. And so he said, go ahead and do it. You went out and accomplished this goal and now a lot of the ActiveRecord code base is your own and so you began maintaining
Starting point is 00:07:26 it um that doesn't get us to full-time at shopify so tell us like the personal slash business end of that same timeline and how you ended up being able to do this as your job uh yeah so i mean i guess the first thing is what is when i decided that this is what i wanted to do uh because it's funny you call it living the dream and to a certain extent i kind of call I guess the first thing is when I decided that this is what I wanted to do. Because it's funny, you call it living the dream, and to a certain extent, I kind of call it living the nightmare. That's the great irony, isn't it? Yeah. But basically, I'm bad at work-life balance.
Starting point is 00:08:05 And I started to realize that if I didn't do this as my full-time job, I had two full-time jobs. So I set out to switch doing it full-time simply because I'm bad at convincing myself to not work. And if I didn't do it during work hours, I was just going to be doing the same amount of work on my own time. So it was purely a move for my own sanity. I sort of really started to get a taste
Starting point is 00:08:26 for actually spending a significant amount of my time doing it. I was working at Thoughtbot as a consultant at the time, and I had some significant downtime between projects, and so that was when I really started to get to try to start spending more of my work hours doing it. And I'd been looking for ways to convince management that it was a worthwhile thing for me to be able to continue doing uh ultimately just the interest didn't didn't align there um so i ended up leaving and i ended up at shopify because uh basically i went to conferences and
Starting point is 00:09:00 begged a lot of people to give me money and Shopify ultimately was a company that saw the value proposition. When you say full-time open source, it's a convenient way of summing up a job with the emphasis on the focus of your job is working on open source. It by no means means, it does not mean that I
Starting point is 00:09:18 spend 100% of my time working on open source. Open source is more than just code. It's a lot more than that. It's like documentation, community, even just research and development, I'm sure, is not just the things you do. Yeah.
Starting point is 00:09:36 How long was this time frame of you petitioning the community, so to speak, with what you wanted to do and how you wanted to do full-time. How long was that span of time? About six months. Not too bad. And you were full-time at ThoughtBot during that time? Yeah.
Starting point is 00:09:54 Let me just do something unprecedented here on the changelog. I'm going to read, Sean, with your permission, I'm going to read Sean's LinkedIn bio. Can we do that? He was reading a changelog to us. Oh man, I haven't touched this in a long time. Go ahead. I'm assuming you haven't changed it for a while
Starting point is 00:10:11 because it sounds like it's still in your petition phase. Oh no, I had that there just to get recruiters to stop spamming me. Oh boy. I enjoyed this. I knew that Sean was at Shopify but I wanted to see where he was previously
Starting point is 00:10:25 Thoughtbot makes a lot of sense because that's probably where you started doing the bike shit and so I saw this on LinkedIn and I thought it was funny he said I'm currently only interested in opportunities that will allow me to focus primarily on open source contributions the best way I can contribute to the community
Starting point is 00:10:41 is with my work on ActiveRecord that includes your company. If you agree, drop me a line. My dream stack is the one where recruiters leave me the hell alone. Please read the previous paragraph. If you think that, quote, focusing on open source contributions and, quote, working at a company that uses open source are the same, then go away. It's interesting there because you're very clearly stating at that time, and probably to this day,
Starting point is 00:11:11 that you were interested and you were set on opportunities that allow you to do open source contributions full-time, or primarily is what you said, which is probably more fair than full-time. Then of course the call to get recruiters to leave you alone is hilarious because I've been on LinkedIn for many years, and the only people who've ever contacted me on LinkedIn
Starting point is 00:11:33 is recruiters. I haven't had a legitimate request ever. So that was kind of funny, and perhaps written back when you were ready to go from a consultant at ThoughtBot to maybe on the path that you're down today. So tell us, at Shopify, you got the job. Can you tell us what your official position is? And you said it's more than just doing open source stuff.
Starting point is 00:12:00 What all is involved in your role at Shopify? Well, my official position is 10x Hacker Ninja Guru. Oh, good. And if you're wondering what that means, it means that I once thought it'd be funny to see what happened if I put that on a government form related to my work permit. And it turns out what happens is that I cannot legally work in Canada with any job title other than 10x Hacker Ninja Guru.
Starting point is 00:12:20 Oh my gosh. That is so true. Oh, wow. So, I mean, my job... So one of the things that... Right now I'm the only person who focuses on Rails as their full-time job. And one of the biggest challenges that we face as a project
Starting point is 00:12:40 is just issues and pull requests come in at a greater volume than we can generally process them. So I try to keep at the very least one of the baseline things I do as part of my job is day-to-day, or I guess really week-to-week, open issue count doesn't go up and open pull request count does not go up. So probably the majority of my day
Starting point is 00:13:01 is just doing issue triage more than anything else and reviewing pull requests and merging or closing them. the majority of my day is just doing issue triage more than anything else. And reviewing pull requests and merging or closing them. Does it feel like you're making significant when you say that the majority of your time is issue triage and when I think about different ways that people work on open source and
Starting point is 00:13:18 the ways that I think companies should invest in open source, I think the way that Shopify is going about it is necessary in terms of pushing the thing forward because certain projects, certain tasks and needs are so intellectually or not stimulating, but stressful or require, they require so much thought that you couldn't possibly one week, one day a week or one week a month or two hours at
Starting point is 00:13:47 the end of your day make significant process, like for instance, on a rewrite of ActiveRecord or a portion of it. And I think we need people who are thinking about these things all the time, because that's how you move software forward is you have to have the entire system in your head. And I like to think, hey, at least Sean Griffin has ActiveRecord in his head and is moving it forward. And then when I hear you say, most of my time I'm triaging issues and hoping they don't, I'm not working, it's not that I'm working them down. It's that I just don't want them to get any bigger. And I wonder, wow, is, are we moving forward or are we just maintaining the status quo? I mean, a little bit of both.
Starting point is 00:14:25 I think that moving forward also has a different meaning for ActiveRecord today than it used to. If you guys go to a RubyConf or RailsConf, unless the tone is very different than it was a few years ago, there'll be a lot of people like, oh my gosh, is Rails
Starting point is 00:14:41 dying? Is Ruby dying? I think the notion of that in general is just sort of silly, but I think a lot of where that comes from is a misinterpretation of the signals that come from just we as a community are no longer at the point where we're going to be hip and breaking new ground.
Starting point is 00:15:02 Ultimately, Rails has shifted to a mature, stable framework. So to me, it's actually, it's interesting because I've been finding it harder and harder as time goes on
Starting point is 00:15:13 to convince people why they should upgrade to new Rails versions because we just don't have as many killer features as we used to. But there is a lot of work that goes into each release
Starting point is 00:15:23 and I think a lot of the things that people are missing... At Shopify, I end up pairing with some people quite a bit on trickier test failures that they run into as part of upgrading Rails versions. And
Starting point is 00:15:37 one of the ones that I've seen a lot recently was I changed how dirty behaves in after-save callbacks in ActiveRecord. So basically, if you ask if an attribute has changed inside behaves in after-save callbacks in ActiveRecord. So basically, if you ask if an attribute has changed inside of an after-save callback in 5.2, that's going to behave the same as if you had that code, rather than in a callback,
Starting point is 00:15:56 if you actually just put that code directly after the call to save, so it behaves as if it was after save. In 5.1 and earlier, the way it works is we do the persistence, we run the after save callbacks, and then we clear the dirty flags. And so in 5.2, the way that's going to work
Starting point is 00:16:15 is we do the persistence, then we clear the dirty flags, and then we run the callbacks. So there's a bunch of new methods that are also more clear on what question you're trying to ask. So the new methods are all named, do you want to know if there is a change
Starting point is 00:16:30 that has yet to be saved, or do you want to know if there is a change that was just persisted? That's nice. And all of the places that I've run into in Shopify that I've inadvertently broken things are places where there's a bunch of code that is in an after save callback
Starting point is 00:16:47 mutating something and then expecting changed attributes to include both the things that were just saved and the thing that has yet to be saved. And that actually just doesn't work in 5.1. It'll only, that change would only be reflected in the do we have an unsaved change method. Anyway, so every time that I've gone in with somebody to look at how to fix this test failure,
Starting point is 00:17:09 ultimately, we've realized that the code was really, really funky to work around this quirkiness. And once we fixed the test failure, we actually were able to just delete a bunch of this hacky code that was working around a Quark inactive record. And I think that's a thing that a lot of people don't realize going from version to version is, for me, a successful Rails version is when you can upgrade and delete a bunch of your hacks. That's definitely progress.
Starting point is 00:17:37 Especially when you talk about moving from a state of franticness or from lack of clarity or conciseness to a state of like a good thought through API. And when you have that many people using different versions, you know, just moving a thing from step one to step 1.5 perhaps requires a lot of work. Such as you say, clarifying those specific, even just the method names in 5.1 versus previously.
Starting point is 00:18:10 And so a lot of work goes into it. On the outside, when you're looking at the new feature, it has less of the wow factor, less of the big wins that we were getting, perhaps in the first 10 years years when there was obviously things missing and huge gains to be made. Certainly, I think probably active storage is the thing that we've added since Rails 5 is the first time we've had something
Starting point is 00:18:38 I think everybody would agree was just a thing that almost every application needs and was obviously missing from Rails. Tell us about Active Storage. This is the cloud storage adapter type stuff? Yeah, it serves a similar purpose to refile or why am I spacing on the name of the other one? That's really popular. I'm still using Paperclip.
Starting point is 00:19:01 Does that make me old school? It does, yeah. Carrier Wave is the other one I was thinking of. Carrier wave, yep. Paperclip is a thing that definitely, like, it used to be carrier wave and paperclip, but I don't think anybody's really using paperclip anymore. Well, you found him.
Starting point is 00:19:19 You know, on that note, though, to pull, maybe this is a wrench to some degree in the conversation, but you said, and maybe it's even something you can't speak about, but you mentioned how you left ThoughtBot because management didn't align, which I'm not going to assume exactly what you said. But it was something about your desire to do this full time. But then like Jared, you use Paperclip. That's a ThoughtBot project, right? Right. full-time but then like jared you use paperclip that's a thoughtbot project right right and
Starting point is 00:19:45 there's so much open source out there around rails that's almost default for many rails developers that's from thoughtbot so you would assume almost that thoughtbot and management of thoughtbot would see eye to eye with your future sean like what you were trying to do and that's in hindsight seeing this and like this conversation just reminded me how prolific ThoughtBot is to Rails and open source. And I don't know. I kind of have a question mark on that. I mean, I don't want to undersell
Starting point is 00:20:15 their dedication to open source at all. Right, of course. But there's a big difference between a developer who is billing versus a developer and works on open source one day a week because Thought billing versus a developer and works on open source one day a week because Thoughtbot just does not bill on Fridays. So there's a big difference between that
Starting point is 00:20:32 and somebody who is not billing at all. And you need to be somebody that was doing this consistently day-to-day, full-time. Yeah. And that just wasn't possible. Well, yeah, exactly. I think, so it's a big question of just, I don't want to be working somewhere and that just wasn't possible. Yeah, exactly. It's a big question of just,
Starting point is 00:20:52 I don't want to be working somewhere because it's charity. I think that for the right company, I very much do earn my paycheck and bring sufficient value to the company to justify my job's existence. So I say our goals didn't align. I mean that it just was not a situation there where I was going to be able to justify my paycheck there.
Starting point is 00:21:12 I shouldn't. Sorry, I should have asked that because I was like, hearing paperclip here in this conversation reminded me of that. They do such great work and that's why I just had that question mark. Yeah. It was probably a couple years ago now, we were on the giant robots podcast and that and back when
Starting point is 00:21:29 ben was hosting it and i even said to him that day that like my career has very much been just following the thought bot and what they do and using their open source tools and even today like our new stuff is not on ruby on rails Our CMS is on Elixir and Phoenix, and yet we're still using some ThoughtPot libraries that happen to be Elixir-based. So couldn't speak more highly of the open-source stuff that they've done through the years. Absolutely.
Starting point is 00:21:57 Yeah. This episode is brought to you by GoCD. GoCD is an open source continuous delivery server built by ThoughtWorks. It provides continuous delivery out of the box with its built-in pipelines, advanced traceability, and value stream visualization. With GoCD, you can easily model, orchestrate, and visualize complex workflows from end to end. It supports modern infrastructure with Elastic On-Demand Agents and Cloud deployments. And their plugin ecosystem ensures GoCD will work well in your unique environment. To learn more about GoCD, visit gocd.org slash changelog. It's open source and free to use, and there's also professional support and enterprise add-ons available from ThoughtWorks. Once again, gocd.org slash changelog. back to active storage now because i've've only read it in brief, but give us, I think this is probably a big deal in terms of like, this is a flagship type of feature.
Starting point is 00:23:30 Tell us a little bit more about it, when it's coming, that kind of stuff. So it's coming to Rails 5.2, which as far as I know, we haven't actually announced a date yet. But it's currently on Rails Master. And it's like I was saying, it serves a similar purpose to Carrier Wave and Refile. And I'm honestly not intimately familiar enough with either of those libraries or Active Storage to tell you specifically how it differs, other than generally, as with most things
Starting point is 00:23:57 when they come into Rails, API that kind of better suits what we see as the Rails way. I think the one thing that it does much more up front than any other solutions thus far is it has a big focus on making direct upload for a service like S3 much easier and more integrated, which is a big deal if you are expecting large file uploads and you're hosting on something like Heroku
Starting point is 00:24:20 where if your user takes more than 30 seconds to upload that file, you're screwed. Or actually, if the combined time for them to upload to your server and also for you to upload to S3 or otherwise transmit to your database or wherever else you're going to store it because you can't store it on the file system of the server. The second part of that transaction is usually pretty fast
Starting point is 00:24:44 because Heroku's on AWS infrastructure. So as long as you're on AWS also, it's fast, but yeah, point taken. Anyway, direct upload is a thing that you end up needing. I used a gem for it in the past, actually, with Paperclip, I think. That worked fine, but now it's built in, out of the box. Yeah, just looking at it from the maintenance point of view, it has an interesting structure just for all the files themselves actually do live on a single table, which is, I don't know if,
Starting point is 00:25:16 I've never looked at refile. I know CarrierWave, as far as I know, doesn't structure things that way. I know Paperclip did it as columns on your table of the model it was attaching itself to. So that's kind of interesting in that you can always just add attachments to a model without needing a database migration. Like I said, I really can't speak to
Starting point is 00:25:35 specifically how it differs from the other libraries out there too much. One last topic on the Rails and full-time aspect before I move on to talking about your new shiny, which is diesel. I was recently on your guys' show on The Bike Shed. If you all don't listen to Sean and Derek Pryor on The Bike Shed podcast, check that out. We'll link up that episode in the show notes where we talk about funding, sustainability, so on and so forth, platforms.
Starting point is 00:26:04 If you're not sick of us talking about that, go listen to that episode. If you are sick of it, well, bear with us a little bit because we've got to get this figured out. One of the things I said on that episode was that I would love for there to be more Sean Griffins out there and more Shopify specifically. Shopify is a company that I think because of its leadership and because of its roots in programming and Rails, I think gets it better than most in terms of the value of having a full-time staffer working on the infrastructure that they rely upon. But first of all, do you think there's a future for more people in roles like yours in corporate? I'm going to say corporate America,
Starting point is 00:26:47 but corporate world? And then secondly, if so, how do we go about, like, I don't know, getting more John Griffins out there? Oh boy, that's a loaded question. Yes, I think there's room for more people with similar roles. I think that for it to be successful, it does need to be a job that is worth having regardless of the PR aspect of it. So it needs to be structured in such a way
Starting point is 00:27:14 that it brings value to the company. There's a couple of things that are always going to exist that bring value for having a full-timer. Number one is I sort of try to avoid specifically doing something in Rails because Shopify needs it. That said, I tend
Starting point is 00:27:34 to fix the problems that I'm most exposed to and I'm exposed to Shopify's problems more than other people's problems, so things tend to get priority that way. And certainly when I do just know something that, regardless of whether of Shopify or not. And certainly when I do just notice something that regardless of whether of Shopify or not that just when I'm pairing with somebody like oh that's bad we should
Starting point is 00:27:49 fix that there's a certain there's a certain benefit to just when you notice something during that sort of work to be able to just go and commit it. And then the other one is just having the resource available of a person who can answer questions that not many other people can
Starting point is 00:28:09 and can generally act as sort of a multiplier on the rest of the team. So you act as a liaison essentially to answering questions in and around Rails to Shopify and the developers there. Yeah, and sometimes it's more beginner questions that other people could have answered, and that's fine. There's also benefits to just having a flexible enough schedule
Starting point is 00:28:34 to be able to spend as much time as people need answering questions. But every now and again, there will be a time where I'll be helping somebody debug something, and we'll spend two or three hours on it, and we'll figure it out. And it'll be the sort of thing where they, it would have taken them a day or two otherwise. And those are the days that I feel really good about, about,
Starting point is 00:28:50 about my job. So that's the part that feels good about giving value back to the company that's, you know, obviously employing you, not just the success you want to have as being full-time open source on rails. Those are kind of two different things to some degree?
Starting point is 00:29:06 Yeah, I mean... Because it sounds like you wanted to be employed somewhere to be full-time doing what you want to do, but also give the value back. So it seems like it's two-headed in some capacity. Right. Like I said, full-time open source does not mean 100% of my time is spent on open source. It's just a nice way to bundle up
Starting point is 00:29:26 and the focus of my job is open source. I think one of the benefits that just comes from it is if somebody does just need somebody to pair with them on a hard problem, whether it's Rails-related or not, I tend to be less bound to deadlines because I don't work on the product. So I'm much more available than most people are for whatever impromptu, for answering questions,
Starting point is 00:29:48 for pairing with somebody on whatever, doing code review. Shopify could probably use a few dozen people like that with the number of developers we have, but there's certainly a lot of value in just having people who are available, senior developers who are available just to help in general,
Starting point is 00:30:07 who the answer's never going to be, I would love to, but I really can't because I have to finish this feature. I've been trying to think of a metaphor that works well when discussing maybe the roles of developers inside of a company. One that gets to a scale where it's beyond three people or a small team, where there's multiple teams of developers doing different things.
Starting point is 00:30:30 And what makes sense in terms of business language that people can use with regards to roles like yours? And I think one metaphor I've been thinking about is in terms of kind of back office, front office type of employees. I'm not sure how business-centered y'all are, but with regard to the people who are out in sales, marketing, HR, HRB, back office, certain customer-facing jobs versus accounting, controllers, those types of roles
Starting point is 00:31:03 where the people up front are the ones, quote unquote, making the money, right? And the people in the back are the ones, quote unquote, saving the money or managing the money. And with developers, I think at a certain point, when so much of your infrastructure is outside of your control, in the case of software as a service, and you get to a certain size, and you have hundreds of thousands, if not case of software as a service, and you get to a certain size, and you have hundreds of thousands, if not millions of lines of code that are maintained
Starting point is 00:31:29 by people that don't work for you. I think having kind of your front office developers, these are the people who are working on new features, new products, so on and so forth. And then having kind of a set of infrastructure people, which is really nice. Like you said, when somebody does need help or needs a pair, you're generally more available because you don't have that deadline. I wonder if that's a metaphor that resonates with you guys or if I'm just barking up the wrong tree.
Starting point is 00:31:58 No, I think it makes a lot of sense. To a certain extent, the reason I've tried to structure this position the way that it is for me is just because at the end of the day, if I'm working on the product, I'm not going to be that much more effective at building a Rails app than any other developer is.
Starting point is 00:32:16 The majority of building a Rails app is boring, which is a good thing. I don't mean boring like I don't want to be doing it, but boring in that just about any developer is going to be able to do an equally good job and take about the same amount of time.
Starting point is 00:32:34 And I do think that there is a unique knowledge set that comes with just maintaining the framework that I want to be able to apply as much as I can. If you had to, what would be a way to template the kind of role you have to others? Like if we can copy and paste what you do there, like is there a job description? How do you outline the type of role?
Starting point is 00:33:02 I mean, obviously you covered the things you just did here now. How do you take that to LinkedIn, GitHub? Obviously GitHub does probably a lot, but Stripe does a lot. How do you take that to the non-deep tech companies and say, here's how you embed somebody? Like Jared said, you have so much leaning on lines of code you don't even control that aren't employed by you. how do you copy and paste what you do and apply it elsewhere? So that's the thing is I don't know that that's the way to grow it. I think that for the people who want to do it full, full time,
Starting point is 00:33:34 it is going to be a very unique job dependent on both the person doing it and what they want out of the role and also what makes sense for the company that's hiring them. I think really the way that we grow more people in open source is we do just have more people who aren't necessarily full-time on it
Starting point is 00:33:52 but are spending some portion of their time. So there are issue triage, for example, is the sort of thing that can generally much more easily be done on a day-a-week scale. Smaller features are something that can be done on a day-a-week scale. Smaller features are something that can be done on a week-a-month scale.
Starting point is 00:34:08 Something that Shopify has been doing recently, which I think is really cool, is we have people within the company submit proposals to the Rails team of stuff that they would like to work on, and then we pick some people, and they come join the Rails team for a quarter and just do open source full-time for one quarter.
Starting point is 00:34:27 It's not everybody in the company does it that way. It's just whoever gets picked. But ultimately, that is the sort of thing people talk about, 20% time, and I would wager 25% time is not a huge jump after that. I think if more companies offered 20-25% time for open source and also gave people the flexibility to spend that time in whatever chunks make sense for the thing
Starting point is 00:34:51 that they're trying to accomplish. If they're just wanting to get their feet wet in open source, Fridays makes a lot of sense. If they're maintaining a project a week, a month might make more sense. If they really want to dig into building a big feature, there are things that you just can't do that take more than a week that you
Starting point is 00:35:07 just cannot effectively do unless you're able to really go heads down, spend your time thinking, spend full time thinking about it. And so allowing people even to go a quarter a year. I think if more companies offered that sort of thing, that would be the better way to grow it
Starting point is 00:35:23 than more positions like mine. Okay, one last question on this and then we'll cut ourselves off. Cause honestly, I think Adam and I could prod you about this particular thing all afternoon. Yeah. But the last thing I'll say is, you know, I joke, I said that you were quote unquote living the dream. You jokingly said, you know, kind of more like living the nightmare, but in all honesty, are you happy with what you've achieved in terms of your role and the ability to do it is it does it play out uh overall good or overall bad another loaded question come on no that's not loaded that's fair no i mean uh i would i i i'm very satisfied with how things have turned out and i I think I'm happier doing what I am right now
Starting point is 00:36:05 than I would be doing something else. Well, that's the safest way to answer that. That's my very political way of saying no, but I'm going to be more miserable doing anything else. Right. No, I mean, so that was the other half of kind of why I went into doing this full time. I found myself getting less and less able to motivate,
Starting point is 00:36:25 get motivated by business problems, which would be a whole rabbit hole. So we don't have to go into that, but that was discussed recently on go time. We mentioned that in the pre-call jokingly, Hey, it's go time, but we have a show called go time and it was discussed on their recent show.
Starting point is 00:36:41 I, I, uh, I think it might've been chase Adams where, or somebody else i can't recall the person but if we do i'll link it up in the show notes that they were just saying that they they uh they enjoyed building building developer tools more serving the developer the developer was their customer rather than the product that the company was delivering you know
Starting point is 00:37:02 so like they felt comfortable in that arena much more than they did, as you had said, which was dealing with business problems. Yeah. Well, then that's why open source as a full-time job can kind of suck because how many developers give every single one of their customers a direct hotline to them? So Sean, you write a Ruby ORM as your day job and for some reason you decided to build a Rust ORM
Starting point is 00:37:27 as what I'll just call your side gig. Tell us about Diesel and why you decided to dive into the world of ORMs once again, this time in Rust. So just to preface one thing, because I've been noticing a lot of people tune out Diesel immediately with that label. So I'd like to free face it with, Diesel is very much more of a query builder first
Starting point is 00:37:48 and an ORM second. That's funny because in our notes I actually have a specific question. Is it really an ORM and do we care to bike shed the acronym for a while? Because we can. And I'm sure people do. It is an ORM in that it does provide
Starting point is 00:38:03 some functionality to map rows in your database table to objects in your system. Okay. But more about the Query Builder. Yes, but it's more about the Query Builder than anything else, because I think that's the more interesting part. Why do people tune out when they hear ORM,
Starting point is 00:38:20 do you think? I think that when people hear ORM, they think they have PTSD flashbacks to the most painful parts of ActiveRecord or Hibernate or Django ORM. And they think about callbacks and over coupling business logic to persistence logic and all that good stuff.
Starting point is 00:38:44 Fair enough. Anyway, so you had asked what made me want to jump back into it. So Rust was a language that, I want to jump into it because Rust, basically. And Rust was a language that I was sort of, before I ever actually tried writing a line of it, was aware of and interested in, just because I was very into various functional programming languages, mostly Scala and Haskell. And I found Rust interesting
Starting point is 00:39:12 because everybody's trying to solve the problem of shared mutable state, and most functional languages are going about that by removing mutability, and I found Rust, if nothing else, novel because it went about fixing it by removing mutability and I found Rust, if nothing else, novel because it went about fixing it by removing sharing. And so it
Starting point is 00:39:29 piqued my interest if for no other reason than because it was doing something novel. So I had inadvertently, towards the end of my time at ThoughtBot, I had inadvertently become the 3D rendering engine guy.
Starting point is 00:39:48 So I was on a client project. Basically, I wrote a 3D rendering engine, and it turns out when you do one of those, there's a lot of 3D rendering engine contracts that come out of the weeds. And why I ended up having to write one is a story that'll take me 10 minutes to tell. But basically, out of necessity, I ended up having to write one is a story that will take me 10 minutes to tell. But basically, out of necessity, I ended up having to write one for a different project
Starting point is 00:40:07 at sort of the beginnings of WebGL being supported by browsers. And then the projects kept coming. Anyway, so we run this project that was C++. And it was for mobile. And because of the polycount requirements, we couldn't take the overhead of a framework like Unity. So it was kind of raw OpenGL engine. And Rust had just gone 1.0 at this point in time, and so for fun on nights and weekends,
Starting point is 00:40:39 I sort of ported it to Rust just to get a feel for the language. And got sold on it, if nothing else, as a replacement for C++ because I had a segfault that I was having a bear of a time tracking down. And this was like me just one-to-one, as close as I could, porting the C++ code over. And this was not modern C++. This was a guy with a decent understanding of C trying to write C++. So, you know, not using templates in any meaningful way. This was a guy with a decent understanding of C trying to write C++.
Starting point is 00:41:09 So, you know, not using templates in any meaningful way. Certainly not using smart pointers the way I should have been. So it was generally pretty easy to kind of directly port to Rust, but then I finally figured out where the segfault was coming from because it wouldn't compile. And so I never shipped the Rust port, but just the act of porting it fixed a bug in my code. So I sold on it for a replacement for C++
Starting point is 00:41:29 immediately there, but I don't generally write C++ because I'm very bad at it. So that wasn't terribly interesting. But I did realize how amazing its type system was as part of that project. And so Rust type system is basically, to a certain extent,
Starting point is 00:41:48 it is what if you took Haskell's type system, removed higher-kinded types, but then in exchange gave certain other types of genericism in type class instances, and you end up with a Rust type system. And so that was really cool. And so then I started wondering, hmm, I wonder if Rust could work as a high-level language. So Diesel was me trying to answer that question originally.
Starting point is 00:42:16 That's an interesting concept. Well, first of all, maybe you should write a book, because two ways that you described Rust to me just now were a bit more tangible. And we've had shows on Rust. Steve Klabnick's taught us about Rust. You did Cats and it never sticks very well with me. That was the episode that actually originally got me interested in Rust. Oh, very cool.
Starting point is 00:42:37 So it's the circle of life. Nothing we love here more than that. That's awesome. That's right. So random aside on Haskell. So RustConf was this weekend. Nothing we love here more than that. That's awesome. That's right. Random aside on Haskell, RustConf was this weekend. I was there giving a talk.
Starting point is 00:42:55 The talk was about Rust's type system. It was called Type System Tricks for the Real World. The actual goal of the talk was to explain the concept of monomorphization in a way that was accessible to people who were new to both programming and new to Rust. I'm not going to recap the whole talk. I don't know how long editing is, so it may not
Starting point is 00:43:14 even be online when this goes up. If it's up, there'll be a link in the show notes, but if not the video of it will be up in a few weeks. there's this case of infinitely sized types. That's kind of an error that you really only run into if you're trying to implement a singly linked list in Rust, where the naive way you'd write it is you have a struct or an enum of some type where one of the fields is the same type. And in a language like Haskell, that's sort of implicitly behind a pointer. Everything's
Starting point is 00:43:43 heap allocated, so the size is known and that's fine. And that's how you solve the problem in Rust as well. But in Rust, if you kind of naively just write a struct where one of the fields is the same type as the struct itself, then that has an infinite size because the size of a struct is the sum of the size of all of its fields. And so the example that everybody uses for this is singly linked lists because that's really the only time you ever practically run into this. But in every example I've ever seen,
Starting point is 00:44:10 you always have it be like, it's a list of bytes specifically, because you don't want to deal with generics explaining this problem. And I've always kind of hated that because number one, not everybody knows or cares what a singly linked list is and number two nobody's ever going to be implementing a list of bytes and i had this revelation i was working on this talk if i make if i make it uh specifically a list of type car i can call it list string and number one it becomes more tangible to people who don't know what singly linked lists are and number number two, I get to make fun of Haskell a lot during this talk. And there were like
Starting point is 00:44:49 three people in the audience who did Haskell as their full-time job, and they thought it was really funny. That's when you're really talking to a niche. First, you're at a RustConf, and at RustConf, you get three people in an audience that understand your joke.
Starting point is 00:45:05 Well, I mean, no. So, funny other story, which I'll tell in just a second. But no, I mean, I think a lot of people got the joke because, like, just representing string as a singly linked list of characters is kind of a bad idea in general. And most people seem to just, at the very least, laugh. But there were the three people, just because I think everybody who does Haskell will agree that that choice has caused so much pain to the Haskell community and so they laughed much more because of that so it wasn't the three people got it
Starting point is 00:45:35 the three people were hit by it basically my very first conference talk I ever gave my first slide was a joke that was really only funny if you knew Haskell. And it was at a Ruby conference, and I way overestimated the number of people in Ruby who know Haskell.
Starting point is 00:45:54 Jessica Kerr found it very funny. I know that because she was the only person in the entire room who laughed. You should put that on a t-shirt or something. Jessica Kerr thinks I'm funny, you know? She thought that one joke was funny, if nothing else. This episode is brought to you by TopTow.
Starting point is 00:46:39 TopTow is the best place to work as a freelancer or hire the top 3% of freelance talent out there for developers, designers, and finance experts. In this segment, I talk with Josh Chapman, a freelance finance consultant at TopTal, about the work he does and how TopTal helps him legitimize being a freelancer. Take a listen. Yeah, in my arena within TopTal, I specialize in everything from market research to business plan creation to pitch decks to financial modeling, valuation. And then that leads very naturally into fundraising strategy, capital raising strategy, investor outreach, closing a deal, deal negotiation, how to value the company, how to negotiate that.
Starting point is 00:47:22 And all those skill sets that I have continued to hone over on the TopTal side are ones that I actually deploy every single day in my own company. Freelancing can sometimes be seen as not legitimate or subpar work. Now, I would argue that when you work with a company like TopTal, they put so much vetting into not only the companies that you work with,
Starting point is 00:47:44 but also the talent that you work with, but also the talent that you work with, which I'm on the talent side, that it adds a level of legitimacy that isn't seen across other platforms. And that for me, as the talent side, is incredibly fruitful and awesome to be a part of, right? I enjoy the clients. I enjoy the other talent that I get to talk to. I enjoy the TopTal team. And that creates an overall positive experience, not only for TopTal, but for me as the talent and for the client as the company on the other side. And that is really not seen or is the experience across other platforms in the freelance market. So if you're looking to freelance or you're looking to gain access to a
Starting point is 00:48:22 network of top industry experts in development, design, or finance, head to toptal.com, that's T-O-P-T-A-L.com, and tell them Adam from The Change Law sent you. For those wanting a more personal introduction, email me, adam at changelog.com. you had mentioned that diesel was an attempt to uh i'm paraphrasing your words now was basically present rust as a higher level language or not It was to figure out if it could even be that. Yes. And I think that and I think maybe some other efforts around web frameworks and such things are starting
Starting point is 00:49:14 to cast Rust more in a general purpose light. Whereas when it was first presented to the community and it's been around for a little while now, but we've always said this is a systems language. Sure. And I remember you saying on the bike chat at some point
Starting point is 00:49:30 that you think that has perhaps done a disservice to the language. I was just curious if you could expand on that. Yeah, so I think the problem with just referring to it as a systems language is the things that systems languages can do and the things that Ruby can do. This isn't like a Venn diagram where, you know, or even a, it's not like two disjoint sets, right? So a systems language, being a systems language does not prevent the language from doing anything
Starting point is 00:49:58 that a high-level language does. It's just that historically languages that were able to do the sort of things that systems languages are able to do, which is generally just be able to control memory allocation, have been painful to use for higher level tasks. C++ developers may disagree with me here, but that's fine. Like Go, for example, was originally presented to the world as a systems programming language. Yes. And they eventually reframed it as a general purpose programming language because ultimately, even though technically general purpose is less general, like if you're going to classify things,
Starting point is 00:50:32 general purpose is less general purpose than systems because the term general purpose programming, Ruby's a general purpose programming language. Nobody is going to claim that you can write an operating system in Ruby. I mean, you can, it just won't be a very good one or a very fast one at least. That's one of the
Starting point is 00:50:48 reasons I appreciated how Swift was presented to the world because it was very clearly from the start it's supposed to be everything from a single line script
Starting point is 00:50:58 that you execute just in time to build an operating system with it. And so it had this huge ambition, but they clearly stated it from the very beginning, at least from the release date.
Starting point is 00:51:10 Whereas with Go and Rust, they've kind of been trying to not really figure out what it is, but realize that maybe perhaps it's been cast in a light that people tend to put it into a corner and say, oh, you're just for that. And so they don't think of it as a tool they can grab. I think that the Rust team overestimated Rust's appeal to C++ developers
Starting point is 00:51:28 and underestimated its appeal to the broader audience. It's interesting because you're coming to it as a Rubyist who had C++, some skills, but you liked it because of it. I do not have C++ skills, let's be clear. Well, I said some skills, because you were building a thing with C++, so there's some skills right there. I think that's telling, right?
Starting point is 00:51:53 That the only reason I decided to actually take a look at it was as a replacement for C++. When I was aware of it before that, I never realized what it could do outside of, oh yeah, you would use it for everything you would use C for. Right. So you set out to see if it was good at such things as being an ORM, and so you started building Diesel, and that was a while ago. Tell us where you got with that, where Diesel's at in its life,
Starting point is 00:52:23 and just open up that a little bit for us yeah so uh i shipped diesel thanksgiving day 2015 because i was trying to avoid family um family's listening oh boy i i hope not uh I doubt it. So, Diesel's coming up on its second birthday. Right now, I guess where it's at in its lifespan is we are looking to ship 1.0, and right now our target date is November 23rd, which will be Diesel's second birthday. That by no means is going to be Diesel's feature complete,
Starting point is 00:53:01 but it is a commitment to stability of the API. And basically, all of the features that I expect to be able to implement in the near future, and by the near future, I would call that within two to three years, that I also expect to require breaking changes are done. So the features that have been on the 1.0 milestone are not the sort of things that are super pressing compared to some of the other things, but are the things I expect to require breaking changes.
Starting point is 00:53:34 One of those is refactoring most of the stuff that's actually gone now, but refactoring our error types to be a slightly better structure. It's one of those things that's actually been on the issue tracker since June. It's a low-priority thing.
Starting point is 00:53:50 It's one of those, yeah, there's a few tweaks we want to make here, but there's no pressing need for it. It's not like our current error handling API is super bad, but it's one of those that is going to be a breaking change, so we need to do it before 1.0 or commit to not doing it for a while. Ironically, as I've been going through this milestone, actually I've been finding more and more things. I'm like, you know what?
Starting point is 00:54:13 Actually, I think I can do this backwards compatibly. So I'm going to take this off the milestone. You mentioned that you started this Thanksgiving 2015. That's when I released 0.1. Okay. I started on I released 0.1. Okay. I started on about six months before that. And you mentioned that you got into this as a desire to learn more about Rust, right? So was this your learning thing for Rust, or is that not where you started?
Starting point is 00:54:36 No, no. Certainly my learning thing for Rust, at least for the very basics of it, was that 3D rendering project. Certainly I learned a lot more about rust as part of writing diesel um but i i definitely had a reasonable understanding of the language when i started the project so were you scratching an itch when you did it or what was why why is it even existing i mean it was very much so. It exists because there was not a good ORM at the time. I was originally going to... My original plan was actually to go full-on web framework,
Starting point is 00:55:13 which, you know, if I had infinite time, would have happened. And ORM was the first logical part of that. But it was very much like I just wanted to explore Rust as a high-level language. I didn't know when I started if Diesel was going to ever be a library that was good enough to ship. But I wanted to see if Rust could work as a high-level language,
Starting point is 00:55:34 if it was a language I wanted to spend more time with. And so to me, high-level means web, and pretty much every web application out there needs to interact with a database, and I wanted a library that made it easier to interact with a database. So it wasn't so much to scratch and itch, per se, because I didn't have an existing application in Rust that was missing a good database library.
Starting point is 00:55:54 That said, good open source libraries are not built in a vacuum, so I very quickly realized I was going to need an application. So for me, that application was crates.io, which was the, it's Russ's version of rubychems.org, which is a surprise, it does some surprisingly complex interactions with the database. I would assume that that's relatively straightforward. You would think,
Starting point is 00:56:18 but they're doing some interesting things in the database that a lot of people would have otherwise done in their native language. For example, one of the things that Decel supports as a result of crates.io is the ability to use arbitrary user-defined SQL functions in the query builder.
Starting point is 00:56:40 And that was because crates.io has a function called canon crate name, which is just where they canonicalize the name replacing underscores and hyphens and changing the casing, and they do that in the database. That was just one of those, like, okay, so, you know, we would have just,
Starting point is 00:56:57 you would have made sure that you always called .downcase and, you know,.gsub in your scope in Rails. I don't actually know that there's a huge benefit to doing it in the database
Starting point is 00:57:09 because you also just have to remember to call this function. But it was, that was how they were doing it. And so Diesel supports arbitrary SQL functions. And that turned out to be, I mean, number one,
Starting point is 00:57:17 just because it's a good, like, it's a good thing to support. But it turned out to be a really good move because there's a bunch of functions that have non-trivial signatures to write when you actually try and figure out the type signatures of it. Like just lower.
Starting point is 00:57:34 Actually, I think we have lower in Diesel. I'm trying to think. There's a really common one where like depending on the type that you pass. Coalesce is probably a good one because Coalesce has no meaningful type signature that you could write in most languages. Certainly, Rust does not have variadic length functions. So figuring out how to write Coalesce would have been a pain in the ass. But because we support arbitrary user-defined SQL functions
Starting point is 00:57:58 and it's really easy to do so, you just define Coalesce with the actual signature you need for that one case, where are you going to call it? And we were able to just punt that issue entirely. Yeah, it saves a bunch of code. And yeah, it's certainly, I mean, and saving code is a benefit, for sure. It's one of those things, like, if I had a good idea in my head
Starting point is 00:58:14 of how to reasonably write coalesce in a way that wasn't painful for people to use, I would do it, but I don't, so that's okay, because you can still do coalesce, and it's fine. Right, it saves you from coming up with a painful solution, right? Yeah.
Starting point is 00:58:29 Whereas people can all just do it the way that they need to. So I imagine, Adam asked, why does it exist? And you said it wasn't really to scratch an itch, but you wanted to try out the novelty of Rust. You've already tried Rust. You wanted to provide a thing in the Rust ecosystem that didn't exist, which is one of the ways
Starting point is 00:58:43 that languages get adopted, is people have tools that can solve their problems, especially if you're trying to build web stuff and there's no web stuff or database stuff available. It's sometimes a non-starter for folks. I was wondering if a lot of diesels, maybe not the genesis of it, but maybe the design of diesel and the feature set are a product of your experience working with ActiveRecord and the history there and how much of it is a product of Rust's design.
Starting point is 00:59:18 How much of it is because of where you've been with ActiveRecord and so the things that you've seen that you don't want to do, or the good ideas that you do want to do, and then how much of it falls out from Rust's properties as a platform. Yeah, so it's a combination of both. So I have joked that Diesel is my apology for ActiveRecord, even though, of course, I'm not responsible for a lot of ActiveRecord's design. I do see the effects of its design quite a bit.
Starting point is 00:59:47 So certainly one of the big things I wanted to avoid are things that are common sources of issues on the Rails code base. And there's two kinds of issues that come in, right? It's either a person has an actual bug, and there are features in Rails that are just more bug-prone than others, generally because they're more complex. Or it's because a lot of people misunderstand a feature. And so we closed those issues, but I still noticed them getting open. So I was very much not trying to write active record for Rust. It was very much I wanted to build an ORM that was true to Rust.
Starting point is 01:00:21 And so I spent a good bit of time thinking about what that meant. And to me, that means safety. So Rust is, you know, touts that it's a memory-safe language, but it's also a very type-safe language. So the defining vision behind Diesel was
Starting point is 01:00:40 Diesel should disallow an incorrect SQL query at compile time. Now, what I thought that was going to be originally and what it turned out to be are two completely different things. I originally envisioned this being like you just had some fragments of SQL as strings, and we provided an API to kind of stitch those together, and then we connected to your database at compile time
Starting point is 01:01:01 and asked the database, like, hey, is this a valid query? What we ended up with does none of that. The smallest little inkling of that that came through is that we have an optional feature where you can, we have a bunch of data structures that we generate that represent your database schema, and we can generate that for you at compile time if you give us a database URL.
Starting point is 01:01:24 That's the only part of that design that actually survived. Because it was bad design or bad idea or there was a better way of doing it? Didn't need it? Just didn't need it.
Starting point is 01:01:38 Turned out that Rust-type system is flexible enough I can represent a SQL query in a way that is very close to one-to-one with the underlying SQL in pure Rust and give more meaningful error messages that way. The main reason you want a query builder to begin with is that
Starting point is 01:01:56 SQL strings are not very composable. And they're hard to reuse. So you'll notice that a lot of Diesel's APIs will look very similar to Rails. For example, if you want to pass a select clause, it's going to be.select.
Starting point is 01:02:11 Instead of passing us some symbols, basically, if you're just passing a list of columns, it'll look exactly like it does in Rails, but there'll be an extra set of parentheses because you're passing us a tuple. And then you'll remove all of the colons, because each column on your table is an
Starting point is 01:02:27 actual type. And so you're passing us just those structs. But then the other big difference, though, is that then when you want to select a clause that's more complex than just a list of columns, you're going to just do whatever you want with the query builder, because you were never passing
Starting point is 01:02:43 a list of columns, you were passing us a arbitrary SQL expression that is valid for the get from clause. So if you want to call lower on one of those columns you just lower and then that same column but that's all still Rust code. We just support a much much
Starting point is 01:03:00 wider range of what is possible in SQL. We don't support everything that's possible in SQL. I don't think we ever will support everything that's possible in SQL, but I'd like to get to like 95% of at least ANSI. I definitely don't want Diesel to be chasing every backend specific feature till the end of time. That is something that third party libraries can do.
Starting point is 01:03:23 And from the get go, I've tried to design this in a way that there is a solid foundation of APIs for people to add additional plugins. Crates.io does full-text search stuff, so sort of as my canary, I maintain a full-text search crate
Starting point is 01:03:37 for Diesel that supports for just the various operators and types that are required for Postgres full-text search. So the point being, I guess the goal was to have it be as productive as ActiveRecord is, but catch errors at compile time, within reason. There are certain things
Starting point is 01:03:53 that we just don't catch. One of the biggest ones being we don't know what your check constraints are. So certainly inserts are probably the least checked part of diesel and updates, because we don't know enough about your database schema to actually fully verify are all of the invariants represented
Starting point is 01:04:10 in Rust type system and I don't really care to because that's getting into the really really high cost low gain type of things it's heavy lifting to get that done and then the other things are like we sort of assume that all four given type in SQL,
Starting point is 01:04:30 the equivalent type in Rust, all values can be mapped between the two, which is not always true. The two examples that really come to mind are like, Postgres, even though it claims it accepts any UTF-8 string, will not allow strings that contain null characters. And there actually is a type in Rust called Cstring that very specifically represents a string
Starting point is 01:04:50 that does not contain null characters, but we just take a normal Rust string because it was too painful to disallow that. And then like Chrono, the commonly used datetime library, I don't remember what the mapping is, but basically either Postgres supports dates that are earlier or Chrono accepts dates that are earlier, but then whichever one isn't that allows dates that are later.
Starting point is 01:05:15 So there's a subset of dates that are accepted between the two, but it's something like anywhere from 20,000 BC to 23,000 AD. So not something I'm worried about. But anyway, so we want to prevent runtime errors within reason. So then the other half of your question was, how much of it came
Starting point is 01:05:36 from Rust versus how much of it came from Rails or ActiveRecord? And so there's a lot of things that are kind of lessons learned from ActiveRecord. Certainly there's no semb lot of things that are kind of lessons learned from ActiveRecord. Certainly, there's no semblance of dirty or validations or callbacks. Things like timestamps are handled purely at the database.
Starting point is 01:05:52 We provide database-level helpers that you can call on your migrations. CreatedAt doesn't need anything other than a default in the database. And then UpdatedAt is handled by a trigger. And because I can never remember the syntax for creating a trigger, there's a diesel manage updated at function
Starting point is 01:06:08 which takes the name of your table as a string in your migration to the SQL function that will create the trigger and set up the trigger for you. And so as a result of that, that means all of our functions around inserts and updates use the returning keyword so that way we actually reflect what was stored in the database.
Starting point is 01:06:28 A big thing that was just sort of a learning from, not just maintaining Rails, but also my time as a consultant, is that it's unidiomatic in Diesel to use the same type for reading from the database as you use for writing. You guys mentioned you do Elixir and Phoenix, so this will probably be familiar to you guys. So we have a separate struct that you'll implement
Starting point is 01:06:51 a trait called change set for, mostly just by putting derive change set on it, or derive as change set on it. And the design of DSLs meant that your structs which implement queryable are meant to be one-to-one with the queries that you're executing. And that may or may not be one-to-one with your database tables. And then your structs which implement insertable or as change set, and oftentimes you will have one struct which implements both,
Starting point is 01:07:17 those are meant to be one-to-one with your API endpoints or web forms. Because in my experience, the needs of those two things tend to diverge over time and they change for different reasons. So the design of Diesel, it's possible to have those two on the same struct, but it's a little bit painful and it's unidiomatic. And the design of Diesel is meant to kind of gently nudge you
Starting point is 01:07:38 to separate those early on to make your life a little easier later on. That actually sounds pretty nice coming from where I've been. those early on to make your life a little easier later on. That actually sounds pretty nice coming from where I've been. Having those separations, I think, definitely would grow better with an application than the way ActiveRecord
Starting point is 01:07:56 does it. What about the other way? Is it anything from Diesel, any gleanings from Diesel that have found their way back into the Rails code base or perhaps will be oh yeah i mean all of i mean i i can't actually like there's only a few instances i can list of like explicit cases you know it's very hard for me to talk about all the places where just like because i've made diesel i'm better equipped to maintain rails um but there are
Starting point is 01:08:21 definitely a few concrete cases uh i'm eventually looking to get rid of ActiveRecord Relation. And by get rid of, I don't mean deprecate, I mean move to a gem. And continue to maintain that gem, because I don't want to do Rails 2.3.3.0 all over again. But move it to a gem and start to explore alternate query builders, because ultimately Relation uses a full SQL query as its unit of composition.
Starting point is 01:08:46 And I think that is the wrong level of abstraction for a query builder to operate at. And I think that's most apparent in how long relation or took us to add and why it was so tricky, which there's a conference talk where I went into the details of it. We don't need to recount all of that here. Anyway, so diesel is very much
Starting point is 01:09:03 kind of what I'm imagining an eventual future Rails query builder might look like. So there's that. A slightly more concrete example is I'm working on a new Postgres driver for
Starting point is 01:09:18 Diesel right now. The current Postgres driver is built on libpq, which is the C library for interacting with the Postgres wire protocol that is shipped with Postgres itself. It is also what the pgRubyGem is built on top of. And very specifically, so Postgres allows you to transmit values
Starting point is 01:09:37 as either binary representation or text representation. And we use text representation always because the binary representation is generally undocumented and not considered to be stable, even though it is effectively stable, at least for the most common types. The only thing that is actually documented is that the binary representation of numeric types
Starting point is 01:09:57 is network-endian and everything else. You go look at the C source code. Diesel right now always uses the binary representation and there are certain data types that I've just not added support for yet because I don't want to figure out their binary representation.
Starting point is 01:10:17 On the flip side of that the binary representation of the timestamp type in Postgres is a signed 64-bit integer representing the number of microseconds since January 1st, 2000, and an unsigned 32-bit integer representing the, or I'm sorry, the number of seconds,
Starting point is 01:10:33 and then an unsigned 32-bit integer representing the sub-second portion of that, which is, I believe, microseconds, but it's actually dependent on a compiler flag. But basically you can assume it's microseconds, but it's actually dependent on a compiler flag. But basically you can assume it's microseconds. So time or date time is the most expensive kind of core type supported by Rails in terms of typecasting. Because the difference and the difference in performance
Starting point is 01:10:57 between turning those two numbers into a Ruby time object versus doing arbitrary string parsing is enormous. So the actual main driver for me doing this for Diesel is adding async IO, which Rails would not be able to take advantage of. But then one of the other drivers is libpq allows you at the per query level to say I would like all of the results back
Starting point is 01:11:23 as binary versus text. But the wire protocol itself actually allows you to specify per column whether you would like the results back as binary or text. So when I finish this driver for diesel, I'm then going to pull that out and write a Ruby wrapper for that and ship that as an optional new Postgres adapter for Rails, which may or may not become the default, depending on how well shipping
Starting point is 01:11:45 a Rust dependency of Rails ends up going. And then the final, most concrete way that diesels influence Rails, about three weeks ago, I made a change to Rails where I changed how we handled bind parameters and fixed a bazillion longstanding issues that were mostly gone in 5.0,
Starting point is 01:12:05 but kind of still hung around and now are completely impossible. And it was a change to A-REL and basically just how we manage bind parameters with our AST. Since we're running short on time, I won't get into the technical details, but the way I implemented it
Starting point is 01:12:18 was I took some code from Diesel and I pasted that into A-REL and I converted that to Ruby syntax and then just followed the test failures until until everything was green the old copy paste and now i've realized i can do that and it works legit strategy that's that's just my strategy now that's your first step on all bugs just copy some stuff out of diesel paste it in yeah first implement it in diesel and then copy paste it into rails that's hilarious all right so that's probably a little more wordy than you were looking for no that it those are great concrete examples and i just love
Starting point is 01:12:56 seeing you know basically the fruits of labor in one place get applied across especially in a project like rails which so many people benefit from your work there so that is good stuff. We're running real short here. Two really quick questions and we'll call it a day. The first one is DieselHits1.0 are you ever planning on picking up the full web framework
Starting point is 01:13:18 in Rust? No. I had a baby. I can't do that. I don't have time for that anymore. You have a baby. Ruby takes up all my time now. My baby's named Ruby by the way. Your baby't do that. I don't have time for that anymore. You have a baby. Ruby takes up all my time now. My baby's named Ruby, by the way. Your baby is named Ruby. Yes. That's a great name. I'm not sure if this is going to be a thing or not, but
Starting point is 01:13:33 if I can just plug this and then maybe we'll cut it if it doesn't end up happening. Also, if you're interested, I have these really cute baby Ruby stickers, which are for sale on DevSwag and there's a link in the show notes if you want to support my development of Rails and Diesel. There you go. Get out there and support Sean. Buy a baby Ruby sticker
Starting point is 01:13:50 or 10, if it's a thing. Last question. Hypothetically, you're stranded on a desert island. You only get one programming language that begins with RU. Which programming language do you pick and why?
Starting point is 01:14:10 I'm trying to think of a third programming language that begins with R-U to give a facetious answer. Yeah, if I would have said R, you could have just said R. Yeah. Probably Rust because Rust can do everything Ruby can, but the inverse is not true. Smackdown. There you have it. And I also, like, I love Ruby,
Starting point is 01:14:28 but I do often find myself working through problems in Ruby by prototyping it in Rust. Good answer. Sean, thanks so much for joining us. It's a lot of fun. Yeah, thanks for having me. All right. Thank you for tuning into The Change Log this week.
Starting point is 01:14:44 If you enjoyed this show, share it with a friend. Rate us on Apple Podcasts. And thanks to our sponsors, Linode, GoCD, and TopTile. Also, thanks to Fastly, our bandwidth partner. Head to Fastly.com to learn more. We host everything we do on Linode cloud servers. Head to Linode.com slash ChangeLog. Check them out.
Starting point is 01:15:02 Support the show. The Change Log is hosted by myself, Adam Stachowiak, and Jared Santo. Editing iselog. Check them out. Support the show. The changelog is hosted by myself, Adam Stachowiak, and Jared Santo. Editing is done by Jonathan Youngblood. And the awesome music you've been hearing is produced by Breakmaster Cylinder. You can find more episodes just like this at changelog.com
Starting point is 01:15:17 or by subscribing wherever you subscribe to podcasts. Thanks for listening. Thank you.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.