The Changelog: Software Development, Open Source - Enter the Matrix (Interview)

Episode Date: March 9, 2020

Matthew Hodgson (technical co-founder) joined us to talk about Matrix - an open source project and open standard for secure, decentralized, real-time communication. It's open source, it's decentralized, it's end-to-end-encrypted, and it's also self-sovereign. Matrix also provides a bridge feature to bridge existing platforms and communication silos into a global open matrix of communication. A recent big win for Matrix was Mozilla's announcement of switching off its IRC network that it had been using for 22 years and now uses Matrix instead.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com slash changelog. Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com. What we love most about Linode is their independence and their commitment to open cloud.

Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.com slash open. Again, linode.com slash open. What's up? Welcome back, everyone. You are listening to the ChangeLog, a podcast featuring

the hackers, the leaders, and the innovators in the world of software development. I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog. On today's show, we're talking with Matthew Hodgson about Matrix, an open source project and open standard for secure, decentralized, real-time communication. It's open source, it's end-to-end encrypted, it's decentralized, and it's also self-sovereign. Matrix also provides a bridge feature to bridge existing platforms and communication silos into a global open matrix of communication. And of course, we talked through all this. A recent big win for Matrix was Mozilla's announcement of switching off its IRC network that had been using for 22 years and now uses Matrix instead. matthew we're here to talk about matrix maybe

not the matrix but matrix which is an open source uh secure decentralized communication standard protocol software there's a lot here first of all thanks for joining us well thank you very much for having me on the show it's a privilege and I'm really looking forward to trying to tease apart all the various different bits and bobs we have in Matrix because you're right, there's lots of different bits there. There's a lot going on and a lot of buzz going on as well, especially since recently Mozilla adopting Matrix and killing off their 22-year-old IRC dependency.

Before we get to that, maybe we'll talk about the idea itself, where it came from, and the birth of these bits and bobs. Yeah, sure. And yeah, it's topical. I guess it's just yesterday that Mozilla turned off their MozNet, one of the older IRC networks, and honestly, we felt pretty sad about it. But we can talk about that in a bit. So I guess, in terms of trying to explain matrix itself, as you said,

it's an open source project. And we provide an open standard for secure decentralized communication. So you could call it an open standard or an open protocol. And you also get an open network out of it. So what we've tried to do is to basically build the missing communication layer of the open web. Because there are many, many different ways to communicate online these days. And almost all of them are proprietary closed silos like WhatsApp and Slack and Discord and Facebook and so many other ones. And we had been building these as a professional team for about 15 years for our sins, typically building them actually for big telcos. So tier one telcos like Tim Brazil, the biggest mobile network in Brazil,

and Singtel in Singapore, and other guys. And I used to run a unified communications business unit for an outfit called Amdocs who are a large multinational telecoms supplier and we got fed up with building yet more proprietary silos and taking this problem of all of these little islands which can't talk to one another which go and trap their users into these silos and instead replace it with something completely different and try to do the impossible follow the holy grail build an entirely new communications protocol put it out there completely open on the internet for anybody to build on and participate in and i hope that if we can pull it off we might

have something that could replace email, could replace the phone network, and be a new kind of common denominator way for people to talk in real time. And that could be instant messaging, could be VoIP video, could be in virtual reality, could be IoT data, any kind of conversational real time data, have a way to synchronize it from A to B. So on one level, it sounds like a crazy moonshot and a bit over-ambitious. And certainly when we started it five years ago now, actually five and a half years ago, September 2014 is when we launched it, it felt totally nuts. But since then, we've just been plugging away at it,

trying to make it all work. And so far, it seems cautiously positive. When you said sad regarding Mozilla and Matrix, was that because of IRC going away? Yeah. I mean, I'm an IRC geek. I'm sure that many people here have spent their lives on IRC. I started adminning a network back in 99. It was a Tolkien network, actually, because that's how cool I am.

Cool to me. It's still going strong now, 20 odd years later, and I still have my IRC set up. I have my Z and ZNC running, or ZNC, and all that stuff. Actually, I access it via Matrix these days, because we bridge into existing networks like IRC and all that stuff. Actually, I access it via Matrix these days

because we bridge into existing networks like IRC and XMPP from Matrix. But I really get the IRC thing. Our earlier companies were founded from IRC. We started working together because we knew each other on a university underground IRC channel in like 2009 and 98 and so

and so basically grew up on irc ended up creating matrixes in some ways very heavily inspired by irc but built on 21st century kind of technology and feature sets and with a slightly bigger vision in terms of not just being chat but being an open fabric but yes i first had to say i'm i've got a very very soft spot for it and i felt in some ways really sad that we were the reason that they turned it off entirely frankly i wish that we'd left it going and kept it bridged but they wanted to make use of all of the moderation stuff that we have in matrix and try to get the best possible sort of starting point experience

and try to minimize confusion due to somebody on IRC saying, oh, I can't see that emoji because my client doesn't do emoji. Or I can't do threads because IRC doesn't do threads and that sort of thing. So I see why they did it, but still feel like we accidentally killed an old friend. It's just funny to see you be sad because we're so enamored with progressing,

you know, that you don't often stop to see the dead as supposed to be very graphic, you know, like dead technologies or things that fall by the wayside as we progress. And so we're so, I guess, attracted to change and the future and modern and 21st century, as you just said, that we, I was just surprised to actually see you be sad that IRC went away. Yeah, well, as I said, it's sentimental value. And also, I mean, this is where we spend the entire show talking about IRC and feeling sentimental about it. But seriously, I have a weird theory that the lower the bandwidth you have to communicate, in some ways, the more intimate and more emotionally connecting a communication technology can be. Yeah, I think what happens is that the brain fills in the gaps. So if you're on a big glossy 1080p video feed like this, it's pretty unambiguous on

what's going on. You can see each other's body language. You've got all of your amazing audio. And it's like real life. It's very high bandwidth. But then if you take all of that away, and you end up with a really, really, really low bandwidth, kind of minimalist thing like IRC,

there were friends, particularly from the Tolkien network, who I'd never met in real life, and became incredibly close friends with, i'd never met in real life and became incredibly close friends with and um subsequently did meet in real life and i ended up introducing folks today who then got married so i mean it's not rocket science everybody knows how well-known relationships work now it's a little bit less obvious in the late 90s and i think the fact that the more minimal you are, the less bells and whistles that has a certain appeal.

And it's not just grungy retro things because the Tolkien geeks I hung out with weren't techies. They were total randoms who just happened to like Lord of the Rings before the films came out. I hasten to add. So actually, and it's just to be united for a commonality of interest in the books. And it was just really interesting that the more minimal, the more the mind.

It's like reading. The difference between reading and watching a film. Sure, you can watch a film, super, super rich, and you have a shared experience with somebody. But it's never going to be as vivid as the imagination you have when you read a good book. And the more that is left unsaid, the more vivid your imagination is. I can agree with that. There's been several books where I've read it, and I like the movie because, hey, you get an hour and a half bite-sized version of the book,

and that's nice because they put a visual to things and kind of paint in what your imagination leaves out. Well, don't they always disappoint you,? Because it's not what you imagined. Kind of. The one I would say in particular that I was still very happy with was Ready Player One. As a book, it was amazing. But as a movie, maybe it's Spielberg. I don't know.

But I mean, it was still amazing to me. It's a controversial one. Jared and I never agree on movies. That's why I say it to me at the end there because we almost never agree. I like movies he hates. Well, I didn't see that movie so I'm not going to comment on it. I saw the five minute clip you showed me in your home theater and I enjoyed that because it was like a spectacle.

I just think that in my experience almost every time a movie's been based on a book that I previously loved, the movie has been a letdown versus your experience. And that's where I can agree with Matthew, is that what you see on a film may never be as vivid as what you can imagine. And I do agree with that.

So I think the same is true, honestly, on IRC and segueing beautifully into Matrix, where we have a much, much, much richer set of primitives to work with. You have your stickers, you have your emoji, you have your animated GIFs,ifs inline images inline file transfer um full video conferencing full voice conferencing and we've even done 3d video calling in vr over matrix just to try to spell out to people but it's not just um wow text chat but uh and also you can't have a project called matrix which doesn't have a vr element let's face it so we built virtual worlds in Matrix and using Star Trek TNG Season 2 holodecks in WebVR

and went and took iPhone X. And I modded WebRTC to support depth channel as well as video channel. And so you would set up two video calls over Matrix, one for depth depth one for the actual video and then get a kind of dot cloud slash mesh of the person sitting in vr just to spell out to people that look you can use this for synchronizing any type of real-time data in this instance it's just the call signaling it wasn't the actual call media which goes over srtp the standard encrypted real-time protocol thing that i guess we're using to talk right now but the rest of it and the scene geometry was coming out of matrix no that's the other end of the spectrum doesn't get much richer than that to be in a virtual world with somebody and you put a vive

on or even use a project daydream style and headset and it was super compelling you could get eye gaze correction happening for line of sight correction because in vr you just physically move your head around until your eyes are lined up with the other person you don't need to mess around with voodoo like apple tried and failed with facetime to kind of slightly re-render everything and ironically they are actually doing it the same way that we were doing it but humans naturally gravitate to fix their line of sight. Anyway, long story. Basically, we've gone to the other extreme.

But in the end, a lot of the most productive conversations are the ones which are completely indistinguishable from IRC. And I think one of the reasons Mozilla likes Matrix is that the IRC routes are very visible. And if you want to use it and play with it and hook up wechat to it as a really nice command line irc client um somebody wrote a really surprisingly impressively good matrix module for it complete with end-to-end encryption and typing notifications and all this sort of thing and people can use it and pretend that they're in a kind of modern irc and if they do that and it works for them then that's a good result absolutely so i was trying to think back i couldn't find it on our search adam but i was trying to find that

conversation we had a couple years ago about the decentralized social web and we talked about the success of email and the failure of irc in terms of mass uh not mass production but what's it called critical mass yeah so email hit critical mass irc never did and i remember we had a fun conversation just kind of hypothesizing what held irc back and why email succeeded where irc failed and matthew i'm curious because of your love for the for the agent and nostalgia for it and still i agree that a, you can get a lot done with plain old text. I'm curious your thoughts on that. Do you think email had a advantage over IRC and usability?

Or why do you think that email hit critical mass and IRC was always, you know, the dwelling place of geeks and nerds? That's an absolutely excellent question. And it's something that we literally obsess about on a daily basis these days, because you can only look to the past and learn from it to avoid making the same mistakes as we go and build our protocol out in matrix. I mean, there are many aspects to it. I guess one thing with email is that there was nothing before it. As in you had to have a bootstrap first commonality to let people talk to one another. So whatever the first thing was that could bridge that gap in a global way would have amazing first mover advantage, as we've seen. Secondly, it's open federation, whereas IRC is closed federation nowadays. In

the very dim days of IRC, right at the beginning, when there was only one network, it was actually open federation and anybody could just pitch up and try to peer in or set up a S line or whatever it is to go and connect to another server. And it was a disaster because it didn't scale and you got abuse. And then there was the creation of Fnet, I think is the first fork out of the original IRC network when they Jupiter'd off a server called Jupiter to split it off, and so on and so forth. But it was basically a closed federation. So you couldn't just jump on board. Probably the other major thing is the lack of a killer app for IRC. And when we look at where open standards for communication

have happened before, like SIP for VoIP, and XMPP, obviously for chat and IRC. There's never, for any of them, been a no-brainer killer app like Netscape was for the early web or like, I hate to say it, but Outlook Express was in the early days of email, relatively early days of email. Let's face it, every single Windows box had Outlook Express in there and people used it. And then obviously Gmail has come along and Hotmail came along and people moved to the web i use it as well

in anger but um my point is that people were really taking time to bootstrap the protocol where i guess pine would have been the early email equivalent and mutt to some extent but pine was pretty mainstream. Like when I was at university, every student had a Pine login and was expected to, you know, fire a putty on their Windows box or even use an actual VT100 somewhere, at the risk of sounding ancient, to go and check their email via Pine. And people made clients that worked that were usable by normal people.

And I think there's an elitist thing, honestly, on IRC, that geeks think, well, this just needs to be good enough for me and my developer mates. And in fact, the geekier it looks, the cooler it is. And I don't want any of these mainstream normies going and screwing up my special IRC network. And that attitude kind of prevails. And on XMPP, similarly, nobody sat down and said,

right, I'm going to write the Netscape equivalent for XMPP. So this is something that we very consciously tried to fix with Matrix, that we built the protocol, but also invested more time than anything else on Riot, which is the reference app or the flagship app that we built as the core team to try to be something that looks and smells like Discord or Slack or WhatsApp. And people who don't give a stuff about decentralization don't care about open standards and the open web or anything like that. They just want a secure version of

slap that they have something that they can use so that's i think a reason why we might be succeeding better in terms of mainstream adoption than some say irc or even x and pp and one of the things that email almost had to get right in that the bar was so much lower at the beginning that any app that actually allowed you to do email, even if it's like the original Unix mail client, the horrible command line thing that takes one letter of actions, even that was a killer app when it launched. Email is very personal too.

I mean, it still prevails, right? Many people build businesses around email newsletters alone. So it's still very much personal, still very much a commerce opportunity. Definitely killer app. Would you say that that kind of bleeds into this idea that adoption is driven by ease of use? Would that kind of bleed into ease of use, the killer app idea? Yeah, there's usability. Yeah. I mean, obviously it needs to actually be there for a purpose and yeah with email email

is still the only lowest common it's the only commonality between any people out there on the net it's true yeah i mean you've got the phone network and so you can piggyback on top of that for say what with whatsapp or something but um if you're not piggybacking on top of the e164 numbering system that you use on the phone network you basically start with email just like i think that we got in touch here with email coming in saying hey do you want to come on the show and if you want to reach out to somebody that's how you do it so i think obviously it needs to be usable you need to have a usable app on top too. And it's very true for us because we're competing against these very glossy apps

like Discord who don't need to worry about any of the standards, the APIs, the ecosystem, the open source, a governance model to let people contribute to how the thing evolves, et cetera. They can spend all of that time and effort on glossy interactive UI. And so we have to do that and more

as well as the rest of it to keep up to succeed. Well, you mentioned that you got tired of building silos, right? And that's what Discord is kind of, is a silo, right? You go and make your own account. It's a silo. Can't bridge other things. You can't connect, you know, conversations

from one community to another. You can't create public channels. You can't connect conversations from one community to another. You can't create public channels. It's missing all that stuff. It is. I mean, it's a nice experience, and they seem to be doing pretty well. But it's the epitome of a silo, unfortunately. And that's the last thing that we want Matrix to be. I mean, for context, Matrix has got about 40,000 deployments that we want matrix to be i mean for context matrix has got about 40 000

deployments that we're aware of one way or another on the public network right now and some of them are massive with millions of users some of them are little raspberry pi like i've got a pc in the corner of the room here with my personal server which runs on it and it's the opposite of a silo. It feels like an ocean, a web, if you will. Where are we as humanity then short-sighted when it comes to, I guess, massive communities? I mean, you seem to have some deep thoughts on this stuff. So if you kind of got into Matrix and the solution by being tired of building silos,

what are we missing as humankind when it comes to networks of large communities with massive amounts of knowledge and silos? I hope nobody is listening to this, hoping that we're going to be geeking out about like HTTP versus CoAP or JSON versus CBOE or what is the ratchet we use in our end-to-end encryption? But no, it's an excellent question. I'm very happy to talk about what we're missing as a species. Let me answer with a story, which is about two years into Matrix, it was kind of working. We've got thousands of servers out there and probably hundreds of thousands of users on

the network. And we were trying to figure out where it would go from there. And honestly, we started to see some level of abuse popping up on the network. Started to see our first spam. Started to see people being unpleasant to one another. As we started to introduce the end-to-end encryption, it became obvious that an open, secure, decentralized encrypted communication network that anybody can participate in without any restrictions whatsoever can encourage some pretty colorful to say the least behavior from the human species and if this thing continued to scale as it has since we've got like 15 address 15 million addressable users on it now you're going to end up with a massive global community that just keeps

growing and if we do the right thing, it should be bigger than email. And just like email has got massive abuse with spam, and all the various different flavors of spam that you see. Likewise, in matrix, we could disintegrate into an absolute firestorm of spam, abuse, and everything in between. And it was literally almost a light bulb going on, or almost a light bulb going off, that we thought that we were building a decentralized communication network. And we built one, and we realized that actually the problem that we have to solve is nothing whatsoever to do with decentralized communication,

and absolutely everything to do with decentralized reputation. Going and giving users the ability to empower users to basically tune out the stuff they don't want to see. And it might be that they don't want to see porn, it might be that they don't want to see political content might not be spam, it might be illegal stuff that they don't want to see. Or they might want to discover alternatively, the best open source content or the best tech content or whatever it might happen to be. And that is an entirely separate problem in its own right, which is, first of all, nothing to do with matrix. It's a problem that permeates society in general.

It's a problem that we see all over the internet and the web today. It's a problem that is biting Facebook monumentally on the ass because they just haven't ever spent the time to figure out how to ethically and sustainably let their users choose what to see. Instead, they algorithmically choose the optimal information that will addict them to come back and watch more ads. And it was this horrible realization that basically the successive matrix is contingent on solving the problem of global communities, which is the problem of empowering people to basically control the filtering algorithms that select what they can see. So that's probably not the answer you were expecting. But in terms of where I see the problem of large global communities, it's because they rip themselves

to shreds, both in real life and online, if they end up being fed propaganda and disinformation designed to addict them and designed to keep engagement at the cost of the individuals. And people don't see it coming. I mean mean people have finally wised up to it in facebook and twitter to some extent in the last couple of years but we were panicking about this like four or five years ago before the cambridge analytic stuff saying oh my god if we don't go and figure out a way to stop well it's either just going to be total chaos and explode or it will get taken over by nasty folk and be considered a dark web kind of bad reputation where matrix is the place you could go if you want to find bad stuff which obviously is going to impact its ability to go mainstream not to mention just make the

world a worse place so we've been spending an awful lot of time thinking about solving the small problem of disinformation abuse propaganda and propaganda, and fake news on the internet. Because if we don't, the project fails. If you're building out a digital store or e-commerce site, what's the one thing you have to get right? Ding, ding, ding, ding. Yes, taking payments. That's right. I've got good news for you, though. Square APIs and SDKs make taking payments, managing orders, catalogs,

customers, inventories, or even employees painless. With Square, you can build commerce apps that go beyond payments. They support iOS, Android, Flutter, and React Native for in-app mobile payments or to integrate with Square readers from your own app. Square also has payment forms for embedding a checkout experience directly into your website. Check out tutorials and explainers on Square's YouTube channel for developers at youtube.com slash squaredev. You can learn about key concepts like item potency, how to take digital wallets in an online shop,

or how to store and charge a card on file. Again, learn more and get started today at youtube.com slash square dev. So Matthew, you called it a moonshot and you also described all the ways in which it's hamstrung versus proprietary competitors for that moonshot, which is the foundation and the encryption and decentralization and all the things of open source governance. And so you have effectively a hamstrung moonshot you're trying to make. And so the question that comes to my mind is, how do you fund multiple year hamstrung moonshot like this? How are you going about it?

Oh, good question. I mean, you're right that it's harder to do this than in a conventional model. And ironically, we have a side-by-side comparison between the same team who were building these proprietary silos originally. And in some places, they were more sophisticated than Matrix. We had things like streaming file transfer

so you could like live stream video as a file upload and watch it in real time as it downloaded really really cute really fun and then we did matrix and i'm guessing it's about six to ten times more manpower required in time and effort to build something we had already built at this time as open source and do it in public with open governance and open standard so in terms of the funding question we were lucky in that the first couple of years we did it inside amdocs so we took the unified communications team wholesale and suddenly pivoted overnight with the blessing of, of the powers that be to go and create matrix on the basis that if we were successful,

MDocs can one day go to AT&T and say, Hey guys, do you want to buy a billion dollar carrier grade matrix deployment from us? And if any luck, there will be good reason for AT&T to stop messing around with this 150 year old PSTN stuff and actually come and jump on board Matrix instead. And they reasoned that if anybody would pull this off, it would be an existing proven team who were profitable and successfully delivering

carrier-grade telco stuff. So that's how we were funded for the first two or three years. And then it got to the point that we were successful enough that other folks were building on top of the protocol. Particularly Ericsson spun up a business unit called the Contextual Communication Cloud Business Unit, who were providing enterprise communication tools based on top of Matrix. Also Talos, the French defense company, was spinning up a unit called Citadel, who provide military grade

communication on top of matrix and mdocs were saying well guys congratulations you're slightly hamstrung but not very hamstrung moonshot is wildly successful and why the hell are we the only people paying the bill for this and also more practically, the thing would never ever work if it had a single corporate sponsor who was funding and driving the entire thing. Because it would be as if AT&T had invented the project and is the owner of all of the open source intellectual property inside an asset lock and has a neutral board of guardians, as we call ourselves, who are practically speaking the directors of the non-profit foundation of which two are myself and my co-founder amandine who came up with the idea of matrix back in about 2013 with me and then three independents we've got the marconi professor of communication from cambridge university over here where a lot of the team started out and also we've got ross who is a decentralization privacy expert lawyer in washington who works for neo-america as a i guess the policy expert in the space and also jutta steiner who is the ceo Parity in Berlin, who are one of the major players in the Ethereum blockchain space.

And it turns out that Parity as a company runs entirely on Matrix and has done for about three, four years now, which is petrifying given how immature we were three or four years ago. But basically, we got independent people from the community to come on board and look after, basically be custodians to make sure that nobody screws it up, including myself, including Amandine as founders and the creators of it. Because, yeah, we need to keep the lights on. And the way we do that was to then set up a standalone startup called New Vector. It was originally called vector vector matrix get it

then we spun out of amdocs or parted ways shall we say with amdocs and so we were on a new vector so we called it new vector it's a terrible name well naming is hard so we don't blame you yeah it is just had the word new to it then they did that in Silicon Valley. The show. They were spinning off Silicon Valley startups and China as a whole thing. It was funny. This is like new Pied Piper. It wasn't anything special. I hate to say it, but I've never

watched an episode of Silicon Valley because as far as I can tell, it is way too close to the truth. Particularly the whole decentralized stuff. You should do it. I'm only thinking about New England. I'm thinking about New Mexico. I mean, this is an established naming convention.

Yes. Possibly even New York. Yeah, New York. There you go. I suspect we won't keep the name forever, to say the least. But for now, it's done okay. And we got the whole team, apart from, in the end, one guy to come over to basically be hired by us to work on it.

And we keep the lights on by providing professional services and SaaS hosting for Matrix because everything is open source, Apache licensed. You can run it yourself. Often, it's just easier to ask the guys who wrote it to run it for you and so we've got a big kubernetes deployment which kind of leverages the economies of scale and all the expertise that we have in running the servers so for instance we host mazilla's server for them they could obviously run it themselves but the fact that they have the option to migrate off at any point and run it themselves is kind of good enough it's a safety net they're not being you know trapped into a vendor like slack um or however it might be and that's the value prop that we provide on the sas

platform which is called modular.im by the way if anybody is interested in getting an excellent matrix server hosted by the creators and the experts and then separately we do professional services consulting work for anybody who is doing large-scale matrix deployments and needs advice on how to get it right and that started off with the french government who reached out a few years ago now because they realized they had a security risk that everybody was using telegram for their internal political and public sector operations which has some fairly questionable encryption and heritage in terms of using a russian communication app if you are yourself a nation state surely you should be using something that is under your own control and you don't have to trust anybody sitting in a different country and so they went and googled for decentralized communication systems or self-sovereign encrypted

communication and we're basically the only one even now who comes up and so they pinged one of our android developers on github that well-known way for the government to tap you on the shoulder and he said oh my god man you have no idea who this is but they say that for the government to tap you on the shoulder and he said oh my god Matthew I have no idea who this is but they say they're for the government and he's based in France so it was literally his French his local government reaching out saying it's the Ministry of Digital saying they want to use Matrix everywhere help help and so went over to Paris and met with them really really nice bunch and amazing first conversation with them where you think everybody's going to be very political and public sector and i think the conversation began with one of the folks in the

room asking how we did antivirus which is good question because if you're end-to-end encrypted how are you going to get an antivirus server in there to go and check where the attachment runs are you going to have to start running client-side? Are you going to install Norton Antivirus on your Android? You know, how are you going to do it? And it turned out that he'd gone and set up Fuse as user space file system on his matrix server and put it for an antivirus layer there, not for the end-to-end encrypted stuff,

but basically to independently be the first person I knew who had actually integrated antivirus properly into Matrix, which I thought was, ah, this is going to be a good relationship if we're coming at it from, you've hacked up your own fuse-based Matrix distribution. So we've had a lot of fun working with them and provide a lot of services to them

for keeping their deployment running, which is now, I think, 60 servers split over all 16 ministries, and it targets 5. a half million employees because turns out more than 10 percent of France works for the government who knew and it's probably our biggest success story so far in terms of getting people off whatsapp and telegram and email for that matter and just having a massive cross-organizational encrypted system they've had e2e actually turned on by default from the outset.

They were the first people in the Matrix environment to do so. And so a good example is that we built out the enterprise E2E stuff for them where we do do it server-side. And what we do is to send the single message key, not even the message key, the attachment key for the attachment that needs to be scanned is sent to the antivirus server which is completely operationally separate from the communication server so you do break the e2e in the most strategically limited manner possible just to let the av server scan that one file and then proxy and give you a result as to whether you are allowed to proxy it through to the device or not.

And that's an example of the sort of thing we did for them. Very cool. So it sounds like you have the old-fashioned way you have customers. I mean, beautiful. Yeah, I mean, when we set up New Vector, it was 2017, and it was right at the peak of the ICO frenzy. And an awful lot of people were saying, oh my my god you've actually got a working decentralized product if you put a token in that you can be gazillionaires and we did consider it honestly quite seriously in fact i even sat down with

the team at one point and said guys we're not doing very well at finding seed funding for new vector we're gonna do an ico and everybody kind of head desks and looked suitably cautious and worried about it i'm glad we didn't honestly for many many reasons but it's nice honestly to do it the old-fashioned way with good old-fashioned customers and avoid matrix ever being a kind of paper play thing where people are speculating or gaming it to try to get more of the magic beans right that being said you do have a decentralized network that's sending encrypted communications around the world especially now in governments even so very important data even though it's not financial transactions so there has to be some aspect of trustlessness in there.

Or I'm curious how the decentralization occurs and how you trust the servers or if this is like a thing that you sign up as a matrix server and there's a centralized trust factor. I guess maybe at that point we're getting to the nitty-gritty of how matrix runs and how the network operates. So go ahead and let us have it. Sure, very, very happy to talk the nitty-gritty of how matrix runs and how the network operates so go ahead and let's have it very very happy to talk the nitty-gritty so the decentralization aspect of it was shamelessly

ripped off git and in fact at data structure level chat room in matrix is almost identical to a git repository it is a merkle dag of, which are signed into a directed acyclic graph. And you start off with the origin block, effectively the origin event, which is called an m.room.create event. And then as people participate in the room, they add nodes into that graph. If there weren't any races,

it would be just a linear list of nodes but in practice you get races all the time because people talk over the top of each other and you have net splits and somebody might go offline and their server might not be able to talk to the rest of the network and so you actually end up with quite interesting tree-shaped tags as the conversations bifurcate and then rapidly reconverge again. And the novelty on matrix is that the conversation is replicated over all of the participating servers. So just like in Git, everybody has a clone of the same repository and is constantly pushing and pulling commits between copies of the different repositories.

So in matrix, every server, every node on the network maintains a copy of the conversations its users are participating in. So it's decentralized in that the conversations are decentralized over the servers. In some ways, this is pretty old hat. It's very similar to Usenet. Anybody familiar with NNTP and Usenet should be familiar with the idea of conversations being replicated over a whole bunch of participating servers. Obviously, for Usenet, it was done almost as a flood globally, and it didn't have the cryptographic integrity that you get from signing it into a signed data structure like a Merkle DAG

or a Git repository. I'd like to point out that we have nice hash agility in Matrix and like Git. That's basically the way it operates. And the clients can be very, very thin. The simplest Matrix client is curl. And you want to receive a message, you curl slash sync on your matrix server. It will block until there's a message to receive, and then it will return the JSON of that message. And the message can be anything you like. It's just freeform JSON. And likewise, if you want to send a message, it's a single HTTP hit from curl.

Curl minus X post to slash send message. And you give it a message type you want it to send and some JSON body, and you send it into the room where you asked it to go. So, I mean, as a joke on Hacker News a few months ago, I wrote a four-line or I think five-line matrix client in Bash and curl just to really try to spell out to people that your matrix client can be as stupid and simple as possible. It's the equivalent of telnetting into an IRC server.

And everybody says, oh, the reason email and IRC is so great is I can telnet into it. And I say, well, our matrix, well, you probably could telnet if you really want to, but you'd have to type out the HTTP headers. So why not just use curl? But it's the same idea.

On the other extreme though, you can have a really thick client like Riot, which has got offline support it has local um cache um as of your conversations it has um end-to-end encryption built in obviously to the client and all of this good stuff at which point they're up in hundreds of thousands of lines of code but the nice thing about matrix is that it can be anywhere in between depending on where you want to put things to actually answer answer your question, heaven forbid, on the decentralization piece, the big novelty, other than replicating these conversations between the servers,

is how you stop people misbehaving. Because it has to be Byzantine fault tolerant. If I spin up a matrix server and I deliberately add a bunch of bugs into it, and I inject an event that says that, hey, I've kicked you from this room. Everybody needs to decide whether I was allowed to kick you or not. So can I ban you? Have I got ops? All this sort of thing. And for that, we have a proof-based consensus mechanism. It's not quite a consensus mechanism. We call it state resolution, because the primitives

you get in each room is, first of all, the history of the messages in the timeline of that room, the conversation itself. And then a subset of those events are key value data. And it's things like who's in the room, what permissions do they have? What's the name of the room? What's the topic of the room? And again, it's free for you can put any key value data you like into it. But it's the key value data that's important semantically for the events in that room. And basically, if I were to kick you from this conversation, if we were having it in matrix rather than zoom, then I would go and say, Hey, I'm gonna kick you. And the reason I can kick you is because I was given ops at this point by this guy who created the room. And you have to always provide the proof, a chain of events, the auth events, as we call it, the proof that you're

allowed to do the operation that you do. And in fact, you do it on everything you do. When you send a message, you can say, hey, I'm sending you a message. And by the way, the reason I can send this message is that I was invited by this guy. And he was allowed to invite me because of that. And it goes back to the beginning of the room. room typically it's like five or six events in the chain and everybody executes the copy of the off chain um against their view of the room and if they conclude that yes indeed this event was allowed to happen they accept it into their copy of history and if you get a conflict in terms of some kind of race, some kind of contradiction, then we have state resolution kick in, which goes and defines how you merge together these key values in a deterministic manner, such that everybody ends up concluding the same state of the room.

And that's hard. We got it wrong this time. And it was a couple of years before we realized that it was exploitably wrong and some really lovely people both discovered the vulnerabilities and then started to break rooms and exploit them and we had a fairly stressful summer back in 2018 going and doing an emergency rethink of the merge resolution algorithm to fix the thin code that we had and the thin codes that we had there were like three or four shortcomings things like having got a tie break incorrect also having a parameter called depth that should have been untrusted but

we were trusting it a bit more than we should which was liable to people then lying about it and abusing and so forth and so we shipped um at the beginning of 2019 um state res v2 which gets it right and is what matrix is today and so far we haven't seen any disasters with people discovering that they can inject malicious events in order to hijack rooms again felt a lot like the early days of irc if you ever saw a net split where after the net split half the people got ops and they come back and there's a race before the ops gets taken away but if they script it during the race they can hijack the channel it was literally oh my god we've reinvented the mistakes of f net in 1998 all over again but just like irc finally got its act together we did too. Can you

imagine what it takes to build or test that you know like something decentralized like how do you even test like state resolution? Yeah again excellent question so on testing it we have a test strip called SciTest which has got about 850 integration tests. It's a totally separate code base to any of the servers. And it exercises one or more servers and clients to basically go for as much of the API surface as we can. And in terms of APIs, you have the client server API, which is the one I explained earlier for just implementing clients, then you've got the server to server interface, which is the much more exotic one, which has got both the cryptographic hashes flying around the merkle dag as well as state resolution and there are three other ones one for identity lookups one for push and one for

application services application services being clients on steroids that can bridge you into other networks and link through to irc and xmP and that sort of thing. So Citus primarily tests, well, it actually tests all of those these days. It's a bit of a weird project. It's written in Perl. It's not entirely deliberate that it was written in Perl, but it's one of these things that began in Perl and is stuck. And before you know it, you've got 850 odd tests, which are critical to any development of the platform stuck in your CI system. Every time somebody commits something to one of our server implementations, it gets run through the gamut. As it happens, it doesn't have that much in the way of state resolution tests. And we actually

have separate test jigs for that. And it's very topical because we have a next generation server implementation going on at the moment called Dendrite our first one was written in python and it's called synapse and it's lots of python and twisted dendrites written in go and it's ended up being written by almost totally different people to synapse and it had a hiatus of almost two years when we realized we didn't have bandwidth to both work on it and work on synapse but we're now at the point that Synapse is stable enough and we have enough manpower to actually work on Dendrite. And so for the last week, one of the lead developers on Dendrite, Neil Alexander, has been implementing StateRes V2 for Dendrite,

playing catch up with where we were a year and a half ago on Synapse. And he's been writing a lot of tests, a lot of unit tests, lots of test jigs running side test against it also writing more documentation i don't know if you've ever read the git merge documentation but somewhere in the git source repo there's a doc that explains how their three-way three-way merge thing works it's the equivalent of our state resolution it's surprisingly involved and terse and if you get it wrong the whole thing's just going to fall apart in an absolute um firestorm so we um constantly

re-documenting it i think we've now got five different write-ups from different members of the team each one advances the state of the art to better explain how it works and i think there's also a formal proof written in latexX, so it must be true, trying to explain the algorithm and almost do a formal justification as to how state res works. So it's by far the only academically complicated bit of it. And I think that we've got it right. Just like Git in practice is quite complicated, but it works very well in reality. Matrix is also quite complicated, but hopefully it's kind of necessarily complicated. It's not over-engineered for the sake of navel-gazing, but it's the minimum level of complexity needed to be able to merge together decentralized chat rooms in a deterministic, consistent manner.

How often do you think about internal tooling? I'm talking about the back office apps, the tool the customer service team uses to access your databases, the S3 uploader you built last year for the marketing team, that quick Firebase admin panel that lets you monitor key KPIs, and maybe even the tool that your data science team had together so they can provide custom ad spend insights. Literally every line of business relies upon internal

tooling, but if I'm being honest, I don't know many engineers out there who enjoy building internal tools, let alone getting them excited about maintaining or even supporting them. And this is where Retool comes in. Companies like DoorDash, Brex, Plaid, and even Amazon, they use Retool to build internal tooling super fast. Retool gives you a point, click, drag and drop interface that makes it super simple to build these types of interfaces in hours, not days. Retool connects to any database or API. For example, to pull data from Postgres, just write a SQL query and drag and drop a table onto the canvas. And if you want to search across those fields, add a search input bar and update your query,

save it, share it. It's too easy. Learn more and try it free at retool.com slash changelog. Again, retool.com slash changelog. Again, retool.com slash changelog. So, Matthew, let's imagine I want to enter the Matrix. I was just going gonna pun totally intended you went there i did i had to didn't have to but i had to the geek in me had to we have a slack system people are on discord hey some people are still using irc and of course the question

always comes down like this is cool this is interesting i like decentralized i want to be self-sovereign i have all these things that like, but what's it actually like to use it? And what does switching look like? So from our perspective, we have a Slack community. We've been on Slack for years now, and it's what most Slack communities are like. It's got channels. It has people in those channels. You can invite people to join. they join, etc., etc. What is Matrix like to use as a user? I saw there are communities, there's rooms. Explain it to us.

Well, I mean, it depends on the app that you use. And many people use Riot, which was the flagship app that we built to give people a kind of easy onboard. And you can install it on Android, off Google Play or F-Droid or iOS or the web, or as currently an Electron app on desktop and all the various desktop operating systems. And it looks and smells a lot like Slack, honestly. You have lots of group chats. You have public chats, more like you'd get on IRC. You have direct messages.

You have private DMs and private group chats. You can group them into communities, but this is a horribly badly implemented feature at the moment. We're about to rip it out and fix it, but we don't recommend people go anywhere near communities at the moment. We only keep them around because people have used them and depend on them, but it's something used them and depend on them. But it's something that we're working on fixing. And then within that, you have all the building blocks you'd expect out of a modern chat system. Typing notifications, read receipts, really important to have little chat heads which go Tetris patterning down the right-hand side of the screen to show you who has read up to a given point that Slack doesn't have.

I've no idea how Slack survives without it because it's so useful to know when people have read your messages. And you have file transfer. End-to-end encryption is the other big, unique thing that most people aren't used to, that you can turn on encryption, which uses the Signal-style encryption double-r ratchet algorithm.

We actually extended it with something called Megom to support large encrypted group conversations, whereas WhatsApp and Signal, I think, only go up to 256 participants. We're good for at least thousands of devices. And thanks to Megom. And so you have the slight surprise when you enable this of needing to verify that you're talking to the people you think you're talking to this is not much point in encryption if it turns out that you were talking to some attacker all along who was pretending to be the other person and at first we got the ux for this terribly wrong and went and

forced everybody to have to manually verify every device they ever spoke to which gets very tedious very quickly started off being kind of comparing public keys then we switched to looking at comparing emoji so you calculate a shared secret of emoji between the two devices and check that both people are looking at the same shared secret. And then more recently, we've done a huge sprint of work. It's been like four or five months now, going and implementing cross-signing, which means that if I verify one of my devices, and you have verified me, then you will automatically trust the new device. So it starts to feel a bit more like, say, PGP, where if you want to be sure that you're talking to the right person, you verify them once. But then it should last for the rest of time unless they go and revoke their keys and start over with a new identity.

Turns out it's hard to do in a decentralized world. And also the UX around it to try to make it friendly to non-technical users. The idea that you scan a QR code on an existing device in order to kind of cross-sign it and verify the ownership is hard to get right. But we're almost there. We also have to do it across all the platforms simultaneously. And as it happens, Riot is totally different apps and codebases on web, iOS, and Android. We deliberately didn't use React Native or Flutter or anything, but instead have built it from the ground up independently with different teams of people in each place, mainly so that we can modularize it and factor out the SDK as a component other users can build on for their own matrix apps but ends up being a lot of work but that's one of the other big differentiators whatever things where people find lots of work to

be done around the room directory which at the moment is unfiltered so you get a lot of random stuff is there just a single namespace for rooms so I created a JS party room and it's like pound JS party colon matrix.org. Is that global? So that was global. However, it's a bit unusual in that because the rooms get replicated over the servers, you can define multiple aliases as we call them. So it could be that alias on matrix.org, but the same room could also be tagged as hash whatever on Mozilla.org or on GNOME.org or KDE.org or any of the other matrix servers flying around the internet if there are users participating in that room from that place. So you get a local namespace on that server, but the rooms themselves are spread across the whole thing. I see. So the room exists across the entire network,

but this particular way of referencing it is local to the matrix.org domain or server? To the domain, yep. And you get one server per domain at the moment. And are there sub-silos? Like I'm thinking back now, I'm thinking about Mastodon a little bit. I'm trying to like have a frame of reference where there's kind of like subsections of that like you have maybe

that's the community's idea but it's not baked out yet where for instance in our slack setup we have changelog is our top level slack community and then inside there we have rooms and so those are local uh is there any sort of like namespacing in that way where you could say, maybe it's at the domain level, like this is Mozilla's domain. So all the rooms are Mozilla's. So you're right. That's how communities are supposed to act. So it's not like Mastodon where you have a server,

which is very, you know, it really sets the tone for what goes on on that server. It's a lot more islandy with links between the islands when you need them, whereas in Matrix, again, it's more like Git. It's not like a Git repository on GitHub as a particular flavor due to being on GitHub. It could equally well be sitting on GitLab or on Gitea or whatever.

So instead, yeah, you have to create your own communities to group these rooms together. And you can do that by just using your server as a way to gather together a particular namespace. And that's a pretty clunky way of doing it. And you can only have one namespace per server by just saying, hey, all the aliases on gnome.org are probably going to be to do with gnome.

And communities are basically a way of maintaining these room directories and also groups of users which can span multiple servers. And the thing that went wrong with them is that we implemented them in a bit of a rush as a new API. It was like a quick, quick, quick, we've got to add communities because this is unusable without them. So we need to have an API that allows you to create a group and allows you to invite people into the group. And the API, they're called groups rather than communities so you create a group and you invite people you're going to need a way to like ban people you need to have moderators in that and it needs to have a name and a topic and it's got a set of rooms associated with it and we went and implemented all of this as an entirely new api

surface before kind of realizing one day that hang on we've got rooms already they've got members who you invite they've got topics and names they've got power rules and they're decentralized and it was at that point the whole thing kind of ground to a halt having realized we'd done it horribly wrong and we went and focused on other things and we're literally now coming back to it to rip it out and at least experiment with the idea of what we call rooms as groups. So you just create a room and you nest rooms in it in a beautiful Matroska way. And if you kind of join the top level of the room, then you would get a room directory that shows you the other rooms which are nested within it. And it becomes a much, much, much more powerful way to have kind of hierarchies of communities and you can see that it can map onto an active directory tree or ldap or whatever other

hierarchy of users you might have or hierarchy of rooms you might have flying around could be your imap hierarchy could be news groups could be a discord server split into different partitions whatever discord calls them okay the bridging concept we spoke about a little bit but this is the idea i'm just going through some of the the jargon used around the matrix.org website trying to understand all the all your bits and bobs so bridging is bringing in other protocols or other networks yep precisely so this is how you have access to traditional irc uh there are about 60 or 70 of them now most of them built by the community on the core team we started with irc because as i mentioned we kind of like irc and

we also wanted to bootstrap it um uh with the irc community and in some ways the irc bridge is the most sophisticated one lots of people on irc use a use matrix is basically a great big bouncer decentralized bouncer in the sky and i think we do about 25 of the traffic on free node um these days much to the i know i think free node are a bit non-plussed about this on one level it means that people are using free node and irc lives on on a level i suspect they'd probably prefer it if they were just talking irc rather than coming in from another land. But it is what it is and people seem to like it. We also did XMPP via Bifrost, which is our next-gen bridge toolkit, which is actually an application framework to build bridges. implement your bridge for say x and pp as a module that plugs into a general all-purpose

you know rainbow bridge like a bifrost in norse mythology slash marvel mythology would look like and we also did slack and we did getter and i always forget the last one those are probably the important ones to be honest and. And then everything else, Telegram, Discord, GroupMe, Facebook, Signal, WhatsApp, all of this stuff comes in from the wider community. And we made the bridges super easy to write. And the Slack bridge is about 50 lines of JavaScript, because it's basically just taking webhooks from Slack, turning it into the matrix HTTP API, and then back again. So they also vary massively in quality and also methodology. We always try to go for what we call double puppeted bridges, which are ones where on the remote side, the users puppet matrix users. So you can't tell that they're not real matrix users. And likewise, the matrix users puppet other side users.

So the people on the other side aren't aware that they're talking to people on matrix. So for instance, there is a Slack puppeting bridge where you can give the bridge your Slack credentials and it basically logs into Slack as you and it mirrors everything it sees in Slack into matrix and vice versa. So it's the ultimate vibe

that people don't realize that you're a dog. I like that term, double puppet. I've never heard of that before. Well, we had to make all of this up because nobody had ever gone into bridging at the same level of detail that we had. And I actually wrote a blog post in 2017 called How Do I Bridge The Let Me Count The Ways, which tries to enumerate about five, six, seven different flavors, which goes from the worst bridge bots, which you just have a bot that joins a room. And I'm sure you've seen it on IRC that somebody joins it from Discord. And you have a bot called Discord bot.

And it just goes and blathers out whatever's happening on Discord. And you can't really see who's talking and it dumbs it all down and then the next level up would be using virtual users so that you actually create irc connections to represent the people on discord and then it goes all the way through to double puppeting and single puppeting and all that stuff so bridging is tough because it's really useful but it's always a bit of a compromise there's always an impedance mismatch between the capabilities you get in one protocol and the big capabilities you get in matrix and so we have people who go from irc to slack var matrix and from irc to x and pp and it works pretty well but you're always going to lose a bit of the experience along the way and also operationally it's a lot of hops for the data to go through.

And all it needs is for the bridges to be running on different servers. Yeah, you can basically easily end up with four or five different hops between the users, which inevitably starts to rack up in terms of latency and just risk of things going wrong. So it's a thing. At the moment, we've got one guy working full time

on them. We're trying to hire somebody else to come and join him. And sometimes they are invaluable. Sometimes they're a pain in the ass. Sometimes they really upset people because they are not as native as a real thing. But they're also why we called matrix matrix as some kind of structure in which to bridge together the existing silos. Matrix is Latin for uterus rather unexpectedly. It's where something grows. It's also where the word mother and maternal and maternity matrix is all the same route etymologically. And so, yeah, we wanted to basically build a substrate in which things would grow and link together the other things which were out there to bridge the islands.

Well, given you came from a world where you were like, well, we're building a bunch of silos, now you can actually build silos, but bridge to them. Hey. Yeah, I mean, whenever anybody comes up with a new protocol and very enthusiastically says, oh, we are creating the one true messaging app, we say, oh, okay, that's cool, good.

Good luck with that, and we'll bridge. And we'll bridge. And we do. And if we don't, then somebody in the community will go and do it. And it's certainly one approach to take. Obviously, in an idle world, we would all be natively doing the same thing, but we're not on a crusade to force everybody to use Matrix.

We're a lot more pragmatic. And I think that XMPP suffered a bit from hoping that everybody would get natively on the network and everybody will have Jabber IDs on their business cards. And we are much, much, much more pragmatic. And if I want to find somebody on Matrix, I look them up based on their phone number or their email address, because those are the IDs that people already have in their address books. You could also use the matrix ID, but much as like at Matthew colon matrix.org.

Ideally, we wouldn't be relying on those. And similarly, sure, it'd be nice if people were natively on matrix, but some of the best communities I'm in are half Slack and half matrix. And all of the normal people sit on Slack and all the geeks sit on Matrix, and eventually, hopefully, Matrix will get so good that people will naturally come and join us in the wider ocean rather than being trapped into proprietary Slack land, not least because Matrix hosting is about five times cheaper than Slack.

Just dropping that in there. Well, that's what Jared left out when he mentioned our Slack history is that we're on a free Slack. And so we obviously have a lot of limitations in terms of integrations and just different things that we can do because even history,

we lose a lot of history. And if you're trying to foster a community with history, because that's important, you kind of miss out on that if you're on free Slack. Yeah. So at some point, while we don't expect Slack to give us free because it does cost money to run,

we'd love to find a more affordable way. We just can't afford the affordable even. It's quite expensive. Right, because they charge per user, and we invite everybody to freely join our community so if we were going to pay six dollars plus per user per month we would not be able to invite everybody to come join it's crazy expensive and they're basically just holding your history hostage i find it's i mean from a businessman capitalist perspective i appreciate

very much the model they have but from an end user perspective it's like the least open and pleasant thing imaginable on the web so i mean you could just go and use a free matrix server you can use the matrix.org one as you're using them today the disadvantage is that it tends to be a bit overloaded it's got easily a hundred thousand concurrent users at any given point and the python code base does not scale as well as it should. We haven't yet migrated over to the Go one. So that's the disadvantage. And also it feels a little bit like a public swimming pool.

Lots of people running around. And sometimes the kids go and wee in the pool. Sometimes, every time. That's a good analogy. Just expect it. It's going to be there that's right somebody told me if you ever get a pool expect them to pee in the pool it's going to happen you know the terrible terrible thing about chlorine um in your eyes that the reason that

your eyes go red with chlorine in the pool isn't actually the chlorine it's because it reacts with urine so next time you get red eyes after going to the swimming pool, that's what's going on. That's why I wear goggles around my eyes. I just keep them closed. I'm never going swimming again, so I'm not going to have that experience. Well, you mentioned earlier in the call, we got Mozilla who recently made this transition from RSE

to what you're doing with Matrix, and very publicly had mentioned that you mentioned the French government. What are some other big adoption, big wins that you've got on your belt? Where are you at, I suppose, with, I guess, your happiness with adoption? We're pretty happy. In terms of growth, it's good. As I said, we're up in tens of thousands of users, 15 million-odd addressable users.

In terms of active users, it's hundreds of thousands daily. And I haven't actually checked the monthly ones, but it must be similar to that. All of those are guesses because it relies on people phoning home with reports and stats. And when you install your matrix server, it allows you to opt in or not to do that. And we guess at least half of the people say, hello, why would I tell them how many people are on my server? So we're running a bit blind, but those are the lower estimates. In terms of big wins, France is not counted in those stats. They have hundreds of thousands of daily active now. They're not quite up at saturation of their five and a half million users, but it's growing pretty steadily.

Other countries have followed them. So Germany has got several matrix initiatives going now, including the Ministry of Defense or the Bundeswehr, as they call it, who unexpectedly announced themselves on Christmas Eve, thanks to I think, a freedom of information request from a journalist who heard something about it and asked them to clarify what they were up to. And it turns out that they're in the middle of a trial to deploy matrix in that context. And also, the Federal Data Protection Minister, who's I guess, the head of GDPR for Germany has taken a shine to matrix and has been enthusing about how there should be a panan government matrix network so that all the governments and public sectors can collaborate together that would be nice wikipedia has a matrix server

gnome and kde have them as also red hat i'm not sure we're allowed to talk about but they do phone home stats so we do know that they are running a big matrix server at least they have a point just don't talk about it some of the Rust community went to Matrix rather than Discord. We're hoping that the rest of the Rust guys will see the light and come back eventually. So it's honestly a weird mix of open source, lots of cryptocurrency projects like Tezos and Status,

Ethereum, obviously Parity I already mentioned, open source cryptocurrency types, also activists and people who really care about privacy and encryption and then on the flip side governments and that's kind of the main uptake right now and we haven't yet shifted into the more mainstream people who are using slack today but that is the big challenge that we're tackling really over the course of this year, trying to get the UX glossy enough in Riot,

trying to make sure that features like communities are there. And in terms of first-time user experience, right now there's quite a lot of WTF movements, so to speak, for new users where we just haven't quite polished it as much as we should, or we've let a bit of the protocol shine through into the user interface. For instance, right now you can invite users into DMs. If you're used to it, it's really useful. Like somebody DMs you and you think, ah, you're not talking to the right person, so let me field a question and then invite in the correct person to talk to you. Or if you want to invite a bot

into your DM so that it can translate it for you or something like that, it's actually kind of useful. but the level of panic that some people have when they're coming over from slack and discover that the one-to-one that they were having with somebody can actually be magically upgraded not even into a dm but into a private chat but it could even be made a public chat if you wanted to just because you can it's a classic example of just because you can doesn't mean you should yeah so lots of work basically on ux to broaden to make it more mainstream and we've just hired two more designers to join our existing very long suffering one to both of him have experience in the professional

app design space for communication apps to try to de-geek Riot and make it more mainstream. You mentioned sort of the cadre of people behind Matrix.org Foundation, but you hadn't really enlightened us on your team size and things like that. Can you share a bit more about, you know, who's doing all, I can't imagine you're doing all this work so who else is with you no i don't think you'd want me to so um the core team when we set up new vector as the spin out in 2017 was 11 people and since then we have scaled up to 41 i think as of this morning. So we've taken on funding as well as the revenue I mentioned earlier. We've taken on funding both seed from Status,

the Ethereum decentralized communication project who wanted to support us because they like decentralization and wanted the project to survive. And then more recently from VCs in London and specifically notion dawn capital and first minute who i think we're very lucky to have got as investors because they get the mission and the idea as well as see the commercial possibility in that we could become

the netscape of the matrix ecosystem and if matrix takes off properly it's a bit of a long shot as you said it's a bit of a slightly hamstrung moonshot but we're quite close to the moon at this point and assuming we don't overshoot or crash into it hopefully we'll have a beautiful landing and that's allowed us to scale up the team significantly for the foundation we don't hire anybody. It's just the directors basically looking after the governance and the IP and the evolution of the protocol. But for people paid to work full time on matrix, well, that's what we do as NuVector. And then lots of other contributors from other companies and individual open source contributors, probably five, six, 700 into the core projects and easily thousands into the wider projects which people happen to have built on top if you can wave a magic wand and a year from now magical things happen i guess regarding adoption you know moon landing future where you're going to what would what would happen to make you know i suppose the killer app the the killer use case whatever it might be what would be the next big step for matrix by far the biggest

thing that we can do right now is to fix the first time user experience in riot and it's tantalizingly close if we can go and get the ux so that people will try using it rather than Slack and think, oh, this is actually a bit better. Then ignore all of the open source, all the decentralization, all of that stuff. Just from a pure pragmatic user perspective, you've got to have that that does know what you need it to do. And we would need to, I mean, that's more than anything else what I'd like to have happen. Secondarily, we need to fix the reputation and abuse challenges.

We've made some good work on that. We have the concept of shared block lists so that if there are abusive folks, you can choose to ban them from your room or server or community or user and then kind of pull that information together. A bit like email blacklists, but hopefully a little bit more nuanced. And that is improving things a bit, but it needs to be a lot more nuanced if it's going to solve the kind of grayer scale things.

If you ban somebody for spamming a room, that's pretty concrete. But if you just happen to want to dial down the NSFW content in a room, that's getting a bit more grazed to what somebody considers safe for work, et cetera. So we need to have that subtlety to support that. And finally, end-to-end encryption

has been an epic to make work. We are the only decentralized, real-time encrypted system that I know with end-to-end encryption. And it turns out the reason people don't tend to do that is it's really hard to get the edge cases to work. And people are very unforgiving of random messages that don't decrypt because there was a race between sending the keys or one of the servers went down in the middle, or there was a net split and the server who sent

you the message didn't know you were even in the room. And so, of course, it didn't encrypt it for you. From a user's perspective, if you don't get the message, that's the one thing you had to get right. So we've spent so much time trying to get that right and turn it on by default. We get a lot of flack in the privacy space for not having it on by default, particularly as WhatsApp and Wire and Signal are obviously all encrypted by default these days. So we aim to get it on for the end of January. It slipped.

Best quote I heard is 10 days from now is our next mission, but we're still working through that work. So I really, really hope a year from now that's old history and everybody is used to everything being end-to-end encrypted. Other than that, on the commercial side, obviously more governments and big, big, big players adopting it to the extent that it becomes the default way to do chat in an industry rather than email or phone calls or jumping on Slack and paying huge amounts of money would be a good outcome. I've already mentioned France and Germany.

The US is also using matrix in production in various public sector environments. Fortunately, we can't really talk about them, but some of them are really, really interesting places where the decentralization is super useful. In fact, one we can talk about is the Texas Department of Emergency Management and Public Safety, where they previously were using WhatsApp for coordination in the field. And they're very extensively using Matrix now

to coordinate in a disaster scenario, work in that domain. And yeah, I mean, it would be great if open source projects didn't use a proprietary thing like Slack and instead use Matrix. And if governments did it too, then between the geeks at FOSDEM and the public sector folk at whatever the open source conference equivalent is for the public sector, they actually do have open government conferences, which are pretty interesting as all the people compare how to build open public infrastructure then perhaps the rest of the world would meet in between and it would just keep growing oh the other thing i'd wave magic wand is to migrate to dendrite and kill off our python code base and move to go because the scalability nightmare we

have on python main not python's fault but we just built it as a, it doesn't scale horizontally, whereas the Go thing is very, very horizontally scalable and is shamelessly inspired by some of our, shall we say, high volume competition in the proprietary space. That would be great because we waste far too much time debugging Python when I would just like it to be efficient. How big of an undertaking would that be? It's not that, well, for the Go migration, it's pretty close. It's really frustrating, actually.

We probably got it 80, 90% of the way there before we just had to pull people off it to focus on the fact that France was going live on Synapse and we had to do this, this, this, and this, and a whole bunch of other commitments like that. And it's only in the last two, three months that we have rehired people who can focus exclusively on it. But frankly, they're on fire. Oh, there's an entire thing I haven't mentioned at all, which they've also been working on. The reason, one of the ways we got Dendrite development going again

is for peer-to-peer matrix rather than client-server. So as of today, and you heard it here first, we had our first ever live traffic over peer-to-peer matrix. What we did was to take the Go code base, compile it to WebAssembly, run it in a service worker on your browser, and stick it into Riot. So you literally just go and pull the server to run client-side peer-to-peer. We used libptp from the IPFS protocol labs guys as the network transport, and you just spin up Riot, and it is a real paradigm shift.

It's really trippy in that you don't change a single line of the client code. It's just talking to an invisible matrix server. It has a URL and everything. And the service worker intercepts those outbound HTTP requests and suddenly wormholes it into WebAssembly and gets Dendrite to run locally to service the traffic. So that's been the slightly unexpected route that

we've taken in order to get the Go thing finished by saying, hey guys, it's not just a boring version two of Synapse, which is going to run faster. Instead, it's a whole different beast. It's a test jig for peer-to-peer experimentation. And suddenly, whoosh, we've made huge progress. So it's probably only a matter of months until dendrite now is good to go in production although migrating over the 40 odd thousand synapses to a totally new experimental code base it's a bit like apache versus nginx is how i think of it apache is going to be around forever it's not that fast everybody knows it is mature. It's got lots of random things built on top of it. And meanwhile, the cool kids will just go

and build on top of Nginx. And I see the same thing happening with Synapse and Android. Are you able to use TinyGo for that? I would think that the WebAssembly executable thing would be rather large. But maybe with TinyGo, it's not. It is.

It is 25 megabytes of WASm at the moment which is yeah so not usable from a client perspective yeah i haven't i'm not a go expert by any means myself i haven't looked at tiny go but if it compels down to wasm why not it could be interesting one of the biggest challenges we had to storage because Dendrite was written specifically for Postgres originally and used lots and lots of Postgres specific SQL features like sequences and returning keywords and stuff. And you're not going to be running Postgres in browser client side. I mean, that would just be silly enough to do Dendrite without Postgres. So what I ended up doing was to write a SQLite driver in Wasm that actually calls over the JavaScript gap into the browser to go and use SQLite, ironically, in Wasm running in SQL.js.

So first of all, well, it turned out to be relatively easy to go and write a go driver for sqlite that would go and terminate in the browser but the hard thing was to then rewrite all of dendrite to not use assumptions that it was running on postgres but the guys finished that in the last couple of weeks and glued it all together using lib p2p on the javascript side so it ends up being this unholy mishmash of Go and JavaScript. And I was shocked and stunned, but it does actually work.

So right now there is a testnet with two people on it talking to one another over peer-to-peer matrix, pinching themselves to believe that it actually exists. But that is going to just change everything when it becomes mainstream because you're no longer going to need a server.

You don't need to pick your matrix.org. You don't need necessarily to pay for hosting, although obviously we hope that people will still want a permanent home somewhere for their data, and it will in some ways make it more valuable. It's a useful thing to have. But watch this space to see how that evolves. Lots of unsolved problems, particularly in routing and

scalability and discovery and all that sort of stuff, but good problems to have. Well, I'm glad you remember that part because that's super interesting and definitely exciting. So Matthew, we're ending up against our time. Thanks so much for joining us today. All the hard work

you've been putting in for many years on this slightly hamstrung but much more pure moonshot that you're on. We hope you have a good landing on the moon. Everybody's safe and sound. And you hit your goals. It looks like you're well on your way. And Matrix

is very interesting, very cool. All the links to all the things related to what we talked about, of course, are in your notes, listeners. So check those out um what's the best the very best place for somebody who wants to get involved from a developer perspective maybe they want to check out this p2p stuff is there a waypoint for community members collaborators or is it all inside of the matrix's matrix guardians of the galaxy thing what your guardianship is there collaboration points for people it's super open collaboration i mean the guardians are a safety net they don't

actually they're not gatekeepers it's just they're the safety net so that if somebody did go rogue that they can step in and say oh we're not going to accept these evolutions to the protocol so if you want to get involved come on to hash matrixmatrix-dev on colonmatrix.org, or on matrix.org, as we say. And that's one of the main developer hangouts, or even just hashmatrix on matrix.org, the first ever room on matrix. And it's a little eclectic as all sorts of randoms find their way into matrix for the first time, but it's pretty cool. Last question, how did you get the matrix.org domain? Because that is an awesome domain.

Yeah. Honestly, we were thinking of names for matrix. It started off being called synapse as a code name, but we ended up keeping that for the server. And we're just trying to think of something that really describes what we were doing. Thought of matrix and thought, no, I can't possibly call it that because of the film. It would just be silly.

And then realized it had been already at that point it that because of the film it would just be silly and then realized it had been already at that point 16 years since the film which is no now it's been like 21 years or something if we really want to feel old and um realized that actually perhaps it might be all right and what's is the domain available went on to it and it was like parked on CEDAW or something. We paid $3,000 for it, for memory, which seems ludicrously good, honestly. I would expect it to be like 50k. Yeah, I was guessing 50, 100k, but for whatever reason, possibly because it used to be, unfortunately,

a hardcore porn website at some point. And so I think its reputation hurt. Little backlinks to get you. Yeah, honestly. Get some interestinginks get you. Yeah, honestly. Get some interesting 404s. Yeah, the SEOs and some of the referrers coming in are quite special. But I don't know whether that really factored in

or we just got super lucky. So I'm happy. It's a good name. Well, you're about to get another spike because Matrix 4 is in production. That's right. If you incognito search for the word Matrix on Google right now, I think last time I checked, not that I do this every night before going to bed,

because that would be sad. Last time I checked a few months ago, we were up at number 11, so almost on the front page. And to be competing against Fundamentals of Mathematics and Keanu Reeves is no mean feat. So we thought we were almost there. But obviously, Matrix 4 is just going to knock us completely off the search rankings forever. Yes, it will.

But if you search for Matrix chat or communications or decentralized communication, you'll come up. So at least Riot's got a really memorable and unique name, which isn't ever going to get confused with anybody else. So that helps a lot too yeah yeah well if the new movie is as bad as the last one maybe it'll just go by the wayside and you'll still you reckon straight to video it's possible i mean i hope not i hope it's good but i have my doubts yeah i don't know it might work out uh but i i was hoping again so that um particularly they might pull it back

together in the third one but it really wasn't to be unfortunately no it just kept going down yep but let's see they might do it hopefully four is on the way up and uh john wick slash keanu reeves and all this fanfare of the last several years has really the last decade really will have uh brought back some new desire for matrix movies the matrix movies no i hope so for their sake but on the flip side i hope we don't get warner brothers phoning up saying how dare you yeah right it's a dictionary word so hopefully we'll be all right and we we haven't got our VR stuff properly off the ground yet. So hopefully nobody will take offense. And if one of the brothers is listening to this or Googling it,

then please just ignore this whole piece of the conversation. We do have transcripts, so it might be possible. It's fine. It won't be on page two, though. It's not going to hit page two. Well, Matthew, thank you so much for, I guess, really, just your excitement for this. I mean, it's so cool to see someone like you

be so knowledgeable about these protocols to lead this and be a guardian of it. So thank you for your time today. I'm easily excited. Luckily, not doing much in the way of actually um building it these days um so thank you so much for having me on and for enduring my long-winded and off-topic answers it's been a pleasure really fun thank you same here all right let us know in the comments at changelog.com slash 384 if you're're planning to replace your Slack, your IRC, your Discord, or other with Matrix.

We're super curious to hear what the community thinks about this. Of course, you can comment on all of our episodes at changelog.com. Head to your show notes and click discuss on ChangeLog News. We'd love to hear from you. Support us by telling your friends, send a text, tweet, Insta story, whatever. Pick your flavor of influence. We would appreciate it. And this episode was hosted

by jared santo and myself adams takovic our beats are produced by the beat free break master cylinder and we're brought to you by some awesome partners fastly linode and roll bar oh and one more thing we have a master feed that brings you all of our podcasts in one single feed it's the easiest way to listen to everything we ship head to changelog.com slash master to subscribe or search for ChangeLog Master in your podcast app. You will find us. Thanks for listening. We'll see you next week. I almost asked this during the show, but this might be good for a break or a teaser or a treat at the end. If one wanted to move away from Slack at some point and they were on the free plan, could they subscribe to the pain plan for one month, get their history and move away and suck all that history into Matrix? Good question. I'm glad you didn't ask it because the answer is that we don't yet bridge history. We should do.

And I think there are some importers. Actually, you're right. I think somebody has written an importer now that will pull in your history, and that might work as a way to get at your history. But I'll need to double-check it. The bridge itself doesn't do it. You'd have to do a kind of GDPR export and then rehydrate it in Matrixland.

But it's surprisingly hard at the moment in Matrix to retrospectively do history because of the DAG it would be like rewriting git history you'd have to rebase the whole thing in order to rewrite time and yeah theoretically yes practically i'm not sure many people do i guess that's the downside of it being git right like that that difficulty if it wasn't for that then it might be a little easier yeah but we can work around it one thing you can do is to actually uh start going backwards in time so you rather than progressing the dag forwards in time you can create a branch and have it chronologically go backwards so you can kind of splice it in that way it's a bit of a head screw but could even be like a i mean since it's immutable it's not going to go anywhere and then we're going to go back and edit their stuff.

Maybe it's a way to nested archive or something like that. Yep. You know, like embedded archive, like into something like that where it's like, it's done. We stamped it. It's done, but we wouldn't bring it with us. And that's interesting. Yeah. I mean, your bridging move is awesome for adoption, but at a certain point what you're going to have is a bunch of people

wanting to jump ship and better than bridging is like escaping so if you can provide an escape hatch with history and like you know make that as painless as humanly possible you could get a lot more adoption off of people that are like you know what I'm done with slack boom