Y Combinator Startup Podcast - #84 - João Batalha and Luís Batalha

Episode Date: July 6, 2018

João Batalha and Luís Batalha are cofounders of Fermat’s Library.Fermat’s Library is a platform for annotating papers. Each week they send out a paper annotated by their community. Some recent papers were Birds and Frogs by Freeman Dyson and Von Neumann's First Computer Program by Donald Knuth.They’ve also built a Chrome Extension call Librarian for the arXiv which allows you to get direct links to references, do BibTeX extraction and make comments on papers.You can find them at FermatsLibrary.com.Read the transcript on our blog.The YC podcast is hosted by Craig Cannon.

Hey, how's it going? This is Craig Cannon, and you're listening to Y Combinators podcast. Today's episode is with Joao Battaglia and Luis Battaglia, co-founders of Fermat's Library. Fermat's Library is a platform for annotating papers. Each week, they send out a paper annotated by their community. Some recent ones include Birds and Frogs by Freeman Dyson and Von Neumann's first computer program by Donald Knuth. They've also built a Chrome extension for the archive called Librarian, which allows you to get direct links to references, do BibTech's extraction, and and make comments on papers. You can find them at formaths library.com. All right, here we go.

You guys are brothers, right? Yeah, we go. Yeah, okay. He's the older one. I'm two years younger. Okay. And what made you want to start for Mott's Library? So just for the people that don't know what it is,

Vermont is a platform for annotating papers. And so if you want to think about it, we imagine a PDF view in your browser, and then you have annotations on the site. that support latex and markdown. And so you can add annotations in parts of papers that you think are particularly tough to understand or you think you could add more content there.

But so it's something that we've done. The four of us that started for Maud, we all have a technical background. And so after college, we kept on reading papers. And every once in a while, we had this internal journal club where we would read a paper and presented to the others. So I remember, for instance, presenting a few years back,

presenting the Bitcoin paper to Louise and Mika, which don't have a CS background. And so you kind of have to go into, for the Bitcoin, you might have to go into, okay, what's a hash function? What's a public key encryption? And so we were already doing this. And we knew that you also have this behavior offline in places like universities. And so we wanted to take that experience and bring.

it online. We thought there was a lot of content that you end up producing while you're trying to read a paper, which can be the most, like, the most dense piece of content that the human can read sometimes, right? The language can be incredibly Spartan. And sometimes, like, there's a step in some paper that they say, oh, this should be obvious, but then you look at it and it's like, okay, I don't get it. And so we knew that there was a lot of content there that you end up producing while

trying to understand a paper. And we wanted to bring that online. Because, Luis, you were in physics before. I studied physics together with Mika and Joanne and Timer went to MIT. Timor studied economics and you studied CS. Yeah. So a lot of the papers are around physics, math, economics, biology, CS, right? Yeah.

Yeah. Because that was, you kind of like solved the cold start by just annotating yourself. Exactly. Right? And now it's more about getting the author in there. Exactly. That was the kind of the growth act. Our first paper was the Bitcoin paper.

Yep. And still the most commented, right? Yeah, that one has a good number of comments. It has been there for the longest and it is, it was quoted and or just there are a bunch of news sites that have pointed back to it. Oh, okay. It's like, okay, if you want to read it, go to the annotated version. But we had a few, like, cool people comment there. Lawrence Laszig.

Comment on the Bitcoin paper. A bunch of people from the Bitcoin community. Exactly. Yeah. But the larger goal with Vermont is to try to move things in the right direction, meaning move science towards what people call open science. And so that encompasses the number of things from open data, which means just sharing the data that you've used for publishing,

whatever research you might be publishing, And you want to share that and make that easily accessible to people so that if they want to replicate the results that you got or use it in their own research, they have an easy time doing that. So that's open data. You also have just publishing the code that you used or the algorithms that you've used and making those more easily available to people. There's also open publishing, which means just publishing in papers that are not behind or in journals that are not behind paywalls. So there's a lot of things that are within open science, all of those. And then there's also, so we want to push things in that direction and also try to build a platform that makes it easier for people to collaborate. And we think there are a lot of things that could be happening nowadays where people could be collaborate, scientists could be collaborating remotely a lot more than they are.

or that's at least the way we think. But it's starting to change where we've had for the paper, the Erdos. Yeah. I think this is actually a trend. We're seeing more and more people collaborating online around papers. So, for instance, there's this famous example around a problem called the Erdos discrepancy. And this problem is a famous problem that was posed by Paul Erdos, which is like this famous mathematician, 80 years. years ago. And Terence Tao, the Fields Medalist, was trying to solve the problem. And he put

it on his blog that he was trying a certain approach to solve the problem. And then there was this guy from Germany that just wrote a comment there, like the size of a tweet. And he said that the Erdosh problem had a Sudoku-like flavor. And that's some of the machinery that they were using to solve the Sudoku problem could be used there. And that was actually the key to correct the problem. And they ended up publishing a solution to the Yardos, the screpancy problem, which was probably one of the biggest milestones in number theory in 2016. And that was all thanks to a comment on his blog and to the fact that they were collaborating online around solving that problem, which is also a polymath problem. The polymath project was a project started by these

other fields medalists called Tim Gowers. And they were trying to, uh, It was actually a social experiment to see if it was possible to solve math problems online and, you know, collaborating around math problems online. And yeah, and they were able to solve it. And thanks to that comment. Because you kind of see, right, you look at GitHub and then you think of the impact that GitHub has had for open source. Open source, of course, existed much before GitHub. But it has really allowed a lot more people to come in and be able to get into open source.

or open source and start contributing. And there are a number of other really interesting platforms. You have Wikipedia just for more general knowledge or you have Stack Overflow for just programmers helping each other. And we think that there could be something similar to that, but for science in general. Right. Well, because did you listen to the Rogan with Peter Attia?

Parts of it. Mika listened to that. Yeah. Yeah, that was a really good one. And he talks about, I don't know if they're talking about the archive in particular around publishing papers, but he talks about having full-time staff. Oh, yeah. Just scrubbing the data looking for interesting information coming out. And again, like in the context of Stack Overflow, that's the place where like programmers find specific answers to problems.

Whereas with the archive, like, good luck. Yeah. Good luck finding that stuff. Yeah. And so have you guys thought about addressing like, just discoverability in the context of particular fields? It's a really tough problem. Yeah.

Like, for instance, paper recommendations, it's really hard to. Because you're just doing one a week right now. Yeah. In addition to the browser extension. And we also have our tool that is used internally at universities and research groups for people that they're reading papers together and they add annotations. Yeah. But for now,

we have the weekly journal so we release a paper every week that we select and we annotate it or somebody in the community annotates it and then we have the archive extension that adds a bunch of features on top of archive like

like BibTech extraction reference extraction and comments and eventually definitely like recommendation and recommendation engine or making it easier to discover papers that are relevant to you. That's something we definitely want to add onto our archive extension.

But it's a tough problem. It is. Yeah. Initially, we started formats as a, as John said, as a journal club. And then we saw that,

you know, people like the interface, the commenting interface and liked reading the annotation. So now we are starting to expand and turn formats into more of a platform. And that's why we, we decided to do the archive Chrome extension. Because archive, for people that don't know what it is, it's basically a place where papers leave before they go to journals in the form of preprints. So there are like drafts before they go to journals.

And what we did is we built a Chrome extension that basically allows people to see all the commenting interface on archive papers. And so you don't have to go to another website. You're just reading archive papers and you see the comments on the site. if you have the Chrome extension installed. Well, and a lot of these papers don't even have comments on the page. Like, best case, you're emailing the author? Exactly. Yeah.

Yeah. They don't have. Okay. So what archive does, it's basically they just host papers. Yeah. That's the core functionality of archive. And so one of the things that we noticed is that especially for areas like machine learning and deep learning archive is super important because the papers are, the new papers are coming,

are coming out at such a high rate that people don't wait before the papers go to journals before they start working on top of it and using the stuff that other people discover. So all the papers are published on archive. And so you need a way to distinguish good quality work from bad work if you are reading a paper on archive that hasn't been peer reviewed or something about machine learning. and I think that's why the librarian extension is so important in fields such as machine learning. So does the librarian extension have a rating mechanism as well? Like how do you distinguish good from bed work?

Right now it's only through the comments. But we are actually thinking about implementing some sort of rating system for papers. And we're probably going to also, we've been thinking about that for a while now. And it's not, we're probably going to run a few surveys to our audience. to, because you could do it in a number of ways, like rating a paper, you could do it. Obviously, there's likes or dislikes or up votes and downvotes. So you could either just have an holistic rating for the whole paper. You could also imagine rating it on a number of different aspects of the paper.

It could be about, okay, how big is their data set if they're using some data set or what do you think about their methods? So you could have a more complex rating system. And so we've been thinking about that a lot and we're just trying to figure out what makes the most sense there. But that's also definitely, like we would love to add that to archive or to the Chrome extension for our code.

Yeah. So how do you think the collaboration plays out then? Because I understand how, you know, say for instance, you know, you're a physicist, you start commenting on someone else's paper, you start a discussion that creates a new project, right? Do you think you'll go further than there?

Like are we, do you talking about like forking and that kind of stuff? Yeah, that's, I think you could, there's, there's a lot of things that you could do if you, once you have a platform that is more, that has more people in it and that they're doing more stuff in it. And so that's why the way you've been growing format is with a goal far in the future where we are a much broader platform. And so right now, but right now we were focused. mostly on solving problems that people have nowadays. And actually we were largely inspired for our archive extension by the survey that the archive guys did where they had, I don't know how many people, but they survey the people that use archive and then published a paper where they describe the problems that those people

reported while using archive and the things that they most wanted to see, the features that they most wanted to see. And then the archive folks just said, hey, we're just going to, we're going to be the platform to build upon. And we're not going to do all of these things that people would like us to do. But here it is, this is what people want to see. If there's anybody else that wants to work on this, here are the results of the survey. And since then, they've actually done a pretty great job of, like, building an API and wanting to become more of a platform. And so there's a lot of ways that we envision that you could have.

collaboration around science. And so, yeah, like forking, like forking a paper or forking some type of research. Or data. Exactly. Or data. There's a lot of things that you could do there. It's not something that we're focused on right now. Right now we're just trying to solve these problems that people have pointed out and

create a place where people can just post comments and discuss around a paper. An example of the problems that people mentioned was like, for instance, reference extraction. So if you go to a PDF, you have at the bottom of the paper, you have the references that they used. And most of the times when people want to search the references, they have to copy the text in the PDF, put it on Google and try to find the link to the paper. And one of the things that we did with our Chrome extension is we allow that. They just click on a button in the Chrome extension and then they see a list of references with links to the papers. So that was one of the the features that was most requested by the archive users.

And our idea was initially we wanted really to convince people to install the Chrome extension. And so let's solve the hair on fire problems that they are describing here. And then once we have people using the Chrome extension and then we can expand into like open collaboration around papers since they're already there. Yeah. So that was the growth. Do you guys know of anyone working on publishing negative results? this is something I've been fascinated with. And like basically the problem is that like as an academic, you're not incentivized

to publish negative results because you want to publish things that have high impact so you can get a job or a tenure position or just get people to even care about your work. Right. Yeah. So they don't publish. Do you know anyone like working on that? And I know researchers that are they're studying that field a lot. But unfortunately for some of these things, you just, that's, that's the very. very large problem and people are becoming more aware of that and with that you

like you have negative results you also have like people doing a lot of research into like p-value hacking yeah yeah so so p-value it's essentially um a standard that people use in order to know if the results that you have obtained you have obtained out of some experiment that you've run are worthy of being published and um and so so and that is worked for the most part that has worked fine until now, but, or I mean, that's arguable, but, but people are looking into it and, and, and thinking, okay, should we do things differently and should we be much more stricter with what's considered the golden standard to publishing? And we've thought of doing things there with, with Vermont, really, just just so that if you're

looking at a paper to have an idea, okay, how relevant is this paper. This is more specific for certain areas, like if you're talking about medicine or biology, where that is really important, like the statistical significance of the results that you're presenting. That's all, right? That's the most important thing. And so we've thought of doing something with Vermont there, either via some API where you could, like, send us the DOI of a paper and we would send you like some information regarding the B value or something or with the Chrome extension where, um, where you'd see that that information displayed very prominently saying, hey, like, these, this might be, there might be some peak value hacking year or this, or this is very solid research. Because there is a very big problem and people are realizing how prevalent it is, especially in things like economics and biology, biology, nutrition. I mean, it came about, I was just talking to a friend who's doing a PhD at Cambridge in bio.

That's a big thing. Yeah. And only by attending a conference in the States, did he realize that there was someone in Australia working on the exact same problem as him concurrently. And they're failing at the same types of experiments, but because they don't publish them. Like, no one knows the results. No one knows the methods. And essentially, like, these, you know, traveling salesman type problems that people are so excited

about quantum for, like trying all these permutations, are how. happening at a smaller scale, but no one's publishing anything. So the progress isn't happening. Yeah. And part of it is just the way research is done and you come into it and you're trying to find some correlation usually. Yeah. You will be trying to find some trend in the data.

And you, and you, whether you, you know, you, you are going to usually have that bias. You're trying to find some correlation in publishing that. And so, yeah, you might need to change things. dramatically in order to get people to start publishing negative results, which are like, could be incredibly useful for other researchers. Yeah. But there are a bunch of people working on that. There's this researcher at Stanford.

I'm forgetting his name. It's John. And then I forget his last name. But he actually just went on this podcast, Econ Talk. Oh, really? I love Econ Talk. Yeah. So you should listen to that podcast.

And actually, Timor has been talking. to that professor. I think he's a professor at Stanford. And he, yeah, and he is analyzed more this subject, but more relating to economics, I believe. But yeah, he's found a lot of the things that we're talking about here. They're prevalent also in economics. Cool.

Let's go into the Twitter questions. So we have a ton of questions. You guys are very popular on Twitter. So congrats on your great following. Let's see. Let's start with something broad. Tanner Goblinstein asks, what are the most interesting papers you read in the past couple of years that are not widely known?

That's interesting. I end up reading all sorts of papers from different areas. How do you get the papers, actually? It's just like a random walk. Really? Yeah. It's a random walk. It's funny.

Yeah. Or sometimes you'll think. For instance, a few months ago, I got like a Fitbit to track, like my sleep. And so I wanted to read papers about sleep. And so that just got me into like a random walk around like research around sleep. And then I found a bunch of interesting things. I ended up annotating a paper about a big study in Finland that was done in regards to the association between sleep and mortality. There are a bunch of really interesting things that I learned from there, for instance, that, like, if you sleep less than seven hours, that's associated with higher mortality.

But if you sleep more than eight hours, that is also associated with higher mortality. Really? Yeah. So have you changed your life based on that? Yeah. No, I try. Well, not that I was usually more on the end of not sleeping enough. But there was also another thing from that research that apparently sleep quality doesn't matter as much, at least for mortality.

Which is kind of counterintuitive, but it seems that just sleep quality is very closely related to the amount of sleep that you're getting. Okay. So like seven hours of like okay sleep versus seven hours of great sleep. That's kind of hard to distinguish. Seriously? So you could like sleep on an airplane your whole life and live as long? Yeah. Yeah.

Apparently. Maybe your life will be a little bit more miserable. But so it's hard sometimes to pick the favorites. But there is one, for instance, there's one that is also kind of random, but the paper published in the 90s about the Simpsons paradox and the hot hand phenomena in basketball. So the hot hand phenomenon in basketball is, right, you think that, okay, because they just made, they just made a field goal, like the next one, they have a higher chance of making it. And so there's this researcher that in the 90s looked at a data set from the Celtics to see if for free throws, if that was true. And so before they had asked students at Stanford and Cornell, like 100 students, if they thought that, okay, if they just made the first free throw, for the second one, are they higher? Did they have a higher chance of making it or not?

And there was something like 68 of the 100 students that were asked at agreed. And they thought that that was true. And these are like people from Stanford and Cornell. And so then they looked at this. And so what they found back in the 90s, what they found was that actually that seemed not to be the case, right? That from your second free throw is not, you're not more likely to make it. if you made the first one. But what they found is that you're just more likely to make it on your second one.

Objectively. Significantly. Yeah. Okay. And so this was done in the 90s with like, I don't know how many free throws, but maybe like 5,000. They looked at some data from the Celtics. Just across the Celtics.

Yeah. And then I went and got a data set from Cagle with like six under a thousand free throws. Free throw shots? And I re-ran the same, right, re-ran the same algorithms that they ran for the study in the 90s and then looked at what the results were. And yeah, and so the pattern is pretty clear that just on their second free throw, they're just much better at it significantly, regardless of their first one. And yeah, it doesn't matter as much that it doesn't matter if they made their first one or

if they missed. Yeah. Yeah. And then that paper kind of then tried to explain why people think that there is a hot hand phenomena, and that is related to the Simpsons paradox, which for people that are not that don't know what the Simpsons paradox is, it's also really kind of changed my worldview a little bit once I learned more about the Simpsons paradox. But it's basically what it says is that you can get two valid conclusions out of the

same data depending on how you split it. So an example, an example is for, is that between 2000 and like 2013, the average or the median wage for high school dropouts in the U.S. is dropped. For high school graduates, it also dropped. For people with an undergrad degree, it dropped. And for people with a graduate degree or higher, it also dropped. So across the board, for all of those segments, the median wage dropped. but in aggregates it went up.

And so you look at it and it's like, okay, what's going on here? And it turns out is that what happened is that a lot more people got a degree. So they just shifted towards higher education. So that's why you get on average it going up. And then for each one of these segments, it goes down. And so the Simpsons paradox is that depending on how you cut the data, you might get different results. But in this case, it's pretty easy to understand that you should be like what the right way.

What's the right way to look at this data? But in some other case, it's not clear whether or not you should include this variable and cut the data in some different way. And so relating it back, like for this basketball issue, what it was is that if you looked, the results were different whether you looked on a player by player or if you looked at the Once you collapse it all into the same table, you get different results rather than when you looked at it player by player. And so, yeah, if you collapse it, I think I forget exactly the way it went, but if you collapse, it might have been that you indeed saw. You didn't see the hot hand phenomenon, but if you looked at it player by player, you saw it.

And so they're arguing that that's why people add the idea. That's why you get like 68 students out of 100 saying that they believe in. the hot hand phenomenon. Yeah, yeah, yeah. And so, yeah, so some of the papers, like, that's really random. It's just like, it's funny. You're getting these just like little tidbits of trivia out of it all. Yeah.

Is it, has it been relevant to you in terms of physics? I mean, you're basically, you're working on software now, right? Yeah. But I, yeah, I also end up discovering really cool physics papers. So, for instance, my two favorite papers are actually, they were written by Freeman Dyson. One of them is when he proposed the concept of a Dyson sphere, it's just one page. And he basically explained an advanced civilization would need more energy than the energy that we can generate on Earth.

So we would have to go to a star and build a cap around the star to extract the energy of a star. But it's funny because it's like with really simple math and physics equations, he was able to derive, okay, is these spheres? stable, is it going to eat indefinitely? And so it's a really interesting paper. And the other one that I really like is, is one about Feynman's derivation of Schrodinger equation and also written by Freeman Dyson. And it just shows, you know, Feynman's intuition about quantum mechanics. And it's also really simple and easy to read, even if you don't have a physics background. But one of the things that I noticed from, from like trying to find papers and annotating all these papers was that, you know, in the 60s and and all like through the 20th century, all these discoveries and all

these papers were mostly like one, two pages. And yeah, like the, it's, it's so funny. And also fairly simple to read. But the discovery of the neutron, it's like maybe one column just, the discovery. of the Positron, the Dyson Sphere paper. They're really, really short papers and fairly accessible. Why do you think they've gotten so long? Is it sort of like, you know, David Foster Wallace citing a million things because he doesn't have confidence? I think it's also a consequence of a field developing.

Yeah. You just have, you know, more complex questions. And so it's harder to write. And they're also a little bit more detailed as to the methodology and the format of papers has gotten a little bit more formal in that sense where people follow us a very specific format. And I think that has added on to it. But yeah, nowadays they tend like the gravitation wave that we annotated. That's relatively, that's what, like 15 pages. Maybe.

It would be interesting to analyze like the constraints in terms of size that the journals were imposing. Yeah. Like 50 or 60 years ago compared to what they're doing now. If they are like forcing people to write, they were forcing people to write shorter pages. Shorter papers back then. Not sure. But, but I mean like if the discovery of the depositron paper was published today, I bet it wouldn't be just a single column. Yeah.

Well, are they intended to be more reproducible now? Good question. Maybe. Maybe. Yeah, I think Or maybe it's just more complex problems that they are tackling now. Yeah. It might be,

It might be the case. Yeah. Yeah. It's definitely not going back. It seems like you don't really see a trend anywhere of shorter papers. But yeah, it's interesting. Yeah, you go back to the 60s and 50s. And it's pretty nuts.

Man, glory is. Yeah. All right, cool. So let's go to another question. Polaris 7 asks. what are the necessary ingredients in a good and impactful science writing? This is also a good question. I don't think that I'm qualified to do or like I haven't published that many papers to to know that.

But one of the things that we noticed, or at least I noticed from reading papers is that sometimes it's not like the discovery paper that is the most impactful paper. So for instance, I just remember when quantum electrodynamics was discovered, there were three guys working on that problem. So Feynman, Schwinger, and Tomonaga, and they were sort of working independently on that problem and publishing papers on quantum electrodynamics. And the most impactful paper was actually published by Freeman Dyson, who at the time took, took, the time to analyze all the work and kind of unified the work of Feynman, Tomonaga and Schwinger, wrote a paper that helped other researchers understand what quantum electrodynamics was back then and helped really spread their work. So it was actually the most impactful paper.

So in other words, yeah, clear writing. Exactly. Yeah. clear writing. It's also, I mean, the question here is impactful scientific writing. And so you have, of course, writing papers. And then you also have just scientific writing in the sense of making some concept more, explaining that to a general, more general audience.

And so I think there, there's also, yeah, it's also the same where you want to make it clear and you want to make it accessible. But for instance, even like something like the Bitcoin paper where it is like, And I mean, I studied photography in college and even like it took me a few reads through it to to actually get it. And it's a beautiful paper, but it's definitely not, it's a very Spartan language. And you want to read every sentence. And so it can be very challenging to approach it. And I think definitely it all, you always benefit if you can make it as clear and accessible as possible.

because you never know the audience that is going to be end up reading your paper you of course you can expect that the people in your field are going to read it but sometimes things can can be useful especially like interactions between math and physics

things can be useful in different fields and so I think it's always beneficial for science if you try to make it as accessible what does impact mean yeah that's a question as well yeah did you see that one from Adam.

Adam Baybutt asks the basically metrics for value ad. Exactly. What does impact mean? If it's the number of citations that you get or just the number of people that, you know, learn about a certain subject because of a paper.

So in that way, a review paper can have a really big impact compared to a discovery paper. And so it's one of the problems that we also think about a lot. these metrics and what are the incentives in science and what makes people, you know, want to publish a paper or, you know, why should people worry about clarifying a paper and making it understandable to as many people as possible? Do they have the incentives to do that? How can you create incentives to do that? Right.

And then sometimes, you know, if you're just, if the metric is just a number of citations, sometimes it's not aligned. to making the paper understandable and comprehensible to a large audience. I mean, is that a question that you guys have to tackle? Because, you know, on one hand, you want to illuminate these papers that people could potentially learn from. But then on the other hand, you're running a site with content, right? And you want things that are going to capture attention. So I saw you have the Charlie Munger posts on there, right?

Nika annotated the Charlie Munger paper. Okay. Our other co-founder. Yeah, yeah, yeah. So it's like squarely non-technical paper, but Charlie Munger has millions of fans across the world. Exactly. Right. So you kind of have to balance those two things.

Yeah. And yeah, it's not easy. And citations are definitely a proxy, right? If the paper is getting cited a lot, it has some sort of importance. But it's definitely not perfect. And if you look at the most cited papers in these different fields, you might be surprised that there might not be the ones that you expect it to be. I certainly remember. I remember looking at, like, most cited papers in computer science.

And they're definitely very impactful, but you might have, some of them, I remember reading through those 10 and some of them I had never heard about before. And so, yeah, and sometimes very important, well, this is more specific for certain fields, very important concepts or discoveries never really get published in one paper that then gets a ton of citations. or that knowledge gets spread in some other way. And so there are, yeah, citations are not perfect. But I wouldn't say that we have a great answer for that.

What's a better proxy and how you should go about it. And I don't think anybody really right now has a better answer to, or not that we've heard about. But, yeah, it's an interesting problem. We'll see what people start using. in the future. Because, yeah, you could measure impact or how many, how many people are talking about it on social media. Many blog posts are reading about this paper.

Or if you have code, you know, if you have a public repo, how many forks do you have on your repo? Yeah, or like for certain. And then it depends on field by field, right? So if you take bio, then bio papers or can have a very direct, can be used very directly, say, in an industry, right? You can publish a paper about a drug and then that can be used worldwide and save lives. So for that field, maybe you can, there are a bunch of other metrics that you could use there to calculate the impact of a paper.

But for the more traditional science, like physics and math, sorry. Yeah, that's, it's hard. Okay. Question up top. Arcelon Yarvesi asks, it's basically about working in public and in the speed of publishing. They say, since scientific papers usually go through scrutiny and evaluation before getting published, how do you cope with not being always updated and up to speed in a world with daily news and contributions? It's kind of relates to what we were talking about before in relation to people publishing to the archive before.

before they really test it out. Where do you guys fall in that dynamic of like publishing as soon as possible, like with something like machine learning where things are just getting put out all the time versus going through a peer review for getting something out? And this kind of loops into peer review, which is the whole world onto itself that people are talking a lot about. For us generally, or say for a weekly journal, we generally are not publishing the most recent research.

And there is definitely, like sometimes there's a lot of us having to get to catch up to even, I remember annotating a paper about like this machine learning algorithm to play one-on-one poker. And this was like out of my league. I had to go like spend a good amount of time there researching it. And also figuring out, okay, how relevant is this? I also don't because, you know, I'm not in the field, so it's hard for me to gauge, okay, what's the impact of this paper? So, yeah, there's, there's, sometimes it takes us a lot of, a lot of reading up before we can actually say, okay, this is worth, it's worth publicizing and making and avving our audience or it's words, our stamp of approval and say, hey, you should read this. I think you, you like it.

It can take a while sometimes. But in the future, like, looping back to peer review, that's also something that I think the system nowadays does not seem to be perfect for the way things work nowadays. And we would love to see either via Vermont or some other platform to try to tackle that and try to do something to make peer review. a better system or to change it significantly. I think there's a lot of work left to be done there, which can have a very significant impact in science, right? That's part of like the most, one of the most important aspects of science is just, okay,

having a very skeptical mindset, looking at it with a very critical eye, and seeing, okay, is this, is this something that we can build upon? Is this something that we're going to add to our foundations to build more science upon this? And so that's a very important. an aspect of science. And I think it's, um, it, it's not perfect and could be better.

So Anvil Rotterdam asks, have you ever thought about building a tool for annotating books? Something like what Patrick Collison was talking about in the thread where he basically says, I'd pay a lot more for books if I could see the highlights, annotations and marginalia of friends or people I follow. Yeah. It's, it's, I think it's actually a really, really good question. And we have a friend, Jess Riddle, from the Prelimatory Institute, is a researcher there that writes about this, wrote about this on his blog. And I think that besides annotating academic papers, it also makes total sense to annotate books. And especially kind of introductory books, introductory books about science. and he gives this example of a book that is used by thousands of students to learn classical mechanics called Goldstein.

And there is a section on that book where they talk about this transformation called the Legendre Transform. And he does a bad job at explaining what it is. But apart from that section, the rest of the book is awesome. It's really nice if you want to learn classical mechanics. But if I want to write the book that does a better job at explaining the Legendre transformation, it has to be net better than the Goldstein book so that anyone will adopt that book. Otherwise, people just keep using the Goldstein book. So it would make sense for books to be annotated and also be open source so that in that sense,

you would just commit a new chapter, a new explanation. for that and keep all the other chapters and then just change that bit instead of having to write a new book and then convince people to adopt your book just because of that. So I think it makes total sense to do that more introductory. And we've thought about that, the type of things that you could do. If you add some platform where you could have books that kept being updated and you could have, okay, this is the standard for learning calculus where you just, you know, this is constantly being up to the idea. You're adding exercises to it. People are forking it. Like, if you need

more information about this, you're not understanding it. You could deep dive into it and you have a bunch of additional content that is attached to it really feels like something that should exist. And we've thought about it like, about doing something with Vermont for that. Yeah, it's just, it's so many things. But just in terms of copyright, are there massive issues there? Or is that possible? I think some of the, you might be facing some of the same challenges that Wikipedia is facing to an extent. Then, yeah, it's, it's, it would, I think it would depend a lot on the, on the format that is used.

I do think there's For something like this You'd probably benefit From having some editor Or like a team of editors To curate and to see Okay what like Should we add this?

Should we not to an extent To be some to be a curating voice In terms of copyright yeah You could run into some issues there Well some of these especially The classic books like On Electromagnety is more like They're out of copyright yeah

Yeah A lot of things are Yeah I mean my impression was that these are maybe even like current books coming out like popular fiction even as annotated by oh yes famous person um so i mean maybe if they gave away their notes for free and they were just the layer on top yeah but if you wanted to you know resell your own version of the book yeah that's interesting there's also some right there is some some legislation well there's fair use where you can use a piece of content if you're adding onto it or like right this is why you can you can

have like a video on YouTube with a snippet from a movie if you're reviewing it there's some some precedent there for for doing this type of thing but yeah but for more general books I also agree that that it'd be amazing because we were just talking about this we've talked about this for a while now but right because you read a book and the the purpose of that book is not only to, for you to absorb all the knowledge that is there, but it's also to get you thinking about what's being talked about in the book. And then you might reach some other conclusion. You might go on a tangent. And when you're reading it, that knowledge might never be shared with anybody else.

You might just read it yourself and you think, and okay, oh, this just made me think about something else. And it would be really like there's a lot of knowledge that is being lost. And it would be great if you could capture it in some way. The Amazon Kindle highlights site is one of the saddest things I've ever seen. Yeah. Have you ever done that? We have Kindles, but we haven't even. Oh, yeah.

So there's a whole web interface for looking at all of your highlights across all of your Kindle books. It's not good. So do you use it for anything? I mean, sometimes I go back. So like the best way that I've found for me personally to retain is to buy the audiobook and go through a book a couple times. and then my retention goes way up. But occasionally I'll be just like, what was that passage in, you know, whatever book? And I'll go back on to Amazon and you can like dig.

It's from Amazon. Yeah. And you can dig through your highlights from your Kindle. I think I've seen like a startup that does that in a better way. Kind of pulls all your highlights and organizes them. Yeah. I remember looking into this. But what I started doing is, well, if I'm ever, I also use.

And so that's, I don't do, I don't, usually don't, don't write annotations via Kindle some way or highlighting. I usually don't use it for that. But if I'm reading a physical book over the path, whereas before, maybe I would never write anything, now I try to like write a lot more there. Yeah. And then at some point, if I have time to try to go through, try to go through the books, see where I wrote things and then write that in some notebook.

Right. And because there is like just going through that exercise of looking what you highlighted can be very helpful. Yeah, I mean, I was an English major in college. So like I've forgotten more books than a lot of people ever read in college. And one of my professors actually recommended this, which is basically take a five by seven index card. And as you're reading the book, you're making little notes, right? you're like, all right, this character does this or like this is an important point. And then at the end, you basically write a paragraph to your future self describing your memories of the book and what happens and like important ideas.

And that can really like trigger it for you to retain. But past that like, I know. But I remember in school like in or back in Portugal, we all have to read this epic poem that is like, it's called the Luselish. And it was written by a poet back in the day. And it's about the Portuguese going from Portugal all the way to India. The Portuguese discoveries. The Portuguese discoveries. And so I remember we had a version.

You had the original version, which is pretty thick. And then we also had the version that had annotations on the side for each verse, not for all of them, but for a lot of them. And that made such a big difference, right? Because you're reading in all Portuguese, which by itself is already hard to tell. and he's using, he's making references that you have no clue about. So much historical context in every word almost. The names of all, like India was not called India. So everything is different.

And you're reading it through the first time you go, it sounds great. It rhymes. But you don't understand a lot of the context, the context behind it. And if you go through it and, okay, you read through it. And then on the side, you have all this rich. content that really only adds on it to your experience and makes it much more memorable. You can map it out in your mind and and create much more connections. It really enriches your experience. And of course, you have this because in this case, this is an epic poem that everybody has to read. And so there's a large incentive to publishing the annotative version of this book that is no longer under copyright. And so there you can have those type of things. But for for a lot of of more recent books, I think there would be, you could benefit a lot for man being that to some extent, right?

Where you can, if you want to, if you're reading through these few pages and you love what the author is talking about here, you want to dig deeper into this topic that he's talking about right now. There should be some place where you could do that. But yeah, it's just nobody has actually built this. I mean, I think that like defaults toward the blogosphere for most people. Yeah. They just like some people summarize books and like write Amazon reviews.

Yeah, yeah. But then the thing there is that, and sometimes that content does exist, but being able to find it easily, having that in your fingertips can make the whole difference. Right. Even if you, yeah, maybe you could spend like a minute like searching on Google and you'll find the content there you're looking for. But if it was right there, you could just click and it would pop up. And you'd see it, then it would be much more likely that you would end up reading that content. Those type of things make a big difference being right there.

Do you find that annotations sometimes are best done by someone who is not the author of a paper? What's interesting is that the authors of the paper sometimes, you know, they are not going to know where people are going to struggle understanding the paper oftentimes. times. I remember when I was annotating the Ethereum white paper, right, written by Vitaleek. I went through it and then I emailed them and it's super quick to reply. And you reply back with some of the questions that he gets the most about Ethereum. Makes sense. But when you're writing it, you have no clue for you. You've worked it out in your mind, some steps you might skip because you just have internalized.

them by so much. So you only get, you only know where people are going to struggle once you put it out there and you start getting questions. And so, yeah, so sometimes the authors are not the best. Every time we talk with an author,

I think it's easier for them to answer questions about their papers than to annotate the paper. But then if you have another person annotating a paper, I think it's easier for them because, but yeah, with the authors, we see that a lot. Yeah.

just ask me questions, I'll answer them. But sometimes I don't know how to enhance or add content to my own paper. Yeah. You guys could provide that service for sure. You could like reverse engineer clear papers. Yeah, yeah. It's kind of worth noting that this is a side project for you guys. Yeah.

How, I mean, I have so many questions about like how you go about building this thing. That's like definitely consuming a lot of your time. I mean, it has to. between finding reading papers making all those like graphics and tweets and stuff that you guys do how do you find that balance like what what's your whole philosophy around this yeah so yeah it definitely takes its time it is something that we we actively tried to do after college and while we were working while we are just before doing for my reading reading papers and staying up today It's something that we tried to do anyway.

And so we were already looking into research before as just something that we would enjoy. And then we found it good to have some sort of peer pressure amongst ourselves to present papers to each other. Because that really forces you to understand something well, right? I think it was fine. He has some quote where you don't understand something until you can explain it. Some freshman in college. Yeah, yeah. And so that is, that's very true. And so we tried to do that amongst each other. And then, and so then we got to Fermat and we thought, okay, maybe we can bring this, this online. And so we were already spending an healthy amount of time doing this type of stuff. But it is. But with Vermont, you have to, like the first version of Vermont, we kind of build it over the weekend and we tried to just make it, just put it out there as fast. as possible.

And then it's mostly like late at night. I'll be trying to fix bugs. People in Akron News don't seem to think that it's a side project. And they're pretty harsh on it. So, yeah, so there are definitely bugs. And sorry about that. We try to fix them when we have time. Yeah, but it definitely takes its time.

But I think it's also something that all of us really like doing. And I mean, I start looking at Wikipedia. articles about quantum computing and then I like spent three hours clicking on articles and articles and articles and then I found like five papers to annotate and I've produced like 10 or 15 tweets. So it's something that we really enjoy doing. Yeah. And so it's, it's, you know. I think that's the real genius of it, right?

It's like basically figuring out a way to turn your, I mean, if you have the desire, turn your, what would be your hobby anyway? Exactly. And having a forcing function. Because this type of thing is really easy to let go, right? Because sometimes, yeah, sometimes you might not feel like understanding a paper to the point where you could annotate it. They're like, it takes a while to get a good grip, especially if it's not an area that you're super familiar with. Of course.

And so it's not that, yeah, that's definitely not the type of effort that it just you do on a Saturday night, right, unless you add a forcing function that you know that, you know that, you. that within a couple of weeks you're going to be putting this to a lot of people. That's my favorite part of the podcast. With the software stuff, it's pretty easy for me to just like, it could be anyone in the room and we can do a podcast. But when we do physics ones or anything or math or something,

I'm just like, oh my God, you have to take a couple days, just reading. Yeah, yeah. I'm not, obviously I couldn't even become an expert if I dedicated a week to it,

but I want to be conversant to a certain extent. And that part's fun. Yeah. Same with us. Like you definitely feel the pressure. when you're writing these annotations. Yeah. Because people,

and people will call you up on it. And you'll be like, okay, this is wrong or you missed this. And so when you're writing it, you want to be really careful, make sure that what you're saying is correct. And you know that you might have some,

somebody that actually, a college kid or whoever, that is reading through that paper and then is going to use your annotation to help him understand. And so you have the responsibility. We feel that responsibility towards, those people to do a good job at it. And when we put an annotation, we want to stand by it and you want it to be of quality.

And it's funny, it's like the more you annotate a paper, this is like a circle. And the more you annotate a paper, there are more people there are that are at the edge of starting to understand what the paper is about. So you start getting more and more questions because the circle expands. And then you just have more people that are like starting to understand this topic about number theory or physics or whatever. So you get more and more questions about the paper. So it's like and then when do you stop explaining a certain concept? So it's like you want to annotate a paper about number theory.

Okay. Do you have to explain what a prime number is, for instance? Or do you have to explain what a rational number is? So it's really interesting once you start thinking about that, like how deep do you go? Yeah. Well, you got to be careful about those YouTube videos then, because if you get discovered on YouTube as an explainer series, good luck. People started asking you. Yeah, yeah, yeah, no.

We've done a few of those, but yeah. We've annotated a paper that it was, I think it was a proof of the irrationality of the square root of two. And then there was this, I think was 14-year-old kid from Russia. that because of that paper, he came out with an alternative proof for that. And he sent us that proof. And I read the proof, and it was apparently... It's legit? Yeah.

And I told him to submit that to a journal, a Matt journal. And I think he did it. I haven't heard back from him, but we should reach out to him to see if he actually was able to publish it. So it's also nice to see how we can inspire people sometimes to do these. of things. And I also think, especially with Twitter, one of the things that we learned is that learning something, learning a concept or learning a fact is really, really addictive. And we see that on Twitter almost every day. People come back and we have hundreds of thousands of users that read our tweets. And I think that's why people really like when they have a good teacher

and when they can go to a class and really learn something. I think the problem is that usually that requires a lot of effort from people. You either have to go to a class or you have to read a book to learn something. And I think what we're able to do with our Twitter account was to provide that same feeling, acquiring a quantum of knowledge, but at the cost of reading a tweet, which is really easy for the reader. Sometimes it's really hard to make those tweets. It requires a lot of reading and thinking, how can you explain something with just these characters and an image maybe.

But, you know, once you get to that and once you're able to teach someone a fact or something, people really like that. And I think it's something that, you know, there should be more people exploring that on Twitter. It's a very particular medium. But there's a lot of people that are. attracted by that. You might not, yeah, you might not, a few years ago, I would have been very surprised, but now you have all of these scientific, be it explainers, but you have people that have millions of followers.

And what they're following for is for scientific content, they just want to learn. Yeah. And so that, that's something very uplifting that we've learned, that there's a lot of people out there that want to learn. I think it's too easy to get down on those people. They were just like, oh, you know, this is like, like, base it's fun facts or whatever but like at the end of the day like that's good yeah people are excited to learn they want to learn and then you like extrapolate it out a little bit more and you look at

someone like dan carlin doing the hardcore history podcast like i think if you would objectively like written that down you're like all right i'm going to produce 25 hours of content about the cons and people are going to be into it i would have told you no fucking way and then you look at it and it's like millions and millions and millions of downloads yeah that's pretty cool. There's some things you look at it and it really catches you by surprise. I mean, this is parallel, but it's like Wikipedia, for instance. If somebody had pitched Wikipedia to me before Wikipedia existed, I would have never guessed that it would be possible. Yeah. Because, right, like, how are you going to do this? Like, no incentive, just people are going out of goodwill,

they're going to add content to it. And it's going to be good content, reliable, things that you can used to learn. And that's just, right, that's not something that you would initially think would fit with human nature. But people surprise you positively, right? And the same goes for like stack overflow. Like people just out of goodwill, they will go out and explain, you know, or try to help you solve your problems.

Like there's something to be said that like humans have like some, some untapped fountains of good will that we might not be leveraging as much as we could. You know, you see bright spots here and there and like Wikipedia or Stack Overflowers. Some projects that if you pitched them to me before they existed, I would be very skeptical that they would be able to get to the point that they are today. Of all the parallel universes, we are in the universe where Wikipedia exists. Exactly. There's got to be a lot of parallel universes where Wikipedia doesn't.

It didn't survive. Yeah. Yeah, well, I mean, it's like when you talk about you guys expanding, you almost don't have to over-engineer the incentive mechanism. You know, if you believe that is true. Yeah. Like, annotating more papers is objectively interesting. Exactly. Yeah, you, yeah, for sure.

We have people, I think, you know, we'll always have people that are going to be interested in consuming the content and reading. Then you have the other side. How do you create incentives for people to annotate the papers? I think that's a different, a different game. But yeah. Some things is just that it takes some time. And we are totally, when we started this, we knew that it would take time until people cared at all about what you were doing.

And then it takes even more time to make any sort of impact on the issues that we care about. But for a lot of these things, even say if you look at archive, archive was started and like, it's my age. It's like started August 9091. And it is taken a long time to get to where it is today. And if you look at the graph of submissions for archive, it's completely almost linear. Yeah. There's no startup exponential acoustic growth.

It's like completely linear. But it's arguably one of the things that has had the most or that has impacted the making of science or the distribution of science the most. But it just took a while to grow. And it seems like it's just going to keep growing linearly, but sometimes that's what you need. And so we are totally mindful of that. And we know that this might take a really long time until you can get to do what our ultimate vision is and to build that out. But, you know, some things, they just take some time. So do you feel, do you feel pressure to achieve like profitability or even like sustainability in the business?

Not at all. We never really thought about that. Yeah. Because also probably because this is a side project, we never really thought about monetizing or achieving profitability. It is like for some of these communities, you know, like Stack Overflow, it's a for-profit company. And I think it does a great job at what it does. And I'm probably happy that it is a for-profit company because they're just more independent. And if they have a good leadership that takes it in the right direction, it's great because they don't need donation to ask for donations to keep going. Wikipedia is a nonprofit and they've been doing great.

So it's possible to do it both ways. We've just, because we have very limited resources, we try to focus all of our attention in the areas that are the most important into what we're trying to achieve. So, right? So that means like we have to prioritize it. So meaning like our next step is going to be building the Chrome extension for archive versus doing anything else because we think that's what has the biggest impact. So that's why we never dealt into profitability and we just paid the costs ourselves. There's just server costs because we do all the work. So it's never something that that has been in our minds a lot.

and we think you could build these type of platforms either for-profit or non-profit. So, yeah, just something we will kind of defer it further down into the future. It's a good question. For instance, could have archive survived if they were a startup, for instance? Yeah, right. If they were for-profit. Yeah. Could they raise money with that kind of linear growth?

If they were not inside a university. Yeah, it's a good question. Yeah. I mean, plenty of companies without, you know, startup growth, raise money and become profitable or sustainable. Yeah. Right. But you're just like, okay, what are you going to charge for? Yeah.

People are care because, yeah, I mean, archive is great because it's open. Exactly. Right. And so many other journals may be dying out because they're not. Yeah. Absolutely. Yeah. One of the trends that we've also noticed is a lot of a lot of people building journals on top of archive.

And we are even collaborating with, we, um, with a few journals, one of them being the quantum journal, which is an overlay journal on top of archive on the quantum physics category. And what they do is basically, so what is a journal is just a list of links to papers in so they don't have any hosting costs. They just have a page where they just have the links to all the papers that they decided to publish and all the papers are on archive. So it's completely open.

And what we are, we, they, what we, our partnership with them is basically all the papers they have the, Fermat's library commenting interface. But we are seeing more and more of these journals popping up. So for instance, the Erdoch discrepancy solution was published on one of these open journals called discrete analysis. And I think it's, it's totally possible that these open journals get to a point where they, they have, you know, a reputation like science or nature. As long as you convince people to, you know, publish their papers on these journals, it's, it's, there's nothing about science or nature that, that, you know, is unique to them

and that prevents these, these open journals to get to, to, to that point. Yeah. Of course, it's also going to take time. Exactly. But, but, uh, but I think it's, it's totally possible. Yeah, exactly. It is, um, I mean, a lot of people talk about this, right? where you have journals that put content behind paywalls

and that kind of might have been funded with public funds. And so, right, there's that whole discussion about that. And it is a tricky system to get out of because it is sort of in a stable equilibrium to a sense, right? Because if you're a researcher, you need that publication in nature or whatever to get your postdoc position in a renowned university. And so you have incentives for, for the status quo to persist.

But there are a few ways that you could get out of it, right? As Louise was mentioning, one way is for these open journals to start gaining more reputation, right, so that if, okay, getting published in discrete analysis, it's a big deal. And it has a lot of reputation attached to it. And once that starts to happen, like, you get more and more people just putting it all out there on archive and publishing it all in open journals. The other ways that you could get out of the system would be for specific fields like what we were talking about

in machine learning where you have an incentive to publish as fast as possible because the field is just moving so quickly. And nowadays journals or big conferences might take years sometimes. Yeah, a lot of time. for submitting it until it actually gets out there, right? If you're submitting to NIPS or whatever for machine learning,

it takes a long time for it to actually be officially published. And so you also have that incentive that if it's open publication, you can move much faster. And so it is sort of a tricky equilibrium to get out of. And that's why these companies that make billions of dollars in revenue. And for instance, one way to get out of it was just, And I think it's one of the ways that they are probably, you know, it's probably the way to get this open, open journals to be as popular as nature or science is to convince really people that already have a tenure or really famous scientists to publish on those journals. You already have your position.

You already have your Fields Medal, your Nobel Prize, just publish on an open journal. That's what Terence Stahl did with the other discrepancy, and I think that's what other people are doing. And Tim Gowers, which is the – it's a field medalist also, this mathematician, which founded the discrete analysis Open Journal. And I think you wrote a blog post a while ago, and his mission was to convince famous mathematicians and people in these situations to publish on open journals. Yeah, because, right, for the young researcher that is trying to get a position and a – We're competitive field, then you need, right? Because if you want to get your postdoc in a renowned university, you need to have that. So that's what's keeping it alive.

So these big names endorsing the open journals. I think that's going to be the growth act to increase the reputations of these open journals. Absolutely. And it was interesting because it is a problem. And we definitely believe that that's the right direction. And while you're in the U.S., right? Like while I was studying at MIT, you don't even realize it because if you're within the MIT network, everything is open. Yeah.

Right. So you just, you're accessing it. And when I was an undergrad, I didn't even realize. In other words, if you're literally on the MIT Wi-Fi, you have access to these journals that are paywalled. And you don't even see, okay, this would be $30 if I was like five blocks down that way. But Louise was studying in Portugal. And so we would talk. And then you'd compare like even like even in Portugal where, right?

You have well-funded universities, but the research groups might not be able to afford all the journals. And so sometimes you just have a lot of trouble accessing research. And so this is not in the U.S. It is like big institutions have access to it. But like in a lot of other parts of the world, the fact that a lot of research is being published in non-open journals has a significant impact. Well, especially when like legit CS papers are written by people. who aren't associated with any university, right? They're just like hobbyists writing things.

Like, why would they have 100 journal subscriptions? Exactly. It's impossible. Yeah, I remember even like researchers in my other researchers in my research group. Sometimes they would have to go through CERN to get VPN through CERN to get access to these papers. Yeah. Or like I would have to email you and ask you to send me some PDFs. Yeah, yeah.

You're a good brother. Yeah, I contribute it. Five bucks. So if someone wants to contribute or help out, what can they do to help you guys? I think there are a few ways that you can help us out. You can annotate a paper on Vermont's Library. And so email us, team at Vermont's Library. Exactly.

If you want to annotate a paper there. Can spread the word. And if you're at a university, then we are. If you have a journal club. if you have a research group and you want to annotate papers and share them among your peers when you create an account on format

now you can also upload your own papers you have that option and then you can share with whoever you can create your own lists and so we have people at universities that use us already like or be it for classes and students have to read papers and so they

will post annotations on format or just within research group and they all decide to read a paper. So if you're at university, and if you want to use this, it's completely free, so you just need to sign up. Yeah.

Yeah, those are the two main ways that you can help us out. We're also taking cryptocurrency donations. So there's that. But really, like, most of our costs are just server costs so we don't have to pay salaries to anybody. So, yeah.

That's about it. That's the way to help us. Cool. All right. Thanks, guys. Thank you for having us. All right. Thanks for listening.

So as always, you can find the transcript and the video at blog. dot ycombinator.com. And if you have a second, it would be awesome to give us a rating and review wherever you find your podcast. See you next time.