The Shintaro Higashi Show - What Is Peer Review?

Episode Date: January 6, 2025

In this episode, Shintaro learns about peer review from Peter. They tackle common misconceptions, explain how peer review parallels qualifying rounds for Judo competitions, and discuss the strengths and flaws of the system. From navigating PubMed to addressing the challenges of reviewer anonymity, this episode offers an engaging and accessible take on an often misunderstood topic. Whether you're a researcher, a student, or just curious about how science works behind the scenes, this discussion will leave you with a clearer understanding of what makes peer-reviewed research tick. (00:00:00) Introduction (00:00:56) PubMed and Its Role (00:04:02) Understanding Peer Review on PubMed (00:05:41) Comparing Journals: The "Olympics" of Academia (00:08:41) How Peer Review Works: Submission to Acceptance (00:13:12) The Role of Peer Reviewers (00:16:02) Challenges of Reviewer Anonymity (00:21:34) The Debate Around Funding and Conflicts of Interest If you're in business, then you have customer churn. Whether you're building a startup, growing a mom & pop shop, or operating in a fortune 500 powerhouse, Hakuin.ai measures, predicts, and improves your customer retention. https://hakuin.ai

Alright, so PubMed, government day. Yeah, government day. Let's just say for instance, peak of COVID. Yeah. I'm a little bit skeptic about this thing. Let me go and research myself, do my own research as you know, people would say, right? But you know, some people do their own research and actually read these papers.

Yeah. Okay. If there was a paper that says, okay, you know, the vaccine is this and that and it talks negatively of what the government has an agenda of. They could exclude it, correct? Hello, everyone. Welcome back to the Shintaro Higashio with Peter Yu. We have a very interesting episode, something a little bit more academic.

Peter wants to do something that's not so, what's the word? Meathead-ish? No, no. I used. No, I think judo people, you know, there's a lot of like, you know, cerebral components to judo and I think this type of topic will help with your judo to acting. And then it was interesting because we're gonna talk about the peer review system, I think, because last time when we talked about PhD, some people asked in the comments like what the peer review system is

I did briefly comment Comment on that but not too much in detail But also interestingly you asked me you looked up a paper on PubMan And then you were asking how legit it is how to tell right? Yeah, and then uh, you know, so my initial thing, the way I came across this paper was estrogen conversion situation with,

it's like an aromatase inhibitor sort of a thing with Cialis, and whether or not Cialis can lower your estrogen levels, right, estradiol levels when you're taking to Sashone, right? So then I was looking at it and I was like, oh, look how many people have participated in this, it must be legit. And then I saw that it was like funded by the company that makes it.

I was like, this isn't legit, you know? And then what do I know? I don't really read, I read, you know, articles like this, but I don't really know what I'm talking about. I kind of read the abstract most of the time, skim the rest of it. I don't really care about like whether or not I kind of read the abstract most of the time skin the rest of it I don't really care about like whether or not the study was a shit or not. Yeah so you know so you tell me a little bit more about

this now what is this thing with the peer reviewed? So let's kind of yeah so we're gonna use that as an example to kind of walk it through I did look that up you did send that paper to me just for so that I because first of all the disclaimer is that I'm not in medical field. I just know a little bit about the medical research is conducted, which has a lot of parallels in my field too. A lot of the science. But you're qualified to talk about it because your wife's a dermatologist. Yeah, so that helps. So she's done medical research too, so we talk about that a lot. And I do AI research, which has a lot of parallels, you know, a lot of the same processes. We just play different sports. She's more of a, I don't know,

she's probably playing more of a, like, a sexy sport than I play judo kind of thing. Anyway. All right. All right. So, okay, let's go to start with my how he found it you went to PubMed right I actually googled it and it was on PubMed okay so I'll tell you I was like okay at least this thing is on PubMed it's legit so I thought first

so I'll give you a little bit of a PubMed so I actually looked it up PubMed is very used widely but it PubMed is more like Judo Inside. In the sense that, it's like you know how Judo Inside has competition results for everything. Basically, they just competition high quality competition to like local random ones too, right? Yeah. So PubMed is like that. Any... Well, Judo Sons have local whatever I mean yeah it's to a certain extent PubMed is not gonna have everything but it does have some it doesn't guarantee that every study published on PubMed is peer reviewed

okay okay so peer reviewed is another class in itself, so I'll go into that so but yeah So you can it's made pump that is maintained by your tax pay taxes It's a bill maintained by the government and they kind of gathers all the life sciences and biomedical research It's like judo inside for those things and everyone has access to everyone has access to because it's you pay you pay for it It's yeah, so if I'm like skeptical about certain types of vaccines I could go on PubMed and read some articles. Yeah, but now you have to know how to go through the results, right? So it's basically like asking like someone had a question, who is the best judo player right now? And the person goes to judo inside, it looks up all the competition results

yeah, probably going to be Teddy Renair Teddy Renair also, but maybe there's an athlete, a random athlete that just has more wins than Teddy Renair because that person just played in the local circuit or something just like, it's a convoluted example so that's the kind of thing, like you need, you would need someone who is very familiar with the judo competition scene to kind of interpret all these results on judo inside for you, right? I don't think so, man. Because you see like Teddy Vernier won three Olympics, this and that. Yeah.

And you see the results. And then you see that I took second at Liberty Bell one year. I'm in my pot. You know what I mean? But people don't necessarily know the difference between Liberty Bell and the Olympics. What? Of course they do. But that's the thing. Do you know the difference between different medical journals? The Olympics, dude? Every single person knows that the Olympics are more legit than the Liberty Bell Championships. So things like, for example, so that's a good point So in that sense that the good parallel will be like the lip the the New England Journal of Medicine That's like the Olympics right? So if you find a paper in pub, man Yeah, that's published on New England Journal. You know, that's fucking rigid legit, you know

that's a hardest journal to get into but fucking legit legit you know that's the hardest journal to get into but there are other journals that are very local like specific to the field like Liberable is a decent tournament in the US but you wouldn't know that unless you are in the American judo scene you know that's true that's kind of like the thing so first you go to PubMed you basically one quick way to verify if it's legit or not is to check where it's published right okay okay all right so PubMed government yeah government day let's just say for instance peak of COVID yeah I'm a little bit skeptic about the thing let me go and research myself do my own

research as you know people would say right But you know some people do their own research and actually read these papers Yeah, okay, if there was a paper that says okay You know the vaccine is this man it talks negatively of what? The government has an agenda of they could exclude it correct Well, it's who can exclude it. What do you mean PubMed not, not Publish. No, PubMed is not like that. It's a database. Everything gets on it. It's like Judo and stuff. Pretty much, you can kind of, for the practical, for all intents and purposes, you can kind of say there's no... So when they were making fun of ivermectin and stuff like that, like I could look up papers during that time period,

with ivermectin having favorable results, even though it's against what they want us to do. Yeah, if it was published in these journals and like, so here a good parallel would be that you're doing research in judo. For example, like who is the best, like who's the current like best American judo player in 73 or something like it's something like that and then you go to judo inside and look at Jack Inatsuka yeah well yeah you know because you're an expert but now like someone who doesn't know judo much is going to look through the judo inside results or like you want to compare which which is Jack Inatsuka better or Ohno Shohei better kind of like some question like that, right? Yeah

Say those two players have similar results. I'm not just saying Jack or Oh No Shohei, but some similar result then you want to see who's better You have to now go into the matches and see on you know If they played against each other in what settings you have to see the content of the match that's kind of like reading the papers and see and you can see that you just have to like have the base knowledge of judo to interpret how these matches have played out right all right but not every paper on PubMed is peer- no just like not all not it's played to me peer reviewed so the peer review

system is basically kind of like the basic qualifications mmm like a qualifying qualifying rounds for a competition or criteria no you know what I mean yeah so that means it doesn't necessarily mean that that's the absolute truth that you have to see that so just because someone entered the Olympics Doesn't mean that the person is like the best judoka. Let me ask you a question Yeah, you know I'm not getting my answers questions answered. Yeah, well I haven't even asked any questions, but I have questions in my head that aren't yeah

So someone could peer review it and say this paper's trash. So I'll yeah so I'll go into the... But it'll still get the stamp of approval it's peer review. That's where it gets into the details of how it's conducted right. So the okay let me kind of go step back and why we started having the peer review. You gotta be a little bit more faster because I am getting bored. I love you bro. Okay I'll go okay so let's start yeah the peer review just doesn't work like oh someone says no or someone says yes and then goes in. The peer review is for so in the competition there are like judges that kind of their rules and like qualification rounds that

basically allow people in or not right? All right so I'm interested in this theory I write a paper yeah okay I design a study I write a paper I just send it to PubMed or what happens? I said no where do I send it? So basically you can so you want to basically enter a competition. That means you want to go into the journal basically. So you send the paper to the journal. The journal or the conference has like deadlines, you know, recurring deadlines. Then the editor of the journal usually, editor decides if the paper gets in or not, right? But the editor doesn't have all the expertise because the now science is so specialized, just like sports have been very specialized, right?

Okay, yeah. So you asked peer reviewers who are, so the editor basically picks peer reviewers to judge the paper, to recommend or not. Usually a set of people. But what if I get a bunch of set of people that are just really dumb and they don't know what they're talking about? The editor picks the... The editor's job is to pick the qualified peer reviewers who are experts in the field.

He just choose? Yeah, so it basically goes on to... There's basically criteria for... Each journal has a criteria that kind of determines the prestige of the journal how rigorous the peer review system is Just like on a list of guys who can peer review this stuff. It depends on the journal and then everyone it's like depends it Depends on the field too. So for my field Because the pace is so fast

You'd up example, a conference, you need to have at least three papers published in this conference. You know what I mean? To become, to be selected as a peer reviewer. Oh, okay, okay, okay. All right, so you can't peer review because you have two papers published?

I can peer review because I have published multiple papers. I'm not the first author in all the papers, but I have published multiple papers into some top conference. So I can peer review. I thought you just did one for your proposal thing. I thought you said you have to do defend it and then you have to do three. So thesis defense is completely different from this system.

Yeah. So this is more of a... Show us how little I know. I mean, it's very opaque. Unless you're in it, it's really hard to navigate through. I learned this through trial and error throughout my tenure as a PhD student. You have multiple papers.

You could peer review someone's thing. Yeah, in my field that qualifies, that makes me qualified. Not all fields. But there's so many things that, so you're doing AI in video, what about if they do like an AI thing that's completely unrelated to your field, how would you know that the results are legit? I just based on that. I wouldn't be assigned that paper to review. The editor, in the conference they call the chairs, they pick and choose based on my profile. You should peer review this paper. So is it possible to like,

all right Peter, are you gonna review this paper? Is it possible to what? But you have to like sign up to be a peer reviewer? That's right. So a lot of times, so that's another question, how to encourage peer reviews because it's kind of like volunteer position. All right, so let's just say the paper that you're in, the journal that you're in for AI, okay? You are specialized in AI and video and there's a lot of, how many people are qualified to be a peer reviewer in your field in that journal?

I don't have the number but they're quite a few, right. I don't know. It's a hundred? More than a hundred of course. A thousand? More than a thousand. I'll say I mean it's a big field so yes that they're more than a thousand. I guess there's a lot of people who can review. All right so let's just say ten thousand. Sure yeah. I'm not sure. So there's ten thousand people who can peer review. Yeah, probably more than that. Okay, let's just say 50,000. Yeah. How many of those guys gonna sign up to be a peer reviewer? So, peer, being a peer reviewer, you can put that on your CV or your resume. That counts as a service. It doesn't count as much as the actual publishing. Is it like working the scoreboard at a local tournament?

Yeah, kind of like being a judge. Or like a rules judge. Referee commission. Oh, that's exactly, that's a great comparison. It doesn't count as much as your competition win, but it counts. Yeah, okay. And then some people argue they should count more. But that's another debate.

So how many people actually I don't have the number but they do get enough so that like each paper might the conferences I usually publish you usually get at least three reviews and then one way three people review your thing that's it three yeah there are three or more I mean it it's like know each other No, that's so most peer review systems are double blinded double blind. So I don't know who wrote it The author doesn't know who reviewed only the editor chance like all right, so let's just say right There's a paper that comes out it goes across your desk because you're in that field Yeah, and it disbues all the stuff that you've written about. Yeah that field and it disproves all the stuff that you've written about. You could just be like, that's garbage.

Of course, that's the downside of being double blind. Because you can say whatever you want because there's anonymity. But that's why you have multiple people. So it's like this, it's not like a bulletproof, like foolproof system. Of course it's not. Are the qualification criteria for Olympics foolproof? Of of course, it's not it's are the qualification criteria for Olympics pool foolproof. Of course not No, they're like but you have layers and layers of this protection so that you minimize the risk of that happening And if you keep giving this trash reviews you'll be you'll be banned from being a peer reviewer

Okay, so someone reviews your peer reviewer now So I get to review. Oh, so the author if I'm the peer reviewer One of the peer reviews the author gets to read it the judge the editor gets to read it All the other reviewers also get to read it, but they're all anonymized has there ever been a time It's like all right, you know judge look at this pure of course You can't even spell the words correctly These guys are moron get this thing out of here. Does that happen? Yes, I've written letters like that to editors saying hey, it's an anonymous letter letter I would like a secret letter you can most journals and conferences allow that and you basically say it's a kind of a joke

It's always the for some reason reviewerer 2 that's always giving the trash reviews It's like a joke and then of course you can flag the reviewer and then if you get flagged a lot And it's legitimate. Yeah, you get you the editor can choose to ban you from being a reviewer. So this is purely Pro bono situation where no one's getting paid. Yeah, so that's another issue. So some people, that's why it's a little sometimes hard to recruit reviewers. And because of that, people are thinking about, oh, maybe there should be an independent company

that pays peer reviewers and then kind of put a stamp of approval kind of thing. But also, it kind of introduces different dynamics dynamics so it's not a bad one. Ready for this one? Yeah. Leave it to me to fix something that doesn't need to be fixed. I'm not an expert in it. I'm open to ideas.

Randomize in a lottery system every now and then a pure reward gets $10,000. Oh just kind of like a lottery. It's like a incentivize, not by volume, not by doing, I get paid X amount, it doesn't affect the thing, but every now and then you get a thank you bonus for being part of the thing. Maybe that's it, I've never heard that. Maybe I'll like start telling people,

but proposing this idea. Leave it to me. Yeah. But again, so big, you know, how judo rules always change the qualification. People are always tricking the peer review system. But the idea is simple. Like there's so many research papers coming out and there's so few spots for journal publication. So how do we ensure?

Yes. So let's just say not your field specifically because there's no sample size really. But if you're doing medication stuff like we're talking about like Cialis, the number of sample size kind of matter, right? So like when you're a peer reviewer, can you say like, all right, first of all, they studied three guys who are out of shape. You know, this has nothing to do with anything. Like this shouldn't even be allowed because it's just like this.

Of course, that's a very valid criticism can you say hey the company who makes the product funded this thing this should this should be conflict of interest there so that's a little different because again it's double blind so the reviewers don't know that this was funded by someone. We don't even know their identity. Nothing about just the content. Right? So that question is up to the editor. And now just because a company funded the search study doesn't mean it's automatically invalid. Just like, just because like Judo Federation funders ono show a Doesn't invalidate the fact that I want to show is a good judo cup judo person all right You know not to keep interrupting you but I have a lot of questions. Yeah, yeah

You just said just because x y and z it doesn't mean but they're still a conflict Yeah, it's like you're spending hundreds of million dollars on research. Phase one, phase two, phase three. All right, we finally get it. Now we're ready to make money on this thing. Here's a paper to prove the efficacy of this thing. Data's kind of like not really doing what we want it to do. Let's just get rid of some of these data and then just put this one and then, you know, right?

There's no way for the peer review to know that the data's not fabricated, right? is fabricated so now the peer review doesn't it's just kind of like the qualifying rounds right? of course getting into a top journal is an accomplishment in itself like just like qualifying the Olympics is an accomplishment but once it's published now everything is public the identities of authors who funded them now the real shit happens the real competition happens people will dig at try to like tear this thing down left

and right someone will definitely mention that like oh this was funded by I love how you never fired up we're talking about grapple but now you're so fired up what's up with that Peter no I'm fired up I you never fired up when we're talking about grappling, but now you're so fired up. What's up with that Peter? No, I'm fired up. I'm especially fired up because I love finding these inner sections between my two interests, you know, I think it's cool. There are a lot of good parallels, but Just like you have to duke it out once you qualify for the Olympics. You have to actually have a match. That's when once it's accepted people start you qualify for the Olympics you have to actually have a match that's when once it's accepted people start you know criticizing it that's the real battle

like if you're gonna survive so let's just say this company funds its own research and says if you take this drug you'll be ten times stronger sure yeah obviously it's skeptical but then the research shows that it's you know methodologies is correct seems to be correct everything is correct you know peer review this is a good paper this is legit it gets out there gets published now all of a sudden people like I don't know about this data let me redo this study of course people do that okay we produce that's good to know that and then they could look at it and be like hey this company who's making this stuff funded this it seems fishy let me test this on

my own I'll explain people do explain to you why people do it it's like a goal mind for the... it's just like people doing research so, someone makes a crazy claim and it gets published in a top journal, right? but if you disprove that, you'll get so much reputation for that reputation? yeah, like you'll be like, oh my god god you're the guy who disproved this crazy theory oh so this is it you're basically like oh Terry Renear wipe the floor but you're the guy who beat Terry Renear

mmm tore it down okay it's kinda like that so what happened in your field recently that was like that someone disproved some big thing that was like wow I think chat gbt is kind of like that so there was a huge debate whether the approach taken by chat gbt was actually going to work that well a lot of debate a lot of division I was actually on the when I started my PhD I was on the side that I chat gbt would approach taken by chatapie wouldn't work this well.

But it did work. It did work. So what is the mechanics behind it that works? It's like a large language model, right? So I definitely want to dedicate an episode for that. But basically the debate was that the chat open AI theory was that if we give enough, a lot of data, train a huge model, it'll do crazy things like reasoning,

and just like, you know what chatropathy does, it's kind of mind blowing, right? All these crazy cognitive tasks. But on the other side, people are basically saying, you can't go to the moon by building a tall ladder. Like keep building a ladder. People argue that you need a fundamentally different approach to solve this problem.

There was a huge debate. But Chachypitty basically kind of pushed the discussion to the other side because it works so well. It works really good. I used it like the other day I was like I got some weird like AST results back I was like hey man I told Chajimod like you know the medications that I'm on do you think my medications to be causing these enzymes to be elevated and it was like you know it gave me a reason like a good answer.

I was like holy like, you know, it gave me a reason like a good answer. I was like, holy moly, you know. Can you even, the fact that it works so well, like because so that's LLM is like a very like close to my research. I use LLM like I know how like mathematically it works and that's my job to know but yeah I've and I read the whole history like what what language model is and then all to like, I know I can kind of give you the whole history too. The fact that it works so well, it's just still mind-boggling to me. It's wild, man. What about churn? Can you tell us about churn? Churn? Oh yeah, so when you have... When you have, hold on, I gotta find that.

I'm so bad. Jesus, Peter. when you have hold on I'm so Jesus Peter when you have when you're running a business right mom pop shot judo dojo you know you have customer churn and our friends company Hakuin.ai Hakuin yeah we can post the link but Hakuin can help you with customer churn thank you drew

thank you thank you Jason, Levan, JudoTV Higashi brand yeah thank you drew thank you Jason Levan judo TV he got you brand yeah thank you so much for all your support all right let's do another episode about judo now yeah yeah I mean so that I did did they answer a question on how to like it did I get what peer reviewed is now and I think it's more legit than I thought I honestly thought it was just a bunch Of guys in a room like y'all guys forget this guy. We like no guy. It was a lot more You know, I never thought of it like that and then just all the peer-reviewed journal now

I kind of get why it is but I see its flaws too. Like I know about it So thank you for teaching me that yeah, so it's it's all it's kind of like yeah basically How judo rules always change is the same thing. People are always tweaking. Because ultimately what we want to find out is good scientific results get published. If you guys are watching this, please comment

in the YouTube section whether you like these kinds of conversations. I mean, I enjoy them. But it's so far from what we're doing. So I definitely want to hear feedback from our listeners to see if this is something that we should keep pursuing Peter's very happy about it you know he almost quit on me so I want to talk about you know I wanted to talk about politics but Peter was like no we're gonna talk about academia am I all right oh yeah I don't

know know much about possible I do know more about academia I guess so I don't know much about possible. I do know more about academia I guess so I don't know much about politics. Is that right Peter? I mean not enough to like really comments on it authoritatively But yeah, so yeah, if you guys like it I mean I tried all I try I try to like kind of relate make it relatable and I think yeah There's a lot of value. So yeah, let me know if you guys, how you guys think and yeah, we'll see you guys in the next episode.