WHOOP Podcast - Study proves WHOOP is “excellent” at sleep tracking.

Episode Date: February 26, 2020

A new study by the University of Arizona has proven that WHOOP is outstanding at tracking sleep. Kristen Holmes and Emily Capodilupo talk about what this means and why WHOOP is excellent when compared to polysomnography, the gold-standard in sleep evaluation. Researchers also found that wearing WHOOP actually improves sleep habits. Kristen and Emily discuss setting a new bar for sleep tracking (2:10), how this study came to be (3:23), why they expected the results to be strong (6:03), how WHOOP is built to get the best data possible (8:29), how WHOOP is helping people get better sleep (11:48), what polysomnography is and why it's considered the gold-standard in sleep tracking (14:54), why sleep is so difficult to evaluate (18:04), and how rewarding the results are (23:05).Support the showFollow WHOOP: www.whoop.com Trial WHOOP for Free Instagram TikTok YouTube X Facebook LinkedIn Follow Will Ahmed: Instagram X LinkedIn Follow Kristen Holmes: Instagram LinkedIn Follow Emily Capodilupo: LinkedIn

Hello folks. Welcome to the WOOP podcast. I'm your host Will Ahmed, the founder and CEO of WOOP, and we are on a mission to unlock human performance. At Woop, we build technology across hardware and software and analytics that's designed to understand your body. And this podcast is really an opportunity for me to interview individuals who are high performing, professional athletes, coaches, trainers, high performing executives, you know, name it. And this is a special week because we just got third-party validation, the University of Arizona. We are now published in the Journal of Clinical Sleep Medicine. And it shows that WOOP is the most accurate, non-invasive sleep monitor on the market. Pretty excited about that.

You know, we founded Woop in 2012. We always knew that accuracy was going to be pretty core to what we were trying to deliver. And this validation for us is proof that it was worth investing all that time and capital and energy to make our product more accurate than other products. We gave up a lot in order to be more accurate. And so this episode between Kristen Holmes and Emily Capitaluppo, they're going to talk about what a major milestone this was for us at Woop, and they're going to talk in great detail about the study conducted by the University of Arizona. Effectively, researchers tested the whoop accuracy against the gold standard in sleep tracking. That's a test known as polysumography, and found that whoop was excellent by comparison.

It also found that whoop can actually improve sleep habits. So for those of you interested in your sleep or about sleep validation in general, I think you're going to get a lot out of this podcast. Kristen and Emily certainly deserve a lot of credit. They've poured years of time and energy and hard work into this research. So we're super thrilled with these results. And without further ado, here are Kristen and Emily. Hi, Whoop crew. Kristen Holmes, VP of Performance Science here at Woop.

If you didn't know it already, we care a lot about sleep. Sleep is the key to recovery and one of the best things people can do to improve their mental and physical health. It's vital that wearable devices are as accurate as possible, considering the immense value of sleep and the widespread adoption of tools to track sleep. The bar should be high. It is kind of a historic moment here at WOOP. In a recent study conducted by the University of Arizona Health Sciences Center for Sleep and circadian sciences show that Woop is, well, really good at sleep. The results, which were recently published in the Journal of Clinical Sleep Medicine,

validate our sleep accuracy while also demonstrating the power of Woop to change behavior. here to elucidate further is the architect of our sleep algorithms vice president of data science and research emily capitolupo hey christian hi emily this is really a watershed moment for whoop emily you've been at whoop for seven years as the first employee so i don't think people recognize the significance of that you've been here for you know just since the beginning and you have been grinding away in the most rigorous way imaginable to ensure the accuracy of our sleep. Talk a little bit about the process to kind of get to a point where we're in a position

to enter into this validation process. Sure. So I think one thing that's, you know, maybe our users don't fully understand is just how much goes into a validation study. So we actually started working with the University of Arizona on this particular study three years ago. Gosh, yes. It's been a long time.

It's been a really long time. We didn't start the actual data collection, obviously three years ago, right? So for the study itself, so just going back three years, first you kind of have to form that partnership. You go through like a legal process, including something called getting approval by the Institutional Review Board or IRB. Because testing, whoop requires testing on human subjects, even though whoop is completely safe, they're still, you know, really, really important legal protections for research. done on humans and so there's a lot of review and kind of back and forth and just process that goes so it takes months so that was you know before you even get permission to start doing the research then you have to recruit subjects in this particular study we recruited 34 subjects 32 of whom

ended up completing the protocol which was a 14-day protocol and then all the data needs to be analyzed and then you know sort of written up and so you create like a manuscript which is like the, call it like a first draft of the publication. And then you go and you try and kind of basically shop that around to journals. And so different journals are kind of looking for different things, different topics. And so you have to find a journal who's interested in publishing your paper. And then you go through a process called peer review, which basically includes they send out your manuscript typically to two other authors who are in your similar space and, you know, kind of experts in the field. And they give you back feedback. You have to respond to the

feedback and you kind of keep going until everybody signs off that, you know, the paper as written is good science and should go out into the world. And then it has to get published. And so from, you know, when the study is completed until it exists in the world, which just happened for us about a week ago now, is months. And so in order to get this paper, which it might just seem like, you know, a couple of pages of validation, it was three years of process in the making, but I actually want to go back even further than that. So how did we even get to the point three years ago? We were ready to kind of go down this validation pathway. You know, by the time we were sort of ready to go validate, you know, third party, this external validation that we

did at the University of Arizona, we'd already done hundreds of internal validation studies. So we actually, you know, we knew exactly what they were going to say before they said it. those sort of value that our members saw in getting a third party to validate it was, you know, if whoop says whoop is great, that sort of only carry so much weight. And being on the front lines of talking to partners, I can tell you that everyone was really interested in this third party validation. Yeah, so we knew how accurate everything was and sort of had for several years before we went down this route. But just, you know, for different research opportunities and just expanding the sort of use case. for a whoop, it was important that, you know, some of those things get done in sort of a more formal way. But before that, we'd sent hundreds of people to a local sleep lab in the Boston area

where they underwent polysymognography while wearing usually four whoop straps. We needed to make sure that we got lots and lots of data because these studies are also extremely expensive. And so they would do the sleep study and then we would take the data off of the PSG data and then the WOOP data and then we wrote these machine learning algorithms that trained WOOP to recognize what WOOP data looked like during different sleep stages because one of the things that's particularly challenging

about any device that is trying to do sleep staging from the wrist is that these different sleep stages are defined by changes in brain waves and you can't read brain waves from the wrist so you're measuring heart rate and heart rate and movement which are sort of all downstream correlates of what's going on in your brain, but you're actually not making a direct measurement of any of those things.

It's just probabilities. No, it's like downstream markers. You know, similarly to how, like, the EKG measures heart rate by measuring, like, the electrical conductivity. So literally, like, that, you know, an instruction was sent from the SA node to contract. And then you measure the, like, contraction of the muscles. And then a PPG uses, measures blood flow, right? It's just like a downstream effect, but there's noise that interferes.

You know, anytime you're measuring a downstream effect of something, instead of the thing itself, there's an opportunity for other things to affect that downstream outcome. And you have to kind of clean those things out in order to, like, get to the original signal. Okay. So whoop was actually built from the ground up all the way from the hardware to be optimized to get the most accurate heart rate data to get the most accurate sleep data that we possibly could. and certain optimizations that we've made

from different studies that we've done over time have changed everything from the form factor of the wrist to even like the material of the band to make it easier to get like a clean heart rate signal because from heart rate from the PPG signal on whoop we get the heart rate, we get heart rate variability, we get respiratory rate, all of those things come from heart rate and so if the signal that the physical device is able to read

is cleaner all of those things become more accurate. So then the inputs to the sleep staging algorithm are cleaner and so that we're better able to stage sleep. So shout out to signal processes. Yeah, for sure. I think, like, I mean, all the things they have to count for to get a. Yeah. So I think, you know, one thing that's sort of easy to not appreciate is that, you know, while this paper in so many ways looks like it's a validation of sleep, it's really a validation of the whole platform and everything that goes into it. Because, you know, if you took our algorithms on a sort of less well optimized piece of hardware, you'd get less good results.

Right. And so, you know, everything that sort of we'd been doing from, you know, when I joined whoop seven years ago, we didn't even have a working prototype. So we weren't working on sleep back then. We kind of knew we wanted to,

but we weren't doing that yet. And so, you know, we had to kind of mess around a little bit and try and fail and iterate to figure out how to get that that hardware optimized so we could optimize the signal processing algorithm

that gets, you know, the heart rate signal so that we can get all the sleep pieces, right? Which is why, you know, it's so exciting every time that we were working on the next generation of hardware or next generation of the signal processing algorithm, sort of the ceiling of how accurate the sleep algorithm can be gets like raised to that next bar. So sort of three years ago when we finally decided to like go for it and to kick off this third

party validation process, it was because we'd sort of reached the level as far as like the hardware signal processing and then sleep algorithm that we thought we needed to be at to go to market because, you know, this was also right around the time that we launched a consumer product. So, Emily, why don't we just outline the study design and methodology quickly for folks so they can really understand, like, what it was we were looking at? Yeah, so, you know, we keep saying sleep validation study, but it actually was a little bit more than that. So the subjects actually wore whoop for two weeks.

So it was a 14-day study. So they had two seven-week periods, and on that seventh night, so right in the middle of them, They underwent polysemnography testing, and I'll explain that in a moment. But it was sort of randomized whether or not you sort of were in the experimental group or the control group in the first or the second week. But if you were in the experimental group, you got the full user whoop experience, so sleep scores, you know, access to your data, recovery scores, all that wonderful stuff. and in the control week, you would report on your sleep quality and log your bed times and wake times, but you didn't wear whoop. What was really interesting about that sort of those two-week periods where they didn't do the truth studies, there's no polysomography testing. They did show that just by wearing whoop, self-reported measures of sleep quality improved.

And this is something that we talk about a ton. We see it over and over anecdotally with our clients that they, They report improving their behaviors and sleeping better. We also see it in the data that, like, over time, time in bed increases, sleep consistency improves. You know, people start to practice better sleep hygiene because once you start- Yeah, because once you start telling somebody this, like I say this all the time, but there's something really powerful about telling people that they're like getting a B-minus in sleep and they hate it and then they improve.

So it's like you have to quantify something in order to kind of know what you're trying to work on. And I, what I love about this is that, you know, there's a lot of research out there. that says, you know, wearing a sleep tracker is going to make you more anxious and not and actually hinder sleep. You know, and I just want to call that out that we see the opposite effect. And I think, you know, it's a shout out to a product in terms of the UI and, you know, the design of the app and how elegant that is in terms of delivering feedback in a way that is really encouraging and helps you navigate the behaviors that are going to lead to better

sleep. So it's not just telling you you're bad at sleep, but it's, hey, you do excellent. Y and Z, and you're going to get better results. So it's, it's, I think, I think we can, with the results, we can kind of safely not bucket ourselves with those other products that, you know, maybe increase anxiety around sleep. Yeah. And so I think like, you know, we set out really to kind of quantify the accuracy of Woop. And so this point was, you know, not really the, from our perspective, like the main objective of the study. But I actually think it's, it's really worth talking about the fact that just seven days, which is most of our whoop users are on for, you know, months,

if not years. And so, you know, seven days such a short period of time, the fact that it's like, just seems to be correlated with this improved sleep quality is really a powerful result. Yeah. Like you said, I think it shows the power of the product and all the kind of good work that's gone into it. So I definitely kind of want to highlight that finding because I think it is so interesting. And somewhat, I think, unique. I don't think it's something that people think about as much. Like, they think about quantifying everything, but just, like, the fact that, you know, we're really driving positive behavior change is special. Because ultimately, at the end of the day, I mean, that's, you know, if you can, you know, I know we're talking, we're going to talk about how

accurate we are, which is a phenomenal. But, you know, let's say you're off on accuracy, you know, percent, you know, to the left or right. The fact that we're actually modifying behavior is, to me, that's the that's the keys of the kingdom right like if you're actually modifying your behavior that's in the end that's that's what you want to drive toward right yeah and i think that's that's just an unbelievable finding um that i think really is an important piece of this validation study you sometimes people ask me like okay so like i know my sleep stages you know so what like what like what am i supposed to do with this information and i think like the data shows that there is something powerful that happens when you're informed yes yeah even just you know seven days worth

I want to talk a little bit about what a polysymognography is because we're going to throw that word around a lot when we talk about the sleep study. In a polysynography testing, subjects sleep in a laboratory, and they simultaneously wear an EKG, which measures heart rate, an EOG, which measures eye movements, which is electro-oculogram, an EMG, or electromyalogram, which measures motion, so they actually measure on your chin and either measure like grinding your teeth and stuff like that. that and then also like sometimes on your legs for things like restless leg syndrome and stuff like that and then in EEG electroencephalograms so that measures brainwaves and then sometimes PSGs will also include other sensors sometimes you get like pulse ox or SPO2 and respiratory rate and breathing things like that and so we had these 32 subjects do one night each of this PSG testing while wearing whoop and then compared the

results of the polysomography testing, which is the gold standard to what whooops algorithms output. And we also measured the accuracy of whoop's heart rate, heart rate variability, and respiratory rate as compared to the gold standards collected during the polysomnography testing. And, you know, I think one outcome that was, you know, extremely exciting for us is that heart rate, heart rate variability and respiratory rate were all extremely accurate within one unit of truth on average throughout the sleep. So it's really exciting that, you know, those are the inputs that our whole sleep algorithm is built on.

So if those measurements weren't accurate, nothing else could possibly be as accurate. And then we also measured sort of our ability to detect sleep and wake, of course. It's our sleep validation study. And then slow wave sleep and REM sleep. And all of those agreement to the gold standard was really, really high. Emily, just to go back to methodology real quick, the fact that there are two scores I think is really important and relevant and just points to the rigor of the study. Can you talk about why that's important? I think one thing that's somewhat unusual about polysynography testing, we keep calling it the gold standard, which is true. But you would expect sort of something you might not expect about a gold standard measurement is that it's actually done by hand. It's an extremely manual process. So people go to school to become polysumography technicians. And they learn to sort of look at the sleep in 30-second chunks, and then they, like, manually, based on the patterns that they're seeing, that they've been trained to identify, classify the stage into the different sleep stages. So wake, light, sleep, slow-wave sleep, and REM.

And as you can imagine with any subjective manual process, two individuals both experienced trained in different labs who score the exact same sleep will agree on about 76% of the epochs or those 30 second chunks throughout the night, which is pretty bad in the scheme of things. and so like what that kind of tells you is that there's a lot of sleep is sort of in this gray area that's maybe kind of not quite one sleep stage or another and sort of physiologically the explanation for this is that it's not really true that sleep are these discrete categories you're not either in slow wave sleep or light sleep it's a spectrum and so there's like lighter light sleep and deeper slow wave sleep and then there's like this thing in the middle that's like deep light sleep or light deep sleep you know kind of um right and so when when you have two scorers and they're looking at sort of a a light light sleep or a deep deep sleep you know they're much more likely to agree because it's you know kind of a clean or an easy case and when you're right near that threshold where it's like kind of looks like light sleep and kind of looks like slow wave sleep you're much more likely to get that disagreement where like one just happens to go one way and the other one happens to go another way. It's kind of like if you're looking at colors and you know,

somebody says like is this like red or blue, it's pretty easy because then it gets to like purple and you have to pick one, right? Because you're not allowed to call it purple. You have to pick red or blue. And so you're going to get disagreement. And that's exactly what kind of is going on with these sleep staging. And so the way that we got around this, which is something other similar studies have done in the past, it's like a sort of recognized way of handling this problem is we had two people score the whole sleep and then we looked at the places where the two scorers agreed and then compared whoop to those periods of agreement because the sort of assumption is that in places where two humans disagree then how could woup possibly agree

because we're only going to say one thing and they've said two different things so you can't win but in places where the two humans agree then it's sort of probably true or at least increases is the probability by a significant amount that that was, in fact, that sleep stage. And so whoop should probably get that right as well. And so our comparison was done to the agreed upon periods of the sleep. So I think, like, one thing, you know, our members might be thinking, like, when listening to this is like, oh, does that mean that, like, you know, it's wrong 25% of the time, like, is whoop incorrectly stating sleep?

And I think that, like, that's why you have to understand that it's a spectrum. so it's like you know if we call purple uh red sometimes and blue the other amount of time then you know we might be a little bit wrong here a little bit right there so overall one of the things that the study looked at is like is there a bias or sort of are we calling it purple or red half the time and blue half the time such that it kind of like washes out in the total sleep time that we're calling slow wave sleep and total sleep time that we're calling light sleep is still at least still correct. So in addition to like looking at each 30 second chunk and making sure we agreed with the consensus epochs from the two scores, we also looked at like over the whole course

of the sleep or we identifying like the same amount of total light sleep, slow wave sleep and REM sleep. Obviously it wouldn't be good if we had this bias and we were telling you like every night that you got 30 more minutes of REM sleep than you actually got. But it doesn't really matter if sort of like I started this period of REM sleep 30 seconds early and ended it 30 seconds early because you still get the same amount in the end. And just maybe just remind users, you know, what are the things that you're looking at to kind of stage the sleep?

So our intervals, you know, just kind of go through the different things that we're looking at. Yeah, so we look at the accelerometer a little bit. Most of the information that it provides is sort of redundant information we're already getting from the heart rate. Which is just movement.

Yeah, sorry, the accelerometer measures movement. Technically, also position. And then we also look at, So from our heart rate signal, the PPG signal, we derive heart rate, heart rate variability, and respiratory rate. And then from those four inputs, so the motion, heart rate, heart rate variability, and respiratory rate, we derive dozens of proprietary features that then get passed to our machine learning algorithm, which sort of turns them into sleep stages. It's pretty cool to look at the graph, you know, because it really just shows how granular a view. we're actually looking at the body in terms of what's happening physiologically and then being

able to map probabilities, you know, based on, you know, where you sit across these different markers. Yeah. And some of these markers, you can kind of just look at them, just plot them over time, and you can almost see the sleep stages. Like respiratory rate tends to be a little bit lower and more even during slow wave sleep and, like, higher and more variable during REM sleep. And so you can almost kind of like see that change and be like, oh, like, that's kind of cool. And then some of the other features that we look at, you can't really see you with the naked eye, but the algorithms are able to derive that information from them. The fact that we can actually detect these different stages is just phenomenal,

but also just the very simple fact that we know when you're awake versus when you're asleep. So, Emily, we've been obviously working on this for three years, waiting for the outcome, and you've obviously been in minimally evolved and just transferring data and making sure they had all the information that they need to be able to execute this. so it finally comes out how do you how do you feel yeah i mean it's just it's really great to kind of see everybody's hard work you know formally recognized in this way again like it's one thing when we do all of our own analysis and you know sometimes you go after specific populations and we have a lot of control over all of the internal validation that we do and then we have this idea of like

how good whoop is but then when you go to do this like third party validation study you almost have to like hand over your baby and like all the keys and then just trust that like everything you've been doing internally you know is actually going to kind of hold up in this world where you have no control over it to hand hand these keys over to you know dr sigh and his team at the university of arizona and then for them to publish and show that like everything that we thought about the product you know really does like stand up in this sort of rigorous test of its accuracy and performance was just really exciting and you know we're also really really excited it feels like this new level you know for whoop and for our members who are curious you know

they invest a lot and you know we want to be as transparent as possible with them you know they should know like where we're good and they should know where we're not and where we're working to make improvements and so I think it's a really exciting step forward in terms of transparency you know we encourage our members to ask questions yeah definitely you know to hold us accountable so you know please please read the paper ask questions about the paper and we're not stopping here you know we're going you know continue down this path of doing research i mean it's core to who we are as a company and and we just want to keep getting better and yeah thank you know one kind of maybe teaser to kind of leave them with is we've been working on this for three years but it's not like we started working

on this and then waited for it to finish for to do anything else right we've got there's lots more in the pipeline you know obviously we can't talk about all of it yet but kind of stay tuned i think there are a couple of fast follows that we're really excited to talk about about, you know, as they start to come out in the next couple months. Yeah, absolutely. Well, Emily, thank you so much. This has been really fun and just, yeah, super pumped for what's next. Thank you.

If you're not already a whoop member, you can join our community for as low as $30 to begin. We provide you with 24-7 access to your biometric data, as well as analytics across strain, sleep, recovery, heart rate variability, and more. The membership comes with a free whoop strap three-parts. We offer 612 and 18 month memberships. The more you sign up for, the more you save. If you enter the code Will Ahmed at checkout, that's WI-L-L-A-H-M-E-D, will give you 15% off a membership just for listening to this podcast. And for our current members, you can upgrade to the Woop Strap 3.0 and get access to all the new Woop Live features by following the link in your Woop app.

If you're out of contract, you'll literally get the 3.0 for free when you commit to another six months. Check out whoop.com slash the locker for show notes and more, including links to relevant topics from this conversation and others. Make sure to subscribe, rate, and review the Whoop podcast on iTunes, Google, Spotify, or wherever you like to listen. We'd love to hear your feedback. You can find me online at Will Ahmed. I try to respond to everyone who reaches out. And you can also follow at Whoop on Instagram, Twitter, and Facebook. You can email The Locker at Whoop.com with any thoughts, ideas, or suggestions you may have.

Thank you again to all our listeners, to all our Whoop members. We love you.