Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas - 358 | Solo: Vacuum Energy and the Cosmological Constant

Episode Date: June 22, 2026

The most surprising discovery in fundamental physics during my career as a scientist was undoubtedly the acceleration of the universe, announced in 1998. The most straightforward explanation for these observations is a positive cosmological constant, or vacuum energy. I talk about the origin of the idea with Einstein, how quantum physicists started to think about it and understand the "cosmological constant problem," as well as how its discovery also raised the "coincidence problem." This is the first of two connected solo episodes; the next will be on theories of dark energy that are not the cosmological constant. Blog post with transcript: https://www.preposterousuniverse.com/podcast/2026/06/22/358-solo-vacuum-energy-and-the-cosmological-constant/ Support Mindscape on Patreon.

Hello everyone and welcome to the Mindscape podcast. I'm your host, Sean Carroll. In physics, which is a very broad field, it includes, you know, atomic physics, plasma physics, condense matter physics, biophysics, particle physics, gravity, cosmology, all these things. Different fields, of course, have different rates of progress, both because of theoretical ideas and experimental input. If you focus just on fundamental physics, and let's not argue about what that means or should mean, I take it to mean figuring out the most fundamental laws of nature the best we can. So not including things like biophysics and condensed matter physics, not because they're not important.

It's just a different thing. I once tried to get the field to change the name fundamental physics to the name elementary physics, the most elementary laws of physics being the idea, but no one agreed with that. That's okay. Fundamental physics is here to stay. In this subfield of fundamental physics, which includes sort of particle physics and gravity, things like that there haven't been a lot of helpful, experimental, surprising discoveries in the past several decades. Arguably, since the 1970s, there have been a few here and there, masses of neutrinos,

things like that, also things that we expected, like the Higgs boson, gravitational waves, but in terms of true surprises, it's been few and far between. That makes it hard to make progress in fundamental physics because experiments are what drive us to great ideas. And there's one, of course, shining counter-example to this idea that there haven't been many surprises, which is the acceleration of the universe. Back in 1998, we were told by results from two competing astrophysics groups looking at type 1a supernovae, using them to measure the expansion rate to the universe and how it changed over time, that against all expectations, the universe is actually accelerating, not decelerating in its expansion rate. And of course, this was a big deal. Nobel Prizes were handed out and the whole bit. We still don't understand with complete confidence what is going on with the acceleration

of the universe, which is great for fundamental physicists such as myself, for theorists who are trying to figure out what's going on. It would be nice if we had more than that one fact, the acceleration of the universe. It would be helpful to our project of inventing theoretical models to explain it. But it doesn't slow us down. We're going to invent them anyway. There is a leading candidate, the cosmological constant, as proposed by Einstein many, many years ago, but there's other ideas as well. So recently, through a set of circumstances that I won't go into, I fell behind personally in my recording of podcasts and I'm doing traveling and things like that.

And so even though I just did a solo podcast on quantum mechanics not that long ago, I figured I can fill in by doing some more solo podcasts. I figured just one solo podcast, but what you're going to do is you're going to get two and two separate weeks. What happened was I went to Patreon, where I sometimes turn for advice from my Patreon listeners. If you ever curious about what it would be like to be in the cool in crowd of Patreon listeners, just go to patreon.com slash Sean M. Carroll and you can sign up. And I asked them what will be a good topic for a solo podcast. And I gave them some suggestions. And I think that a pretty clear tendency, even though there's a lot of diversity in the answers, was for the suggestion of theories of dark energy and the accelerating universe.

And in some sense, it's shocking. I haven't done that already. This is something that I do know something about. Indeed, my first ever attempt at publishing a trade book and writing a popular level book and getting it published was I proposed a book on the accelerating universe and dark energy. And no one wanted to publish it. This was like 2006, maybe, and people were already like, well, we've done that already. That's old news. 1998, 2006, the ship has sailed.

So I don't think that was right. I think there's plenty of room for a really good book on that topic, but I never did write it. So plenty to say about that. And I thought that was a very good choice on the part of my Patreon listeners. So I thought, sure. And then I started sketching it out. And I realized, you know what? There's actually too much to say.

about this topic. So I've broken it into two episodes. I know that I have in the past gone on a great length in solo episodes, but I do think that there's sort of a natural dividing line here. You can talk about the cosmological constant as Einstein came up with it and as we really wondered about what it means in relationship to quantum field theory and what the observations are telling us about it. And then you can go beyond that to what if it's not a cosmological constant. What if it's something else? What if it's something dynamical, a modification of gravity, or something like that? So those will be the two solo podcasts. I'm going to talk in this one that you're listening to right now about the cosmological constant, the energy of the vacuum, because that's an equivalent idea. They're precisely identical.

And then I'm going to do another one on dynamical dark energy, which on the one hand, contains a whole bunch of really exciting, interesting ideas and would be a tremendously exciting, exciting, discovery where we defined evidence for it. On the other hand, there is no evidence for it right now, so it's still completely speculative. We're a little bit less constrained in that part of the world, so we can let our imaginations roam. It's all great physics. It's all cutting edge. It's all very perplexing and confusing. And that's exactly what you like as a working physicist. You want to have problems you don't know the answer to, struggling to figure out what those answers are. So I'm going to try in these two podcasts to sort of let you in on both what the questions are and why certain answers look the way they do. Certain other

answers are less promising, things like that. Hopefully to give you roadmaps you can understand better what we're trying to do in cosmology and fundamental physics with this problem and what might happen next. So with that in mind, let's go. When we're talking about the cosmological constant, the energy and see the vacuum, or whatever, there's a very obvious starting point for us, which is Einstein. He's the guy who came up with the idea back in the day. And it provides a fun excuse to think historically about how these ideas came about. Because it's always useful to think historically because we have.

different pre-existing ideas than they had back then, right? We have different things that we think are really important, and other people in different historical eras might not have cared about that much, or they might have had other things that they care about that we don't care about very much. So that's very much the case for Einstein, who was absolutely not the shut up and calculate kind of guy. He was someone who had very strong ideas about how things should be, and he thought them through in a very rigorous and careful way. But he was open to changing his mind and realizing that he was wrong.

But he definitely had a style that opened his mind to different possibilities. So you picture that around 1915, Einstein had put the finishing touches on the general theory of relativity. The general theory of relativity says that spacetime, which was an idea that went back to Minkowski, right after Einstein had figured out special relativity in 1905, his old professor Minkowski had pointed out that it was better formulated as a theory of a unified space time. And Einstein didn't like that suggestion by Minkowski, but he eventually came around to realize that was a good idea.

And in fact, that we could incorporate gravity into the ideas of relativity by allowing space time to have a geometry, by allowing it to be curved. And that was the basis for his theory. theory of general relativity. So that's 1915. And I just can't even imagine how much fun it must have been to be a physicist at that time and to have this new playground to run around in and explore new things. Of course, Einstein had the head start on everybody else. So many of the fun ideas that one could explore in the context of general relativity were first explored by Einstein very soon after he invented the theory.

He didn't do everything, like the Short Shield solution, the metric tensor, the curvature of space time for something like the solar system or a black hole or whatever, was found only two years after Einstein invented the theory, but Einstein himself didn't find it. It was Short Shield, of course, who did it. One thing that Einstein did do was to think about the universe as a whole. And now it's kind of interesting. This is why I mentioned the philosophical predisposition

that different people have, Einstein was heavily influenced by Ernst Mach, who was a physicist and a philosopher of physics, to be honest, in Germany at the time, who had very definite ideas about the relationship between space and time and matter and things like that. I would say that Mach's influence on Einstein was maybe inspirational, like maybe he nudged Einstein in the right direction to say some good things. But ultimately, the answers that Einstein came up with are completely disconnected from any of Mach's ideas. And it's a weird historical thing that people keep trying to fit it in. People keep trying to have Mach's principle, which connects the rest frame of the universe to the existence of matter in the universe, to general relativity, Einstein's theory,

even though they're not really actually very closely connected. But I bring this up because Einstein thought that they were closely connected. And when he first turned his attention to the universe as a whole, he was very influenced by Mach. And there was also in the back of Einstein's mind, and people today forgot about this, but there was this puzzle that people knew about in Newtonian physics, in Newtonian gravity.

The puzzle is the following. You can sort of imagine solving Newton's equations for gravity, using this way of writing things that's actually due to Pierre Simone Leplas in terms of the gravitational potential. So rather than just saying that the sun reaches out 93 million miles to the earth and pulls it via the gravitational field,

you can say, well, I shouldn't even say by the gravitational field because that's not how Newton talked. Via the gravitational force, via action at a distance in Newton's way of thinking about it, Laplace replaced Newton's equations with a field equation, and then there's no action at a distance. The sun is pushing the gravitational potential field. The Earth moves in it.

All that makes perfect sense if you're talking about the solar system, but it was puzzling when you try to talk about the universe as a whole. Because it turns out that if you just imagine a universe filled with an absolutely constant density of matter, it is undefined what potential you should have. There are multiple solutions that work equally well. had a constant to it, things like that. And it didn't seem to be a good starting point for investigating the universe. So Einstein had that in mind when he started doing relativistic cosmology. And therefore, rather than just assume that the universe was spatially flat,

because remember the whole point of general relativity is it's the geometry of space and time that defines what we think of as gravity, Einstein said, well, what if space is finite? What if space doesn't go to infinity? Then you can both help understand how to get out of this puzzle in Newtonian gravity, and you can be consistent maybe, he thought, with Mach's principle, that it's the matter in the universe that is fixing what we think of locally as our standard of rest in our inertial frames and things like that. So Einstein had these ideas, and he was pretty convinced, as far as I can tell,

I'm not a super expert on the history here. so maybe someone can correct me. But as far as I could tell, like he thought that there were just really good philosophical reasons to think that space is finite, not infinite. Personally, myself, I'd rather be open-minded about that. I don't have any strong pre-existing opinions about whether space is finite or infinite.

I'll let the data or a better theory decide that. But anyway, he tackled the problem of cosmology, and he said, look, let's just start by assuming that matter is distributed uniformly throughout the universe. He had no idea whether that was true or not. There was some indication from cosmology circa the 19-teens that maybe there was some uniform distribution of matter in the galaxy, but we didn't even know in 19, 15, 16, 17 about the existence of other galaxies, right? That idea didn't, while the idea was there, but the knowledge, the data, didn't come

along until the 1920s. So it was sort of open season for speculating. There wasn't enough data to really tie you down. So Einstein said, okay, let's imagine a uniform universe, one where the density of matter is statistically the same everywhere. He knew that the density of matter is not the same on the earth as in the sun and things like that. You're imagining that you're thinking about very, very large scales throughout the universe and the average density of matter could be the same everywhere. And he said, okay, what if I take a spherical universe? And by spherical, of course, we know that space is three-dimensional on large scales. So that means a three-dimensional sphere.

That's no problem to differential geometry and general relativity. So space in Einstein's model is a three-dimensional sphere. And time is just a line. Time goes from wherever it goes on into the future. And he said, okay, I'm going to plug this into my equation called Einstein's equation, and I'm going to ask what that implies for the evolution of the universe, a constant density in the spherical universe. And what he found was that two things happen. You plug into Einstein's equation, and you get an equation for either the expansion or contraction of the universe. Space

wants to change its size. That's not really something that was very well defined in the Newtonian point of view, but in general relativity, it makes perfect sense. You have a three-dimensional sphere and the three-dimensional sphere can get bigger or smaller. And it turns out that both that geometry of space and the existence of matter contribute to this change in the size of the universe. And you can find a solution where at one moment of time, the universe is not changing size. Okay. So you can set the sort of the velocity of the universe, if you want to call it that, to zero by balancing the curvature of space versus the matter, what you can't do is keep it there.

So basically what Einstein realized is that the equations he was looking at kind of implied a universe that would start from somewhere. It was all very vague at the time, right? We're talking about 1917 or so. Start from somewhere, expand, and then reach a point of zero expansion rate, but then just start collapsing again. This is very familiar to modern cosmology. students. This is a closed,

finite time universe that we can absolutely think about. It doesn't seem to be the universe we live in as far as we know, but as far as Einstein knew, he wasn't sure. So he seemed to find, again, you could find a solution where the universe is not expanding, but it would instantly start

either expanding or contracting, depending on how you set up the different numbers. Imagine buying your kit a toy only to find that the batteries aren't included, or buying furniture, but it's missing the tools you need to build it. That's so frustrating, right?

Now imagine that exact same feeling, but you're paying way more. That's what buying business software usually feels like. It's fragmented, disconnected, and incredibly expensive. Odu completely changes that. Odu is a complete, fully integrated business suite where all your apps actually talk to each other. We're talking automated lead routing for your sales teams,

seamless AI integration, and an intuitive point-and-click website builder. It's everything your business needs to scale, saving you time, headaches, and money. So stop piecing your software together. Go to odoo.com. That's ODO.com to learn more. Now that I live somewhere with real seasons, there's a difference between the winter wardrobe and the summer wardrobe. When summer arrives, I'm looking for clothes that are lighter and more breathable, but still put together. That's why I keep coming back to Quince.

They focus on high-quality essentials that feel and look amazing, like breathable linen and soft organic cotton. They're well-made basics, but without the luxury markup. Quince has European linen pants and shirts that look as good as they feel. Their teas are soft and easy to wear, and their lightweight cotton sweaters are perfect for cooler summer nights. And the whole range is perfectly affordable. Quince clothes are priced 50 to 80% less than what you'd find at similar.

brands. That's because Quince works directly with ethical factories and cuts out the middlemen, so you're getting premium materials without the markup. I have a blue, long-sleeved quince linen shirt that is one of the first things I reach for in the closet. It's effortless and goes with everything. So elevate your summer wardrobe by going to quince.com slash Minescape for free shipping on your order and 365-day returns. Now available in Canada too. That's Q-U-I-N-C-E-com slash Minescape for free shipping and 365-day returns. Quince.com slash mindscape. And so that was puzzling. And there's this somewhat of a myth that says that Einstein was just philosophically devoted to the idea of a static universe. As far as I can tell, that's not true. What is closer to the truth, though again,

I'm not a historian here, is that Einstein asked his astronomer friends. He said, wait, is the universe expanding? And no, no, no, it seems to be constant. Seems to be more or less. static as far as we can tell. Again, they didn't know there were other galaxies. They were just looking at stars within our galaxy. So Einstein was said, you know, he said, I'm going to listen to the data. And he went back and said, can I find a different solution to the equations where the universe is not expanding or contracting. And to do it, he changed his equation. He added a new term to Einstein's equation for the spatial, the space time geometry in general relativity. These days you might go like, well, that was a little bit of a panic move there, Professor Einstein. Like, you didn't stick by your guns. You went and changed your equation right away. But you have to realize, you know, it was Einstein who invented the equation, right? Like, it wasn't sacred to him.

It hadn't been tested and put through decades of hoops that it all successfully passed. He had written it down. He can write down a new one. You know, why not? It was a very adventurous time in theoretical physics in that sense. So he added a new term for the equation. And, you know, he didn't do it just willy-nilly. He said, well, what am I allowed to add to this equation?

There's, of course, famously the metric tensor in general relativity. If you want to know more details about some of this general relativity stuff, please pick up my book. The biggest idea is in the universe, volume one, space, time, and motion, what we explain what all these things mean. And he realized that he could just add a term that was proportional to the metric tensor. The metric tensor is what tells you distances and times in the space-time geometry that you have in general relativity. There's absolutely nothing wrong with adding such a term.

It's 100% compatible with the principles of relativity, with the symmetries, with all the language that he had, the notation, etc. So it's fine. You can just add it. And there is a constant. There's a new constant of nature, which Einstein called the cosmological term, or we would call the cosmological constant. So he added it to his equation. And very roughly speaking, very informally, there's the effect of the cosmological constant. If this number, the cosmological constant, is positive, it resists the tendency of matter to pull the universe together. It sort of pushes the universe apart.

So by cleverly choosing the value of the cosmological constant, Einstein could find a solution to his equations, where space was a three-dimensional sphere, matter was uniformly distributed, and there was a positive cosmological constant that kind of resisted the gravitational field of the matter to give you a static universe over time. And indeed, this solution, which is perfectly good, exact solution to Einstein's equations, is now known as the Einstein static universe. Now, almost right away, people realized this was not a good idea. For one thing, people talk about it as if the Einstein-Static universe is unstable.

That's a little bit not quite accurate. I mean, I get it. I know what they mean. It means something real. See, what you're doing is you're balancing the density of matter versus the amount of the cosmological constant. And you have to balance it exactly. If there was a slight mismatch between the proper value of the matter, is cosmological constant, the universe could be almost static for a long time, but ultimately

you begin to expand or contract. That's not quite an instability, but that's just pointing out that any deviation from this very, very precise number, which had no good reason to be true, would lead you to a non-static universe. There's a better objection, which I don't know whether anyone raised at the time, which is the universe is not perfectly smooth, right? As I said, Einstein knew that the density of matter on Earth is different than in the sun or an interstellar space. And so at best, this assumption that the matter density of the universe is constant is an approximation, right? It can't be exactly right. There are inhomogeneities in the universe.

And those in homogeneities will generally grow with time if you solve the equations. Overdense regions, regions in which there's a little bit more matter than average, will pull more matter onto them. And that is an instability. Denser regions grow more dense, less dense regions grow less dense, and the universe becomes lumpier and lumpier over time. The only reason that's a problem, I mean, we think that's exactly what does happen in the real world. That's the origin of structure in the universe. But in the modern world, we know that that is only been going on for a finite period of time because there was a big bang about 14 billion years ago. Einstein, remember, was proposing a solution which had supposed to, was.

supposed to last for infinity years. There was no Big Bang in his model. So why hasn't all the structure already formed? Why isn't the universe sort of arbitrarily in homogeneous was a question for him that he wouldn't have been able to answer? So anyway, that those sort of puzzles became a little bit less pressing 10 years later or 12 years later, 10 years later, actually, I think, when Edwin Hubble and others put forward the data that said that the universe is not static after all. I mean, Hubble actually did two things. First, he showed that these little fuzzy nebulae that people were taking pictures of in their telescopes are, in fact, whole separate galaxies. They're very, very far away.

Then he compared the distances of those galaxies to their red shifts and showed that the universe is expanding. He was very cautious about interpreting these data in the light of general relativity. He was an observer at heart, not a theorist. So he just said, I have a relationship between Redshift and distance. And of course, all the theorists like Einstein, but also people like LaMetra and DeSitter and others, instantly knew that what he was seeing was the expansion of space. So there's some controversy in the historical literature about whether Einstein said that this was the greatest blunder of his life to have invented the cosmological constant rather than just,

saying that his theory predicted that the universe should be expanding or contracting. He could have predicted the expansion of the universe and then he could have become famous. Who knows? But we don't know. That was a story that was apparently told by George Gamov who liked telling stories. I think it might very well be true. Einstein certainly did say that, you know, he should get rid of the cosmological constant once he realized the universe was expanding. You don't need it.

The whole point of it was to make the universe step. and with the universe is expanding, you can do that without any cosmological constant there. The problem with that move, the problem with saying, you know, I don't need it anymore, is who cares whether you think you need it anymore? It's there as an open possibility, right? Like I said, it was a perfectly legitimate addition to the equations. So the scientific thing to do is not to say, I don't need it, I'm going to set it aside. But to say, is it there? Ask the question. What would the effects of this weird thing be? How would we know it? Can we put either observational limits on it or could we even detect it or something like that? And really ever since then, ever since Einstein suggested it, there was a sort of a cycle of phases of people suggesting that the cosmological constant could help with this particular problem that we have in observational cosmology and realizing, oh, we don't need. didn't go away. And so it waxed and waned quite a bit. Okay. So that was the situation. And honestly,

you know, cosmology was not at that time in 1920s, even in 30s, 40s, 50s, maybe 60s, not the most respectable part of physics. The ratio of theory to data was quite large. And so it was hard to make progress, just as it's hard to make progress right now in all sorts of questions in fundamental physics. And so people moved on to other things. The really good physicists were thinking about quantum mechanics and particle physics and things like that. And so let's talk about that. Let's talk about quantum mechanics and particle physics and things like that. It was very quickly realized, well, I shouldn't say very quickly, pretty quickly realized by people like Lemaître, Georges Lemaître, a famous cosmologist and really the pioneer of the idea of the Big Bang theory. By the 1930s,

Lometra had thought about the cosmological constant, and he had realized that you don't need to think of the cosmological constant as a change in Einstein's equation. You can just include it in terms of the original equation that Einstein wrote down back in 1915. And this is a point that I think is really perfectly transparent, but people sometimes fumble it, so it's worth getting exactly right here. when Einstein wrote down his equation in 1915, there's a left-hand side and a right-hand side. The left-hand side are some symbols. I'll tell you what the symbols are. It's r-mu-nu minus 1-half-r-g-mu-new. These are symbols that are constructed from the metric tensor and the remon tensor,

and they all have to do with the curvature of space time. And then there's the right-hand side to Einstein's equation, where you see 8 pi-g-tt-t-munu, And 8 and pi and G are all constants of nature. There's also like a 1 over C to the 4th or something like that. But I don't ever remember where the speed of light goes because it's set equal to 1 in my brain. So to me it's just 8 pi G. G is Newton's constant of gravity.

It goes back to Isaac Newton and his famous inverse square law. 8 and pi are numbers that you are familiar with. T.U. Nu is the energy tensor, the stress energy tensor or the energy momentum tensor, different people call it different things. It doesn't matter. It basically sums up and includes all the different forms of energy and momentum and stress and strain and all those things, pressure and density that could be causing space time to curve. Okay. So the right hand side says that there's matter and stuff. The left hand side says there's space time curvature and the equation sets them proportional to each other. So when Einstein first invented the cosmological constant, he put it on the left hand side.

He grouped it in with the curvature of space-time terms. He wrote lambda times little G-mu-new, where little G-mu-new is the metric tensor. What Lometra realized is you can just do math. That is to say, you can subtract. You can move Lambda-G-Mu-New from the left-hand side to the right-hand side of the equation, make it minus Lambda G-Mu, and then just identify that as a new form of energy. Indeed, it is the form of energy that would exist even in empty space. So the metric says you can think of the cosmological constant as the amount of energy in the vacuum,

a.k.a. the vacuum energy. And he talked about the different properties of this vacuum energy. It's constant. It never changes by hypothesis. There is a pressure that is associated with the energy density. Really, there's two numbers in cosmology that you need when you have some new source. of energy momentum, you need the energy density and you need the pressure. And for the vacuum energy, Lemaître pointed out that the pressure was negative. That is to say it's more like a tension. The classic example is if you have a rubber band and you pull it, right, the rubber band pulls back on you. That is a negative pressure. So pressure pushes, negative pressure pulls you back. Okay. And so that was fine, you know, Lemaître realized that you don't know. need to think of the cosmological constant as a change in Einstein's equation. You can think of it

or identify it with the energy density of empty space. And what I want to emphasize is there literally isn't any difference between those two ideas. It's not, and some people still insist that it is, but they're just wrong. It's not as if either you're changing the geometry side of Einstein's equation or you're adding vacuum energy. There's no difference between those two things. They're literally exactly the same thing. So from that perspective, it's once again making the cosmological constant term even more inevitable sounding, right? Like, okay, now there is a number. There's a constant of nature, the energy density of empty space, and you've got to go measure it. I mean, maybe you can set it to zero for convenience as a simplifying assumption to start, but it is in principle

something you should go out and constrain by taking data. Now, The other thing I want to mention is sort of jumping ahead to more modern times a little bit, but this negative tension thing, sorry, negative pressure thing, which is tension, leads to some weird explanations for why the cosmological constant makes the universe accelerate. So you will hear things that sound something like the following. Ordinary matter or radiation and things like that have positive energy density and positive pressure. So they make the universe decelerate. They basically pull things together.

If you imagine a universe full galaxies and it's expanding, all of those galaxies are pulling on all the other galaxies. So the expansion rate would naturally want to slow down. And this particular way of saying things, which I don't like, so don't trust me on this, says, but now the vacuum energy has a negative pressure and therefore it pushes things apart. And you're like, wait a minute. Where did that therefore come from? After all, you just told me that the idea of,

negative pressure is tension which pulls things together. It doesn't push things apart. And then you're told, aha, but what I mean is the gravitational effect of negative pressure is to pull things, is to push things apart because the expansion of the universe responds not to just the energy density, but the energy density plus three times the pressure, which is true. And that number three, three times the pressure, is the number of dimensions of space. okay, so it's not arbitrarily chosen.

It's because space is three-dimensional, but it's three times the pressure. So row plus 3P, where row is the energy density, p is the pressure, that's the number that comes into the acceleration equation in general relativity

for the acceleration of the universe. The problem with that explanation, I mean, it's perfectly true, mathematically, row plus 3P is negative if you have a cosmological constant because for the cosmological constant, p equals minus, minus row. The pressure is precisely minus the energy density. And therefore, row plus 3p is minus 2 row. So if you have a positive energy density in the vacuum, you have a negative amount of energy density plus three times the pressure. And that's what pushes the universe apart. The problem with that as an explanation is that where do I get any intuition for the claim that row plus 3P is appearing on the right hand side of my equation?

It's just magic, right, unless you've actually gone through the math. You're not actually explaining anyone, anything to anybody. So I don't like that explanation. There's another way of explaining it, which is the following. The equation that cosmologists actually solve for the expansion rate of the universe, and this is all done post-Einstein. Like Einstein had the idea that the universe could be a three-dimensional sphere, but then other people like Robertson and Walker,

and then followed by Friedman and Lemitra and others, really generalized that idea to all sorts of different possibilities and they got a very general equation now known as the Friedman equation. And the Friedman equation, in fact, there's several Friedman equations, but the one that everyone solves

sets the expansion rate of the universe, which is given by the Hubble parameter, right? We now call the Hubble parameter because Hubble measured it. In Hubble's plot of velocity times distance, velocity versus distance, his equation says that they're proportional. velocity is a constant times distance for a distant galaxy, and that constant is known as the Hubble parameter. And so that same constant appears in the cosmological version of Einstein's equation

in the Friedman equation, and it's a way of judging how fast the universe is expanding. So the Friedman equation says that the Hubble constant squared is proportional to the energy density of the universe. Maybe, you know, if you also have a term for the curvature of space, if you want to include that. These days, we know that the curvature of space is close to zero, so we don't need to keep talking about that for today's purposes. So anyway, H squared, H is the Hubble constant. We square it is proportional to the energy density.

That's it. That's the equation. Okay. There's no appearance of pressure in there. And you say, well, how do I get the difference between accelerating and decelerating universe out of that equation? And the answer is how the energy density changes with time matters. So imagine a universe that has nothing but vacuum energy in it,

nothing but cosmological constant, okay? No matter to worry about makes our lives easy. That says that row, the energy density is just a constant. That says that H is a constant because h squared is proportional to row. H is proportional to the square root of row. And if row is a constant, so is the square root of row. So what the equation is telling you is just H is a constant. The expansion rate of the universe is a constant. And you might now get confused for a different reason. You might say, wait a minute, you are telling me that this means the universe

is accelerating, and now you're telling me it means the expansion rate is a constant. That doesn't sound like accelerating to me. And that's because you haven't grown up with non-nuclidean geometry. Sorry about that. You're thinking of the expansion rate is somehow a velocity. But it's clearly not.

In an expanding universe, if you look at a galaxy using Hubble's law, a galaxy, a billion like years away is moving apart an apparent velocity of some number, whereas a galaxy twice that far away is moving away twice as fast. There's different velocities for different galaxies. So the expansion rate of the universe cannot possibly be thought of as a velocity. Okay. What it is is a rate of expansion. It's basically the answer to the question, how quickly does the universe double in size, right? In fact, the Hubble time, which is one over the Hubble Conce,

is almost exactly that, the doubling time for the universe. And so to say that H is a constant, which is what you get for vacuum energy, says that the amount of time in which the universe doubles in size is a constant. So in other words, from any one size of the universe, it can double in size and then double again, and then double again and double again. So it goes from its initial size to twice that, to four times that, to eight times that, to 16 times that and so on. That is what we call exponential expansion, because you keep multiplying by two at every time step. They told two friends and they told two friends, et cetera. So a constant energy density leads to a constant expansion rate, which shows up as an exponential

expansion to the universe, which is obviously accelerating. So that chain of logic is longer than the negative pressure business, but it actually makes sense, unlike the negative pressure her business. Then I get a pressure business, like I said, it's not wrong. It's completely true. It's just sort of making it sound easier than it really is. Okay, sorry, I got to get off my chest because I'm trying to fix all of the mistakes that I hear out there in the world. There's probably too many mistakes out there for me to fix all of them, but I can do my little part. Okay, so that's the state of play in the 1930s. Einstein had come up with his equations. He had to come up with the idea of the cosmological constant. Lemaître interpreted the cosmological constant as an energy density of the vacuum. But Lemaître wasn't doing quantum mechanics.

This is all still perfectly classical, okay? Make sure that we distinguish different ideas in our head. So Lemaître is just pointing out that this number Einstein had invented could be interpreted as what is the energy density in empty space. If I take a cubic centimeter of space and I remove all the stuff, so there's no particles, there's no radiation, no dark matter, as we would now say. truly empty, you can ask yourself how much energy is in that empty cubic centimeter of space. And according to Einstein and Lemaître, that's a new constant of nature.

That's not something you can just say, oh, it should be zero because it's empty, right? It's something you have to go out there and measure. Now, at this era, the 1930s and so forth, like I said, a lot of the excitement in physics was in quantum mechanics, particle physics, things like that. People knew and loved gender relativity, but they didn't think about it too much. It was not front and center in the minds of the leading theoretical physicists. People did know about it. Wolfgang Pauly wrote a whole textbook on general relativity when he was preposterously young.

But it wasn't what they were thinking about on a day-to-day circumstances. They were thinking about quantum field theory and quantum mechanics. And guess what? The idea of vacuum energy rears its head in a different way in quantum field theory. And this is, again, something that people get wrong a little bit. So let me try to explain it the best I can. Forget for the moment about quantum field theory. We'll get back to that.

Think about the simplest physical system we can think of the simple harmonic oscillator. Okay. So literally a weight on a spring attached to a wall bouncing back and forth, simple harmonic motion. In physics, we describe this in terms of a potential energy as a function of X. If X is the location of the weight on the spring, the potential energy looks like omega-square. X squared, where Omega is the frequency, literally the frequency of the thing bouncing back and forth. And so this is a very, very simple physical system with an omega-square to X-square type of potential energy, an ordinary kinetic energy. It's simple because it's simple to write down, but you

can solve it, and it's kind of beautiful, and it illustrates a lot of very basic ideas. And so if you take a quantum mechanics class, like I was teaching here at Johns Hopkins last semester, one of the very first things you will do, and one of the things you'll keep doing at higher and higher levels of sophistication throughout your physics career, is study the simple harmonic oscillator. So in classical physics,

what that means to study the simple harmonic oscillator is you have a ball and it's rolling in this hill and the hill looks like a parabola, omega squared, X squared, and you say, well, if I let the ball go with zero velocity at a certain point on the hill, can I solve its equations of motion going backward and forward?

And the answer is yes. And they look like signs and cosines, sinusoidal functions for the oscillation of the particle back and forth in this potential. of course it would be very natural to do the quantum mechanical version of this. What does that mean? That means you don't have a little ball moving into potential. That means you have the Schrodinger equation, which means you have a wave function in that potential, and you're going to solve the Schrodinger equation in that kind of potential and see what you get.

And it's very, very instructive to do that. What you get by solving the Schrodinger equation in the simple harmonic oscillator potential is a series of states with, fixed constant energy, you can have any energy you want. But if you want a definite energy, remember in quantum mechanics when you measure something, you're not sure what answer you're going to get generically. If you measure the position of a particle, it could be some uncertainty there. If you measure the energy, there can be some uncertainty there. But there's a special set of quantum states where the energy is perfectly well-defined and known, the eigenstates

of energy. And for the harmonic oscillator, these eigenstates are exactly, exactly equally spaced in energy. So there's a lowest energy state, and there's a state that has one more energy, whereby one I mean H-bar times omega. Remember, omega is the frequency of the harmonic oscillator. H-bar is Planx constant. So H-bar-O-Mega is the size of the energy step

from the lowest-energy state to the next energy state. It's also the size of the energy step from the first energy state to the second and from the second to the third, etc. They are exactly equally separated energy amounts between the different levels, as we say, or different excited states of the simple harmonic oscillator. Okay?

So that's just one of the things that makes it a very, very nice system. Other kinds of systems with other kinds of potentials will also have different kinds of energy states, but they won't be equally spaced in energy between them. So all that's very well known. You'll study that to death in your quantum. the mechanics class, but there's one thing that is kind of mysterious and it's going to depend on your professor's inclination how much you're going to think about it, which is that the lowest

energy state, if you start with an ordinary simple harmonic oscillator with a potential omega squared x squared, by the way, it's usually one half omega squared x squared, but don't worry about the half. So omega squared x squared, okay, um, what is the lowest energy state? Well, classically, That's the particles just sitting there at x equals zero, so it's zero potential energy, and also zero kinetic energy. It's not moving. That's the lowest energy state is zero energy. That's how we're trained to think when we grow up. Quantum mechanically, solved Shortinger's equation, that turns out not to be right. The lowest energy state of the simple harmonic oscillator doesn't have zero energy quantum mechanically, if it did classically. There is a difference

between the classical lowest energy and the quantum mechanical lowest energy, and that difference is one half H bar omega. So H bar omega is the gap between one energy state and the other. One half H bar omega is the starting point. So the lowest energy state is one half H bar omega. The next one is three halves H bar omega and then five halves H bar omega, et cetera, for infinitely far into the higher energy states. this phenomenon that the quantum mechanical energy of the ground state of the lowest energy state is one half H bar omega is called the zero point energy of the harmonic oscillator.

And so like I said, it's going to depend on the predispositions of your professor how much you're going to fret about that. I think that the answer is you shouldn't fret too much. And yet the reason why is the number, one half H. bar omega is not to be interpreted as the ground state energy of the simple harmonic oscillator. What it's to be interpreted as is the difference between the ground state energy of a classical harmonic oscillator and a quantum mechanical one. Okay. So quantum mechanics raises the ground state energy by one half H bar omega, but it doesn't tell you what it was to begin with. I could have imagine starting with a simple harmonic oscillator whose potential was not

one-half omega-squared x squared, but rather one-half omega-squared x-squared minus one-half h-bar omega. Nothing can stop me from doing that. And classically, if all you're doing is studying the rocking of a ball in a hill or something like that, adding a constant to the overall energy makes no difference whatsoever. It does not show up in any of the motions of the particle. it has no effect on the physics. It's completely irrelevant. All that matters is how the energy changes from point to point. So two things. One is the actual zero point energy is completely arbitrary. I can set it to be whatever I want. It's not one-half-h-h-bore-Omegas, it's whatever I choose it to be. And number two, the number one-half-h-h-h-bore omega isn't even anything intrinsic to nature. It's a difference between a quantum mechanical theory and a classical theory. But nature is just quantum mechanical. Nature doesn't. and know about the classical theory, nature says you're quantum mechanical from the start. Why were you starting with a classical theory and then quantizing it? You should have just started with the quantum mechanical theory.

And in that case, the relevant fact is that the energies of different energy states are separated by H-bar Omega. Okay. Nevertheless, people kind of get hung up about this a little bit. They start thinking that, you know, quantum mechanics tells you that there is energy density, in the vacuum. Don't get me wrong. There could be, right? That's what I'm trying to say. It's a constant of nature. It's a fact about reality. What is the zero point energy of the simple harmonic oscillator? It's not a fact about your quantum mechanical calculations. You can choose the vacuum energy to be whatever you want. Okay. So why do we care about that? Why do we care about the simple harmonic oscillator?

You know, it's a very nice thing to be able to solve things precisely, but we want to get into the messy real world with electrons and protons and stuff like that. So we have to upgrade our game a little bit and start thinking about quantum field theory. At this point, I will encourage you to read volume two of the biggest ideas in the universe, quanta and fields where we talk about precisely quantum field theory. If you want to know what that means, it just means that there are fields, stretch throughout all space and time, and they vibrate and you quantize them, and that's what gives rise to particles and things like that.

So since the 1930s, at least getting into high gear in the 1950s, quantum field theory has been our best attempt at understanding the world at a deep level, at least particle physics and the world other than gravity. Gravity is a special thing, but okay, that's a more difficult topic to get into, not something we understand very well. Okay, so quantum field theory sounds like a very large journey away from the simple harmonic oscillator, but here, the wonderful news. When you start studying quantum field theory, just like in other areas of physics, you're going to start simple and you're going to work your way up. So what does it mean to start simple? It means to start with a field that is just doing its own thing, a field that is not interacting with other fields, not even interacting with itself. And those are all very well-defined statements and you can specify what they mean. So we talk about a free field to be one that is not

interacting with other fields or itself. Okay. That's one step. Let's think about free field theory, no interactions, just make our lives easy. Then we'll put in the interactions later. It's an ancient move. It goes back to Galileo saying let's ignore air resistance. We can put it back in later, right? We're saying let's ignore interactions between fields. We can put them back in later. The second move, which is something that is less ancient, but is absolutely always what you do when you study fields, is you start thinking about waves. And that means you start thinking about waves of definite wavelength, what we call modes of the field. And again, I'm not going to get into details if you haven't read the book. If you have read the book, this is all perfectly transparent. But you can think about what is happening in a vibrating quantum field rather than thinking about what's happening at each point in space. You can think about what's happening at each wavelength. And this is not a weird thing. This is called Fourier analysis. And it's literally what your ears do when you hear sounds. You hear different pitches because your ear. are saying, well, how much vibration is there at this frequency and how much is there at that frequency and so forth. So let's do that. Rather than focusing on the behavior of the quantum field at one point, let's focus on the behavior of a single mode with a fixed wavelength everywhere

of this quantum field. Okay. Guess what happens? Miraculously, each mode of a free quantum field obeys an equation which is exactly the same as the equation of a simple harmonic oscillator. That's why we're doing all this. That's why I was wasting my time telling you about the simple harmonic oscillator because free non-interacting quantum fields very naturally behave like a collection of an infinite number of simple harmonic oscillators. Not one at every point in space, but one at every wavelength. Indeed, one at every wave vector, we say, because a mode of the field has both a wavelength, but also a direction in space.

That's the wave vector. So every wave vector is a mode, and every mode acts like a simple harmonic oscillator for a free field theory. And that means that every mode of the harmonic oscillate, of the quantum field has the same kind of zero point energy contribution that the regular harmonic oscillator had. And everything that we just said,

about the quantum mechanical harmonic oscillator really is still true in quantum field theory. That is to say it's both true that if you start with zero energy in the classical theory and then you quantize it, you get a new addition to the energy of the vacuum, which is one half H bar omega. Now there's every value of omega, okay? So you don't just because there's different wavelength modes of the field. They all have their own vibrations. And what that means is that you don't just have energy density in empty space of one-half-h-h-bar-O-Mega. You have the sum, or the integral, if you like, of all possible omegas of one-half-h-h-bar-O-Megga.

And what does that equal? It equals infinity. So this is one of the famous infinities of quantum field theory. This is perhaps the most basic infinity of quantum field theory. The energy of empty space, the vacuum energy, is infinites. if you're very, very naive about quantum field theory. But, but before you get too excited about that, everything I said about the quantum mechanical oscillator is still true. I could choose the vacuum energy to be whatever I want.

It doesn't need to be infinity. I can just choose it to be zero. Okay. This is one of the steps in what we call the renormalization of the quantum field theory. It's perfectly legitimate. No one forced me to have a certain value of the vacuum energy to begin with, okay? So the lesson is not that there's an infinite amount of energy in empty space in quantum field theory. The lesson is that there's an arbitrary amount of energy in empty space in quantum field theory.

And guess what? That's just the lesson of the vacuum energy that Lemaître had talked about. There's a new constant of nature, the vacuum energy, and we have to go measure it. So it's a very delicate back and forth kind of thing. Quantum field theory says, okay, there can be vacuum energy and empty space. It does not say what it has to be, but there's another level that says, well, maybe it lets you have and guess as to what it could be. Like, maybe it lets you sort of estimate a natural value. Like, there's nothing natural about zero.

You could set it to zero. You could say, I'm just going to say there's zero energy in the vacuum. You're allowed to do that. But there's no reason to do that from the point of view of quantum field theory. some number and you can wave your hands and you can say like, okay, I have sort of natural guesses for how big that number should be. So the first person to go down this road was Jakov Zeldovich, who was a physicist or Russian physicist. And in the 1960s, he made this point very clear that the vacuum energy of quantum field theory is precisely what goes into the vacuum energy of

relativity, the cosmological constant. And this is the beginning of the connections between quantum field theory and vacuum energy. There were some precursors there. I shouldn't jump right ahead to Zeldavich. Again, it's a fascinating story because people don't know what's going on, so they cling to certain beliefs

and they get rid of others and they learn something, but don't put it in the right context and all this stuff. You know, Richard Feynman, of course, became famous for Feynman diagrams in quantum field theory. And Feynman diagrams are

are little pictures of particles moving, but they're 100% derived from quantum field theory. Okay? They're not derived from thinking about particles. The particles come out of quantum field theory as predictions once you quantize these fields. But these ideas don't come out of a vacuum, if you'll excuse the pun. And Feynman had this previous idea with John Wheeler, who was his advisor at Princeton, the Wheeler-Findman absorber theory of radiation. You may have heard of this. This is part of the package of ideas where Wheeler came up with the idea that maybe positrons, which are anti-particles of electrons, are just electrons moving backward in time. And therefore, there's only one electron anywhere that sort of moves forward and backward in time and goes through the whole universe. People love this idea.

It's not right. It was a good guess. It was a fun idea. But these days, we know, no, that's not right. What's actually right is quantum field theory. The reason why all electrons are the same is not because they're the same electrons, because they're all vibrations in the same underlying quantum field. But okay, they didn't know that. They weren't sure.

They were trying new things. That's what physicists do. And so Wheeler and Feynman and their absorber theory really tried to think of electrons as particles, not as fields. Even though quantum field theory already existed at the time, they were open to changing that. And one of the things in the back of their minds was they knew about this fact. Acumenogy business, they didn't know it was the cosmological constant. They didn't know as far as I know. They didn't know it had anything to do with gravity or anything like that, but they were just

kind of worried about the fact that when you quantize field theory, you get an infinite contribution to the energy. Again, I think that a more correct attitude is to say that infinity is not the energy. It's the difference in energy between the classical theory and the quantum theory. And it's the quantum theory that is right. So who cares about the classical theory? but still they were a little bit worried. And so they thought if they could replace quantum field theory with the theory of particles,

that would make this vacuum energy problem go away. It didn't work, of course, but that was part of their motivation. It eventually led to Feynman diagrams, which is pretty good. The other example is Phil Anderson, Phil Anderson, a Nobel Prize winning condensed matter physicist, who arguably was the first person to really propose what we now call the Higgs mechanism. The Higgs mechanism says if you have some scalar fields, you have some field with no directionality in space or anything like that. It's just a number at every point in space.

And imagine that it couples in a certain way to other fields like electrons and photons and stuff. And this field that is now called the Higgs field, which Anderson thinks it should have been called the Anderson field. Imagine that this field gets a non-zero value in empty space. So would you imagine that this field, the scalar field, the scalar field, field, which we now call the Higgs field, has some potential energy function, which says how much energy is in the field simply as a virtue of its value. As opposed to, you know, fields have kinetic energy if they're changing in time and they have gradient energy if they're changing in space, the potential energy is just how much energy they have because they have a value. Okay. So if the

minimum energy is not at zero field value, which is no reason why it should be, this is the famous Mexican hat potential. Then the field will roll down to the minimum, which is not at zero, and it will sit there and it will affect other fields. And Anderson knew this and he sort of mentions it in words in one of his papers, but he didn't pursue the idea. That's why he doesn't really get credit. He doesn't get his name on it for the Higgs mechanism. Why didn't he pursue the idea? Part of it was because he was really a condensed matter physicist, not a particle physicist. That's where his interests lay. But also because I think this is. my impression, he knew that there would be a tremendous energy density associated with that field,

changing its value in empty space. So you're proposing that there's a field pervading the universe that carries an enormous amount of energy, and no one has ever noticed it before. I'm not sure that he ever, again, connected it to gravity or the cosmontical constant, but it bugged him. So he moved on to do other things. That's okay. He won the Nobel Prize for other things. Anyway, this, all of which is to say, like by the 1960s and Zeldavich, people had put the story together. If you just care about the zero point energy of the quantum field, it doesn't matter.

You can make it whatever you want. It's a constant of nature. And if all you're doing is particle physics, it's not even an observable constant of nature. It's just like the ball rolling back and forth in the quadratic potential and the parabolic potential, the harmonic oscillator. If I lift the potential up and down

by adding a number to it, It doesn't change the behavior of the particle. Likewise, if I just change the energy density of empty space, it doesn't change particle physics in any way. What it does change is gravity. Gravity cares about the total amount of energy in the universe, and Zeldovich appreciated that. So he pointed out, you know what, we have a problem here, folks,

because there is a number, the energy density of empty space. We have no idea what it is. We have no idea what it should be, I should say. But you can sort of wave your hands about natural energy scales, blah, blah, blah, and Zelda points out the natural scale for this number is much bigger than it can possibly be given the observations. We're talking about an enormous discrepancy. So let's like skip ahead to the modern way of thinking about these things. The modern way of thinking about these things actually puts some meat on the bones of all this talk about natural values and stuff like that.

And that framework that we're talking about here is effective field theory. In the effective field theory framework, you admit you don't understand everything. You say, look, I don't know the theory of everything. I don't know whether it's string theory or something else. I don't know what's going on at very, very high energies, which in quantum field theory corresponds to very, very short distances in space time. Say, I don't know what's going on. Happily, quantum field theory has this feature that even if you don't know what's going

on of very short distances, you can still find an effective theory that only includes particles and fields and phenomena at low energies and long distances. This is called the infrared theory, because you think about infrared things as long distances, long wavelengths, ultraviolet things as short distances, short wavelengths. So following Ken Wilson and others, you say, I have an ultraviolet cutoff. That's basically the shortest distance that I'm claiming to understand. I don't know what goes on at even shorter distances, but I'm going to bundle all those effects up and ask what they do in my infrared theory. Okay. And then there's some rules for how big you expect the parameters in the infrared theory to be, et cetera, et cetera. And basically

some parameters in the infrared theory are what's called relevant. They get more and more important as you go to longer distances and more in the infrared. Some are irrelevant. They're not that important. And the cosmological constant, the vacuum energy, is super duper relevant. It very much shows up in the infrared. And you can use these techniques of effective field theory to say, okay, how big do we expect the cosmological constant to be? And the answer is hilariously bigger than it can be. And I say this even knowing that the 1960s, you know, our cosmological observations were not nearly as precise as they are today. They didn't need to be. The different, the discrepancy between the effective field theory prediction for the vacuum energy and what you actually observe it to be is just so hilariously large that you don't need very precise observations.

The most dramatic way of saying it is if you run your effective field theory, you put your ultraviolet cut off all the way up at the plank scale, and you compare it to our limitations, let's say in the 1980s before we actually found the vacuum energy, how much bigger is the predicted vacuum energy than the limit? answer is the famous factor of 10 to the 122 times bigger. One followed by 122 zeros. Okay. So not the number 122. That would be easy. We could deal with that. But one followed by 122 zeros. That's 10 to the 122. It's a very, very big discrepancy. Okay. That discrepancy is known as the cosmological constant problem. And again, as I keep emphasizing, you're 100% allowed to solve. the cosmological constant problem in the following way. Just say that's the way it is. The cosmontal constant in the infrared, the measurable quantity is something we measure.

Okay. I mean, maybe you had an expectation for what it should be. Your expectation was wrong. Tough cookies, you know. And that's an attitude that you can have and you can get away with it. It's completely compatible with the data. Maybe that's just how things are. Sometimes physics is like that.

I think a lot of theoretical physicists are going to want to say, maybe that's true, but maybe it's not. Maybe this huge discrepancy between our expected value for the vacuum energy in effective field theory and the observer value is a clue to some new physics that we don't yet understand. So even though it's intellectually allowed for you to say the cosmontual constant just is what it is, I measure it, then I get all of my life, maybe you're missing an important. clue that nature is trying to give you to develop a better theory. I think that's what a lot of physicists these days would think. So that came about in the 1960s, and by the 1980s, once we'd put the standard model of particle physics in place, people began turning their attention to gravity and cosmology. Okay. In the 1980s were when the first superstring revolution happened,

and it also was the time when people really began thinking about the cosmological constant problem very hard because they'd solve all the problems in experimental particle physics. They're moving on to the universe. And so why is the observed vacuum energy? So just to make it super duper clear, if the cosmautical constant or the vacuum energy were anywhere close to its natural value as far as particle physics is concerned, the universe would have exploded apart into nothingness in a tiny fraction of a second after the big bang. The expansion rate of the universe driven by such a vacuum energy would be so big that you couldn't make an atom, much less a molecule or a planet or a star or anything like that. So it's not like you have to do some delicate cosmological measurements and maybe you have error bars on those and maybe you're wrong about this. The discrepancy is just so big that it's absolutely overwhelmingly obvious. Furthermore, you don't even know whether the cosmological constant is positive or negative theoretically, right? You're not making a prediction for its sign. You're making a prediction. You're making a prediction. for its absolute magnitude, if anything.

It's just a guesstimate more than a prediction. It could be negative and that would mean it would make the universe re-collapse in a tiny, tiny fraction of a second. And that's also ruled out by the data since the universe is around 14 billion years since the Big Bang. So by the 1980s, theoretical physicists had started thinking seriously about this cosmological constant problem. And I say this because I entered graduate school in 1988 and now from the rest of this, podcast, solo podcast, and for the entirety of the next one on dynamical dark energy, it's going to be a very Sean-centric view of all these problems. Not because I am central to the story of dark energy in the accelerating universe,

but dark energy in the accelerating universe are central to me because I did a lot of work on it and it wrote a lot of papers on and things like that. So I'm not going to try to give you a completely unbiased view of anything. I'm going to tell you what was happening around. me when I was thinking about these things because I'm not doing a lot of research for these solo podcast. Sorry about this, but I do have some experience that I can share with you here. So there's a bunch of things in the late 1980s, early 1990s, where people were thinking about the cosmological constant problem. And I'm going to mention four ideas, four attitudes,

for, let's say, strategies you can take to address the cosmological constant problem. One of them is a little bit anachronistic. It came along later. But it gives you a few for what's going on. None of them are very good, okay? None of them are like obvious slam dunks. That's why it's still an interesting problem. One that got an enormous amount of attention is supersymmetry. Supersymmetry in the 1980s especially was incredibly popular.

It really just had grown popular in particle physics in the 70s. And in the 80s, people were still very excited about grand unification and also connections. of supersymmetry with string theory. So it was like the hot topic in theoretical particle physics circles. And supersymmetry, unlike other symmetries, does have something to say about the cosmotrial constant. So what do I mean by that? Remember, the absolute value of the vacuum energy, as we were talking about for just the harmonic oscillator oscillator, right?

All it matters is how it changes from place to place, except when gravity comes. into the game. So for the rest of particle physics, for the standard model of particle physics, for the electroweak force and the strong force and the Higgs mechanism and all that stuff, you can ignore gravity and you don't need to worry about the cosmological constant. Supersymmetry is different because supersymmetry is a mixture of a space-time symmetry, or it's a way of mixing space-time symmetries with what we call internal symmetries. super symmetry is sort of a unique way of having a non-trivial interaction between spacetime symmetries and things like gauge invariants that you get in the standard model of particle physics. So supersymmetry because it relates particles of different spins, right?

Spin zero and spin one-half particles, for example, spin is a spacetime feature. It's a space-time property of the quantum mechanical field. So supersymmetry in some very rough sense knows about space time. If you talk to like supersymmetry experts, they will start telling you things like the supersymmetry transformation is the square root of momentum or something like that. And I'm not sure if that's very enlightening or not, but the point is it is related to space time, supersymmetry, okay? So it's related to both particle physics and to gravity.

That's very exciting all by itself. And because it's related to space time, super symmetry knows. something about the vacuum energy, by which I mean, in a given theory of supersymmetry, the vacuum energy is not arbitrary anymore. In regular old quantum field theory, it's arbitrary. It's a constant nature, the vacuum energy, you're going to measure it or whatever. In a supersymmetric theory, it's determined by what is going on in the rest of the model. And people very quickly realized that there were certain versions of supersymmetry where the vacuum energy would be exactly zero.

You could actually solve the puzzle of why the vacuum energy is small if you had perfect supersymmetry. They also realized you could have negative vacuum energy, which seemed weird, but okay, at the time, they didn't bother that with that too much. They did, though, instantly realize a problem with this, which is that supersymmetry is not exact in the real world as we know it. If it were, there would be an electron with a certain mass in charge, and there would also be a spin zero partner of the electron with the same mass and charge. No such particle exists. This would be called the selectron by super symmetry aficionados, but it hasn't been found. This is not a huge puzzle.

This is just because probably supersymmetry is broken if it exists. Broken symmetries are super easy to have in particle physics. That's what the Higgs mechanism does. That was something that particle physics understood very well. So you have super symmetry and then you break it and you could even estimate the energy scale. at which it had to be broken and things like that. The problem there is that, again, if you break supersymmetry,

it does care about the vacuum energy, and you generally not only do not get zero for the vacuum energy, but you do get a number that is incompatible with the data. So supersymmetry was intriguing for the cosmological constant problem because it seemed to have implications for it, but the implication seemed to be bad. It seemed to take away your freedom, to make the cosmontal constant whatever you wanted.

Now, that's not necessarily a sign that you're on the wrong track. Maybe you just haven't filled in all the details yet. And so people put a lot of work into seeing could they come up with a supersymmetric model that explained the small value of the cosmontrial constant? And the answer was, no, they hadn't been able to do that quite yet. So that was one avenue. Another avenue is wormholes and Euclidean quantum gravity. And I mentioned this because this was one of those.

ideas in physics that it got very popular for a very brief period of time and then it went away. And the person who really put it together was Sidney Coleman, who's a professor at Harvard, who was my quantum field theory professor and was on my thesis committee. And it was a great guy, a brilliant scientist. And usually it's very funny. And Sydney himself was a little bit chagrined about this, like his style as a theoretical physicist. The reason why he's not as famous as Stephen Weinberger, Sheldon Glashow, his colleagues at the time, was just that he was very cautious. He was, you know, doing well established calculations in quantum field theory. He was not speculating about some crazy model. And then the one time that he did a little speculating about

some crazy model was this cosmontal constant stuff. And it got him a lot of publicity. But it wasn't really well grounded because it was just speculating. And ultimately it went away. But I got to tell you about it because even though it went away, and even though people don't really think it's the right answer, it was, it's really brilliant. I mean, it really worked. And if it had worked, it didn't really work. If it had worked, it was sort of offer an explanation the super intellectually satisfying way,

which is kind of intriguing. So here was the idea. People, like I said, were beginning to think about gravity as well as particle physics. And what is the sense? simplest way to get there. I didn't really plan out this part of the episode that deeply. Maybe I should have planned it out more deeply. But one of the ways you can do quantum gravity, we know that you don't have an exact theory of quantum gravity, right? We don't know all the details, but we know a lot about quantum mechanics. You know a lot about gravity. You can make some reasonable statements that you think are telling you something about quantum gravity. And that was done by people like Stephen Hawking and Jim Hardell and others. And they did it via what is called the path integral approach to quantum gravity. So basically, the first thing you do, and this is already a suspicious step, but you think of space time, but instead of four-dimensional space time, you just think of four-dimensional space. This is called doing Euclidean quantum gravity. So four-dimensional space, that is to say, all the four dimensions are the same.

There's not one picked out to be time and three picked out to be space. And by a path integral, you say, you calculate the quantum mechanical probability of the universe being in a certain state by adding up all of the different ways you can get there. This is one of Feynman's ideas, the path integral formulation of quantum mechanics. And so in gravity, since gravity is the theory of four-dimensional space time, the proposal from Hawking and others

was that you could calculate the wave function of the universe by summing over all of the four-dimensional spaces that have a boundary that is the universe that you're looking at right now. And it was just a mathematical trick. It wasn't, you know, the universe is not really Euclidean. It was just supposed to be a wave calculating the wave function of the universe. So people had this idea, all right, let's sum over all the four-dimensional geometries and calculate the wave function of the universe, right?

It's a good way to spend your time while you're waiting for the particle accelerators to actually give you useful information. And they realized, okay, what's the first thing you do? What's the simplest thing you do? Well, you're thinking of how to approximate an integral or a sum over all four-dimensional geometries. That sounds hard, but let's look at what we're summing up, and it's Einstein's equation, right? It's Einstein's expression for what we call the action of general relativity. It's a number you can attach to every four-dimensional geometry. And in, well, I'm not going to go into all the details, but you want to find a saddle point or an extreme of the action, and that's the classical equations of motion.

And so what they found was the simplest solution is just a big spear. a big four-dimensional spear, okay? That's a simple contribution to the path integral of quantum gravity. Fine. That's easy. We can do that. But then what about like the next simplest thing? And some genius, I don't know who it was, but someone said, look, if I have not one sphere,

but I have two spheres. So they're both very smooth. The idea being that if you wiggle the spear, it gets less and less of a good solution to Einstein's equation and therefore less and less important in the quantum mechanical calculation. But this person says if I have two spheres, I could connect them by a tiny wormhole. And that tiny wormhole would have almost no contribution to the action because it's just tiny wormhole, right? So two spheres are almost as good of a contribution to Euclidean quantum gravity as one spear is. And then once you say that, well, okay, I have three spears or I could have two spears with two wormholes connect.

them, etc. So there's an infinite number of things, like an add-up. And this was exactly the kind of thing that Coleman was really good at. So he and others, Steve Giddings, Andy Strominger and other people worked on this, Joe Polchinsky. They thought about, you know, how they calculate the wave function of the universe as a sum over spheres connected by all these wormholes. And people realize that this was a huge problem, actually, because if you have a wormhole connecting two big spears, it's not just gravity that gets involved. What if an electron falls down the wormhole? What if it goes from one sphere to another, right? And this is a lot of, I'm going to try to cut this short because we could get out of hand. Like I said, it didn't really pay off anyway. But the point is, it would seem that this is worrisome for real world particle physics. It would seem like in the real world, you might be able to have processes where like a photon goes down a wormhole and disappears. And that would be bad. And what Coleman was able to, to figure out was that in fact, that's not the observable impact of these wormholes. In fact, what happens is you can't just have an electron go down the wormhole because it would look like charge is disappearing in the universe and that's bad. What you could have is an electron and a positron going down the wormhole and a photon coming out. So that would conserve charge and energy and everything like that.

And that wormhole therefore looks from far away where you don't know it's a wormhole, it looks just like an ordinary particle physics interaction. So really, Coleman says, the effect of these wormholes is not to swallow up particles and make them disappear. It's to contribute to all of the existing parameters of particle physics, like the fine structure constant that tells you how strong the electromagnetic interaction is, like the Higgs mass, like all these other things. And like the vacuum energy. Okay. So in other words, Coleman says, these wormholes. holes add a new contribution to the vacuum energy.

So you have whatever classical contribution you have, you have the zero point energy, you have the Higgs field, and now you have the wormhole contribution to the vacuum energy. And what he showed by a calculation was the contribution of the wormholes was exactly to cancel out the contribution of everything else. It was truly beautiful. It was very, very convincing and provocative when it first appeared. And this is the late 80s, about when I arrived in grad school.

Sadly, the whole thing never lasted because, as Coleman himself said, very, very clearly, he was not bombastic about it. He says, look, this is a house doubly built on sand, okay? Because we're talking about Euclidean quantum gravity and wormholes. These are things we don't understand very well at all. And the more and more people looked at it, the less and less sensible and reproducible it all appeared. So it didn't burn out, but it faded away. You know, the idea that wormholes provided the successful resolution of the cosmological constant problem. There were enough obstacles to it really working.

Like some people said, it actually doesn't make the cosmological constant small, makes it big. It was very hard to figure these things out because no one knows what the rules are. So that kind of went away. But again, I'm trying to give you a flavor for what kinds of ideas were being bandied about at the time. Another idea I wanted to mention, which came out much later, I mean, 10 or 15 years later, is something called self-tuning. And this is interesting because I did have a little, I wrote a little paper about this. I didn't invent it.

Other people invented the idea. But this is once the idea had come along in the 2000s that maybe we live on a brain, B-R-A-N-E brain, right? Not B-R-A-I-N brain. So a brain is like a three-dimensional, there's different dimensionality of brain. let's say a three-dimensional surface embedded in higher dimensions. So if you have higher dimensions, extra dimensions of space, like you do in string theory, maybe we don't live in all of them.

Maybe we live on a three-dimensional brain embedded in them. And by specially picking the way in which the brain interacts with the rest of the world, you can find parameters or you can find dynamics, I should say, that will cancel the vacuum energy on the brain. So basically, if you have like, let's say, a cosmological phase transition where you have a change of physics from one phase to another. Ordinarily, that would be associated with a change in the vacuum energy, like a change in the average energy in water and ice is different.

That's why ice cubes float. Same thing is true for space time and quantum fields. When there's a phase transition, the energy changes. And so this self-tuning idea, what they were able to do was to show that they could pick a set of fields, etc., with the property that when you had a phase transition on the brain, it would not change the vacuum energy, the vacuum energy, or would not change the effective observed value of the vacuum energy that you would measure by doing cosmological experiments. And this was called self-tuning. The vacuum energy sort of goes away. Okay. And now, this turns out, again, it didn't work in the dustbin of

history. It was a good idea, like many good ideas, don't quite work. I remember when I gave a talk on it at the University of Texas, and Stephen Weinberg was in the audience. And Weinberg was an expert on the Cosmological Constant. He had written an extremely influential review article in the late 80s, where he went through all the different ways of solving the Cosmetrical Constant Problem that had been proposed at the time. And he actually proved a theorem that said, basically, you can't just invent some new fields and solve the Cosmachial Constant Problem. There will always be some fine-tuning involved, at least in the context of ordinary quantum field theory. And so when I explained in my seminar, I was explaining the idea of self-tuning, which again, I didn't invent, but I'll tell you what I did about it in a

second. And he's asking questions. He's like, where do you know, how did you do this? Like, I thought I had a theorem and said this couldn't be done. And I eventually explained it to him. And he goes, that's not self-tuning. You tuned it. And he was completely correct in the sense that you had to choose other parameters of the theory, not the vacuum energy, but other parameters of the theory. very carefully so that the vacuum energy would cancel in a certain way. My paper with Laura Mercini, which I wrote when I was a beginning professor at Chicago, was not about whether or not it was fine-tuned or not. I thought it was fine-tuned from the start, but I thought it was interesting because I wanted to know,

like, we have the Friedman equation, right? We have this equation that says the expansion rate of the universe feels the total energy density of the universe. It doesn't just feel the energy density of matter and radiation. how does it know to feel the energy density of matter radiation, but not the energy density of vacuum energy. How does that work? And what we showed was, and I sort of guessed this ahead of time and then we show that it was true. Effectively, what was going on was that the self-tuning mechanism had replaced the energy density in the Friedman equation. The energy density is given the letter row, the Greek letter row, I don't know why, for historical reasons, with the combination

row plus p, where row is the energy density and p is the pressure. Now remember, for the vacuum energy, row plus p is zero, because p equals minus row. The pressure is exactly minus the energy density. For ordinary matter, there's no pressure. So cosmologically, you could get an ordinary acting universe at late times, where we're dominated by matter density, and then the vacuum energy would just disappear from the equation. The problem with that is that there's also radiation. In the early universe, the universe is radiation dominated, in fact.

Radiation is more important than matter. And at the moment of Big Bang nucleosynthesis, when we are fusing protons and neutrons into helium and other light elements, the universe is actually more radiation than matter. And if you change the Friedman equation by replacing row with row plus p, radiation has pressure, it has positive pressure. In the case of radiation, the pressure is one-third of the energy density. So you're replacing row with something like four-thirds row, and that's not allowed experimentally. I remember Shammat Kachru, the string theorist, was one of the people who first proposed the self-tuning idea. And I gave a talk at Stanford, and he was in the audience.

And he said, you're telling me that I've made a prediction about something that can be experimentally ruled out? That's awesome. And it was experimentally ruled out, but it was also theoretically not that attractive. So again, it went away. But I'm trying to give you a feeling for what was attempted those days. And I'll give you one last one, one last attempt at solving the cosmontial constant problem. Of course, you know this one. It's the anthropic principle.

And this was suggested by a number of people, but it was Stephen Weinberg who really did it carefully. And he showed that if you live in a multiverse with different parts of the multiverse, having different values of the cosmological constant, but everything else the same, then you would naturally not get a big cosmological constant in those parts of the universe that were hospitable to life. For exactly the reason we talked about, if the cosmontal constant's too big, it blows galaxies apart, if it's negative, it collapses the universe very quickly. So there's a small window, relatively speaking.

And to his eternal credit, he made these predictions in the 1980s before the cosmological constant was discovered. And he said, look, the natural expectation, if this anthropic principle is on the right track, is that there is a non-zero value for the cosmological constant that we will eventually discover because there's no reason, no symmetry for it to be exactly zero. It turned out to be right. And he was more or less in the ballpark of the correct answer. That doesn't mean the anthropic principle is right, but it does mean that compared to those other things that I was telling you about, it did a better job of making a prediction and coming out right. Okay, so there we were in, you know, circa 1990. We had the idea of the cosmological constant.

We related it to the vacuum energy. We had tried to explain why it was small without a lot of progress. Now, I come into the game because Bill Press, who was a professor at Harvard at the time, is at the University of Texas now. He was in the astronomy department. And he was a very well-respected theoretical cosmologists. and he had been invited by Alan Sandidge, who was the editor of a journal called Annual Reviews of Astronomy and Astrophysics.

Bill was invited by Sandidge to write a review article about the cosmontical constant. I have no idea why. It was not like a particle physics review article. That's Weinberg had already done that. This was the astronomy version of the review article. And okay, so Bill for some reason said yes. And then for some reason, I was a grad student in the astronomy department at the time. and Bill asked me to collaborate with him.

I mean, I know why that happened because Bill didn't want to write anything about super symmetry or wormholes or anything, and he knew that that was important to the story. So he was going to do some cosmology. He was going to, like, make some plots. He was a very numerical guy. He was one of the authors of numerical recipes, which was at the time before you could ask LLMs to do all your coding for you, numerical recipes was a very helpful way to do scientific computation. And so Bill did a wonderful job at figuring out what the effect of the cosmological constant would be on, you know, cosmological observations, growth of large scale structure, a million different things.

He actually didn't just do numerical calculations. He derived formulas and semi-olytical approximations and things like that that were super useful for working cosmologists. And then we eventually realized that neither he nor I were an expert in the observational side of things. So we ask Ed Turner, who is a professor of Princeton and astronomer, to help us with that. And here's where it gets a little embarrassing because what Ed was really an expert in was a technique called statistics of strong gravitational lensing. The idea is that if you have a very far away galaxy or quasar, it can be gravitationally lensed by an intermediate. a distance galaxy or quasar, and the probability of that happening is something you can calculate. And you can calculate how that probability changes as a function of the cosmological constant.

If you have a non-zero cosmological constant, that can change the statistics of how frequently you will get lensed quasars and galaxies in the background. And when we started writing our article around 1990, this was thought to be the best way to constrain the cosmological. constant. Like ever since Einstein had come up with the idea, people were trying to measure the value of the cosmontrial constant. They had not done so historically and notoriously, it was very difficult to get precision measurements in cosmology. This gravitational lensing story was thought to be a promising way forward. Now, as it turned out, within 10 or 15 years after we wrote our article, there were two methods that were really useful for measuring the cosmotial constant. One was supernovae distances

and the other one was antisotropies in the cosmic microwave background. We didn't mention either one of these in a review article. This is what happens when you're in the right place at the wrong time. We were a little bit too early to be able to do that. So we were superseded by later articles, but in the meantime, we were useful for quite a time there. So my job was to write about the theory of the cosmontrial constant. I wrote both about all these things I've just been telling you about,

the anthropic principle and wormholes and stuff like that. But also, I just tried to explain the issues with naturalness and fine-tuning with the cosmontual constant. So there is the idea that the value of the cosmontial constant is mysterious. Why is it so small? That's the cosmontal constant problem. But there's also the idea, well, what if it's not exactly zero? So this is 1990. We didn't know that it was not exactly zero, but we were open to the possibility.

We thought it probably was zero, honestly. In the back where everyone's mind, all the theoretical physicists anyway, there was this idea, look, I don't know why the cosm partial constant is so small, but in the space of all possible theories that I haven't yet thought of, more of them make it exactly zero than make it 10 to the minus 122 times its natural value, but not 10 to the minus 123 times its natural value. Like maybe there's some symmetry or some dynamics or some mechanism or something like that that makes the cosmological constant zero. I can't imagine that it's just beyond the scope of detectability right now. Of course, the one big counter example to that was the anthropic principle. But what if you had detected? So I and others were saying, you know, back there in 1990, what if we do detect the cosmontual constant? There was one thing that everyone recognized, which is that if you know how fast the universe is expanding,

if you know the Hubble constant, then there's a. certain amount of energy density in the universe that would be compatible with the universe being spatially flat. So remember Einstein, ever since Einstein, you can think of the three-dimensional geometry of space changing through time in a cosmological setting. And that three-dimensional geometry of space could be flat, it could be positively curved like a sphere, or negatively curved like a saddle. And that zero curvature option right in the middle is sort of the special point, right? It's the middle point between the other two things. It's what's called the critical density of the universe. The critical density is the amount of energy you need

to satisfy Einstein's equation with a flat universe. So you might like on the basis of theoretical prettiness prefer the critical density. And in the early 1980s, we had the idea of the inflationary universe scenario. From Alan Gooth and others, we had the idea that you could explain why the universe is so smooth and homogeneous and isotropic, because you can imagine that the universe underwent a period of super fast expansion very early on that basically smoothed all the wrinkles and flattened everything out. And at the time, 1990, inflation didn't make a lot of predictions other than the critical density should be the density of the universe. The universe should be spatially flat. Meanwhile, astronomers had looked

for the density of the universe. They tried to measure it. They measure the gravitational density of galaxies and clusters and whatever. They couldn't find it. They kept getting numbers that were close to 0.3 times the critical density. So a lot of astronomers just thought the universe was not spatially flat. It was negatively curved. So one of the motivations, astronomically, for taking the cosmological constant seriously, is, you know, technically maybe the universe has the critical density, but point. 0.3 of it is matter, and 0.7 of it is the cosmological constant. We knew in 1990 that that would be allowed by the data.

It would be a little bit convenient. It would be a good story for inflation. But it would be weird. And I'll tell you the reason why it would be weird. It's something which we call the coincidence problem. The point is that the universe has been expanding for a long time since the Big Bang, 14 billion years. And as the universe expands, matter and radiation, which we know exist in the universe, dilute away, right?

The density of matter and radiation goes away as the universe gets bigger and bigger. Whereas the vacuum energy does not dilute away. Its energy density stays exactly fixed. So just to compare it to matter, by matter cosmologists just mean any collection of slowly moving particles, slow compared to the speed of light. So dark matter is matter, but stars are matter, galaxies are matter, etc.

matter. Matter scales away as the volume of space increases, right? Because the number of galaxy stays the same, but the volume of space goes up. So the amount of galactic matter per volume goes down. Meanwhile, the vacuum energy remains constant. So if you compare the value of the vacuum energy and matter today to what they were in the early universe, whatever they are today, there was an enormously bigger amount of matter and enormously smaller amount of vacuum energy in the early universe because the matter has to have declined ever since. And in the future, it will be almost all vacuum energy. So if the vacuum energy is of the same order of magnitude within a factor of 10 of the matter density in the universe today, that means it was completely invisible in the

early universe and will be completely dominant in the late universe in the future. And And we just happen coincidentally to live in the one time in the history of the universe when these two numbers are comparable. So this is called the coincidence problem. And I was really convinced in this coincidence problem. I made some plots that got into the paper that explained how hilariously unlikely it would be for the cosmological constant to actually be of the same order of magnitude as the matter density. But it doesn't matter how much theoretical uncomfortableness you have. Eventually you're going to have to go look at the data. and that's what people did.

So in 1992, the Kobe satellite discovered the anisotropies in the cosmic microwave background. So the cosmic microwave background radiation left over from the Big Bang was discovered in the 60s, but between the 60s and the 90s, whenever you measured it, it was the same temperature, the same intensity in every direction of the sky. It looked perfectly smooth. We all knew that it couldn't be perfectly smooth because we knew there's galaxies and the current. universe, those galaxies had to come from somewhere. And that implied there must have been tiny ripples in the universe at early times, which is what the microwave background is reflecting.

So eventually we're going to find them and the Kobe satellite did find them. Again, Nobel Prizes all around. That was in 1992. And it was, I remember it vividly that this was one of the first times in my life that I got to feel like an insider because Bill Press, who was an insider, and also an inveterate gossip. So he knew before the official announcement that Kobe had found these antisotropies and he told everyone at lunch. And then I got to tell people in the physics department

like Sidney Coleman and others that Kobe had discovered the temperature and isotropies in the microwave background. Everyone knew that was a huge deal. And I got to be the bearer of news and feel like I was a little bit ahead of the game. So I'm not above feeling good

because of those kinds of things, even though they're pretty meaningless in the ultimate. standard of things. But anyway, it was again a kind of an interesting historical moment because people knew about the prospect of finding antisotropies in the my great background. Jim P. Bowles and others and Zeldavich and others had worked out to all the math in the 70s and the 60s for that matter for what those antisotropies should look like. But people hadn't really thought it through like the rest of the community hadn't really absorbed. So what can we

do with this data that's going to come in? Okay, we're going to, we have this telescope, the Kobe satellite, which had detected the existence of these amisontropies, but it was very primitive. It only just found that they're there. It didn't really do a great job of the details. It was not a very highly infocused image or anything like that. So we knew that future telescopes would be bringing us enormously richer data about the microwave background on isotropies. And soon it was figured out that indeed these data that were going to come in would be very, very informative about cosmological parameters like the Hubble constant, like the cosmological constant, like the dark matter density, all of those things.

That turned out to be true. It took another 10 years between Kobe and other ways of measuring the microwave background before we really started to squeeze information out of it. But it was clear even in 1992 that that was going to be coming soon. The other thing, of course, that was data-oriented, which turned out to be decisive, was measuring the Hubble expansion rate, the Hubble diagram, that is to say, velocity of distant things versus their red shifts. No, that's not true. Velocity of redshift is the same thing. Distance of distant things versus their red shifts in a new way, namely using Type 1A supernovae.

So I had nothing to do with that except I was plugged in. Again, I was in the right place at the right time, and my friends were very involved with this. So the idea here is that there's different kinds of supernovae. I don't understand all the different classifications. That's real astronomy. But there are type 2s and type 1s, and the type 1s have subdivisions, type 1A, type 1B, etc.

My office mate, Klein Schmidt, who was a graduate student of Robert Kirchner, who was a professor at the Harvard Astronomy Department. He might have been the chair at that time. He was certainly a chair at some point, Bob Kirshner. And Bob was like the guy in observational supernova cosmology. Back in those days, you know, you don't remember. You kids are too young, but we didn't have that many supernovae, right? They were hard to find.

In a big galaxy, you might be able to see one supernova per century. And so if you don't look at many, many, many galaxies, you're not going to see any unless you're very lucky or they're very close. So there had been a couple. There had been a few supernovae that had been found, more than a few, but it was not a systematic procedure for finding them. And Brian Schmidt, my office mate, worked as his thesis project on type 2 supernovae and measuring the Hubble constant using what's called the expanding photosphere method.

Basically you model the expanding type 2 supernova and use geometry and black body radiation and physics to compare the brightness versus the distance and things like that. And it was pretty good as a method for measuring the Hubble constant, but it was not quite good enough. There's just too many uncertainties in the sort of sphericity of a type 2 supernova and things like that. The other thing you could do is type 1A supernovae. There, the physics was a little bit less understood.

Maybe still is. I'm not really up on it. But for some reason, it seemed as if type 1A supernovae were almost standard candles. That is to say, almost every type 1A supernova is approximately the same brightness. There is an understanding for why this is true. A supernova, well, sorry, let's go backward. The type 2 supernova happens when a red giant star runs out of fuel, collapses, and bounces.

And different red giants are different. So that's going to be different for different type 2 supernova. A type 1 supernova is supposed to be when a white dwarf. has a companion star that is gradually dribbling mass onto it, and eventually it hits the Chandra Sagar limit. It collapses and blows up, and that's a type 1. Unlike the red giant explosions when they give out their nuclear fuel, the Shandra Saker limit is universal.

All the white dwarfs have the same Shandra Saker limit. So you can kind of see why Type 1A supernovae should be the same brightness. And they're very bright. That's the good thing. Now, they're not exactly the same brightness, and that's an important thing. I'll talk about that in a second, but they're approximately. So that means you can go approximately how far away they are, right? At least relative to other type 1A supernovae.

And if you find a type 1a supernova in a galaxy that you know the distance to already, you can figure out what that absolute brightness is. So in Bob's mind, Bob worked on type 1A supernova also, Bob Kirchner. And the graduate student who he put on that task was Adam Reese, who was not my office mate, but was in the floor below us. So Harvard and where I was a grad student was like the center for cosmological supernova observations at that time. But the focus was on measuring the Hubble constant, not the cosmological constant. The Hubble constant is a natural first thing to try to measure. There was an idea, though, what if we didn't wait around to find supernovae,

by accident, but we made a dedicated program to go look for them. And this idea is very, very telling historically. It's a very good lesson in the history of science. It wasn't the astronomers who had that idea. It was the particle physicists. Saul Promoter in particular was a particle physicist who said, like, why are you just waiting around for these supernovae to happen? Like, let's get a big telescope, look at a lot of galaxies, and let's, in a

deterministic, no-aheadable way, find a bunch of supernovae. and measure their distances. That would be the better way to do it. Particle physicists think big, right? They don't want to like take pictures of galaxies and hope something interesting happens. They want to be systematic and go over the whole sky. And I remember having a long phone conversation with Saul and Ariel Gubar, who was his co-author on one of the very early papers where they talk about could we measure the cosmontrial constant doing this.

Because I was still a grad student at the time and a nobody, but I had been an author on this review article with Bill Press and Ed Turner. that they knew about. So I became friends with Saul, et cetera, and of course I was also friends with Bob and Brian and Adam and everything. And I still think there's a good history of science book to be written. This would not be by me, but by somebody about everything that went down in that time. Because basically, Saul and his collaborators formed the high,

sorry, formed the supernova cosmology project, the SCP. That's what they called it. And they were largely ex-particle physicists who said, damn the torpedoes, we're going to do this. And they, you know, had the ability to get a lot of money from the Department of Energy. And they had, you know, good expertise at computer programming to reduce the data and things like that. And they just went out and did it. The astronomers, including Bob Kirshner, very noticeably, were very skeptical that it would ever work. Because for good reasons, not for bad reasons.

They weren't just protecting their turf. They knew the pitfalls. They knew the possible ways everything could go wrong. They knew that not all the type 1A supernovae were the same. They had data saying that. There was a method that Bob and others actually pioneered, and Adam was involved with it, and I think Bill Press was involved with it, that tried to standardize the candles.

So not all type 1A supernovae are exactly the same brightness, but if you plot the amount of time it takes for them to get brighter and dimmer, that's related to their absolute brightness. This is the Phillips relation after Mark Phillips. And so they were trying to figure out how to reduce a heterogeneous sample of many supernovae down to a homogeneous sample and make them all behave the same. Then they would be actually acting as standard candles. So they're standardizable candles, even though they're not standard. And the point was they were so impressed with all the difficulties that they thought that, you know, this pole motor thing was never going to work.

And I, my understanding is, my understanding is very incomplete because it's just from my own knowledge. It was Nick Sunsef and Brian Schmidt. Brian and Adam had both by this point graduated and moved on to other things. But, you know, they said, you know what? I think we can do it. In fact, I think that these are the astronomers saying, I think we can catch up because we know astronomy better than those particle physicists do. And so Saul and his group started first, but needed to. learned the astronomy, Brian and Nick and their group, which eventually became known as the

high Z supernova team, Z is for Redship, so high Redshift super high Redshift supernova team, started later but caught up, okay, and they were very good at reducing the data and whatever. And so Adam and Bob and many other people joined that group as well. And what happened is they were both right. Saul was absolutely right in saying that it could be done and starting it. and Brian and Adam and Bob and Nick and all those other people were also writing that you need to be careful and get it right and you know do it correctly. And they were plugging away. And I remember very, it's very cute story.

I remember getting in the mail a list, a set of papers that the astronomy group had written, the high C supernova team. And so they were advertising their work. So they like, we didn't have the archive. I guess we did have the archive back then, but still people would mail papers to each other. So they had put a bunch of the papers they had written into like a spiral notebook and mailed it to people who they thought might be interested. And the title page was the high Z supernova project measuring the deceleration of the universe. The irony being that what they actually ended up measuring was the acceleration of the universe. But no one expected that at the time.

They thought they would measure the Hubble constant and maybe they could measure the critical density. or the density, I should say, the density parameter of the universe, which would tell us did we have the critical density or not. And so it was in 1998 that both teams came out with their results. But let me tell you one more thing about 1997. In 1997, I was a postdoc at the Institute for Theoretical Physics in Santa Barbara. And I was doing, you know, some crazy things with topological defects and extra dimensions and stuff like that. you know, who knows what I was working on at the time.

But Phil Lubin was, is a observational cosmologist, a microwave background guy who's a professor in Santa Barbara. And he organized a little workshop December 1997. And basically it was on determining cosmological parameters, like I said, the Hubble constant, cosmological constant, etc. from the cosmic microwave background. They didn't have the data to do that in 1997, but they were working on how to do it. So they had a workshop to do that. And Phil asked me, could I give a talk at the workshop? I knew nothing about how to extract cosmological parameters from the microwave background,

but he said, could you do us a favor and give a talk on measuring cosmological parameters in all the ways that are not the cosmic microwave background? So from galaxies and supernovae and things like that. So I said, sure, and I did that. And I, you know, I gave this talk. And it was only because I was forced to do it because I was giving the talk, but I went through an enormous amount of papers on modern observational cosmology from very different perspectives. And it was super informative because over and over again. So this is 1997.

The favorite theorist's model of the universe was the critical density of matter, the good old flat cosmontory universe. the observer's favorite universe was an open universe with only 0.3, the critical density of matter. And over and over again, there were from different angles things in the data saying the theorist's favorite universe can't be right. We didn't know what it was. I gave a talk and I said, look, I don't know what's wrong, but there's enough data here already in 1997 to say that the theorist's favorite model of omega equals 1, as we would say, the density premise, the universe is exactly the critical density, and it's all made of ordinary matter, and that's what inflation predicts and things like that, that model can't be right. Maybe it's because the universe is open, and there's not enough matter to make the critical density. Maybe it's because there's a cosmological constant.

Maybe it's because dark matter is warm or mixed rather than cold, or maybe there's a tilt in the spectrum of things. There's a whole bunch of different possibilities. I didn't know which one it was, but I knew that something was going on. And that was the mood in cosmology at the time. So in February 1998, when the Hisei Supernova team led by a paper that Adam Reese was the first author on, said, you know what, we did our measurements, it looks like there is a non-zero cosmological constant. And a couple months after that, Prometer and his group came out and said, yep, that's what we got to.

We're both getting exactly the same answer. This was a very dramatic claim. You know, Adam and others have explained in interviews that they didn't believe it themselves when they got it. They're like, it's very easy to make a mistake. Maybe your computer program was wrong. Maybe you put in the wrong number somewhere. You don't want to write a paper saying the universe is accelerating unless you know you're on the right track. But eventually they couldn't make it go away.

So they published their paper. And I had a little bit of an inside scoop, right? I was friends with all these people. So I got to, I got copies of the plots and I got to give talks in Santa Barbara telling everyone the news and everyone was super duper excited. But they were willing to accept it very, very quickly exactly because it made everything snap into place. It's again, a super interesting lesson for the history of science. Not all claimed dramatic discoveries are created equal. If you claim a dramatic discovery that is literally flying in the face of everything,

we think we know, people are correctly going to be skeptical of it. Whereas if you claim a dramatic discovery that makes everything fit together that we think we know, people are much more likely to believe it right away. Not everyone did. There were some grumpy people who said, you know, they didn't believe the supernova stuff. That's okay. There's always a place in physics for a couple grumpy people as long as eventually they realize that the data become overwhelming. And then, long story short, it's already pretty long, but in the early 2000s, we did get the cosmic microwave background data

from both ground-based telescopes and from the W-Map satellite and elsewhere, and the CMB confirmed the picture of the universe that have been put together by the supernova data, where 0.7 of the critical density is the vacuum energy, the cosmological constant, and 0.3 of it was the matter, just like the astronomers have been saying all along.

So that is the still best fit 20 years later. Lambda CDM cosmology. Lambda is the letter we give to the cosmological constant. CDM is for cold dark matter. And that's the best we can do for fitting the universe and it fits it very, very well. It's not 100% perfect. You know, we talked to both Adam Reese and Mark Kamienkowski at different times here on Mindscape. And they explained to us that there are things like the Hubble tension that don't completely fit in.

but that's okay. That's what drives progress in science. The Hubble tension is something where we don't have a good theoretical model. It's not making everything snap together. So we're not sure whether that's going to go away or not. But roughly speaking, we have a model that fits really well. And the microwave background in the early 2000s confirmed everything that the supernovae had been telling us. So where does that leave us?

Well, it's a good fit to the data. The coincidence problem, the fact that, you know, we live in the unique era in the universe, where both matter and vacuum energy seem to be relevant is still very much with us. The way I like to put it is you should think about the coincidence problem and the cosmological constant problem in the following way. The best solution that we have on the market right now is the anthropic principle. And that's not a good solution. That's a very messy solution. There are things about it we don't understand theoretically.

It involves a lot of speculative physics. It's still better than any of the other alternatives. Again, not to say that it's right. But that is the state of the art. One thing that is, you know, still percolating in the back of people's minds is the cosmontal constant problem, different than the coincidence problem, right? Why is the vacuum energy so small? That's just been made harder by the fact that it's not zero, right? Again, in the minds of both theorists, it was easier to think of something that would set it all the way to zero than something that would set it to 10 to the minus 122 times the plank scale.

But that's apparently where it is. It might be, if you want to think dramatically, it might be indicating that this whole effective field theory paradigm fails when gravity and cosmology are in the game. On the one hand, if you just say it casually, it doesn't sound that crazy. Like, yeah, gravity is weird. Quantum gravity is weird. It's not an effective field theory.

Okay. On the other hand, like, you know, even gravity should fit into the effective field theory paradigm from everything that we know, so it's not really clear why it doesn't. On the third hand, some people had boldly said, yeah, maybe that's what's going on and tried to figure out what that means for the cosmontual constant problem.

I would say Tom Banks and Willie Fishler, who I mentioned earlier in the previous solo podcast, as really pushing the idea that the dimensionality of Hilbert space is a finite number. They've also been saying, you know, if you take that whole idea seriously, it completely changes the cosmotrial constant problem because the dimensionality of Pilbilt space is related to the cosmological constant.

And it therefore is not an ordinary parameter in an effective field theory. So I'm saying these words out loud without really understanding them or expecting that you understand them. But I'm just trying to let you in on how some physicists are thinking about these big issues right now. It really is, it's only one number that we've measured, the cosmological constant. So it's very hard to build detailed models on top of that. We don't have that much guidance. But that one number is spurring us to new ideas. It does imply if the cosmontrial constant is truly a constant, it does imply that we know about the future of the universe now. Like when I was your age, we were still taking very seriously the idea of maybe the universe would recalapse someday. We don't know.

If it's truly a cosmontual constant, then that is making the universe accelerate right now, There's no reason for that not to continue forever. The cosmontal constant doesn't go away. It's a constant. The density of energy and empty space is a number. It is about 10 to the minus 8 ergs per cubic centimeter. That will never go away. The universe will get emptier and colder and more desolate and lonely forever and ever.

So enjoy the time you have in the universe right now. That would be the lessening. Or the other possibility is maybe the cosmational constant is zero. maybe we were right all along back in the 80s when we thought that in the space of all possible ideas we haven't come up with yet, it is easier to think of an idea that sets the cosmological constant to zero than to some small number. And you say to yourself, well, wait a minute, you just told me that the data don't allow that, but that wasn't quite true. The data say there is something making the universe accelerate, but maybe that thing is not the cosmological constant. The cosmological constant is a very, very good fit to the data right now, but we were surprised when we found it. And so we should be open-minded.

And being open-minded means we should think about the possibility that what is making the universe accelerate is not the cosmological constant, but something else. So the term dark energy is the catch-all term for something else, or even including the cosmological constant. The cosmontal constant itself is a form of dark energy. But there are other forms. There's other theoretical models for kinds of dark energy that might change with time just a little bit. The cosmontal constant is constant. Dynamical dark energy can change just a little bit. And that opens up an enormous number of possibilities.

If I were you, I would return for the second installment of this solo exploration of what's making the universe accelerate next time here on the Mindscape podcast. Thanks for listening. Talk to you next time. The