The simplest explanation for dark energy is that it is the cosmological constant. This means that any volume of space, whether it is a perfect vacuum or filled with matter, has an intrinsic fundamental energy associated with it. As the volume increases so does this vacuum energy. It is the energy density of an empty vacuum. This energy has negative pressure, meaning that this force results in expansion, against what you might intuitively think - positive pressure in fluid mechanics makes gases expand. Surprisingly, general relativity can describe both attractive gravity and this repulsive type of force. In fact, Einstein thought that the universe was static and he called upon this repulsive force to counteract gravity, which wants to collapse matter in on itself. He called this force the cosmological constant and latter, upon learning that the universe is expanding, threw the notion out, a rare blunder for an amazing genius, but then not quite. It's back in vogue but for an entirely new reason - to explain why the expansion rate is increasing!
Let's briefly compare the workings of gravity with vacuum energy. Gravity in general relativity doesn't depend just on the distribution of matter in space. It depends on both mass and energy (recall their equivalence). This means that all kinds of energy including photons as well as the momentum of objects and particles determine the gravitational force, in addition to rest mass. And it means both the energy density and the momentum density of space must be calculated. The gravitational force at any point in space can be described in detail using an energy-momentum tensor. I describe what tensors are in the article Gauge Theory, if you are curious about them. This is a rank-2 tensor that encodes the energy density in matter, the energy flux, the momentum density and the momentum flow (pressure) at any point in space. It is a force that is proportional to energy density plus 3 times pressure (G is the gravitational constant and c is the speed of light), shown below to give you a basic idea what the formula looks like.
Quantum theory tells us that the pressure of dark energy is equal and opposite the energy density at any given point in space. This is sometimes explained by treating space like a gas of particles. In the quantum world, virtual particles and their antiparticles pop in and out of existence all the time. This imparts a certain amount of energy into the system - vacuum energy. Vacuum energy increases as the volume increases, so work must be done to expand the universe's inventory of vacuum energy. Work must go in, in other words. This expenditure of work makes the force repulsive rather than attractive, in contrast with gravity, which can do work - work comes out. Going back to the gas laws, this is what lends vacuum energy its negative pressure.
Now we can describe vacuum energy by using the following simplified equation, which becomes p + 3p = p - 3p. Pressure (p) is equal and opposite energy density(p)) so we can swap those therms out. This leaves us -2p, a negative pressure. An analogy is a rubber ball that is inflated past its normal pressure. It will want to explode.
These equations are efforts to try to understand how dark energy works in spacetime. Dark energy does not fit well into our current theoretical framework. Trying to calculate the energy density of dark energy using quantum field theory or general relativity give you either zero or infinity for answers, respectively. Trying to reconcile these theoretical answers with the (very well established) observational measurement of about 6 x 10-10 J/m3 is a problem that points yet again to the fact that general relativity and quantum mechanics don't yet fit together (and almost every physicist thinks they should - logically, there should be a single unified theoretical framework that describes all behaviours in the universe).
Intense negative pressure was also created very early in the universe's evolution, during cosmic inflation. However, the mechanism there, according to most physicists, seems to be attributed to the fact that the particle field, which was created at that moment, was initially way out of equilibrium.
Quantum vacuum fluctuations (virtual particles popping in and out of existence) suggest that even a perfect vacuum is teaming with energy fluctuations at the quantum level. Normally we would think that this kind of quantum effect would be smoothed out or erased at the macroscopic level, and vacuum energy would even out to zero on large scales, but because vacuum energy is quantized it cannot diminish down to zero - it must have some non-zero lowest possible energy, called zero-point energy.
Is Vacuum Energy Useable?
Gravitational energy is usable so why not vacuum energy? There seems to be a vary large and continuously growing supply of it, right? There is a very real and measurable energy associated with a vacuum, as evidenced by the well-verified Casmir effect illustrated in the diagram below right.
Physicists can measure a tiny suction force created between two very close metal plates in a perfect vacuum where the longest of these quantum waves do not have the physical space required to form. Because some EM waves are eliminated between the plates, that space between them contains less vacuum energy than the space outside them. This (very tiny) unbalanced force pushes (or sucks if you want) the plates closer together. Sometimes people extrapolate from this example to the idea that dark energy is directly creating the suction force. It is not - the force is better described as an imbalance of forces. However, it does prove that even vacuums contain energy fields and energy fields contain energy, which can be manipulated to do work on a system.
This is also direct evidence of a physical force arising from a quantized field. In this sense, we must think of a vacuum as having an underlying complex structure of some sort that supplies these fields. Many physicists use the analogy that the underlying quantum fields in a vacuum act like a kind of web made up of interconnected vibrating balls. You can think of these balls as the various boson force particles in quantum field theory. The strong, weak and electromagnetic forces can all be described this way. Not gravity - a quantum framework for general relativity doesn't yet exist.
This ball-string arrangement makes each point in space a kind of harmonic oscillator. Each field is quantized so it must also have a lowest possible energy or ground state. It must retain at least a tiny bit of "jiggle" to it. The uncertainty principle requires that this zero-point energy be greater than the minimum of its classical potential energy (which can go down to zero). A fascinating analogy can be drawn for matter. Even at absolute zero, where there is no useable energy in a system (this means that it cannot do any work), there must still be some tiny minimal jiggle of atoms and this is why liquid helium will not freeze, ever (but it will solidify under tremendous pressure). This is one example where you can actually "see" what is usually a sub-microscopic quantum effect.
The Casimir effect is another example. It has only been measured for photons because only that force is measurable at this experimental scale, but in theory, all bosons, including photons, produce an attractive Casimir-type force while fermions (particles of matter) produce a negative or repulsive force (arising from the Pauli exclusion principle).
The Casmir effect has led many people to wonder whether we could harness vacuum energy to do work. DARPA (a U.S. defense research department), for example, launched a $10 million project in 2009 to attempt to harness the energy behind the Casmir effect.
Let's examine the thermodynamics of the Casimir effect a little closer. Potential energy in the form of vacuum energy is converted into kinetic energy - the plates move together if they are free to do so, and heat energy can be created as well if the plates then collide. This line of reasoning is analogous to what happens in free fall. An object's gravitational potential energy is converted into kinetic energy, and then heat is created through friction when the object hits the ground. A problem arises right at the beginning with this potential energy. Is the Casimir effect potential energy coming from the EM field or from vacuum energy? I mentioned previously that work is required to expand the universe's inventory. Is there energy, therefore, available to do work? The majority of physicists consider the idea of harnessing dark energy a non-starter, analogous to the classic perpetual motion machine.
Thermodynamic problems also arise when we think about the expanding universe and the ever-increasing supply of vacuum energy that comes with it. If the universe is an isolated system, it's total energy should be conserved; it should stay the same (this assumes there is no interaction with other universes if they exist).
It might be interesting here to investigate the idea of energy conservation further by revisiting the CMB photons traveling through space in the expanding universe: Imagine this finite number of photons shooting around in all directions. As the volume of space expands, the photons redshift. This means that the energy of each photon (and therefore, the total EM radiation energy) decreases, resulting in a loss in total photon energy over time. Assuming that few of the photons lost energy through interactions with other particles, where did that energy go? I've read some online claims that the energy goes into some unknown quantum field or perhaps into the gravitational field. Perhaps this is a trick question. We observe the photon energy as decreasing because we are looking at the photons from our reference point. If we were moving alongside the photons, they would appear to maintain exactly the same energy they started out with. In fact, it seems to me that neither us nor nor the photons would have experienced any time passing at all, according to special relativity. Could dark energy be an example of a relativistic situation as well? The Casimir experiment would seem to claim otherwise. Nothing there is traveling at relativistic speeds. But then is the Casimir effect showcasing not dark energy but the quantized nature of EM energy?
One last question: Is that unknown field, that some online sources suggest, the vacuum energy field? Or, is it better to simply think of the photon energy as conserved in the sense that a much larger space now heated to 2.73 K (current CMB temperature) is thermodynamically equivalent to a much tinier space that was once heated to 4000 K (recombination epoch temperature)? This is simply the gas pressure-temperature law at work, and it means that no energy is lost when volume is taken into account.
Physicists maintain that the total energy of the universe is conserved, even though it seems that the expansion of space, with its ever-increasing supply of dark energy, messes with the rules a bit. Perhaps it is more accurate to say that quantum mechanics messes with essentially classical thermodynamics laws? Dark energy as the cosmological constant probes the non-zero energy requirement of the quantum uncertainty principle. None-the-less, most physicists claim that vacuum energy does not violate the laws of thermodynamics because it is energy that is not available to do work. However, if you do a quick Google search, you will find many recent published papers online that suggest ways that we could harness vacuum energy. Wikipedia deals with this conundrum briefly. I had fun examining some of them but I too am not convinced (a least as far as I can understand the problem).
Still, does the question for you remain - if dark energy draws from a seemingly inexhaustible reservoir of vacuum energy, does this break the second law of thermodynamics? Does the requirement of not being able to do work hold for you or not? Was my earlier statement that the two pie charts represent identical total mass-energy correct? Is the universe truly an isolated system? Or, do thermodynamic conservation laws fall short of explaining the quantum universe? It seems to me that there is still plenty of room to play with the nature of dark energy itself before we must start rewriting physics. Dark matter, on the other hand (tackled in the previous article) seems far more likely to eventually force a rewrite of gravity, depending on what we see in upcoming collider experiments.
Perhaps all this wrangling isn't necessary. The cosmological constant is not the only contender for dark energy, as we'll see next in Dark Energy Part 12.