Figure 1: The cosmological constant was originally introduced by Einstein in 1917 as a repulsive force required to keep the Universe in static equilibrium. In modern cosmology it is the leading candidate for dark energy, the cause of the acceleration of the expansion of the universe.

In the context of cosmology the cosmological constant is a homogeneous energy density that causes the expansion of the universe to accelerate. Originally proposed early in the development of general relativity in order to allow a static universe solution it was subsequently abandoned when the universe was found to be expanding. Now the cosmological constant is invoked to explain the observed acceleration of the expansion of the universe. The cosmological constant is the simplest realization of dark energy, which is the more generic name given to the unknown cause of the acceleration of the universe. Its existence is also predicted by quantum physics, where it enters as a form of vacuum energy, although the magnitude predicted by quantum theory does not match that observed in cosmology.

History

The cosmological constant first appeared in a 1917 paper by Einstein entitled "Cosmological Considerations in the General Theory of Relativity" (Einstein 1917), in which he motivates its introduction into the field theory of general relativity by the need to stabilize the universe against the attractive effect of gravity:

"The term is necessary only for the purpose of making possible a quasi-static distribution of matter, as required by the fact of the small velocities of the stars" (Einstein 1917).

At the time, observations of our universe were limited primarily to stars in our own galaxy, so there was indeed observational evidence justifying the assumption that the universe was static.
Einstein's goal was to obtain a Universe that satisfied Mach's principle of the relativity of inertia (for a historical discussion see Pais 1982, Sect 15e), and construct a cosmology that was finite, yet stable against gravitational collapse. The attempt proved futile, as shortly thereafter de Sitter (1917) demonstrated an empty universe solution to Einstein's equations (allowing inertia relative to space empty of matter) and Friedmann (1922) derived solutions to Einstein's equations that corresponded to an expanding universe. These results could be considered a prediction that the universe must be expanding or contracting, a remarkable implication of general relativity that was later borne out by observation.
When Hubble observationally discovered the expansion of the universe Einstein finally abandoned the cosmological constant completely (Einstein 1931).

In the intervening years the cosmological constant came in and out of vogue as new observational results repeatedly seemed to require it, but then were explained in other ways. As of the early 1990s there were tantalising hints that the cosmological constant might again be needed. The universe appeared to be younger than the oldest stars it contained, a feature that was remedied if the universe was currently in an accelerating phase. Number counts of galaxies indicated that the volume contained within a solid angle at high redshift was larger than expected in a decelerating universe. Theoretical arguments from inflation and later observational results from the cosmic microwave background radiation indicated that the universe should be flat, but observations of large scale structure indicated that the matter density was inadequate to achieve this -- vacuum energy could make up the shortfall.

Figure 2: The mass-energy composition of the Universe. The cosmological constant is one possible form of dark energy, which is believed to be behind the acceleration of the expansion Universe.

This set the stage for the discovery of the accelerating universe by two teams in 1998/1999. The High-Z supernova team and the Supernova Cosmology project both discovered that high-redshift supernovae were fainter than expected for a decelerating universe and that the difference could be explained if there was a cosmological constant of just the right magnitude needed to make the universe flat.

This was a dramatic convergence of observation and theory. Since then increasingly accurate probes have confirmed to high precision the need for dark energy, but the nature of the dark energy is now the issue being investigated. As of 2010 the measured properties of dark energy remain consistent with those of a cosmological constant. However, massive observing efforts are underway to test whether this is the correct explanation for the acceleration or whether some other sort of dark energy, perhaps one that changes with time or one that is motivated by some form of quantum gravity, is needed to explain the acceleration we see.

Greatest Blunder

In his autobiography, My World Line, George Gamow reported on a conversation he had with Einstein about the introduction of the cosmological constant into the field equations.

Much later, when I was discussing cosmological problems with Einstein, he remarked that the introduction of the cosmological term was the biggest blunder of his life.(Gamow 1970).

This second hand account has become one of the most frequently repeated quotes in cosmology.

The physics of the cosmological constant

To explore more deeply the nature of the Universe, we must use the mathematical language in Einstein's general relativity to relate the geometry of space-time (expressed by the metric tensor, \(g_{\mu\nu}\)) to the energy content of the universe, (expressed by the energy-momentum tensor, \(T_{\mu\nu}\)).

Einstein Field Equations

Arguably, one of Einstein's most significant discoveries was that the distribution of energy determines the geometry of space-time, which is encoded in his field equation,

where \(G\) is the gravitational constant.
Although this is the simplest form of the equations the freedom remains to add a constant term. This "cosmological constant" was what Einstein added in order to achieve a static universe, and it is given the symbol \(\Lambda\ .\)

Vacuum Energy

Figure 3: The equation of state, \(w\ ,\) describes the relationship between pressure and density in a material according to \(w=p/\rho\ .\) Here are some examples of the equation of state for common fluids. When matter is at rest (pressureless dust) it has \(w=0\ ,\) but as it picks up velocity (\(v\)) its equation of state increases until \(w\rightarrow1/3\) as \(v\rightarrow c\ .\)

Vacuum energy arises naturally in quantum mechanics due to the uncertainty principle. In particle physics the vacuum refers to the ground state of the theory -- the lowest energy configuration. The uncertainty principle does not allow states of exactly zero energy, even in vacuum (virtual particles are created). Since in general relativity all forms of energy gravitate, this ground state vacuum energy impacts the dynamics of the expansion of the universe.

Vacuum energy should not have any dissipative processes such as heat conduction or viscosity, so it should take the form of a perfect fluid,

\[\tag{3}
T_{\mu\nu} = (\rho + p)U_{\mu}U_{\nu}+p g_{\mu\nu}.\]

In order to maintain Lorentz invariance, vacuum energy should also have no preferred direction. Therefore the first term in the perfect fluid energy tensor must be zero, requiring

\[\tag{4}
p^{vac}=-\rho^{vac},\]

which corresponds to an equation of state \(w^{vac}=p^{vac}/\rho^{vac} = -1\ ,\) and results in an energy-momentum tensor for vacuum energy,

Equivalence of Cosmological Constant and Vacuum Energy

We can split the energy-momentum tensor into a term describing the matter and energy, and a term describing the vacuum, \(T_{\mu\nu} = T_{\mu\nu}^{matter} +T_{\mu\nu}^{vac}\ .\) Einstein's equation including vacuum energy becomes,

So vacuum energy and the cosmological constant have identical behaviour in general relativity, as long as the vacuum energy density is identified with,

\[\tag{7}
\rho^{vac} = \frac{\Lambda}{8\pi G}.\]

Cosmology

Figure 4: Aleksandr Friedmann (1888-1925) derived the famous equation which bears his name in 1922, shortly before his untimely death aged just 37 from Typhoid fever. George Gamow (1904-1968) was one of his students.

In an homogeneous, isotropic universe the geometry is defined by the Friedamnn-Lemaître-Robertson-Walker metric (FLRW metric) and the dynamics of the universe are governed by the Friedmann equations (Friedmann equations). The dynamics are driven by the energy content of the universe and the equation of state of the components that make up the energy density. The equation of state relates density \(\rho\) to pressure \(p\) according to \(w = p/\rho\ .\) The cosmological constant enters these equations in the following way, where \(a\) is the scale factor of the universe normalized to 1 at the present day, \(H=\dot{a}/a\) is Hubble's constant (an overdot represents differentiation with respect to time), and \(k\) is the curvature of the universe given by +1, 0, and -1 for positive, flat, and negative curvature respectively,

These equations are more concisely written by considering both the cosmological constant and curvature as forms of energy density (\(\rho_\Lambda = \Lambda / 8\pi G\) and \(\rho_k = -3k/8\pi G a^2\)). Then Eqs. (8) and (9) become

The current energy density of each component, \(\rho_{i0}\ ,\) is often represented as a fraction of the critical density, \(\rho_{\rm c}=3H_0^2/8\pi G\ ,\) which is the energy density required to close the universe (also calculated at the present day). Denoting this \(\Omega_i = \rho_{i0}/\rho_{\rm c}\) and using Eq. (12) allows us to write

For pressure to do work there needs to be a pressure gradient -- a relatively high pressure region next to a relatively low pressure region -- that will then cause movement from high pressure to low. In a homogeneous universe there are no pressure gradients, so a positive pressure does no work and has no expanding effect (there are no low-pressure regions for it to push matter into). On the contrary, in general relativity all forms of energy gravitate so pressure effectively pulls, strengthening the attractive force of gravity (thus the factor of \(p\) in Eq. (9), which does not appear in Newtonian gravity). The cosmological constant has negative pressure, \(w=-1\ ,\) so its general relativistic contribution counteracts the normal force of gravity and provides an outwards acceleration.

Observational evidence

Observational evidence for the accelerating universe is now very strong, with many different experiments covering vastly different timescales, length scales, and physical processes, all supporting the standard \(\Lambda\)CDM cosmological model, in which the universe is flat with an energy density made up of about 4% baryonic matter, 23% dark matter, and 73% cosmological constant. For more detail and references see the review by Frieman, Turner and Huterer 2008.

The critical observational result that brought the cosmological constant into its modern prominence was the discovery that distant type Ia supernovae (0<z<1), used as standard candles, were fainter than expected in a decelerating universe (Riess et al. 1998, Perlmutter et al. 1999). Since then many groups have confirmed this result with more supernovae and over a larger range of redshifts. Of particular importance are the observations that extremely high redshift (z>1) supernovae are brighter than expected, which is the observational signature that is expected from a period of deceleration preceding our current period of acceleration. These higher-redshift observations of brighter-than-expected supernovae protect us against any systematic effects that would dim supernovae for reasons other than acceleration.

Prior to the 1998 release of the supernova results there were already several lines of evidence that paved the way for the relatively rapid acceptance of the supernova evidence for the acceleration of the universe. Three in particular included:

Figure 5: The relative size of the universe as a function of time for a flat universe made entirely of matter (red) and one made of 30% matter and 70% cosmological constant (green). In both cases the zero of time corresponds to the present day, and that has been defined so that the slope matches the current expansion rate of the universe (Hubble's constant is taken to be 70 km/s/Mpc). Both types of universe would have initially decelerated, but the universe with the cosmological constant later switched and started accelerating. The cosmological constant universe is older because it took longer to reach its present rate of expansion (13.5 Gyr) than the matter-only universe (9.3 Gyr).

The universe appeared younger than the oldest stars.

Stellar evolution is well understood, and observations of stars in globular clusters and elsewhere indicate that the oldest stars are over 13 billion years old. We can compare this to the age of the universe by measuring the universe's rate of expansion today and tracing that back to the time of the big bang. If the universe had decelerated to its current speed then the age would be lower than if it had accelerated to its current speed (see Figure 5). A flat universe made only of matter would only be about 9 billion years old -- a major problem given that this is several billion years younger than the oldest stars. On the other hand, a flat universe with 74% cosmological constant would be about 13.7 billion years old. Thus the observation that the universe is currently accelerating solved the age paradox.

There were too many distant galaxies.

Galaxy number counts had already been used widely in attempts to estimate the deceleration of the expansion of the universe. The volume of space between two redshifts differs depending on the expansion history of the universe (for a given solid angle). Using the number of galaxies between two redshifts as a measure of the volume of space, observers had measured that distant volumes seemed too large compared with the predictions of a decelerating universe. Either the luminosity of galaxies or the number of galaxies per unit volume was evolving with time in an unexpected way, or the volumes we were calculating were incorrect. An accelerating universe could explain the observations without invoking any strange galaxy evolution.

The observed flatness of the universe despite insufficient matter.

Using measurements of temperature fluctuations in the cosmic microwave background radiation (CMB) from when the universe was ~380,000 years old one can conclude that the Universe is spatially flat to within a few percent. By combining these data with accurate \(H_0\) measurements and/or measurements of the matter density of the universe, it becomes clear that the matter in the Universe only contributes approximately 23% to the critical density. One way to account for the missing energy density would be to invoke a cosmological constant. As it turns out, the amount of cosmological constant needed to explain the acceleration observed in the supernova data, was just what was needed to also make the universe flat. Therefore the cosmological constant solved the apparent contradiction between the matter-density and CMB observations.

Unresolved issues

Cosmological Constant Problem

The cosmological constant problem arises because, using naive naturalness arguments in quantum field theory, one cannot explain why the observed cosmological constant is so small. Quantum mechanical calculations that sum the contributions from all vacuum modes below an ultraviolet cutoff at the Planck scale give a vacuum energy density of \(\rho_\Lambda\sim10^{112} {\rm erg/cm}^3\ .\) This exceeds the cosmologically observed value of \(\rho_\Lambda\sim10^{-8} {\rm erg/cm}^3\) by about 120 orders of magnitude. See for instance Weinberg (1989) and Carroll (2004), Section 4.5.

Coincidence Problem

The cosmological constant is not diluted as the universe expands, whereas the density of matter drops in inverse proportion to the volume. This means that there is only a fleeting moment of cosmological time during which the matter density will be of comparable magnitude to the vacuum energy density. Many argue that to be living in that moment is too unlikely to be coincidence. This has been called the coincidence problem, and has motivated theories beyond the cosmological constant with more general forms of dark energy that may change with time.

Dark energy or cosmological constant

These unresolved issues have motivated the current observational effort to test whether the cosmological constant is a valid cause of the acceleration of the universe. Other theories, such as fledgling theories of quantum gravity (e.g. brane-motivated cosmologies), naturally produce dark energy candidates with properties different from the standard cosmological constant (Padmanabhan, 2003). Phenomenological theories such as quintessence have also been proposed, which have a time-varying value of dark energy. Although these models are designed partially to negate the coincidence problem by having dark energy solutions that can track the matter density, these then suffer a new fine-tuning problem as they introduce additional parameters whose values need to be fine-tuned to produce the evolution needed (Weinberg 1989, 2000).

Anthropic Solutions

Many argue that the coincidence problem is most simply solved by anthropic considerations. That is, were the value of the cosmological constant much higher or lower than the observed value it would disrupt structure formation in the universe and humans would not exist. Although many argue against anthropic solutions on philosophical grounds in preference for solutions that invoke some deeper physical principles, anthropic arguments are gaining more prominence, especially in light of the emergence of the string landscape. It had been hoped that string theory would give a well motivated fundamental explanation for the values of the constants of nature, but now it seems ours is only one of many possible solutions, which we find ourselves in by chance constrained by anthropic requirements (Polchinski, 2006).

Dark gravity

Dark energy also encompasses the possibility that there is no additional energy density component to the universe, but rather that the equations of general relativity need revision. In this sense general relativity might be a limit of a more complete theory of gravity in the same way that Newtonian gravity is a low-energy limit of general relativity. This possibility is also known as dark gravity.

Further reading

For a straightforward overview, including observational constraints and theoretical considerations see Carroll (2001).

For an excellent historical summary of the cosmological constant see Section 3 of Frieman, Turner, and Huterer (2008), and continue reading for a quantitative review of dark energy in general, both the theory and the observational evidence.

For a technical summary, especially of the multi-faceted theoretical possibilities for the cosmological constant, see Padmanabhan (2003).