October 2023

The Most Beautiful Theory

“Alfred, it’s spinning.” Roy Kerr, a New Zealand-born physicist in his late 20s, had, for half an hour, been chain-smoking his way through some fiendish mathematics. Alfred Schild, his boss at the newly built Centre for Relativity at the University of Texas, had sat and watched. Now, having broken the silence, Kerr put down his pencil. He had been searching for a new solution to Albert Einstein’s equations of general relativity, and at last, he could see in his numbers and symbols a precise description of how space-time—the four-dimensional universal fabric those equations describe—could be wrapped into a spinning ball. He had found what he was looking for.

When this happened, in 1962, the general theory of relativity had been around for almost half a century. It was customarily held up as one of the highest intellectual achievements of humanity. And it was also something of an intellectual backwater. It was mathematically taxing and mostly applied to simple models with little resemblance to the real world, and thus not widely worked on. Kerr’s spinning solution changed that. Given that pretty much everything in the universe is part of a system that spins at some rate or other, the new solution had relevance to real-world possibilities—or, rather, out-of-this-world ones—that previous work in the field had lacked. It provided science with a theoretical basis for understanding a bizarre object that would soon bewitch the public imagination: the black hole.

General relativity was presented to the Prussian Academy of Sciences over the course of four lectures in November 1915; it was published on December 2^nd that year. The theory explained, to begin with, remarkably little, and unlike quantum theory, the only comparable revolution in 20th-century physics, it offered no insights into the issues that physicists of the time cared about most. Yet it was quickly and widely accepted, not least thanks to the sheer beauty of its mathematical expression; a hundred years on, no discussion of the role of aesthetics in scientific theory seems complete without its inclusion.

When gravity fails

Today its appeal goes beyond its elegance. It provides a theoretical underpinning to the wonders of modern cosmology, from black holes to the Big Bang itself. Its equations have recently turned out to be useful in describing the physics of earthly stuff too. And it may still have secrets to give up: enormous experiments are underway to see how the theory holds in the most extreme physical environments that the universe has to offer.

The theory was built on the insights of Einstein’s first theory of relativity, the “special theory”, one of a trio of breakthroughs that made his reputation in 1905. That theory dramatically abandoned the time-honoured description of the world in terms of absolute space and time in favour of a four-dimensional space-time (three spatial dimensions, one temporal one). In this new space-time, observers moving at different speeds got different answers when measuring lengths and durations; for example, a clock moving quickly with respect to a stationary observer would tell the time more slowly than one sitting still. The only thing that remained fixed was the speed of light, c, which all observers had to agree on (and which also got a starring role in the signature equation with which the theory related matter to energy, E=mc²).

Special relativity applied only to special cases: those of observers moving at constant speeds in a straight line. Einstein knew that a general theory would need to deal with accelerations. It would also have to be reconciled with Isaac Newton’s theory of gravity, which relied on absolute space, made no explicit mention of time at all, and was believed to act not at the speed of light but instantaneously.

Einstein developed all his ideas about relativity with “thought experiments”: careful imaginary assessments of highly stylised states of affairs. In 1907 one of these provided him with what he would later refer to as his “happiest thought”: that someone falling off a roof would not feel his own weight. Objects in free-fall, he realised, do not experience gravity. But the curved trajectories produced by gravity—be they the courses of golf balls or planets—seemed to imply some sort of pushing or pulling. If golf balls and planets, like people falling off roofs, felt no sort of push or pull, why then did they not fall in straight lines?

The central brilliance of general relativity lay in Einstein’s subsequent assertion that they did. Objects falling free, like rays of light, follow straight lines through space-time. But that space-time itself is curved. And the thing that made it curve was mass. Gravity is not a force; it is a distortion of space-time. As John Wheeler, a physicist given to pithy dictums about tricky physics, put it decades later: “Space-time tells matter how to move; matter tells space-time how to curve.”

The problem was that, in order to build a theory on this insight, Einstein needed to be able to create those descriptions in warped four-dimensional space-time. The Euclidean geometry used by Newton and everyone else was not up to this job; fundamentally different and much more challenging mathematics was required. Max Planck, the physicist who set off the revolution in quantum mechanics, thought this presented Einstein with an insurmountable problem. “I must advise you against it,” he wrote to Einstein in 1913, “for in the first place you will not succeed, and even if you succeed no one will believe you.”

Handily for Einstein, though, an old university chum, Marcel Grossmann, was an expert in Riemannian geometry, a piece of previously pure mathematics created to describe curved multi-dimensional surfaces. By the time of his lectures in 1915 Einstein had, by making use of this unorthodox geometry, boiled his grand idea down to the elegant but taxing equations through which it would become known.

Just before the fourth lecture was to be delivered on November 25^th, he realised he might have a bit more to offer than thought experiments and equations. Astronomers had long known that the point in Mercury’s orbit closest to the sun changed over time in a way Newton’s gravity could not explain. In the 1840s oddities in the orbit of Uranus had been explained in terms of the gravity of a more distant planet; the subsequent discovery of that planet, Neptune, had been hailed as a great confirmation of Newton’s law. Attempts to explain Mercury’s misbehaviour in terms of an undiscovered planet, though, had come to nought.

Famous long ago

Einstein found that the curvature of space-time near the sun explained Mercury’s behaviour very nicely. At the time of the lectures, it was the only thing he could point to that general relativity explained and previous science did not. Martin Rees, Britain’s Astronomer Royal, is one of those who sees the nugatory role played by evidence in the development of the theory as one of the things “that makes Einstein seem even more remarkable: he wasn’t motivated by any mysterious phenomena he couldn’t explain.” He depended simply on his insight into what sort of thing gravity must be and the beauty of the mathematics required to describe it.

After the theory was published, Einstein started to look for ways to test it through observation. One of them was to compare the apparent positions of stars that were in the same part of the sky as the sun during a solar eclipse with their apparent positions at other times. Rays of light, like free-falling objects, trace straight lines in space-time. Because the sun’s mass warps that space-time, the positions of the stars would seem to change when the rays skirted the sun (see diagram).

In 1919 Arthur Eddington, a famed British astronomer, announced that observations of an eclipse made on the West African island of Principe showed just the distortion Einstein had predicted (one of his images is pictured). “LIGHTS ALL ASKEW IN THE HEAVENS”, read the New York Times headline, adding helpfully that “Nobody Need Worry”. Einstein, while pleased, had faith enough in his idea not to have been on tenterhooks. When asked what he would have done had Eddington found a different result, he replied, “Then I would feel sorry for the good Lord. The theory is correct.”

As far as the rest of the world was concerned, Eddington’s result put general relativity more or less beyond doubt. But that did not make it mainstream. For one thing, it was hard to grasp. At a public event, Eddington was momentarily stumped by the suggestion that he “must be one of the three persons in the world who understand general relativity”. When the silence was taken for modesty, he replied “On the contrary, I am trying to think who the third person is!”

General relativity also seemed somewhat beside the point. The quantum revolution that Planck had begun, and that Einstein had contributed to in one of his other great papers of 1905, was bearing fascinating fruit. Together with a blossoming understanding of the atomic nucleus, it was at the centre of physicists’ attention. Special relativity had a role in the excitement; its most famous expression, E=mc², gave a measure of the energy stored in those fascinating nuclei. General relativity had none.

What it offered instead was a way to ask questions not about what was in the universe, but about the structure of the universe as a whole. There were solutions to the equations in which the universe was expanding; there were others in which it was contracting. This became a topic of impassioned debate between Einstein and Willem de Sitter, a Dutch physicist who had found one of the expanding-universe solutions. Einstein wanted a static universe. In 1917 he added to his equations a “cosmological constant” which could be used to fix the universe at a given size.

That became an embarrassment when, in 1929, an American astronomer put forward strong evidence that the universe was, indeed, getting bigger. Edwin Hubble had measured the colour of the light from distant galaxies as a way of studying their motion; light from objects approaching the Earth looks bluer than it would otherwise, and light from objects receding looks redder. Hubble found that, on average, the more distant the galaxy, the more its light was shifted towards the red; things receded faster the farther away they were. The evidence for an expanding universe these redshifts provided led Einstein to reject the cosmological constant as the “greatest blunder of my life”.

The theory had other implications at which its architect initially baulked. In the 1930s nuclear physicists worked out that stars were powered by nuclear reactions, and that when those reactions ran out of fuel the stars would collapse. Something like the sun would collapse into a “white dwarf” about the size of the Earth. Bigger stars would collapse yet further into “neutron stars” as dense as an atomic nucleus and just 20 kilometres or so across. And the biggest stars would collapse into something with no length, breadth or depth but infinite density: a singularity.

Finding singularities in a theory is highly distasteful to the mathematically minded; they are normally signs of a mistake. Einstein did not want any of them in his universe, and in 1939 he published a paper attempting to show that the collapse of giant stars would be halted before a singularity could be formed. Robert Oppenheimer, a brilliant young physicist at Berkeley, used the same relativistic physics to contradict the great man and suggest that such extreme collapses were possible, warping space-time so much that they would create regions from which neither light nor anything else could ever escape: black holes.

Oppenheimer’s paper, though, was published on the day Germany invaded Poland, which rather put the debate on hold. Just a month before, Einstein had written to Franklin Roosevelt highlighting the military implications of E=mc²; it would be for realising those implications, rather than for black holes, that Oppenheimer would be remembered.

In part because of Oppenheimer’s government-bewitching success, new sorts of physical research flourished in the post-war years. One such field, radio astronomy, revealed cosmic dramas that observations using light had never hinted at. Among its discoveries were sources of radio waves that seemed at the same time small, spectacularly powerful and, judging by their redshifts, phenomenally distant. The astronomers dubbed them quasars and wondered what could possibly produce radio signals with the power of hundreds of billions of stars from a volume little bigger than a solar system.

Roy Kerr’s solution to the equations of general relativity provided the answer: a supermassive spinning black hole. Its rotation would create a region just outside the hole’s “event horizon”—the point of no return for light and everything else—in which matter falling inward would be spun up to enormous speeds. Some of that matter would be squirted out along the axis of rotation, forming the jets seen in radio observations of quasars.

Disappear like smoke

For the first time, general relativity was explaining new phenomena in the world. Bright young minds rushed into the field; wild ideas that had been speculated on in the fallow decades were buffed up and taken further. There was talk of “wormholes” in space-time that could connect seemingly distant parts of the universe. There were “closed time-like curves” that seemed as though they might make possible travel into the past. Less speculatively, but with more profound impact, Stephen Hawking, a physicist (pictured, with a quasar), and Roger Penrose, a mathematician, showed that relativistic descriptions of the singularities in black holes could be used to describe the Big Bang in which the expansion of the universe began—that they were, in fact, the only way to make sense of it. General relativity gave humans their first physical account of creation.

Dr Hawking went on to bring elements of quantum theory into science’s understanding of the black hole. Quantum mechanics says that if you look at space on the tiniest of scales you will see a constant ferment in which pairs of particles pop into existence and then recombine into nothingness. Dr Hawking argued that when this happens at the event horizon of a black hole, some of the particles will be swallowed up, while some will escape. These escaping particles mean, in Dr Hawking’s words, that “black holes ain’t so black”—they give off what is now called “Hawking radiation”. The energy lost this way comes ultimately from the black hole itself, which gives up mass in the process. Thus, it seems, a black hole must eventually evaporate away to nothingness.

Adding quantum mechanics to the description of black holes was a step towards what has become perhaps the greatest challenge in theoretical physics: reconciling the theory used to describe all the fields and particles within the universe with the one that explains its overall shape. The two theories view reality in very different ways. In quantum theory, everything is, at some scale, bitty. The equations of relativity are fundamentally smooth. Quantum mechanics deals exclusively in probabilities—not because of a lack of information, but because that is the way the world actually is. In relativity all is certain. And quantum mechanics is “non-local”; an object’s behaviour in one place can be “entangled” with that of an object kilometres or light-years away. Relativity is proudly local; Einstein was sure that the “spooky action at a distance” implied by quantum mechanics would disappear when a better understanding was reached.

It hasn’t. Experiment after experiment confirms the non-local nature of the physical world. Quantum theory has been stunningly successful in other ways, too. Quantum theories give richly interlinked accounts of electromagnetism and of the strong and weak nuclear forces—the processes that hold most atoms together and split some apart. This unified “standard model” now covers all observable forms of matter and all their interactions—except those due to gravity.

Some people might be satisfied just to let each theory be used for what it is good for and to worry no further. But people like that do not become theoretical physicists. Nor will they ever explain the intricacies of the Big Bang—a crucible to which grandiose theory-unifiers are ceaselessly drawn. In the very early universe, space-time itself seems to have been subject to the sort of fluctuations fundamental to the quantum world (like those responsible for Hawking radiation). Getting to the heart of such shenanigans requires a theory that combines the two approaches.

There have been many rich and subtle attempts at this. Dr Penrose has spent decades elaborating an elegant way of looking at all fields and particles as new mathematical entities called “twistors”. Others have pursued a way of adding quantum bittiness to the fabric of space-time under the rubric of “loop quantum gravity”. Then there is the “Exceptionally Simple Theory of Everything”—which isn’t. As Steven Weinberg, one of the unifiers whose work built the standard model, puts it, “There are so many theories and so few observations that we’re not getting very far.”

Dr Weinberg, like many of his colleagues, fancies an approach called superstring theory. It is an outgrowth of the standard model with various added features that seem as though they would help in the understanding of space-time and which its proponents find mathematically beguiling. Ed Witten of the Institute for Advanced Study (IAS) in Princeton, Einstein’s institutional home for the last 22 years of his life, is one of those who has raised it to its current favoured status. But he warns that much of the theory remains to be discovered and that no one knows how much. “We only understand bits and pieces—but the bits and pieces are staggeringly beautiful.”

This piecemeal progress, as Dr Witten tells it, offers a nice counterpoint to the process which led up to November 1915. “Einstein had the conception behind general relativity before he had the theory. That’s in part why it has stood: it was complete when it was formulated,” he says. “String theory is the opposite, with many manifestations discovered by happy accident decades ago.”

Entangled up in the blue

And the happy accidents continue. In 1997 Juan Maldacena, an Argentine theoretician who now also works at the IAS, showed that there is a deep connection between formulations of quantum mechanics known as conformal field theories and solutions to the Einstein equations called anti-de Sitter spaces (similar to the expanding-universe solution derived by Willem de Sitter, but static and much favoured by string theorists). Neither provides an account of the real world, but the connection between them lets physicists recast intractable problems in quantum mechanics into the sort of equations found in general relativity, making them easier to crack.

This approach is being gainfully employed in solving problems in materials science, superconductivity and quantum computing. It is also “influencing the field in a totally unexpected way,” says Leonard Susskind, of Stanford University. “It’s a shift in our tools and our methodology and our way of thinking about how phenomena are connected.” One possibility Dr Maldacena and Dr Susskind have developed by looking at things this way is that the “wormholes” relativity allows (which can be found in the anti-de Sitter space) may be the same thing as the entanglement between distant particles in quantum mechanics (which is part of the conformal field theory). The irony of Einstein’s spooky quantum bête noire playing such a crucial role has not gone unremarked.

There is more to the future of relativity, though than its eventual subsumption into some still unforeseeable follow-up theory. As well as offering new ways of understanding the universe, it is also providing new ways of observing it.

This is helpful because there are bits of the universe that are hard to observe in other ways. Much of the universe consists of “dark matter” which emits no radiation. But it has mass, and so it warps space, distorting the picture of more distant objects just as the eclipse-darkened sun distorted the positions of Eddington’s stars. Studying distortions created by such “gravitational lenses”—both luminous (pictured, with Einstein) and dark—allows astronomers with the precise images of the deep sky today’s best telescopes provide to measure the distribution of mass around the universe in a new way.

Another form of relativity-assisted astronomy uses gravitation directly. Einstein’s equations predict that when masses accelerate around each other they will create ripples in space-time: gravitational waves. As with black holes and the expanding universe, Einstein was not keen on this idea. Again, later work has shown it to be true. A pair of neutron stars discovered spinning around each other in the 1970s are exactly the sort of system that should produce such waves. Because producing gravitational waves requires energy, it was realised that these neutron stars should be losing some. And so they proved to be—at exactly the rate that relativity predicts. This indirect but convincing discovery garnered a Nobel prize in 1993.

As yet, though, no one has seen a wave in action by catching the expansion and contraction of space that should be seen as one goes by, because the effects involved are ludicrously small. But researchers at America’s recently upgraded Laser Interferometer Gravitational-wave Observatory (LIGO) now think they can do it. At LIGO’s two facilities, one in Louisiana and one in Washington state, laser beams bounce up and down 4km-long tubes dozens of times before being combined in a detector to make a pattern. A passing gravitational wave that squashes space-time by a tiny fraction of the radius of an atomic nucleus in one arm but not the other will make a discernible change to that pattern. Comparing measurements at the two sites could give a sense of the wave’s direction.

Step into the light

The aim is not just to detect gravitational waves—though that would be a spectacular achievement—but to learn about the processes that produce them, such as mergers of neutron stars and black holes. The strengths of the warping effects in such cataclysms are unlike anything seen to date; their observation would provide a whole new type of test for the theory.

And history suggests there should be completely unanticipated discoveries, too. Kip Thorne, a specialist in relativity at the California Institute of Technology and co-founder of LIGO, says that “every time we’ve opened a new window on the cosmos with new radiation, there have been unexpected surprises”. For example, the pioneers of radio astronomy had no inkling that they would discover a universe full of quasars—and thus black holes. A future global array of gravitational-wave observatories could open a whole new branch of observational astronomy.

A century ago general relativity answered no one’s questions except its creator’s. Many theories are hit upon by two or more people at almost the same time; but if Einstein had not devoted years to it, the curvature of space-time which is the essence of gravity might not have been discovered for decades. Now it has changed the way astronomers think about the universe, has challenged them to try and build theories to explain its origin, and even offered them new ways to inspect its contents. And still, it retains what most commended it to Einstein: its singular beauty revealed first to his eyes alone but appreciated today by all who have followed. “The Einstein equations of general relativity are his best epitaph and memorial,” Stephen Hawking has written. “They should last as long as the universe.”