Heat

Entropy is missing information

Entropy and information are the same quantity, counted in different units.

What does it mean for something to be disordered?

Put ten coins on a table, shaken and flat, and ask what you could say about them without looking.

You could say "all heads", but that is one arrangement out of 1,024. You could say "five heads and five tails", and that is true of 252 of those arrangements. If all you know is that the table looks about half and half, you have a long list of ways it could be, and you cannot tell them apart. Physicists call "about half and half" a macrostate, and each exact arrangement of heads and tails a microstate. Entropy is how many microstates hide behind the macrostate you can see.

The same count is information, seen from the other side. Suppose a friend knows the exact arrangement and you have to find it by asking yes-or-no questions. Each question can at best cut the list of possibilities in half. Twenty questions are enough to pick out one item from 1,048,576, and ten are enough for the coins. One yes-or-no answer is one bit. For equally likely arrangements, the missing information is the number of binary answers needed to identify one.

So a table of coins you have not looked at and a memory you have not read are the same kind of thing. Both are a state you have only described roughly, with many exact ways to fill in the rest. When you look and find out which one it is, the missing information drops by exactly the number of bits you learned. Nothing on the table moved; what changed was what you know. Whether that makes entropy partly a fact about the person looking is something physicists still argue about, and the last fold of this page says how. The numbers come out the same either way.

A hot gas illustrates the connection to heat. Its temperature and pressure specify a macrostate with an enormous number of possible molecular arrangements. Heat transfer carries entropy, and the Vigil's radiators remove entropy along with energy.

Guess first Shake a box of a hundred fair coins. Would you ever expect to see all of them come up heads?

No. All heads is one arrangement out of 1.3 × 10³⁰, so you would wait longer than the age of the universe many times over. The arrangement you see will always look unremarkable, close to half and half, because nearly all the arrangements do: 96 percent of them have between 40 and 60 heads. Macrostates with many compatible arrangements are more likely, which illustrates the statistical basis of the second law.

Ludwig Boltzmann found this idea in 1877, and Max Planck later wrote it as S = k log W. S is the entropy, W is the number of microstates, and k, which Planck added, is a constant that turns "how many" into joules per kelvin. The logarithm converts the count of arrangements into entropy.

Each unseen coin doubles the ways the world could be and adds one bit to the entropy. Illustrative: the squares stand for arrangements, and a real system has astronomically more of them.

How big is a bit, in joules per kelvin?

In physical units the formula reads:

S = k × ln W

W is the number of arrangements consistent with what you know.
ln is the natural logarithm. It turns a count into an exponent, so that two independent things, whose counts multiply, have entropies that add.
k is Boltzmann's constant, 1.380649 × 10⁻²³ joules per kelvin.

A bit is a factor of two in W, so one bit of entropy is k ln 2, or 9.570 × 10⁻²⁴ joules per kelvin. Everything else follows by counting bits:

Thing	Arrangements	Entropy in bits	Entropy in J/K
One bit	2	1	9.6 × 10⁻²⁴
One byte	256	8	7.7 × 10⁻²³
A hundred coins	1.3 × 10³⁰	100	9.6 × 10⁻²²
A resident's mind (illustrative)	2 to the power 1 × 10¹⁷	1 × 10¹⁷	9.6 × 10⁻⁷
A mole of liquid water, 18 g, at 25 °C	too many to write	7.3 × 10²⁴	70

The bottom two rows compare stored information with thermal entropy. A mole of water is a little more than a tablespoon, and its thermal entropy, measured by chemists as heat flow over temperature, is 70 joules per kelvin. Divide by the size of a bit and that is 12 bits of missing information for every molecule in the glass. The bits are all in how the molecules are sitting and moving, which nobody records.

Weigh a mind against that. The entropy in a whole mind's 1 × 10¹⁷ bits is the same as the thermal entropy of only 0.25 micrograms of water. Thermal entropy can therefore represent much more missing information than an explicit memory of modest mass.

Is every bit a coin flip?

Shannon, who built the mathematics of information in 1948, noticed that the count changes if the outcomes are not equally likely. A coin that lands heads 90 percent of the time surprises you less than a fair one. Its average information per flip is 0.47 bits, not 1, so a long record of its flips can be squeezed down by 53 percent without losing anything. In general the entropy is the average surprise: rare outcomes carry more bits, common ones fewer.

Is heat just entropy that leaves?

When heat $Q$ flows into something at temperature $T$ , its entropy rises by $Q/T$ . One joule of heat at 300 K carries 0.0033 joules per kelvin of entropy. One joule at the radiators' 30 K carries 0.033, which is 10 times as much. At lower temperature, the same energy transfer carries more entropy. The Vigil's first petawatt, rejected at 30 K, carries 3.3 × 10¹³ joules per kelvin of entropy away every second. In bits, that is 3.5 × 10³⁶ per second, the same figure as the number of bits it could forget each second at that temperature, as The cost of forgetting derives from the same entropy balance. The way the Vigil turns some of that flow into work on the way out is the subject of Recycling heat.

With JavaScript on, this is a toy: eight boxes with one molecule each, a byte's worth. Click a box to look inside and watch the entropy fall by k ln 2 each time. Without it, the table above gives the same counts: eight unknown boxes are 256 arrangements, eight bits, and 7.7 × 10⁻²³ joules per kelvin.

Open everything

Why is the entropy a logarithm, and where does Shannon's formula come from?Counting, Stirling's approximation and the demon's bookkeeping

Why a logarithm

Take two independent systems with $W_1$ and $W_2$ arrangements. Together they have $W_1 W_2$ , since any arrangement of one can go with any of the other. Entropy should be additive: the entropy of two separate things is the sum of their entropies. A function $f$ with

$f(W_1 W_2) = f(W_1) + f(W_2)$

is a logarithm, and the constant out front picks the units. Choose $k$ and a natural logarithm and you get joules per kelvin. Choose 1 and a base-2 logarithm and you get bits. The two differ by the fixed factor $k \ln 2$ per bit.

From counting to Gibbs

Boltzmann's count assumes every microstate is equally likely. Usually they are not: a system in contact with a heat bath prefers low-energy states. Josiah Willard Gibbs gave the formula for that case, and a counting argument shows where it comes from. Imagine $N$ copies of the system, with $n_i = N p_i$ of them in state $i$ . The number of ways to hand out the states among the copies is

$W = \frac{N!}{\prod_i n_i!}$

Stirling's approximation, $\ln n! \approx n \ln n - n$ , gives

$\ln W \approx N \ln N - \sum_i n_i \ln n_i = -N \sum_i p_i \ln p_i$

so the entropy per copy is the Gibbs entropy

$S = -k \sum_i p_i \ln p_i$

When all $W$ states are equally likely, $p_i = 1/W$ and this collapses to $k \ln W$ .

Shannon's version

Shannon asked a different question: what is the right measure of the uncertainty of a source that emits symbols with probabilities $p_i$ ? He asked for three things: it should vary smoothly with the $p_i$ , it should grow with the number of choices when they are equally likely, and if a choice is broken into stages, the total should be the weighted sum of the parts. Those requirements leave only one form, up to a constant:

$H = -\sum_i p_i \log_2 p_i \ \text{bits}$

This is the Gibbs formula with $k$ removed and the logarithm in base 2. They are the same quantity: $S = (k \ln 2)\, H$ . The formulas share the same probability measure. Gibbs and Shannon were each counting how many questions it takes to name an exact state, in different units, and one bit of $H$ is worth $k \ln 2$ joules per kelvin of $S$ .

Maxwell's demon and Szilard's engine

In 1867 James Clerk Maxwell imagined a tiny doorkeeper who watches the molecules of a gas and opens a door between two halves only for fast ones going one way and slow ones the other. With no work, the demon sorts hot from cold, which the second law forbids. Leo Szilard reduced the puzzle in 1929 to its smallest case: a box with a single molecule, in contact with a bath at temperature $T$ . Slide a partition into the middle. If the demon learns which half the molecule is in, it can hook a weight to the partition on that side and let the molecule push the partition, which acts as a piston, out to the far wall. Isothermal expansion from $V/2$ to $V$ gives

$W_{\text{out}} = \int_{V/2}^{V} \frac{kT}{V'} \, dV' = kT \ln 2$

One bit of knowledge became $kT \ln 2$ of work, and the box is back where it started. If this repeated for free, the demon would be a perpetual-motion machine.

Where the cost hides

For decades the answer was thought to be the measurement. Brillouin argued in 1951 that to see a molecule the demon needs light hotter than its surroundings, and that the scattering generates entropy. Bennett's analysis of 1982, building on Landauer's work, showed that this is not necessary: a measurement can be done reversibly, by copying the molecule's side onto a blank memory with no waste at all. Look at the entropy of the molecule $G$ and the memory $M$ together, with mutual information $I$ between them:

$S(G, M) = S(G) + S(M) - I$

Before the measurement the memory is blank, $S(M) = 0$ and $I = 0$ . Afterwards the molecule on its own still looks like a fifty-fifty coin, so $S(G) = k \ln 2$ , the memory holds one bit, $S(M) = k \ln 2$ , and they are perfectly correlated, $I = k \ln 2$ . Given the memory, the molecule has no missing information left, $S(G) - I = 0$ : that is the drop the story describes when you look. But the joint total is still $k \ln 2$ . Measuring did not destroy the missing information; it moved it into the memory as a record. Then the engine runs. The molecule expands and loses its correlation with the memory, so $I$ falls to zero and the joint entropy has risen to $2k \ln 2$ . The extra $k \ln 2$ came from the bath, which gave up $kT \ln 2$ of heat as the work came out.

The cycle can repeat only if the memory is wiped. That sends its $k \ln 2$ of entropy back into the bath, and costs at least $kT \ln 2$ , exactly the work the engine produced. The full cycle produces no net work. Measurement can be reversible, while erasing the record restores the entropy to the bath. The derivation of that erasure cost has its own page: The cost of forgetting.

What is still argued, and what has been tested?Maximum entropy, fluctuation theorems, feedback and the quantum case

Whose ignorance is it?

If entropy is missing information, whose information? E. T. Jaynes argued in 1957 that statistical mechanics is a method of inference: given what you know about a system, choose the probabilities with the largest entropy that agree with it. Maximize $H$ with the average energy fixed and the Boltzmann distribution drops out. On this view entropy is a property of a description, and two observers who know different things assign different entropies to the same gas. frontier

Other physicists emphasize that measurable macroscopic quantities define the coarse-graining. Once that choice is fixed, the entropy calculation is objective. The interpretation remains disputed, while both approaches give the same numerical results here.

The second law as an average

For a large system the second law is a certainty. For a molecule-sized one it is a statement about averages. Christopher Jarzynski proved in 1997 that if you drag a system between two states, in contact with a bath at temperature $T$ , the work $W$ you put in on repeated runs obeys

$\left\langle e^{-W/kT} \right\rangle = e^{-\Delta F/kT}$

where $\Delta F$ is the change in free energy. Individual runs may take less work than $\Delta F$ , in apparent violation of the second law; the equation says they are rare enough to balance. Gavin Crooks sharpened this in 1999: the probability of a given amount of work going forward, compared with the probability of the same work coming out when the process runs in reverse, differs by exactly the factor $e^{(W - \Delta F)/kT}$ . Larger apparent violations are exponentially less probable.

Information as a resource

These theorems assumed nobody was watching and steering. Takahiro Sagawa and Masahito Ueda added the demon in 2008: a controller that measures a system and then acts on what it learned. Their generalized second law turns the information $I$ gained by measurement into a budget: the work you can extract is at most $-\Delta F + kT\, I$ , with $I$ counted in natural units, so one bit is $\ln 2$ . Knowing something lets you take that much more from the bath, and nothing more. In 2010 they extended Jarzynski's equality the same way.

It has been tested. In 2010 Shoichi Toyabe and colleagues in Tokyo used a tiny particle jostled by heat on a spiral staircase of electric potential. A feedback loop watched the particle and put a barrier behind it every time it hopped up a step, so it climbed and gained more free energy than the work done on it. They measured the energy gained and the information used, and the data matched the generalized equality. Landauer's principle accounts for the heat released when the controller's memory is reset.

Quantum entropy

For a quantum system described by a density matrix $\rho$ the entropy becomes $S = -k\,\mathrm{Tr}(\rho \ln \rho)$ , von Neumann's version of the Gibbs formula. A pure state has zero entropy. If a pure state is split in two, each half can have large entropy because of how the halves are entangled with one another. Entanglement entropy is central to the information problem for black holes, whose thermodynamic entropy is proportional to horizon area. Black holes glow and How much can a region remember? develop these questions. Whether black-hole entropy counts something hidden in the hole, and whether information escapes in the radiation, is argued in 2026 and not settled. frontier

Why does entropy start low?

If the laws of physics run equally well backwards, why does entropy always rise toward the future? The standard answer is that the early universe began in an extraordinarily special, low-entropy state, and that everything we call time's arrow follows from that. Why it began that way is open. frontier

A good review of the whole field, in the form it had by 2015, is Parrondo, Horowitz and Sagawa's Thermodynamics of information.