The root meaning of probability was originally "approbation." Something probable was deserving of belief. This connotation has been lost in modern times as the understanding of mathematical probabilities has become widespread. Although some epistemologists still connect "degrees of belief" with epistemic probabilities.

Probabilities are a priori theories. Statistics are a posteriori experiments.

The a priori probability of two dice showing two sixes is 1/36. A priori probability assumes that we have no information that favors one outcome over the others. This is the principle of indifference or principle of insufficient reason. Specifically, it assumes that all outcomes are equally probable. In short, all other things being equal, things are equal.

If we did have information about a difference, we would adjust our probabilities. The increase in our information or "state of knowledge" is the essential idea in Bayesian probability.

And how might we come by such additional information? By running experimental trials and gathering data on the frequencies of real outcomes. This is the work of statistics.

In the theory of measurements and measurement errors, we find a dispersion in measurements when we are measuring the same thing over and over. These are measurement errors. It is very important to distinguish measurement errors from real variations in the thing measured. But this is not easy, because errors and the natural distribution in values of a measured property normally follow the same distribution function.

If the values of some property in a population is the consequence of independent random events, for example coin flips that result in heads instead of tails, the distribution can be shown to follow the well-known bell-shaped curve or "normal" distribution.

Observational errors themselves are the result of independent random factors, so they follow the same distribution.

This distribution function was discovered in the 1720's by the French mathematician Abraham de Moivre. It was identified with measurement errors by astronomers measuring star positions in the early nineteenth century, notably by the Belgian astronomer Adolphe Quételet.

An unfortunate choice of terms led to this being called the "law of errors." Mathematicians and philosophers both jumped to the erroneous conclusion that if random events followed "laws" that they must in fact be determined by those laws.

De Moivre's work was a famous book called "The Doctrine of Chances." In the 1738 edition he found that in the limit of large numbers of tosses of a coin, the discrete binomial expansion of (p - q)ⁿ could be approximated by a continuous mathematical curve (the modern bell curve).

This was an era when mathematical analysis and continuity carried an aura of deterministic causal truth, compared to the chance nature of individual random events.

The curve was first called the "normal distribution" by Charles Sanders Peirce, who better than any other philosopher articulated the difference between a priori probabilities and a posteriori statistics.

For reasons that are philosophically and physically very deep (basically the fundamental discreteness of nature), we find a similar random distribution of many physical, biological, and social characteristics. Note that the distribution of such characteristics is ontological, not epistemic. It is not a matter of what humans can know, but how things are in a mind-independent external world..

Social scientists in the early nineteenth century argued that randomness in many populations was governed by this "law." Adolphe Quetelet was a Belgian astronomer who developed the error distribution as a way to analyze and reduce the data from astronomical observations. He then found the same distribution function in many human statistics, the number of marriages and suicides per year, for example. He found "norms for many human characteristics, such as height, and promoted the concept of the average man (l'homme moyen) . Quetelet and other social scientists mistakenly concluded that the "lawfulness" of these normal distributions affirmed the fundamental deterministic nature of the world.

To the modern sensibility, an argument that all events are determined because their characteristics are random and normally distributed seems illogical and a bit perverse. But such was the philosophical and religious commitment of nineteenth-century scientists to the many forms of determinism.

If just one single event is determined by chance, then indeterminism would be true, some philosophers say, which would undermine the very possibility of certain knowledge. Some go to the extreme of saying that chance makes the state of the world totally independent of any earlier states, which is nonsense, but it shows how anxious they are about chance.

In statistical mechanics, the state of least information (equilibrium) can be achieved in many more ways than states with recognizable information. The equilibrium macrostate has the most microstates, each assumed to have equal a priori probability. If a system begins in an initially ordered state (high information), it tends to evolve toward states of disorder (absence of microscopic information). The increase of entropy (second law of thermodynamics) is then seen to be a decrease of information. The mathematical equations for entropy and information turn out to be identical apart from their sign and arbitrary constants.

Both are proportional to the logarithm of the number W of microstates consistent with the macrostate information.

Both James Clerk Maxwell and Ludwig Boltzmann were familiar with the work of Quetelet and his English colleague Thomas Buckle. Maxwell, and probably Boltzmann too, used the above law of distribution of random events in the limit of large numbers to help them derive the law for the distribution of molecular velocities in a gas.