Thursday, June 12, 2014
Wassily Hoeffding 1914-1991
In honor of the 100th anniversary of the birth of Wassily Hoeffding today, let's recall his incredibly useful inequality.
Prob(|X-pn|>εn) < 2e-2ε2n
where X is the random variable representing the number of heads from n independent coin flips each with a probability p of being heads. Informally the sum of many independent events will, with extremely high probability, be very close to the expected value. Notice that as long as ε = 1/o(√n) the probability goes down exponentially in n and this is tight as one expects X to be about O(√n) away from pn for constant p.
Back in the 80's, probabilistic computation started to play a major role in algorithms, cryptography, learning theory and interactive proofs. To analyze these computations one needed to deal with Chernoff bounds and it was difficult to find clean and easy statements of the bounds one could use in proofs. One of my officemates in grad school, Bob Sloan, took it upon himself to write up a short document giving Hoeffding's inequality and a few generalizations. Having that write-up was like holding Excalibur in my hands, ready to do battle with probabilistic complexity.
Later Alon and Spencer gave a nice treatment in their Probabilistic Methods text and these days you can get all the formulas you want on Wikipedia.