Benford’s Law

Send to Kindle

By Emily Chen

Here’s a puzzle for you: what does the graph on the left represent?

The level of motivation to fulfill New Year’s resolutions over the month of January?  Christmas cheer vs. the number of times the same song is repeated on the radio? 

Though the above explanations are plausible, the graph is actually an illustration of Benford’s Law, one of the most mysteriously universal rules known to scientists today.  Benford’s Law, in short, describes the probability that the first digit of a long number, such as 123456, is a specific number (in this case, the first digit is 1.)  As the graph illustrates, when you have a large sample of numbers, the 1 is the first digit about 30% of the time, 2 is the digit about 18% of the time, and so on. This seems very strange initially—after all, why should certain numbers appear as first digits more than others? This search for that answer has a history spanning over a century.

In 1881, when scientists still used logarithm tables, a particularly observant astronomer named Simon Newcomb noticed that the pages in the beginning of the books of log tables were more worn than the pages at the end.  Newcomb worked on describing this phenomenon mathematically, and published his model, proposing that the probability of a number, N, being the first digit of a longer number, was log(N+1) – log(N). For Newcomb, it both started and ended with his book of log tables.

More than fifty years later, the physicist Frank Benford decided to put Newcomb’s model to the test.  He used over 20,000 numbers from a wide variety of sources, including Reader’s Digest, population sizes, and the surface areas of rivers.  His results confirmed Newcomb’s model.  And as it happens in history, the law was named not after its originator but its publicizer.

Benford’s law applies to a huge variety of data sets, and has been supported by datasets ranging from genomic data to addresses. Indeed, Benford’s law has been verified to such an extent that violations of it may be a sign of fraud. It is used in audits to check for fudged or fabricated numbers, and as the applications grow, more sophisticated models involving predictions of the second or third digits have been developed.  

We can’t go applying Benford’s law willy-nilly, though. In datasets where there is a significant psychological component, for example, the data may be skewed toward “simpler” numbers such as multiples of 5 or 10.  For example, if a group of volunteers were asked to list how many minutes of sleep they got a night, they might list multiples of 60 (an hour) or 30 (half an hour) more often.  And for certain distributions, such as the distribution of heights or IQs across populations, Benford’s law simply doesn’t apply. However, Benford’s law does work very well for sets of numbers resulting from calculations or combinations of other numbers.  So if you were to multiply the IQ of each person by their height to form a new set of numbers, Benford’s law might well apply. And of course, the larger the dataset, the more closely mathematical trends (like Benford’s law) fit the data.

So now we come to be big question: why does Benford’s law work, and why does it apply to so many different datasets? We still don’t know for sure. But scientists, being scientists, have come up with a few possible explanations. 

One possibility is that Benford’s law results from the prevalence of functions involving exponential growth. But though this explains some instances where it applies, it leaves many other questions unanswered.   Another explanation is that Benford’s law applies to combinations of certain distributions that occur naturally (the distributions, like height and IQ, that Benford’s Law doesn’t apply to!) and we see these combinations in many datasets. This, too, brings up a lot of questions. 

The answer we are left with is one that is both exciting and frustrating to scientists: we don’t know. But never before has mankind had access to such large datasets and such powerful mathematical modeling tools. Perhaps we will be the ones to finally crack the mystery of Benford’s Law.

Note: For the more mathematically inclined, Terence Tao wrote an excellent blog post on Benford’s Law and other universal laws, which can be found here:  http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/. 

For the not-so-mathematically -inclined, the application of Benford’s Law to tax fraud is demonstrated here: http://www.intuitor.com/statistics/Benford’s%20Law.html

Copyright 2012 Yiqing Chen.

 

This entry was posted in Mathematics, Statistics. Bookmark the permalink.

One Response to Benford’s Law

  1. Reggie says:

    Yes, yes, the IRS uses Benford’s Law on numbers in your 1040 SO DON’T MAKE THINGS UP!!!

Add Comment Register



Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>