Goodhart’s law, Changing Wikipedia, and the Hawthorne Effect

I’ve been interested in Goodhart’s law for a long time, and in the past couple years even wrote a few articles about it. So I’ve left a column on Tweetdeck running with a search for Goodhart’s law, to see how it is used and discussed.

If you’re not familiar, the popular paraphrase of Goodhart’s law is “When a measure becomes a target, it ceases to be a good measure.” This quote bothered me for a long time, since it is a significant generalization of Goodhart’s original phrasing, “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” This was confusing until I saw a tweet saying that the popular paraphrase is known as “Strathern’s Variation,” and I found that others had noted the same thing. This prompted me to investigate.

Digging through the Wikipedia edit history, I found a reference to Strathern that had since been edited out, citing a 2007 publication, “Wireless Communications: The Future.” This wasn’t available online, and I was fairly sure I had seen the quote before then anyways. Digging, I found the origninal source; a 1997 paper by Strathern. So on August 4th, I edited Wikipedia to include the fact that the frequently quoted paraphrase of Goodhart’s law is actually hers, and added a link.

From August 1st-4th, I count 14 mentions of the term “Goodharts Law” on Twitter. That’s probably par for the course; it gets mentioned around 100 times a month. But before August, I can find 1 time that Strathern has been mentioned referring to the quote this year — the one prompting my investigation — as opposed to 3 in the month ending September 2nd — and another several dozen in the week since due to bots retweeting a Techcrunch article that leads with the quote. This isn’t yet statistically significant, but it’s an interesting impact to notice.

The problem with writing this article, then, is that it brings further attention to the issue — and that highlights the difference between Goodhart’s law and the Hawthorne Effect, an earlier and simpler claim that paying attention affects measurement. The appearance of the article potentially warps how well my measure represents the effect of the original edit, but it’s not placing any pressure on the measure.