On sampling bias
Politicians and journalists are often unaware that the opinions they hear are a biased sample of all opinion.
Sometimes, it’s more important to remember the question than the answer - a fact which is especially true when thinking about political opinions.
Let’s start with the story of Abraham Wald’s work for the US air force. During WWII, many of its planes were being shot down so it wanted to armour-plate the planes. Because such plating was heavy, only part of the planes could be so armoured. But which parts? They looked at the damage sustained by returning planes and saw lots of holes in the wings and tail, and thought that these areas were those most needing protection.
Enter Wald. The damage you see, he said, is only a sample of all the damage that our planes are suffering. And it’s a biased sample. We don’t see planes that have been hit in the fuselage because they don’t make it back. Shots in the wings, on the other hand, are survivable. We shouldn’t worry about armouring them, therefore; its the fuselage we must protect.
This was an early example of recognising the importance of sampling bias.
Now, the thing here is that such bias is obvious when it’s been pointed out*. The moment we say it, it’s clear that the planes that return home differ from those that get shot down. But this much wasn’t obvious to the USAF at the time, which was thinking of armouring the wings. The trick isn’t so much the answer but the question.
We must always ask of anything: is this all we’re seeing, or just a sample? And if it’s a sample, is it a biased sample? And if so, in which way?
We often forget these questions; investors, for example, fail to see that a short run of good performance by a fund manager might be a lucky biased sample of their actual performance.
Which brings me to my fear - that the same is true in politics. Politicians and journalists might, like the USAF before Wald stepped in, be drawing wrong conclusions from biased samples.
An obvious example of what I mean is the vox pop. These are egregious case of sampling bias. This is partly because the sort of people who are on the street in the middle of the afternoon are a biased sample of all people: they’re not at work or trapped at home by a disability. Worse still, the interviewees are themselves selected by editors for those who give answers they want**. The true and honest answer in many cases would be: “Why are you asking me rather than somebody who knows? Fuck off you daft twat.” But that would be left in the editing suite.
This, however, is only one case of many where the opinions we see are a biased sample. Here are some others:
- Financial markets. Investors are a diverse bunch: it takes differences of opinion to make a market. This means there’ll always be someone who’s very bearish, or bullish, or hyper-critical of the government. And this is not to mention that those willing to speak to the media aren’t necessarily a representative sample of all investors. Hard facts about markets are price moves; individual opinions are likely to be evidence of sampling bias.
- X/Twitter is obviously unrepresentative of opinion generally. Not only does the platform attract racists and repel their opponents, but also its algorithms boost rightist tweets and downgrade others.
- MPs’ surgeries. Those who attend these are disproportionately likely to be cranks or those with bees in their bonnets - not all, obviously, but a larger proportion than is found in the general public.
- Legacy media. When newspapers sold millions, it was reasonable to suppose that their opinions were shared by lots of voters. Today, however, hardly anyone buys a print paper: the combined sales of the Mail, Sun, Express and Telegraph are probably less than 1.5 million, just 3% of the electorate. Newspapers now cater to a minority of cranks. We’ve no reason to suppose therefore that their opinions or interests are representative of those of the wider electorate.
- Groupthink. Like many of us, politicians move in narrow circles - in their case, of advisors, journalists and fellow pols. This leaves them vulnerable to groupthink and professional deformation. For example what Simon Wren-Lewis called mediamacro after 2008; or overconfidence about the efficacy of top-down policies; or simply an over-estimation of the extent to which people pay attention to politics.
- Lobbyists. People who organize themselves to form pressure groups or pay lobbyists are a biased sample of all people. As Mancur Olson pointed out, interests that are concentrated among a few people are more likely to get organized than those that are spread across everyone. If ten people stand to gain £1m each from a particular policy they are more likely to form a pressure group than a million people each of whom would gain £20, even though the aggregate gain is larger in the latter case. Lobbyists therefore - by virtue of their very existence - are a biased guide to what is really in the public interest.
You probably won’t be surprised by any of this. Sampling bias is often obvious once it is pointed out. But the thing is that we have to be alert to it. Remembering the question - “is this a representative sample?” - is the difficult thing. The answers are often easy.
Which brings me to my concern: are MPs actually alert to the problem? And it’s a serious one, because all these points imply that the information MPs get - from the media, their constituents or lobbyists - is biased***?
Several things make me suspect not. One is that, under Starmer, Labour’s anti-immigration policy courted the votes of the right without actually winning any. That’s consistent with the party mistaking voices on Twitter and legacy media for wider opinion. It made the mistake of which Bertie Wooster accused Roderick Spode:
The trouble with you, Spode, is that just because you have succeeded in inducing a handful of half-wits to disfigure the London scene by going about in black shorts, you think you’re someone. You hear them shouting “Heil, Spode!” and you imagine it is the Voice of the People. That is where you make your bloomer. What the Voice of the People is saying is: “Look at that frightful ass Spode swanking about in footer bags! Did you ever in your puff see such a perfect perisher?
Two other facts suggest governments neglect sampling bias: the complexity of the tax system; and regulatory capture. Both are consistent with governments kowtowing to lobbyists, unaware that these are unrepresentative of the wider public interest which requires strong regulation and a simple tax system.
Of course, none of this would matter so much if politicians were popular. But they’re not - which is a further data point suggesting that they’re listening to the wrong voices.
Now, I might be mistaken here. It could be that ministers in the Starmer government weren’t really interested in accurately gauging public opinion or (often a different thing!) the public interest. Maybe all they wanted was to defeat the left and get well-paid jobs after politics.
If politicians do want an accurate picture of public opinion, however, then they have a problem, because the traditional means of discovering it are now prone to significant sampling bias.
* Though Wald’s mathematics (pdf) was not so obvious!
** Years ago, a friend was stopped in the street by a voxpopper doing a piece on a survey that had shown Oakham to be one of the best places to live in the country. “What’s it like living here?” she asked. “It’s good”, he said. “Oh, I’m having a bad afternoon” she replied. “I can’t find anyone to say anything bad. “OK” he replied, “there’s not much for young people to do.” Guess what appeared.
*** A further problem is that a lot of public opinion is itself formed by the sampling bias, and not just because of media bias. But that’s another story.
the planes could be so armoured. But which parts? They looked at the damage sustained by returning planes and saw lots of holes in the wings and tail, and thought that these areas were those most needing protection.
Enter Wald. The damage you see, he said, is only a sample of all the damage that our planes are suffering. And it’s a biased sample. We don’t see planes that have been hit in the fuselage because they don’t make it back. Shots in the wings, on the other hand, are survivable. We shouldn’t worry about armouring them, therefore; its the fuselage we must protect.
This was an early example of recognising the importance of sampling bias.
Now, the thing here is that such bias is obvious when it’s been pointed out*. The moment we say it, it’s clear that the planes that return home differ from those that get shot down. But this much wasn’t obvious to the USAF at the time, which was thinking of armouring the wings. The trick isn’t so much the answer but the question.
We must always ask of anything: is this all we’re seeing, or just a sample? And if it’s a sample, is it a biased sample? And if so, in which way?
We often forget these questions; investors, for example, fail to see that a short run of good performance by a fund manager might be a lucky biased sample of their actual performance.
Which brings me to my fear - that the same is true in politics. Politicians and journalists might, like the USAF before Wald stepped in, be drawing wrong conclusions from biased samples.
An obvious example of what I mean is the vox pop. These are egregious case of sampling bias. This is partly because the sort of people who are on the street in the middle of the afternoon are a biased sample of all people: they’re not at work or trapped at home by a disability. Worse still, the interviewees are themselves selected by editors for those who give answers they want**. The true and honest answer in many cases would be: “Why are you asking me rather than somebody who knows? Fuck off you daft twat.” But that would be left in the editing suite.
This, however, is only one case of many where the opinions we see are a biased sample. Here are some others:
- Financial markets. Investors are a diverse bunch: it takes differences of opinion to make a market. This means there’ll always be someone who’s very bearish, or bullish, or hyper-critical of the government. And this is not to mention that those willing to speak to the media aren’t necessarily a representative sample of all investors. Hard facts about markets are price moves; individual opinions are likely to be evidence of sampling bias.
- X/Twitter is obviously unrepresentative of opinion generally. Not only does the platform attract racists and repel their opponents, but also its algorithms boost rightist tweets and downgrade others.
- MPs’ surgeries. Those who attend these are disproportionately likely to be cranks or those with bees in their bonnets - not all, obviously, but a larger proportion than is found in the general public.
- Legacy media. When newspapers sold millions, it was reasonable to suppose that their opinions were shared by lots of voters. Today, however, hardly anyone buys a print paper: the combined sales of the Mail, Sun, Express and Telegraph are probably less than 1.5 million, just 3% of the electorate. Newspapers now cater to a minority of cranks. We’ve no reason to suppose therefore that their opinions or interests are representative of those of the wider electorate.
- Groupthink. Like many of us, politicians move in narrow circles - in their case, of advisors, journalists and fellow pols. This leaves them vulnerable to groupthink and professional deformation. For example what Simon Wren-Lewis called mediamacro after 2008; or overconfidence about the efficacy of top-down policies; or simply an over-estimation of the extent to which people pay attention to politics.
- Lobbyists. People who organize themselves to form pressure groups or pay lobbyists are a biased sample of all people. As Mancur Olson pointed out, interests that are concentrated among a few people are more likely to get organized than those that are spread across everyone. If ten people stand to gain £1m each from a particular policy they are more likely to form a pressure group than a million people each of whom would gain £20, even though the aggregate gain is larger in the latter case. Lobbyists therefore - by virtue of their very existence - are a biased guide to what is really in the public interest.
You probably won’t be surprised by any of this. Sampling bias is often obvious once it is pointed out. But the thing is that we have to be alert to it. Remembering the question - “is this a representative sample?” - is the difficult thing. The answers are often easy.
Which brings me to my concern: are MPs actually alert to the problem? And it’s a serious one, because all these points imply that the information MPs get - from the media, their constituents or lobbyists - is biased***?
Several things make me suspect not. One is that, under Starmer, Labour’s anti-immigration policy courted the votes of the right without actually winning any. That’s consistent with the party mistaking voices on Twitter and legacy media for wider opinion. It made the mistake of which Bertie Wooster accused Roderick Spode:
The trouble with you, Spode, is that just because you have succeeded in inducing a handful of half-wits to disfigure the London scene by going about in black shorts, you think you’re someone. You hear them shouting “Heil, Spode!” and you imagine it is the Voice of the People. That is where you make your bloomer. What the Voice of the People is saying is: “Look at that frightful ass Spode swanking about in footer bags! Did you ever in your puff see such a perfect perisher?
Two other facts suggest governments neglect sampling bias: the complexity of the tax system; and regulatory capture. Both are consistent with governments kowtowing to lobbyists, unaware that these are unrepresentative of the wider public interest which requires strong regulation and a simple tax system.
Of course, none of this would matter so much if politicians were popular. But they’re not - which is a further data point suggesting that they’re listening to the wrong voices.
Now, I might be mistaken here. It could be that ministers in the Starmer government weren’t really interested in accurately gauging public opinion or (often a different thing!) the public interest. Maybe all they wanted was to defeat the left and get well-paid jobs after politics.
If politicians do want an accurate picture of public opinion, however, then they have a problem, because the traditional means of discovering it are now prone to significant sampling bias.
* Though Wald’s mathematics (pdf) was not so obvious!
** Years ago, a friend was stopped in the street by a voxpopper doing a piece on a survey that had shown Oakham to be one of the best places to live in the country. “What’s it like living here?” she asked. “It’s good”, he said. “Oh, I’m having a bad afternoon” she replied. “I can’t find anyone to say anything bad. “OK” he replied, “there’s not much for young people to do.” Guess what appeared.
*** A further problem is that a lot of public opinion is itself formed by the sampling bias, and not just because of media bias. But that’s another story.
I got vox popped once and gave a detailed two minute long answer, which alarmed the journalist and obviously didn’t appear in the news— they’d have lost all their viewers, understandably. But it made me more sympathetic to the public.
I do think the fact that nobody reads the papers genuinely hasn’t seeped through, though. I think in that case there’s possibly a Kuhnian thing going on?
The Older People Who Know How Things Are Done need legacy media to matter as much as it once did, because otherwise their knowledge of How The World Really Is won’t actually work and their grip on power becomes fragile. I think the whole Starmer story makes sense through a Kuhnian lens
As a legacy media reader of the FT and Observer I count myself a well informed dinasaur