Statistical Liars

The Brit medical magazine Lancet and the state-funded BBC both use statistics to tell lies.

The origin of the expression “lies, damned lies and statistics” is unknown, but expresses a fundamental truth – by dishonest sampling methods and misinterpretation, statistical surveys can prove whatever the rogue chooses.

For example the Brit Lancet medical journal last year published a paper claiming 655,000 Iraqis had been killed since liberation. That’s been widely debunked in the blogosphere, and yesterday hit the MSM:

One critic is Professor Michael Spagat, a statistician from Royal Holloway College, University of London…

“The authors ignore contrary evidence, cherry-pick and manipulate supporting evidence and evade inconvenient questions,” contends Professor Spagat, who believes the paper was poorly reviewed. “They published a sampling methodology that can overestimate deaths by a wide margin but respond to criticism by claiming that they did not actually follow the procedures that they stated.” The paper had “no scientific standing”.

Did he rule out the possibility of fraud? “No.”

Today, the BBC published this:

Israel, Iran and the United States are the countries with the most negative image in a globe-spanning survey of attitudes toward 12 major nations. Canada and Japan came out best in the poll, released Tuesday.

The survey for the British Broadcasting Corp.’s World Service asked more than 28,000 people to rate 12 countries – Britain, Canada, China, France, India, Iran, Israel, Japan, North Korea, Russia, the United States and Venezuela – as having a positive or negative influence on the world.

Canada had the most positive rating in the survey of 28,389 people in 27 countries, with 54 percent viewing it positively and 14 percent negatively. It was followed by Japan and France.

That’s not unexpected – the weak always envy and fear the strong.

Still, how did the BBC figure this out? Clothes peg on nose, I downloaded their detailed results and methodology to find out. It’s junk – sample selection varies arbitrarily from country to country (some national, some sample cities), sample size varies bizarrely (Lebanon got 1,200, the US 1,000), samples are tiny (half the respondents were asked about 6 nations, the other half about the other 6). Plus about 6 of the nations surveyed are fear states, raising the French Joke Problem.

But the biggest error is the failure to include a reference question. That’s one that allows you to determine the respondent’s prior bias – for example all US political polls (should) ask a) whether the respondent voted at the last election and b) which party they voted for. That way you can correct for oversampling by comparing with actual votes at the last election.

Party affiliation wouldn’t have worked in this survey since lot of the nations are fear states and don’t have political parties.

So I’d have used a reference test along the lines of “Do you consider your country to be a force for good or bad?”. That gets you a reference level to correct the results with.

The BBC also pulls the same trick as Lancet – not checking against other results. For example the BBC says the EU is considered a force for good by a plurality of all EU nations surveyed, whereas at about the time of the survey:

The annual Eurobarometer poll…shows that on average, only 50 percent of Europeans consider EU membership “a good thing”, down from 54 percent earlier in the year. The traditionally Euroskeptic British are no longer the most hostile, having been overtaken by the Austrians. Only 32 percent of Austrians, and 33 percent of Brits, say EU membership is good for their country. They are followed by Latvia (36 percent), Finland (38 percent) and Sweden and Hungary (both 39 percent).

None of this statistical quibbling is likely to effect the underlying reality that, say, most Europeans envy and fear Americans and Israelis.

But it would be nice if – just for once – the left used good science.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: