See No Evil (Part 17 of Cognitive Biases)

Our 17th installment features the selection bias, selective perception in general, and the self-fulfilling prophecy.


Selection Bias

There’s a growing argument that telephone polls, once the gold standard of scientific opinion surveys, are becoming less reliable. More and more people are refusing to participate, meaning that the actual sample becomes to some extent self-selected: a random sample of people who like to take polls. People who don’t like to take polls are underrepresented in the results, and there’s no guarantee that class feels the same as the class answering.

Selection bias can happen in any scientific study requiring a statistical sample that is representative of some larger population: if the selection is flawed, and if other statistical analysis does not correct for the skew, the conclusions are not reliable.

There are several types of selection bias:

• Sampling bias. Systemic error resulting from a non-random population sample. Examples include self-selection, pre-screening, and discounting test subjects that don’t finish.
• Time interval bias. Error resulting from a flawed selection of the time interval. Examples include starting on an unusually low year and ending on an unusually high one, terminating a trial early when its results support your desired conclusion or favoring larger or shorter intervals in measuring change.
• Exposure bias. Error resulting from amplifying trends. When one disease predisposes someone for a second disease, the treatment for the first disease can appear correlated with the appearance of the second disease. An effective but not perfect treatment given to people at high risk of getting a particular disease could potentially result in the appearance of the treatment causing the disease, since the high-risk population would naturally include a higher number of people who got the treatment and the disease.
• Data bias. Rejection of “bad” data on arbitrary grounds, ignoring or discounting outliers, partitioning data with knowledge of the partitions, then analyzing them with tests designed for blindly chosen ones.
• Studies bias. Earlier, we looked at publication bias, the tendency to publish studies with positive results and ignore ones with negative results. If you put together a meta-analysis without correcting for publication bias, you’ve got a studies bias. Or you can perform repeated experiments and report only the favorable results, classifying the others as calibration tests or preliminary studies.
• Attrition bias. A selection bias resulting from people dropping out of a study over time. If you study the effectiveness of a weight loss program only by measuring outcomes for people who complete the whole program, it’ll often look very effective indeed — but it ignores the potentially vast number of people who tried and gave up.

In general, you can’t overcome a selection biases with statistical analysis of existing data alone. Informal workarounds examine correlations between background variables and a treatment indicator, but what’s missing is the correlation between unobserved determinants of the outcome and unobserved determinants of selection into the sample that create the bias. What you don’t see doesn’t have to be identical to what you do see.


Selective Perception

Expectations affect perception.

We know people are suggestible: several studies have shown that students who were told they were consuming alcohol when they weren’t still got drunk enough their driving was affected.

In one classic study, viewers watched a filmstrip of a particularly violent Princeton-Dartmouth football game. Princeton viewers reported seeing nearly twice as many rule infractions committed by the Dartmouth team than did Dartmouth viewers. One Dartmouth alumnus did not see any infractions committed by the Dartmouth side and sent a message that he’d only seen part of the film and wanted the rest.

Selective perception is also an issue for advertisers, as consumers may engage with some ads and not others based on their pre-existing beliefs about the brand. Seymour Smith, a prominent advertising researcher, found evidence for selective perception in advertising research in the early 1960s. People who like, buy, or are considering buying a brand are more likely to notice advertising than are those who are neutral toward the brand. It’s hard to measure the quality of the advertising if the only people who notice it are already predisposed to like the brand.



Self-Fulfilling Prophecy

A self-fulfilling prophecy is a prediction that directly or indirectly causes itself to become true, by the very terms of the prophecy itself, due to positive feedback between belief and behavior. The term was coined by sociologist Robert K. Merton, who formalized its structure and consequences in his 1949 book Social Theory and Social Structure.

A self-fulfilling prophecy is initially false: it becomes true by evoking the behavior that makes it come true. The actual course of events is offered as proof that the prophecy was originally true.

Self-fulfilling prophecies have been used in education as a type of placebo effect.

The effects of teacher attitudes, beliefs and values, affecting their expectations have been tested repeatedly. A famous example includes a study where teachers were told arbitrarily that random students were "going to blossom". The prophecy indeed self-fulfilled:  those random students actually ended the year with significantly greater improvements.


For previous installments, click on "Cognitive Bias" in the tag cloud to your right.