There’s a joke I have often shared amongst my nerdier friends if our conversations have ever turned to anecdotal evidence. It originates from the brilliant webcomic Saturday Morning Breakfast Cereal (SMBC):
For those who have ever had to argue with someone over the validity of anecdotal evidence, it feels like an all too familiar jab at this experience (in the words of Gandalf from Lord of the Rings, “that wound will never fully heal. He will carry it the rest of his life”). But to those who have limited experience within the framework of science, or for whom their thoughts have never turned to the subjectivity of anecdotes, this may seem like an almost untenable position to hold. Isn’t science, after all, based on anecdotes? How else would we know to investigate a claim, if we had never heard of the claim to begin with? And even if that weren’t the case, if it’s so bad, why do so many careers – journalism and politics in particular – almost seem to rely upon them? Here, I will seek to explain why scientists divorce their work from anecdotal evidence, in addition to outlining why anecdotal evidence forms such a terrible foundation for our beliefs and the deleterious impact it has on our society more generally, referencing specifically the “Sudanese gangs” incident during the 2018 Victorian state election.
First, it must be outlined what I mean when I say “anecdote”. By that, I mean a personal account, specifically in this instance about the efficacy of a “cause and effect” relationship. So, for example, I drank some coffee and felt tired afterwards, therefore the coffee made me tired. Syllogistically, one could write this as:
1) I did A, and then B happened,
2) therefore B happened because of A.
Science is not predicated on anecdotes. Anecdotes are a bad form of evidence, if they are even accepted as evidence at all. They are far, far too prone to the post hoc ergo propter hoc (before this, therefore because of this) fallacy. After all, in my example prior, maybe I had decaf coffee and had just had an exhausting day. Maybe I’m a coffee fiend and I need an octuple-shot 16 oz. caramel macchiato with four equals to feel any effect anymore. There is also the possibility coffee really does make me tired. Or maybe I’m just lying to you to provoke a reaction (hands up if you’ve ever had this conversation: “coffee? No thanks, that stuff doesn’t even work on me. Did I ever tell you the time…”). The point is, anecdotes just aren’t very reliable datapoints. They are open to confabulation, deceit, or subconscious cognitive biases.
That doesn’t mean anecdotes can’t be good starting points for further research. Maybe you’ve heard that willow bark is an effective form of pain relief. As it turns out, the bark of willow trees really does help to relieve pain because it contains the chemical salicin. Investigating chemicals like salicin eventually led Charles Frédéric Gerhardt to synthesise acetylsalicylic acid, more commonly known as aspirin. But scientists don’t regard anecdotes as evidence, hence why – if they do hear of an interesting and theoretically sound anecdote – they research whether it’s true or not. The subjectivity of anecdotal “evidence” leads to an incapability of performing any useful statistical analysis on them. You can’t really produce any models from anecdotes, either. For that, you need empirically collected objective data. Scientists are human and are prone to mistakes in logic. Science, as a tool, aims to separate our subjective experiences away from objective reality. That is why, both for the good of a scientists’ reputation and the efficacy of their research, anecdotes are rightly disregarded. There is a reason the only medical “cures” which are based on anecdotal “evidence” are things like homeopathy and naturopathic remedies, and why anti-vaxxers rely so heavily on anecdotes; scientists have done these studies, and they found no objective data supporting these conclusions.
That is science though. Why should we be so clinical in our approach to our own beliefs? It all relates to issues of representation, controls and sample size. Imagine this: you are mugged by someone with a beard. Then you hear all over the news that, according to politicians, there is a major problem with bearded people mugging random civilians. There are gangs of these bearded fellows roaming the streets. The politicians know this, of course, because they claim all their constituents called them up and told them their own personal accounts. The media too is getting calls about such anecdotes. Well, you’ve been mugged yourself by a man with a beard, so you come to the immediate conclusion that beards cause people to commit more crimes. You are now, in your mind, terrified and resentful towards bearded men. You call up your own representative and your favourite local current affairs program, A Present Affair, to report your own experience. There are three major underlying issues with this approach.
First, you are biased towards reporting your experiences. No one is reporting the tens of thousands of bearded men who go about their lives contributing positively to the community, but at this point everyone would be reporting every remotely bad experience they’ve ever had with these men. These experiences would also be tainted by people who naturally hate men with beards and will make up stories about them, and by minor events which are boosted to seem worse than they really were (an accidental bump in the street can become a deliberate shove upon faulty recollection). This is a problem with the representation of this data.
Second, the politicians and the news almost certainly did not consider alternative factors that explain the data and did not implement any controls to account for this. Let’s say people with beards really were disproportionately represented in the crime statistics. What if the bearded population tends to be younger than other groups of people, something which is correlated with higher rates of criminality? Or what if the bearded population is more dense in areas of lower socio-economic class, which is also correlated with higher rates of criminality? Or what if we didn’t control for the fact that people with beards are overwhelmingly men (no offence to Lettie Lutz), and hence our statistics weren’t compared to other groups of men like it should have been but against the population more generally? If we don’t consider how these factors colour our anecdotal “evidence”, then we might miss out on the real underlying cause of this behaviour.
Finally, our sample size simply might be too small. How many people with beards are there in, say, my home state of Queensland? Well, according to the 2017 census, there are about 2.4 million males living there. If we guess that 5% of those people have beards (a very generous underestimation), then that leaves about 120,000 people with beards. How many reports would we need of men with beards terrorising local communities before there would be sufficient justification that we should be afraid of bearded men? 10? 100? Even if there were 1,000 genuine reports of specific individuals, would that be enough to justify our vilification of that community? Probably not, because the caveat is always that there are 119,000 other bearded men who aren’t terrorising local communities. Of course, statisticians have many ways to account for sample sizes much smaller than the actual population, but since journalists typically aren’t well versed in statistical methodology, it would be highly doubtful that they have properly taken this consideration into account.
This example seems silly, of course. We know that just because men have beards, that doesn’t mean the beards lead to higher criminality. But of course, my example shouldn’t be too unfamiliar to any Australians, given this was exactly what the Liberal National Party (LNP) and the commercial news media partook in against the Sudanese population in the state of Victoria. Why did the LNP do this? Because there was an election soon, and the LNP wanted to scare people into voting for them as the political party that was “tough on crime”. Why did the commercial news media report on it? Because people wanted to consume more of these stories (I suspect out of confirmation bias), and they could make more money if more people were tuning in. The sad thing is people easily bought into the dog-whistling, the anecdotes and the shoddy statistics. People legitimately believed that because of where you were from – which they could infer from the colour of your skin – you were more likely to commit gang-related crimes. This is the general repugnant effect that falling for anecdotal evidence can have, however. People, based almost entirely on their skin colour, were now vilified by others, not because of their actions, but because of the actions of a minority of their population which had been overblown and shoved down our throats.
There was no evidence to support this claim beyond one datapoint that showed the Sudanese population was overrepresented in the crime statistics. There are multiple explanations for this beyond where they were from, such as the age distribution and the socio-economic class density of this group. Another possible explanation relates to sample sizes; the Sudanese population is only about 6,000, and so even if only one or two gangs legitimately existed, that would severely impact the overall statistics. None of this changes the fact that you were 11 times more likely to be burgled under a circumstance of aggravation, 51 times more likely to be burgled without a circumstance of aggravation, or 34 times more likely to be seriously assaulted by someone who wasn’t from Sudan in the year 2016-2017. You won’t find that reported by the commercial news media, but you will find it reported by the Victorian Crime Statistics Agency. Overall, people from Sudan – who make up 0.1% of the Victorian population – only committed 1% of all crimes for that year. That is an overrepresentation, but not one that is without reasonable explanation. Despite all this, people believed in the anecdotes over the objective facts.
It is true that, sometimes, anecdotes are correct. But it is also true that, if an anecdote is correct, then objective data and statistics will bolster it. We shouldn’t outright ignore personal anecdotes – consideration of people’s personal experiences is vital to empathising and understanding other’s perspectives, and can sometimes lead to the discovery of scientific breakthroughs or genuine societal issues – but we shouldn’t consider their claims valid until we can back it up with good science or good statistics. If we let people with an agenda dictate what beliefs we argue over, and what evidence is acceptable to substantiate those beliefs, then the argument is already moot. As a collective, we should be better than that. As a society, we deserve better than that. As a country, we are better than that.