Cherry picking data is a logical fallacy where only information that support’s one’s conclusion is shared; contrary information is either ignored or suppressed. It’s also called “the fallacy of incomplete evidence,” which probably makes it more clear what is going on.
Scientists are supposed to watch out for this; the process of evaluating data specifically includes performing statistical evaluations of all the findings to determine the possibility that the results are actually a coincidence rather than the result of the intervention. (Nerdly aside: in scientific papers you’ll see something like “p=.005” to indicate this; that means that, statistically, there are 5 chances in 1000 that the results were a coincidence, and conversely, 995 chances out of a thousand that the same results will happen the next time. A p value of less than .05 is considered statistically significant.)
Ahem.
So cherry picking is a bad thing, scientifically speaking. In politics it happens all the time; you’ll hear this factoid or that statistic being trotted out, and when you dig into it you find that it’s not at all representative of reality.
A great example of this is in the area of anthrogenic global warming (climate change caused by humans). As I’ve written in previous posts, there is a rather vocal group (predominately Republicans) who deny that our climate is changing (getting warmer), and even if it is, it’s not caused by human activity. They have pointed to an increase in arctic ice to prove their point; if the arctic ice pack is growing, it’s pretty difficult to say the environment is getting warmer. And to support this they say that the size and thickness of the arctic sea ice was higher in 2009 than it was 20 years prior to that, in 1989.
Here’s the thing: the data they point to are simply wrong. Below is a graph showing the measurements of the thickness and extent of the arctic sea ice in 1989 (blue line) and again in 2009 (red line).
Even a casual observation would show that the ice is less in 2009 than it was even 20 years prior. But for one (very) brief moment (in March, where the arrow is pointing), the lines crossed and it seemed to show that the ice pack was ever-so-slightly greater in 2009 than 1989.
This is classic cherry picking. It is obvious when you look over time that the ice pack is getting smaller. It would clearly take more sophisticated statistical analysis to know exactly how much smaller, how fast it’s happening and whether it’s stable, accelerating or slowing, but it’s definitely smaller.
This would be exactly like having your bank account be overdrawn every day for a month except for paydays, and then attempting to convince the bank they were being unfair for charging you overdraft fees by pointing only to paydays as evidence of your sterling record-keeping. It’s just plain wrong.
There are lots of examples of the use of this type of bad science, whether in politics, the news, or in religion today. A good example in religion is the so-called “Bible code.” In the book of the same name, the author maintains that a sophisticated statistical analysis of the words and letters in the Bible (particularly the Hebrew Scriptures, or Old Testament) reveals patterns that make startlingly-accurate predictions of events today, written thousands of years before they happened.
The problem is that if you perform exactly the same analysis on Moby Dick, for example, the same types of patterns emerge. And while most people in Western society would agree that Moby Dick is one of the great works of recent history, I doubt if we’d say it contains predictions of the future.
At its most destructive (and least honest), cherry picking starts with having a point you want to make, and then finding and only reporting the evidence that supports your pre-determined position. The most honest way is to first gather as much evidence as you can, and after careful (and objective) review, determine what the evidence is telling you. Scientists work very hard to do just that; and they don’t always get it right. Confirmation bias shows up frequently, even when it’s being guarded against.
Avoiding this confirmation bias is challenging and takes time, so (not very surprisingly) not many people are willing to go looking for it in their own reasoning. It’s way easier to start with your conclusion and cherry pick your “facts.” In fact, I would suggest that we all do it by default, unless we actively and specifically guard against it.
It’s surprisingly difficult.
About BigBill
Stats: Married male boomer.
Hobbies: Hiking, woodworking, reading, philosophy, good conversation.
Picking cherries
Cherry picking data is a logical fallacy where only information that support’s one’s conclusion is shared; contrary information is either ignored or suppressed. It’s also called “the fallacy of incomplete evidence,” which probably makes it more clear what is going on.
Scientists are supposed to watch out for this; the process of evaluating data specifically includes performing statistical evaluations of all the findings to determine the possibility that the results are actually a coincidence rather than the result of the intervention. (Nerdly aside: in scientific papers you’ll see something like “p=.005” to indicate this; that means that, statistically, there are 5 chances in 1000 that the results were a coincidence, and conversely, 995 chances out of a thousand that the same results will happen the next time. A p value of less than .05 is considered statistically significant.)
Ahem.
So cherry picking is a bad thing, scientifically speaking. In politics it happens all the time; you’ll hear this factoid or that statistic being trotted out, and when you dig into it you find that it’s not at all representative of reality.
A great example of this is in the area of anthrogenic global warming (climate change caused by humans). As I’ve written in previous posts, there is a rather vocal group (predominately Republicans) who deny that our climate is changing (getting warmer), and even if it is, it’s not caused by human activity. They have pointed to an increase in arctic ice to prove their point; if the arctic ice pack is growing, it’s pretty difficult to say the environment is getting warmer. And to support this they say that the size and thickness of the arctic sea ice was higher in 2009 than it was 20 years prior to that, in 1989.
Here’s the thing: the data they point to are simply wrong. Below is a graph showing the measurements of the thickness and extent of the arctic sea ice in 1989 (blue line) and again in 2009 (red line).
Even a casual observation would show that the ice is less in 2009 than it was even 20 years prior. But for one (very) brief moment (in March, where the arrow is pointing), the lines crossed and it seemed to show that the ice pack was ever-so-slightly greater in 2009 than 1989.
This is classic cherry picking. It is obvious when you look over time that the ice pack is getting smaller. It would clearly take more sophisticated statistical analysis to know exactly how much smaller, how fast it’s happening and whether it’s stable, accelerating or slowing, but it’s definitely smaller.
This would be exactly like having your bank account be overdrawn every day for a month except for paydays, and then attempting to convince the bank they were being unfair for charging you overdraft fees by pointing only to paydays as evidence of your sterling record-keeping. It’s just plain wrong.
There are lots of examples of the use of this type of bad science, whether in politics, the news, or in religion today. A good example in religion is the so-called “Bible code.” In the book of the same name, the author maintains that a sophisticated statistical analysis of the words and letters in the Bible (particularly the Hebrew Scriptures, or Old Testament) reveals patterns that make startlingly-accurate predictions of events today, written thousands of years before they happened.
The problem is that if you perform exactly the same analysis on Moby Dick, for example, the same types of patterns emerge. And while most people in Western society would agree that Moby Dick is one of the great works of recent history, I doubt if we’d say it contains predictions of the future.
At its most destructive (and least honest), cherry picking starts with having a point you want to make, and then finding and only reporting the evidence that supports your pre-determined position. The most honest way is to first gather as much evidence as you can, and after careful (and objective) review, determine what the evidence is telling you. Scientists work very hard to do just that; and they don’t always get it right. Confirmation bias shows up frequently, even when it’s being guarded against.
Avoiding this confirmation bias is challenging and takes time, so (not very surprisingly) not many people are willing to go looking for it in their own reasoning. It’s way easier to start with your conclusion and cherry pick your “facts.” In fact, I would suggest that we all do it by default, unless we actively and specifically guard against it.
It’s surprisingly difficult.
About BigBill
Stats: Married male boomer. Hobbies: Hiking, woodworking, reading, philosophy, good conversation.