There are 5 unlabeled glasses of wine from 5 different wineries in front of you. Your research-task is to do a blind tasting and assign each glass to the correct winery.

You guess wrong on one of the glasses.

If the rules are that all of the glasses must have a winery assigned, the consequence is that you are forced to answer the wrong winery for two of the glasses.

There are three ways you can report this result:

1. 3/5 correct

2. Missed two glasses

3. Missed one glass

Which illustrates how you can choose to publicly report research results depending on the reader response you want to evoke. Headlines you can choose range from “Most people guess right when wine tasting” to “People cannot reliably taste the difference between wineries”.

So most science news headlines are inaccurate one way or the other.

If you gather data from a study-group, you risk reporting headlines like “4/5 wines identified correctly in wine-tasting test”, which is an impossible result given your study set-up (only 5/5, 3/5 and 1/5 are possible outcomes).

So some science news headlines are wrong.

If you in your effort to fix these problems in your study, open up for the possibility that your research participants can report their “lived experience” from the wine tasting, you risk ending up with headlines like “A wine for any occasion – how differences in wine tasting experiences are dependent on gender and social status”. Which is probably stretching how much you can generalize from your data, but nevertheless probably correct given how you decided to do your data-analysis.

So some science headlines are confusing and possibly misleading.

Fact remains though: You guessed wrong on one of the glasses

So, my plea. Be a rational and skeptical reader. Please don’t forget the underlying facts and please ensure open access to the data so that anyone and everybody, can scrutinize if they wish.