[HN Gopher] Why most published research findings are false (2005)
___________________________________________________________________
Why most published research findings are false (2005)
Author : Jimmc414
Score : 85 points
Date : 2022-10-19 17:39 UTC (5 hours ago)
(HTM) web link (journals.plos.org)
(TXT) w3m dump (journals.plos.org)
| apienx wrote:
| Related: https://en.m.wikipedia.org/wiki/Half-life_of_knowledge
|
| Psychology is the worst offender. Human behavior is quite hard to
| model. ;-)
| JacobThreeThree wrote:
| The journal linked in the OP is apparently one of the better
| and more rigorous journals.
|
| For example:
|
| >Public Library of Science, PLOS ONE, was the only journal that
| called attention to the paper's potential ethical problems and
| consequently rejected it within 2 weeks.
|
| https://www.theguardian.com/higher-education-network/2013/oc...
| slt2021 wrote:
| Is this research finding ("Why most published research findings
| are false") also false?
|
| Which means published research findings are true? But then it
| will turn all research findings false, which is impossible
|
| I am confused
| est wrote:
| you are confused with the concept of "most"
| realaleris149 wrote:
| including this one?
| planetsprite wrote:
| predictable joke
| Sakos wrote:
| Is it a joke?
| fastaguy88 wrote:
| This was a very controversial paper when it was published,
| perhaps because of its incendiary title. But the paper is much
| more subtle than the title suggests. Basically the idea is that
| if you try to test phenomena that are completely unexpected, your
| prior odds are low, so even if you get a positive result, there
| is a good chance the result is incorrect. So there is a danger
| that by trying to ensure more correct results, scientists may
| avoid paradigm changing experiments, which would be a loss.
|
| Follow up analysis by Jager and Leek (2014)
| https://academic.oup.com/biostatistics/article/15/1/1/244509
| suggests the false discovery rate is closer to 14% than 50%.
| 6gvONxR4sf7o wrote:
| I'd love to know the correlation between failure to replicate
| and public virality of findings. I wouldn't be surprised of the
| more exciting findings replicate at a different rate from
| average.
| H8crilA wrote:
| That's kind of the point. More exciting statements are more
| often "proven" because they're more often attempted to be
| proven.
| nextos wrote:
| A good way to address these issues is to frame all experiments
| as multilevel models. See [1] for a long discussion from Andrew
| Gelman et al on why this is advisable.
|
| Surprisingly, many people working in statistics still ignore
| the James-Stein theorem, which provides a theoretical
| justification for multilevel models. In layman terms, said
| theorem shows that if you are simultaneously estimating many
| random variables you should borrow information across variables
| [2]. Estimating them one by one is suboptimal and does not
| minimize the global mean squared error.
|
| Multilevel models "shrink" individual effect sizes by looking
| at the overall distribution of effect sizes and provide much
| more realistic estimates.
|
| [1] https://arxiv.org/abs/0907.2478
|
| [2] https://en.wikipedia.org/wiki/Stein%27s_example
| themitigating wrote:
| That's a huge difference yet anti-intellectuals will contiune
| to post this paper until the end of time as proof that you
| can't trust science
| timr wrote:
| It's also wrong. There have now been large-scale initiatives
| to reproduce papers, and they're getting higher failure rates
| than 14%.
|
| The statistics underlying all of this make a number of
| implicit assumptions that may or may not be true IRL (for
| example: that journals publish papers independent of the
| salaciousness of claims made within them). If Science picks
| out only the top-5% most-sensational claims for publication,
| then you can't assume that a 95% CI is a safe threshold.
| You've probably got to increase it to a much higher value to
| have any prayer of getting past the inherent bias in such a
| process.
| serial_dev wrote:
| Mentioning that not everything that's published in a peer
| reviewed publication is automatically 100% correct is not
| anti-intellectual.
|
| Mistakes can be made, data can be limited or misinterpreted,
| scientists can be corrupted. Theories, "common sense" can
| change: what we thought we'd know for sure have been proven
| wrong. If you are a scientist, you know that.
|
| Only because I don't automatically take every "scientific"
| finding at face value, it doesn't mean I don't trust science.
|
| I trust science, I just don't trust _every single_ research,
| experiment, scientist, or journal. Actually that 's in
| itself, in a way, science.
| afpx wrote:
| "Science" currently incentivizes the wrong things. How do
| you change the incentives (back)?
| chithanh wrote:
| I think it is a misconception that switching to the
| "right" incentives will solve problems. The very
| existence of incentives is not conducive for knowledge-
| based work.
|
| https://hbr.org/1993/09/why-incentive-plans-cannot-work
| hbn wrote:
| Yup. Blindly "trusting the science" is inherently anti-
| science
|
| There is plenty of things that we took as scientific fact
| in the past that turned out to be false. Science is
| supposed to question existing notions and test/prove new
| theories. If you just assume that we're in a post-science
| era where we've got it all figured out, and our current
| theories are all correct, that's not science, it's dogmatic
| faith.
|
| Ignaz Semmelweis was destroyed by the medical community for
| his insane idea that doctors should wash their hands before
| performing medical procedures. He was attacked about to the
| point where he ended up having a nervous breakdown and was
| committed to an asylum where he was beaten to death by the
| guards. Serves him right for questioning the science!
| dekhn wrote:
| We've optimized since Ignaz; now, you only have to infect
| and then cure yourself to win a Nobel after being doubted
| (https://www.nytimes.com/2005/10/11/health/nobel-came-
| after-y... the scientist actually infected themself with
| helicobacter pylori, got sick in the predictable way, and
| cured it with antibiotics)
|
| The process by which DNA was convincingly demonstrated to
| be the molecule of hereditary was fairly complex; an
| early experiment that was complex and hard to understand
| did so, but people didn't completely believe it so a
| later experiment that was easier to understand was done.
|
| For the longest time, the establishment believed the
| functionality of the ribosome (a critical subsystem that
| translates mRNA into protein) was carried out by its
| protein subunits. Although convincing data was published
| in the 1960s, the general belief did not change until the
| crystal structure of the ribosome was published showing
| that RNA formed the catalytic component.
|
| And my personal favorite, it was considered unpossible
| that prions could be caused by proteins that misfolded
| and caused other proteins to misfold, it required
| absolutely heroic efforts in the face of extraordinary
| pressure to establish the molecular etiology of prions in
| the minds of the establishment.
|
| It's hard to change your mind. Some people never will.
| d0mine wrote:
| "Science advances one funeral at a time."
| https://quoteinvestigator.com/2017/09/25/progress/
| foobarbecue wrote:
| Medicine was not at all scientific until recently.
| There's a great podcast series about the rise of
| evidence-based medicine (over authority-based medicine)
| that I loved, but I can't seem to find it now... Anybody
| know what I'm thinking of?
| [deleted]
| yucky wrote:
| The Replication Crisis is real, denying that is anti-
| intellectual.
|
| https://en.wikipedia.org/wiki/Replication_crisis
| Retric wrote:
| Calling something a crisis is arbitrary. So it both is and
| isn't a crisis based on completely arbitrary metrics.
| yucky wrote:
| I encourage you to dig in to the sources posted. The
| majority of published research can't be and/or hasn't
| been replicated. That's a crisis by any definition,
| especially with the amount of idiots running around
| bleating about "the science is settled" or "trust the
| science" or whatever catch phrase.
| Retric wrote:
| The degree to which the science is settled varies wildly
| by topic and that's ok. Nutrition is perceived to be
| filled with a lot of junk, but without dietary vitamin C
| you will die. That's settled even inside a field filled
| with debate.
|
| Individual papers where never intended to be the final
| arbiters of truth, that's not their role. If nobody
| thinks things are worth looking into again then stuff
| stays in a very nebulous state which is no worse than
| where things where before a paper was published.
| [deleted]
| treeman79 wrote:
| It's not that you can't trust "science" it's that people are
| fallible.
| diognesofsinope wrote:
| I mean, the failure of a lot of experiments to replicate is
| extraordinarily well documented...
|
| Not to mention the corollaries from the Wikipedia article (ht
| tps://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
| ):
|
| "In addition to the main result, Ioannidis lists six
| corollaries for factors that can influence the reliability of
| published research.
|
| Research findings in a scientific field are less likely to be
| true,
|
| 1. the smaller the studies conducted.
|
| 2. the smaller the effect sizes.
|
| 3. the greater the number and the lesser the selection of
| tested relationships.
|
| 4. the greater the flexibility in designs, definitions,
| outcomes, and analytical modes.
|
| 5. the greater the financial and other interests and
| prejudices.
|
| 6. the hotter the scientific field (with more scientific
| teams involved)."
|
| These are all reasonable criticisms from my own area of
| expertise: applied econometrics.
| nonameiguess wrote:
| Being skeptical of cutting edge research is a lot different
| than distrusting science. There are plenty of people out
| there denying special relativity, evolution by natural
| selection, or believing all of western medicine is invalid,
| based on extrapolation from stuff like this.
|
| I won't speak for all textbooks, but generally stuff you
| find in there should not be the same as what you find in
| journals, and is much more settled. Big caveat that that
| isn't necessarily true for younger sciences without long-
| established theory, say exercise physiology or social
| psychology, but something like a chemistry textbook is
| pretty damn trustworthy.
|
| And those are what people who aren't actually scientists
| should mostly be educating themselves with, not newspaper
| science reporting sections.
| glofish wrote:
| This followup paper seems to summarize p-values reported in 77K
| papers and uses the distribution of these p-values to compute
| the FDR.
|
| Alas everyone knows that the p-values reported in accepted (!)
| papers are questionable at best - any analysis that uses them
| is on shaky foundation.
|
| The misuse of p-values in science is well known and an endemic
| problem - so what do we learn from a re-analysis of made up
| numbers?
| Manu40 wrote:
| Not to be pedantic, but considering how many people I have
| run into who take all research studies as if they are holy
| gospel; you may want to reconsider saying "everyone knows".
|
| Just my 5 cents on the matter.
| glofish wrote:
| good point, what I really meant is
|
| _by now everybody should know that p-values are
| questionable at best_
| user3939382 wrote:
| Related, if you're interested in this topic:
| https://www.youtube.com/watch?v=z6IO2DZjOkY
|
| Basically, the corruption of medical science and practice.
| Dawnyhf5 wrote:
| elmolino89 wrote:
| While using the software packages released last week may not be
| the greatest idea, I am leery of any research paper using 10
| years old genome assembly, Ensembl annotation release in the
| 40ish (we are at #105 ) or clearly outdated program versions with
| X updates in the last 5 years. Also if Fedex/UPS were tracking
| packages with bordering on cavalier attitude observed in some
| labs, often we would be getting bags of guano ordered by some
| horticulturalist instead of a book of our choice. Or even an
| empty bag, since QC may be an afterthought and hard wet lab work
| may still produce unusable crappy data.
|
| On the other hand the brand new technologies are rather
| expensive, good quality human tissue samples hard to get so
| scrapping the bottom of the barrel trying to justify grant $$$ is
| unavoidable.
| stonemetal12 wrote:
| >Ensembl annotation release in the 40ish (we are at #105 )
|
| Were there any major bug fixes that would impact their work in
| that time window? Maybe for them it is done instead of
| outdated.
| elmolino89 wrote:
| Sure. Some genes were retired because there was not enough
| support. Earlier Ensembl versions used older genome assembly.
| Which means: some genes "jumped" from one chromosome to
| another, or got properly stitched residing before partially
| on floating contigs.
|
| Just to be clear: I am not saying that anything done in 2022
| using hg19 is 100% wrong. Just that it is a bit like using a
| stretched shoelace 50cm (+/- 5cm) to measure your corridor
| when you have a decent tape measure in your pocket.
|
| There are microarrays used in Big Science projects with ~1/3
| of probes not matching human transcripts from latest ENSEMBL.
| Since most of them map to the current genome who knows what
| is the meaning of the signal from such probes. Unannotated
| exons? Retained introns? But some probes do not even map to
| the genome => some silly splicing error(?) packed in a
| plasmid in the 90ies?
| T3RMINATED wrote:
| hackandthink wrote:
| Ioannidis paper really impressed me back then. I was surprised
| when Ioannidis supported shady Covid research. Ideology can catch
| everyone. Beware.
|
| https://sciencebasedmedicine.org/what-the-heck-happened-to-j...
| rossdavidh wrote:
| Ioannidis was the first major name to point out that the
| mortality rate for covid-19 was 0.25-0.5%, at a time when
| figures as high as 1 in 8 were being cited. Abundant evidence
| since then, including CDC's official estimates in the US,
| suggest he was correct. So, you know, beware, ideology can
| catch everyone. Even you.
| janef0421 wrote:
| The reasonableness of a statement should be based on how well
| it is supported at the time it is made, not whether it is
| later demonstrated to be correct.
| trention wrote:
| The first estimates from Ioannidis from the flawed (in every
| possible way) California paper were less than 0.1% IFR. He
| later revised them to 0.16% IFR, not sure if subsequent
| "revision" (=admission of being wrong) was done afterwards.
| Bear in mind, that was his IFR estimate for March-April 2020,
| when a lot of treatment was being done wrong (=intubate
| early) and even steroids were not supposed to be used for
| treatment outside RCTs.
|
| Meanwhile, the CFR at Diamond Princess was at 2.6%, so
| refrain from the idiocy "1 in 8 IFR estimates" as only
| completely uninformed people will fall for them.
| [deleted]
| daze42 wrote:
| There appears to be an error in the correction posted in August
| of this year. See if you can spot it.
| usgroup wrote:
| This one?
|
| https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
|
| Lol yes, research finding Yes, true relationship No is missing
| a left bracket in the denominator.
| H8crilA wrote:
| Here's an XKCD summary of the paper: https://xkcd.com/882/
|
| It really is that simple.
| dang wrote:
| Related:
|
| _Most Published Research Findings Are False (2005)_ -
| https://news.ycombinator.com/item?id=19016399 - Jan 2019 (39
| comments)
|
| _Why Most Published Research Findings Are False (2005)_ -
| https://news.ycombinator.com/item?id=18106679 - Sept 2018 (40
| comments)
|
| _Most Published Research Findings Are False-But Little
| Replication Goes Long Way_ -
| https://news.ycombinator.com/item?id=9337355 - April 2015 (3
| comments)
|
| _Why Most Published Research Findings Are False_ -
| https://news.ycombinator.com/item?id=8340405 - Sept 2014 (2
| comments)
|
| _Why most published scientific research is probably false
| [video]_ - https://news.ycombinator.com/item?id=6661710 - Nov
| 2013 (53 comments)
|
| _Most published research results are false_ -
| https://news.ycombinator.com/item?id=2207750 - Feb 2011 (15
| comments)
|
| _Why Most Published Research Findings Are False_ -
| https://news.ycombinator.com/item?id=1825007 - Oct 2010 (40
| comments)
|
| _Lies, Damned Lies, and Medical Science_ -
| https://news.ycombinator.com/item?id=1793838 - Oct 2010 (27
| comments)
|
| _Lies, Damned Lies, and Medical Science_ -
| https://news.ycombinator.com/item?id=1787055 - Oct 2010 (2
| comments)
|
| _Why Most Published Research Findings Are False (2005)_ -
| https://news.ycombinator.com/item?id=833879 - Sept 2009 (2
| comments)
| hdesh wrote:
| Dang (pun intended), is there a trick to find all these similar
| articles?
| zaik wrote:
| https://hn.algolia.com/
| trention wrote:
| I am not completely sure whether most of Ioannidis' own research
| is "false" - but all of his covid takes (in whatever form) are.
| janef0421 wrote:
| That's totally normal. An individual study is unlikely to have
| enough statistical power to make a definite conclusion, and an
| individual line of evidence is insufficient to confirm a theory.
| No conclusion can be stated with any certainty until it is
| verified repeatedly, and no theory is fit for use until several
| consistent lines of evidence conform to its predictions. In fact,
| even calling these findings "false"is often inaccurate, as most
| researchers don't make strong claims.
___________________________________________________________________
(page generated 2022-10-19 23:00 UTC)