[HN Gopher] Why isn't there a replication crisis in math?
___________________________________________________________________
Why isn't there a replication crisis in math?
Author : jseliger
Score : 113 points
Date : 2022-02-02 18:26 UTC (4 hours ago)
(HTM) web link (jaydaigle.net)
(TXT) w3m dump (jaydaigle.net)
| svat wrote:
| Despite the title (a title with a question in it invites people
| to comment without reading the post, even more than the usual
| already high level), this is a really good post IMO, with
| valuable insights into not just mathematics but also the
| replication crisis elsewhere. (And it _does_ discuss Mochizuki 's
| claimed proof of the _abc_ conjecture, and links to MathOverflow
| on formalization with proof assistants, to a recent paper
| discussing Vladimir Voevodsky 's views, etc.) This from the first
| part is sound:
|
| > _The replication crisis is partly the discovery that many major
| social science results do not replicate. But it's also the
| discovery that we hadn't been trying to replicate them, and we
| really should have been. In the social sciences we fooled
| ourselves into thinking our foundation was stronger than it was,
| by never testing it. But in math we couldn't avoid testing it._
|
| But the post doesn't stop there! The second part of the post
| (effect sizes etc), with the examples of power posing and the
| impossibly hungry judges, is even more illuminating. Thanks!
| tabtab wrote:
| because practical applications of such works are usually several
| decades away. A bad medical study can kill people today. There's
| not a lot of incentive to check nor complain about consequences
| far into the future.
| mjw1007 wrote:
| I think it's a good article.
|
| But it's worth considering the possibility that mathematics could
| have fallen, and could still fall, into a state where false
| results are frequently published, and isn't 'protected' by
| anything special in the nature of the field or its practitioners.
|
| Just as you might find yourself asking "why did city A fall into
| the grip of organised crime, and not city B?". You might look for
| answers in the methods of police recruitment or a strong history
| of respect for the rule of law or anything like that, but it
| might turn out that the answer is really just "city A got
| unlucky".
| zitterbewegung wrote:
| Math doesn't have a replication crisis at all it has a
| comprehension crisis.
|
| Since proofs are programs one can basically say that mathematical
| theorems are incredibly detailed software that is completely open
| source and invites people to identify programs that don't work
| and or fix issues.
|
| A famous one is Fermats last theorem which needed a fix but was
| largely right.
|
| Others have said that it takes 6 months to a year to get
| published. The other thing with math is the fact that you can get
| completely scooped and your work is worthless.
|
| Edit: I am using "proofs are programs" very loosely and yes
| Theorems are much more than programs as other commenters have
| pointed out.
| jaydaigle wrote:
| Interesting! My experience is that scooping is less of an issue
| in math than in any of the science fields I have friends in.
| Papers are lower-stakes, there's less money involved, and if
| two of you are working on the same project you can just co-
| author.
|
| (And if you have an independent paper, that can _also_ get
| published; your paper is distinct even if the result isn't. I
| think the PT HOMFLY polynomial was independently proven in like
| four different papers published within two years (and it's
| named so that all eight authors get credit).
|
| But also, publication lags shouldn't lead to more scooping,
| because you can put it up on the arXiv at the beginning of the
| publication process, not the end. In my experience the paper is
| treated as "real" once it hits the arXiv; the acceptance is
| mostly a formality that lets us put it on our promotion packet.
|
| But also, publication times don't lead to scooping generally
| because you
| bawolff wrote:
| > Since proofs are programs one can basically say that
| mathematical theorems are incredibly detailed software that is
| completely open source and invites people to identify programs
| that don't work and or fix issues.
|
| I would say its more like pseudocode. There can be quite a
| large gap between a normal proof, and a machine checkable
| proof, which is the computer program version.
| pthread_t wrote:
| > The other thing with math is the fact that you can get
| completely scooped and your work is worthless
|
| Why math specifically? One would think this applies in
| virtually all fields.
| zitterbewegung wrote:
| Increase in development time (publication can take 6 months
| to a year).
| contravariant wrote:
| A more notorious example of the comprehension crisis would be
| Mochizuki's claimed proof of the abc conjecture. So far fairly
| few people are willing to claim they both understand and agree
| with the several hundred pages of 'proof'.
| zitterbewegung wrote:
| I was tempted to use that but Fermat's last theorem is known
| to the general public for much longer and has a resolution.
| exdsq wrote:
| https://en.wikipedia.org/wiki/Shinichi_Mochizuki
|
| It's a fun rabbit hole to go down :)
| mb7733 wrote:
| > Since proofs are programs one can basically say that
| mathematical theorems are incredibly detailed software that is
| completely open source and invites people to identify programs
| that don't work and or fix issues.
|
| I don't think it's that straightforward; proofs in papers are a
| mix between explanation in natural language and mechanical
| steps. Not every step of deduction can feasibly be written out.
| That's part of why computer aided proofs are not that popular
| in math.
| Quekid5 wrote:
| An interesting perspective on programs-as-proof is I-forget-
| his-name-but... the mathematician who made really bold claims,
| if _only_ you 'd study under his tutelage for a number of years
| to understand this whole new terminology he invented.
|
| With programs-as-proof it really wouldn't matter. It's either
| "computer says yes" or "compu'er says noooo".
|
| EDIT: Whoop, sibling post mentioned, it's Mochizuki.
| morelisp wrote:
| This doesn't seem a very generous description of Mochizuki's
| work. You don't "need" to study under him for a number of
| years and there's no evidence he's being obscurantist. The
| proof is long and has a lot of novel techniques he's
| invented, and he works primarily in Japanese. You can
| reasonably side with e.g. Scholze's interpretation without
| thinking Mochizuki is disingenuous or some kind of scammer.
|
| He's considerably less esoteric than e.g. Grothendieck was
| even during his more "public" years.
| Quekid5 wrote:
| I have nothing invested in whether or not any given
| mathematician is right or wrong. I just picked a random
| example of a controversial proof -- the point was more that
| proof-as-computation _could_ settle and any all disputes.
|
| It might not lend more understanding to people not invested
| in "field X" (or even people who _are_ invested in field
| X!), but it would be _proof_.
|
| Proof in the current world of math is quite intangible.
| k__ wrote:
| Aren't programs generally more complex than math proofs?
|
| Like, the more accidental edge cases people produce, the less
| they understand the program.
| nitwit005 wrote:
| The math department at my university tried to read through the
| proof of Fermat's last theorem, as a for-fun activity. They
| eventually gave up because they realized it would take too much
| time.
| hgibbs wrote:
| Having a paper published can take a very very long time, a year
| is quite short and 6 months is basically the minimum wait. My
| last paper has taken 2 years to be accepted from the first time
| I submitted it, despite it being largely the same as the
| initial submission, accepted as part of my thesis over a year
| ago, and having several citations. It is very frustrating, and
| it also means that easier (and less original) work is easier to
| publish.
| the_watcher wrote:
| The social science and medical replication crisis seems like it
| would be far more impactful than a mathematics crisis, right?
| Politicians, policy-makers, doctors, etc. all make decisions
| based on potentially flawed or outright incorrect studies in a
| way that I don't think is true for the equivalents in math,
| simply because there aren't decisions and policies up for debate
| related to much of them (if I am wrong about this, please correct
| me).
| smartscience wrote:
| If a flawed mathematical paper were used as the basis for what
| then became a flawed cryptography algorithm, I can see that
| having impact if the bad guys noticed the flaw first. But yes,
| I expect examples like that would be comparatively rare.
| not2b wrote:
| In cryptography the math is almost always the strongest part,
| and it is the side-channel attacks and implementation
| mistakes that let the bad guys in. When it is the math, the
| flaw is often that the algorithm has all the desirable
| properties proved in a number of papers, but has some
| exploitable structure that analysts can turn into an attack.
| raphlinus wrote:
| This is a thoughtful and thought-provoking blog post. I think
| it's worth asking similar questions of computer science. I think
| you'll find some math-like patterns -- there's basically no
| chance Quicksort or any other fundamental algorithm paper will
| fail to replicate -- and some patterns which will fail to
| replicate, like in software engineering.
|
| Some of the early results on pseudorandom generators and hash
| functions aren't holding up well, but I think that's just
| progress. We understand the problem a whole lot better than we
| did back then.
|
| Perhaps more interesting is the literature on memory models. The
| original publications of the Java and C11 memory models had lots
| of flaws, which took many years to fix (and that process might
| not be totally done). I worry that there are a bunch of published
| results that are similarly flawed but just haven't gotten as much
| scrutiny.
| Analemma_ wrote:
| The parts of CS that are the most math-like (which include
| fundamental algorithms) don't have a replication crisis, but
| the ones that are the most social-science like probably do, or
| would. I would bet large sums of money that a lot of the
| literature on stuff like "does OOP lead to better code", "does
| IDE syntax highlighting lead to fewer bugs" etc. would fail to
| replicate if anyone bothered trying.
|
| The thing is, the general sense I get is that people in CS
| already have so little confidence in these results that it's
| not even considered worth the time to try and refute them.
| Which doesn't exactly speak well of the field!
| Beldin wrote:
| Some measurements are interesting and valuable without being
| replicable. For example, the number of online devices or the
| number of websites using wordpress. Take the same measurement
| at a later point in time and the results are different. Yet I
| wouldn't call those fields maths-like.
| Karrot_Kream wrote:
| Research into this stuff is very young and so I think it's
| fair to be skeptical of the results. I'm hoping we'll
| eventually come up with more rigorous, reproducible results.
| sterlind wrote:
| I worry about ML papers in particular. models are closely
| guarded, often impractical to train independently due to
| ownership of the training/test set, or computing power or
| details left out of the paper. there's no way to
| mathematically prove any of it works, either. it's like
| social science done on organisms we've designed.
| ahelwer wrote:
| In competitive programming you could basically assume the
| pseudocode in a paper is not literally correct and requires
| some tweaking to work, despite a "proof" of its correctness.
| Particularly with string algorithms.
| sterlind wrote:
| long time no see!
|
| there's a couple levels there:
|
| rote translating pseudocode into your target language isn't
| likely to pan out well.
|
| so instead you run the pseudocode in your mind, develop an
| intuition on how it works, and that's the "replication" bit
| this post talks about with reviewing math papers.
|
| but both the pseudocode and your code will likely have edge
| cases you didn't handle. this isn't a problem for math -
| that's the category of common trivial/easily fixable proof
| errors that don't really affect the paper. but they're a
| problem for machines that run them literally.
|
| maybe a good compromise strategy for formal verification is
| to declare the insight of the algorithm - recurrence relation
| or whatever - as an axiom, and then use the prover to whack
| the tricky edge cases.
| AussieWog93 wrote:
| From my experience in ML, I'd suspect that the "crisis" isn't
| that the research is false so much as it's useless (algorithm x
| with parameter set w happens to work well on one particular
| dataset, conclusion: I have revolutionised the field).
| yodsanklai wrote:
| This isn't unique to ML. A lot of research is about adding an
| epsilon to an existing paper, which probably doesn't
| interesting anyone except a small community working in their
| very own niche topic.
|
| But does it mean there's a crisis? maybe that's just a way to
| foster an environment that will let great ideas emerge.
| fshbbdssbbgdd wrote:
| I'd rather be in the world where we have too many papers
| tweaking the details of power posing and exactly measuring
| how much each contributes to the effect. At least we'd know
| the effect is real.
| yodsanklai wrote:
| Yes, I'm convinced tons of published results are flawed! I
| heard top researchers tell their students "don't spend too much
| time on the proofs, nobody reads them"). And much CS scientific
| papers don't get a lot of attention. But it's not necessarily
| bad, other researchers builds on top of this work and results
| consolidate over time.
| xenonite wrote:
| Isn't this a misunderstanding? I suspect they rather meant to
| avoid spending too much time on language in these parts.
| yodsanklai wrote:
| No, it's not. In that specific case, the supervisor thought
| the value of the paper didn't lie in the proofs, plus it
| was a rank B conference. He rather has his student working
| on a different paper than spending 1 week on the proof.
| skybrian wrote:
| There was that time when it was discovered that nearly all
| published binary searches and mergesorts were wrong. [1]
|
| And yet, the concepts of binary search and merge sort are fine.
|
| I think that's quite similar to the situation in math papers?
| Because math isn't executable, a math paper being
| "significantly" wrong would be like discovering that a program
| uses a fatally flawed algorithm and is trying to do the
| impossible. It can't be fixed.
|
| Programs that can't be fixed seem rare?
|
| [1] https://ai.googleblog.com/2006/06/extra-extra-read-all-
| about...
| thethirdone wrote:
| Rather than "wrong", I would describe those implementations
| as "not fully general". They work perfectly when `n < some
| large number` as opposed to `n < some large number * 2`. The
| latter is the best you can do with the function signature,
| but that is somewhat arbitrary. You could easily choose a 64
| bit index and exceed all practical needs.
| scoofy wrote:
| The paper here seems to make absolutely zero distinction between
| deductive and inductive reasoning, which should be the entire
| point.
|
| Math is an arbitrary framework, built from arbitrary axioms. It
| is deductive. Thus, all proofs are simply deduction. The
| knowledge here is _positive_ knowledge, we can show things are
| true, false, or undecidable. There may be errors, but those are
| errors in execution.
|
| Psychology is _not built on a framework_. It is inductive, we are
| trying to find axioms that map to the data we collect. Thus, all
| papers are trying to add /build it's arbitrary framework. The
| only knowledge here is _negative_ knowledge, falsification, we
| know only what is a failed hypothesis. There _will_ be errors,
| both in execution, and there will also be statistical errors in
| experimental results.
|
| The entire point of the replication crisis is that we don't
| publish or pay attention to results that are boring, so the
| framework we build is built on skewed data. We don't reject
| previously popular papers that are now unfalsifiable (the idea
| that the _now unfalsifiable_ Ingram experiment is still taught in
| every university psychology dept should be outrageous). The
| boring results need to be weight statistically against the
| interesting results, but aren 't, etc. Nobody out there is
| arguing whether or not the axiom of choice is a _true_ axiom? It
| sort of doesn 't matter, it can't matter, because it's arbitrary
| by definition.
|
| You can't have a replication crisis inside of a deductive
| framework _without changing the framework_. This doesn 't happen
| too often, but we did see this during the shift from neutonian to
| einsteinian physics. The study of philosophy of science is fairly
| obscure, but is the center of this discussion.
| jcranberry wrote:
| I think you're wrong on some of these points, and the author
| was trying to make these points but was not explicit.
|
| I remember reading an article in the intelligencer by a
| mathematician who essentially said there are certain
| conjectures where if they were to be proved false rather than
| be thrown into a sea of uncertainty, mathematicians i would
| quickly move to investigate a readjustment of basic axioms
| rather than accept that those conjectures are incorrect.
|
| Then there are fields of mathematics around selecting different
| axioms. Investigating the ramifications of whether you take the
| undecidable "continuum hypothesis" as true or false. And then
| theres model theory and such. Presumably they study models of
| interest and not arbitrary ones.
|
| You're mostly correct that the methodology is mostly deductive
| but the point is that what we choose to use isn't arbitrary
| because there are things in math which are more important than
| axioms as they are things believed to be "real".
| stavros wrote:
| Exactly, I can't understand what the author was thinking. If a
| paper shows that 2+2=4, what would the replication problem be?
| You write the paper out again and 2+2 turns out to equal 5?
|
| "The results can't be replicated" is different from "the logic
| here is wrong". So different that this article starts from an
| entirely invalid premise.
| [deleted]
| gus_massa wrote:
| > _When I've served as a peer reviewer I've read the papers
| closely and checked all the steps of the proofs, and that means
| that I have replicated the results._
|
| The side effect is that math papers have an insane long time to
| publication. Perhaps 6 months, or 1 year or more if you are
| unlucky.
|
| In physics, the publication time is like 3 month. Something like
| 1 month for the first review and then two months for making small
| changes suggested by the referee and discussing with the editor.
|
| As a side^2 effect, some citation index of the journals count
| only the citations during the first year. But the papers that
| have the citation are sleeping over the reviewer desk during that
| year, so the number is lower than the real number.
| lacker wrote:
| The root of the replication crisis in social sciences is not just
| that many papers fail to replicate, but that there is no way to
| clearly resolve a result that fails to replicate. A paper claims
| that pre-K increases test scores a decade later, another paper
| claims it doesn't, and there's no clear resolution. The
| disagreement just festers, with both sides citing research that
| supports their opinion. The argument often "spills out" into the
| public sphere.
|
| In mathematics and computer science, there are many errors in
| published papers. However, once you point out an error, it's
| usually pretty straightforward to resolve whether it's really an
| error or not. Often there is a small error which can be fixed in
| a small way. Exceptions like the abc conjecture are rare.
| amelius wrote:
| Because math is not science?
|
| https://philosophy.stackexchange.com/questions/14818/is-math...
| curt15 wrote:
| It did:
|
| https://en.wikipedia.org/wiki/Italian_school_of_algebraic_ge...
| otabdeveloper4 wrote:
| Because math is not a science.
| iqanq wrote:
| More like because maths are an actual science.
| dilawar wrote:
| A bit cynical take is that it is harder to mislead in theory?
| Your mistakes will get caught and you risk losing your reputation
| so it less likely that someone will publish a proof without
| double triple checking. Theory subject has faster verificarion
| loops.
|
| Verification and review is harder in experimental works and it
| might take few decades before someone finds an error, or even
| collect resources to verify the claim. So maybe it's harder to
| fight 'publish or perish' deamons in experimental sciences?
| keithalewis wrote:
| "A mathematician's fame is proportional to the number of
| incorrect proofs they have published."
| deltaonefour wrote:
| This gets at the heart of the difference between science and
| math. What is actually science? and what is actually math? You
| may think you know, but most people don't.
|
| Did you know? That in Science, and therefore reality as we know
| it... NOTHING can be proven to be true. Proof is the domain of
| logic and mathematics, NOT science.
|
| There is a huge difference between science and mathematics.
|
| Math is an imaginary game involving logic. You first recursively
| assume logic is true then you assume some additional things are
| true called "axioms." You assume the entire universe only
| encapsulates Logic and the axioms as part of your theory and that
| is everything that exists in your theoretical universe. You
| derive or prove statements that you know must consequently be
| true in your made up universe based off of logic and the
| "axioms." These statements are called "theorems." That is math...
|
| For example the "pythagorean theorem" is a statement made about
| geometry assuming that the axioms of geometry are true and logic
| is true.
|
| So of course if it's a logical game, there's no real replication
| crisis. All theorems in mathematics are proven true based off of
| logic. You can't replicate conflicting results if a result is
| PROVEN to be true via pure logic.
|
| Science is different. Nothing can ever be proven to be true.
| There are basically two axioms in science. We assume logic is
| true, just like in math. And we assume that the mathematical
| theory of probability is a true model for random events happening
| in the real world. Then we assume everything else about the
| universe is completely unknown. This is key.
|
| Because the domain of the universe is unknown... at any point in
| time we can observe something that contradicts our initial
| hypothesis. ANY point in time. That means even the theory of
| gravity is forever open to disproof. This is the reason why
| NOTHING can be proven. To quote Einstein:
|
| "No amount of experimentation can ever prove me right; a single
| experiment can prove me wrong."
|
| So sure mathematicians can make mistakes... and I know the author
| is talking about higher level details.... but at its most
| fundamental level, assuming all other attributes are ideal... it
| is fundamentally impossible for math to have a replication crisis
| while science, on the other hand, is fundamentally forever open
| to these crisis so long as the universe is unknown.
|
| The most interesting thing to me in all of this however is that
| within science, probability is a random axiom. We have no idea
| why probability works... it just does. It's the driver behind
| another fundamental concept: Entropy and the arrow of time. For
| some strange reason Logic in our universe exists side by side
| with probability as a fundamental property.
| mcphage wrote:
| > All theorems in mathematics are proven true based off of
| logic. You can't replicate conflicting results if a result is
| PROVEN to be true via pure logic.
|
| You can publish a paper in mathematics that claims to prove
| something, but is mistaken. A paper claiming that a theorem is
| proven, is not the same thing as the theorem _being_ proven.
| However, that 's not often the case--why that is, is an
| interesting & meaningful question.
| Barrin92 wrote:
| why not turn the premise from the article around. Instead of
| suggesting that math might also have a replication crisis, why
| not question whether the whole replication crisis thing was
| overblown and effectively more of an ideological attack on a few
| disciplines.
|
| Errors in science, disagreement and lack of reproducibility I
| think are common and prevalent but it doesn't necessarily imply
| that a discipline as a whole doesn't make progress. The obsession
| with statistical accuracy and 'science as bookkeeping' mentality
| seems fairly new to begin with, and science did just fine before
| we even had the means to verify every single thing ever
| published.
|
| It kind if ignores the dynamic nature of science. Most of what is
| published probably has close to zero impact regardless of whether
| its right or wrong, but paradigm changing research generally
| asserts itself. Science is evolutionary in that sense, it's full
| of mistakes but stumbles towards correct solutions at uneven
| tempo. In a sense you can just look at it like VC investment.
| Nine times out of ten individual things don't work, but the
| sector overall works, the market economy is full of grifters and
| failed businesses, but it doesn't matter that much.
|
| So, maybe half of math is bullshit but so is everything else but
| in math people just say "whatever" until they find something
| good, whereas in psychology people use it as an opportunity to
| hack away at it.
| AussieWog93 wrote:
| The difference between the two examples you have it that
| venture capital and the market economy openly embrace their
| flaws, whereas science and academia refuses to acknowledge (or
| manage) the "humanness" of the system and projects a hyper-
| enlightened ideal both internally and to the outside world.
| burnished wrote:
| This isn't turning the problem around in the same sense. In
| order to effectively turn a problem around you need to use its
| complement, and then it should be a binary proposition. Your
| example where you suppose an idealogical attack fails here
| because it is not the only other explanation, you haven't
| turned the problem around in the same way that turning "what is
| the probability this action had an effect" around can become
| "what is the probability that this action had no effect".
| quanticle wrote:
| _It kind if ignores the dynamic nature of science. Most of what
| is published probably has close to zero impact regardless of
| whether its right or wrong, but paradigm changing research
| generally asserts itself._
|
| The problem is that psychology, unlike physics, doesn't really
| have a paradigm. There's the old saying, "Extraordinary claims
| require extraordinary evidence." But for that heuristic to be
| effective, you need a standard for what counts as
| extraordinary. Extraordinary compared to what? In phsyics,
| there are two well-established paradigms (relativity and
| quantum mechanics), which establish what counts as ordinary,
| and what counts as extraordinary. So, for example, if you're
| making a claim that the distribution of dark matter in the
| cosmos more clumpy than predicted by existing models, of that
| the energy level of a particular field is 12 MeV rather than
| 10, those are ordinary claims, which can be accomodated by
| tweaks to the existing paradigm. But if you're saying that the
| speed of light has varied over the history of the universe, or
| that all subatomic particles are actually tiny vibrating
| string-like structures, well, that's going to require a lot
| more evidence.
|
| In psychology, it's much more difficult to have that kind of
| intuition. Take the concept of priming, for example. Is
| claiming that people walk more slowly when they're encouraged
| to think of things that make them feel old extraordinary? It
| makes a certain sort of intuitive sense, but, on the other
| hand, there's absolutely no causal mechanism suggested. So when
| a number of priming studies fail spectacularly under
| replication [1], I don't know what to think. I don't have a
| good sense for how much of psychology is overturned by the
| replication failure, in the same sense that I'd have for
| physics if it turned out that e.g. the speed of light is a
| variable rather than a constant.
|
| [1]: https://mindhacks.com/2017/02/16/how-replicable-are-the-
| soci...
| pdonis wrote:
| _> In phsyics, there are two well-established paradigms
| (relativity and quantum mechanics), which establish what
| counts as ordinary, and what counts as extraordinary._
|
| Actually, I would say that these well-established paradigms
| establish what counts as _extraordinary_. In other words,
| relativity and QM are _examples_ of extraordinary claims that
| we believe _because_ we have extraordinary evidence for them.
| Both of these theories say all kinds of extraordinary things,
| and most people who first encounter the theories start out
| thinking they can 't possibly be true. We believe them not
| because they are just ordinary, but because we have taken the
| time and effort to accumulate extraordinary evidence for
| them.
|
| In that light, the replication crisis in other areas of
| science is easily explained: they allow extraordinary claims
| to be published _without_ the extraordinary evidence that
| those claims would require. So of course many of those claims
| turn out to be wrong.
| mjfl wrote:
| because math doesn't do experiments.
| medstrom wrote:
| Exactly. A math study is not a "study" in the sense of "hey i
| saw a funny pattern in some data maybe it's a sign of my pet
| theory" - it's literally already proven when published. There's
| nothing more to do.
| Jtsummers wrote:
| Or as noted in the article:
|
| > But one of the distinctive things about math is that our
| papers aren't just records of experiments we did elsewhere. In
| experimental sciences, the experiment is the "real work" and
| the paper is just a description of it. But in math, the paper,
| itself, is the "real work".
|
| And
|
| > And that means that you can replicate a math paper by reading
| it.
| horsawlarway wrote:
| I think that means that the word "experiment" isn't the right
| term for what most mathematicians do.
|
| I'd say most times it's "modeling", not "experimenting"
| [deleted]
| nimih wrote:
| https://www.experimentalmath.info/ would beg to differ, I
| think.
| horsawlarway wrote:
| This is my take.
|
| Science attempts to describe reality. Math attempts to create
| rules/axioms.
|
| They're not the same pursuit, although they can often be useful
| together.
| NovemberWhiskey wrote:
| This seems the obvious answer. The replication crisis isn't
| about published material being _wrong_ , it's about the
| inability to reproduce the results of experiments or studies in
| a repeatable fashion.
|
| It's not like you make a hypothesis in math and then need to go
| away and interview a sample of 1,000 circles and report back
| that, controlling for ellipses that may be misreporting as
| circles, the ratio of the circumference to the diameter is 3.2
| +/- 0.1 (p<0.05).
| bunje wrote:
| "Mathematics is the part of physics where experiments are
| cheap." - V.I. Arnold
| schuyler2d wrote:
| You could also argue that Math _had_ its replication crisis in
| the 17th-19th centuries. E.g. infinite series "proofs" that were
| eventually shown to be flawed methodologies.
|
| This and other crises led to grounding modern mathematics with
| set theory, Zermelo-Fraenkel axioms, etc and understanding what's
| possible (e.g. Godel's theorem).
|
| Psychology and other social sciences are barely a century old.
| fshbbdssbbgdd wrote:
| A more recent example is the Italian school of algebraic
| geometry, where it was discovered in the mid 20th century that
| many claimed proofs were faulty (and some "proven" results were
| incorrect).
| zmgsabst wrote:
| Mathematics _is_ having a replication crisis and people pay so
| little attention they don't know.
|
| That replication crisis has led to efforts in formal
| verification such as HoTT, Lean, etc.
|
| https://homotopytypetheory.org/
|
| https://xenaproject.wordpress.com/2021/06/05/half-a-year-of-...
| Victerius wrote:
| How is math experiencing a replication crisis?
|
| > This site serves to collect and disseminate research,
| resources, and tools for the investigation of homotopy type
| theory, and hosts a blog for those involved in its study.
|
| > Exactly half a year ago I wrote the Liquid Tensor
| Experiment blog post, challenging the formalization of a
| difficult foundational theorem from my Analytic Geometry
| lecture notes on joint work with Dustin Clausen.
|
| ???
| dwohnitmok wrote:
| No by and large mathematics is not having a replication
| crisis. As the blog post states:
|
| > Question: Was the proof in [Analytic] found to be correct?
|
| > Answer: Yes, up to some usual slight imprecisions.
|
| This has been the case for almost all math formalization
| efforts. Even when (very rarely) proofs were revealed to be
| incorrect, the result was salvageable.
| thrown_22 wrote:
| That's because most math proofs are treading on well
| understood grounds and only extending them slightly. E.g.
| it would be like a psychologist asking how an results of a
| well proven result would be different if all participants
| wore red shoes.
|
| When you enter truly new grounds mathematicians don't even
| agree if the distinctions being made have a meaning, let
| alone if they are true.
| guerrilla wrote:
| Here's [1] the lecture where Vladimir Voevodsky talked about
| the problem and his experience with it but like the blog
| says, he didn't and they don't consider it a crisis. Even
| HoTT (and other TT) people present it as how things could be
| much better, not about how things are terrible.
|
| 1. https://youtu.be/E9RiR9AcXeE
| umvi wrote:
| My completely subjective opinion is that at the highest levels of
| math, there are only a handful of people that are even capable of
| peer reviewing, and their time is in high demand.
|
| Wiles's proof of Fermat's Last Theorem is like 120 pages long and
| he first delivered it disguised as a class to a bunch of grad
| students who barely understood any of it and hence gave no
| feedback. Because this is Fermat's Last Theorem which is famous,
| eventually people in the math community that understood Wiles's
| work reviewed it and found an error. Had it been a 120 page proof
| of some not famous problem like random chessboard thought
| experiments, it probably could go years without anyone seriously
| looking at it.
| verisimi wrote:
| "We get away with it becuase we can be right for the wrong
| reasons--we mostly only try to prove things that are basically
| true."
|
| Apart from the replication crisis, the other crisis that is not
| really talked about is funding. Academic funding basically comes
| from one of 3 (connected) sources - government, corporations and
| the military. Somehow or other - these 3 sources have pretty much
| the same or non-conflicting aims. These aims relate to power and
| control. This is actually the largest crisis, IMO.
|
| Given that information, we can re-assess the quoted statement the
| author makes.
|
| Perhaps its not that they are proving things that are "basically
| true". Its that right or wrong do not matter. What does matter is
| that the answers provided meet the agenda of those funding the
| study. The answer is not _that_ important as long as it is
| supportive of whatever agenda is in play. I believe this is the
| case for the replication crisis in science also.
|
| A replication "crisis" is only a crisis if you are attempting to
| achieve truth and greater understanding. But truth and
| understanding are only ostensible reasons, not the actual ones.
| What these studies are actually doing is creating a parallel
| construction - the aim is actually for studies to appear
| 'truthey', without actually being so. What studies should actual
| do is increase the funder's power, wealth extraction abilities,
| etc.
|
| If you doubt this and think that truth matters, consider this.
| Surely we should have cracked the best diet for people by now?
| But there is no common understanding of what is good or bad to
| eat - if anything there is more confusion. The reason of course,
| is that there's no money in recommending whole foods or whatever.
| However, there is money in drugs to make people 'better'. And
| money in making diet so confusing that people eat themselves into
| trouble.
|
| Anyway, if you are in the business of governing or monetising the
| masses, truth and understanding is the last thing you want. Far
| better to have a story that gives you control, or extracts money.
| Such is life under fascist governance (where fascist =
| corporation + governance working together).
| Iwan-Zotow wrote:
| Huh?!?
|
| Please read about abc conjecture and whole saga wrt proof,
| Shinichi Mochizuki, Peter Scholze, etc, etc, etc
| sgillen wrote:
| Did you read the article? This saga is explicitly mentioned and
| does not detract from the authors point IMO.
| ocschwar wrote:
| There is one corner of math that does have a replication crisis.
| Just as we compare programming languages by how "ergonomic" they
| are to learn and use, mathematicians do come up with novel
| notation systems to try to improve the ergonomic state of their
| field, and since "ergonomics" is another way to say "esthetics",
| and is proved or disproved by user testing, that is where
| replication gets hard.
|
| The inventor of category theory's wiring diagrams, for example,
| has claimed that he could get middle schoolers to understand
| them. I suspect that success has not been replicated.
| Simplicitas wrote:
| Steven Pinker takes a decent crack at "Statistical Significance"
| in his new book Rationality
| (https://en.wikipedia.org/wiki/Rationality_(book)), which
| underpins a lot of this and is mentioned in this piece. And I'm
| still grappling with this part of the book. lol
___________________________________________________________________
(page generated 2022-02-02 23:00 UTC)