An Uneven Ledger: How Scientific Disciplines Choose Whether to Hold Themselves Accountable
In the years following the emergence of what many researchers now call the replication crisis, a curious pattern solidified across American scientific institutions. Psychology, medicine, and certain corners of economics underwent something resembling a public reckoning — preregistration mandates, registered reports, large-scale replication consortia, and heightened editorial scrutiny became fixtures of the professional landscape. Meanwhile, other disciplines — theoretical physics, qualitative sociology, clinical nutrition, and significant portions of macroeconomics — continued operating with comparatively little external pressure to verify whether their published conclusions held up under independent examination. The question of why this divergence exists, and what it reveals about science as a social institution, deserves considerably more attention than it has received.
The Disciplines That Rebuilt Their Floors
The field of psychology serves as the most documented case study in disciplinary self-correction. The Reproducibility Project, coordinated through the Center for Open Science and published in Science in 2015, found that fewer than half of 100 prominent psychology studies replicated at a statistically meaningful level. The fallout was substantial: grant agencies began rewarding open data practices, journals introduced replication-specific publication tracks, and professional organizations revised ethical guidelines around data sharing. The damage to the field's reputation was real, but so was the corrective response.
Biomedical science followed a parallel, if slower, trajectory. Concerns raised by researchers at Bayer and Amgen — who found that the majority of landmark cancer biology findings could not be reproduced in industrial settings — prompted the National Institutes of Health to reform its reporting standards for preclinical research. Rigor and reproducibility language became embedded in grant review criteria. The field's proximity to clinical application, where unreplicable findings can translate directly into failed drug trials and patient harm, created a practical urgency that accelerated reform.
What these fields share is a combination of high public visibility, measurable downstream consequences, and sufficient funding infrastructure to support the overhead costs of verification. When failures become visible to patients, regulators, and congressional appropriations committees, the institutional calculus around accountability shifts.
The Disciplines That Declined the Invitation
Contrast this with domains where replication norms remain largely aspirational, selectively applied, or actively resisted. Clinical nutrition research, for instance, has produced decades of contradictory findings on dietary fat, sodium, carbohydrates, and cardiovascular outcomes — yet the field has not undergone anything resembling the structural reform that psychology experienced. Partly, this reflects the genuine difficulty of conducting controlled dietary studies in free-living populations. But it also reflects a disciplinary culture in which expert consensus, rather than independent replication, functions as the primary currency of credibility.
Qualitative social science presents a different but related challenge. Methodological traditions that privilege interpretive depth over quantitative precision are not inherently incompatible with rigor, but they do complicate the application of conventional replication frameworks. When a finding is understood as context-dependent and theoretically situated, the demand for identical reproduction across independent samples can appear epistemologically naive rather than scientifically necessary. Critics argue that this framing, however philosophically coherent, has also functioned as a convenient shield against external accountability.
Macroeconomics occupies perhaps the most politically complicated position. High-profile replication failures — most notably the dispute over the Reinhart-Rogoff debt threshold findings, which had influenced austerity policy in multiple countries — demonstrated that the stakes of unreplicable economic research can be substantial and broadly distributed. Yet the field's methodological pluralism, its reliance on observational data that cannot be experimentally reproduced, and its entanglement with ideological commitments have made systematic replication efforts difficult to institutionalize.
Incentive Architecture as Destiny
The divergence between high- and low-accountability disciplines cannot be explained by methodological differences alone. Incentive structures — who funds replication, who publishes it, and who receives career credit for conducting it — determine whether verification becomes a professional norm or a marginal activity.
In fields with substantial federal funding streams and regulatory oversight, the infrastructure for accountability exists because external parties have demanded it. The Food and Drug Administration's requirements for clinical trial replication prior to drug approval represent the most formalized version of this dynamic. When regulators can withhold market access, the incentive to produce reproducible findings is direct and enforceable.
In fields that operate without comparable external scrutiny — or whose outputs are difficult to translate into measurable policy outcomes — the replication imperative is largely self-generated, which means it depends on the willingness of professional communities to impose costs on themselves. That willingness, in turn, depends on whether a field's leadership perceives replication failure as a threat to institutional legitimacy or as a manageable internal matter that can be handled quietly through editorial discretion.
The funding dimension compounds this dynamic. Replication studies are expensive, time-consuming, and — in a grant ecosystem that rewards novelty — professionally unrewarding for the researchers who conduct them. Fields with larger average grant sizes and more robust funding ecosystems can absorb this overhead more readily. Fields operating on thinner margins are structurally disadvantaged in building replication capacity, regardless of their normative commitments to reproducibility.
The Legitimacy Gap
What emerges from this comparative analysis is not simply a technical problem of verification infrastructure, but a legitimacy gap that runs through the scientific enterprise. Research produced in high-accountability disciplines is, by definition, subject to a form of external stress-testing that research produced in low-accountability disciplines is not. Over time, this asymmetry creates a stratified scientific landscape in which some bodies of knowledge have been genuinely pressure-tested and others have not — but where the distinction is rarely made explicit in policy discussions, media coverage, or public communication.
This matters for reasons that extend beyond academic integrity. When policymakers draw on scientific consensus to justify interventions in public health, education, economic management, or environmental regulation, they are not always drawing from a uniformly verified evidentiary base. The confidence attached to a finding that has survived independent replication is epistemically different from the confidence attached to a finding that has never been subjected to it — but that difference is frequently invisible in the translation from research to policy.
Toward Structural Honesty
Addressing this asymmetry requires more than exhorting individual disciplines to adopt better practices. It requires that funding agencies, journals, and professional associations make the accountability standards of each field visible and legible to external audiences. Disclosure of replication rates, data availability, and methodological transparency should be treated not as supplementary information but as core components of scientific communication.
Some disciplines have begun moving in this direction voluntarily, and the movement toward open science infrastructure — preregistration repositories, data archiving mandates, registered replication reports — offers genuine tools for broadening accountability norms. But voluntary adoption has clear limits. The fields most in need of external accountability are, almost by definition, those least likely to self-impose it.
The replication crisis did not strike all of science equally, and the reforms it prompted have not distributed themselves evenly. Recognizing that unevenness — and refusing to treat disciplinary self-governance as a sufficient substitute for genuine verification — is a precondition for a scientific enterprise that can credibly claim to know what it says it knows.
