summary of correction in research

Loading metrics

Open Access

Perspective

The Perspective section provides experts with a forum to comment on topical or controversial issues of broad interest.

See all article types »

Correction of scientific literature: Too little, too late!

* E-mail: [email protected] (LB); [email protected] (EB); [email protected] (JH); [email protected] (GM-K)

Affiliation Faculty of Information and Technology, Monash University, Clayton, Victoria, Australia

Affiliation Harbers Bik LLC, San Francisco, California, United State of America

Affiliation Cipher Skin, Denver, Colorado, United State of America

Affiliation School of Health and Society, University of Wollongong, Wollongong, New South Wales, Australia

• Lonni Besançon, 
• Elisabeth Bik, 
• James Heathers, 
• Gideon Meyerowitz-Katz

Published: March 3, 2022

• https://doi.org/10.1371/journal.pbio.3001572
• Reader Comments

The Coronavirus Disease 2019 (COVID-19) pandemic has highlighted the limitations of the current scientific publication system, in which serious post-publication concerns are often addressed too slowly to be effective. In this Perspective, we offer suggestions to improve academia’s willingness and ability to correct errors in an appropriate time frame.

Citation: Besançon L, Bik E, Heathers J, Meyerowitz-Katz G (2022) Correction of scientific literature: Too little, too late! PLoS Biol 20(3): e3001572. https://doi.org/10.1371/journal.pbio.3001572

Copyright: © 2022 Besançon et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The author(s) received no specific funding for this work.

Competing interests: EB has received consulting fees from publishers and research institutions, and gets donations through Patreon.com . All authors have been involved with error-checking research and have been the target of reprisals in a number of ways.

Traditionally, scientific progress has relied on trust and the relatively slow cycle of peer review, publication, and citation of research data. The current Coronavirus Disease 2019 (COVID-19) pandemic not only accelerated the speed of research but also brought to light some severe shortcomings of the scientific publication process, such as failures to quickly address errors or to catch and prevent scientific misconduct.

Within months of the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) virus being identified, disease progression to COVID-19, viral transmission routes, and treatment options were being carefully studied, and effective vaccines had begun development. This was one of the most impressive scientific achievements of the modern era [ 1 ]. While the goal of disease mitigation has been rightfully praised for its comparative speed, organization, and safety, the mechanics of the publication system that sustained it have been far from ideal. We believe it is true to say the response to COVID-19 has succeeded in spite of, rather than because of, the present publication system.

During the COVID-19 pandemic, many basic quality control and transparency principles have been violated on a regular basis [ 2 ]. This is perhaps most apparent in the Surgisphere debacle [ 3 ], in which global policy on COVID-19 treatment was changed overnight on the basis of a database that later turned out not to exist. Although the Surgisphere retraction happened quickly, it was far slower than the change in medical practice, which was immediate, and represents a best-case scenario in which a high-profile paper was immediately interrogated and investigated. The stories of hydroxychloroquine and ivermectin, both widely promoted based on poor quality or even fraudulent studies [ 4 ], are further concerning accounts of how the scientific publishing process has failed to exercise basic quality control.

In the usual course of scientific investigation, these stories would be something of a footnote—fraud, malfeasance, and mendacity within median research studies are interesting to meta-scientists, methodologists, and theoreticians, but rarely have an impact outside of the scientific community. By contrast, high-profile studies with global implications receive a great deal of collective attention and scrutiny if they are untrustworthy, which normally limits their influence. However, in the setting of the present pandemic, poor research has sometimes been instantly applied to health policy after being published or simply publicized, and used in treatment regimens shortly after. There is an immediacy to the impact of scientific papers that was rarely present prior to the pandemic.

Traditional responses to such research issues (such as post-review letters, notes of concern, and even retractions) are woefully inadequate to address these problems. Nowadays, preprints and peer-reviewed research papers are rapidly shared on online platforms among millions of readers within days of being published. A paper can impact worldwide health and well-being in a few weeks online; that it may be retracted at some point months in the future does not undo any harm caused in the meantime. Even if a paper is removed entirely from the publication record, it will never be removed from the digital space and is still likely to be cited by researchers and laypeople alike as evidence. Often, its removal contributes to its mystique. For example, a retracted and proven false study on vitamin D from 2020 [ 5 ] that was pulled from the preprint server it was hosted on was still being cited uncritically as recently as November 2021 .

All of these issues are compounded by the glacial pace at which scientific correction is likely to occur [ 6 ]. Identifying flaws in a paper may only take hours, but even the most basic formal correction can take months of mutual correspondence between scientific journal editors, authors, and critics. Even when authors try to correct their own published manuscripts, they can face strenuous challenges that prompt many to give up. Worse still, while editors and authors might gain financial and career benefits from ignoring errors, scientific critics are explicitly discouraged by the academic community from performing this work.

The authors of this Perspective have all been involved in error detection in this manner. For our voluntary work, we have received both legal and physical threats and been defamed by senior academics and internet trolls. While many scientists personally support error-checking work [ 7 ], academia as a whole seems to view error-checking as a dirty footnote to the achievement of publication. Even when papers are retracted, individuals who have spent endless hours explicating the problems therein are left with no formal career benefits whatsoever—if anything, they face substantial retaliation for correcting errors.

This system is unsustainable, unfair, and dangerous, and the pandemic has acted to magnify its unsuitability and numerous limitations. Rather than being a disappointing footnote, error-checking should be supported and funded by government agencies and research institutions. Public, open, and moderated review on PubPeer [ 8 ] and similar websites that expose serious concerns should be rewarded with praise rather than scorn, personal attacks, or threats (either legal or on the reviewers’ lives ). Importantly, retraction should not always be seen as a failure. While some papers are retracted for reasons of serious research misconduct, other papers are retracted because of unintentional errors [ 9 ]. Scientists must acknowledge that any process comes with an error rate, and correcting mistakes should not limit careers, but instead enhance them [ 10 ]. Consequently, we propose some solutions that could improve the current error-checking and correction system in scientific publishing ( Box 1 ).

Box 1. Approaches to destigmatize and speed up the scientific correction process

• Editors should issue an Expression of Concern within days after serious and verifiable concerns have been raised either privately or on a public forum. If the concerns are publicly raised, the Expression of Concern should link to them, otherwise, it should summarize the key points raised privately.
• Committee on Publication Ethics (COPE) guidelines for editors and journals should provide a timeline for responding to concerns about published papers, with, for instance, a maximum of 90 days to publicly highlight concerns, contact the authors, get a response from the authors, and publish it.
• Public, open, and post-publication peer review should be considered and rewarded by hiring and promotion committees as well as by funding bodies. Applications for funding or positions should consider such correction efforts made by scientists just as much as they consider the publication of new research results.
• To further establish an error-checking culture, scientists should be trained to recognize mistakes (including their own). Institutes and funding agencies should allocate time for error-checking, and institutions and journals should promote corrections and retractions as much as they promote new research findings.
• Notices of retractions or corrections could be linked to the researchers that initially raised concerns and attributed a DOI that links to their careful rebuttal of the original paper (even if it’s their own).
• Journal webpages should directly link to discussions on PubPeer rather than rely on the use of an external plug-in.
• DOI versioning already allows for concurrent versions of documents to coexist and should be adopted by publishing venues to allow for easy and fast correction of papers when needed, along with meta-data on the changes between versions.
• Critics who raise professional, nondefamatory concerns about a preprint or published paper should have explicit, paid legal protection (e.g., provided by their institutions or professional societies) against threats issued by the critiqued authors or their institutions.

Understandably, there are legal concerns when discussing the retraction and/or correction of scientific papers. We do not wish to minimize these issues, as they may be serious for both publishers and academics alike. However, many such problems stem from an environment in which retraction is seen as a career-ending calamity, and, thus, our proposed improvements ( Box 1 ) may alleviate many of the legal concerns faced by the academic community.

Viewing the retraction and correction of scientific papers as a failure is a self-fulfilling prophecy: As the status of a paper rarely changes post-publication, it is seen as something exceptional or immoderate instead of a normal part of the scientific process. The alternative to addressing this problem is to continue to maintain a scientific commons that is unable to deal with the rapid dissemination and correction of research that is needed in the digital age.

• View Article
• PubMed/NCBI
• Google Scholar
• 8. Barbour B, Stell BM. PubPeer: Scientific assessment without metrics. In: Biagioli M, Lippman A, editors. Gaming the Metrics: Misconduct and Manipulation in Academic Research. The MIT Press; 2020. p. 149–155. https://doi.org/10.7551/mitpress/11087.001.0001

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 15 June 2023

A meta-analysis of correction effects in science-relevant misinformation

• Man-pui Sally Chan   ORCID: orcid.org/0000-0003-2984-0487 1 &
• Dolores Albarracín   ORCID: orcid.org/0000-0002-9878-942X 2  

Nature Human Behaviour volume  7 ,  pages 1514–1525 ( 2023 ) Cite this article

4002 Accesses

14 Citations

614 Altmetric

Metrics details

• Communication
• Human behaviour

Scientifically relevant misinformation, defined as false claims concerning a scientific measurement procedure or scientific evidence, regardless of the author’s intent, is illustrated by the fiction that the coronavirus disease 2019 vaccine contained microchips to track citizens. Updating science-relevant misinformation after a correction can be challenging, and little is known about what theoretical factors can influence the correction. Here this meta-analysis examined 205 effect sizes (that is, k , obtained from 74 reports; N  = 60,861), which showed that attempts to debunk science-relevant misinformation were, on average, not successful ( d  = 0.19, P  = 0.131, 95% confidence interval −0.06 to 0.43). However, corrections were more successful when the initial science-relevant belief concerned negative topics and domains other than health. Corrections fared better when they were detailed, when recipients were likely familiar with both sides of the issue ahead of the study and when the issue was not politically polarized.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

111,21 € per year

only 9,27 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

The psychological drivers of misinformation belief and its resistance to correction

Understanding and combatting misinformation across 16 countries on six continents

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Data availability.

The data that support the findings of this study are openly available in OSF at https://osf.io/vkygw/ .

Code availability

All code for data analyses associated with the current submission is available at https://osf.io/vkygw/ . Any updates will also be published in OSF.

Ahmed, W., Downing, J., Tuters, M. & Knight, P. Four experts investigate how the 5G coronavirus conspiracy theory began. The Conversation https://theconversation.com/four-experts-investigate-how-the-5g-coronavirus-conspiracy-theory-began-139137 (2020).

Heilweil, R. The conspiracy theory about 5G causing coronavirus, explained. Vox (2020); https://www.vox.com/recode/2020/4/24/21231085/coronavirus-5g-conspiracy-theory-covid-facebook-youtube

Pigliucci, M. & Boudry, M. The dangers of pseudoscience. The New York Times (2013); https://opinionator.blogs.nytimes.com/2013/10/10/the-dangers-of-pseudoscience/

Gordin, M. D. The problem with pseudoscience: pseudoscience is not the antithesis of professional science but thrives in science’s shadow. EMBO Rep. 18 , 1482 (2017).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Townson, S. Why people fall for pseudoscience (and how academics can fight back). The Guardian (2016); https://www.theguardian.com/higher-education-network/2016/jan/26/why-people-fall-for-pseudoscience-and-how-academics-can-fight-back

Caulfield, T. Pseudoscience and COVID-19—we’ve had enough already. Nature https://doi.org/10.1038/d41586-020-01266-z (2020).

Article   PubMed   PubMed Central   Google Scholar  

Pennycook, G. & Rand, D. G. Lazy, not biased: susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition 188 , 39–50 (2019).

Article   PubMed   Google Scholar  

Vraga, E. K. & Bode, L. Defining misinformation and understanding its bounded nature: using expertise and evidence for describing misinformation. Polit. Commun. https://doi.org/10.1080/10584609.2020.1716500 (2020).

Lewandowsky, S. et al. The Debunking Handbook 2020. Databrary https://doi.org/10.17910/b7.1182 (2020).

Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592 , 590–595 (2021).

Article   CAS   PubMed   Google Scholar  

Garrett, R. K., Weeks, B. E. & Neo, R. L. Driving a wedge between evidence and beliefs: how online ideological news exposure promotes political misperceptions. J. Comput.-Mediat. Commun. 21 , 331–348 (2016).

Article   Google Scholar  

Lazer, D. M. J. et al. The science of fake news: addressing fake news requires a multidisciplinary effort. Science 359 , 1094–1096 (2018).

Wyer, R. S. & Unverzagt, W. H. Effects of instructions to disregard information on its subsequent recall and use in making judgments. J. Pers. Soc. Psychol. 48 , 533–549 (1985).

Greitemeyer, T. Article retracted, but the message lives on. Psychon. Bull. Rev. 21 , 557–561 (2014).

McDiarmid, A. D. et al. Psychologists update their beliefs about effect sizes after replication studies. Nat. Hum. Behav. https://doi.org/10.1038/s41562-021-01220-7 (2021).

Yousuf, H. et al. A media intervention applying debunking versus non-debunking content to combat vaccine misinformation in elderly in the Netherlands: a digital randomised trial. EClinicalMedicine 35 , 100881 (2021).

Kuru, O. et al. The effects of scientific messages and narratives about vaccination. PLoS ONE 16 , e0248328 (2021).

Anderson, C. A. Inoculation and counterexplanation: debiasing techniques in the perseverance of social theories. Soc. Cogn. 1 , 126–139 (1982).

Jacobson, N. G. What Does Climate Change Look Like to You? The Role of Internal and External Representations in Facilitating Conceptual Change about the Weather and Climate Distinction (Univ. Southern California, 2022).

Pluviano, S., Watt, C. & Sala, S. D. Misinformation lingers in memory: failure of three pro-vaccination strategies. PLoS ONE 12 , 15 (2017).

Maertens, R., Anseel, F. & van der Linden, S. Combatting climate change misinformation: evidence for longevity of inoculation and consensus messaging effects. J. Environ. Psychol. 70 , 101455 (2020).

Chan, M. S., Jones, C. R., Jamieson, K. H. & Albarracin, D. Debunking: a meta-analysis of the psychological efficacy of messages countering misinformation. Psychol. Sci. 28 , 1531–1546 (2017).

Janmohamed, K. et al. Interventions to mitigate vaping misinformation: a meta-analysis. J. Health Commun. 27 , 84–92 (2022).

Walter, N. & Tukachinsky, R. A meta-analytic examination of the continued influence of misinformation in the face of correction: how powerful is it, why does it happen, and how to stop it? Commun. Res. 47 , 155–177 (2020).

Walter, N., Cohen, J., Holbert, R. L. & Morag, Y. Fact-checking: a meta-analysis of what works and for whom. Polit. Commun. 37 , 350–375 (2020).

Walter, N. & Murphy, S. T. How to unring the bell: a meta-analytic approach to correction of misinformation. Commun. Monogr. 85 , 423–441 (2018).

Walter, N., Brooks, J. J., Saucier, C. J. & Suresh, S. Evaluating the impact of attempts to correct health misinformation on social media: a meta-analysis. Health Commun. 36 , 1776–1784 (2021).

Chan, M. S., Jamieson, K. H. & Albarracín, D. Prospective associations of regional social media messages with attitudes and actual vaccination: A big data and survey study of the influenza vaccine in the United States. Vaccine 38 , 6236–6247 (2020).

Lawson, V. Z. & Strange, D. News as (hazardous) entertainment: exaggerated reporting leads to more memory distortion for news stories. Psychol. Pop. Media Cult. 4 , 188–198 (2015).

Nature Microbiology. Exaggerated headline shock. Nat. Microbiol. 4 , 377–377 (2019).

Pinker, S. The media exaggerates negative news. This distortion has consequences. The Guardian (2018); https://www.theguardian.com/commentisfree/2018/feb/17/steven-pinker-media-negative-news

CDC. HPV vaccine safety. U.S. Department of Health & Human Services https://www.cdc.gov/hpv/parents/vaccinesafety.html (2021).

Jaber, N. Parent concerns about HPV vaccine safety increasing. National Cancer Institute https://www.cancer.gov/news-events/cancer-currents-blog/2021/hpv-vaccine-parents-safety-concerns (2021).

Brody, J. E. Why more kids aren’t getting the HPV vaccine. The New York Times https://www.nytimes.com/2021/12/13/well/live/hpv-vaccine-children.html (2021).

Walker, K. K., Owens, H. & Zimet, G. ‘We fear the unknown’: emergence, route and transfer of hesitancy and misinformation among HPV vaccine accepting mothers. Prev. Med. Rep. 20 , 101240 (2020).

Normile, D. Japan reboots HPV vaccination drive after 9-year gap. Science 376 , 14 (2022).

Larson, H. J. Japan’s HPV vaccine crisis: act now to avert cervical cancer cases and deaths. Lancet Public Health 5 , e184–e185 (2020).

Soroka, S., Fournier, P. & Nir, L. Cross-national evidence of a negativity bias in psychophysiological reactions to news. Proc. Natl Acad. Sci. USA 116 , 18888–18892 (2019).

Baumeister, R. F., Bratslavsky, E., Finkenauer, C. & Vohs, K. D. Bad is stronger than good. Rev. Gen. Psychol. 5 , 323–370 (2001).

Kunda, Z. The case for motivated reasoning. Psychol. Bull. 108 , 480–498 (1990).

Kopko, K. C., Bryner, S. M. K., Budziak, J., Devine, C. J. & Nawara, S. P. In the eye of the beholder? Motivated reasoning in disputed elections. Polit. Behav. 33 , 271–290 (2011).

Leeper, T. J. & Mullinix, K. J. Motivated reasoning. Oxford Bibliographies https://doi.org/10.1093/OBO/9780199756223-0237 (2018).

Johnson, H. M. & Seifert, C. M. Sources of the continued influence effect: when misinformation in memory affects later inferences. J. Exp. Psychol. Learn. Mem. Cogn. 20 , 1420–1436 (1994).

Wilkes, A. L. & Leatherbarrow, M. Editing episodic memory following the identification of error. Q. J. Exp. Psychol. Sect. A 40 , 361–387 (1988).

Ecker, U. K. H., Lewandowsky, S. & Apai, J. Terrorists brought down the plane!—No, actually it was a technical fault: processing corrections of emotive information. Q. J. Exp. Psychol. 64 , 283–310 (2011).

Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N. & Cook, J. Misinformation and its correction: continued influence and successful debiasing. Psychol. Sci. Public Interest 13 , 106–131 (2012).

Nyhan, B. & Reifler, J. Does correcting myths about the flu vaccine work? An experimental evaluation of the effects of corrective information. Vaccine 33 , 459–464 (2015).

Nyhan, B., Reifler, J., Richey, S. & Freed, G. L. Effective messages in vaccine promotion: a randomized trial. Pediatrics 133 , e835–e842 (2014).

Nyhan, B. & Reifler, J. When corrections fail: the persistence of political misperceptions. Polit. Behav. 32 , 303–330 (2010).

Rathje, S., Roozenbeek, J., Traberg, C. S., van Bavel, J. J. & van der Linden, S. Meta-analysis reveals that accuracy nudges have little to no effect for U.S. conservatives: regarding Pennycook et al. (2020). Psychol. Sci. https://doi.org/10.25384/SAGE.12594110.v2 (2021).

Greene, C. M., Nash, R. A. & Murphy, G. Misremembering Brexit: partisan bias and individual predictors of false memories for fake news stories among Brexit voters. Memory 29 , 587–604 (2021).

Gawronski, B. Partisan bias in the identification of fake news. Trends Cogn. Sci. 25 , 723–724 (2021).

Pennycook, G. & Rand, D. G. Lack of partisan bias in the identification of fake (versus real) news. Trends Cogn. Sci. 25 , 725–726 (2021).

Borukhson, D., Lorenz-Spreen, P. & Ragni, M. When does an individual accept misinformation? An extended investigation through cognitive modeling. Comput. Brain Behav. 5 , 244–260 (2022).

Roozenbeek, J. et al. Susceptibility to misinformation is consistent across question framings and response modes and better explained by myside bias and partisanship than analytical thinking susceptibility to misinformation. Judgm. Decis. Mak. 17 , 547–573 (2022).

Bolsen, T., Druckman, J. N. & Cook, F. L. The influence of partisan motivated reasoning on public opinion. Polit. Behav. 36 , 235–262 (2014).

Hameleers, M. & van der Meer, T. G. L. A. Misinformation and polarization in a high-choice media environment: how effective are political fact-checkers? Commun. Res. 47 , 227–250 (2020).

Guay, B., Berinsky, A., Pennycook, G. & Rand, D. How to think about whether misinformation interventions work. Preprint at PsyArXiv https://doi.org/10.31234/OSF.IO/GV8QX (2022).

Hove, M. J. & Risen, J. L. It’s all in the timing: interpersonal synchrony increases affiliation. Soc. Cogn. 27 , 949–960 (2009).

Tesch, F. E. Debriefing research participants: though this be method there is madness to it. J. Pers. Soc. Psychol. 35 , 217–224 (1977).

Tanner-Smith, E. E. & Tipton, E. Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Res Synth. Methods 5 , 13–30 (2014).

Tanner-Smith, E. E., Tipton, E. & Polanin, J. R. Handling complex meta-analytic data structures using robust variance estimates: a tutorial in R. J. Dev. Life Course Criminol. 2 , 85–112 (2016).

Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. , https://doi.org/10.18637/jss.v036.i03 (2010).

van Aert, R. C. M. CRAN—package puniform. R Project https://cran.r-project.org/web/packages/puniform/index.html (2022).

Coburn, K. M. & Vevea, J. L. weightr: estimating weight-function models for publication bias. (2021); https://cran.r-project.org/web/packages/weights/index.html

Fisher, Z. & Tipton, E. robumeta: an R-package for robust variance estimation in meta-analysis. ArXiv . https://doi.org/10.48550/arXiv.1503.02220 (2015).

Sidik, K. & Jonkman, J. N. Robust variance estimation for random effects meta-analysis. Comput. Stat. Data Anal. 50 , 3681–3701 (2006).

Hedges, L. V., Tipton, E. & Johnson, M. C. Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Methods 1 , 39–65 (2010).

JASP Team. JASP (2022); https://jasp-stats.org/

Higgins, J. P. T., Thompson, S. G., Deeks, J. J. & Altman, D. G. Measuring inconsistency in meta-analyses. Br. Med. J. 327 , 557–560 (2003).

Higgins, J. P. T. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21 , 1539–1558 (2002).

Tay, L. Q., Hurlstone, M. J., Kurz, T. & Ecker, U. K. H. A comparison of prebunking and debunking interventions for implied versus explicit misinformation. Br. J. Psychol. 113 , 591–607 (2022).

Tappin, B. M., Berinsky, A. J. & Rand, D. G. Partisans’ receptivity to persuasive messaging is undiminished by countervailing party leader cues. Nat. Hum. Behav. , https://doi.org/10.1038/s41562-023-01551-7 (2023).

Traberg, C. S. & van der Linden, S. Birds of a feather are persuaded together: perceived source credibility mediates the effect of political bias on misinformation susceptibility. Pers. Individ. Dif. 185 , 111269 (2022).

van Bavel, J. J. & Pereira, A. The partisan brain: an identity-based model of political belief. Trends Cogn. Sci. 22 , 213–224 (2018).

Kahan, D. M. Misconceptions, misinformation, and the logic of identity-protective cognition. SSRN Electron. J. https://doi.org/10.2139/SSRN.2973067 (2017).

Levendusky, M. Our Common Bonds: Using What Americans Share to Help Bridge the Partisan Divide (Univ. Chicago Press, 2023).

Voelkel, J. G. et al. Interventions reducing affective polarization do not improve anti-democratic attitudes. Nature Human Behaviour , 7 , 55–64 (2023); https://doi.org/10.31219/OSF.IO/7EVMP

Ecker, U. K. H., Hogan, J. L. & Lewandowsky, S. Reminders and repetition of misinformation: helping or hindering its retraction? J. Appl. Res. Mem. Cogn. 6 , 185–192 (2017).

Schwarz, N., Sanna, L. J., Skurnik, I. & Yoon, C. Metacognitive experiences and the intricacies of setting people straight: implications for debiasing and public information campaigns. in. Adv. Exp. Soc. Psychol. 39 , 127–161 (2007).

Ecker, U. K. H., Lewandowsky, S. & Chadwick, M. Can corrections spread misinformation to new audiences? Testing for the elusive familiarity backfire effect. Cogn. Res Princ. Implic. 5 , 41 (2020).

Kappel, K. & Holmen, S. J. Why science communication, and does it work? A taxonomy of science communication aims and a survey of the empirical evidence. Front. Commun. 4 , 55 (2019).

Fischhoff, B. The sciences of science communication. Proc. Natl Acad. Sci. USA 110 , 14033–14039 (2013).

Winters, M. et al. Debunking highly prevalent health misinformation using audio dramas delivered by WhatsApp: evidence from a randomised controlled trial in Sierra Leone. BMJ Glob. Health 6 , 6954 (2021).

Registered replication reports. Association for Psychological Science http://www.psychologicalscience.org/publications/replication (2017).

Vraga, E. K., Kim, S. C. & Cook, J. Testing logic-based and humor-based corrections for science, health, and political misinformation on social media. J. Broadcast Electron. Media 63 , 393–414 (2019).

Vijaykumar, S. et al. How shades of truth and age affect responses to COVID-19 (mis)information: randomized survey experiment among WhatsApp users in UK and Brazil. Humanit. Soc. Sci. Commun. 8 , 1–12 (2021).

Anderson, C. A., Lepper, M. R. & Ross, L. Perseverance of social theories: the role of explanation in the persistence of discredited information. J. Pers. Soc. Psychol. 39 , 1037–1049 (1980).

Sirlin, N., Epstein, Z., Arechar, A. A. & Rand, D. G. Digital literacy is associated with more discerning accuracy judgments but not sharing intentions. Harv. Kennedy Sch. Misinformation Rev. , https://doi.org/10.37016/mr-2020-83 (2021).

Arechar, A. A. et al. Understanding and reducing online misinformation across 16 countries on six continents. Preprint at PsyArXiv https://psyarxiv.com/a9frz/ (2022).

Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: experimental evidence for a scalable accuracy-nudge intervention. Psychol. Sci. 31 , 770–780 (2020).

Jahanbakhsh, F. et al. Exploring lightweight interventions at posting time to reduce the sharing of misinformation on social media. in Proc. ACM on Human–Computer Interaction vol. 5, 1–-42 (Association for Computing Machinery, 2021); https://doi.org/10.1145/3449092 (2021).

Pennycook, G. & Rand, D. G. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proc. Natl Acad. Sci. USA 116 , 2521–2526 (2019).

Gesser-Edelsburg, A., Diamant, A., Hijazi, R. & Mesch, G. S. Correcting misinformation by health organizations during measles outbreaks: a controlled experiment. PLoS ONE 13 , e0209505 (2018).

Mosleh, M., Martel, C., Eckles, D. & Rand, D. Promoting engagement with social fact-checks online. Preprint at OSF https://osf.io/rckfy/ (2022).

Andrews, E. A. Combating COVID-19 Vaccine Conspiracy Theories: Debunking Misinformation about Vaccines, Bill Gates, 5G, and Microchips Using Enhanced Correctives . MSc thesis, State Univ. New York at Buffalo (2021).

Koller, M. Rebutting accusations: when does it work, when does it fail? Eur. J. Soc. Psychol. 23 , 373–389 (1993).

Greitemeyer, T. & Sagioglou, C. Does exonerating an accused researcher restore the researcher’s credibility? PLoS ONE 10 , e0126316 (2015).

Hedges, L. V. & Olkin, I. Statistical Methods for Meta-analysis (Academic, 1985).

Hedges, L. V. Distribution Theory for Glass’s estimator of effect size and related estimators. J. Educ. Stat. 6 , 107 (1981).

Borenstein, M., Hedges, L., Higgins, J. & Rothstein, H. Introduction to Meta-analysis (Wiley, 2009).

Lakens, D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t -tests and ANOVAs. Front. Psychol. 4 , 863 (2013).

Morris, S. B. Distribution of the standardized mean change effect size for meta-analysis on repeated measures. Br. J. Math. Stat. Psychol. 53 , 17–29 (2000).

Hart, W. et al. Feeling validated versus being correct: a meta-analysis of selective exposure to information. Psychol. Bull. 135 , 555–588 (2009).

Lord, C. G., Ross, L. & Lepper, M. R. Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence. J. Pers. Soc. Psychol. 37 , 2098–2109 (1979).

Seifert, C. M. The continued influence of misinformation in memory: what makes a correction effective? Psychol. Learn. Motiv. 41 , 265–292 (2002).

van der Linden, S., Leiserowitz, A., Rosenthal, S. & Maibach, E. Inoculating the public against misinformation about climate change. Glob. Chall. 1 , 1600008 (2017).

Ecker, U. K. H. et al. The psychological drivers of misinformation belief and its resistance to correction. Nat. Rev. Psychol. 1 , 13–29 (2022).

Ecker, U., Sharkey, C. X. M. & Swire-Thompson, B. Correcting vaccine misinformation: A failure to replicate familiarity or fear-driven backfire effects. PLoS One, 18 , e0281140 (2023).

Gawronski, B., Brannon, S. M. & Ng, N. L. Debunking misinformation about a causal link between vaccines and autism: two preregistered tests of dual-process versus single-process predictions (with conflicting results). Soc. Cogn. 40 , 580–599 (2022).

Guenther, C. L. & Alicke, M. D. Self-enhancement and belief perseverance. J. Exp. Soc. Psychol. 44 , 706–712 (2008).

Misra, S. Is conventional debriefing adequate? An ethical issue in consumer research. J. Acad. Mark. Sci. 20 , 269–273 (1992).

Green, M. C. & Donahue, J. K. Persistence of belief change in the face of deception: the effect of factual stories revealed to be false. Media Psychol. 14 , 312–331 (2011).

Ecker, U. K. H. & Ang, L. C. Political attitudes and the processing of misinformation corrections. Polit. Psychol. 40 , 241–260 (2019).

Sherman, D. K. & Kim, H. S. Affective perseverance: the resistance of affect to cognitive invalidation. Pers. Soc. Psychol. Bull. 28 , 224–237 (2002).

Golding, J. M., Fowler, S. B., Long, D. L. & Latta, H. Instructions to disregard potentially useful information: the effects of pragmatics on evaluative judgments and recall. J. Mem. Lang. 29 , 212–227 (1990).

Viechtbauer, W. & Cheung, M. W.-L. Outlier and influence diagnostics for meta-analysis. Res. Synth. Methods 1 , 112–125 (2010).

Borenstein, M. in Publication Bias in Meta-analysis: Prevention, Assessment, and Adjustme nts (eds Rothstein, H. R., Sutton, A. J. & Borenstein, M.) 194–220 (John Wiley & Sons, 2005).

Duval, S. in Publication Bias in Meta-analysis: Prevention, Assessment, and Adjustme nts (eds Rothstein, H. R., Sutton, A. J. & Borenstein, M.) 127–144 (John Wiley & Sons, 2005).

Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R. & Rushton, L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J. Clin. Epidemiol. 61 , 991–996 (2008).

Stanley, T. D. & Doucouliagos, H. Meta-regression approximations to reduce publication selection bias. Res. Synth. Methods 5 , 60–78 (2014).

van Assen, M. A. L. M., van Aert, R. C. M. & Wicherts, J. M. Meta-analysis using effect size distributions of only statistically significant studies. Psychol. Methods 20 , 293–309 (2015).

Pustejovsky, J. E. & Rodgers, M. A. Testing for funnel plot asymmetry of standardized mean differences. Res. Synth. Methods 10 , 57–71 (2019).

Maier, M., Bartoš, F. & Wagenmakers, E. J. Robust Bayesian meta-analysis: addressing publication bias with model-averaging. Psychol. Methods , https://doi.org/10.1037/met0000405 (2022).

Download references

Acknowledgements

We thank D. O’Keefe, who assisted in the inter-rater reliability. Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH114847 (D.A.), the National Institute on Drug Abuse of the National Institutes of Health under Award Number DP1 DA048570 (D.A.) and the National Institute of Allergy and Infectious Diseases of the National Institutes of Health under award numbers R01AI147487 (D.A. and M.S.C.) and P30AI045008 (Penn Center for AIDS Research [Penn CFAR] subaward; M.S.C.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This research was supported by the Science of Science Communication Endowment from the Annenberg Public Policy Center at the University of Pennsylvania. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and affiliations.

Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA, USA

Man-pui Sally Chan

Annenberg School for Communication, Annenberg Public Policy Center, School of Arts and Sciences, School of Nursing, Wharton School, University of Pennsylvania, Philadelphia, PA, USA

Dolores Albarracín

You can also search for this author in PubMed   Google Scholar

Contributions

D.A. initiated the project, and M.S.C. supervised the project. Both M.S.C. and D.A. contributed to the theoretical formalism, developed the coding scheme and performed the coding reliability. M.S.C. took the lead in the data curation, preparing the analytical plan and performing the analytic calculations. Both M.S.C. and D.A. discussed the results and contributed to the final version of the manuscript.

Corresponding author

Correspondence to Man-pui Sally Chan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Human Behaviour thanks Jon Roozenbeek, Sander van der Linden and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary analyses and results, Tables 1–5 and Fig. 1.

Reporting Summary.

Supplementary table.

PRISMA 2020 Checklist.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Chan, Mp.S., Albarracín, D. A meta-analysis of correction effects in science-relevant misinformation. Nat Hum Behav 7 , 1514–1525 (2023). https://doi.org/10.1038/s41562-023-01623-8

Download citation

Received : 09 May 2022

Accepted : 09 May 2023

Published : 15 June 2023

Issue Date : September 2023

DOI : https://doi.org/10.1038/s41562-023-01623-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Engaging with conspiracy believers.

• Karen M. Douglas
• Robbie M. Sutton
• Daniel Toribio-Flórez

Review of Philosophy and Psychology (2024)

Psychological inoculation strategies to fight climate disinformation across 12 countries

• Tobia Spampatti
• Ulf J. J. Hahnel
• Tobias Brosch

Nature Human Behaviour (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Corrections, retractions and updates after publication

Taylor & francis journal article correction and retraction policy.

Sometimes after an article has been published it may be necessary to make a change to the Version of Record . This change will be made after careful consideration by the journal’s editorial team, with support from Taylor & Francis staff to make sure any necessary changes are done in accordance with both Taylor & Francis policies and guidance from the Committee on Publication Ethics (COPE).

Aside from cases where a minor error is concerned, any necessary changes will be accompanied by a post-publication notice, which will be permanently linked to the original article. These changes can be in the form of a Correction notice, an Expression of Concern , a Retraction , and in rare circumstances, a Removal .

The purpose of linking post-publication notices to the original article is to provide transparency around any changes and to ensure the integrity of the scholarly record. Note that all post-publication notices are free to access from the point of publication.

Read on for our full policy on corrections, retractions, and updates to published articles.

Version of Record

Each article published by Taylor & Francis journals, or journals published by us on behalf of a scholarly society, either in the print issue or online , constitutes the Version of Record (VoR): the final, definitive, and citable version in the scholarly record.

The VoR includes:

The article revised and accepted following peer review, in its final form, including the abstract, text, references, bibliography, and all accompanying tables, illustrations, data.

Any supplemental material.

Recognizing a published article as the VoR helps to provide further assurance that it is accurate, complete, and citable. Wherever possible it is our policy to maintain the integrity of the VoR in accordance with STM Association guidelines:

Publishing tips, direct to your inbox

Expert tips and guidance on getting published and maximizing the impact of your research. Register now for weekly insights direct to your inbox.

STM Guidelines on Preservation of the Objective Record of Science

What should I do if my article contains an error?

Authors should notify us as soon as possible if they find errors in their published article, especially errors that could affect the interpretation of data or reliability of information presented. It is the responsibility of the corresponding author to ensure consensus has been reached between all listed co-authors prior to requesting any corrections to an article.

If, after reading the guidance, you believe a correction is necessary for your article, please contact the journal’s Production Editor, or contact us .

Post-publication notices to ensure the accuracy of the scholarly record

Correction notice.

A Correction notice will be issued when it is necessary to correct an error or omission, where the interpretation of the article may be impacted but the scholarly integrity or original findings remains intact.

A correction notice, where possible, should always be written and approved by all authors of the original article. On very rare occasions where there is a need to correct an error made in the publication process, the journal may be required to issue a correction without the authors’ direct input. However, should this occur, the journal will make best efforts to notify the authors.

Please note that correction requests may be subject to full review, and if queries are raised, you may be expected to supply further information before the correction is approved.

Taylor & Francis distinguishes between major and minor errors. For correction notices, major errors or omissions are considered changes that impact the interpretation of the article, but the overall scholarly integrity remains intact. Minor errors are considered errors or omissions that do not impact the reliability of, or the readers’ understanding of, the interpretation of the article.

Major errors are always accompanied by a separate correction notice. The correction notice should provide clear details of the error and the changes that have been made to the Version of Record. Under these circumstances, Taylor & Francis will:

Correct the online article.

Issue a separate correction notice electronically linked back to the corrected version.

Add a footnote to the article displaying the electronic link to the correction notice.

Paginate and make available the correction notice in the online issue of the journal.

Make the correction notice free to view.

Minor errors may not be accompanied by a separate correction notice. Instead, a footnote will be added to the article detailing to the reader that the article has been corrected.

Concerns regarding the integrity of a published article should be raised via email to the Editor or via the Publisher .

Read our reference guide to the type of changes Taylor & Francis will correct using a correction notice.

Retractions

A Retraction will be issued where a major error (e.g., in the methods or analysis) invalidates the conclusions in the article, or where it appears research or publication misconduct has taken place (e.g., research without required ethical approvals, fabricated data, manipulated images, plagiarism, duplicate publication, etc.).

The decision to retract an article will be made in accordance with both Taylor & Francis policies and COPE guidelines. The decision will follow a full investigation by Taylor & Francis editorial staff in collaboration with the journal’s editorial team. Authors and institutions may request a retraction of their articles if they believe their reasons meet the criteria for retraction.

Retractions are issued to correct the scholarly record and should not be interpreted as punishments for the authors.

The COPE guidance can be found here .

Retraction will be considered in cases where:

There is clear evidence that the findings are unreliable, either as a result of misconduct (e.g., data fabrication or image manipulation) or honest error (e.g., miscalculation or experimental error).

The findings have previously been published elsewhere without proper referencing, permission, or justification (e.g., cases of redundant or duplicate publication).

The research constitutes plagiarism .

The Editor no longer has confidence in the validity or integrity of the article.

There is evidence or concerns of authorship for sale.

Citation manipulation is evident within the published paper

There is evidence of compromised peer review or systematic manipulation.

There is evidence of unethical research, or there is evidence of a breach of editorial policies.

The authors have deliberately submitted fraudulent or inaccurate information, or breached a warranty provided in the Author Publishing Agreement (APA).

Where the decision has been taken to retract an article, Taylor & Francis will:

Add a “retracted” watermark to the published Version of Record of the article.

Issue a separate retraction statement, titled ‘Retraction: [article title]’, that will be linked to the retracted article on Taylor & Francis Online.

Paginate and make available the retraction statement in the online issue of the journal.

Expressions of concern

In some cases, an Expression of Concern may be considered where concerns of a serious nature have been raised (e.g., research or publication misconduct), but where the outcome of the investigation is inconclusive or where due to various complexities, the investigation will not be completed for a considerable time. This could be due to ongoing institutional investigations or other circumstances outside of the journal’s control.

When the investigation has been completed, a Retraction or Correction notice may follow the Expression of Concern alongside the original article. All will remain part of the permanent publication record.

Expressions of Concern notices will be considered in cases where:

There is inconclusive evidence of research or publication misconduct by the authors, but the nature of the concerns warrants notifying the readers.

There are well-founded concerns that the findings are unreliable or that misconduct may have occurred, but there is limited cooperation from the authors’ institution(s) in investigating the concerns raised.

There is an investigation into alleged misconduct related to the publication that has not been, or would not be, fair and impartial or conclusive.

An investigation is underway, but a resolution will not be available for a considerable time, and the nature of the concerns warrant notifying the readers.

The Expression of Concern will be linked back to the published article it relates to.

Article removal

An Article Removal will be issued in rare circumstances where the problems cannot be addressed through a Retraction or Correction notice. Taylor & Francis will consider removal of a published article in very limited circumstances where:

The article contains content that could pose a serious risk of harm if acted upon or followed.

The article contains content which violates the rights to privacy of a study participant.

The article is defamatory or infringes other legal rights.

An article is subject to a court order.

In the case of an article being removed from Taylor & Francis Online, a removal notice will be issued in its place.

Updates and scholarly discussion on published articles

An addendum is a notification of an addition of information to an article.

Addenda do not contradict the original publication and are not used to fix errors (for which a Correction notice will be published), but if the author needs to update or add some key information then, this can be published as an addendum.

Addenda may be peer reviewed, according to journal policy, and are normally subject to oversight by the editors of the journal.

All addenda are electronically linked to the published article to which they relate.

Comment (including response and rejoinder correspondence)

Comments are short articles which outline an observation on a published article. In cases where a comment on a published article is submitted to the journal editor, it may be subject to peer review. The comment will be shared with the authors of the published article, who are invited to submit a response.

This author response again may be subject to peer review, and will be shared with the commentator, who may be invited to submit a rejoinder. The rejoinder may be subject to peer review and shared with the authors of the published article. No further correspondence will be considered for publication. The editor may decide to reject correspondence at any point before the comment, response and rejoinder are finalized.

All published comments, responses, and rejoinders are linked to the published article to which they relate.

Pop-up notifications

If deemed necessary by the Publishing Ethics & Integrity team , a pop-up notification may be temporarily added to the online version of an article to inform readers an article is under investigation. This is not a permanent note (unlike an Expression of Concern, Correction, or Retraction notice), but is to indicate an investigation is in progress. Please note, these are not added to every article under investigation.

Updating and retracting articles on F1000Research

On F1000Research, authors can revise, change, and update their articles by publishing new versions, which are added to the original article’s history on the platform. The versioning system is user-friendly and intuitive, with new versions (and their peer reviews) clearly linked and easily navigable from earlier versions. Authors can summarize changes in the ‘Amendments’ section at the start of a new version.

As stated in the F1000Research Retraction policy , articles may be retracted from F1000Research for several reasons, including research misconduct and duplicate publication, but the retracted article will usually remain on the site. Retracted articles are not ‘unpublished’ or ‘withdrawn’ so that they can be published elsewhere; usually the reasons for the retraction are so serious that the whole study, or large parts of it, are not appropriate for inclusion in the scientific literature anywhere.

Last updated 10th July 2024: Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > The Cambridge Handbook of Cognition and Education
• > Correcting Student Errors and Misconceptions

Book contents

• The Cambridge Handbook of Cognition and Education
• Copyright page
• Contributors
• How Cognitive Psychology Can Inform Evidence-Based Education Reform
• Part I Foundations
• Part II Science and Math
• Part III Reading and Writing
• Part IV General Learning Strategies
• 16 When Does Interleaving Practice Improve Learning?
• 17 Correcting Student Errors and Misconceptions
• 18 How Multimedia Can Improve Learning and Instruction
• 19 Multiple-Choice and Short-Answer Quizzing on Equal Footing in the Classroom
• 20 Collaborative Learning
• 21 Self-Explaining
• 22 Enhancing the Quality of Student Learning Using Distributed Practice
• Part V Metacognition

17 - Correcting Student Errors and Misconceptions

from Part IV - General Learning Strategies

Published online by Cambridge University Press:  08 February 2019

Access options

Save book to kindle.

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Correcting Student Errors and Misconceptions
• By Elizabeth J. Marsh , Emmaline Drew Eliseev
• Edited by John Dunlosky , Kent State University, Ohio , Katherine A. Rawson , Kent State University, Ohio
• Book: The Cambridge Handbook of Cognition and Education
• Online publication: 08 February 2019
• Chapter DOI: https://doi.org/10.1017/9781108235631.018

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sage Choice

Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project

Julia m. rohrer.

1 International Max Planck Research School on the Life Course, Max Planck Institute for Human Development, Berlin

2 Department of Psychology, University of Leipzig

Warren Tierney

3 Department of Organizational Behavior, INSEAD, Singapore

Eric L. Uhlmann

Lisa m. debruine.

4 Institute of Neuroscience and Psychology, University of Glasgow

5 Laboratory of Experimental Psychology, KU Leuven

6 Institute of Psychology, Leiden University

Benedict Jones

Stefan c. schmukle, raphael silberzahn.

7 Sussex Business School, University of Sussex

Rebecca M. Willén

8 Institute for Globally Distributed Open Research and Education (IGDORE)

Rickard Carlsson

9 Department of Psychology, Linnaeus University

Richard E. Lucas

10 Department of Psychology, Michigan State University

Julia Strand

11 Department of Psychology, Carleton College

Simine Vazire

12 Melbourne School of Psychological Sciences, University of Melbourne

Jessica K. Witt

13 Department of Psychology, Colorado State University

Thomas R. Zentall

14 Department of Psychology, University of Kentucky

Christopher F. Chabris

15 Autism and Developmental Medicine Institute, Geisinger Health System, Danville, Pennsylvania

Tal Yarkoni

16 Department of Psychology, University of Texas at Austin

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In 13 written statements, researchers from diverse branches of psychology share why and how they have lost confidence in one of their own published findings. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

Science is often hailed as a self-correcting enterprise. In the popular perception, scientific knowledge is cumulative and progressively approximates truth more accurately over time ( Sismondo, 2010 ). However, the degree to which science is genuinely self-correcting is a matter of considerable debate. The truth may or may not be revealed eventually, but errors can persist for decades, corrections sometimes reflect lucky accidents rather than systematic investigation and can themselves be erroneous, and initial mistakes might give rise to subsequent errors before they get caught ( Allchin, 2015 ). Furthermore, even in a self-correcting scientific system, it remains unclear how much of the knowledge base is credible at any given point in time ( Ioannidis, 2012 ) given that the pace of scientific self-correction may be far from optimal.

Usually, self-correction is construed as an outcome of the activities of the scientific community as a whole (i.e., collective self-correction): Watchful reviewers and editors catch errors before studies get published, critical readers write commentaries when they spot flaws in somebody else’s reasoning, and replications by impartial groups of researchers allow the scientific community to update their beliefs about the likelihood that a scientific claim is true. Far less common are cases in which researchers publicly point out errors in their own studies and question conclusions they have drawn before (i.e., individual self-correction). The perceived unlikeliness of such an event is facetiously captured in Max Planck’s famous statement that new scientific truths become established not because their enemies see the light but because those enemies eventually die ( Planck, 1948 ). However, even if individual self-correction is not necessary for a scientific community as a whole to be self-correcting in the long run ( Mayo-Wilson et al., 2011 ), we argue that it can increase the overall efficiency of the self-corrective process and thus contribute to a more accurate scientific record.

The Value of Individual Self-Correction

The authors of a study have privileged access to details about how the study was planned and conducted, how the data were processed or preprocessed, and which analyses were performed. Thus, the authors remain in a special position to identify or confirm a variety of procedural, theoretical, and methodological problems that are less visible to other researchers. 1 Even when the relevant information can in principle be accessed from the outside, correction by the original authors might still be associated with considerably lower costs. For an externally instigated correction to take place, skeptical “outsiders” who were not involved in the research effort might have to carefully reconstruct methodological details from a scant methods section (for evidence that often authors’ assistance is required to reproduce analyses, see e.g., Chang & Li, 2018 ; Hardwicke et al., 2018 ), write persuasive e-mails to get the original authors to share the underlying data (often to no avail; Wicherts et al., 2011 ), recalculate statistics because reported values are not always accurate (e.g., Nuijten et al., 2016 ), or apply advanced statistical methods to assess evidence in the presence of distortions such as publication bias ( Carter et al., 2019 ).

Eventually, external investigators might resort to an empirical replication study to clarify the matter. A replication study can be a very costly or even impossible endeavor. Certainly, it is inefficient when a simple self-corrective effort by the original authors might have sufficed. Widespread individual self-correction would obviously not eliminate the need for replication, but it would enable researchers to make better informed choices about whether and how to attempt replication—more than 30 million scientific articles have been published since 1965 ( Pan et al., 2018 ), and limited research resources should not be expended mindlessly on attempts to replicate everything (see also Coles et al., 2018 ). In some cases, individual self-correction could render an empirical replication study unnecessary. In other cases, additionally disclosed information might render an empirical replication attempt even more interesting. And in any case, full information about the research process, including details that make the original authors doubt their claims, would help external investigators maximize the informativeness of their replication or follow-up study.

Finally, in many areas of science, scientific correction has become a sensitive issue often discussed with highly charged language ( Bohannon, 2014 ). Self-correction could help defuse some of this conflict. A research culture in which individual self-corrections are the default reaction to errors or misinterpretations could raise awareness that mistakes are a routine part of science and help separate researchers’ identities from specific findings.

The Loss-of-Confidence Project

To what extent does our research culture resemble the self-correcting ideal, and how can we facilitate such behavior? To address these questions and to gauge the potential impacts of individual self-corrections, we conducted the Loss-of-Confidence Project. The effort was born out of a discussion in the Facebook group PsychMAP following the online publication of Dana Carney’s statement “My Position on Power Poses” ( Carney, 2016 ). Carney revealed new methodological details regarding one of her previous publications and stated that she no longer believed in the originally reported effects. Inspired by her open disclosure, we conducted a project consisting of two parts: an open call for loss-of-confidence statements and an anonymous online survey.

First, in our open call, we invited psychological researchers to submit statements describing findings that they had published and in which they had subsequently lost confidence. 2 The idea behind the initiative was to help normalize and destigmatize individual self-correction while, hopefully, also rewarding authors for exposing themselves in this way with a publication. We invited authors in any area of psychology to contribute statements expressing a loss of confidence in previous findings, subject to the following requirements:

• The study in question was an empirical report of a novel finding.
• The submitting author has lost confidence in the primary/central result of the article.
• The loss of confidence occurred primarily as a result of theoretical or methodological problems with the study design or data analysis.
• The submitting author takes responsibility for the errors in question.

The goal was to restrict submissions to cases in which the stigma of disclosing a loss of confidence in previous findings would be particularly high; we therefore did not accept cases in which an author had lost faith in a previous finding for reasons that did not involve his or her own mistakes (e.g., because of a series of failed replications by other researchers).

Second, to understand whether the statements received in the first part of the project are outliers or reflect a broader phenomenon that goes largely unreported, we carried out an online survey and asked respondents about their experience with losses of confidence. The full list of questions asked can be found at https://osf.io/bv48h/ . The link to the survey was posted on Facebook pages and mailing lists oriented toward scientists (PsychMAP, Psychological Methods Discussion Group, International Social Cognition Network, Society for Judgment and Decision Making (SJDM), SJDM mailing list) and further promoted on Twitter. Survey materials and anonymized data are made available on the project’s Open Science Framework repository ( https://osf.io/bv48h ).

Results: Loss-of-Confidence Statements

The project was disseminated widely on social media (resulting in around 4,700 page views of the project website), and public commentary was overwhelmingly positive, highlighting how individual self-correction is aligned with perceived norms of scientific best practices. By the time we stopped the initial collection of submissions (December 2017–July 2018), we had received loss-of-confidence statements pertaining to six different studies. After posting a preprint of an earlier version of this manuscript, we reopened the collection of statements and received seven more submissions, some of them while finalizing the manuscript. Table 1 provides an overview of the statements we received. 3

Overview of the Loss-of-Confidence Statements

Note: JIF = 2018 journal impact factor according to InCites Journal Citation Reports. Citations are according to Google Scholar on April 27, 2019.

In the following, we list all statements in alphabetical order of the first author of the original study to which they pertain. Some of the statements have been abbreviated; the long versions are available at OSF ( https://osf.io/bv48h/ ).

Statement on Carlsson and Björklund (2010) by Rickard Carlsson

In this study, we developed a new way to measure mixed (in terms of warmth and competence) stereotypes with the help of the implicit association test (IAT). In two studies, respondents took two IATs, and results supported the predictions: Lawyers were implicitly stereotyped as competent (positive) and cold (negative) relative to preschool teachers. In retrospect, there are a number of issues with the reported findings. First, there was considerable flexibility in what counted as support for the theoretical predictions. In particular, the statistical analysis in Study 2 tested a different hypothesis than Study 1. This analysis was added after peer review Round 2 and thus was definitely not predicted a priori. Later, when trying to replicate the reported analysis from Study 1 on the data from Study 2, I found that only one of the two effects reported in Study 1 could be successfully replicated. Second, when we tried to establish the convergent and discriminant validity of the IATs by correlating them with explicit measures, we committed the fallacy of taking a nonsignificant effect in an underpowered test as evidence for the null hypothesis, which in this case implied discriminant validity. Third, in Study 1, participants actually took a third IAT that measured general attitudes toward the groups. This IAT was not disclosed in the manuscript and was highly correlated with both the competence and the warmth IAT. Hence, it would have complicated our narrative and undermined the claim that we had developed a completely new measure. Fourth, data from an undisclosed behavioral measure were collected but never entered into data set or analyzed because I made a judgment that it was invalid based on debriefing of the participants. In conclusion, in this 2010 article, I claimed to have developed a way to measure mixed stereotypes of warmth and competence with the IAT. I am no longer confident in this finding.

Statement on Chabris and Hamilton (1992) by Christopher F. Chabris

This article reported a divided-visual-field (DVF) experiment showing that the skilled pattern recognition that chess masters perform when seeing a chess game situation was performed faster and more accurately when the stimuli were presented briefly in the left visual field, and thus first reached the right hemisphere of the brain, than when the stimuli were presented in the right field. The sample was large for a study of highly skilled performers (16 chess masters), but we analyzed the data in many different ways and reported the result that was most favorable. Most critically, we tried different rules for removing outlier trials and picked one that was uncommon but led to results consistent with our hypothesis. Nowadays, I would analyze these types of data using more justifiable rules and preregister the rules I was planning to use (among other things) to avoid this problem. For these reasons, I no longer think that the results provide sufficient support for the claims that the right hemisphere is more important than the left for chess expertise and for skilled visual pattern recognition. These claims may be true, but not because of our experiment.

Two other relevant things happened with this article. First, we submitted a manuscript describing two related experiments. We were asked to remove the original Experiment 1 because the p value for the critical hypothesis test was below .10 but not below .05. We complied with this request. We were also asked by one reviewer to run approximately 10 additional analyses of the data. We did not comply with this—instead, we wrote to the editor and explained that doing so many different analyses of the same data set would invalidate the p values. The editor agreed. This is evidence that the dangers of multiple testing were not exactly unknown as far back as the early 1990s. The sacrificed Experiment 1 became a chapter of my PhD thesis. I tried to replicate it several years later, but I could not recruit enough chess master participants. Having also lost some faith in the DVF methodology, I put that data in the “file drawer” for good.

Statement on Fisher et al. (2015) by Ben Jones and Lisa M. DeBruine

The article reported that women’s preferences for wearing makeup that was rated by other people as being particularly attractive were stronger in test sessions in which salivary testosterone was high than in test sessions in which salivary testosterone was relatively low. Not long after publication, we were contacted by a colleague who had planned to use the open data and analysis code from our article for a workshop on mixed effect models. They expressed some concerns about how our main analysis had been set up. Their main concern was that our model did not include random slopes for key within-subjects variables (makeup attractiveness and testosterone). Having looked into this issue over a couple of days, we agreed that not including random slopes typically increases false positive rates and that in the case of our study, the key effect for our interpretation was no longer significant. To minimize misleading other researchers, we contacted the journal immediately and asked to retract the article. Although this was clearly an unfortunate situation, it highlights the importance of open data and analysis code for allowing mistakes to be quickly recognized and the scientific record corrected accordingly.

Statement on Heyman et al. (2015) by Tom Heyman

The goal of the study was to assess whether the processes that presumably underlie semantic priming effects are automatic in the sense that they are capacity free. For instance, one of the most well-known mechanisms is spreading activation, which entails that the prime (e.g., cat) preactivates related concepts (e.g., dog), thus resulting in a head start. To disentangle prospective processes—those initiated on presentation of the prime, such as spreading activation—from retrospective processes—those initiated on presentation of the target—three different types of stimuli were selected. On the basis of previously gathered word association data, we used symmetrically associated word pairs (e.g., cat–dog; both prime and target elicit one another) as well as asymmetrically associated pairs in the forward direction (e.g., panda–bear; the prime elicits the target but not vice versa) and in the backward direction (e.g., bear–panda; the target elicits the prime but not vice versa). However, I now believe that this manipulation was not successful in teasing apart prospective and retrospective processes. Critically, the three types of stimuli do not solely differ in terms of their presumed prime–target association. That is, I overlooked a number of confounding variables, for one because a priori matching attempts did not take regression effects into account (for more details, see supplementary statement at https://osf.io/bv48h/ ). Unfortunately, this undercuts the validity of the study’s central claim.

Statement on Lucas and Diener (2001) by Richard E. Lucas

The article reported three studies that examined the types of situations that extraverts enjoy. Our goal was to assess whether—as intuition and some models of personality might suggest—extraverts are defined by their enjoyment of social situations or whether extraverts are actually more responsive to the pleasantness of situations regardless of whether these are social. We concluded that extraversion correlated more strongly with ratings of pleasant situations than unpleasant situations but not more strongly with social situations than nonsocial situations once pleasantness was taken into account. There are two primary reasons why I have lost confidence in this result. First, the sample sizes are simply too small for the effect sizes one should expect ( Schönbrodt & Perugini, 2013 ). I do not remember how our sample size decisions were made, and the sample sizes vary substantially across studies even though the design was essentially the same. This is especially important given that one important effect from the third and largest study would not have been significant with the sample sizes used in Studies 1 and 2. We did report an internal meta-analysis, but I have become convinced that these procedures cannot correct for other problematic research practices ( Vosgerau et al., 2019 ). Second, many participants were excluded from our final analyses. Two participants were excluded because they were outliers who strongly affected the results. We were transparent about this and reported analyses with and without these outliers. However, the results with the outliers included do not support our hypothesis. We also excluded a second group because their results seemed to indicate that they had misinterpreted the instructions. I still find our explanation compelling, and it may indeed be correct. However, I believe that the appropriate step would be to rerun the study with new procedures that could prevent this misunderstanding. Because we would never have been motivated to look for signs that participants misunderstood the instructions if the results had turned out the way we wanted in the first place, this is an additional researcher degree of freedom that can lead to unreplicable results.

Statement on Schmukle et al. (2007) by Stefan C. Schmukle

The original main finding was that the implicit gender self-concept measured with the IAT significantly correlated with second-digit/fourth digit (2D:4D) ratios for men ( r = .36, p = .02) but not for women. We used two different versions of a gender IAT in this study (one with pictures and one with words as gender-specific stimuli; r = .46), and we had two different 2D:4D measures (the first measure was based on directly measuring the finger lengths using a caliper, and the second was based on measuring the scans of the hands; r = .83). The correlation between IAT and 2D:4D was, however, significant only for the combination of picture IAT and 2D:4D scan measure but insignificant for other combinations of IAT and 2D:4D measures. When I was writing the manuscript, I thought that the pattern of results made sense because (a) the research suggested that for an IAT, pictures were better suited as stimuli than words and because (b) I assumed that the scan measures should lead to better results for psychometric reasons (because measurements were averaged across two raters). Accordingly, I reported only the results for the combination of picture IAT and 2D:4D scan measure in the article (for all results, see the long version of the loss-of-confidence statement at https://osf.io/bv48h/ ). In the meantime, I have lost confidence in this finding, and I now think that the positive association between the gender IAT and 2D:4D is very likely a false-positive result because I should have corrected the p value for multiple testing.

Statement on Silberzahn and Uhlmann (2013) by Raphael Silberzahn and Eric Uhlmann

In 2013, we published an article providing evidence that the meaning of a person’s name might affect the person’s career outcomes. In a large archival data set with more than 200,000 observations, we found that German professionals with noble-sounding last names such as Kaiser (“emperor”), König (“king”), and Fürst (“prince”) were more often found as managers compared with German people with common, ordinary last names such as Koch (“cook”) or Bauer (“farmer”). We applied what we believed to be a solid statistical approach, using generalized estimating equations first, and during the review process applied hierarchical linear modeling and controlled for various potential third variables, including linear controls for name frequency. A postpublication reanalysis by Uri Simonsohn using an expanded version of our data set identified a curvilinear name-frequency confound in the data, whereas we had used only linear controls. Applying the improved matched-names analysis to the larger data set conclusively overturned the original article’s conclusions. Germans with noble and nonnoble names are equally well represented in managerial positions. We subsequently coauthored a collaborative commentary ( Silberzahn et al., 2014 ) reporting the new results. This experience inspired us to pursue our line of work on crowdsourcing data analysis, in which the same data set is distributed to many different analysts to test the same hypothesis and the effect-size estimates are compared ( Silberzahn et al., 2018 ; Silberzahn & Uhlmann, 2015 ).

Statement on Smith and Zentall (2016) by Thomas R. Zentall

We have found, paradoxically, that pigeons are indifferent between a signaled 50% reinforcement alternative (leading half of the time to a stimulus that signals 100% reinforcement and otherwise to a stimulus that signals 0% reinforcement) over a guaranteed 100% reinforcement alternative. We concluded that the value of the signal for reinforcement (100% in both cases) determines choice and, curiously, that the signal for the absence of reinforcement has no negative value. More recently, however, using a similar design but involving extended training, we found that there was actually a significant preference for the 50% signaled reinforcement alternative over the 100% reinforcement alternative ( Case & Zentall, 2018 ). This finding required that we acknowledge that there is an additional mechanism involved: the contrast between what was expected and what was obtained (positive contrast). In the case of the 50% reinforcement alternative, 50% reinforcement was expected, but on half of the trials, a signal indicated that 100% reinforcement would be obtained (“elation,” analogous to the emotion felt by a gambler who hits the jackpot). Choice of the 100% reinforcement alternative comes with an expectation of 100% reinforcement, and because 100% reinforcement is obtained, there is no positive contrast and no elation. The recognition of our error in not acknowledging that positive contrast has led to a better understanding of the motivation that gamblers have to gamble in the face of repeated losses and occasional wins.

Statement on Strand et al. (2018) by Julia Strand

The article reported that when participants listened to spoken words in noise, the cognitive resources necessary to understand the speech (referred to as “listening effort”) were reduced when the speech was accompanied by dynamic visual stimulus—a circle that modulated with the amplitude of the speech. When attempting to replicate and extend that work, I discovered an error in the original stimulus presentation program that was responsible for the observed effect. The listening-effort task we used was based on response time, so the critical comparison was participant response times in conditions with and without the visual stimulus. There was an unintentional delay set in the timer of the condition without the visual stimulus, leading to artificially slowed response times in that condition. We contacted the journal, and they invited us to submit a replacement article. Given that the timing delay affected every observation for one condition in a systematic way, it was straightforward to reanalyze the data and present the results as they would have been without the error. The original article was not retracted but now links to the new article ( Strand et al., 2020 ) that presents the corrected results.

Statement on Vazire (2010) by Simine Vazire

In this article, I suggested a model in which self-reports are more accurate than peer reports for traits that are low in observability and low in evaluativeness, whereas peer reports are more accurate than self-reports for traits that are high in observability and high in evaluativeness. The main issue was that I ran many more analyses than I reported, and I cherry-picked which results to report. This is basically p -hacking, but because most of my results were not statistically significant, I did not quite successfully p -hack by the strict definition. Still, I cherry-picked the results that made the contrast between self-accuracy and peer accuracy the most striking and that fit with the story about evaluativeness and observability. That story was created post hoc and chosen after I had seen the pattern of results.

Statement on Willén and Strömwall (2012) by Rebecca M. Willén

In this study, I evaluated the criteria used by Swedish courts for assessing credibility of plaintiffs’ accounts. The main reasons for my loss of confidence in the results reported are listed below.

First, the main coder (myself) was not blind to the veracity of the statements. In addition, the main coder had also conducted the interviews, which means that she might have been influenced by the memory of nonverbal cues that were not supposed to have influenced the codings. The second coder was blind and did indeed come to different conclusions in his codings. These differences may have been a consequence of the conditions and nonverbal cues being known to the main coder, and this possibility remained undisclosed in the article.

Second, all four hypotheses described as confirmatory in the introduction of the article were in fact not formalized until after the data had been collected. It could be argued that the first three hypotheses were “obvious” and thereby implicitly already decided on. The fourth hypothesis, however, was far from obvious, and it was the result of exploratory analyses made by myself.

Finally, no gender differences were predicted, and gender was never planned to be analyzed at all. The gender findings are thus the result of exploratory analyses. This fact is, however, never made very explicit; instead, these unexpected results are highlighted even in the abstract.

That said, I do think there is reason to believe that one particular main finding is worth trying to replicate: “False and truthful confessions by 30 offenders were analyzed, and few significant effects were obtained.” That is, true and false statements by criminally experienced offenders might be more difficult to distinguish than true and false statements provided by the typical participants in deception and interrogation research (i.e., undergraduates without criminal experience).

Statement on Witt and Proffitt (2008) by Jessica K. Witt

The article reported that squeezing a rubber ball interferes with the processes necessary for the perceiver’s ability to reach to a target to affect perceived distance to the target (Experiment 3a). Participants judged the distance to targets that were beyond the reach of the arm, then picked up a conductor’s baton and reached to them. One group of participants applied a constant, firm pressure on a rubber ball while making their distance judgments, and another group did not. There are two primary flaws that cast doubt on the findings. One concerns the methodology. The sample sizes were small, so statistical power was likely to be quite low. The other concern regards the statistical analysis. The analysis reported in the article used an incorrectly specified model. Specifically, we calculated the mean estimated distance for each participant at each distance for a total of 10 estimates per participant, then analyzed these means as if they were independent observations. This inflated the degrees of freedom, which resulted in lower p values. When the data are analyzed correctly, the statistical significance of the critical effect of ball squeeze on estimated distance depends on whether or not an outlier is removed (for full results, see long version of the loss-of-confidence statement at https://osf.io/bv48h/ ). Model misspecification and low sample sizes also applied to Experiments 1, 2, and 3b. For Experiment 1, when the data are analyzed correctly, statistical significance depends on the exclusion of two outliers. For Experiment 2, the critical effect of tool condition was not significant; no outliers were identified. There were only 4 participants per condition, making the experimental outcomes inconclusive. For Experiment 3b, the article originally reported a null effect; upon reanalysis, the effect was still null. Experiment 4 is believed to have been analyzed correctly on the basis of the reported degrees of freedom, but those data have been lost and therefore cannot be confirmed. With such low statistical power, little confidence can be had that the reported data support the idea that squeezing a ball can interfere with the effect of tool use on estimated distance.

Statement on Yarkoni et al. (2005) by Tal Yarkoni

This study used a dynamic decision-making task to investigate the neural correlates of temporally extended decision-making. The central claim was that activation in areas of right lateral prefrontal cortex strongly and selectively predicted choice behavior in two different conditions; peak between-subjects brain-behavior correlations were around r = .75. I now think most of the conclusions drawn in this article were absurd on their face. My understanding of statistics has improved a bit since writing the article, and it is now abundantly clear to me that (a) I p -hacked to a considerable degree (e.g., the choice of cluster thresholds was essentially arbitrary) and that (b) because of the “winner’s curse,” statistically significant effect sizes from underpowered studies cannot be taken at face value (see Yarkoni, 2009 ). Beyond these methodological problems, I also now think the kinds of theoretical explanations I proposed in the article were ludicrous in their simplicity and naivete—so the results would have told us essentially nothing even if they were statistically sound (see Meehl, 1967 , 1990 ).

Discussion of the Loss-of-Confidence Statements

The studies for which we received statements spanned a wide range of psychological domains (stereotypes, working memory, auditory perception, visual cognition, face perception, personality and well-being, biologically driven individual differences, social cognition, decision-making in nonhuman animals, deception detection) and employed a diverse range of methods (cognitive tasks, implicit and explicit individual differences measures, archival data analyses, semistructured interviews, functional MRI), demonstrating the broad relevance of our project. Overall, the respective original articles had been cited 1,559 times as of April 27, 2020, according to Google Scholar, but the number of citations varied widely, from nine to 740. The reasons given for the submitters’ loss of confidence also varied widely, with some statements providing multiple reasons. Broadly speaking, however, we can group the explanations into three general categories.

Methodological error

Five of the statements reported methodological errors in the broadest sense. In three instances, submitters (Jones & DeBruine; Silberzahn & Uhlmann; Witt) lost confidence in their findings upon realizing that their key results stemmed from misspecified statistical models. In those three cases, the submitters discovered, after publication, that a more appropriate model specification resulted in the key effect becoming statistically nonsignificant. In another instance, Carlsson reported that upon reconsideration, two studies included in his article actually tested different hypotheses—a reanalysis testing the same hypotheses in Study 2 actually failed to fully support the findings from Study 1. Finally, Strand lost confidence when she found out that a programming error invalidated her findings.

Invalid inference

Four of the statements reported invalid inferences in the broadest sense. In two cases (Heyman and Yarkoni), the submitters attributed their loss of confidence to problems of validity—that is, to a discrepancy between what the reported results actually showed (a statistically significant effect of some manipulation or measure) and what the article claimed to show (a general relationship between two latent constructs). In a similar vein, Zentall lost confidence in a conclusion when a follow-up experiment revealed that an extension of the experimental procedures suggested that the original mechanism was not sufficient to account for the phenomenon. Although the latter loss-of-confidence statement might be closest to normative assumptions about how science advances—new empirical insights lead to a revision of past conclusions—it also raises interesting questions: At what point should researchers lose confidence in a methodological decision made in one study based on the results of other studies that are, in principle, also fallible?

Seven of the statements (Carlsson, Chabris, Lucas, Yarkoni, Schmukle, Vazire, and Willén) reported some form of p -hacking—specifically, failing to properly account for researcher degrees of freedom when conducting or reporting the analyses. We hasten to emphasize that our usage of “ p -hacking” here does not imply any willful attempt to mislead. Indeed, some of the submitters noted that the problems in question stemmed from their poor (at the time) understanding of relevant statistical considerations. The statement by Lucas also highlights how subtle researcher degrees of freedom can affect analyses: Although the justification for a specific exclusion criterion still seems compelling, the researcher would not have been motivated to double-check data points if the desired results had emerged in the initial analysis.

Results and Discussion of the Anonymous Online Survey

Overall, 316 scientists completed the survey. Most (93%) reported being affiliated with a university or a research institute, and all career stages from graduate students to tenured professors were represented. We did not limit the survey to particular fields of research but asked respondents to indicate their department (if applicable); 43% did not report a department, 37% worked at a psychology department, and the remaining respondents were distributed over a broad range of fields (e.g., business, economics, medicine). Almost all respondents reported working either in Europe (44%) or the United States (47%). Figure 1 provides an overview of the survey results.

An overview of the findings from the loss-of-confidence survey.

Almost half of the respondents (44%) reported losing confidence in at least one of their findings. Another 14% were not sure whether they had lost confidence according to our definition for a variety of reasons. For example, some reported that their confidence in one of their own research articles was low to begin with; some had lost confidence in their theoretical explanation but not in the general effect—or conversely, in the effect but not in the theory; others doubted whether their results would generalize to other contexts. Respondents who reported losing confidence were then asked to elaborate on the case for which they felt most responsible. 4 Of the respondents who stated that they had experienced a loss of confidence, more than half (56%) said that it was due to a mistake or shortcoming in judgment on the part of the researchers, and roughly one in four (28%) took primary responsibility for the error.

Strikingly, the primary reason indicated for a loss of confidence was self-admitted questionable research practices (e.g., p -hacking and selective reporting; 52%). However, a broad variety of other reasons were also reported. The loss of confidence was a matter of public record in fewer than a fifth of the reported cases (17%), and if it was a matter of public record, the outlets primarily chosen (statement in later publication, conference presentation, social media posting) were not directly linked to the original research article. Respondents whose loss of confidence was not public reported multiple reasons for the lack of disclosure. Many felt insufficiently sure about the loss of confidence to proceed (68%). Some stated the belief that public disclosure was unnecessary because the finding had not attracted much attention (46%), expressed concerns about hurting the feelings of coauthors (33%), or cited the lack of an appropriate venue (25%), uncertainty about how to best communicate the matter (25%), and worries about how the loss of confidence would be perceived (24%).

On the whole, these survey results suggest a nuanced view of losses of confidence. Researchers may start to question their own findings for a broad variety of reasons, and different factors may then keep them from publicly disclosing this information. Collectively, the responses suggest that a sizeable proportion of active researchers has lost confidence in at least one of their findings—often because of a recognized error of their own commission.

Note that our respondents do not constitute a representative sample of researchers. Furthermore, estimating article-level rather than researcher-level loss of confidence requires assumptions and extrapolations. 5 Thus, caution should be exercised when interpreting the specific numerical estimates reported here. Nevertheless, one can attempt a very conservative extrapolation: More than 1 million academic articles are currently published each year ( Jinha, 2010 ). Supposing that at least a third of these are empirical research reports, and that even just 1% of these reports are affected, that still leaves thousands of articles published each year that will eventually lose the confidence of at least some of their authors—often because of known errors yet typically without any public disclosure.

General Discussion

The Loss-of-Confidence Project raises a number of questions about how one should interpret individual self-corrections.

First, on a substantive level, how should one think about published empirical studies in cases in which the authors have explicitly expressed a loss of confidence in the results? One intuitive view is that authors have no privileged authority over “their” findings, and thus such statements should have no material impact on a reader’s evaluation. On the other hand, even if authors lack any privileged authority over findings they initially reported, they clearly often have privileged access to relevant information. This is particularly salient for the p -hacking disclosures reported in the loss-of-confidence statements. Absent explicit statements of this kind, readers would most likely not be able to definitively identify the stated problems in the original report. In such cases, we think it is appropriate for readers to update their evaluations of the reported results to accommodate the new information.

Even in cases in which a disclosure contributes no new methodological information, one might argue that the mere act of self-correction should be accorded a certain weight. Authors have presumably given greater thought to and are more aware of their own study’s potential problems and implications than a casual reader. The original authors may also be particularly biased to evaluate their own studies favorably—so if they have nonetheless lost confidence, this might heuristically suggest that the evidence against the original finding is particularly compelling.

Second, on a metalevel, how should one think about the reception one’s project received? On the one hand, one could argue that the response was about as positive as could reasonably be expected. Given the unconventional nature of the project and the potentially high perceived cost of public self-correction, the project organizers (J. M. Rohrer, C. F. Chabris, T. Yarkoni) were initially unsure whether the project would receive any submissions. From this perspective, even the 13 submissions we ultimately received could be considered a clear success and a testament to the current introspective and self-critical climate in psychology.

On the other hand, the survey responses we received suggest that the kinds of errors disclosed in the statements are not rare. Approximately 12% of the 316 survey respondents reported losing confidence in at least one of their articles for reasons that matched our stringent submission criteria (i.e., because of mistakes that the respondent took personal responsibility for), and nearly half acknowledged a loss of confidence more generally.

This suggests that potentially hundreds, if not thousands, of researchers could have submitted loss-of-confidence statements but did not do so. There are many plausible reasons for this, including not having heard of the project. However, we think that at least partially, the small number of submitted statements points to a gap between researchers’ ideals and their actual behavior—that is, public self-correction is desirable in the abstract but difficult in practice.

Fostering a culture of self-correction

As has been seen, researchers report a variety of reasons for both their losses of confidence and their hesitation to publicly disclose a change in thinking. However, we suggest that there is a broader underlying factor: In the current research environment, self-correction, or even just critical reconsideration of one’s past work, is often disincentivized professionally. The opportunity costs of a self-correction are high; time spent on correcting past mistakes and missteps is time that cannot be spent on new research efforts, and the resulting self-correction is less likely to be judged a genuine scientific contribution. Moreover, researchers may worry about self-correction potentially backfiring. Corrections that focus on specific elements from an earlier study might be perceived as undermining the value of the study as a whole, including parts that are in fact unaffected by the error. Researchers might also fear that a self-correction that exposes flaws in their work will damage their reputation and perhaps even undermine the credibility of their research record as a whole.

To tackle these obstacles to self-correction, changes to the research culture are necessary. Scientists make errors (and this statement is certainly not limited to psychological researchers; see e.g., Eisenman et al., 2014 ; García-Berthou & Alcaraz, 2004 ; Salter et al., 2014 ; Westra et al., 2011 ), and rectifying these errors is a genuine scientific contribution—whether it is done by a third party or the original authors. Scientific societies could consider whether they want to more formally acknowledge efforts by authors to correct their own work. Confronted with researchers who publicly admit to errors, other researchers should keep in mind that willingness to admit error is not a reliable indicator of propensity to commit errors—after all, errors are frequent throughout the scientific record. On the contrary, given the potential (or perceived) costs of individual self-corrections, public admission of error could be taken as a credible signal that the issuer values the correctness of the scientific record. However, ultimately, given the ubiquity of mistakes, we believe that individual self-corrections should become a routine part of science rather than an extraordinary occurrence.

Different media for self-correction

Unfortunately, good intentions are not enough. Even when researchers are committed to public self-correction, it is often far from obvious how to proceed. Sometimes, self-correction is hindered by the inertia of journals and publishers. For example, a recent study suggested that many medical journals published correction letters only after a significant delay, if at all ( Goldacre et al., 2019 ), and authors who tried to retract or correct their own articles after publication have encountered delays and reluctance from journals (e.g., Grens, 2015 ). Even without such obstacles, there is presently no standardized protocol describing what steps should be taken when a loss of confidence has occurred.

Among the participants of the Loss-of-Confidence Project, Fisher et al. (2015) decided to retract their article after they became aware of their misspecified model. But researchers may often be reluctant to initiate a retraction given that retractions occur most commonly as a result of scientific misconduct ( Fang et al., 2012 ) and are therefore often associated in the public imagination with cases of deliberate fraud. To prevent this unwelcome conflation and encourage more frequent disclosure of errors, journals could introduce a new label for retractions initiated by the original authors (e.g., “Authorial Expression of Concern” or “voluntary withdrawal”; see Alberts et al., 2015 ). Furthermore, an option for authorial amendments beyond simple corrections (up to and including formal versioning of published articles) could be helpful.

Thus, it is not at all clear that widespread adoption of retractions would be an effective, fair, or appropriate approach. Willén (2018) argued that retraction of articles in which questionable research practices (QRPs) were employed could deter researchers from being honest about their past actions. Furthermore, retracting articles because of QRPs known to be widespread (e.g., John et al., 2012 ) could have the unintended side effect that some researchers might naively conclude that a lack of a retraction implies a lack of QRPs. Hence, Willén suggested that all articles should be supplemented by transparent retroactive disclosure statements. In this manner, the historical research record remains intact because information would be added rather than removed.

Preprint servers (e.g., PsyArXiv.com ) and other online repositories already enable authors to easily disclose additional information to supplement their published articles or express their doubts. However, such information also needs to be discoverable. Established databases such as PubMed could add links to any relevant additional information provided by the authors. Curate Science (curatescience.org), a new online platform dedicated to increasing the transparency of science, is currently implementing retroactive statements that could allow researchers to disclose additional information (e.g., additional outcome measures or experimental manipulations not reported in the original article) in a straightforward, structured manner.

Another, more radical step would be to move scientific publication entirely online and make articles dynamic rather than static such that they can be updated on the basis of new evidence (with the previous version being archived) without any need for retraction ( Nosek & Bar-Anan, 2012 ). For example, the Living Reviews journal series in physics by Springer Nature allows authors to update review articles to incorporate new developments.

The right course of action once one has decided to self-correct will necessarily depend on the specifics of the situation, such as the reason for the loss of confidence, publication norms that can vary between research fields and evolve over time, and the position that the finding takes within the wider research. For example, a simple but consequential computational error may warrant a full retraction, whereas a more complex confound may warrant a more extensive commentary. In research fields in which the published record is perceived as more definitive, a retraction may be more appropriate than in research fields in which published findings have a more tentative status. In addition, an error in an article that plays a rather minor role in the context of the wider research may be sufficiently addressed in a corrigendum, whereas an error in a highly cited study may require a more visible medium for the self-correction to reach all relevant actors.

That said, we think that both the scientific community and the broader public would profit if additional details about the study, or the author’s reassessment of it, were always made public and always closely linked to the original article—ideally in databases and search results as well as the publisher’s website and archival copies. A cautionary tale illustrates the need for such a system: In January 2018, a major German national weekly newspaper published an article ( Kara, 2018a ) that uncritically cited the findings of Silberzahn and Uhlmann (2013) . Once the journalist had been alerted that these findings had been corrected in Silberzahn et al. (2014) , she wrote a correction to her newspaper article that was published within less than a month of the previous article ( Kara, 2018b ), demonstrating swift journalistic self-correction and making a strong point that any postpublication update to a scientific article should be made clearly visible to all readers of the original article.

All of these measures could help to transform the cultural norms of the scientific community, bringing it closer to the ideal of self-correction. Naturally, it is hard to predict which ones will prove particularly fruitful, and changing the norms of any community is a nontrivial endeavor. However, it might be encouraging to recall that over the past few years, scientific practices in psychology have already changed dramatically ( Nelson et al., 2018 ). Hence, a shift toward a culture of self-correction may not be completely unrealistic, and psychology, with its increasing focus on openness, may even serve as a role model for other fields of research to transform their practices.

Finally, it is quite possible that fears about negative reputational consequences are exaggerated. It is unclear whether and to what extent self-retractions actually damage researchers’ reputations ( Bishop, 2018 ). Recent acts of self-correction such as those by Carney (2016) , which inspired our efforts in this project, Silberzahn and Uhlmann ( Silberzahn et al., 2014 ), Inzlicht (2016) , Willén (2018) , and Gervais (2017) have received positive reactions from within the psychological community. They remind us that science can advance at a faster pace than one funeral at a time.

Acknowledgments

We thank Michael Inzlicht, Alison Ledgerwood, Kateri McRae, and Victoria Savalei, who all contributed to the initial draft of the project concept, and Nick Brown, who proofread an earlier version of the manuscript. C. F. Chabris contributed to this work while he was a Visiting Fellow at the Institute for Advanced Study in Toulouse, France. Additional material appears at https://osf.io/bv48h/

1. Guidelines to promote openness (e.g., Nosek et al., 2015 ) might partly reduce this asymmetry and thus make it easier for third parties to spot flaws.

2. An archived version of the website can be found at https://web.archive.org/web/20171212055615/https://lossofconfidence.com/ .

3. Readers are cautioned to infer nothing about original authors who did not join or sign a loss-of-confidence statement about their own articles. In some cases, these authors approved of the submission but did not get involved otherwise; in others, they had already left the field of research.

4. Respondents who were not sure whether they had experienced a loss of confidence could also answer the follow-up questions. However, many decided not to answer, and for those who answered, responses are hard to interpret given the broad variety of scenarios they were referring to. Thus, we decided to restrict the following analyses to respondents with an unambiguous loss of confidence.

5. In the survey, we also asked researchers to indicate in how many of their articles they had lost confidence. An analysis of these numbers suggested that respondents had collectively lost confidence in more than 10% of their publications in total or more than 7% counting only those articles in which they had lost confidence because of an error for which they took primary responsibility. Of course, these are extrapolations based on retrospective self-reports, and we cannot assume respondents can give perfect estimates of the relevant quantities. For this reason, a number of our key analyses focus on the respondents’ description of the one case for which they felt most responsible.

Transparency

Action Editor: Chris Crandall

Editor: Laura A. King

Author Contributions: T. Yarkoni and C. F. Chabris initialized the project in 2016. J. M. Rohrer managed the project starting from 2017, launched the corresponding website, and took the lead in writing the manuscript. W. Tierney and E. L. Uhlmann took the lead in designing the loss-of-confidence survey, and numerous authors provided feedback and edits on the survey content. L. M. DeBruine, T. Heyman, B. Jones, S. C. Schmukle, R. Silberzahn, E. L. Uhlmann, R. M. Willén, and T. Yarkoni submitted loss-of-confidence statements during the first round of data collection. R. Carlsson, R. E. Lucas, J. Strand, S. Vazire, J. K. Witt, T. R. Zentall, and C. F. Chabris submitted statements at a later point in time. All the authors provided critical feedback and helped shape the manuscript. Authorship order was determined by the following rule: lead author (J. M. Rohrer); authors who led the survey [W. Tierney and E. L. Uhlmann]; authors of loss-of-confidence statements received during first round of data collection, in alphabetical order; authors of loss-of-confidence statements received later, in alphabetical order; and senior authors [C. F. Chabris and T. Yarkoni]. All of the authors approved the final manuscript for submission.

Declaration of Conflicting Interests: The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Funding: J. K. Witt is supported by the National Science Foundation (Grant BCS-1632222), and W. Tierney and E. L. Uhlmann’s work was supported by an R&D grant from INSEAD. Part of this research was conducted while T. Heyman was a postdoctoral fellow of the Research Foundation-Flanders (FWO-Vlaanderen).

Publisher Correction: Summary of Research: What Is the True Impact of Cognitive Impairment for People Living with Multiple Sclerosis? A Commentary of Symposium Discussions at the 2020 European Charcot Foundation

• Publisher Correction
• Open access
• Published: 19 March 2024
• Volume 13 , page 501, ( 2024 )

Cite this article

You have full access to this open access article

• Sarah A. Morrow 1 ,
• Paola Kruger 2 ,
• Dawn Langdon 3 &
• Nektaria Alexandri 4  

284 Accesses

Explore all metrics

The Original Article was published on 20 February 2024

Avoid common mistakes on your manuscript.

Publisher Correction: Neurology and Therapy https://doi.org/10.1007/s40120-024-00579-9

In the sentence beginning ‘This is a summary of an original published...’ in this article, the text ‘You can access the full article for free, here: https://doi.org/10.1007/s40120-023-00519-z ’ has been removed and the corrected sentence now reads ‘This is a summary of an original published article entitled “What is the true impact of cognitive impairment for people living with multiple sclerosis? A commentary of symposium discussions at the 2020 European Charcot Foundation” [ 1 ]’.

In this article reference 1 was missing and should have been as given below.

The original article has been corrected.

Morrow, S.A., Kruger, P., Langdon, D. et al. What is the true impact of cognitive impairment for people living with multiple sclerosis? A commentary of symposium discussions at the 2020 European Charcot Foundation. Neurol Ther. 2023;12:1419–1429. https://doi.org/10.1007/s40120-023-00519-z .

Article   PubMed   PubMed Central   Google Scholar  

Download references

Author information

Authors and affiliations.

Department of Clinical Neurological Sciences, Western University, London, ON, Canada

Sarah A. Morrow

Patient Advocate, Rome, Italy

Paola Kruger

Department of Psychology, Health and Wellbeing, Royal Holloway, University of London, London, UK

Dawn Langdon

Global Medical Affairs, Neurology and Immunology, The Healthcare Business of Merck KGaA, Darmstadt, Germany

Nektaria Alexandri

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sarah A. Morrow .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/ .

Reprints and permissions

About this article

Morrow, S.A., Kruger, P., Langdon, D. et al. Publisher Correction: Summary of Research: What Is the True Impact of Cognitive Impairment for People Living with Multiple Sclerosis? A Commentary of Symposium Discussions at the 2020 European Charcot Foundation. Neurol Ther 13 , 501 (2024). https://doi.org/10.1007/s40120-024-00596-8

Download citation

Published : 19 March 2024

Issue Date : June 2024

DOI : https://doi.org/10.1007/s40120-024-00596-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Advertisement

• Find a journal
• Publish with us
• Track your research

Help | Advanced Search

Computer Science > Computation and Language

Title: when can llms actually correct their own mistakes a critical survey of self-correction of llms.

Abstract: Self-correction is an approach to improving responses from large language models (LLMs) by refining the responses using LLMs during inference. Prior work has proposed various self-correction frameworks using different sources of feedback, including self-evaluation and external feedback. However, there is still no consensus on the question of when LLMs can correct their own mistakes, as recent studies also report negative results. In this work, we critically survey broad papers and discuss the conditions required for successful self-correction. We first find that prior studies often do not define their research questions in detail and involve impractical frameworks or unfair evaluations that over-evaluate self-correction. To tackle these issues, we categorize research questions in self-correction research and provide a checklist for designing appropriate experiments. Our critical survey based on the newly categorized research questions shows that (1) no prior work demonstrates successful self-correction with feedback from prompted LLMs in general tasks, (2) self-correction works well in tasks that can use reliable external feedback, and (3) large-scale fine-tuning enables self-correction.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs

About The BMJ

• Resources for authors
• Forms, policies, and ethics

Corrections and retractions

Corrections.

We try hard not to make mistakes, but errors — both by editors and by authors — do creep into the journal. We publish corrections when necessary, and, to ensure that corrections are handled consistently, one editor deals with them all.

We set no time limit for notifying errors or publishing corrections. We always try to contact the author of the original article unless the error is very obvious, and we publish all corrections as soon as we can.

If your article is published on thebmj.com, and you discover a mistake between online and print publication we will normally correct this by making the article correct in the printed version. Only in exceptional circumstances will we publish a corrected version of an article.

In the print journal we publish corrections either singly in the relevant section or, more usually, grouped together in a box of "Corrections and clarifications." On thebmj.com we indicate in red that an article has a correction and provide a direct link to it.

If you want further advice about our policy or would like to notify us about the need for a specific correction please email our Publishing Services team. Please give as much detail as possible about errors, including any views on how they might have arisen.

Retractions

Retractions are considered by journal editors in cases of evidence of unreliable data or findings, plagiarism, duplicate publication, and unethical research. We may consider an expression of concern notice if an article is under investigation.

A replacement version of the online article is posted containing just the metadata, with a retraction note replacing the original text. The PDF may be removed or replaced with a version watermarked with “Retracted.” A retraction notice is published in print.

• Publishing model
• Editorial staff
• Advisory panels
• Explore The BMJ
• BMJ Student
• How green is The BMJ?
• Sources of revenue
• Article types and preparation
• Article submission
• Authorship & contributorship
• Competing interest policy
• Copyright, open access, and permission to reuse
• Patient consent and confidentiality
• Research Ethics
• BMJ papers audit
• Guidance for new authors
• BMJ Christmas issue
• Resources for advertisers and sponsors
• Resources for BMA members
• Resources for media
• Resources for subscribers
• Resources for readers
• Resources for reviewers
• About The BMJ app
• Poll archive
• International jobs

This week's poll

Read related article

See previous polls

• Open access
• Published: 11 January 2021

25 years on, the written error correction debate continues: an interview with John Truscott

• Hassan Mohebbi   ORCID: orcid.org/0000-0003-3661-1690 1 , 2  

Asian-Pacific Journal of Second and Foreign Language Education volume  6 , Article number:  3 ( 2021 ) Cite this article

7580 Accesses

10 Citations

6 Altmetric

Metrics details

Introduction

Since the publication of ‘the case against grammar correction in L2 writing classes’ in 1996, written corrective feedback has become a controversial issue in second language writing instruction (Lee, 2020 ; for a review, study Reinders & Mohebbi, 2018 ) as far as almost every single article investigating the effect of feedback on L2 writing improvement refers to this paper (citation = 2480). On the one hand, Truscott, in a series of research publications (Truscott, 1996 , 1999 , 2001 , 2004 , 2007 , 2009 , 2010 , 2016 ; (Truscott: The efficacy of written corrective feedback: A critique of a meta-analysis, unpublished); Truscott & Hsu, 2008 ) argued against the perceived and claimed positive effect of written corrective feedback on improving writing accuracy of L2 learners. On the other hand, some well-known scholars such as Bitchener ( 2008 , 2012a , 2012b ), Bitchener and Ferris ( 2012 ), Bitchener and Knoch ( 2008 , 2009a , 2009b , 2009c , 2010 , 2015 ), Ellis ( 2009 ), Ferris ( 1995 , 1997 , 1999 , 2002 , 2003 , 2004 , 2010 , 2012 , 2014 , 2015 ), Guenette ( 2007 ), Hyland ( 2010 ), and Lee ( 2013 , 2016 ), to name but a few, have studied the potential role of written corrective feedback in L2 writing from different perspectives to provide support for the positive role of correction in L2 writing and offer strategies for L2 language writing teachers. After 25 years, we may have a better picture of the research findings to evaluate correction effectiveness in L2 writing. Therefore, we interviewed Professor John Truscott to have his feedback on feedback research.

Many thanks for giving me this opportunity to interview you.

First of all, the first question I would like to pose is the primary motivation for writing the paper ‘The case against grammar correction in L2 writing classes (1996)’?

That paper was the result of several things coming together. One was my own experience as a writing instructor. I was conscientiously correcting my students’ errors but not seeing any meaningful benefits to their writing. I heard similar concerns from frustrated colleagues, unhappy that their students kept making the same mistakes after being repeatedly corrected on them. At the same time, I was doing a lot of reading on various topics in second language learning/teaching, in an effort at self-education, and correction was one topic that caught my eye. There was lots of evidence that correcting was a bad idea, a view that fits well with my experience and my understanding of the nature of language and learning, and was also suggested by the complexities of the feedback process. However, what I saw in the literature and in practice was an unquestioning acceptance of correction. This situation called for a strong statement, telling teachers that there is a choice to be made and presenting the case for the no-correction option.

I should maybe add that error correction has always been a secondary interest for me, a detour from my main interests, which are in cognitive science, especially as it relates to language. People in writing instruction have occasionally noted that I disappear from the field for several years at a time. That is because I am pursuing other interests.

Did you expect to receive responses from other researchers? What made you write a response to Ferris, after 3 years, in 1999?

No, I thought the paper would probably be ignored, like all the experimental work that was already pointing to the futility of correction. Regarding my response to Ferris, the immediate reason for writing it was that the editors invited me to, which of course was appropriate since her paper was explicitly a critique of my earlier paper. I accepted the invitation to show that everything in the original 1996 paper remained valid and that there was no genuine case for correction.

What did you find most challenging in Ferris’s argumentation for feedback and correction?

Well, to be blunt, I did not find anything particularly challenging. If I have to pick out one thing, I suppose it would be drawing out the implicit assumptions in that argumentation, mainly the burden of proof assumption. I hope that readers interested in this topic will go through those two papers together, compare them point by point, and refer back to the 1996 paper where appropriate.

If you could turn the clock back, would you add or remove any argument or counter-argument to your paper in 1996?

I would say that paper has stood up pretty well over the years. I might adjust some details of the presentation and change the emphasis in places, but I do not see a need for any substantive changes in the arguments.

In the past 25 years, we have seen many published papers that indicate the positive effects of error correction. Ferris, Bitchener, Knoch, Ellis, Guenette, and Lee were the leading researchers. Do you reject all the findings? Is there not any research paper that touched your heart?

First, I would disagree with the statement that those papers indicate positive effects of error correction. I have discussed that at length in various places and will not go into the specifics here. Interested readers might look at the paper I recently uploaded to ResearchGate, “The efficacy of written corrective feedback: A critique of a meta-analysis”. Is there any paper among them that especially impresses me? No, not really. Some clearly have their strengths, in design and methodology, for example, and in the interesting data, they produced. However, I do not see any study that has made a meaningful case that correction is effective. More importantly, the issue is what the overall body of research says. In my judgment, it says, quite clearly, that correction does not work– it does not help learners improve their ability to write accurately.

After 25 years, if you would like to summarize the main gaps of the studies supporting the positive effect of written corrective feedback, what would they be?

I am not sure about “gaps”, but there is a long list of problems with the claims that studies have shown positive effects. Again, I would refer readers to my various papers on this subject, especially the most recent one, on ResearchGate. I am not very comfortable giving a short, superficial summary of these big issues … but here goes.

Many studies are so narrow and specialized and/or artificial that they have little relevance to teaching; this is what made it possible for them to get good results. A couple of others were not actually about second language learning (titles should not always be trusted). Several studies commonly cited evidence that correction works did not address the question but looked instead at how different types of correction compare to each other. Others compared groups that were given a learning task plus correction on it to a group that did not get the learning task (or correction on it, of course), and the authors then invalidly concluded that the correction was responsible for observed differences in the performance of the groups. In the past, favourable reviews of correction often included studies which showed only that learners could (sometimes) use the corrections they receive to revise the writing on which they received them. Such studies were lumped together with research that looked at the effect of correction on learning. I think the field is finally getting over this bit of folly. Finally, we should not forget that there is a substantial body of studies that clearly found correction unhelpful or even harmful, findings that are often inappropriately dismissed or misrepresented.

In this context, I would like to note an interesting observation that has been made in meta-analyses of correction work, both written and oral –an apparent paradox in the research findings. The strongest results come from studies that did very little correcting, typically providing students with feedback only one time. The weakest results come from studies that provided feedback several times over an extended period. For a correction-skeptic like me, this seemingly bizarre finding makes perfect sense. The longer, more serious studies were efforts to test correction as it is done in language classes and as it affects learners’ ability to use the language accurately in realistic ways. They found that correction does not work. The experiments using brief treatments and obtaining strong results were typically very artificial, with little relevance to actual teaching and learning; and this artificiality is what made their strong results possible.

So if you see this body of research as pointing to the failure of correction as a teaching tool, then there is no paradox; the facts are as expected. But if you want to tell teachers that the research supports the use of correction, then you have a problem. The advice you offer should apparently be something like this: “Give your students corrective feedback on one assignment and then be sure to avoid giving any further feedback in the class”. This appears to be a logical consequence of the claim that the research has found correction effective. If it is not, then those who make this claim need to offer a convincing explanation for the paradox. The absence of such an explanation is a serious gap.

You have mentioned that one of your main research interests is the cognitive aspect of language learning. Do you think that a possible reason for learners’ failure to learn from feedback is that they do not have explicit and implicit knowledge of the teacher’s target structure?

Yes, certainly. One part of the case against correction is that in order for a given instance of correction to succeed a great many different things, all have to go right. Failure in any one of them can render the correction ineffective or even harmful. Knowledge of the target structure is one of the areas where things can go wrong.

Don’t you think that you should have done more experimental research to provide counter-evidence in the response of the proponents of written corrective feedback?

I do not think there is any shortage of experimental research showing the futility of correction. The issue is how this body of research has been (mis)interpreted. The literature has always been filled with statements that the evidence supports correction. The more critical assessments say that significant support for the practice exists, but findings are inconsistent. What is needed, now more than ever is a critical voice challenging these overly optimistic claims. Most importantly, teachers need to know that there is a choice to be made, and they need to be presented with both sides of the issue.

That said, I do think worthwhile experiments can be done, and I am interested in doing some.

If you want to call for papers to examine the effectiveness of written corrective feedback, what would be the main research questions?

The main answer is the one that I have always given. Research should try to identify special, limited ways in which correction might be useful. The question of error types is especially interesting. I wrote a paper on this long ago, speculating on what types might be correctible and suggesting that experiments should be done on them. But that has not been pursued. In their published reports, I would also like to see authors present much more information on how different error types responded to correction in their experiments.

A second type of research that I would suggest is work that seeks to clarify the implications of existing research findings. One example is the study that Angela Hsu and I did on the distinction between feedback as input for revision and feedback as a tool for learning, showing that findings commonly treated as evidence for learning actually have nothing to say about it. Another is Ekiert and di Gennaro’s conceptual replication of the Bitchener and Knoch research ( 2021 ). They found that the latter’s narrow focus made the treatment look much more successful than it actually was (and in the process raised doubts about its value even within that narrow focus).

Today, we have many options to get learners’ writing corrected by software and give them great support. What is your stance regarding the potential contribution of technology to L2 writing instruction? Do you not think that technology can compensate for the weakness in teachers’ practice and strategies in writing classes?

I think the potential of technology in this area, and others, is worth exploring. Computers are very good at keeping track of the errors each learner has made, for example, and they do not suffer from inattention and fatigue problems. In the context of the overall case against correction, I am skeptical about how far such things will take us. More generally, the use of technology in education has produced many false hopes and disappointments in the past. Whether current efforts will prove different is an open question and an interesting one.

How do you draw the picture of the road ahead in this field of research?

Based on the current state of things, I would say the prospects are not very good. I see a field in the grip of a kind of groupthink, with discussion dominated by researchers who share a favourable view of correction and continually reinforce each other’s core beliefs on the subject, beliefs that reflect tradition and intuition, and further reinforced by these factors. In my judgment, one result is that support for correction is seen where it does not exist, and contrary evidence is dismissed or downplayed. Not a healthy situation for the development of a research field. But of course, my own biases on this topic have always been clear.

As you may endorse, writing is an essential skill. When you reject written corrective feedback, what is your replacement? Do you not accept any kind of strategies that have been proposed, namely focused, unfocused, direct, indirect, and metalinguistic feedback?

Writing is a very important skill. So if writing instructors feel obliged to devote considerable time and energy – theirs and their students’ –to practices that do not work, then changes are needed.

How can we replace correction? The question is a familiar one, but to me, it is an odd thing to ask. When we talk about replacing something, we assume that something serves a useful function; we then ask what else could serve that function equally well. My thesis is that correction is not serving any function. So the question of how it can be replaced is to me quite odd. On the other hand, if the question is simply how we can fill the time that is currently devoted to correction, I do not think teachers need any advice from me. There is never enough time in a writing class to do everything as extensively or intensively as it could be done; abandoning correction will allow teachers to spend more time on whatever they think is most in need of that additional time.

As for the different ways to correct, I do not find the distinctions among direct, indirect, and metalinguistic feedback particularly interesting – we should be about equally skeptical toward all of them. Focused correction is a more interesting topic since it ties in with the question of error types. If it can be shown that particular types actually do benefit from correction, then feedback that focuses on those types might be appropriate, subject to questions of practicality. On the other hand, unfocused feedback sees correction as a general purpose tool, which it is not.

What about teachers’ pedagogical knowledge, their language proficiency, and their writing assessment literacy? Is it not possible to find fault with teachers concerning the alleged ineffectiveness of correction?

Limitations in teachers’ knowledge and ability certainly belong on the list of things that can make correction ineffective. However, I do not see this as finding fault with teachers, and I do not see any prospects of changing the situation in any meaningful way. Teachers are humans, languages are insanely complex, time is limited, and we have no science of writing instruction to guide teacher training. It is noteworthy that proponents of correction have not achieved any consensus on providing feedback, even on fundamental issues like comprehensive vs selective or direct vs indirect vs metalinguistic.

As the last question, what is your suggestion for language teachers? What should they do in teaching writing? How should language learners study writing?

First, I should acknowledge that apart from the special topic of error correction, I am not an authority on writing instruction. For most familiar practices, I do not have any critique to offer or any strong endorsement. I would like to see some serious research into the effectiveness of standard practices and possibly some changes in response to its findings, but I do not have anything to say on that. On a more positive note, I think the role of input deserves more attention than it commonly gets in writing instruction. If you want to learn to produce good written English (for example), you need very extensive experience seeing and processing good written English, getting a feel for what good writing looks like. I’m afraid this rather obvious point has been obscured by an excessive concern with errors and grammar rules.

Many thanks for your time and your responses.

Availability of data and materials

Not applicable.

Bitchener, J. (2008). Evidence in support of written corrective feedback. Journal of Second Language Writing , 17 (2), 102–118.

Article   Google Scholar  

Bitchener, J. (2012a). A reflection on ‘the language learning potential’ of written CF. Journal of Second Language Writing , 21 (4), 348–363.

Bitchener, J. (2012b). Written corrective feedback for L2 development: Current knowledge and future research. TESOL Quarterly , 46 (4), 855–867.

Bitchener, J., & Ferris, D. R. (2012). Written corrective feedback in second language acquisition and writing . New York: Routledge.

Book   Google Scholar  

Bitchener, J., & Knoch, U. (2008). The value of written corrective feedback for migrant and international students. Language Teaching Research , 12 (3), 409–431.

Bitchener, J., & Knoch, U. (2009a). The value of a focused approach to written corrective feedback. ELT Journal , 63 (3), 204–211.

Bitchener, J., & Knoch, U. (2009b). The relative effectiveness of different types of direct written corrective feedback. System , 37 (2), 322–329.

Bitchener, J., & Knoch, U. (2009c). The contribution of written corrective feedback to language development: A ten month investigation. Applied Linguistics , 31 (2), 193–214.

Bitchener, J., & Knoch, U. (2010). Raising the linguistic accuracy level of advanced L2 writers with written corrective feedback. Journal of Second Language Writing , 19 (4), 207–217.

Bitchener, J., & Knoch, U. (2015). Written corrective feedback studies: Approximate replication of Bitchener & Knoch (2010a) and Van Beuningen, De Jong &Kuiken (2012). Language Teaching , 48 (3), 405–414.

Ekiert, M., & di Gennaro, K. (2021). Focused written corrective feedback and linguistic target mastery: Conceptual replication of Bitchener and Knoch (2010). Language Teaching, 54 (1), 71–89.

Ellis, R. (2009). A typology of written corrective feedback. ELT Journal , 63 (2), 97–107.

Ferris, D. R. (1995). Student reactions to teacher response in multiple-draft composition classrooms. TESOL Quarterly , 29 (1), 33–53.

Ferris, D. R. (1997). The influence of teacher commentary on student revision. TESOL Quarterly , 31 (2), 315–339.

Ferris, D. R. (1999). The case for grammar correction in L2 writing classes: A response to Truscott (1996). Journal of Second Language Writing , 8 (1), 1–11.

Ferris, D. R. (2002). Treatment of error in second language student writing . Ann Arbor: The University of Michigan Press.

Google Scholar  

Ferris, D. R. (2003). Response to students writing: Implications for second language students . Mahwah: Lawrence Erlbaum.

Ferris, D. R. (2004). The “grammar correction” debate in L2 writing: Where are we, and where do we go from here? (and what do we do in the meantime … .?). Journal of Second Language Writing , 13 (1), 49–62.

Ferris, D. R. (2010). Second language writing research and written corrective feedback in SLA: Intersections and practical applications. Studies in Second Language Acquisition , 32 (2), 181–201.

Ferris, D. R. (2012). Written corrective feedback in second language acquisition and writing studies. Language Teaching , 45 (4), 446–459.

Ferris, D. R. (2014). Responding to student writing: Teachers’ philosophies and practices. Assessing Writing , 19 , 6–23.

Ferris, D. R. (2015). Written corrective feedback in L2 writing: Connors & Lunsford (1988); Lunsford & Lunsford (2008); Lalande (1982). Language Teaching , 48 (4), 531–544.

Guenette, D. (2007). Is feedback pedagogically correct? Research design issues in studies of feedback on writing. Journal of Second Language Writing , 16 (1), 40–53.

Hyland, F. (2010). Future directions in feedback on second language writing: Overview and research agenda. International Journal of English Studies , 10 (2), 171–182.

Lee, I. (2013). Research into practice: Written corrective feedback. Language Teaching , 46 (1), 108–119.

Lee, I. (2016). Teacher education on feedback in EFL writing: Issues, challenges, and future directions. TESOL Quarterly , 50 (2), 518–527.

Lee, I. (2020). Utility of focused/comprehensive written corrective feedback research for authentic L2 writing classrooms. Journal of Second Language Writing , 49 , 100734.

Reinders, H., & Mohebbi, H. (2018). Written corrective feedback: The road ahead. Language Teaching Research Quarterly , 6 , 1–6.

Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning , 46 (2), 327–369.

Truscott, J. (1999). The case for “the case against grammar correction in L2 writing classes”: A response to Ferris. Journal of Second Language Writing , 8 (2), 111–122.

Truscott, J. (2001). Selecting errors for selective error correction. Concentric: Studies in English Literature and Linguistics , 27 (2), 93–108.

Truscott, J. (2004). Evidence and conjecture on the effects of correction: A response to Chandler. Journal of Second Language Writing , 13 (4), 337–343.

Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of Second Language Writing , 16 (4), 255–272.

Truscott, J. (2009). Arguments and appearances: A response to Chandler. Journal of Second Language Writing , 18 (1), 59–60.

Truscott, J. (2010). Some thoughts on Anthony Bruton’s critique of the correction debate. System , 38 (2), 329–335.

Truscott, J. (2016). The effectiveness of error correction: Why do meta-analytic reviews produce such different answers? In Y.-n. Leung (Ed.), Epoch making in English teaching and learning: A special monograph for celebration of ETA-ROC’s 25th anniversary , (pp. 129–141). Taipei: Crane.

Truscott, J., & Hsu, A. Y. (2008). Error correction, revision, and learning. Journal of Second Language Writing , 17 (4), 292–305.

Download references

Acknowledgements

I am thankful to John Truscott for accepting the interview.

About the authors

Short Biography (John Truscott)

John Truscott is a professor in the Institute of Learning Sciences and Technology and the Center for Teacher Education at National Tsing Hua University in Taiwan. His primary research interest is the Modular Cognition Framework (formerly known as MOGUL, or Modular Online Growth and Use of Language), a broad cognitive framework aimed at bringing together research and theory from a variety of areas in order to better understand language as a part of the human mind. He has also published extensively on the topic of error correction and in other areas of second language acquisition, bilingualism, and linguistics, including form-focused instruction, vocabulary learning, extensive reading, the nature and roles of conscious and unconscious learning, and a variety of other theoretical topics. He is the author of Consciousness and Second Language Learning (2015, Multilingual Matters) and co-author, with Mike Sharwood Smith, of The Multilingual Mind (2014, Cambridge University Press) and The Internal Context of Bilingual Processing (2019, John Benjamins).

Hassan Mohebbi holds a PhD in TEFL. His main research interests are written corrective feedback, assessment literacy, first language use in SLA, and teacher’s pedagogical knowledge. He has co-edited special issues with Christine Coombe for Language Testing in Asia and Language Teaching Research Quarterly journals. He is an editorial board member of Asian-Pacific Journal of Second and Foreign Language Education (Springer), Language Testing in Asia (Springer), Innovation in Language Learning and Teaching (Taylor and Francis), Language Teaching Research Quarterly (EUROKD), Frontiers in Psychology, and Frontiers in Communication. https://publons.com/researcher/1445975/hassan-mohebbi/ .

Author information

Authors and affiliations.

SAM Language Institute, Ardabil, Iran

Hassan Mohebbi

European knowledge Development Institute, Ankara, Turkey

You can also search for this author in PubMed   Google Scholar

Contributions

Hassan Mohebbi had the idea of the paper, and John Truscott accepted the interview invitation. The interview was conducted in different stages, in which it was revised, and new questions were posed. The author(s) read and approved the final manuscript.

Authors’ information

No interest.

Corresponding author

Correspondence to Hassan Mohebbi .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mohebbi, H. 25 years on, the written error correction debate continues: an interview with John Truscott. Asian. J. Second. Foreign. Lang. Educ. 6 , 3 (2021). https://doi.org/10.1186/s40862-021-00110-9

Download citation

Received : 26 October 2020

Accepted : 05 January 2021

Published : 11 January 2021

DOI : https://doi.org/10.1186/s40862-021-00110-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Correction Notices

Errors can occur in published journal articles. Some errors require the publisher to not only correct the article but also issue a correction notice : a formal, public announcement of the correction that alerts readers to the changes to the published work. A correction may also be called an erratum (plural: errata ) or a corrigendum (plural: corrigenda ). The guidance on this page applies to corrections published with any of these names.

Errors requiring a correction notice

Minor typographical errors (e.g., spelling and grammar mistakes) can be corrected in the digital version of an article but do not usually require a correction notice. However, more substantive errors do need formal, public correction. These include rearranging the order of authorship, adding information to the author note, replacing an entry in the reference list, and altering data or results. Additional examples of changes needing a correction notice are shown in the sample correction notices on this page.

Correction notices are covered in the seventh edition APA Style Publication Manual in Section 12.22

Process of correcting a published article

If you detect an error in your published article (including an online first article), the first step is to inform the editor and publisher of the journal of the error. The editor and publisher will determine whether a formal correction notice is needed. The formal correction notice would serve to correct the knowledge base for current and future users of the information in the published article.

If a correction notice is needed, you are responsible for writing it. In your communication with the journal editor, submit a proposed correction notice that outlines what the error was, what the correct information is, and whether some or all versions of the original article have been corrected. The correction notice should contain the following elements:

• the article title
• the names of all authors, exactly as they appear in the published article
• the full journal name
• the year, volume number, issue number, page numbers, and DOI of the article being corrected
• the precise location of the error (e.g., page number, column, line, table, figure, appendix)
• an exact quotation of the error or, in the case of lengthy errors or an error in a table or figure, an accurate paraphrasing of the error
• a concise, clear wording of the correction, or in the case of an error in a table or figure, a replacement version of the table or figure

Once approved, the correction notice is created in the journal’s official template using the information provided by the author. This correction notice is usually published with a DOI both in print and online. The correction notice is also appended to the article’s record in research databases so that readers will retrieve it when they access the article or the database record for the article. Oftentimes, a corrected version of the article is also posted online and noted as being corrected on the first page.

Citing a corrected article

If you are citing an article that has been corrected, it is not necessary to note that the article has been corrected in the in-text citation or reference list entry. Simply write a standard reference list entry for the work, and ensure you do not reproduce any errors from the original. Readers will be informed of the correction when they retrieve the cited work.

Sample correction notices

The following are examples of correction notices published in APA journals. Use these as examples when writing your own correction notice to send to a journal. Note, however, that different journals may have different policies for addressing errors, corrections, and retractions to published articles, so always consult the editor of the journal in which your published article appeared.

• Sample Correction Notice — Armenta et al., 2013 (PDF, 20KB)
• Sample Correction Notice — Hecht et al., 2016 (PDF, 40KB)
• Sample Correction Notice — Fetvadjiev and He, 2018 (PDF, 50KB)

Credits for sample correction notices

From "Where Are You From? A Validation of the Foreigner Objectification Scale and the Psychological Correlates of Foreigner Objectification among Asian Americans and Latinos: Correction to Armenta et al. (2013),” 2015, Cultural Diversity and Ethnic Minority Psychology , 21 (2), p. 267 (https://doi.org/10.1037/cdp0000050). Copyright 2015 by the American Psychological Association.

From “Parsing the Heterogeneity of Psychopathy and Aggression: Differential Associations Across Dimensions and Gender: Correction to Hecht et al. (2016),” 2017, Personality Disorders: Theory, Research, and Treatment , 8 (1), p. 13 (https://doi.org/10.1037/per0000225). Copyright 2017 by the American Psychological Association.

From “The Longitudinal Links of Personality Traits, Values, and Well-Being and Self-Esteem: A Five-Wave Study of a Nationally Representative Sample: Correction to Fetvadjiev and He (2018),” 2019, Journal of Personality and Social Psychology , 117 (2), p. 337 (https://doi.org/10.1037/pspp0000246). Copyright 2019 by the American Psychological Association.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

An assessment of error-correction procedures for learners with autism

Affiliation.

• 1 Texana Behavior Improvement Center.
• PMID: 24114225
• DOI: 10.1002/jaba.65

Prior research indicates that the relative effectiveness of different error-correction procedures may be idiosyncratic across learners, suggesting the potential benefit of an individualized assessment prior to teaching. In this study, we evaluated the reliability and utility of a rapid error-correction assessment to identify the least intrusive, most effective procedure for teaching discriminations to 5 learners with autism. The initial assessment included 4 commonly used error-correction procedures. We compared the total number of trials required for the subject to reach the mastery criterion under each procedure. Subjects then received additional instruction with the least intrusive procedure associated with the fewest number of trials and 2 less effective procedures from the assessment. Outcomes of the additional instruction were consistent with those from the initial assessment for 4 of 5 subjects. These findings suggest that an initial assessment may be beneficial for identifying the most appropriate error-correction procedure.

Keywords: assessment; autism; conditional discrimination; error correction.

© Society for the Experimental Analysis of Behavior.

PubMed Disclaimer

Similar articles

• Examination of efficacious, efficient, and socially valid error-correction procedures to teach sight words and prepositions to children with autism spectrum disorder. Kodak T, Campbell V, Bergmann S, LeBlanc B, Kurtz-Nelson E, Cariveau T, Haq S, Zemantic P, Mahon J. Kodak T, et al. J Appl Behav Anal. 2016 Sep;49(3):532-47. doi: 10.1002/jaba.310. Epub 2016 May 6. J Appl Behav Anal. 2016. PMID: 27150389
• Comparing the effectiveness of error-correction strategies in discrete trial training. Turan MK, Moroz L, Croteau NP. Turan MK, et al. Behav Modif. 2012 Mar;36(2):218-34. doi: 10.1177/0145445511427973. Epub 2011 Nov 30. Behav Modif. 2012. PMID: 22133991
• Using an abbreviated assessment to identify effective error-correction procedures for individual learners during discrete-trial instruction. Carroll RA, Owsiany J, Cheatham JM. Carroll RA, et al. J Appl Behav Anal. 2018 Jul;51(3):482-501. doi: 10.1002/jaba.460. Epub 2018 Apr 19. J Appl Behav Anal. 2018. PMID: 29675825
• Caring for children and adolescents with autism who require challenging procedures. Souders MC, Freeman KG, DePaul D, Levy SE. Souders MC, et al. Pediatr Nurs. 2002 Nov-Dec;28(6):555-62. Pediatr Nurs. 2002. PMID: 12593340 Review.
• Autism from developmental and neuropsychological perspectives. Sigman M, Spence SJ, Wang AT. Sigman M, et al. Annu Rev Clin Psychol. 2006;2:327-55. doi: 10.1146/annurev.clinpsy.2.022305.095210. Annu Rev Clin Psychol. 2006. PMID: 17716073 Review.
• A Decision-Making Tool for Evaluating and Selecting Prompting Strategies. Cowan LS, Lerman DC, Berdeaux KL, Prell AH, Chen N. Cowan LS, et al. Behav Anal Pract. 2022 Jun 8;16(2):459-474. doi: 10.1007/s40617-022-00722-8. eCollection 2023 Jun. Behav Anal Pract. 2022. PMID: 35698480 Free PMC article.
• Adapting Direct Services for Telehealth: A Practical Tutorial. Bergmann S, Toussaint KA, Niland H, Sansing EM, Armshaw G, Baltazar M. Bergmann S, et al. Behav Anal Pract. 2021 Oct 12;14(4):1010-1046. doi: 10.1007/s40617-020-00529-5. eCollection 2021 Dec. Behav Anal Pract. 2021. PMID: 34659652 Free PMC article.
• A Tutorial for the Design and Use of Assessment-Based Instruction in Practice. Kodak T, Halbur M. Kodak T, et al. Behav Anal Pract. 2021 Jan 19;14(1):166-180. doi: 10.1007/s40617-020-00497-w. eCollection 2021 Mar. Behav Anal Pract. 2021. PMID: 33732586 Free PMC article.
• Comparing Error Correction to Errorless Learning: A Randomized Clinical Trial. Leaf JB, Cihon JH, Ferguson JL, Milne CM, Leaf R, McEachin J. Leaf JB, et al. Anal Verbal Behav. 2020 Feb 19;36(1):1-20. doi: 10.1007/s40616-019-00124-y. eCollection 2020 Jun. Anal Verbal Behav. 2020. PMID: 32699736 Free PMC article.
• Evaluating the use of programmed reinforcement in a correction procedure with children diagnosed with autism. Carneiro ACC, Flores EP, da Silva Barros R, de Souza CBA. Carneiro ACC, et al. Psicol Reflex Crit. 2019 Nov 15;32(1):21. doi: 10.1186/s41155-019-0134-3. Psicol Reflex Crit. 2019. PMID: 32026010 Free PMC article.
• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources, other literature sources.

• scite Smart Citations
• Genetic Alliance
• MedlinePlus Health Information

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• How We Do Our Work
• COVID-19 Criminal Justice Responses Tracker
• Publications
• CJI Newsletter
• Annual Reports

Each year, CJI plans, facilitates, and hosts the Institutional Corrections Research Network (ICRN) / National Corrections Reporting Program (NCRP) Annual Meeting, sponsored by the National Institute of Corrections (NIC) and the Bureau of Justice Statistics (BJS).

The meeting brings together corrections researchers from state agencies and federal partners to share information about data analysis tools and internal research. Participants generally include data suppliers to NCRP, members of ICRN, and staff from Abt Associates, NIC, BJS, and CJI.

The goals of this annual meeting are to:

• Provide recommendations for a national research agenda and to assist the corrections field in further developing infrastructure to have high-quality data and share it through national partnerships; and
• Further the work of the corrections field in its understanding and application of research by bringing together agency-based researchers to discuss issues and share insights on research conducted within agencies that operate correctional institutions.

2021 ICRN Conference 

The eighth annual meeting took place virtually on May 25, 2021, featuring presentations from member jurisdictions regarding various topics related to data and research. Presentations included:

• Welcome: Michael Kane, CJI; Danielle Kaeble, Bureau of Justice Statistics; Ann Carson, Bureau of Justice Statistics; Shaina Vanek, National Institute of Corrections
• National Corrections Reporting Program Data: Melissa Nadel and Walter Campbell, Abt Associates
• Updated Assaults Model: Eric Ballenger, Indiana Department of Corrections
• Unpacking Recidivism: Tama Celi, Virginia Department of Corrections
• Predicted Probability of Recidivism Model: Alejandra Livingston, Nevada Department of Corrections

View the full 2021 agenda here.  

2020 ICRN Conference 

The seventh annual meeting took place virtually on June 12, 2020, featuring presentations from member jurisdictions regarding their use of data and research in corrections department responses to the COVID-19 pandemic. The group also heard from BJS on recent and upcoming data collection efforts. Additional virtual meetings may take place.

Presentations:

• Proposed BJS Data Collection on COVID‐19 in State and Federal Prisons
• Colorado Department of Corrections (presented by Morgan Jackson)
• Hawaii Department of Public Safety (presented by George King)
• Idaho Department of Correction – Idaho’s Research Plan (presented by Janeena White)
• Indiana Department of Correction – COVID-19 Data Initiatives (presented by Sarah Schelle)
• Pennsylvania – Huntingdon Confirmed and Suspected Positive COVID‐19 Cases (presented by Bret Bucklen)
• Virginia Department of Corrections – Research During COVID-19 and Expanded Survey Questions (presented by Tama Celi)

2019 ICRN Conference Summary

The seventh annual data providers meeting took place at the Robert A. Young (RAY) Federal Building St. Louis, Missouri on September 19 and 20, 2019.

The meeting began with a presentation from Anne Precythe, director at the Missouri Department of Corrections, that highlighted the importance of using risk data to drive decision-making. Director Precythe’s presentation was followed by presentations on research studies using NCRP and CES data by Abt Associates, and the current and future plans at BJS by Danielle Kaeble (BJS). The introductory presentations were followed by plenary sessions on exploring recidivism and research in corrections. These sessions were followed by breakout sessions and small group discussions about topics including technical violations, classification assessments, restrictive housing, and NCRP data collection and results. Throughout the day, participants had the opportunity to informally network, discuss the dynamic relationship between criminal justice policy and data, and participate in discussions on research and performance management efforts across states.

The second day began with another round of breakout sessions on topics including BJS publications, parole supervision, the NCRP analysis tool, the National Institute of Justice Correction’s Portfolio and new Corrections Strategic Research Plan, hate crimes, and sex offenses, and educational strategies in corrections. Participants later had an opportunity to listen in on upcoming BJS initiatives. The meeting ended with a discussion of key points, lessons learned, and action steps for moving forward.

Read the full 2019 ICRN Meeting Summary

• Past, Present, and Future of NCRP Research (Ryan Kling, Abt Associates)
• The Impact of Recent and Cumulative Conditions of Confinement on Recidivism (Brian R. Kowalski, Ph. D., Bureau of Research and Evaluation, Ohio Department of Rehabilitation and Correction)
• Using Prison Days and Total Costs as an Alternative to Return to Prison Measures (Mark Harris, Ph. D., Research Manager, Wyoming Department of Corrections; Ethan Harris, MS, Statistics Instructor, Casper College)
• Explaining Sex Offender Recidivism: Accounting for differences in correctional supervision (Zach Baumgart, Carisa Bergner, and Megan Jones, Wisconsin Department of Corrections)
• Idaho Prison Compstat (Janeena White, Idaho Department of Correction)
• Pracademic Research: Engaging Corrections Practitioners in Research Through Innovation and Experimentation (Kristofer Bret Bucklen, Ph. D., Director of Planning, Research, and Statistics, Pennsylvania Department of Corrections)
• The Technicalities of Technical Violations (Clarissa Dias, Ph. D., Senior Research Analyst; Maria Stephenson, Research Development Manager; Georgia Department of Community Supervision)
• The Effect of Technical Violation Revocations on Serious Criminal Recidivism (Gerry Gaes, William Rhodes, Abt Associates)
• The Development and Validation of the Minnesota Severe and Frequent Estimate for Discipline (MnSafeD) (Grant Duwe, Director of Research and Evaluation, Minnesota Department of Corrections)
• NCRP for Newcomers (Tom Rich, Mike Shively, Abt Associates)
• The Path to Eliminating Restrictive Housing in Delaware (Philisa Weidlein-Crist, Lead Data Analyst; Miranda Mal, RNR Planner; Delaware Department of Correction)
• Location, Location, Location: What the NCRP tells us about where prisoners serve their sentence, where they’re from, and where they reoffend (Melissa Nadel, Ph. D.; Walter Campbell, Ph. D.; Abt Associates)
• Research Capacity within Department of Corrections and Local/Regional Jails (Dr. Hefang Lin, Research Statistician, Orange County Corrections; Tama S. Celi, Ph. D., Chief of Research, Policy, and Planning, Virginia Department of Corrections)
• Research Update from BJS: Source and Use of Firearms Involved in Crimes and Time Served in State Prison, 2016 (Danielle Kaeble, BJS Statistician)
• Massachusetts Parole Board: PRO Supervision (Gina Papagiorgakis, Director of Research and Planning, Massachusetts State Parole) 
• The New On-Line NCRP Data Analysis Tool (Tom Rich, Abt Associates)
• Corrections Research at NIJ: Providing Guidance in a Time of Change (Eric Martin, Social Science Analyst; David Mulhausen, Ph. D., Director of the National Institute of Justice; Chris Tillery, Office of Director, Office of Research, Evaluation, and Technology) 
• Hate Crimes and Female Sex Offenders: Exploring the NCRP’s Unique Capabilities for Research on Rare and Specialized Crime Types (Christopher Cutler, Melissa Nadel, Michael Shively, Ryan Kling; Abt Associates)
• Correctional Education Study Findings: FY2013 Releases (Tama Celi, Ph. D., Yan Jin, MS; Virginia Department of Corrections)

Stocks could face the steepest correction since the 2022 bear market as earnings kick off, analysts say. Here's what investors should watch for.

• Second quarter earnings season could trigger the most painful stock correction since 2022, according to NDR.
• The research firm warned of a shift from accelerating to decelerating growth in heading into 2025.
• "Another high beat rate may be required to justify the rally," analysts said.

Earnings season has officially kicked off this week, and it could bring the most painful correction for stock prices since the 2022 bear market.

That's according to Ned Davis Research, which offered a preview of what will matter most during the deluge of second-quarter earnings results over the next few weeks.

"The biggest risk could be a shift from accelerating to decelerating year/year growth toward the end of 2024 and into 2025," NDR strategist Ed Clissold said in a Thursday note.

That means that as strong as profit results might be this quarter, the future success of the stock market will largely hinge on company outlooks for the second half of the year.

Here's what investors should look out for during the second quarter earnings season, according to NDR.

Second-half growth estimates

The typical pathway of Wall Street earnings growth estimates is for them to be overly optimistic at the start of the year, only to slowly be revised lower towards the end of the year.

Therefore, it's not a matter of whether analysts will cut their second-half earnings growth estimates but rather by how much they will cut.

"Last year, the growth rate was revised down 4.8% points, much less than the long-term average of 8.1%. It is one of the reasons why the S&P 500 surged 24.2%. So far in 2024, consensus has only been revised down 1.3% points, again one of the reasons for the 18.1% year-to-date gain," Clissold said.

Current analyst projections suggest S&P 500 earnings growth of 5.7% in the second quarter, 19.2% in the third quarter, and 19.6% in the fourth quarter.

And those rosy growth estimates could ultimately be setting the stock market up for failure, especially considering expectations for a slowdown in the US economy's growth rate during the second half of this year.

Consensus earnings beats

Since the start of the now 18-month-old bull market, at least 78% of S&P 500 companies have exceeded consensus estimates, which is historically high.

That trend of breadth within company earnings beats will have to continue if the next inevitable stock market correction is to be pushed further down the road.

"Another high beat rate may be required to justify the rally," Clissold said. "Management teams have guided the Q2 year/year growth rate down to 5.7% from 7.0% at the end of May. The lowered bar makes a high beat rate more attainable."

Accelerating growth

"The concept that earnings growth is good for stocks seems intuitive. It is true, but with an important caveat. Investors look ahead, and they often view extremely strong year/year earnings growth as unsustainable," Clissold said.

With earnings growth surging in recent quarters, how sustainable that growth rate is remains a top question for investors, as decelerating growth is rarely rewarded with higher stock prices.

"Earnings are in the sharp acceleration phase, and consensus estimates are calling for them to remain there through Q3. During Q2 earnings season, watch for whether expected year/year EPS acceleration comes to fruition and for guidance on how long it can continue," Clissold said.

The Magnificent 7 stocks

Since the start of this bull market, much of the S&P 500's earnings growth has been driven by a handful of mega-cap tech companies like Nvidia , Amazon , and Meta Platforms.

"Five of the seven grew by at least 20% versus Q1 2023, and three grew by at least 100%," Clissold said of the mega-cap tech's earnings growth.

As strong as that growth has been, it sets a high bar for these companies to continue to post fast enough growth that impresses investors.

"The hurdle is high. Consensus is calling for five members of the Mag 7 to post slower growth rates in Q2 than in Q1. Even strong beats may not be enough for Mag 7 growth rates to continue to accelerate," Clissold said.

The other 493 stocks

For the bull market to continue, the other 493 S&P 500 stocks need to start pulling their weight in terms of earnings growth, and this earnings season could be the quarter it finally happens.

The 493 companies are expected to grow earnings by 1.1% in the second quarter, compared to first-quarter expectations of a 5.7% decline. These companies ultimately posted first-quarter earnings growth of 0.3%.

"Analysts are banking the Mag 7 to continue to drive earnings growth, but the rest of the market to participate more. The bar is noticeably lower outside the mega-cap favorites," Clissold said.

• Main content

Nvidia: Prepare For A Correction In H2 2025 Or H1 2026

• Nvidia's fundamental growth, driven by AI and data center demand, will likely slow in 2026 due to market saturation, including the completion of initial AI build-outs and double-ordering by customers.
• A potential price correction is likely in H2 2025 or H1 2026; competition and tech advancements may impact NVDA's long-term dominance.
• High valuation compared to peers; current Hold rating with potential Strong Buy opportunity in 2026-2027 as undervaluation emerges, opening up high long-term growth potential in robotics and automation.

I've covered Nvidia ( NASDAQ: NVDA ) twice before, and in my first analysis of the company, I was skeptical about the valuation—albeit, at that time, I arguably should have been bullish. In April 2024 , I provided follow-up coverage, and I rated the stock a Buy. Since then, the investment has gained ~60% in price. Now, I think the investment is certainly overvalued in the near term, and I think there is some potential for the stock to contract significantly in price in late fiscal 2025/early fiscal 2026. It is important for investors to begin to ascertain how the market is going to react once the company begins to report slower YoY growth, which Wall Street analysts have forecasted is likely to truly begin in 2026. It is somewhat speculative to ascertain whether the market will sustain the high valuation, but my own perspective is that NVDA is in for somewhat of a correction, and I think this could be relatively steep and could begin somewhere in H2 2025.

Please note that throughout this analysis, years always refer to fiscal periods, not calendar years.

Operational Analysis

Nvidia's surge over the last 12 months has largely been driven by its leadership in AI and data center technologies. The growth is anticipated by Wall Street and independent analysts to continue through 2025, but the consensus is a slowdown is likely in 2026. For fiscal year 2024 , NVDA reported 126% YoY revenue growth. Furthermore, in fiscal 2025 Q1 , the company reported 262% YoY revenue growth. This was largely driven by exponential demand for AI chips and data center solutions. For example, its data center revenue saw a 409% increase in 2023 and continued to grow exponentially into 2024. The company has secured major partnerships with Amazon ( AMZN ), Google ( GOOGL ) ( GOOG ), Meta ( META ), Microsoft ( MSFT ), and Apple ( AAPL ), who have collectively pledged to invest $200B in AI chips and data centers in 2024. This build-out, AI arms race and total attention on the field of AI has positioned Nvidia as one of the undoubtable lynchpins in the semiconductor supply chain, alongside TSMC ( TSM ). I don't see this changing, but in my estimation, there is likely to be a momentary slowdown circa 2026 as major tech companies complete phase 1 of the AI build-out. I think this is going to open up a significant moment of lowered sentiment in the market for NVDA. I also think this will be one of the best times to buy NVDA shares because I think over the next 5-10 years, big tech companies in the United States and the Western world could begin phase 2 of the AI build-out, which I think could involve tax incentives and federal spending support from the United States government among other leading public institutions.

In the near term, continued growth is set to resume for NVDA throughout 2025, with new products being launched, including the Blackwell platform, which supports generative AI at a trillion-parameter scale, and Spectrum-X technology for Ethernet-only data centers. Management has also committed to expanding its presence in autonomous vehicles and edge computing, which are expected to drive further revenue growth.

However, come 2026 the growth prospects could look less appealing as the firm achieves potential market saturation in the AI training market. As I mentioned, as companies complete their initial AI infrastructure setups, the demand for AI training hardware may start to reduce. There is also evidence that some of Nvidia's top customers have double-ordered to secure their near-term needs—once these infrastructure setups are complete, this double-ordering could lead to a significant cyclical downturn that the market has not anticipated fully yet. Furthermore, as I mentioned in my last thesis on Nvidia, while its moat is unlikely to be challenged significantly, emerging competitors like Groq, Cerebras and SambaNova continue to vie for market share through more efficient chip designs, and Amazon and Google are also developing in-house chips. In other words, the market is evolving and becoming more diverse, and I think it is worth remembering that these last few years have been notably lucky for Nvidia—it was largely in the right place at the right time with the right GPUs and infrastructure to support the largely unimaginable AI boom. Moving forward, new players and established tech giants will be trying to consolidate their positions, which further emphasizes why, over the next few years, the fundamental growth story for Nvidia is likely to change forever as phase 1, the AI infrastructure build-out, culminates.

Financial & Valuation Analysis

Nvidia has been nothing short of extraordinary on the financial front over the past couple of years:

However, analyst estimates for 2025 are already lower than the growth delivered in 2024. As we can also see from the following table, 2026 is a significant drop from 2025 in normalized EPS YoY growth if forecasts prove to be accurate. As it has been covered by almost 50 analysts for both years, I'm convinced that the forecasts are likely to come to fruition.

Seeking Alpha

What Nvidia shareholders need to remember is that the valuation is much higher than other big tech companies at the moment. This makes the investment notably more prone to volatility. For example, compare NVDA to the rest of the magnificent seven on normalized PE ratio:

Arguably, NVDA deserves this rich valuation, but it is also worth remembering that the stock is much more prone to volatility if it fails to deliver such high fundamental growth as it has in the past moving forward. I think because the normalized PE ratio is so high, even on a forward basis, where it reduces to 47, the likelihood of a correction is likely in H2 2025 in anticipation by the market of lower fundamental growth for NVDA for fiscal 2026, which is when Wall Street is forecasting much lower YoY EPS growth. While I understand the argument that NVDA has an immense amount of cash and ST investments and masses of free cash flow, giving it the capability to buy back shares aggressively, I think this will not be enough to keep investor sentiment high throughout 2026. However, share buybacks are a worthy strategy to please investors and bolster shareholder value until the next phase of AI spending begins, which I believe will be heavily oriented toward robotics and automation.

A lot of semiconductor companies are cyclical, and NVDA is proving to be no different as it continues to position itself at the epicenter of advanced computational capabilities. I think that the end of the first semiconductor upcycle related to AI and data centers will be in late 2025 and early 2026, and skepticism about high levels of AI spending is already apparent, for example, in a report from Goldman Sachs titled ' Gen AI: too much spend, too little benefit? '. Doubt is starting to surface about whether AI is worth the amount big tech companies and other industries are spending to develop and deploy its capabilities. This argument is warranted in the near term, but I think that once the spending settles, industries will begin to notice new potentials in the technologies, and I believe the focus will shift much more aggressively from LLMs, which are largely information-based tools, to robotics through AI-assisted automation. In my opinion, this is likely to be phase 2 of the AI build-out and NVDA shareholders who buy during the correction that I predict circa 2026 will be able to profit very well when new enthusiasm for intelligent technologies begins. In the interim, if buying at a low valuation, NVDA is likely to continue to keep growing amid support for information-based AI primarily, but I think the current valuation is too high to warrant an investment at this time. This is why my rating for NVDA is a Hold rather than a Buy, but I can see it becoming a Strong Buy at some point in 2026 or 2027.

By H2 2025 or H1 2026, I think the non-GAAP PE ratio could contract to around 40 on a TTM basis as the market reacts to lower growth prospects in the near future. If normalized EPS is $3.67 in January 2026 (in line with the Wall Street consensus), this would make the stock worth $146.80 in July 2026 if the market prices this into the stock approximately 6 months early. That indicates a 12-month CAGR of 16.17%. However, there is also the possibility that the market overreacts and opens up a more significant undervaluation, which I think is not unlikely considering how richly valued NVDA currently is. If the normalized PE ratio contracts to 35 on a TTM basis, the stock would be worth $128.45, meaning a 12-month CAGR of 1.65%. I think in both of these instances; the investment becomes a Strong Buy with a long-term horizon strategy hinged on the thesis of high growth coming in the future related to automation, robotics, and AI demand surging both domestically and potentially internationally through a re-globalized world order in conjunction with China. Please refer to my recent Palantir and S&P 500 analyses for why I think the West could continue to lead the global economy.

NVDA has a notably strong PEG ratio at the moment—currently just 0.09; but this is likely to come under pressure, and this is already evident in the discrepancy between the PEG GAAP TTM ratio and the PEG non-GAAP FWD ratio. I think that as this metric begins to indicate poorer growth performance on a forward basis, institutional investors are going to begin questioning the near-term valuation, and it could trigger heavy selling action down to the retail investor level. As a result of my analysis, I would be looking to hold off from investing until circa 2026, when hopefully the TTM normalized PE ratio is nearer to 40 or 35. Alternatively, if one is already an NVDA investor, I would certainly not sell, and I would hold this stock for the long term.

Risk Analysis

It is prudent to consider that the sentiment toward AI could shift toward the negative over the long term. The result of this shift would be very bad for NVDA. Although I think this is unlikely, what is perhaps more likely is that the infrastructure build-out for AI, automation, and robotics becomes much slower and elongated. I do not think this is the wisest strategy for Western companies to take, especially due to how bullish China's CCP is in supporting AI at the moment (I think it is paramount that the West hold the lead in technology during this time of geopolitical tension). However, it is conceivable that due to high costs and not enough near-term profits from these infrastructure expenses related to AI, companies stop investing in them heavily, at least for some time.

The enormous power requirements of AI data centers are currently straining existing grids . ClearBridge Investments notes that "The path of least resistance is one of higher power prices in markets with outsized AI infrastructure builds". This could slow down expansion plans, especially as we are concerned about net-zero policies and are yet to have a stable renewable energy infrastructure to meet such high demands.

Furthermore, Sam Altman, OpenAI's CEO, has mentioned that the industry might need as much as $7T in investment to build the necessary AI infrastructure. I think some caution around how the money is being spent and why is paramount, and I think there are likely to be delays in spending affecting NVDA's revenues due to a more prudent and measured allocation strategy related to intelligent technology moving forward. This point is also further reinforced by growing regulatory constraints , which I believe are likely to compound as the technologies become more proficient.

There are also long-term risks that could compound related to China that have already made themselves present. For example, Nvidia's business has already been affected by U.S. export controls on advanced AI chips to China. Huawei's Ascend chip is being positioned as a long-term solution for Chinese enterprises, and Nvidia is losing a big portion of the market if it is further inhibited from selling to China. There is both geographic and company diversification related to AI and semiconductors at the moment, as well as a growing notion of isolationism that is hinged on the geopolitical debate. In my opinion, both are somewhat bad for Nvidia over the long term as more companies begin to become viable alternatives to Nvidia, potentially at lower costs for specific workloads, and certain international markets become less accessible. In my opinion, these inhibitions are likely to get somewhat worse in the short term until the West has solidified a dominant lead and moat in AI infrastructure and capabilities, at which point I believe China is likely to be more dependent on the West for such capabilities and cross-border trade will have less friction and be more tenable. These are core reasons why I think bearish sentiment surrounding AI companies is going to build in the near term, but this will open up a significant buying opportunity for investors who capitalize on the lower valuations with a long-term horizon strategy.

In my opinion, NVDA is currently overvalued when taking into consideration the contraction in YoY growth rates expected in 2026. I think the market is likely to begin pricing this contraction in YoY growth into the stock in H2 2025. Therefore, I think investing at the present valuation is not wise, and I think it would be much better to allocate to NVDA during 2026/2027 when I think there will be a more bearish stance on AI as the infrastructure expenses do not prove as profitable in the near term as initially assumed. That being said, I think this sentiment will be transitory, and the potential undervaluation that evolves in 2026/2027 could make for a Strong Buy opportunity. I think over the next 5-10 years, NVDA will likely continue to be one of the biggest beneficiaries of AI growth and be a lynchpin in the advanced semiconductor supply chain, particularly related to robotics and automation capabilities. For these reasons, my rating for NVDA is a Hold for now.

This article was written by

Analyst’s Disclosure: I/we have a beneficial long position in the shares of AMZN, TSLA, GOOGL either through stock ownership, options, or other derivatives. I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it (other than from Seeking Alpha). I have no business relationship with any company whose stock is mentioned in this article.

Seeking Alpha's Disclosure: Past performance is no guarantee of future results. No recommendation or advice is being given as to whether any investment is suitable for a particular investor. Any views or opinions expressed above may not reflect those of Seeking Alpha as a whole. Seeking Alpha is not a licensed securities dealer, broker or US investment adviser or investment bank. Our analysts are third party authors that include both professional investors and individual investors who may not be licensed or certified by any institute or regulatory body.

Recommended For You

About nvda stock.

More on NVDA

Related stocks.

Trending Analysis

Trending news.

• Environment
• Science & Technology
• Business & Industry
• Health & Public Welfare
• Topics (CFR Indexing Terms)
• Public Inspection
• Presidential Documents
• Document Search
• Advanced Document Search
• Public Inspection Search
• Reader Aids Home
• Office of the Federal Register Announcements
• Using FederalRegister.Gov
• Understanding the Federal Register
• Recent Site Updates
• Federal Register & CFR Statistics
• Videos & Tutorials
• Developer Resources
• Government Policy and OFR Procedures
• Congressional Review
• My Clipboard
• My Comments
• My Subscriptions
• Sign In / Sign Up
• Site Feedback
• Search the Federal Register

The Federal Register

The daily journal of the united states government.

• Legal Status

This site displays a prototype of a “Web 2.0” version of the daily Federal Register. It is not an official legal edition of the Federal Register, and does not replace the official print version or the official electronic version on GPO’s govinfo.gov.

The documents posted on this site are XML renditions of published Federal Register documents. Each document posted on the site includes a link to the corresponding official PDF file on govinfo.gov. This prototype edition of the daily Federal Register on FederalRegister.gov will remain an unofficial informational resource until the Administrative Committee of the Federal Register (ACFR) issues a regulation granting it official legal status. For complete information about, and access to, our official publications and services, go to About the Federal Register on NARA's archives.gov.

The OFR/GPO partnership is committed to presenting accurate and reliable regulatory information on FederalRegister.gov with the objective of establishing the XML-based Federal Register as an ACFR-sanctioned publication in the future. While every effort has been made to ensure that the material on FederalRegister.gov is accurately displayed, consistent with the official SGML-based PDF version on govinfo.gov, those relying on it for legal research should verify their results against an official edition of the Federal Register. Until the ACFR grants it official status, the XML rendition of the daily Federal Register on FederalRegister.gov does not provide legal notice to the public or judicial notice to the courts.

Applications for New Awards; Fund for the Improvement of Postsecondary Education-Tribal Controlled Colleges or Universities (TCCUs) Research and Development Infrastructure (RDI) Grant Program

A Notice by the Education Department on 07/18/2024

Document Details

Information about this document as published in the Federal Register .

Document Statistics

Published document.

This document has been published in the Federal Register . Use the PDF linked in the document sidebar for the official electronic format.

Enhanced Content - Table of Contents

This table of contents is a navigational tool, processed from the headings within the legal text of Federal Register documents. This repetition of headings to form internal navigation links has no substantive legal effect.

FOR FURTHER INFORMATION CONTACT:

Supplementary information:, full text of announcement, i. funding opportunity description, ii. award information, iii. eligibility information, iv. application and submission information, v. application review information, vi. award administration information, vii. other information, enhanced content - submit public comment.

• This feature is not available for this document.

Enhanced Content - Read Public Comments

Enhanced content - sharing.

• Email this document to a friend

Enhanced Content - Document Print View

• Print this document

Enhanced Content - Document Tools

These tools are designed to help you understand the official document better and aid in comparing the online edition to the print edition.

These markup elements allow the user to see how the document follows the Document Drafting Handbook that agencies use to create their documents. These can be useful for better understanding how a document is structured but are not part of the published document itself.

Enhanced Content - Developer Tools

This document is available in the following developer friendly formats:.

• JSON: Normalized attributes and metadata
• XML: Original full text XML
• MODS: Government Publishing Office metadata

More information and documentation can be found in our developer tools pages .

Official Content

• View printed version (PDF)

This PDF is the current document as it appeared on Public Inspection on 07/17/2024 at 8:45 am. It was viewed 0 times while on Public Inspection.

If you are using public inspection listings for legal research, you should verify the contents of the documents against a final, official edition of the Federal Register. Only official editions of the Federal Register provide legal notice of publication to the public and judicial notice to the courts under 44 U.S.C. 1503 & 1507 . Learn more here .

Office of Postsecondary Education, Department of Education.

The Department of Education (Department) is issuing a notice inviting applications for new awards for fiscal year (FY) 2024 for the RDI grant program.

Applications Available: July 18, 2024.

Deadline for Transmittal of Applications: September 16, 2024.

Deadline for Intergovernmental Review: November 15, 2024.

For the addresses for obtaining and submitting an application, please refer to our Common Instructions for Applicants to Department of Education Discretionary Grant Programs, published in the Federal Register on December 7, 2022 ( 87 FR 75045 ) and available at www.federalregister.gov/​documents/​2022/​12/​07/​2022-26554/​common-instructions-for-applicants-to-department-of-education-discretionary-grant-programs .

Jason Cottrell, Ph.D., U.S. Department of Education, 400 Maryland Avenue SW, Room 5C122, Washington, DC 20202-4260. Telephone: (202) 453-7530. Email: [email protected] .

If you are deaf, hard of hearing, or have a speech disability and wish to access telecommunications relay services, please dial 7-1-1.

Purpose of Program: The RDI grant program is designed to provide Historically Black Colleges and Universities (HBCUs), TCCUs, and Minority-Serving Institutions (MSIs), including Asian American and Native American Pacific Islander Serving Institutions (AANAPISIs), Alaska Native and Native Hawaiian Serving Institutions (ANNH), Hispanic Serving Institutions (HSIs), Native American Serving Non-Tribal Institutions (NASNTIs), and/or Predominantly Black Institutions (PBIs), or consortia led by an eligible institution of higher education (institution), with funds to implement transformational investments in research infrastructure, including research productivity, faculty expertise, graduate programs, physical infrastructure, human capital development, and partnerships leading to increases in external funding.

For HBCUs and MSIs, the RDI grant program supports institutions in increasing their level of research activity in alignment with the Carnegie Classification designations. For TCCUs, which currently have their own Carnegie Classification, this program seeks to support an increase in research activities, undergraduate research opportunities, faculty development, research development, and infrastructure, including physical infrastructure and human capital development.

Assistance Listing Number: 84.116H.

OMB Control Number: 1894-0006.

Background: TCCUs provide access to a postsecondary education for many of the Nation's American Indian and Alaska Native students. In the fall of 2021, the 35 Title IV degree-granting TCCUs enrolled over 13,000, or 14 percent of, American Indian and Alaska Native undergraduate students. [ 1 ] Between July 2021 and June 2022, 20 of those TCCUs cumulatively conferred 380 bachelor's degrees to American Indian and Alaska Native students, representing 87.4 percent of all bachelor's degrees conferred by TCCUs. [ 2 ]

Because of their central role in educating American Indian and Alaska Native students, it is important for TCCUs to have the resources they need to excel in research activity. Teaching and research go hand in hand in ensuring student  [ 3 ] and institutional success. [ 4 ] Research activity can impact funding, faculty and student recruitment and retention, and student research opportunities, and promote diversity in graduate students and faculty at an institution.

TCCUs play a critical role in educating Native students and provide opportunities to produce research on American Indian issues from an American Indian and Alaska Native perspective. [ 5 ] According to the National Academies, data provided to their committee looking at MSIs and Science, Technology, Engineering, and Mathematics (STEM) showed that 93 percent of the students enrolled in STEM programs at four-year TCCUs in the fall of 2016 were Native American and Alaska Natives. [ 6 ]

However, TCCUs face obstacles in their efforts to sustain and implement extensive research activities. Administrations often have difficulty maintaining research activities due to the young nature of the institutions and Start Printed Page 58358 their lack of research support offices. [ 7 ] One study found that TCCUs' biggest obstacles in developing research activities are scheduling, infrastructure needs ( i.e., lack of space, equipment, and literature), partnership challenges ( i.e., lack of Tribal community knowledge), faculty capacity, and mistrust inside and outside of Tribal communities. [ 8 ] Additionally, recent events like the COVID-19 pandemic have further demonstrated and exacerbated barriers to improvement, including technology infrastructure, funding constraints ( i.e., long-term funding), [ 9 ] and isolation ( i.e., remote areas). [ 10 ] However, one study found that the potential benefits of research activities for faculty and student development—such as knowledge production and dissemination through conferences, collaborations, and presentations—may far outweigh the costs of overcoming these obstacles. For example, faculty have reported that research opportunities have allowed them to introduce to their classes new information that was not previously available. Additionally, many researchers emphasized that Tribal college research is “more culturally sensitive and community-grounded, both in the methods and in the results.”  [ 11 ] Therefore, we focus this competition on eligible TCCUs. In addition, the Department will make awards from unfunded applications submitted by HBCUs and MSIs from the FY2023 RDI program grant competition with the remaining FY2024 available funds.

Priorities: This notice contains one absolute priority which is from the notice of final priorities, requirements, and definitions for this program published elsewhere in this issue of the Federal Register (2024 NFP).

Absolute Priority: For FY 2024 and any subsequent year in which we make awards from the list of unfunded applications from this competition, this notice contains one absolute priority. Under 34 CFR 75.105(c)(3) , we consider only applications that meet this priority.

This priority is:

Funding for Tribal Controlled Colleges and Universities' Research and Development Infrastructure.

Projects proposed by TCCUs to improve their research and development activities, including infrastructure, faculty development, and academic programs.

Requirements: For FY 2024 and any subsequent year in which we make awards from the list of unfunded applications from this competition, the following requirements apply. The requirements are from the 2024 NFP.

Limitation on Grant Awards. The Department will only make awards to applicants that are not the individual or lead applicant in a current active grant from the RDI grant program.

Use of Funds: Grantees must conduct one or more of the following activities:

(1) Providing for the improvement of infrastructure existing on the date of the grant award, including deferred maintenance, or the establishment of new physical infrastructure, including instructional program spaces, laboratories, and research facilities relating to the fields of science, technology, engineering, the arts, mathematics, health, agriculture, education, medicine, law, and other disciplines.

(2) Hiring and retaining faculty, students, research-related staff, or other personnel, including research personnel skilled in operating, using, or applying technology, equipment, or devices to conduct or support research.

(3) Supporting research internships and fellowships for students, including undergraduate, graduate, and post-doctoral positions, which may include providing direct student financial assistance and other supports to such students.

Note: Under 20 U.S.C. 1138(d)(1) , funds made available under FIPSE may not be used to provide direct financial assistance in the form of grants or scholarships to students who do not meet eligibility criteria under Title IV of the Higher Education Act of 1965, as amended (HEA).

(4) Creating new, or expanding existing, academic positions, including internships, fellowships, and post-doctoral positions, in fields of research for which research and development infrastructure funds have been awarded to the grantee under this program.

(5) Creating and supporting inter- and intra-institutional research centers (including formal and informal communities of practice) in fields of research for which research and development infrastructure funds have been awarded to the grantee under this program, including hiring staff, purchasing supplies and equipment, and funding travel to relevant conferences and seminars to support the work of such centers.

(6) Building new institutional support structures and departments that help faculty learn about, and increase faculty and student access to, Federal research and development grant funds and non-Federal academic research grants.

(7) Building data and collaboration infrastructure so that early findings and research can be securely shared to facilitate peer review and other appropriate collaboration.

(8) Providing programs of study and courses in fields of research for which research and development infrastructure funds have been awarded to the grantee under this program.

(9) Paying operating and administrative expenses for, and coordinating project partnerships with members of, the consortium on behalf of which the eligible institution has received a grant under this program, provided that grantees may not pay for the expenses of any R1 institutions that are members of the consortia.

(10) Installing or extending the life and usability of basic systems and components of campus facilities related to research, including high-speed broadband internet infrastructure sufficient to support digital and technology-based learning.

(11) Expanding, remodeling, renovating, or altering biomedical and behavioral research facilities existing on the date of the grant award that received support under section 404I of the Public Health Service Act ( 42 U.S.C. 283k ).

(12) Acquiring and installing furniture, fixtures, and instructional research-related equipment and technology for academic instruction in campus facilities in fields of research for which research and development infrastructure funds have been awarded to the grantee under this program.

(13) Providing increased funding to programs that support research and development at the eligible institution that are funded by the National Institutes of Health, including through their Path to Excellence and Innovation program.

(14) Faculty professional development.

(15) Planning purposes.

Definition: The definition below applies to this competition and is from the 2024 NFP.

Tribal Controlled Colleges or Universities has the meaning ascribed it in section 316(b)(3) of the HEA.

Program Authority: 20 U.S.C. 1138-1138d . Start Printed Page 58359

Note: Projects will be awarded and must be operated in a manner consistent with the nondiscrimination requirements contained in Federal civil rights laws.

Applicable Regulations: (a) The Education Department General Administrative Regulations in 34 CFR parts 75 , 77 , 79 , 82 , 84 , 86 , 97 , 98 , and 99 . (b) The Office of Management and Budget (OMB) Guidelines to Agencies on Governmentwide Debarment and Suspension (Nonprocurement) in 2 CFR part 180 , as adopted and amended as regulations of the Department in 2 CFR part 3485 . (c) The Guidance for Federal Financial Assistance in 2 CFR part 200 , as adopted and amended as regulations of the Department in 2 CFR part 3474 . (d) The 2024 NFP.

Note: The Department will implement the provisions included in the OMB final rule, OMB Guidance for Federal Financial Assistance, which amends 2 CFR parts 25 , 170 , 175 , 176 , 180 , 182 , 183 , 184 , and 200 , on October 1, 2024. Grant applicants that anticipate a performance period start date on or after October 1, 2024 should follow the provisions stated in the OMB Guidance for Federal Financial Assistance ( 89 FR 30046 , April 22, 2024) when preparing an application. For more information about these updated regulations please visit: https://www.cfo.gov/​resources/​uniform-guidance/​ .

Type of Award: Discretionary grants.

Estimated Available Funds: $4,000,000.

Contingent upon the availability of funds and the quality of applications, we may make additional awards in subsequent years from the list of unfunded applications from this competition.

Estimated Average Size of Awards: $2,000,000.

Maximum Award Amount: $2,000,000 for a 48-month project period.

Estimated Number of Awards: 2.

Note: The Department is not bound by any estimates in this notice.

Project Period: Up to 48 months.

1. Eligible Applicants: Eligible applicants are TCCUs (as defined in this notice). Eligible applicants may apply individually or as lead applicants of a consortium with other eligible applicants and/or other partners such as an institution of higher education with an R1 Carnegie Classification, community colleges, or non-profit, industry, and philanthropic partners. The lead applicant must be an eligible applicant.

2. a. Matching Requirements and Exception: Grantees must provide a 1:1 match, which can include in-kind donations. The Secretary may waive the matching requirement on a case-by-case basis upon a showing of any of the following exceptional circumstances:

(i) The difficulty of raising matching funds for a program to serve an area with high rates of poverty in the lead applicant's geographic location, defined as a Census tract, a set of contiguous Census tracts, an American Indian Reservation, Oklahoma Tribal Statistical Area (as defined by the U.S. Census Bureau), Alaska Native Village Statistical Area or Alaska Native Regional Corporation Area, Native Hawaiian Homeland Area, or other Tribal land or county that has a poverty rate of at least 25 percent as determined every 5 years using American Community Survey 5-Year data;

(ii) Serving a significant population of students from low-income backgrounds at the lead applicant location, defined as at least 50 percent (or the eligibility threshold for the appropriate institutional sector available at https://www2.ed.gov/​about/​offices/​list/​ope/​idues/​eligibility.html ) of degree-seeking enrolled students receiving need-based grant aid under Title IV of the HEA;

(iii) Significant economic hardship as demonstrated by low average educational and general expenditures per full-time equivalent undergraduate student at the lead applicant institution, in comparison with the average educational and general expenditures per full-time equivalent undergraduate student of institutions that offer similar instruction without need of a waiver, as determined by the Secretary in accordance with the annual process for designation of eligible Titles III and V institutions; or

(iv) Information that otherwise demonstrates a commitment to the long-term sustainability of the applicant's projects, such as evidence of a consortium relationship with an R1 institution, a State bond, State matching, planning documents such as a campus plan, multi-year faculty hiring plan, support of industry, Federal grants received, or a demonstration of institutional commitment that may include commitment from the institution's board. (2024 NFP)

Note: Applicants seeking a waiver of the matching requirement must provide the waiver request information outlined above within their application.

b. Indirect Cost Rate Information: A grantee's indirect cost reimbursement is limited to 8 percent of a modified total direct cost base. For more information regarding indirect costs, or to obtain a negotiated indirect cost rate, please see www.ed.gov/​about/​offices/​list/​ocfo/​intro.html . (2024 NFP).

c. Administrative Cost Limitation: This program does not include any program-specific limitation on administrative expenses. All administrative expenses must be reasonable and necessary and conform to Cost Principles described in 2 CFR part 200 subpart E of the Guidance for Federal Financial Assistance.

3. Subgrantees: A grantee under this competition may not award subgrants to entities to directly carry out project activities described in its application.

4. Build America, Buy America Act: This program is subject to the Build America, Buy America Act ( Pub. L. 117-58 ) domestic sourcing requirements. Accordingly, under this program, grantees and their subrecipients (subgrantees) and contractors may not use their grant funds for infrastructure projects or activities ( e.g., construction, remodeling, and broadband infrastructure) unless—

(a) All iron and steel used in the infrastructure project or activity are produced in the United States;

(b) All manufactured products used in the infrastructure project or activity are produced in the United States; and

(c) All construction materials are manufactured in the United States.

Grantees may request waivers to these requirements by submitting a Build America, Buy America Act Waiver Request Form. For more information, including a link to the Waiver Request Form, see the Department's Build America Buy America Waiver website at: https://www2.ed.gov/​policy/​fund/​guid/​buy-america/​index.html .

1. Application Submission Instructions: Applicants are required to follow the Common Instructions for Applicants to Department of Education Discretionary Grant Programs, published in the Federal Register on December 7, 2022 ( 87 FR 75045 ), and available at https://www.federalregister.gov/​documents/​2022/​12/​07/​2022-26554/​common-instructions-for-applicants-to-department-of-education-discretionary-grant-programs , which contain requirements and information on how to submit an application.

2. Submission of Proprietary Information: Given the types of projects that may be proposed in applications for the RDI grant program, your application may include business information that you consider proprietary. In 34 CFR 5.11 we define “business information” and describe the process we use in Start Printed Page 58360 determining whether any of that information is proprietary and, thus, protected from disclosure under Exemption 4 of the Freedom of Information Act ( 5 U.S.C. 552 , as amended).

Because we plan to make successful applications available to the public, you may wish to request confidentiality of business information.

Consistent with Executive Order 12600 (Predisclosure Notification Procedures for Confidential Commercial Information), please designate in your application any information that you believe is exempt from disclosure under Exemption 4. In the appropriate Appendix section of your application, under “Other Attachments Form,” please list the page number or numbers on which we can find this information. For additional information please see 34 CFR 5.11(c) .

3. Intergovernmental Review: This competition is subject to Executive Order 12372 and the regulations in 34 CFR part 79 . Information about Intergovernmental Review of Federal Programs under Executive Order 12372 is in the application package for this program.

4. Funding Restrictions: We reference regulations outlining funding restrictions in the Applicable Regulations section of this notice. Additionally, no funds received by an institution of higher education under this section may be used to fund any activities or services provided by institutions that are not eligible as lead applicants in this competition.

5. Recommended Page Limit: The application narrative is where you, the applicant, address the selection criteria and the priority that reviewers use to evaluate your application. We recommend that you (1) limit the application narrative to no more than 50 pages and (2) use the following standards:

• A “page″ is 8.5″ x 11″, on one side only, with 1″ margins at the top, bottom, and both sides.
• Double-space (no more than three lines per vertical inch) all text in the application narrative, including titles, headings, footnotes, quotations, references, and captions, as well as all text in charts, tables, figures, and graphs.
• Use a font that is either 12 point or larger, and no smaller than 10-pitch (characters per inch).
• Use one of the following fonts: Times New Roman, Courier, Courier New, or Arial.

The recommended page limit does not apply to the cover sheet; the budget section, including the narrative budget justification; the assurances and certifications; the one-page abstract, the resumes, the bibliography, or the letters of support; or the waiver request for the matching requirement. However, the recommended 50-page limit does apply to all of the application narrative.

1. Selection Criteria: The selection criteria for this competition are from 34 CFR 75.210 . The points assigned to each criterion are indicated in the parentheses next to the criterion. An application may earn up to a total of 110 points based on the selection criteria. All applications will be evaluated based on the selection criteria as follows:

(a) Significance. (Maximum 25 points)

(1) The Secretary considers the significance of the proposed project.

(2) In determining the significance of the proposed project, the Secretary considers the following factors:

(i) The likelihood that the proposed project will result in system change or improvement. (up to 10 points)

(ii) The extent to which the proposed project involves the development or demonstration of promising new strategies that build on, or are alternatives to, existing strategies. (up to 5 points)

(iii) The importance or magnitude of the results or outcomes likely to be attained by the proposed project. (up to 10 points)

(b) Quality of the Project Design. (Maximum 30 points)

(1) The Secretary considers the quality of the project design.

(2) In determining the quality of the project design, the Secretary considers the following factors:

(i) The extent to which the goals, objectives, and outcomes to be achieved by the proposed project are clearly specified and measurable. (up to 5 points)

(ii) The extent to which the proposed activities constitute a coherent, sustained program of training in the field. (up to 5 points)

(iii) The extent to which the proposed project is designed to build capacity and yield results that will extend beyond the period of Federal financial assistance. (up to 5 points)

(iv) The extent to which the proposed project represents an exceptional approach to the priority or priorities established in the competition. (up to 5 points)

(v) The extent to which the proposed project will integrate with or build on similar or related efforts in order to improve relevant outcomes (as defined this notice), using nonpublic funds or resources. (up to 5 points)

(vi) The extent to which the proposed project will integrate with, or build on similar or related efforts, to improve relevant outcomes (as defined in this notice), using existing funding streams from other programs or policies supported by community, State, and Federal resources. (up to 5 points)

(c) Quality of Project Service s. (Maximum 15 points)

(1) The Secretary considers the quality of the services to be provided by the proposed project.

(2) In determining the quality of the services to be provided by the proposed project, the Secretary considers the quality and sufficiency of strategies for ensuring equal access and treatment for eligible project participants who are members of groups that have traditionally been underrepresented based on race, color, national origin, gender, age, or disability. (up to 5 points)

(3) In addition, the Secretary considers the following factors:

(i) The likely impact of the services to be provided by the proposed project on the intended recipients of those services. (up to 5 points)

(ii) The extent to which the technical assistance services to be provided by the proposed project involve the use of efficient strategies, including the use of technology, as appropriate, and the leveraging of non-project resources. (up to 5 points)

Note: For the purpose of this competition, technical assistance services could include, for example, technical assistance provided to faculty, staff, and students (at all levels) designed to increase research activities, including to expand institutional capacity to secure new funding, support student research experiences, or facilitate faculty professional development.

(d) Adequacy of Resources. (Maximum 15 points)

(1) The Secretary considers the adequacy of resources for the proposed project.

(2) In determining the adequacy of resources for the proposed project, the Secretary considers the following factors:

(i) The adequacy of support, including facilities, equipment, supplies, and other resources, from the applicant organization or the lead applicant organization. (up to 5 points)

(ii) The potential for the incorporation of project purposes, activities, or benefits into the ongoing program of the agency or organization at the end of Federal funding. (up to 5 points)

(iii) The potential for continued support of the project after Federal Start Printed Page 58361 funding ends, including, as appropriate, the demonstrated commitment of appropriate entities to such support. (up to 5 points)

(e) Quality of the Management Plan. (Maximum 10 points)

(1) The Secretary considers the quality of the management plan for the proposed project.

(2) In determining the quality of the management plan for the proposed project, the Secretary considers the following factors:

(i) The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks. (up to 5 points)

(ii) The adequacy of procedures for ensuring feedback and continuous improvement in the operation of the proposed project. (up to 5 points)

(f) Quality of the Project Evaluation. (Maximum 15 points)

(1) The Secretary considers the quality of the evaluation to be conducted of the proposed project.

(2) In determining the quality of the evaluation, the Secretary considers the following factors:

(i) The extent to which the methods of evaluation will provide timely guidance for quality assurance. (up to 5 points)

(ii) The extent to which the methods of evaluation will provide performance feedback and permit periodic assessment of progress toward achieving intended outcomes. (up to 5 points)

(iii) The extent to which the methods of evaluation include the use of objective performance measures that are clearly related to the intended outcomes of the project and will produce quantitative and qualitative data to the extent possible. (up to 5 points)

2. Review and Selection Process: We remind potential applicants that in reviewing applications in any discretionary grant competition, the Secretary may consider, under 34 CFR 75.217(d)(3) , the past performance of the applicant in carrying out a previous award, such as the applicant's use of funds, achievement of project objectives, and compliance with grant conditions. The Secretary may also consider whether the applicant failed to submit a timely performance report or submitted a report of unacceptable quality.

In addition, in making a competitive grant award, the Secretary requires various assurances, including those applicable to Federal civil rights laws that prohibit discrimination in programs or activities receiving Federal financial assistance from the Department ( 34 CFR 100.4 , 104.5 , 106.4 , 108.8 , and 110.23 ).

For this competition, a panel of three external reviewers will read, prepare a written evaluation of, and score all eligible applications using the selection criteria provided in this notice. The individual scores of the reviewers will be added and the sum divided by the number of reviewers to determine the peer review score. The Department may use more than one tier of reviews in evaluating applications. The Department will prepare a rank order of applications for the absolute priority based solely on the evaluation of their quality according to the selection criteria. The rank order of applications will be used to create a slate.

In the event there are two or more applications with the same final score in the rank order listing, and there are insufficient funds to fully support each of these applications, the Department will apply the following procedure to determine which application or applications will receive an award:

First Tiebreaker: The first tiebreaker will be the highest average score for the selection criterion titled “Adequacy of Resources.” If a tie remains, the second tiebreaker will be utilized.

Second Tiebreaker: The second tiebreaker will be the highest average score for the selection criterion titled “Significance.” If a tie remains, the third tiebreaker will be utilized.

Third Tiebreaker: The third tiebreaker will be the applicant with the highest percentage of Pell Grant students enrolled at the lead applicant institution based on the most recent IPEDS data available.

3. Risk Assessment and Specific Conditions: Consistent with 2 CFR 200.206 , before awarding grants under this competition, the Department conducts a review of the risks posed by applicants. Under 2 CFR 200.208 , the Secretary may impose specific conditions and, under 2 CFR 3474.10 , in appropriate circumstances, high-risk conditions on a grant if the applicant or grantee is not financially stable; has a history of unsatisfactory performance; has a financial or other management system that does not meet the standards in 2 CFR part 200, subpart D ; has not fulfilled the conditions of a prior grant; or is otherwise not responsible.

4. Integrity and Performance System: If you are selected under this competition to receive an award that over the course of the project period may exceed the simplified acquisition threshold (currently $250,000), under 2 CFR 200.206(a)(2) we must make a judgement about your integrity, business ethics, and record of performance under Federal awards—that is, the risk posed by you as an applicant—before we make an award. In doing so, we must consider any information about you that is in the integrity and performance system (currently referred to as the Federal Awardee Performance and Integrity Information System (FAPIIS)), accessible through the System for Award Management. You may review and comment on any information about yourself that a Federal agency previously entered and that is currently in FAPIIS.

Please note that, if the total value of your currently active grants, cooperative agreements, and procurement contracts from the Federal Government exceeds $10,000,000, the reporting requirements in 2 CFR part 200, appendix XII , require you to report certain integrity information to FAPIIS semiannually. Please review the requirements in 2 CFR part 200, appendix XII , if this grant plus all the other Federal funds you receive exceed $10,000,000.

5. In General: In accordance with the Guidance for Federal Financial Assistance located at 2 CFR part 200 , all applicable Federal laws, and relevant Executive guidance, the Department will review and consider applications for funding pursuant to this notice inviting applications in accordance with:

(a) Selecting recipients most likely to be successful in delivering results based on the program objectives through an objective process of evaluating Federal award applications ( 2 CFR 200.205 );

(b) Prohibiting the purchase of certain telecommunication and video surveillance services or equipment in alignment with section 889 of the National Defense Authorization Act of 2019 ( Pub. L. 115-232 ) ( 2 CFR 200.216 );

(c) Providing a preference, to the extent permitted by law, to maximize use of goods, products, and materials produced in the United States ( 2 CFR 200.322 ); and

(d) Terminating agreements in whole or in part to the greatest extent authorized by law if an award no longer effectuates the program goals or agency priorities ( 2 CFR 200.340 ).

1. Award Notices: If your application is successful, we notify your U.S. Representative and U.S. Senators and send you a Grant Award Notification (GAN); or we may send you an email containing a link to access an electronic version of your GAN. We also may notify you informally.

If your application is not evaluated or not selected for funding, we notify you. Start Printed Page 58362

2. Administrative and National Policy Requirements: We identify administrative and national policy requirements in the application package and reference these and other requirements in the Applicable Regulations section of this notice.

We reference the regulations outlining the terms and conditions of an award in the Applicable Regulations section of this notice and include these and other specific conditions in the GAN. The GAN also incorporates your approved application as part of your binding commitments under the grant.

3. Open Licensing Requirements: Unless an exception applies, if you are awarded a grant under this competition, you will be required to openly license to the public grant deliverables created in whole, or in part, with Department grant funds. When the deliverable consists of modifications to pre-existing works, the license extends only to those modifications that can be separately identified and only to the extent that open licensing is permitted under the terms of any licenses or other legal restrictions on the use of pre-existing works. Additionally, a grantee or subgrantee that is awarded competitive grant funds must have a plan to disseminate these public grant deliverables. This dissemination plan can be developed and submitted after your application has been reviewed and selected for funding. For additional information on the open licensing requirements please refer to 2 CFR 3474.20 .

4. Reporting: (a) If you apply for a grant under this competition, you must ensure that you have in place the necessary processes and systems to comply with the reporting requirements in 2 CFR part 170 should you receive funding under the competition. This does not apply if you have an exception under 2 CFR 170.110(b) .

(b) At the end of your project period, you must submit a final performance report, including financial information, as directed by the Secretary. If you receive a multiyear award, you must submit an annual performance report that provides the most current performance and financial expenditure information as directed by the Secretary under 34 CFR 75.118 . The Secretary may also require more frequent performance reports under 34 CFR 75.720(c) . For specific requirements on reporting, please go to www.ed.gov/​fund/​grant/​apply/​appforms/​appforms.html .

5. Performance Measures: For purposes of Department reporting under 34 CFR 75.110 , the Department will use the following program-level performance measures to evaluate the success of the RDI grant program:

(a) The annual research and development expenditures in:

(i) Science and engineering.

(ii) Non-science and engineering.

(b) Annual faculty development expenditures.

Accessible Format: On request to the program contact person listed under FOR FURTHER INFORMATION CONTACT , individuals with disabilities can obtain this document and a copy of the application package in an accessible format. The Department will provide the requestor with an accessible format that may include Rich Text Format (RTF) or text format (txt), a thumb drive, an MP3 file, braille, large print, audiotape, compact disc, or other accessible format.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register . You may access the official edition of the Federal Register and the Code of Federal Regulations at www.govinfo.gov . At this site you can view this document, as well as all other Department documents published in the Federal Register , in text or Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access Department documents published in the Federal Register by using the article search feature at www.federalregister.gov . Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department.

Nasser H. Paydar,

Assistant Secretary for Postsecondary Education.

1.  U.S. Department of Education, IPEDS, Fall Enrollment component.

2.  U.S. Department of Education, IPEDS, Completions component.

3.  NSSE. (n.d.). Digging Deeper Into the Quality of High-Impact Practices: HIPs Must be “Done Well” to Achieve Benefits.

4.  Rosowsky, D. (2022, March 2). The Role of Research at Universities: Why it Matters. In Forbes.com.

5.  Stull, G., Spyridakis, D., Gasman, M., Castro Samayoa, A., & Booker, Y. (2015). Redefining Success: How Tribal Colleges and Universities Build Nations, Strengthen Sovereignty, and Persevere Through Challenges.

6.  Espinosa, L.L., McGuire, K., Miles Jackson, L. (2019). Minority Serving Institutions: America's Underutilized Resource for Strengthening the STEM Workforce.

7.  Riley, E.T., Vadiee, N., & Ganguli, A. (2017). The Evolution of Research at Tribal Colleges and Universities. In Tribal College Journal, 29(2).

8.  Mortensen, M. (2001). Survey of Tribal Colleges Reveals Research's Benefits, Obstacles. In Tribal College Journal, 13(2).

9.  Redden, E. (2021, March 15). Trying Times for Tribal Colleges. In Inside Higher Ed.

10.  Stull, G., Spyridakis, D., Gasman, M., Castro Samayoa, A., & Booker, Y. (2015). Redefining Success: How Tribal Colleges and Universities Build Nations, Strengthen Sovereignty, and Persevere Through Challenges.

11.  Mortensen, M. (2001). Survey of Tribal Colleges Reveals Research's Benefits, Obstacles. In Tribal College Journal, 13(2).

[ FR Doc. 2024-15538 Filed 7-17-24; 8:45 am]

BILLING CODE 4000-01-P

• Executive Orders

Reader Aids

Information.

• About This Site
• Accessibility
• No Fear Act
• Continuity Information

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• Medical Devices
• Medical Devices News and Events
• Webinar - In Vitro Diagnostic Products (IVDs) - MDR Requirements, Correction and Removal Reporting Requirements, and Quality System Complaint Requirements - 08/22/2024

Webcast | Virtual

Event Title Webinar - In Vitro Diagnostic Products (IVDs) - MDR Requirements, Correction and Removal Reporting Requirements, and Quality System Complaint Requirements August 22, 2024

Webinar details, webinar materials.

On May 6, 2024, the FDA issued a final rule amending the FDA's regulations to make explicit that IVDs are devices under the Federal Food, Drug, and Cosmetic Act (FD&C Act) including when the manufacturer of the IVD is a laboratory. Along with this amendment, the FDA outlined a policy to phase out, over the course of four years, its general enforcement discretion approach to LDTs.

On August 22, 2024, the U.S. Food and Drug Administration (FDA) will host a webinar for laboratory manufacturers and other interested stakeholders to discuss how to comply with medical device reporting (MDR) requirements, correction and removal reporting requirements, and quality system (QS) requirements regarding complaint files beginning May 6, 2025 (Stage 1 of the phaseout policy).

Add this to my Calendar Outlook users: Click link, select Open, then click Save & Close

If you have questions that you wish to submit for possible discussion during the webinar, please email [email protected] . All questions must be received by July 22, 2024, to be considered for the discussion. Questions will be not be taken during the live webinar.

Registration is not necessary.

Date: August 22, 2024

Time: 1:00 PM – 2:30 PM

Please dial in 15 minutes before the start of the call to allow time to connect. 

We anticipate high attendance for this webinar and there is limited capacity. We encourage you to join early. However, due to the limited capacity we intend to post a recording and transcript as soon as possible following the webinar.

Please use the following link to join the webinar: https://fda.zoomgov.com/j/1616994355?pwd=cWZhS2RucTU4ZUNLbGF5ZFN5Wlo5dz09

Passcode: %KeTf9

Please note: Participants who join the webinar using the Zoom webinar link should use computer audio (listen through their computer speakers and speaking through computer microphone/headset). The dial-in information provided below is for participants who will be joining the webinar by phone only.

• +1 669 254 5252 US (San Jose)
• +1 646 964 1167 US (US Spanish Line)
• +1 646 828 7666 US (New York)
• +1 669 216 1590 US (San Jose)
• +1 415 449 4000 US (US Spanish Line)
• +1 551 285 1373 US (New Jersey)
• International Callers Dial: Please check the international numbers available
• Webinar ID: 161 699 4355
• Passcode: 066992

The presentation, printable slides, and transcript will be available on this webpage and at CDRH Learn under "In Vitro Diagnostics."

If you have questions about this webinar, please contact CDRH's Division of Industry and Consumer Education (DICE) at [email protected] , 1-800-638-2041, or 301-796-7100.