The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade.
Seven reproducibility variables were identified, quality-assured and used to analyse 56,800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64%. Building on empirical reproducibility rates from a prior study, we estimate – inferred from documentation practices, not direct testing – that reproducibility increased from 28% in 2014 to 64% in 2024.
Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements. Many scientific results cannot be reliably reproduced1, and therefore, conclusions cannot be trusted2. This phenomenon is called the reproducibility crisis3 and has been described in a number of studies covering a broad range of disciplines, including psychology4,5,6, economics7, medicine8,9,10, neuroscience11, and genetics12.
Reproducibility concerns also gained prominence in artificial intelligence (AI)13,14, where rapid growth in the field and increasing social importance have intensified the urgency to mitigate the problem. One way to mitigate the reproducibility crisis is to provide detailed descriptions of the research alongside open code and data, and therefore, a cultural shift toward open science15 has emerged, advocating for transparent and accessible knowledge. As a large portion of empirical research in computer science and AI is conducted on computers and experiments are described in code, sharing detailed descriptions of experiments should, in principle, be straightforward16.
Achieving this relies critically on high-quality documentation of experimental methods and artifacts, such as code, data, and workflows17,18,19,20. This has again led communities and venues to demand stronger requirements from authors, reflected in the introduction of reproducibility checklists, for example, at NeurIPS21 and JAIR22. However, the effect of these procedural and communal changes remains difficult to measure, especially at scale, due to the manual labour required to reproduce studies and the tenfold increase in published studies in the past decade, see Fig. 1.
It is widely acknowledged that empirical AI research can fail in subtle ways. Nondeterminism due to variability between execution (or computing) environments can cause results to vary so much that it affects reproducibility23,24,25,26,27. Poor data treatment, including poorly documented datasets, dataset splits, and data leakage, also poses challenges to reproducibility28,29,30.
Extract — continue reading at the source.
