Four researchers built a tool to test the citations in a large number of scholarly articles. In this study, they used it to show how many fabricated citations they found in a random sample of 500,000 peer-reviewed articles.
https://www.pnas.org/doi/10.1073/pnas.2623869123
I'm interested in the frequency of fake citations. But right now I'm more interested in the fact that the tool is not #OpenSource. One good reason is that it depends in part on #Claude. But the authors did not share their Claude prompts or any of the non-Claude code they wrote to supplement Claude.
However, they do make three gestures toward openness here. (1) They wrote analysis code to help crunch their data, and they're willing to share that analysis code on request. (2) They offer to share the tool code with reviewers and editors on request, presumably to aid in the peer-review of their article. But they don't make the same offer to researchers who want to test the tool or use it for new studies. (3) In Appendix 01 they give pseudocode for the citation-testing tool, but not the actual code.
Motivated researchers could use the pseudocode to create their own tool. But it would likely differ in details from the tool used here, blocking attempts at precise replication.
More important, the lack of open code for testing citations prevents the critique and improvement of that code, and slows the evolution of standardized methods for testing citations.
#AI #Citations #Hallucinations #Metaresearch #OpenScience #ScholComm