The ACM Special Interest Group Governing Board Replication Taskforce.
- Behjat, Laleh
- Childers, Bruce
- Eide, Eric (SIGOPS)
- Fursin, Grigori
- Gil, Yolanda (SIGAI)
- Harper, Simon (Taskforce Chair)
- Jones, Alex K.
- Krishnamurthi, Shriram (SIGPLAN)
- Lin, Jimmy (SIGIR)
- Madden, Patrick (SGB Chair)
- McGeoch, Catherine
- Taufer, Michela (SIGHPC)
Replication and independent verification of results are commonplace in many scientific fields. These practices help avoid misunderstandings, provide greater confidence in reported results, and enables the detection of both unintentional error and deliberate fraud. By comparison with fields such as physics, chemistry, and medicine, computer science is clearly lagging in this regard; replication of experiments is rare, and activities such as paper correction or retraction are virtually unheard of.
The charge of the task force is to develop proposals on how ACM can bring current replication and verification practices in line with the rest of the scientific community. Specifically, the following questions are of interest.
- Across the SIGs, what mechanisms are in place to enable replication and verification efforts? Are they sufficient to catch errors or deliberate fraud (and have they in fact done so?). Can these processes be improved or encouraged in some way?
- What are reasonable expectations from authors, with regards to enabling replication and verification? How can these expectations be made compatible with the review process in conferences, symposia, journals? Should there be different expectations during review, and after paper acceptance?
- If published work fails to be replicated, appears to contain significant errors, or even appears to be fraudulent — what mechanisms should be in place to investigate the matter? Who would such concerns be communicated to? Who would perform investigation, and decide the matter? And if errors, deception, or fraud are confirmed — what should the ACM, as a publisher, do? Should there be corrections, annotations noting the errors, paper retraction?
The SIGs, with elected leaders representing the various research communities of computer science, are a logical group to engage in setting policy. The SIG leaders will be familiar with typical practices within their areas of expertise. With control of many of the major publishing venues, this group also has the ability to enact change.
Given the range of research topics and problems considered, there will not be a one-size-fits-all policy. By engaging many SIGs in this effort, a degree of consistency can be achieved across the ACM.
With this in mind this group will be open to all voices across the SGB and will compose SGB members and other invited experts. The document is also intended to evolve (hence the versioning here, and the version releases) and be transparent.
If you would like to make changes (after discussions) please do. Send me changes otherwise. Create changes and start discussions via the issue tracker. Free form updates via the wiki (public).