2024 ISMIR Open Review Survey: A Brief Summary of Results

ISMIR conference proceedings are the primary vehicle for disseminating research in the community, with over 100 papers published annually in recent years. Submissions undergo double-blind reviews, reviewer discussions, meta-reviewer summarization and recommendation, and finally Scientific Program Committee Chair decisions.

In recent years, some research communities have been adopting “open peer review” practices, such as publishing reviews for accepted papers or all papers, publishing discussions between authors and reviewers, or allowing anyone to post comments at any time. A 2017 survey study on open peer review elicited attitudes from reviewers and authors in different fields on a broad range of open peer review configurations.

Within the MIR community, a working group (including a survey subgroup and a webinar subgroup) dedicated to the ISMIR review processes has worked with the ISMIR Board, ISMIR 2024 General Chairs, ISMIR 2024 Scientific Program Committee Chairs, and other community members on this topic. During ISMIR 2023, the survey subgroup released an initial survey on ISMIR open peer review to conference participants and shared results during the Society Meeting. This was followed by in-depth discussions during the Society Meeting and Unconference session, a webinar hosted by the webinar subgroup, and a revision of the initial survey. The revised survey was released to the ISMIR-Community Google Group between March 23 and April 22, 2024.

In this blog post, the survey subgroup reports initial findings from the survey (N=116 responses). These findings represent a subset of 36 total questions, including the last open-ended question. Pie charts and histograms of multiple-choice responses were generated automatically by the Google survey. Answers to the open-ended question were aggregated and summarized with the help of ChatGPT. 

• More than 80% of survey takers prefer some kind of open peer review for ISMIR. The top 3 opinions are:
  • 28.4% favors publishing peer reviews (anonymous) and review discussions (anonymous) for accepted papers
  • 24.1% favors publishing peer reviews (anonymous) and review discussions (anonymous) for accepted and rejected papers
  • 14.7% favors publishing peer reviews (anonymous) for accepted papers
  • 5.2% favors publishing peer reviews (anonymous) for accepted and rejected papers
  • 5.2% favors publishing peer reviews (with reviewer names) for accepted papers
  • 1.7% favors publishing peer reviews (with reviewer names) for accepted and rejected papers
  • 0.9% favors publishing peer reviews (anonymous) and review discussions (anonymous) for accepted and rejected papers. Reject papers' authors remain anonymous.
  • 0.9% suggests publishing peer reviews and discussions (with optional reviewer names) for a chosen set of accepted papers. 
• 0.9% chooses “It is not clear to me yet whether ISMIRshould adopt open review. I would like to see evidence of the benefits of openreview before making a decision”
• 18.1% of survey takers think that ISMIR should not adopt any form of open review.

Support for Open Peer Review

Enhanced Motivation and Quality

• Publishing outstanding reviews could motivate reviewers to enhance their review quality.
• Reviewers might be incentivized to provide more constructive feedback if reviews are public.
• Open reviews could facilitate more constructive criticism.

Mixed Opinions

Need for Evidence

• Some respondents call for evidence of the benefits of open review before making any changes.
• There is a desire for data and research to support any decision towards open peer review.

Suggestions for Adopting Open Review Practices

Step-by-Step Implementation

• A gradual implementation (e.g., starting with publishing meta-reviews) could allow for evaluation and adjustments.
• Educational materials and guidelines for reviewing could support a smooth transition.
• Suggestions to publish anonymized reviews for accepted papers as a reasonable first step.

Rebuttal Opportunities

• Authors having the opportunity to respond to reviews can improve the overall process.

Concerns and Reservations

Negative Impact on Submission

• Concerns that open peer review might discourage submissions due to the fear of public criticism.
• The potential negative impact on the reputation of ISMIR if errors in reviews are made public.

Impact on Junior Researchers

• Open reviews could expose junior researchers to critical feedback publicly, potentially discouraging them.
• There is a risk that critical feedback might harm their development and willingness to submit.

Introducing Bias

• Concern that public reviews could introduce biases, with critical reviews negatively impacting the perception of papers and authors.

Implementation Challenges

• The effectiveness of open review depends heavily on proper implementation and the community's context.

Doubts about Benefits and Appropriateness for MIR

• Doubts about whether open review would actually improve the review quality compared to the current double-blind process, which is seen as effective.
• Concerns about creating new problems.
• Arguments that open review is better suited for larger communities with different contexts.

Other Approaches are Better

• Improving paper quality and better supervision of students should be the focus rather than changing the review system.
• Implementing mechanisms for feedback on reviews or incentivizing high-quality reviews without making them public could be beneficial.

While the number of responses is relatively small, the survey takers come from diverse career stages and have various levels of experience with ISMIR paper submission and open review practices. The majority of survey takers favor some kind of open review practices for ISMIR, while a fair amount of survey takers object to the idea of open review. Answers to the open-ended questions elaborate various justifications on both sides, and the main suggestion at this time for exploring open review is to try it out with a small step which allows for evaluation and adjustments.