Reviewing Guidelines

These reviewing guidelines have been borrowed from a number of conferences, including the ACM CCS 2022, SIGMOD 2021 and others.

When you prepare your reviews, please consider that they serve two critical purposes: (1) To provide authors with feedback on how to improve their paper and proceed with their work, regardless of the paper decision; (2) To justify and document final decisions regarding publication of this version of the paper.

Regardless of whether a review marks a paper as accept or reject, it should be thoughtful, civil, and provide constructive feedback.

Novelty can manifest in many ways that go beyond the traditional requirement of having a new algorithm or new system design. When reviewing papers, we would like to consider many additional aspects of novelty such as having a novel result, novel engineering, or a novel benchmark. That is, even if the algorithms used and the problem itself are not dramatically new, a paper might still be providing new and useful insights.

Papers that have not gone through peer-review and are not formally published can be brought up if a reviewer feels it is relevant to inform the authors, but such papers cannot be considered in a way that diminishes the novelty of the submitted work or in order to ask the authors to compare explicitly against them, e.g., with experiments. This includes papers on arXiv and white papers.

Review template

Please follow this structure:

Review length

A good review should cover at least the following aspects of the paper in detail: problem focus and motivation, technical contributions, experimental evaluation, related work, and overall presentation. Please elaborate with regards to strong and weak points on all these fronts with detailed comments. We expect an ideal review to span at least 4-5 paragraphs, discussing these aspects. A very short review is very likely a bad review. Recall, that one of the purposes of your review is to give authors constructive feedback, and a short review does not accomplish that.

Critiquing the paper’s motivation and focus

A good review should evaluate the importance and relevance of the problem that the authors list. The authors may not agree with your conclusion, but it is vital that they understand your rationale. For this reason, criticisms on the motivation of the problem should be clearly explained and justified.

Examples to avoid:

More constructive:

Critiquing the technical contributions

A big part of your review is to evaluate the technical contributions of the work. Examine the claims made in the paper, judge if they are applicable to the problem, and evaluate their expected importance, applicability, and usefulness. If you believe that some assumptions of the paper are too restrictive and make the problem unrealistic or narrow, explain that. However, you should avoid tainting your evaluation with personal opinions. Judge the work that the authors are presenting, rather than the work that you would have preferred to see.

Examples to avoid:

It is great to bring up additional contributions that you would like to see and ideas that could extend the presented work. But write them as suggestions, rather than a basis of criticism. Your review should judge the contributions that the paper makes, rather than the contributions it does not make.

More constructive:

Critiquing the evaluation

Your review should evaluate whether the experiments validate the claims of the paper. If the paper claims that the proposed algorithm improves on the state-of-the-art, discuss whether the experiments compare the proposed approach with the appropriate methods, and against all expected metrics (e.g., efficiency, accuracy, scalability). Your review should note whether there are any inconsistencies in the results and whether there are observations that are not explained, justified, or discussed. Also judge whether the experimental methods and datasets are appropriate, or if they introduce biases in the results.

Examples to avoid:

More constructive:

In addition, as you consider such remarks, please consider space restrictions. For example, are there any of the existing experiments that you think are less useful and can be dropped?

Critiquing the writing and presentation

You may often find the presentation of a paper unsatisfactory. Please list specific points of possible improvement such as repetitiveness, poor structure, confusing notation, missing examples, or lack of formalism. However, make sure that you provide concrete examples and pointers for your criticisms. For example, explain what notation confused you, which terms were not properly defined, and which statements should be made more rigorous. Communicating vague discontent about the readability of the paper is not constructive, because it doesn’t explain to the authors how they can improve it.

Examples to avoid:

More constructive:

You should ensure that the paper includes sufficient discussion of the related work, explains similarities and differences, and does not omit obvious connections. Keep in mind however that, unless you are reviewing a survey paper, there will always be some citations that could have been added but weren’t. Do not criticize a paper for the omission of a particular citation if it already includes sufficient discussion of the related work in the particular area. Do point out though glaring omissions and important connections that were missed.

Examples to avoid:

More constructive:

Inclusive language

In keeping with efforts on Diversity and Inclusion in our conference and community, please check the paper for language that may further the marginalization, stereotyping, or erasure of any group of people, especially historically marginalized and/or under-represented groups (URGs) in computing. Authors have been given detailed guidelines and examples on this front. Please read this page if you have not already done so.

If you find exclusionary or hurtful language or examples (even if unintentional), point them out in your review and ensure the authors change the text accordingly. Please also be mindful of using inclusive language in your own review text, as well as during the discussion phase.