A good author response should help the reviewers understand the work, as submitted. In practice, it is rare that a response will successfully win a philosophical argument with the reviewers (e.g., if the reviewers simply do not find a result interesting). A more useful goal for a response is to increase confidence that the results presented in the paper are sound. For instance, a reviewer may be concerned about the correctness of a design or experiment because the paper simply omits a relevant detail, perhaps for space. A helpful response would present the missing information needed for the reviewer to evaluate correctness of the design or experiment.
Some key items to consider include:
- Focus first on the questions raised in a comment by the reviewers collectively (if present), followed by specific questions raised in individual reviews, and any major factual misunderstandings. The reviewers will be asked to identify the questions that have the most bearing on their decision. You are not expected to respond to every point in the reviews.
- Factual errors: Although uncommon, reviewers do occasionally misunderstand a fact critical to a paper. For instance, a reviewer might be mistaken about the configuration of a test system that has a first-order impact on the experimental results. In this case, a good response would politely clarify this detail, and, if present, remind the reviewers where in the submission this detail was presented (or apologize for the omission). From there, you may wish to summarize the implications of this detail on the understanding of the results.
- Be "prefix optimal": put the most important clarifications first (generally what is explicitly requested in the reviews or from the most reviewers), and more minor points, or points that seem important to only one reviewer, toward the end. Reviewers may not read the entire response (past the initial 1500 words), and can search a longer response for their particular concern.
- Length is a guideline: It is critical to answer the key questions in the first 1500 words, as reviewers are not required to read past that point, But if more space is required to clearly answer the questions, it is advisable to include this information in the area beyond the first 1500 words. Of course, brevity is still appreciated by the reviewers. Similarly, it can be helpful to write very short, bullet-style responses to minor questions that can be answered briefly (and easily searched by reviewers). However, one can also write too much: sometimes a long or careless response may reveal new concerns that harm the paper's chances of acceptance.
- Make a longer response easily searchable, with reviewer IDs and keywords: Reviewers may wish to search (i.e., Ctrl+F) for a specific point during discussion. Help the reviewers advocate for your work.
- New data: It is ok to include new data in your response if the data directly addresses a concern raised in the reviews. For instance, in a log-structured file system, a reviewer may be concerned about how long the log cleaner ran during the experiments. It would be ok to report this data for experiments already in the paper, or perhaps include an additional stress test.
- The reason most response instructions prohibit new data is to avoid changing the scope of the submission at the end of the review process. For instance, using the response to report new optimizations and improved results is inappropriate, as the reviewers have not had sufficient time to review the new material.
- Additional good advice can be found from ASPLOS '21 (note that response length and some other policies may differ from FAST '22)