USENIX Security '24 Preliminary Call for Artifacts

Artifact Evaluation Information

Overview

A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the availability and reproducibility of experimental results, USENIX Security runs an optional AE. The AEC will review each submitted artifact and also grant Distinguished Artifact Awards to outstanding artifacts accepted to USENIX Security '24.

Process

To maintain a wall of separation between paper review and the artifacts, authors will be given the option to submit their artifacts only after their papers have been accepted for publication at USENIX Security. The artifact registration deadline is around the camera-ready deadline, while the full artifact submission is about two weeks afterward. By the artifact submission deadline, authors can submit their artifacts, Artifact Appendix, and other supporting information of their accepted USENIX Security 2024 paper via the submission form, which will be available here soon, using the provided submission instructions.

Authors define the contents of their artifact submission. For example, software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. Authors choose which badges their artifact should be evaluated towards (one, two, or all three of the following Artifacts Available, Artifacts Functional, and Results Reproduced. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, well documented, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.

Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members are instructed to treat the artifact confidential during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, exploits under embargo, etc., in your artifact. Since we anticipate small glitches with installation and use, reviewers may communicate with authors for a period of up to five weeks after the artifact submission deadline to help resolve glitches while preserving reviewer anonymity.

The AEC will then complete its evaluation and notify the authors of the outcome around six weeks after the artifact submission deadline. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.

Acknowledgements

The AE process at USENIX Security '24 is a continuation of the AE process at USENIX Security '20–'23 and was inspired by multiple other conferences, such as OSDI, EuroSys, and several other systems conferences. See artifact-eval.org for the origins of the AE process.