Sponsored by USENIX, the Advanced Computing Systems Association.
All authors of accepted USENIX Security '24 papers (including shepherd-approved papers) are encouraged to submit artifacts for Artifact Evaluation (AE). Artifacts can be submitted in the same cycle as the accepted paper or in any of the following cycles for 2024. Each submitted artifact will be reviewed by the Artifact Evaluation Committee (AEC). Before submitting your artifact, please check the Artifact Evaluation Information below. Should you have any questions or concerns, you can reach the AEC chairs at firstname.lastname@example.org.
- Notification to authors: Friday, September 1, 2023, 11:59 pm AoE
- Artifact Registration deadline: Friday, October 6, 2023
- Final papers due: Tuesday, October 10, 2023
- Artifact submission deadline: Tuesday, October 24, 2023
- AE bidding: Wednesday, October 25 to Friday, October 27, 2023
- Author discussion period: Monday, October 30 to Wednesday, November 22, 2023
- Reviews due: Tuesday, November 28, 2023
- Artifact decisions announced: Tuesday, December 5, 2023
- Final appendix files due: Tuesday, December 12, 2023
- Notification to authors: Thursday, February 1, 2024, 11:59 pm AoE
- Artifact Registration deadline: Friday, March 1, 2024
- Final papers due: Tuesday, March 5, 2024
- Artifact submission deadline: Tuesday, March 19, 2024
- AE bidding: Wednesday, March 20 to Friday, March 22, 2024
- Author discussion period: Monday, March 25 to Friday, April 19, 2024
- Reviews due: Thursday, April 25, 2024
- Artifact decisions announced: Tuesday, April 30, 2024
- Final appendix files due: Tuesday, May 7, 2024
- Notification to authors: Wednesday, May 8, 2024, 11:59 pm AoE
- Artifact Registration deadline: Monday, June 10, 2024
- Final papers due: Thursday, June 13, 2024
- Artifact submission deadline: Friday, June 21, 2024
- AE bidding: Monday, June 24 to Wednesday, June 26, 2024
- Author discussion period: Thursday, June 27 to Wednesday, July 24, 2024
- Reviews due: Friday, July 26, 2024
- Artifact decisions announced: Friday, August 2, 2024
- Final appendix files due: Friday, August 9, 2024
Artifact Evaluation Committee
Artifact Evaluation Committee Co-Chairs
Phani Vadrevu, Louisiana State University
Anjo Vahldiek-Oberwagner, Intel Labs
Artifact Evaluation Publication Chair
Victor Le Pochat, KU Leuven
Artifact Evaluation Committee
We’re in the process of recruiting the AEC. Please (self-)nominate in case you are interested using our form.
Artifact Evaluation Information
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the availability and reproducibility of experimental results, USENIX Security runs an optional AE. The AEC will review each submitted artifact and also grant Distinguished Artifact Awards to outstanding artifacts accepted to USENIX Security '24.
To maintain a wall of separation between paper review and the artifacts, authors will be given the option to submit their artifacts only after their papers have been accepted for publication at USENIX Security. The artifact registration deadline is around the camera-ready deadline, while the full artifact submission is about two weeks afterward. By the artifact submission deadline, authors can submit their artifacts, Artifact Appendix, and other supporting information of their accepted USENIX Security 2024 paper via the submission form, which will be available here soon, using the provided submission instructions.
Authors define the contents of their artifact submission. For example, software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. Authors choose which badges their artifact should be evaluated towards (one, two, or all three of the following Artifacts Available, Artifacts Functional, and Results Reproduced. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, well documented, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.
Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members are instructed to treat the artifact confidential during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, exploits under embargo, etc., in your artifact. Since we anticipate small glitches with installation and use, reviewers may communicate with authors for a period of up to five weeks after the artifact submission deadline to help resolve glitches while preserving reviewer anonymity.
The AEC will then complete its evaluation and notify the authors of the outcome around six weeks after the artifact submission deadline. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.
The AE process at USENIX Security '24 is a continuation of the AE process at USENIX Security '20–'23 and was inspired by multiple other conferences, such as OSDI, EuroSys, and several other systems conferences. See artifact-eval.org for the origins of the AE process.