Sponsored by USENIX, the Advanced Computing Systems Association.
All authors of accepted USENIX Security '23 papers (including shepherd approved, but not major revisions) are encouraged to submit artifacts for Artifact Evaluation (AE). Artifacts can be submitted in the same cycle as the accepted paper or in any of the following cycles for 2023. Each submitted artifact will be reviewed by the Artifact Evaluation Committee (AEC). Before submitting your artifact, please check the Artifact Evaluation Information below. Should you have any questions or concerns, you can reach the AEC chairs at sec23aec@usenix.org.
Important Dates
Summer Deadline
- Notification to authors: Friday, September 2, 2022, 11:59 pm AoE
- Artifact Registration deadline: Friday, September 30, 2022
- Final papers due: Tuesday, October 4, 2022
- Artifact submission deadline: Tuesday, October 11, 2022
- Answering AE reviewer questions: Monday, October 17 to Friday, November 11, 2022
- Artifact decisions announced: Tuesday, November 22, 2022
- Final appendix files due: Tuesday, November 29, 2022
Fall Deadline
- Notification to authors: Friday, January 27, 2023, 11:59 pm AoE
- Artifact registration deadline: Friday, February 24, 2023
- Final paper files due: Tuesday, February 28, 2023
- Artifact submission deadline: Tuesday, March 7, 2023
- Answering AE reviewer questions: Monday, March 13 to Friday, April 7, 2023
- Artifact decisions announced: Tuesday, April 18, 2023
- Final appendix files due: Tuesday, April 25, 2023
Winter Deadline
- Notification to authors: Monday, May 8, 2023
- Artifact registration deadline: Friday, June 9, 2023
- Final paper files due: Tuesday, June 13, 2023
- Artifact submission deadline: Wednesday, June 20, 2023
- Answering AE reviewer questions: Monday, June 26 to Friday, July 21, 2023
- Artifact decisions announced: Tuesday, August 1, 2023
- Final appendix files due: Tuesday, August 8, 2023
Artifact Evaluation Committee
Artifact Evaluation Committee Co-Chairs
Cristiano Giuffrida, Vrije Universiteit Amsterdam
Anjo Vahldiek-Oberwagner, Intel Labs
Artifact Evaluation Publication Chair
Alexios Voulimeneas, KU Leuven
Artifact Evaluation Committee
Alessandro Baccarini, University at Buffalo, SUNY
Alexander Warnecke, TU Braunschweig
Amel Bourdoucen, Aalto University
Amit Seal Ami, College of William & Mary
Anunay Kulshrestha, Princeton University
Avinash Sudhodanan, Meta
Charles Babu M, CEA LIST, Université Paris-Saclay
Charlie Jacomme, Inria Paris
Christof Ferreira Torres, ETH Zurich
Chuanpu Fu, Tsinghua University
Daniel Arp, University College London
Daniel De Almeida Braga, Université Rennes 1, CNRS, IRISA
David Balash, The George Washington University
Dawei Wang, SKLOIS, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Eleonora Losiouk, University of Padua
Emanuele Vannacci, Vrije Universiteit Amsterdam
Eric Pauley, University of Wisconsin–Madison
Erik Tews, University of Twente
Erin Avllazagaj, University of Maryland College Park
Evan Johnson, University of California, San Diego
Fabian Ising, University of Applied Sciences Münster
Feng Wei, University at Buffalo
Giulio De Pasquale, King's College London
Grégoire Menguy, CEA LIST, Université Paris Saclay
Guangke Chen, ShanghaiTech University, Shanghai, China
Hai Huang, CISPA Helmholtz Center for Information Security
Hailong Hu, University of Luxembourg
Harshad Sathaye, Northeastern University
Hieu Le, University of California, Irvine
Huadi Zhu, University of Texas at Arlington
Hugo Lefeuvre, The University of Manchester
Hyungsub Kim, Purdue University
Imranur Rahman, North Carolina State University
Imtiaz Karim, Purdue University
Insu Yun, Korea Advanced Institute of Science and Technology (KAIST)
Jafar Haadi Jafarian, University of Colorado Denver
Jakob Bleier, TU Wien
Jiacen Xu, University of California, Irvine
Jiahao Cao, Tsinghua University
Joe Rowell, Royal Holloway, University of London
Johnny So, Stony Brook University
Kaushal Kafle, College of William & Mary
Luca Degani, University of Trento
Lucas Franceschino, Inria
Lukas Bernhard, Ruhr-Universität Bochum
Marco Casagrande, EURECOM
Menghao Zhang, Tsinghua University & Kuaishou Technology
Muhammad Haseeb, New York University
Mulong Luo, Cornell University
Nico Schiller, Ruhr-Universität Bochum
Nikolaos Alexopoulos, Technical University of Darmstadt
Nils Bars, Ruhr-Universität Bochum
Ning Luo, Yale University
Ningfei Wang, University of California, Irvine
Nuno Sabino, Carnegie Mellon University, Portugal
Nurullah Demir, Karlsruhe Institute of Technology
Pascal Cotret, ENSTA Bretagne
Peisen Yao, Zhejiang University
Pradyumna Shome, Georgia Institute of Technology
Rachel King, University of Wisconsin—Madison
Raounak Benabidallah, CEA LIST, Université Paris Saclay
Raphael Isemann, Vrije Universiteit Amsterdam
Rodothea Myrsini Tsoupidi, KTH Royal Institute of Technology
Ruoyu Wu, Purdue University
Saiful Islam Salim, University of Arizona
Salwa SOUAF , CEA LIST, Université Paris Saclay
Samee Zahur, Google
Sanket Goutam, Stony Brook University
Sayanton Dibbo, Dartmouth College
Shaohu Zhang, North Carolina State University
Shay Berkovich, BlackBerry
Shengtuo Hu, Meta
Shouvick Mondal, Concordia University
Shu Wang, George Mason University
Shubham Agarwal, CISPA Helmholtz Center for Information Security
Soheil Khodayari, CISPA Helmholtz Center for Information Security
Solmaz Salimi, Sharif University of Technology
Son Ho, INRIA Paris
Soomin Kim, Korea Advanced Institute of Science and Technology (KAIST)
Stefanos Chaliasos, Imperial College London
Sudheesh Singanamalla, University of Washington
Tiago Heinrich, Federal University of Paraná
Till Schlüter, CISPA Helmholtz Center for Information Security
Tobias Scharnowski, Ruhr University Bochum
Vishal Gupta, EPFL
Vladislav Mladenov, Ruhr University Bochum
Weidong Zhu, University of Florida
Weiteng Chen, Microsoft Research
Wenxi Wang, The University of Texas at Austin
Xi Tan, University at Buffalo
Xin'an Zhou, University of California, Riverside
Xu Lin, University of Illinois Chicago
Yang Hu, The University of Texas at Austin
Yanmao Man, ByteDance, Inc
Yilin Ji, Karlsruhe Institute of Technology
Yohan Beugin, University of Wisconsin-Madison
Yu Nong, Washington State University
Yuke Wang, University of California, Santa Barbara
Yusi Feng, University of Chinese Academy of Science
Zahra Tarkhani, Microsoft and University of Cambridge
Zhibo Liu, Hong Kong University of Science and Technology
Zhiyuan Zhang, University of Adelaide
Are you an experienced graduate student, postdoc, or researcher and interested in promoting reusability and reproducibility of security research? Join us! You can apply to be part of the AEC.
Artifact Evaluation Information
Overview
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the reproducibility of experimental results, USENIX Security will run its fourth (optional) AE this year. The AEC will review each submitted artifact and also grant Distinguished Artifact Awards to outstanding artifacts accepted to USENIX Security '23.
Process
To maintain a wall of separation between paper review and the artifacts, authors will be given the option to submit their artifacts only after their papers have been accepted for publication at USENIX Security. The artifact submission deadline is around five weeks after the paper notification date, but a first stub submission to register the artifact is required around two weeks after the paper notification date. By the artifact submission deadline, authors can submit their artifacts, Artifact Appendix, and other supporting information of their accepted USENIX Security 2023 paper via the submission form using the provided submission instructions.
At artifact submission time, authors can provide artifacts including software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. In addition, authors can request their artifact to be evaluated towards one, two, or all three of the following badges: Artifacts Available, Artifacts Functional, and Results Reproduced. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, documented well, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.
Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members will be instructed that they may not publicize any part of your artifact during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, exploits under embargo, etc. in your artifact. Since we anticipate small glitches with installation and use, reviewers may communicate with authors for a period of up to five weeks after the artifact submission deadline to help resolve glitches while preserving reviewer anonymity.
The AEC will then complete its evaluation and notify the authors of the outcome around six weeks after the artifact submission deadline. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.
As the notification of the AE is after the camera-ready deadline, badges cannot directly appear on the published papers. Nevertheless, authors are given the possibility to add the awarded badges and the Artifact Appendix on the author-version PDF that they can later host on their website or an archive. Moreover, USENIX will publish the badges and the appendices on the conference website. Finally, the badges and the artifact appendices will also be published by USENIX after the conference in dedicated proceedings. The deadline to submit the final artifacts, Artifact Appendix with badges, and other supporting information is around one week after the artifact notification date.
Acknowledgements
The AE process at USENIX Security '23 is a continuation of the AE process at USENIX Security '20–'22 and was inspired by multiple other conferences, such as OSDI, EuroSys, and several other systems conferences. See artifact-eval.org for the origins of the AE process.