Mikhail Pravilov, Google
We introduce a novel framework for generating insights about LLM chatbot interactions with rigorous differential privacy (DP) guarantees. The framework employs a private clustering mechanism and DP keyword extraction methods. By leveraging DP tools such as clustering, partition selection, and histogram-based summarization, Urania provides end-to-end privacy protection. Our evaluation assesses quality of the output benchmarked against a non-private Clio-inspired pipeline (Tamkin et al., 2024). The results show the framework's ability to extract meaningful conversational insights while maintaining stringent user privacy, effectively balancing data utility with privacy preservation. Finally, we discuss the practical applications, limitations, and operational challenges of applying Urania in real-world settings.
Authors: Edith Cohen, Vadym Doroshenko, Badih Ghazi, Charlie Harrison, Peter Kairouz, Pritish Kamath, Alexander Knop, Ravi Kumar, Ethan Leeman, Daogao Liu, Pasin Manurangsi, Adam Sealfon, Da Yu, and Chiyuan Zhang
Mikhail Pravilov is a Software Engineer on Google's Anonymization team, developing practical Differential Privacy solutions at scale. A main contributor to the open-source Jax Privacy and PipelineDP4j libraries, he also works on numerous internal anonymization projects. Holding a bachelor's degree in Machine Learning, Mikhail is dedicated to advancing real-world data privacy.
author = {Mikhail Pravilov},
title = {A {DP} Framework for Gaining Insights into {AI} Chatbot Use},
year = {2026},
address = {Santa Clara, CA},
publisher = {USENIX Association},
month = jun
}