Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective

Machine unlearning focuses on efficiently removing specific data from trained models, addressing privacy and compliance concerns while managing computational costs. Although exact unlearning ensures complete data removal equivalent to retraining, it is impractical for large-scale models, leading to growing interest in inexact unlearning methods. However, the lack of formal guarantees in these methods necessitates the need for robust evaluation frameworks to assess their privacy and effectiveness. In this work, we present RULI (Rectified Unlearning via Likelihood Inference), a novel framework designed to address critical gaps in the evaluation of inexact unlearning methods. Unlike existing approaches that emphasize average-case privacy leakage and the recent U-LiRA, RULI introduces a dual-objective attack to measure both unlearning efficacy and privacy risks at a per-sample granularity. Our findings reveal significant vulnerabilities in state-of-the-art unlearning benchmarks, where RULI achieves higher attack success rates, exposing privacy risks underestimated by existing methods. We also underscore the limitations of existing evaluation frameworks and highlight the benefits of targeted analysis for identifying vulnerable samples. Built on a game-theoretic foundation and supported by empirical evaluations, RULI provides a rigorous, scalable, and fine-grained methodology for evaluating unlearning techniques. This work lays the groundwork for future designs of unlearning algorithms, focusing on both practical privacy guarantees and robust efficacy measurements.