Is it a concern or a preference? An investigation into the ability of privacy scales to capture and distinguish granular privacy constructs

Privacy scales are frequently used to capture survey participants' perspectives on privacy, but their utility hangs on their ability to reliably measure constructs associated with privacy. We investigate a set of common constructs (the intended objects of measurement by privacy scales) used in privacy surveys: privacy attitude, privacy preference, privacy concern, privacy expectation, privacy decision, and privacy behavior. First, we explore expert understanding of these constructs. Next, we investigate survey participants' understanding of statements used in privacy scales aimed at measuring them. We ask a balanced sample of Prolific participants in the United States to identify the extent to which different constructs describe each of a set of 30 statements drawn from scales used commonly in the privacy literature and 39 that we developed. Our analysis reveals considerable misalignment between the constructs associated with the statements and participant understanding. Many statements used in scales or that we developed with the intention to measure constructs such as privacy concern, are seen by survey participants as describing other constructs, such as privacy preferences. We also find that no statement uniquely measured any one construct, though some more reliably track their target construct than others. Our findings constitute an epistemological problem for use of scales in the existing literature (are they capturing what we think they capture?) and a practical problem for construction of new scales (how to ensure construct validity in the face of ill-defined constructs and evolving privacy landscape?). We use methods from corpus linguistics to identify characteristics of those statements most reliably associated with their target construct, and provide a set of provisional suggestions for future statement construction. Finally, we discuss the implication of our results for the privacy research community.