Essays on deliberate errors in surveys

Olbrich, Lukas (2024): Essays on deliberate errors in surveys. Dissertation, LMU München: Fakultät für Mathematik, Informatik und Statistik

Abstract

Surveys have always been prone to deliberate errors by the involved actors (e.g., interviewers or respondents). Such errors can bias survey estimates, lead to biased inferences in survey data analysis, and undermine trust in survey data. This dissertation contributes to the literature on preventing and identifying such behavior. The first four contributions focus on deliberate errors by face-to-face interviewers (e.g., fabrication of parts of or entire interviews). The fifth contribution investigates inattentive responding which is a type of deliberate error by web survey respondents. The first contribution analyzes the deterrence effect of interview audio recordings in face-to-face interviews. With respondent consent, interviews are audio-recorded and these recordings are later used to evaluate the interviewers’ behavior. Without recordings, the interviewers’ behavior is not observable. Using detailed timestamp data and multiple analysis approaches, we show that audio recordings substantially reduce the prevalence of likely deliberate interviewer errors. The second contribution illustrates how multilevel modeling can be an effective analysis approach to identify fraudulent interviewers. In particular, the developed model focuses on the fraudulent interviewers’ behavior over the field period. The method is applied to survey data containing verified falsifications and further data without verified falsifications. The model identifies the verified falsifiers in the former dataset and flags multiple suspicious interviewers in the latter. The third contribution proposes another approach to identifying error-prone interviewers. We exploit that adult self-reported height is stable within short timespans and identify interviewers as error-prone if 1) self-reported heights frequently and substantially differ from measured heights or 2) self-reported heights change frequently and substantially over panel waves. Using multilevel models, we apply the identification approach to four survey datasets and identify several error-prone interviewers. The fourth contribution develops a multivariate approach to analyzing interviewer errors. Using data from ten waves of a yearly panel survey conducted in ten countries (i.e., 100 country-years), we apply multiple indicators of interviewer error both on the interviewer and country-year level. To identify exceptional country-years and interviewers, we use isolation forests and show that interviewer errors are particularly prevalent in several country-years. The results led to the exclusion of multiple country-years from the publicly released data and emphasize the importance of taking the fieldwork institute into account when analyzing interviewer errors. The fifth contribution focuses on preventing and identifying inattentive responding (i.e., providing responses without regard to the question content) in web surveys. As a preventive measure, we experimentally tested the efficacy of so-called commitment pledges that ask respondents to commit to providing accurate responses but found no effect on multiple indicators of inattentive responding. Concerning identification measures, we conducted a further experiment on widely used attention checks and show that large proportions of respondents likely pass such checks by chance. As an alternative, we propose a timestamp-based clustering approach to identify clusters of likely inattentive respondents which is applied to multiple datasets. The contributions on measures to prevent deliberate errors may guide practitioners in designing surveys. The developed and tested identification methods may guide practitioners and applied researchers who seek to assess the quality of their data. In sum, this dissertation contributes to avoiding the prevalence and (potentially detrimental) consequences of deliberate errors in surveys.

• Dokumententyp: Dissertationen (Dissertation, LMU München)
• Keywords: interviewer falsification, interviewer effects, paradata, inattentive responding
• Themengebiete: 300 Sozialwissenschaften ; 300 Sozialwissenschaften > 310 Statistik
• Fakultäten: Fakultät für Mathematik, Informatik und Statistik
• Sprache der Hochschulschrift: Englisch
• Datum der mündlichen Prüfung: 27. November 2024
• 1. Bericht­erstatter:in: Sakshaug, Joseph