Dimensions of online survey data quality: What really matters?

Jon Puleston and Mitch Eggers
GMI (Global Market Insite), UK and USA

Introduction

This paper is an examination of the relative impact of survey design, panel quality, and cross-cultural behavior on the accuracy of data when conducting international online survey research. We study all three of these general influences on data quality using seven specific quality improvement techniques in 15 different countries.

Over the past few years a great deal of research has been independently conducted to understand the impact on data integrity of panel quality, survey engagement techniques, and cross-cultural response biases. Few studies, however, have examined all three of these factors together using a comparative cross-national approach. In addition, this research examines other major factors thought to impact data quality such as imposing age-gender quota cells to drive distributional consistency, setting speed limits that remove those who are inappropriately rushing though surveys, and requiring sample sourcing consistency to remove variance caused by differential unobserved panel characteristics. The latter well-researched aspects of quality are familiar to most researchers and provide an ideal comparative baseline that allows us to answer our primary research question: Which of these various data quality improvement techniques make the biggest difference and how do they interact with each other to produce double trouble (multiplicative) for some topics and question types.