Scientists struggle with representativeness of online surveys - Amsterdam, 2 Oct. 2009
How trustworthy is your online survey? The questions is asked often also about the WageIndicator. On October 1 Stephanie Steinmetz and professor Kea Tijdens of AIAS at the University of Amsterdam, the scientific partner in the Netherlands of the WageIndicator presented a case documenting the scientific pitfalls of online surveys.
"Every survey has shortcomings and biases, even if you have the best possible survey," says Steinmetz during the presentation. "What you need to do is explain those as good as possible, so other scientists can take note of that.
Popular
Online surveys, like those of the WageIndicator, have become popular very fast, notes Steinmetz. "In Germany the share of online surveys increased between 2000 and 2007 from three to 27 percent."
To investigate the representativeness of online surveys, and possibly develop methods to correct them, Steinmetz compared the surveys of the German Lohnspiegel and the Dutch Loonwijzer over 2006 with other reference surveys.
Huge savings in both costs and time have triggered off the popularity, although online surveys have also some disadvantages: a part of the population is not online, and even of those online a part decides not to participate. The none-response rate is increasing fast. "Last week I listened to a presentation of a survey with a non-response rate of 90 percent," says Tijdens. "When I started in this field, already a non-response rate of 30 percent was not done."
Both the Dutch and German surveys showed an underrepresentation of both women and the older age groups. In German additionally the higher educated were slightly underrepresented, possibly because the Lohnspiegel is mostly promoted through trade union site, focusing on the lower educated.
Correction methods
Steinmetz investigated two methods to possibly correct the representativeness of online surveys, one where the online results were brought in line with an offline survey for age, gender and education. A more complex correction method was also applied. After the correction, both the online survey and the reference survey of course matched for the criteria there were corrected, but the deviation on other criteria only increased, said Steinmetz.
The difference in the Netherlands was smaller compared to Germany, possibly because in the Netherlands the percentage of the population online was larger than in Germany, making the survey more representative, suggested Kea Tijdens as an explanation.
Improvements
Steinmetz suggested several ways to improve the scientific value of the online surveys, including always having an offline reference survey at hand, although that would reduce the current cost and time advantages of online studies. Another ways was that we would have to live with the idea no survey would be really representative for the general population, including the online ones.
Kea Tijdens expects that the salary results for individual occupations is much closer to the reality. "Although, we need of course more research," she smiles.
A formal paper by Stephanie Steinmetz and Kea Tijdens on their findings will be ready soon.
Fons Tuinstra
