Tuesday, 10 May 2005

by Mads Stenbjerre, Research Director, Zapera
& Jens N. Laugesen, Research Manager, Zapera

A Summary of Five Years of Learnings

Online research has reached a level of maturity in the Nordic region that it still does not enjoy in most other parts of the world. We benefit from a high internet penetration along with a homogenous distribution of internet access across various demographic parameters such as age, sex and occupation.

It is essential to establish and maintain a proper sampling frame – an online access panel – to operate in the online research industry. Five years of operation in the Nordic region have given us experience on various aspects to have efficient access panels operating today. We know that our need of continuous recruitment should be diversified across both online and offline channels. We know how to balance and control the proper number of invitations and the length of surveys and how they influence directly on the response rate. Finally we have gained valuable experience on how to communicate with our respondents and operate a “contract” to create a reasonable level of commitment between the respondents and us.

We have described the sampling and weighting procedures that we use. As case study we have presented the results of the latest online exit poll we have conducted (the third in all). We still do not have the benefit of the many years of experience that CATI based surveys can rely on. But we believe that we can learn from the “mistake” we made in this year’s election in much the same way that Gallup learned from the experiences of 1948 *).

Paying attention to traditional virtues

We have made the case for paying attention to “traditional” virtues such as high response rates and solid sampling procedures, and not touched much upon other potential strategies, such as propensity weighting, intercepts or even other versions of quota sampling in wide use in today’s online research world.

The reason for these choices are to be found in the fact that we attempt to present online research as a replacement for traditional methods (such as CATI), rather than as a supplementary method. The consequence is that we attempt to match quality standards of traditional methods (e.g. high response rates and moderate weighting, etc.).

Ideally, to move further into the centre stage we need to collect experiences in a more systematic manner so that we may expand the areas where we can confidently apply quota sampling approaches and start reducing error margins further, for example through the application of more sampling variables.

But even while we are hopeful for the future we are seeing that the conditions for online research are already changing in the Nordic region. As described in the paper, we know of online access panels comprising a total of at least 450.000 individuals in Sweden today (and similar shares of the population in the other Nordic countries).

Respondents belong to multiple panels

In a recent copy of Research World, David Pring notes that participation levels may be deteriorating rapidly on the internet and that a considerable number of respondents belong to multiple panels (Pring, 2005).

We are seeing the same trends in our part of the world. Fully 43% of Zapera’s respondents are also members of at least one other panel. Why? Because most research institutes in the Nordic region are busy building access panels.

This development is troubling. Due to some respondents’ multiple memberships, the average member of Zapera’s panels participates in 1.8 surveys per month. In addition to the online survey activity, 58% have been contacted at least once by a research agency via telephone in the latest three months.

This development may erode the individual panel provider’s ability to control or guarantee the degree of survey activity that respondents are exposed to. As Pring (2005) notes, this is not in itself an insurmountable problem, but it warrants careful attention.

Never sleep on the laurels

We believe that in the future, online research must continue to pay heed to the “good habits” of traditional research and be able to document research participation, response rates, error margins, etc. We further believe that online research is already a fully qualified alternative to other methods but that we must never “sleep on our laurels”.

As we have learned in the five years that Zapera has conducted research, online research is truly a multidisciplinary approach. And attention to all aspects of the access panel management process must be maintained.

*) Gallup predicted in 1948 that governor Thomas Dewey would be elected as president instead of president Harry Truman. This was however not true as Truman got re-elected. Gallup did not know the population well enough why they used the sampling method wrong. See “The Practice of Social Research” by Earl Babbie (7th Edition 1995) for more details.