It has been posited that the success and future of Internet research will stand or fall by the industry’s ability to demonstrate that such techniques can produce data which are similar to data produced by other 'credible' or more traditional sources, such as telephone and mail.  Given that most concerns regarding online research tend to revolve around the sampling design and representativeness of online respondent populations, this perspective is certainly appreciated.

More specifically, it has been proposed that propensity or demographic score adjustments (i.e., weighting) can be used to compensate for differences between demographic characteristics of the online population relative to those not on online. This is indeed a very interesting proposition and certainly one that holds important implications for the marketing research community.

While much research in this arena appears acceptable (i.e., post-hoc comparisons of data collected from parallel online and telephone surveys/polls), I would argue, however, for a closer look at the internal and external validities of such a proposition.

Foremost, propensity and demographic weighting are relatively new techniques and are still subject to much scrutiny. It could be argued that such techniques are developed for handling non-random assignment of treatments in experiments. Yet, if the 'treatments' are participation and non-participation in an online survey, the online sample contains no non-participants (as online surveys use only 'volunteer samples') that could be used to estimate the propensity score.

But in the same vain telephone surveys often miss those who live in homes without telephones, people who are away from home, and people who refuse to be interviewed. Most telephone and in-person surveys under-represent people who travel a lot and eat out a lot and thus are not at home to be interviewed. Thus, while the conceptual use of 'weights' is well-received, it may be difficult, without further qualification, to justify the use of weights formulated through telephone data, when telephone data themselves are based on 'convenience sampling,' and thus may be as unreliable as online data supposedly are.