Wave Analysis

Dear Folks,
There was an interesting paper by Larry Leslie at Penn State that has important currency for all of us administering surveys and worried about not getting everyone to return the survey duly completed on time. Provided you record the time when you give out the questionnaire (if the same time for all the respondents, then you can effectively ignore this), and then record the exact time each is returned to you, then a simple wave analysis will give you valid and reliable findings from relatively-low response rates. I use this all the time (even though I collect 100% response rates by never using anonymity) because it can show me if there are likely to be any new response or finding were I to administer the survey to more persons or to an additional cohort.

Basically, you analyse the responses returned within the first week - and note all the different responses or categories and the distribution pattern of the frequencies of each response. Then you analyse similarly those returned in the second week. If you don't find any new category, and you find a similar distribution as in the first-week categories, then you can be fairly confident that you have captured the full range of categories and their relative incidence given in the distribution pattern. Then after you analyse those few late-respondents in the third week, if you find the same range of categories and distribution pattern, you can validly and reliably claim you have got the whole range of potential responses and their distribution pattern, even though a few responses are still not returned in the fourth or fifth week.

Actually, there may be a drift in the distribution pattern from early-respondents to late-respondents, allowing you to extrapolate the pattern drift to cover the few very-late/non-respondents. If you find new responses or a new category being given by the second-week respondents, this means that either your sample size was not sufficiently large or that the time allowed for collecting the early-respondents should be longer than one week. It is easy to change the time allowed for being in the early-respondents group, since you have the time-stamp on each return. However it is usually difficult to change the sample size. You need to do a pilot study, and have at least 2 or 3 times as many respondents as you have question items according to the SPSS version 6.1 software guidebook. However Sanjaya Mishra and Santosh Panda (page 29) in an excellent article on questionnaire design cite Tinsley & Tinsley (1987) as suggesting a ratio of 5 to 10 respondents per item - see http://www.AsianJDE.org/2007v5.1.Mishra.pdf , and Tinsley, H.E.A., & Tinsley, D. J. (1987). Uses of factor analysis in counseling psychology research. Journal of Counseling Psychology, 34, 414-424.

A pilot study should never be omitted in any case, period.

I have used wave analysis in all my studies. It can show me three cohorts from different places give a same pattern and therefore gives high confidence in the findings. In other studies, I use it to see the drift from one year to the next, over four or five years, in say the use of computers at home, or in the earlier adoption of collaborative group learning skills.
Here is the full citation ; enjoy -
Leslie, L.L. (1972) ‘Are High Response Rates Essential to Valid Surveys?’, Social Science Research, vol. 1, pp. 323-334.
All Best Wishes
Paul