Internet Reporting

Internet Reporting on the Private School Survey (PSS) and the Schools and Staffing Survey (SASS)

From 1999-2006, the National Center for Education Statistics (NCES) has offered respondents an Internet reporting option on some rounds of the PSS and on some components of the SASS. These included the:

• 1999-2000 Library Media Center Questionnaire (LMC, a component of SASS),

• 2001-02 PSS,

• 2004-05 Teacher Follow-up Survey (TFS, a follow-up survey on a sample of SASS teachers), and

• 2005-06 PSS.

All of these collections follow the same data collection steps:

Mail Phase: First, the questionnaire is mailed to the school (or to the teacher in the case of the TFS). Approximately a week later, a reminder postcard is sent. Approximately 4 to 5 weeks after the initial mailout, a second questionnaire is sent to non-respondents.

Telephone Phase: A few weeks (depending on the survey) after the second mailout, remaining respondents are contacted by phone to complete the questionnaire. The telephone phase lasts one to two months.

Field Phase: Remaining non-respondents are assigned to field staff who attempt to get the questionnaires completed by phone or personal visit.

The Internet option was offered at the beginning of the mail phase along with the mailed questionnaire in the LMC and the 2005-06 PSS. The 2001-02 PSS and the 2004-05 TFS offered the Internet option with some variations in time from the first mailout--either a week before (either with or without mentioning that that would be a mail option forthcoming), or concurrent. The Internet option was offered along with the second mailouts as well. The intended respondents were notified of the Internet options by mail. Except for the LMC, once telephone follow-up began, the Internet option ended.

The text refers to various response rates, which are defined as follows:

The self-administered response rate is the percent of completed returns during the mail phase (before the start-up of telephone follow-up). It is the number of completed mail questionnaires (and Internet, when offered) divided by the number of cases mailed (in every survey, every case was offered a mail questionnaire).

The Internet response rate is the number of completed Internet returns divided by the number of cases offered the Internet option (the number varied by survey).

The final response rate is the number of completed questionnaires from any mode divided by the number of cases in the sample reduced by the number of out-of-scope cases.

The following sections provide the rationale and results for the specific methodologies utilized in conjunction with each survey and makes recommendations for subsequent rounds of PSS and SASS. Specifically, the following sections cover:

• Summary of the impact on self-administered and final response of including an Internet reporting option;

• Summary of the Internet response rates on the LMC, PSS, and TFS;

• Costs of implementing the Internet option;

• Recommendations for the next rounds of PSS and SASS; and

• Synopses and results of each of the four Internet collections

Summary of the impact on self-administered and final response of including an Internet reporting option

This section examines the impact of adding an Internet option on response rates. Adding the Internet option in addition to the mail questionnaire had two potential advantages. First, if it increased self-administered response, then the costs of conducting telephone and field follow-up would be reduced, as there would be fewer cases. Second, there would be potential data processing advantages, as the Internet instrument captured the data, provides some edit checks on the data, and could expedite the release of the final results.

The 2001-02 PSS and 2004-05 TFS were designed to measure the impact of including an Internet option on self-administered response (2001-02 PSS and 2004-05 TFS). These research projects followed a 2000-01 study on the American Community Survey (ACS) that similarly tested the inclusion of an Internet option along with a mail questionnaire.¹ The results of these studies are shown in Table 1. Table 1 also includes response rate results from the LMC study, which included two Internet treatments, and compared the self-administered and final response rates. The LMC study differed from the other three in that it did not specifically measure response between mail only versus mail plus Internet groups.

Table 1. Impact of Including an Internet Option on Response in 4 Studies

Study	Description and Response Rates
2001-02 PSS	Control panel was mail only. Panel 2 offered mail and Internet concurrently. Panel 3 offered Internet one week before mail, with mention of a mail questionnaire. Panel 4 offered Internet one week before mail, without mention of a mail questionnaire.
	Mail only	Mail and Internet concurrently	Internet one week before with mention of mail	Internet one week before without mention of mail	Mail and Internet groups combined
Self-administered Response	55.9%	55.3%	57.5%	57.0%	56.6%
Final Response	95.7%	95.8%	95.3%	95.7%	95.6%
2004-05 TFS	Control panel was mail only. Panel 2 offered Internet one week before mail, with mention of a mail questionnaire. Panel 3 offered Internet one week before mail, without mention of a mail questionnaire. Each panel subdivided into incentive/non-incentive, but not broken out in this table.
	Mail only		Internet one week before with mention of mail	Internet one week before without mention of mail	Mail and Internet groups combined
Self-administered Response	48.8%		42.4%	45.4%	43.9%
2000-01 ACS	Control panel was mail only. Experimental group offered mail and Internet. Results for Nov, Dec and Jan panels combined.
	Mail only	Mail and Internet concurrently
Self-administered Response	43.6%	37.8%
1999-2000 LMC	Entire sample offered both mail and Internet. Half were given greater encouragement to respond by Internet at each stage of data collection.
	Greater encouragement	Less encouragement
Self-administered Response	36%	45%
Final Response	86%	86%

Overall, these studies indicate that adding the Internet option had a neutral or negative impact on initial response rates. When the mail and Internet options were offered concurrently, the results of the TFS and ACS studies showed that the self-administered response rate of the mail-only group exceeded that of the mail plus Internet group. While the self-administered response rates of the mail plus Internet groups exceeded the mail-only group on the PSS study, the difference was slight (and disappeared in the final analysis). While the LMC did not compare mail only versus mail plus Internet, the initial response of the group receiving greater encouragement to report by Internet was significantly lower.

The 2005-06 PSS offered the Internet option to nearly all respondents, but did not specifically include a design to measure self-administered response against a control group. However, the self-administered response rate was 0.8 percentage points lower than the 2001-02 PSS, when 1/3 of the schools were offered an Internet option, and 3.8 percentage points lower than the 2003-04 PSS, when no Internet option was offered.

The TFS and PSS studies also tested the self-administered response rates of Internet plus mail versus mail only when the Internet option was offered one week in advance of the mailout. Each study had two panels of these cases—the first mentioning that there would be a mail option, and the second not mentioning it. In both studies, the better response rates were achieved by the groups who were not informed of the mail option.

The final response rates did not differ between control groups and experimental groups in any study.

Internet Response Rates

Table 2 summarizes the percentage of potential respondents using the Internet response option from the four Education surveys and the ACS. In general, only a small percentage of the respondents chose to respond by Internet rather than paper.

Table 2. Number of Potential Respondents and Percentage Selecting the Internet Option Response in 4 Studies

Collection	Number Mailed	Number Offered Internet	Of Those Offered Internet, Percent Responding by Internet
1999-2000 LMC, Internet Offered With Motivational Messaging	5,736	5,736	21%
1999-2000 LMC, Internet Offered Without Motivational Messaging	5,735	5,735	9%
2000-01 ACS	9596	9596	2%
2001-02 PSS	43,772	14,412	11%
2004-05 TFS	7,738	5,132	20%
2005-06 PSS	37,975	37,643	17%

Costs of Implementing Internet Questionnaires

There was a substantial cost to adding the Internet mode to each of these surveys, including the costs of authoring and testing the Internet instrument, hardware set-up, and help-desk support during the collection period. It was hoped a higher initial response rate would lower follow-up costs that would more than offset these development and management costs. As noted, the only instance of a higher (but very small) initial response rate was on the 2001-02 PSS. But upon further analysis, the 2001-02 PSS study determined that additional, rather than lesser follow-up efforts were required for the Internet groups. The study included the following calculation of follow-up costs by pane, if the response options used in each treatment panel had been offered to the complete set of PSS respondentsl:

Table 3. Potential Non-response Follow-up Cost by Treatment Panel based upon the 2001-02 PSS Internet Experiment

Panel	Total Interviews	Approx. Number of Late-Mail Returns	Approx. CATI Interviews	Approx. Field Interviews	Approx. Late-Mail Return Cost	Approx. Total Nonresponse Follow-up Cost
Mail only	41,377	248	10,160	1,124	$2,480	$374,280
Mail and Internet concurrently	41,377	579	10,506	1,384	$5,790	$423,510
Internet one week before with mention of mail	41,377	1,614	9,901	1,167	$16,140	$389,210
Internet one week before without mention of mail	41,377	1,945	10,506	1,124	$19,450	$398,170

There is also an impact on staff time in adding an Internet reporting option, which in turn impacts the critical path of processing and releasing survey results. For example, on the 2005-06 PSS, the PSS staff spent several weeks testing the instrument and preparing for the additional mode of Internet. Once collection began, staff received numerous calls from respondents related to the Internet option. These same resources could have been applied towards expediting the overall survey processing, and getting the survey results published more quickly.

Recommendations for the next rounds of PSS and SASS

The experience of adding an Internet option concurrently with a mail questionnaire has had a neutral or negative impact on self-administered response. The best results were achieved when the Internet option was offered one week before the mail questionnaire, with no mention of the mail questionnaire. For all the surveys, only a small percent of respondents offered both modes chose to respond by Internet. Finally, there have been significant costs in adding an Internet option. For all these reasons, if Internet is going to be an additional option in these surveys, at an additional cost, it is best to offer it in a different manner than it has been offered, and/or when other advantages are evident.

Many surveys are able to make contact with respondents by e-mail, reducing the amount of time and costs for follow-up. E-mail messages can include a link to the online Internet questionnaire via a secured server, which makes it relatively easy to access and launch. Mail notification, as it has been done on the education surveys, requires the respondent to go to their computer, open their browser, and key in a user name and password. . Thus far, the constraint for using e-mail notification on SASS and PSS has been that e-mail addresses have not been available prior to data collection. But given the potential advantages of e-mail notification, and noting the lack of success in notification by mail, it is recommended to pursue e-mail notification, subject to Privacy Act and data security provisions.

For SASS, it is recommended that an Internet version of the TFS questionnaires be included, with notification and follow-up of the Internet option by e-mail. It would be beneficial because:

• Teachers have e-mail addresses (many have both home and school e-mail addresses);

• The Census Bureau anticipates collecting e-mail addresses for at least half of the sample of teachers in advance of the survey, based on a recent SASS test. So, notification of the option, and subsequent follow-up can be delivered by e-mail; and

• The Internet questionnaire would incorporate both the TFS-2 questions (for teachers who left the teaching profession) and the TFS-3 questions (for current teachers). This would reduce the costs associated with changing questionnaires for ‘switchers’—sample persons who were provided the incorrect questionnaire because their teaching status was reported incorrectly by the school.

• Based on the studies, the Internet option would not mention the mail option. Questionnaires would be mailed to sample persons who do not respond by Internet after two weeks.

For PSS, it is recommended that the 2007-08 survey focuses on the collection of school-level or principals’ email addresses, and that the 2007-08 survey not include an Internet reporting option.

Synopses and Results of Each of the Four Internet Collections

a. 1999-2000 Library Media Center Questionnaire (SASS)

The Library Media Center (LMC) questionnaire is one of several questionnaires collected on the SASS. In 1999-2000, an Internet reporting option was provided to all respondents along with the paper questionnaire. A split panel study was incorporated to determine if highlighting the benefits of Internet reporting would encourage more respondents to choose that mode. Half the sample received a paper form and motivational messaging, encouraging respondents to respond by Internet, in the mailouts and during non-response telephone follow-up calls. The other half also received a paper form but did not receive any motivational messaging; instead they received instructions that provided factual information about accessing the Internet option. There was no mention of the Internet option during this panel’s non-response telephone follow-up calls.

The motivational messaging treatment did achieve its goal of getting more respondents to respond using the Internet questionnaire, but it did so at a high price of needing many extra non-response telephone follow-up calls. Approximately 21 percent of the treatment panel responded via the Internet compared with 9 percent of the control panel. For self-administered response rates, the treatment (the motivational insert) had a dampening effect on the overall response rate in the treatment panel. The motivational insert did produce significantly higher self-administered Internet use rates, but the size of the effect was small and not of practical value, as less than one percent of the schools reported by Internet before the telephone follow-up started.

While this study did not attempt to ascertain the impact on response of including an Internet option (mail-only option versus mail plus Internet options), there was no motivational treatment effect on final response rates. The final response rate in each of the panels was 86 percent. The surprising finding in this study was that the treatment actually suppressed response rates in the initial part of the field period, prior to the telephone calls made to the non-responding cases. The self-administered response rate for the treatment panel was 36 percent compared to a 45 percent response rate for the control group.

For a complete discussion, please see: Nichols, E., Marquis, K., and Hoffman III, R. 2001. “The Effect of Motivational Messaging on Mode Choice and Response Rates in the Library Media Center Survey.” 2001 Proceedings of the American Statistical Association, Survey Research Methods Section [CD-ROM], Alexandria, VA: American Statistical Association.

b. 2001-02 PSS

The 2001-02 round of PSS incorporated an experiment that specifically tested the impact of adding Internet reporting options on response. Approximately one-third of the respondents were offered the option to complete the survey via the Internet in addition to the traditional mail response option. These respondents were randomly assigned to one of the following three treatment panels:

Panel 2: Internet and mail response options offered at the same time (4,804 schools)

Panel 3: Internet response option offered first with mention of arriving mail questionnaire (4,804 school)

Panel 4: Internet response option offered first without mention of arriving mail questionnaire (4,804 schools)

The other two-thirds were only given the traditional mail response option and these respondents were treated as the Control Panel (i.e. Panel 1). The experiment was designed to answer the following questions:

• Does offering an Internet completion option increase the final response rate?

• Does offering an Internet completion option increase the self-administered response rate, in turn decreasing the amount of follow-up needed? In addition, does the number of expected late mail returns by panel have any cost implications?

• Do any of the Internet treatments have a significantly higher response rate than another?

• Does responding via the Internet affect the response rate of key survey items?

• Can the likelihood of an Internet response be predicted by school characteristics?

Staff calculated self-administered, final, and Internet response rates for each treatment panel and the Control Panel. The final results indicated that there were not any significant differences between the self-administered and final response rates of the control and treatment panels. In early stages of the analysis, there were some differences between the self-administered response rates of the control panel (55.9%) and panels 3 (57.5%) and 4 (57.0%). There were also significant differences between each of the Internet-only response rates of each of the treatment panels; panel 4 (21.3%) was higher than panel 3 (17.5%), which was higher than panel 2 (10.6%). However, the significant differences found in the self-administered response rates between the control and treatment panels 3 and 4 disappeared in the final response rate analysis.

For a complete discussion of all the issues, please see: Warner, T. (2004). 2001-02 Private School Survey (PSS) Computerized Self-Administered Questionnaire (CSAQ) Analysis. Internal Census Bureau Memorandum, February 2004.

c. 2004-05 Teacher Follow-up Survey (TFS) Questionnaires (SASS)

The TFS is administered one year after the SASS and obtains follow-up interviews from a sample of the SASS Teacher Survey respondents. A goal of the 2004-2005 TFS was to increase (or at least maintain) self-administered response rates by offering an Internet reporting option (in addition to the paper questionnaire) and by providing a monetary incentive to respondents. Building on the PSS study, six experimental groups were established.

	Groups one and two were not given the option of completing the survey on the Internet and were sent only paper questionnaires throughout the duration of the experiment. Groups three through six were initially given the Internet option and shortly afterwards were given the paper option as well. However, groups three and four were not made aware that they would receive paper versions of the TFS a week later. Groups five and six were told of the forthcoming paper questionnaire in the initial letters they received requesting their participation in the TFS. Finally, these six groups were further broken down into incentive panels. Groups one, three and five were given a $10 incentive card with the first mailing of TFS materials. The remaining groups were not offered any kind of incentive. TFS Internet and Incentive Experimental Groups
Group		1	2	3	4	5	6
Mode Offered		No Internet	No Internet	Internet, without mention of mailout one week later	Internet, without mention of mailout one week later	Internet, with mention of mailout one week later	Internet, with mention of mailout one week later
Incentive Offered		Yes	No	Yes	No	Yes	No

Results of the study indicated that impact on response of adding the Internet option was negative—the self-administered response rate of the mail-only group exceeded the combined Internet groups by 4.8 percent. When the mail group was compared to the Internet groups separately, the mail group’s self-adminstered response was significantly higher than the Internet group with mention of the mail option, but not higher than the Internet group without mention of the mail option.

The overall impact of providing the $10.00 incentive did increase the self-administered response rate by 5.5 percent, but it did not pay for itself (i.e., the monetary cost of the incentives exceeded that savings from the reduction in field costs). Further, the incentive also increased Internet response among both Internet treatment groups, but as noted, the total response of the Internet groups was lower than the mail only group.

For a complete discussion, please see: Tourkin, S., Parmer, R., Cox, S., and Zukerberg, A., 2005. “(Inter) Net Gain? Experiments to Increase Response.” Paper presented at the Annual Conference of the American Association for Public Opinion Research Annual Conference, Miami, Fl., May.

d. 2005-06 PSS

An Internet reporting option was offered to 37,643 of 40,501 schools in the 2005-06 PSS.² Each mailout included a paper questionnaire and notification of the Internet option. The table below compares the self-administered and Internet response rates of the 2005-06 PSS, the 2003-04 PSS and the 2001-02 PSS. The results indicate that highest self-administered response rate, 49.6 percent, occurred in the 2003-04 PSS (no Internet option); the next highest self-administered response rate, 46.5 percent, occurred in the 2001-02 PSS ( Internet option offered to approximately 1/3 of the schools); and the lowest self-administered response rate, 45.7 percent, occurred in the 2005-06 PSS Internet option to nearly all schools). When the Internet option was offered, 11 percent of respondents in 2001-02 and 17 percent in 2005-06 chose to respond by Internet.

Self-administered and Internet Response Rates on PSS, 2001-2006
	Number Mailed	Number Offered Internet	Mail Returns	Internet Returns	Self-administered Response Rate	Internet Response Rate
2001-02 PSS	43,772	14,412	18,755	1,598	46.5%	11.1%
2003-04 PSS	39,387	0	19,532	0	49.6%	NA
2005-06 PSS	37,975	37,643	10,904	6,467	45.7%	17.2%

1 Griffin, D., Fischer, D., and Morgan, M. (2001). “Testing an Internet Response Option for the American Community Survey.” Paper presented at the Annual Conference of the American Association for Public Opinion Research Annual Conference, Montreal, Quebec, May.

2 1,872 ‘area frame search’ schools were added to the PSS at the telephone follow-up stage, and therefore missed the mail/Internet phase. The remaining schools not offered Internet were Amish or Mennonite, and some that were included in a SASS field text that collected the PSS items for those schools.

10