NCES Generic Testing Supp Statement Part A

SUPPORTING STATEMENT

A. Justification

1. Necessity of Information Collection

NCES is requesting that the current clearance for testing of new methodologies for surveys and assessment be extended for three years. We have found that the testing clearance is a helpful vehicle for evaluating questionnaires/assessments and various data collection procedures. In the past, traditional methods typically consist of small "hothouse" field tests accompanied by interviewer debriefing. The information collected through their use is quite limited in its ability to detect and diagnose problems with the instruments and the procedures being tested. The generic testing clearance has allowed NCES to take advantage of new methods that have come into popular use, which are useful for identifying questionnaire/assessment and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these kinds of techniques, employed routinely in the testing phase of NCES surveys, questionnaires and assessments can be simplified for respondents, respondent burden can be reduced, and the quality of the questionnaires and assessments used in continuing and one-time surveys and assessments can be improved. Thus an increase in the quality of the data collected through these surveys has been achieved as well.

NCES is requesting a three-year generic clearance for pretesting, during which NCES will provide periodic reports on pretesting activities. These pretesting activities will include, in addition to methods used in the past, expanded field tests including split sample questionnaire experiments in multiple panels and usability testing of electronic data collection instruments. The focus of these activities will include investigation of item types, research about incentives (cash and non-cash), and mode (telephone, paper and pencil, mail-out and mail-in, etc.) and methodology of questionnaires and assessments, as well as testing of items.

This clearance package is intended to serve as a request for system clearance. In this document we have provided a description of the scope of possible activities that might be covered under this clearance. NCES requests the same conditions that were included in previous clearance agreements, such as the system clearances under which the NCES Quick Response Information System (QRIS) currently operates (OMB control number 1850-0733) and under which NCES currently tests assessment items and testing conditions and performs other developmental efforts on the National Assessment of Educational Progress (NAEP) (OMB 1850-0790). The requested clearance is important to NCES’ use of pretesting activities, because of the length of time required to plan the activities. This system clearance will go through the usual two Federal Register Review periods. OMB would continue to review and comment/clear on proposed studies in a two-week period with no 30-day Federal Register Notice period required. This is our preference for studies under this clearance. This clearance is similar to the testing clearances held by the Census Bureau and the Bureau of Labor Statistics.

Programs that have submitted developmental studies under this clearance include the Early Childhood Longitudinal Studies (ECLS), the National Household Education Surveys (NHES), the Schools and Staffing Survey, postsecondary survey activities, and the National Assessment of Educational Progress. We anticipate that other NCES programs will also be able to use this clearance for their developmental projects.

The specific methods proposed for coverage by this clearance are described below. Also outlined are the procedures currently in place for keeping OMB informed about the identity of the surveys and the nature of the research activities being conducted.

The methods proposed for use in questionnaire and assessment development are as follows:

Field or pilot test. For the purposes of this clearance, we are defining field tests as small data collection efforts generally of 500 cases or less, conducted among either purposive or statistically representative samples, for which evaluation of the questionnaire and/or procedures is the main objective and no plans to publish the data as such are envisioned.

Field tests are an essential component of this clearance package because they serve as the vehicle for investigating basic item properties, such as reliability, validity, and difficulty, as well as feasibility of methods for standardized administration (e.g., computerized administration) of forms. Field test methodology is necessary also to conduct standardized behavior coding of the interaction between the respondent and the interviewer. This methodology does not require any additional data collection above and beyond the field test--it involves applying a standardized coding scheme to the completion of a field interview, either by a coder using a tape-recording of the interview or by a "live" observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give inadequate answers, this suggests there are some other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire, and research ("New Techniques for Pretesting Survey Questions" by Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989) has demonstrated that this is a more objective and reliable method of identifying problems than the traditional interviewer debriefing, which is typically the sole tool used to evaluate the results of a traditional field test.

Interviewer debriefing has advantages as well, since it employs the knowledge of the employees who have the closest contact with our respondents. In conjunction with other methods, we plan to use this method in our field tests to collect information about how interviewers react to the survey instruments.

Field tests conducted under this clearance will involve either purposive or statistically representative samples. Under this clearance a variety of surveys will be pretested, and the exact nature of the surveys and the samples is undetermined at present. However, due to the small nature of the tests, we expect that some will not involve representative samples. In these cases, samples will basically be convenience samples, which will be limited to specific geographic locations and may involve expired rotation groups of a current survey blocks that are known to have specific aggregate demographic characteristics. The needs of the particular sample will vary based on the content of the survey being tested, but the selection of sample cases will not be completely arbitrary in any instance.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested, and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a representative sample of respondents, to learn whether respondents can answer the questions, and whether they interpret them in the manner intended by the questionnaire designers.

Follow-up interviews (or reinterviews). This involves reinterviewing or re-assessing a sample of respondents after the completion of a survey or assessment. Responses given in the reinterview are compared with the respondents’ initial responses for consistency of responses. In this way, reinterviews provide data for studies of test–re-test reliability and other measures of the quality of data collected. In turn, this information aids in the development of improved (more reliable) measures.

Split sample experiments. This involves testing alternative versions of questionnaires, at least some of which have been designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires, randomly assigned to permit statistical comparisons, is the critical component here; data collection can include mail, telephone, Internet, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a control version, preferably, or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire.

In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty persons in a split sample test.

Split sample tests that incorporate questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than field tests using other methodologies. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of NCES data collection instruments.

Cognitive interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents have answering questions. This is frequently the first stage of revising a questionnaire.

Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues (e.g., confidentiality concerns) which people may feel some hesitation about discussing.

Procedures for Clearance

NCES will provide OMB with a copy of questionnaires and debriefing materials in advance of any testing activity. Depending on the stage of questionnaire development, this may be the printed questionnaire from the last round of a survey, a revised draft based on analysis of other evaluation data, or a set of prototype items showing each item type to be used and the range of topics to be covered by the questionnaire. When the time schedule for a single survey permits multiple rounds of testing, the questionnaire(s) for each round will be provided separately. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. For a test of alternative procedures, the description and rationale for the procedures would be submitted. A brief description of the planned field activity will also be provided. OMB may raise requests for information or comments on substantive issues within 10 working days of receipt.

Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. In most cases, this will be Education Sciences Reform Act, Part C, Section 153, although other legislation, such as Title 13 or 15, may apply for surveys conducted in concert with other Federal agencies. A copy of the Education Sciences Reform Act, Part C, Section 153, is attached. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.

2. Needs and Uses

The information collected in this program of developing and testing questionnaires will be used by staff from NCES and sponsoring agencies to evaluate and improve the quality of the data in the surveys and assessments that are ultimately conducted. None of the data collected under this clearance will be published for its own sake.

Because the questionnaires being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics of NCES or other Federal agencies. Data will be included in research reports prepared for sponsors inside and outside of NCES. The results may also be prepared for presentations related to survey methodology at professional meetings or publications in professional journals.

Information quality is an integral part of the pre‑dissemination review of the information disseminated by NCES (fully described in NCES’s Statistical Standards and a draft of the NCES Style Guide, both of which can be found at http://nces.eded.gov/pubsearch). Information quality is also integral to the information collections conducted by NCES and is incorporated into the clearance process required by the Paperwork Reduction Act.

During the past two years the generic testing clearance has been used to:

1. Pilot testing of a survey of teacher knowledge of primary grade reading instruction;

2. Sampling for the field test of the ECLS-K 8^th grade study;

3. Cognitive work in support of the 2007 National Indian Education Study;

4. The NAEP Word Locator Study for recruiting sample, printing materials, and other preparatory activities for this study.

5. Recently Approved: Fieldtesting a proposed extension of the National Assessment of Adult Literacy into measuring functional writing and vocabulary of adults.

3. Use of Information Technology

When the survey or assessment being pretested employs automated methods for its data collection, the research conducted under this submission will also use automated data collection techniques. This clearance offers NCES the opportunity to try innovative technologies that would reduce burden and increase the use of information technology.

4. Efforts to Identify Duplication

This research does not duplicate any other questionnaire design work being done by NCES or other Federal agencies. The purpose of this clearance is to stimulate additional research, which would not be done under other circumstances due to time constraints. This research will involve collaboration with staff from other agencies that are sponsoring the surveys conducted by NCES. The research may also involve joint efforts with staff from other Federal laboratory facilities. All efforts would be collaborative in nature, and no duplication in this area is anticipated.

To the maximum extent possible, we will make use of previous information, reviewing results of previous evaluations of survey data before we attempt to revise questionnaires. However, this information is not sufficient to refine our survey questionnaires without conducting additional research.

5. Minimizing Burden

This research will be designed as relatively small-scale data collection efforts. This will minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests.

6. Consequences of Less Frequent Collection

This clearance involves one-time questionnaire development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the surveys would suffer.

7. Special Circumstances

All the guidelines listed in the OMB guidelines are met. There are no special circumstances.

8. Consultations Outside the Agency

Consultation with staff from other Federal agencies that sponsor surveys conducted by NCES will occur in conjunction with the testing program for the individual survey. Consultation with staff from other Federal laboratory facilities may also occur as part of joint research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance.

9. Paying Respondents

Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance may receive a small stipend. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by the other Federal cognitive laboratories. Respondents for methods that are generally administered as part of field test activities (that is, split sample tests, behavior coding of interviewer/respondent interaction, and respondent debriefing) will not receive payment unless there are extenuating circumstances that warrant it.

Research on incentives that may be conducted under this clearance will likely also involve small nonmonetary incentives. The Office of Management and Budget has noted that effectiveness of such incentives is a worthwhile research topic. We will work closely with OMB on incentives proposed for each research activity.

10. Assurance of Confidentiality

All respondents who participate in research under this clearance will be informed that the information they provide is confidential and that their participation is voluntary. For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer. For personal visit interviews, respondents will also be notified in writing to give them something they can keep and read. For self-administered questionnaires, the information will be included in the mailing package, either on the questionnaire or the instructions. For Internet-based data collections, this information will be displayed prominently, and in a format that allows the respondent to print it out. All participants in cognitive research will be required to sign written notification concerning the voluntary and confidential nature of their participation. We will also inform respondents in writing of the need to have an OMB number. No personally identifiable information will be maintained.

Each respondent will be assured that all information identifying them or their school will be kept confidential in compliance with the legislation (Education Sciences Reform Act (P.L.107-110, 20 U.S.C. §9010):

“SEC. 183. CONFIDENTIALITY.

1. IN GENERAL.—All collection, maintenance, use, and wide dissemination of data by the Institute, including each office, board, committee, and center of the Institute, shall conform with the requirements of section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232g, 1232h).

2. (b) STUDENT INFORMATION.—The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232g, 1232h).”

11. Justification for Sensitive Questions

Most of the questions that are included on NCES questionnaires are not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.

12. Estimate of Hour Burden

We estimate that the number of people involved in our field test, pilot, cognitive, and focus group work will be at most 1,500 per year. We estimate the annual burden hours will be approximately 2.5 hours per person or an annualized 4000 hours. The total estimated respondent burden is 12,000 hours for the period from July 2005 through June 2008. These hours will be distributed as follows:

Respondent burden (hours)

July 2005 - June 2006 4000

July 2006 - June 2007 4000

July 2007 - June 2008 4000

Total 12000

A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time. However, we can project that our activities will likely include testing items, testing data collection modes, and testing incentive payment levels, in the form of “hothouse” field tests, expanded field tests including split sample questionnaire experiments in multiple panels, cognitive labs, reinterviews, and focus groups among students, teachers, parents, and other types of respondents.

13. Estimate of Cost Burden

There is no cost to respondents for participating in the research being conducted under this clearance, except for their time to complete the questionnaire.

14. Cost to Federal Government

There is no way to anticipate the actual number of participants, length of interview, and/or mode of data collection for the surveys to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by divisions conducting the research from their data collection budgets. We will include information about costs in the individual submissions.

15. Reason for Change in Burden

We anticipate no change in burden.

16. Project Schedule

This research program is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the subject of any printed NCES reports; however, it might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may, however, be prepared for presentation at professional meetings or publication in professional journals.

Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

There are no exceptions to the certification.