Supporting Statement Part A clean 06-15-2020

NCEE System Clearance for

DESIGN AND FIELD STUDIES 2020-2023

OMB# 1850-XXXX v.X

Supporting Statement Part A

Prepared by:

National Center for Education Evaluation

Institute of Education Sciences

U.S. Department of Education

Washington, DC

April 2020

TABLE OF CONTENTS

Part A JUSTIFICATION

A.1 Importance of Information 1

A.2 Purposes and Uses of the Data 3

A.3 Improved Information Technology (Reduction of Burden) 3

A.4 Efforts to Identify Duplication 3

A.5 Minimizing Burden for Small Institutions 4

A.6 Frequency of Data Collection 4

A.7 Special Circumstances 4

A.8 Consultations Outside the Agency 4

A.9 Payments or Gifts to Respondents 4

A.10 Assurance of Confidentiality 5

A.11 Justification for Sensitive Questions 5

A.12 Estimates of Burden 5

A.13 Total Annual Cost Burden 5

A.14 Annualized Cost to Federal Government 5

A.15 Program Changes or Adjustments 5

A.16 Plans for Tabulation and Publication 5

A.17 Display OMB Expiration Date 6

A.18 Exceptions to Certification Statement 6

1. JUSTIFICATION

1. Importance of Information

This is a request for a 3-year generic clearance for the National Center for Education Evaluation (NCEE) that will allow it to collect preliminary or exploratory information to aid in study design. Specifically, the clearance will enable NCEE to: (1) field brief, quick turnaround surveys, extract test case administrative data, administer interviews, or conduct “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation (e.g., for a Federal Student Aid Experimental Site evaluation, a short collection to identify the conditions of an experiment in which an interested institution of higher education intends to participate); and (2) develop, test, and improve its survey and assessment instruments, methodologies, and study dissemination strategies.

Other Centers within IES have found that similar generic clearances (e.g., NCES’s use of OMB #1850-0803 v.248) to be helpful vehicles for evaluating data collection plans and procedures, questionnaires, and assessments. It has allowed NCES to take advantage of a variety of methods to identify questionnaire, assessment, and procedural problems in advance, suggest solutions, and measure the relative effectiveness of alternative solutions. Through the use of these techniques, employed routinely in the testing phase of NCES surveys, questionnaires and assessments have been simplified for respondents, respondent burden has been reduced, and the quality of the questionnaires and assessments used in continuing and one-time surveys and assessments has been improved. As a result, NCES reports that an increase in the quality of the data collected through these surveys has been achieved as well. Similar testing clearances are held by the Census Bureau, the Bureau of Labor Statistics, and the National Science Foundation.

NCEE hopes to accrue the same benefits for its impact evaluations and large-scale program implementation studies, if allowed the opportunity to do so. The procedures expected to be used include but are not limited to exploratory surveys and interviews, focus groups, cognitive laboratory activities, pilot testing versions of an intervention or data collection approach, small-scale experiments that explore questionnaire design, incentives, or mode, and usability testing. Conducting these activities, and in particular conducting them earlier and more quickly, would enable NCEE to improve both the quality and turnaround time of its policy relevant evaluations, many of which are congressionally-mandated with ambitious dates for required reporting.

This request for generic clearance provides a description of the scope of possible activities that might be covered and the procedures for keeping OMB informed about them. The generic request will go through the regular two Federal Register posting periods and OMB review and approval, subsequent to which, NCEE asks that OMB review and clear specific proposed generic information collection activities within a two-week period with no Federal Register Notice period required under the generic clearance. This is similar to NCES’s generic clearance process.

The methods proposed for use in NCEE study design and questionnaire and assessment development are as follows:

Design interviews. These may be conducted with individuals either in person, virtually, or via survey. The purpose would be to understand a topical area (such as a policy or program) or an intervention and may be used in the very early stages of developing an experiment or program evaluation and its accompanying data collection materials. They may cover discussions related to administrative records (e.g. what types of records are available, where, and in what format), subject matter, potential study sample, definitions, possibilities for random assignment, implementation approaches or challenges and so forth in order to assess study design and feasibility. Exploratory interviews may also be used in evaluating whether there are sufficient issues related to an existing data collection to consider a redesign.

Field test. For the purposes of this clearance, we are defining field tests as data collection efforts conducted among either purposive or statistically representative samples, for which the main objective is gathering critical information needed to inform the design of an experimental or program implementation evaluation. This may include collecting data as part of a pilot of an intervention, experimental study procedures, or data collection procedures. Field test or pilot collections are an essential component of this clearance package because they serve as the vehicle for ensuring that federal investments in education evaluations, including large-scale data collections, are feasible and appropriate and yield reliable and valid information. Under this clearance a variety of instruments may be used or tested as part of a pilot, and the exact nature of the surveys and the samples is undetermined at present. However, due to the smaller nature of the tests, we expect that some will not involve representative samples. In these cases, samples will basically be convenience samples, which may be limited to specific geographic locations or a subset of those potentially interested in participating in a larger-scale experimental evaluation. The samples may involve expired rotation groups of current survey blocks that are known to have specific aggregate demographic characteristics. The needs of the particular sample will vary based on the type of field test, including the respondent, content of the instruments, or data collection methods, but the selection of sample cases will not be arbitrary in any instance.

Behavior coding. This method involves applying a standardized coding scheme to the completion of an interview or questionnaire, either by a coder using a tape-recording of the interview or by a "live" observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give inadequate answers, this suggests there are other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire, and research has demonstrated that this is a more objective and reliable method of identifying problems than the traditional interviewer debriefing, which is typically the sole tool used to evaluate the results of a traditional field test (New Techniques for Pretesting Survey Questions by Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989).

Interviewer debriefing. This method employs the knowledge of the employees who have the closest contact with the respondents. In conjunction with other methods, we plan to use this method in our field tests to collect information about how interviewers react to the survey instruments.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the instruments being tested and contains questions that probe how respondents interpret survey questions or administrative data extraction requests and whether they have problems in completing the field test data collection.

Follow-up interviews (or reinterviews). This involves re-interviewing or re-assessing a sample of respondents after the completion of a survey or assessment. Responses given in the reinterview are compared with the respondents’ initial responses for consistency. In this way, reinterviews provide data for studies of test–re-test reliability and other measures of the quality of data collected. In turn, this information aids in the development of more reliable measures.

Cognitive and usability interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents have answering questions. This is frequently the first stage in revising a questionnaire.

Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues that are nuanced and conditional on factors not easily anticipated.

Procedures for Clearance

Before any activity is undertaken, NCEE will provide OMB with a generic information collection that includes a supporting statement describing the testing or exploratory data collection to be conducted and its intended purpose. The supporting statement will be similar to the “Volume I Supporting Statement” that NCES provides for each generic information collection under the NCES generic clearance. If relevant, a copy of instruments or debriefing materials will be included. Depending on the specific activity, this may be a printed questionnaire, a set of prototype items, or an interview script. When mini-experiments are conducted, either in small group sessions or as part of a field test, the different version instruments or interventions will be provided or described. NCEE requests that OMB raise comments on substantive issues within 10 working days of receipt.

Data collection for this project is authorized under the legislation authorizing NCEE evaluations. In most cases, this is the Education Sciences Reform Act (ESRA 2002, 20 U.S.C. §9563), though other legislation (e.g., HEA, 20 U.S.C. § 1094a) or Title 13 or 15 may apply for exploratory or pilot data collections conducted in concert with other offices within ED or other Federal agencies. If collections rely upon authority other than ESRA, the relevant authorizing statute will be specified.

2. Needs and Uses

The information collected under this program will be used by staff from NCEE and sponsoring agencies to design high-quality evaluations of federal education programs and improve the quality of surveys and assessments that inform them before those studies are conducted. None of the data collected under this clearance will be published for its own sake, though data collected through this clearance may be used during full-scale experiments and related data files and publications when appropriate. Data will not otherwise be made public, except when included in research reports prepared for sponsors inside and outside of NCEE. The results may also be prepared for presentations related to evaluation or survey methodology at professional meetings or publications on NCEE’s website and in professional journals.

Information quality is an integral part of the pre‑dissemination review by NCEE, which abides by NCES’s Statistical Standards and IES Style Guide (http://nces.ed.gov/statprog/standards.asp). Information quality is also integral to the information collections conducted by NCEE and is incorporated into the clearance process required by the Paperwork Reduction Act (PRA).

3. Use of Information Technology

To the extent possible NCEE will use automated data collection techniques, such as when the survey or assessment being tested employs automated methods for its data collection or to assess the utility of machine learning for qualitative data coding. This clearance offers NCEE the opportunity to try innovative technologies that would reduce burden and increase the use of information technology.

4. Efforts to Identify Duplication

Information collection and analysis under this clearance will not duplicate any other feasibility and design work being conducted by NCEE or other federal and non-federal agencies that may collaborate in this work. Additionally, to the extent possible, NCEE will make use of existing information, including reviewing results of previous evaluations of instruments, methods, and interventions to build upon past efforts.

5. Minimizing Burden

This research will be designed as relatively small-scale data collection and design efforts so as to minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests. The results of the research conducted under this clearance are expected to inform and improve the interventions, methods and instruments utilized in full- scale studies and thereby improve information quality while minimizing burden to respondents.

6. Consequences of Less Frequent Collection

Without the exploratory and design tests proposed as part of this collection, the quality of the full experiments, quasi-experiments, and implementation evaluations would suffer, as would our ability to design studies in a timely manner to meet policy goals.

7. Special Circumstances

There are no special circumstances.

8. Consultations Outside the Agency

Consultation with staff from other Federal agencies that sponsor surveys conducted by NCEE will occur in conjunction with testing individual surveys. Consultation with staff from other Federal cognitive laboratory facilities may also occur as part of joint exploratory research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance. One nonsubstantive public comment was received.

9. Paying Respondents

Respondents for activities conducted in the laboratory (e.g. cognitive interviews and focus groups) under this clearance may receive compensation for travel and participation. This practice has proven necessary and effective in recruiting subjects to participate in such research and is also employed by the other federal cognitive laboratories. Research on incentives that may be conducted under this clearance may also involve nonmonetary incentives. The Office of Management and Budget (OMB) has noted that effectiveness of such incentives is a worthwhile research topic. If incentives need to be proposed for any research activity under this clearance, justification will be provided and NCEE will work closely with OMB on the incentive strategy to be employed. NCEE will typically propose incentives at the level approved by OMB for cognitive laboratories and focus groups. If a higher incentive amount is proposed for approval, a meaningful justification will be provided.

10. Assurance of Confidentiality

If the collection is under the authority of Education Sciences Reform Act of 2002 (ESRA 2002), all respondents who participate in research under this clearance will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer, and in personal visit interviews respondents will also receive this information in writing. For self-administered questionnaires, the information will be included in the mailing package, either as part of communication materials or on the questionnaire or instructions. For Internet-based data collections, this information will be displayed prominently, and in a format that allows the respondent to print it out. All participants in cognitive research will be required to sign written notification concerning the voluntary and confidential nature of their participation. NCEE will also inform respondents in writing of the need to have an OMB number. No participant direct identifiers will be maintained as part of research under this generic clearance.

11. Justification for Sensitive Questions

Questions that are included in NCEE data collections are typically not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in instruments that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.

12. Estimate of Hour Burden

We estimate that the number of people involved in our exploratory, field test, pilot, cognitive, and focus group work will be at most 4,000 per year, the vast majority of which will be contacted as part of screening and recruitment activities preceding the actual research. Given that screening and recruitment activities are included in the burden calculations, we estimate the annual burden hours will be approximately 0.5 hours per person or an annualized 2,000 hours overall. The total estimated respondent burden is 6,000 hours for the 3-year period beginning on the date of OMB approval in 2020:

Time Period	Respondents	Responses	Respondent burden (hours)
2020 – 2021	4,000	4,000	2,000
2021 – 2022	4,000	4,000	2,000
2022 – 2023	4,000	4,000	2,000
Total	12,000	12,000	6,000

A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time. Similarly, the exact respondents are unknown but are likely to include students, teachers, parents, schools, local and state agency officials.

We estimate that the average number of generic information collections that OMB may expect to receive under this clearance is approximately 10 submissions each year.

13. Estimate of Cost Burden

There is typically no cost to respondents for participating in the research being conducted under this clearance, except for their time to complete the requested activity.

14. Cost to Federal Government

There is no way to anticipate the actual number of participants, duration, and/or mode of data collection to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by divisions conducting the research from their data collection budgets. We will include information about costs in the individual submissions.

15. Reason for Change in Burden

No change to the estimated respondent burden is being requested.

16. Project Schedule

This research program is for study design, instrument and procedure development purposes. Data tabulations will be used to evaluate the results of intervention, questionnaire, and methods testing. The information collected in this effort will typically not be the subject of population estimates or other statistics in NCEE reports; more often, it may be published in research and development reports or be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may be prepared for presentation at professional meetings or publication on NCEE website and in professional journals. Due to the nature of this clearance, there is no definite or tentative time schedule for individual testing activities at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

There are no exceptions to the certification.