NHES ATES 2014 Cog Labs Volume 1

Volume I

Request for Clearance for the

2014 National Household Education Survey (NHES)

Adult Training and Education Survey (ATES)

Cognitive Interviews

OMB# 1850-0803 v.82

July 22, 2013

Note:

This submission is for the NHES ATES portion of the cognitive interviews to be conducted in August 2013. However, it described both the NHES ASPA and ATES cognitive interviews as they are both part of NHES and will be conducted simultaneously, on the same schedule, and will use the same household screener that will determine individual’s eligibility for either ASPA or ATES. The clearance of cognitive interviews for these two surveys is submitted separately in order to facilitate their clearance and allow both to meet their very tight timeline (OMB has already reviewed the ATES questionnaire but has not yet had the chance to review the ASPA questionnaire). The request for clearance for the ASPA portion of the cognitive interviews will be submitted to OMB immediately upon OMB’s approval of the ATES portion.

Justification

Beginning in January 2014, the National Household Education Survey (NHES) will field a feasibility test for conducting the After School Programs and Activities (ASPA) survey and the Adult Training and Education Survey (ATES) simultaneously using the same data collection schedule, design, and household screener. The feasibility test will be a two-stage study. In the first stage households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling is performed. Finally, topical surveys are administered to the selected household members. The topical surveys in the feasibility test will be the ASPA survey for parents of children in kindergarten through 8^th grade and the ATES for out-of-high-school adults ages 16 to 65.

The purpose of this submission is to conduct cognitive interviews to refine the instruments for the 2014 data collection. Prior to cognitive interviews both the ASPA and ATES questionnaires have been reviewed by topical experts. Additionally, ATES conducted focus groups on characteristics of education certificates and participation in work-related training, and a response rate pilot study in early 2013, and is using early results to evaluate responses and skip patterns as input to questionnaire revision.

Cognitive testing has been used for ATES and other NHES surveys in past years. The objective of the interviewing in 2013 is to identify and correct problems of ambiguity or misunderstanding in question wording and respondent materials. The ASPA is a new topical component not yet fielded as a mail survey, and ATES had significant revisions to questions about work-related training and non-credit college courses. The cognitive interviews should result in a set of questionnaires that are easier to understand and therefore less burdensome for respondents, while also yielding more accurate information. The primary deliverable from this study will be the revised questionnaires. A report highlighting key findings will also be prepared.

The interviews will focus on three key areas:

Navigation. The NHES:2012 data collection results indicate that respondents had difficulty following certain skip patterns in the questionnaire (with regards to the school, child, and family questions). The cognitive interviews will look at alternative skip patterns, question formats, and evaluate the flow and order of the questions. The revision to the ATES component specifically also required different navigation and skip patterns, which will be evaluated through the cognitive testing.

Question wording. The ASPA topical module has never been fielded in the NHES as a mail survey, although it has been fielded previously as a telephone survey (in 2001 and 2005). Many of the school, child, and family questions will be the same as those tested and fielded recently in NHES:2012. Therefore, the cognitive interview probing will focus mainly on the new topical questions related to after school activities. In particular, the content experts recommended the use of a weekly grid layout to collect information about where the child spends time after school and on weekends. The grid format requires cognitive evaluation. Similarly, there have been significant revisions to the ATES work-related training and non-credit college classes sections and the addition of questions asking the respondent to list information about credentials in a grid format. These two components require cognitive evaluation.

Spanish translation. The ASPA instrument will be translated into Spanish and tested. The expanded household screener has not previously been tested in Spanish so we will use the ASPA interviews to also test its Spanish translation.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include:

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, and

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Interviews are expected to last about 1 hour and will be conducted by trained cognitive interviewers. This submission includes the screener and questionnaires to be tested, and preliminary protocols for conducting the interviews. The research will be iterative, in that question wording and forms design may change during the testing period in response to early findings.

To adequately test the instruments, it is necessary to distribute the cognitive interviews across respondents who can represent the major variations of experience in the target population and, correspondingly, to raise the total number of participants so as to obtain sufficient numbers of similarly-situated respondents.

We propose to conduct a total of 64 interviews, 24 for ASPA and 40 for ATES:

24 parents or guardians of students aged K-8 to participate in the ASPA cognitive interviews:

10 parents or guardians who speak only Spanish

14 parents or guardians who speak English as a primary language:

10 parents or guardians of a child whose family live in a suburban area and 4 in urban area

9 parents or guardians of a child enrolled in public school and 5 in private school

7 parents or guardians of a child enrolled in elementary school and 7 in a middle school

6 parents or guardians whose highest level of education is less than a Bachelor’s

40 adults ages 18 to 65 who are part of the work force (e.g., not retired, not full-time students), with an emphasis on adults who have less than a Bachelor’s degree to participate in the ATES cognitive interviews:

     At least 9 adults who have at least one educational certificate

     At least 9 adults who have at least one professional certification or license

At least 9 adults who have participated in work-related training in the past 12 months

At least 4 adults who have taken at least one college-level course in the past 12 months

At least 2 adults who have taken at least one ESL course in the past 12 months

     At least 2 adults who have taken at least one GED preparation course in the past 12 months

     At least 5 adults who are not asked whether they have these credentials (to represent “general population”)

At least 2 adults who are self-employed

At least 2 adults who are unemployed

Note: The sum of the categories listed is greater than the total because several respondents should fall into multiple categories.

Consultations Outside the Agency

For ASPA, NCES consulted with 8 experts on children’s after school activities in June 2013, who made recommendations for the survey content and reviewed the resulting draft survey. For ATES, NCES has an ongoing consultation with a government interagency working group – Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) – that has been involved with the survey design since its inception. Additionally, GEMEnA conducted 3 focus groups in January 2013 on characteristics of educational certificates and in May 2013 on work-related training to assist in questionnaire development. Finally, NCES conducted a nationally-representative response rate pilot for adults in early 201, the National Adult Training and Education Survey or NATES, and is evaluating response options and skip patterns based on the early results.

Recruiting and Paying Respondents

To assure that participants from all desired populations agree to take part in the cognitive interviews and to thank them for their time and for completing the interview, each will be offered $40.

Participants will be recruited by AIR, and its subcontractors, Nichols Research and Shugoll Research, using multiple sources, including company databases and personal contacts. An example recruitment e-mail is included at the end of this document. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. The items used to screen respondents for participation are included in the submission. We anticipate it will take 3 minutes per screening interview. Interviews will take place in the DC-Metro area and the San Mateo, CA area.

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality pledge before interviews are conducted. This statement is as follows: “The American Institutes for Research is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (ESRA, 20 U.S.C. §9543). Your participation is voluntary. Your responses are protected from disclosure by federal statute (20 U.S.C. §9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.”

No personally identifiable information will be maintained after the cognitive laboratory analyses are completed.

Estimate of Hour Burden

We expect the cognitive interviews to be approximately one hour in length. Most ATES participants will only be exposed to a portion of the protocol because their education and training will only make them eligible to complete certain portions of the questionnaire. For ATES, this will result in 40 burden hours for cognitive interviews. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 480 screenings to yield 40 participants). This will result in 24 hours of burden for the screener, and an estimated total 64 hours of respondent burden for this research.

Table 1. Estimated response burden for ATES Cognitive Interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	480	480	0.05	24
Cognitive Interviews	40	40	1.0	40
Total	480	520	-	64

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for the cognitive research focusing on the topical interviews to begin by August 5, and last until September 5. After the cognitive interviews are completed, data collection instruments will be revised and submitted to OMB in early fall 2013 as part of a request for clearance for the January 2014 nationally-representative feasibility study.

Email text for personal and professional contacts and advertisements

Dear [contact]:

The American Institutes for Research and Shugoll Research/Nichols Research are assisting the National Center for Education Statistics, of the U.S. Department of Education, and other federal statistical agencies in learning more about [job education, skills, training, preparation, and other similar qualifications people may have] [children’s after-school activities]. We are specifically looking for [adults] [parents] to participate in a one-time, voluntary, research interview about these topics.

Participants will take part in a 60-minute cognitive interview on [DATE] and will receive $40 for their participation.

If you or someone you know is interested in participating please contact [contact info].

Thank you for your time and help with this important study.

Sincerely,

Recruitment Contact Info