10-Day Letter (Request for Approval)

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

September 28, 2020

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) (OMB No. 0920-0222, Exp. Date 08/31/2021) plan to conduct a cognitive interviewing study to evaluate a subset of self-administered questions on the National Survey of Children’s Health (NSCH) for the Health Resources and Services Administration (HRSA). The proposed subset of NSCH questions, known as “Healthy and Ready to Learn,” underwent an extensive review and revision in late 2019. HRSA’s Maternal and Child Health Bureau (MCHB) developed and adopted the “Healthy and Ready to Learn” questions and associated indices in the absence of a “gold standard” and with the expectation that the content would evolve over time (Ghandour et al. 2018)¹. The questions have not been otherwise tested since that review, and it is not certain whether, and to what extent, the questions capture the constructs required by HRSA, whether there are redundancies across questions, and how the questions function across various groups, such as child age groups and parental/guardian employment status. The cognitive interviews will explore these avenues in order to help develop standard, comprehensive measures of young-children’s school readiness.

Background: NSCH “Healthy and Ready to Learn” Questions 2020 (Self-administered/English)

The NSCH is an annual, address-based household survey that produces data for key indicators of physical, emotional, and behavioral health of children ages 0-17. The Healthy and Ready to Learn content was first added to the NSCH in 2016 at the request of state-level stakeholders, researchers, and practitioners from around the country. The content was designed to support National Outcome Measure 13, School Readiness, for the Title V Maternal and Child Health Services Block Grant. Previously, there was no single data source that provided a multidimensional, population-based assessment of young children’s school readiness. HRSA’s MCHB developed the Healthy and Ready to Learn questions to fill that gap and develop such a measure for children 3-5 years old. The questions cover four distinct, yet complementary, domains: Early Learning Skills, Physical Wellbeing and Motor Development, Social-Emotional Development, and Self-Regulation. Individual item responses in each domain are scored as “at risk,” “needs support,” or “on-track,” according to age-specific abilities for 3-, 4-, and 5-year-olds (Child Trends 2019). The items scores are then added to create a summary assessment for each domain.

In 2019, the “Healthy and Ready to Learn” questions underwent a re-assessment by a technical expert panel and by the research center Child Trends. The 62 questions that are being submitted for cognitive testing by the CCQDER fall into three categories: items included in prior versions of the NSCH, items revised from prior versions of the NSCH, and newly suggested items.

Fourteen questions to be tested were included on prior versions of the NSCH. These were selected and refined after a review of content from existing surveys such as the National Household Education Survey on School Readiness (National Center for Education Statistics 2007) ², the Early Childhood Longitudinal Study (National Center for Education Statistics 2010-2011) ³, the Early Development Instrument (Offord Centre for Child Studies 2012-2013)⁴, the Australian Early Development Census (Commonwealth of Australia 2012)⁵, previous iterations of the NSCH and the National Survey of Children with Special Health Care Needs (U.S. Department of Health and Human Services et al. 2015)⁶, and the Well Visit Planner based on Bright Futures guidelines for promoting early childhood development (Bethell 2008; 2)⁷. Additionally, HRSA’s MCHB engaged in iterative consultation, meetings, and workgroups with researchers, Federal partners, and stakeholders (Ghandour et al. 2018)⁸. While these items have been used on prior versions of the survey, they have not undergone in-depth cognitive testing within the context of the NSCH.

Twenty questions have been revised from prior survey questions and twenty-eight are new. These questions were revised or added to the “Healthy and Ready to Learn” content as a result of an extensive re-assessment and review that occurred in late 2019 by a technical expert panel and the research institution Child Trends. The purpose of the revisions and additions was to bolster and improve the quality and depth of the four question domains and summary measures.

Plan of Study

The methodological design of the proposed study is consistent with the design of most NCHS/CCQDER cognitive interviewing studies. The purpose is to identify various patterns of interpretation that respondents consider when formulating an answer to a survey question, as well as any difficulties, confusion, or response error that may occur during administration. Interviews are in-depth and semi-structured. The NSCH is designed to be completed at home; therefore, interviewers will provide a copy of the questionnaire to the respondents so they may complete it at the start of each interview (paper copy if in-person or digital document via shared screen if virtual). After respondents finish filling out the survey, interviewers will then begin the semi-structured discussion of survey responses. Analysis will be conducted using the constant comparative qualitative method and will focus on the constructs captured by each question, consistency of patterns across respondent groups, and potential causes of response error. Findings (like all CCQDER studies) will be documented in a final report and made publicly accessible on a searchable website at https://wwwn.cdc.gov/QBank.

Research Questions:

Findings from previous CCQDER studies have shown that parents’ interpretations of questions about their children vary according to their lived experiences and circumstances (Willson 2007⁹; Willson et al. 2018¹⁰). Overall, the proposed cognitive testing for the “Healthy and Ready to Learn” questions will examine constructs captured by survey questions on young children’s (ages 3-5) readiness to start school and how the child’s age and the work status of the parent/guardian influence interpretive patterns. Specific research questions include the following:

Question Redundancy

Are there redundancies in the constructs captured across multiple questions? If so, how can the questionnaire be reduced to be more concise while maintaining depth and breadth within the question domains?

Child’s Age

Do the questions capture consistent constructs for parents/guardians of children ages 3, 4, and 5? Or, are there differences in interpretation and functionality of the questions based on the child’s age?

Question Interpretation and Response Judgment

On what basis do parents make judgments about their children’s health and development? Do these judgments vary by different groups of parents? If so, how?

Study Protocol

The 62 NSCH questions to be evaluated are included as Attachment 1. Because there are more items included than can be adequately explored in a single cognitive interview, initial probing will prioritize the new items, followed by the revised, and finally the original items. As interviewing progresses, attention will be shifted to emerging themes and response difficulties associated with any given item or section. The testing procedure conforms to the cognitive interviewing techniques that have been described above and in CCQDER’s generic clearance package (OMB No. 0920-0222, Exp. Date 08/31/2021).

We propose to recruit 60 adults (aged 18 and over) who are the parent or guardian of a child aged 3-5 years old for interviews lasting up to 60 minutes conducted in-person in the Questionnaire Design and Evaluation Research Laboratory (QDRL), off-site, or virtually through either the Skype or Zoom video conferencing applications. Parental, and especially maternal, employment that is stressful and time-intensive may detrimentally affect school readiness (McPherran Lombardi and Levine Coley 2014¹¹). Stressful or time-intensive employment may also affect how respondents answer the questions. Therefore, we aim to recruit respondents with diverse employment statuses (including working part-time, multiple jobs, or contracting) in order to examine possible differences in question interpretation. Secondarily, we aim to recruit respondents with a roughly even mix of age, race, and educational attainment. The initial goal is to recruit groups in equal proportion, to the extent possible – that is, within the constraints of those willing to participate in the study.  However, because qualitative sampling is based on theoretical relevance more than equal cell sizes, on-going analysis may reveal the need to recruit more from one group than others. 

Recruitment will be carried out through a combination of advertisements, flyers, word-of-mouth, and CCQDER Respondent Database. The advertisement/flyer used to recruit respondents is shown in Attachment 2. The fifteen-minute screener used to determine eligibility of individuals responding to the advertisement/flyer is shown in Attachment 3a. The ten-minute screener used to determine eligibility of individuals from the CCQDER Respondent Database is shown in Attachment 3b. Note that the wording of the template has been approved and is contained within our umbrella package. Only project-specific information has been added to the recruitment and screener documents. It is anticipated that as many as 120 individuals (from flyers and database) may need to be screened in order to recruit 60 participants.

During the screening process, recruiters will inform respondents of consent form, video recording and remuneration procedure. For virtual interviews, recruiters will also assess respondents’ video conferencing capabilities and will work with them to set up the video conference application (Zoom or Skype).

Additionally, for virtual interviews, respondents will be asked if they are willing to take part in a post-interview debriefing discussion, as shown in Attachment 7, lasting no more than 5 minutes. Since the outbreak of COVID-19, CCQDER staff have been exploring ways to conduct qualitative research remotely. Respondent feedback will help to improve and refine remote interviewing protocols and procedures.

Interviews averaging 60 minutes (including the completion of a Respondent Data Collection Sheet (Attachment 5a and 5b)) will be conducted by CCQDER staff members or Strategic Innovative Solutions (SIS) contractors. All interviews – those conducted in the Questionnaire Design and Evaluation Research Laboratory, off-site, or virtually – will be video and audio recorded to allow researchers to review the behaviors and body language of the respondents. These recordings will allow researchers to ensure the completeness and accuracy of their interview notes.

In accordance with the CCQDER Data Storage and Access Policy, CCQDER has determined that this project falls in the “unrestricted” category. Therefore, interviews will be video recorded, and researchers from CCQDER, SIS contractors, and staff from the Health Resources and Services Administration’s Maternal and Child Health Bureau (MCHB) who are working on the project, will use the recordings for research purposes. SIS contractors and MCHB staff viewing/listening to recordings in the QDRL under CCQDER supervision have read and signed a non-disclosure affidavit.

Video or audio recording is required for this project except in the rare case that a study participant initially agrees to be video recorded during the telephone screening, but the participant changes their mind. In that case, they will be asked if they agree to be audio recorded. If they decline to be audio recorded the interview will proceed without recording. In this case the interviewer will depend on their handwritten notes when conducting analysis. In addition, individuals who select “yes” for allowing the recording on the informed consent form, but “no” for retaining the recording for future research (final text before signatures on informed consent form), will be allowed to participate in the study.

NCHS government issued encrypted laptops will be used to video and audio record the interviews conducted off-site. Due to the size of the video recordings, the internal drive of the encrypted laptop is not sufficient for storage of the recordings. Recordings will be saved to an NCHS government issued encrypted flash drive. The encrypted flash drive is FIPS 140-2 compliant and approved for use by OCISO.

CCQDER staff and SIS contractors will also use the NCHS government issued encrypted laptops to input their interviewer notes or notes from the respondent follow-up discussions into Q-Notes. Within 24 hours, a CCQDER staff member will review “Healthy and Ready to Learn” 2020 interview notes and will delete any direct or indirect personal identifiable information (PII) if found.

Extreme care will be taken with all recordings and paperwork from the interviews conducted off-site and virtually. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets. Once the video and audio recordings are transferred to the QDRL Network, the recordings will be deleted from encrypted flash drive. Once deleted, the files are no longer available for use.

Initial retention period of the audio recordings is 5 years after project completion. After the initial retention period, the recordings will be re-evaluated by the CCQDER Director to determine relevance, ongoing usefulness, and qualitative value for likely use in question evaluation research. If it is determined by the CCQDER Director in conjunction with CCQDER project-relevant staff that there is no valid reason to retain the recording, it will be destroyed by designated CCQDER staff. If the interview continues to be of value (defined as ongoing use by research staff, topic relevance, likely use for federal questions evaluation research), reassessment of the recording will occur again in 5 years.

In person:

After respondents have been briefed on the purpose of the study and the procedures that CCQDER routinely takes to protect human subjects, respondents will be asked to read and sign an Informed Consent (Attachment 4a). Only project specific information has been added to the document. Respondents will also be asked to fill in their demographic characteristics on the Respondent Data Collection Sheet (Attachment 5a). This document is contained in our umbrella package.

Virtual:

For virtual cognitive interviews, respondents and interviewers will see and hear each other through the video conference software from their own computer, tablet or cellphone from their respective locations. A recruiter will meet the respondent on the scheduled video conferencing appointment (through Zoom or Skype). The recruiter will make sure the respondent is prepared for the interview and that the video conferencing application is working properly. The recruiter will go over the informed consent information that was previously emailed to the respondent (Attachment 4b) and remind the respondent of the remuneration procedure. If the application fails to work properly, the recruiter will use this script to cancel the interview.  Script: Unfortunately, we cannot move forward if your [video conference application/Skype/Zoom] is not working properly because these interviews will be conducted over video chat. I’m sorry, we have to cancel this interview.  Would it be okay if I added your name, telephone number, age, educational level, and race to our database so that I can contact you about other studies coming up in the future?  If yes, add to database.  If no: OK, thank you for your time.  Your name and any information you gave me will not be added to our database.

Once the videoconferencing software is working properly, the recruiter will collect the information from the Respondent Data Collection Sheet. (Attachment 5b). At this point, the interviewer will join the video conference and the recruiter will leave the meeting.

The interviewer will then ask the respondent to confirm that he or she understands the information in the Informed Consent, and then state that we would like to record the interview. The recorder will be turned on once it is clear that the procedures are understood and agreed upon.

The interviewer will then orient the respondent to the cognitive interview with the following introduction:

[Fill staff name] may have told you that we will be working on some questions that will eventually be added to national surveys. Before that happens, we like to test them out on a variety of people. The questions we are testing today are about your child’s learning, development, and health.

We are interested in your answers, but also in how you go about making them. I may also ask you questions about the questions—whether they make sense, what you think about when you read certain words, and so on.

I will give you the questionnaire, [if virtual: by sharing my screen with you], and you can fill it out as if you received the survey in the mail at home. I’d like you to answer the questions as best you can. Please tell me if:

there are words you don’t understand,

the question doesn’t make sense to you,

you could interpret it more than one way,

it seems out of order

or if the answer you are looking for is not provided.

Afterwards, we will talk about your answers, and I will ask what you were thinking as you figured out how to answer. The more you can tell us, the more useful it will be to us as we try to develop better questions. Okay? Do you have any questions before we start? [If yes, answer questions. If not,] let’s get started.

[If virtual] I will now share my screen with you. Please go ahead and fill out the form, scrolling down as necessary.

If the interview is virtual, when the respondent has finished filling out the form, the interviewer will end the screen-sharing mode and begin discussing respondent answers.

The interviewer will ask the respondent to complete the questionnaire. He or she will also ask emergent probes to better understand the question-response process. Examples of the sorts of probes that may be asked at the interviewer’s discretion include:

• Could you tell me what [term] means to you?

• Why did you answer that way?

• Can you give me an example?

• In your own words, can you tell me what you think this question is asking?

• Was this question easy or hard to answer? Why?

• How sure are you about your answer?

Interviewers may use some or all of these probes, depending upon the content of the interview and how much information the respondents reveal without being prompted. All probes will be administered after the respondent has finished the questionnaire.

After the interview, respondents will be given the thank-you letter (document contained in the umbrella package) signed by the Director of NCHS (Attachment 6), a copy of the informed consent document, and $40. For interviews conducted virtually, the recruiter will send the respondent a “thank you” email, informing them that they will receive a hard copy “thank you” letter and $40 in cash via FedEx within 7 business days.

Virtual Interview Follow-up:

Respondent follow-up: During the screening process for virtual interviews, respondents will be asked if they are willing to participate in a brief follow-up discussion. If the respondent had agreed to participate in a brief follow-up discussion, a member of our operations team will call (by phone) within 2 business days after the interview to conduct the discussion (Attachment 7). The follow-up discussion will not be recorded, but the person conducting the follow-up will take notes. No additional incentive payment will be made to respondents in this regard.

The discussion will be qualitative in nature to include the following topics:

• Familiarity with video-calling platform used.

• Difficulties respondents may have had understanding the set-up instructions.

• Difficulties respondents may have had accessing or using the video-calling platform.

• Assessment of sound quality during interview.

• Assessment of the visual quality (if a camera was used) during interview.

• Level of respondent engagement during the interview - were there any distractions such as the TV, phone, or other people.

• How the respondent felt about the experience immediately after taking part in the interview.

Interviewer follow-up: In order to further inform our understanding of the effectiveness of conducting cognitive interviews remotely, the interviewers will also provide feedback. They will make note of any technical issues that may have arisen as well as make note of their overall impression of the interview format. Topics for the post-interview notes may include the following:

• Describe any set up or technical issues that impacted your ability to conduct the interview effectively.

• Comment on the respondent’s level of engagement during the interview.

• Comment on the ease with which you were able to obtain the respondent narrative.

• Did anything else, in your view, impact the quality of the data obtained?

We propose giving participants $40 for their participation, which is our standard incentive. In total, for this project, the maximum respondent burden will be 93 hours. A burden table for this project is shown below:

Form Name	Number of Participants	Number of Responses/ Participant	Average hours per response	Response Burden (in hours)
Screener (recruited from advertisement/flyer)	80	1	15/60	20
Screener (recruited from respondent database)	40	1	10/60	7
Questionnaire	60	1	55/60	55
Respondent Data Collection Sheet (QDRL in-person interviews or offsite)	5	1	5/60	1
Respondent Data Collection Sheet (virtual interviews)	60	1	5/60	5
Respondent follow-up	60	1	5/60	5
Total	93

Attachments (7)

cc:

S. King

J. Zirger

DHHS RCO

1 Ghandour, R., Moore, K., Murphy, K., Bethell, C., Jones, J., Harwood, R., Buerlein, J., Kohan, M., Lu, M.. (2018). School Readiness among U.S. Children: Development of a Pilot Measure. Child Indicators Research. 12(4), 1389-1411.

2 National Center for Education Statistics. (2007). National Household Education Survey on School Readiness. https://nces.ed.gov/nhes/pdf/sr/07_sr.pdf Accessed 2/11/2020.

3 National Center for Education Statistics. (2010-2011). Early Childhood Longitudinal Study. https://nces.ed.gov/ecls/ Accessed 2/11/2020.

4 Offord Centre for Child Studies. (2012-2013). Early Development Instrument. https://edi.offordcentre.com/ Accessed 2/11/2020

5 Commonwealth of Australia. (2012). Australian Early Development Census. http://www.aedc.gov.au/ Accessed 2/11/2020

6 U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, and National Center for Health Statistics. (2015). State and Local Area Integrated Telephone Survey (SLAITS). https://www.cdc.gov/nchs/slaits/index.htm Accessed 2/11/2020.

7 Bethell, C. (2008). Patient Centered Improvement of Well-Child Care: Developing and Evaluating the Impact of Patient-centered Interventions to Improve Quality and Equity of Recommended Services.

8 Ghandour, R., Moore, K., Murphy, K., Bethell, C., Jones, J., Harwood, R., Buerlein, J., Kohan, M., Lu, M.. (2018). School Readiness among U.S. Children: Development of a Pilot Measure. Child Indicators Research. 12(4), 1389-1411.

9 Willson, S. (2007). Cognitive Interview Evaluation of the Pregnancy Risk Assessment Monitoring System (PRAMS) Phase 6: Results of interviews conducted August-October 2007. National Center for Health Statistics. Hyattsville, MD. https://wwwn.cdc.gov/qbank/Reports.aspx#/Reports/1057 Accessed 2/11/2020.

10 Willson, S., Schoua-Glusberg, A., Corado, D., and Titus, A.. (2018). Cognitive Interview Evaluation of Opioid-Related Questions for Inclusion on the Pregnancy Risk Assessment Monitoring System. National Center for Health Statistics. Hyattsville, MD. https://wwwn.cdc.gov/QBank/Report.aspx?1201. Accessed 2/11/2020.

11 McPherran Lombardi, C. and Levine Coley, R. (2014). Early Maternal Employment and Children’s School Readiness in Contemporary Families. Developmental Psychology. 50 (8) 2071-2084.

7 | Page