State CN Agencies and Local Education Agencies (Directors and Key Staff) - SLT

OMB Approval No. 0584-####

Expiration Date: XX/XX/20XX

Appendix B3. Cognitive Testing Protocol

Introduction

Good morning/afternoon. Thank you again for taking the time to talk with me today. My name is [INTERVIEWER NAME], and I work for Westat, a private research company in Rockville, Maryland. Joining me today is [NAME], who will be taking notes.

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) hired Westat to conduct a study to better understand the second, independent review of applications (IRA) for school meal programs, including the process at the State and LEA level, and assess the effectiveness of that process to reduce certification errors. The main study will include a survey of all State Child Nutrition Directors, telephone interviews with 30 LEA Directors and staff, and a review of household applications for 20 LEAs. Before we conduct the surveys and interviews on a broader scale, we want to test them to ensure that the questions are easy to understand and to answer.

Informed Consent

Before we get started there are a few things I want to mention.

• This is a research study which means that your participation in this discussion is voluntary.

• Refusal to participate will not have any impact on your position or child nutrition programs.

• The thoughts and opinions that you share during our discussion will be used for research purposes only. Your responses will not be used as an evaluation of your work or your staff’s work.

• There are no direct benefits to you for participating in this study.

• Your participation and feedback are important and needed because input from staff like yourself help us refine our [survey/interview questionnaire] to better explore the second review process in different states.

• The names and affiliations of those who participate in these cognitive tests will be listed in the memorandum we submit on these cognitive tests to FNS. The memorandum will explain how we used the feedback from cognitive testing to refine the instruments. Your name will not be associated with your individual responses.

• Once we start the interview, you can take a break, skip questions, choose to make comments off the record, or stop participating at any time. Again, there will be no penalty or loss of benefits to your program.

• Finally, any electronic notes we have will be stored on Westat’s secure server, and any hard copy notes will be stored in a locked filing cabinet. All notes will be destroyed after the project is complete.

• If you have any questions about your rights as a participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about (Evaluation of the IRA study), and a phone number beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last about 90 minutes. Do you have any questions? [ANSWER ALL QUESTIONS]

Do I have your consent to participate?

Procedures

I will ask the questions in each section, and ask that you respond for your own [Region/State/LEA]. I will ask questions as we proceed in order to hear your reactions to the overall content, wording, and flow of the questions.

ADMINISTER COGNITIVE INTERVIEW. NOTE BEGIN TIME: __________________

NOTE DELIVERY, COMPREHENSION AND RESPONSE ISSUES BY TRACKING DIRECTLY ON INTERVIEW GUIDE.

NOTE ANY VERBAL REACTION RESPONDENT SHARES FOR FOLLOW-UP DURING THE DEBRIEF.

FOR ANY NON-VERBAL REACTIONS (E.G., LONG PAUSES), ASK ONLY, Tell me what you’re thinking here. DO NOT PROBE BEYOND THAT UNTIL THE DEBRIEF.

COGNITIVE TESTING RESEARCH QUESTIONS TO REMEMBER:

• Do respondents understand and interpret the questions as intended?

• Does the sequencing of the questions flow well?

• Do the questions apply to all States/LEAs or do we need to tailor certain questions to each respondent?

• Are the questions appropriate for each type of respondent?

• Is there any information about the application collection, review, and submission process that is missing from either the survey or the interview guide?

USE RELEVANT PROBES BELOW TO DISCUSS OBSERVED ISSUES.

Clarification Requests

You asked [FILL] for question XX. Can you say more about what you were thinking there?

Hesitation

Can you say more about your hesitation at question XX?

Confusion

What was it about question XX that seemed confusing to you?

IF NEEDED, What information would you say [CONFUSING TEXT] is asking for?

Answer Changes

What made you decide to change your answer at question XX?

If R said question did not apply

For question XX you said it did not apply. Can you say more about that?

If R could not answer

For question XX, you said you couldn’t answer. Can you say more about that?

Debriefing Questions

NOTE END TIME: _________________________

Thank you. Now I’m going to ask you some questions about the interview overall. Tell me your overall impressions of the questions I just asked you.

Is there anything else from your earlier review of the questions that we have not talked about today?

Which questions did you think were hardest to answer? What makes you say that?

Which questions did you think were easiest to answer? What makes you say that?

Those are all the questions I have for you. Is there anything we haven't discussed that you would like to mention?

Thank you for your time.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXX. The time required to complete this information collection is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.