Summary of Interview Protocol Changes

Attachment B.
summary of interview protocol changes

Attachment B. Summary of Interview Protocol Changes

This table highlights key changes Insight made to the data collection instruments since the previous evaluation. In general, we expect the main areas of focus and the duration of the interviews to be the same. However, we have made some adjustments to reduce burden where possible, making the instruments more useful to the evaluation and more efficient for the grantees by standardizing certain sections across protocols and adjusting questions that respondents found challenging to answer in the past. At the interviewers’ discretion, individual questions may be skipped if they have already been addressed over the course of the discussion to further shorten the interview.

Grant Program/ Form Name	Protocol Section or Topic	Prior Evaluation	Proposed Evaluation
All	Introductory language	Though some language introducing the interview was similar across protocols, the introductory language varied from one protocol to the next.	The introductory text in each protocol was updated and standardized, using a common template adapted as needed for each grant program. This will help ensure a common understanding of the evaluation for all grantees, regardless of program.
All	Transition items	Some protocols addressed the transition to adulthood for young people with ASD/DD, but this was a minor focus (often just a question or two, or a probe).	The transition to adulthood is now a reporting requirement for MCHB under the Autism CARES legislation. Questions have been added throughout the protocols to obtain more information about grantee activities in this area, as it is now a primary objective of the evaluation.
All	Cross-cutting concepts	MCHB was interested in learning about a key set of concepts, or “buzzwords;” however, these were not formally documented in the interview protocols.	MCHB has approved a list of key “buzzwords” they would like to learn more about, and these key concepts have been systematically incorporated into the protocols. For example, MCHB would like to learn more about collaboration among grantees in this evaluation, so questions have been added throughout. This will ensure that relevant information is collected from the start, limiting the needs for follow-up and clarification with grantees.
All	National resource centers	The interview protocols contained questions for some grantees about their respective coordinating centers (ITAC for LEND and DBP; SPHARC for the State grantees). During the interviews, many grantees struggled to provide detailed responses to these questions.	Questions about the national resource centers have been adjusted and streamlined so that the questions are easier and quicker for interviewees to answer. Additionally, we refer to the resource centers primarily through their Autism CARES-specific role (ITAC/SPHARC).
All	Self-evaluation	Programs’ efforts to evaluate their activities were not captured systematically across protocols.	All protocols contain questions about grantees’ self-evaluation efforts, to supplement interview data and prevent further burden by providing interviewers with additional context for grantee activities.
DBP and LEND Interview Protocols	All	The DBP and LEND programs cover similar topics and often had similar questions, but some were worded differently.	Where possible, questions were standardized between the two instruments to ensure consistency, choosing the best format to improve the interview process and streamline the interview process in the event that co-located LEND and DBP programs choose to participate in a joint interview. For example, the “Increasing Awareness” section is now comparable for LEND and DBP.
DBP and LEND Interview Protocols	Wrap-Up	Section was very brief, with just a few general overarching questions.	Section has been expanded slightly to capture newly relevant information; many interviewees are returning grantees and will have rich insights to share about how the grant has affected their program over time. This section also provides a catch-all location for any questions or topics not fully addressed during the interview. In the DBP protocol we include headers so that interviewers can easily skip questions that are irrelevant or that were already covered over the course of the interview; for LEND this section is slightly shorter.
DBP and LEND Interview Protocols	Increasing awareness	Section contained a large number of questions.	Section was streamlined in both interview protocols to improve overall flow of interview and reduce number of repetitive questions.
DBP Interview Protocol	Contributing to research	This section came towards the end of the protocol, following the discussion of improving systems of care, and contained a large number of questions.	The section was moved earlier in the protocol, following the discussion of training activities, to improve overall flow. Past interviews indicate that research activities are a logical extension of training, as grantees typically discuss ways that they developed trainees’ research skills and involved them in research projects. The questions in this section were also streamlined to improve overall flow, reduce unnecessary or repetitive lines of questioning, and adjusting terms or references that were less relevant to grantees (for example, adjusting the way we asked about “translating research into practice”).
State Implementation Grant Program Interview Protocol	All	Several sections begin with the interviewer summarizing their understanding of the grantee’s activities based on review of noncompeting continuation applications and other relevant documents.	Interviewers will still review grantee reports before interviews so they can customize each interview and ask follow-up questions where needed; however, they are not specifically instructed to summarize grantee activities to reduce the amount of time grantees must spend listening, and to allow grantees to focus on the information they feel is most important.
State Implementation Grant Program Interview Protocol	Awareness	This section contained 8 questions about awareness activities, including an item focused on awareness related to family-centered medical homes.	Questions were condensed and streamlined; this section now contains 6 questions related to awareness with probes that can be asked only if necessary. The question about family-centered medical homes was removed; this topic will be addressed in the discussion of improving systems of care, where it is more relevant to grantees.
State Implementation Grant Program Interview Protocol	Training and reducing barriers	The interview protocol contained a separate section for training questions and questions about reducing barriers.	Because grantees often saw these activities as complementary, the two sections were condensed into one section called “Reducing Barriers to Screening, Referral, and Diagnosis Through Training.” Questions were edited and streamlined to better match the types of training efforts and barrier-reducing efforts that grantees typically discussed.
State Implementation Grant Program Interview Protocol	Improving systems of care	This section provided an overview of the State grantees’ overarching goals related to improving systems of care, and provided a series of probes that the interviewer could use at their discretion to explore activities related to each goal.	This section provides additional, specific questions to help guide the interviewer in exploring grantees’ goals, though the interviewer still has discretion to skip questions that have already been answered earlier in the interview or in progress reports. Questions were also added to address collaboration and other issues of importance to MCHB.
State Implementation Grant Program Interview Protocol	Transition from youth to adulthood	N/A	We added a brief section (2 questions) to ensure that the interviews adequately addressed transition, which is a new reporting requirement for MCHB.
State Implementation Grant Program Interview Protocol	AMCHP/SPHARC	This section asked about the activities of the National Resource Center that assists State grantees. We referred to that resource center as AMCHP, which is the name of the actual grantee.	This section still addresses activities of the National Resource Center that assists State grantees, but the questions have been adjusted to better reflect the kinds of activities States traditionally associate with the Center. This section also now refers to the Center as SPHARC (State Public Health Autism Resource Center), its official name, instead of the grantee’s name.
State Implementation Grant Program Interview Protocol	Wrap-up	Section was very brief, with just a few general overarching questions.	Like the LEND and DBP wrap-up sections, this section has been expanded slightly to capture additional summary information and provide a catch-all location for any questions or topics not fully addressed during the interview. We include headers so that interviewers can easily skip questions that are irrelevant or that were already covered over the course of the interview.
State Innovation in Care Integration Grantees	All	In the prior evaluation, all State grantees (State Implementation Grant program) being evaluated had the same grant guidance; only one interview protocol was developed for all State grantees.	In 2016, MCHB will issue a new group of State grants under newly updated grant guidance (the State Innovation in Care Integration grants). Because the new group of State grantees will have very different goals, they require a different interview protocol than prior State grant recipients. The protocol will be similar in length, but will better fit this group’s goals.
Research Network Interview Protocol	All	In the prior evaluation, the interview questions largely mapped to tables grantees were asked to complete prior to the interview as part of the research network questionnaire. However, this was not explicitly called out in the protocol.	To streamline the interview process for both interviewer and interviewee, the interview protocol has been reorganized, reformatted, and text added to more explicitly call out the tables from the research network questionnaire that are being discussed. Questions can be skipped if the information in the completed questionnaire does not need further clarification. Where appropriate, questions have been added to address new MCHB goals.
Research Network Questionnaire	All	Questionnaire tables were not numbered and did not have drop down response categories, requiring grantees to type out information by hand or enter check marks.	All tables now have a number and name to help grantees complete the questionnaire and to facilitate discussion of the tables during the follow up telephone interview. We have added drop down response categories to further reduce the burden on grantees of typing out information, and have provided additional instruction to help grantees navigate the instrument. We made minor adjustments to a few tables to capture more complete data and reduce follow-up interview time – for example, Table 2 now has a column where grantees can enter their collaborators on a given research project so that the interviewer does not need to spend additional time collecting this data during the interview.
Research Program R40 Interview Protocol	Conduct Research Leading to Evidence-Based ASD Interventions	Interview protocol was very brief and focused primarily on addressing goals, challenges, and the populations included in the research study.	This section was expanded to capture additional information about topics of interest to MCHB, including cultural competency and family involvement in research. These items are not lengthy, can be addressed quickly during the interview, and will allow the R40 grantees to better highlight their accomplishments.
Research Program R40 Interview Protocol	Disseminate Information to Health Professionals and the Public	This section contained one question about the final products the R40 grantees expected to disseminate from their research.	The question about research dissemination products was broken out to make it easier for grantees to respond, asking specifically about publications, conference presentations, and guidelines developed if applicable.
Resource Center: ITAC Interview Protocol (formerly AUCD protocol)	Technical assistance	This section did not contain questions about the LEND and DBP programs’ successes and difficulties; that was asked in a later section.	Questions about LEND and DBP programs’ successes and challenges were moved to the “technical assistance” section to improve overall flow, since ITAC typically helps these programs address these challenges and increase successes through TA.
Resource Center: ITAC Interview Protocol (formerly AUCD protocol)	Collaboration	This section did not contain questions about collaboration with SPHARC specifically (the other National Resource Center).	A question was added to collect information about ITAC’s collaboration with SPHARC, specifically. This question will provide important context for the evaluation but will require minimal time to address.
Resource Center: ITAC Interview Protocol (formerly AUCD protocol)	Data Collection and Management Activities	Section contained two questions about data ITAC collected and data ITAC helped grantees collect.	A question was added to ask about any challenges ITAC faced in collecting these data, and to provide a placeholder for other items requiring clarification. These questions require minimal time and like all questions can be skipped if adequately addressed elsewhere.
Resource Center: ITAC Interview Protocol (formerly AUCD protocol)	Resource Production and Dissemination	This section did not contain questions about challenges related to dissemination or feedback received from Autism CARES grantees.	We added questions to address challenges related to dissemination and inquire about feedback Autism CARES grantees provided to ITAC on their dissemination efforts. These questions require minimal time and can be skipped if already addressed.
Resource Center: ITAC Interview Protocol (formerly called AUCD protocol)	Wrap-Up	This section contained one general summary question.	This section was expanded to collect additional information about successes and challenges, and to provide a catch-all for any questions interviewers were unable to address earlier in the discussion. Many of these can be skipped if not needed, but these items provide an important opportunity to capture remaining data.
Resource Center: SPHARC Interview Protocol (formerly called AMCHP protocol)	All	Interview protocol was brief and contained only four sections addressing technical assistance SPHARC provides, the resources SPHARC disseminates, challenges and successes faced by the State grantees, and wrap-up items. It also referred to SPHARC by the grantee’s name (AMCHP)	Interview protocol was expanded to add a section on collaboration with other grantees and data collection and management activities. This will better address all the activities conducted by SPHARC and will mirror what is being collected for ITAC, the other national resource center. Throughout, we have standardized questions in these two protocols to ensure that comparable data is collected. These items can be skipped at the interviewer’s discretion, if already covered elsewhere.