Interview with Nongrantee Evaluators

Attachment 8B

Non-Grantee Evaluator Interview Guide

Form Approved:

OMB No.: 0920-XXXX

Expiration Date: XX/XX/XXXX

Interview Guide

Program Evaluator for Nongrantee Program

Public reporting burden of this collection of information is estimated to average one hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0920-XXXX)

Hello. My name is [full name], and I’m with ICF Macro. We are working with the Centers for Disease Control and Prevention, or CDC, to better understand how States implement their colorectal cancer (or CRC) screening efforts and how these efforts change over time. This interview should take no more than 1 hour of your time. Let me share a copy of our Informed Consent Form with you; you can follow along as I read it aloud.

[Provide the participant with a copy of the Informed Consent Form. Read the form aloud.]

We want to make sure you know that your answers to any questions that I ask you during this interview will be treated in a secure manner. In evaluation reports, manuscripts, and other materials developed to disseminate information about the evaluation, descriptive data for each program will be presented so that various program models can be shared. Specific quotes will not be attributed to any particular individual.

I want to ask for your permission to audiotape our interview. This will allow me to capture all the details of our discussion. The tape will be kept secure and used only by the evaluation team to supplement the notes we take. Your name will be removed from the notes and transcript.

Do you have any questions?

Let’s begin.

Background

1. Can you tell me about your roles and responsibilities in your State health department’s CRC efforts?

2. When and how did you become involved with your State health department’s CRC efforts?

3. Do you work full- or part-time with your State health department’s CRC efforts?

4. What would you say are the goals of the State’s colorectal cancer efforts?

5. What are your specific evaluation duties?

6. How has your involvement in your State’s health department’s CRC efforts changed over time?

7. If the evaluator is employed by another agency—Please briefly describe your agency and how long it has been involved with [name of State]’s health department, as well as with work related to colorectal cancer control and prevention.

Implementation

Now we want to ask you a few questions about your involvement in the implementation of [name of State]’s health department’s CRC efforts. We are particularly interested in understanding what resources, including funding, staff, partnerships, evaluation, and technical assistance, are needed to evaluate the implementation of your State health department’s CRC efforts.

Conducting Evaluation

1. Did you participate in the development or identification of any tools, such as a logic model, an action plan, or an implementation plan, used to guide the implementation of the State health department’s CRC efforts? If so, please describe your participation and the tools.

2. Does [name of State]’s health department have an evaluation plan that specifically addresses the health department’s CRC efforts? If so, how was it developed?

3. Please briefly describe the evaluation plan.

• What role did you play in its development?

• To what extent does the evaluation plan include process measures? To what extent does it include outcome measures?

• Does the evaluation plan include an explicit evaluation of the effectiveness of the State health department’s CRC efforts? If so, what indicators and data sources have been identified?

4. Please describe the process that the State health department uses to make evaluation-related decisions about its CRC efforts?

• Who is involved? What are their roles?

• Are there regularly scheduled evaluation meetings? If so, who participates?

• What are the strengths of this process?

• What are the weaknesses?

• Has this process changed over time? If so, please describe the changes.

5. What additional data, if any, does [name of State]’s health department use to aid in the evaluation of its CRC efforts?

• Do you have data sharing agreements with any partner organizations? If so, please describe.

6. To date, what evaluation activities have been conducted? [Probe by asking about the following:]

• Partnership development/satisfaction

• Building legislative or leadership support for CRC screening

• CRC screening provision/CRC promotion

7. What activities have been most useful? Least useful?

8. Which activities have been the most difficult to conduct?

9. In what ways, if any, have the evaluation findings been used to inform decisions about program planning or implementation? Please provide examples.

10. Have you conducted any evaluation activities related to building State-level legislative or leadership support for colorectal cancer screening? If yes, please briefly describe.

• What are the available data sources?

• What data collection tools or instruments are you using?¹

• How were these evaluation findings used?

• Did you encounter challenges in evaluating these activities? If so, how did you overcome those challenges?

• Can you give examples of how the evaluation findings have been used for program improvement? If so, please describe the examples.

11. Are you aware of any competing issues or priorities that have affected any of your evaluation efforts? If so, please describe them.

12. What strategies have been used to minimize the impact of those competing issues?

Evaluation Infrastructure

1. Besides you, are there other evaluators currently involved in the evaluation?

• If so, what organizations are they affiliated with?

• What are their responsibilities?

• Were they hired specifically to evaluate the State health department’s CRC prevention efforts? Were there any difficulties in hiring them?

• Do you know how their positions are funded? If so, please describe the funding.

• How do all of the evaluation activities fit together?

• How do you envision other evaluators’ responsibilities changing over time?

2. What roles do State health department’s staff members play in evaluating program activities?

• Are they involved to the extent they should be?

• What role, if any, do partners play in evaluation efforts?

3. Have recruitment and retention of evaluation staff been a problem? If so, please describe the problems experienced.

4. What strategies, if any, are in place to minimize turnover or its impact on the program?

5. Are evaluation activities adequately staffed and funded?

6. Do you anticipate needing additional evaluation staff in the future? If so, please describe what role they would play.

7. What resources are used to support the evaluation of the State health department’s CRC prevention efforts? Please describe these resources.

Technical Assistance and Training

1. What technical assistance or training, if any, have you received to assist with your evaluation efforts for the State health department’s CRC prevention efforts?

2. What were the topics of the technical assistance or training?

• Who provided it? Who else participated in the technical assistance or training?

• Was it useful?

• What additional technical assistance or training, if any, would be helpful to you in carrying out your responsibilities related to evaluating the State health department’s CRC prevention efforts?

3. Have you or any other evaluators provided any technical assistance or training to partners or others affiliated with the State health department’s CRC prevention efforts? Please describe what was provided.

• What was the topic?

• Who was involved in the development and delivery of the technical assistance or training?

• To whom was it provided?

• Were there any barriers or facilitators to providing the technical assistance or training?

• How was it funded?

• Was feedback obtained from participants? If so, what were the results?

Conclusion

Now we would like to hear your thoughts on how well you think the State health department’s CRC prevention efforts are working.

1. In your opinion, what are the key strengths of the State health department’s CRC efforts?

2. In your opinion, what are the key limitations of the State health department’s CRC efforts?

3. What has been accomplished by the State health department’s CRC efforts that otherwise would not have been?

4. Overall, do you think the State health department’s CRC efforts use an effective approach to improve colorectal cancer screening? If so, why? If not, why? Is there another approach you would recommend using? Why?

5. What concerns, if any, do you have about future efforts to evaluate the State health department’s CRC prevention efforts?

6. Is there anything else you would like to say about the State health department’s CRC prevention efforts that we have not discussed?

Those are all the questions I have for you at this time. Before we wrap up, do you have any questions for me about anything we’ve talked about?

Thank-you again for taking the time to speak with me. We sincerely appreciate and value your input!

1 Request to see or have copies made of data, data collection tools, or evaluation reports.