Interview Protocol For Grantee Project Directors, Data Managers, and Evaluators

Interview Protocol for Grantee Project Directors, Data Managers, and Evaluators

[Note: Different individuals may respond to different modules (sections) of this protocol, per the interviewer instructions below. Capital letters signify information that interviewer must insert as appropriate for that respondent. Italics are interviewer instructions and are not to be spoken aloud.]

Hello, thank you for making the time to talk with us about your program supported by [NAME OF GRANT]. My name is [Name]. As I explained in the cover letter about this study, I work for Abt Associates, a research organization based in Cambridge, MA. We are conducting a study for the Department of Education that examines the influence of the Department’s technical assistance activities on grantees’ performance reporting and evaluations. We are focusing on grants within the Charter Schools Program: State Educational Agencies (CSP SEA) program and the Voluntary Public School Choice (VPSC) program.

Participation in the interview is voluntary and nonparticipation will have no impact on you or your grant. The interview will take no more than one hour of your time. The intent of this interview is not to evaluate the performance of your grant. Rather, we are trying to understand how grantees perceive the TA support that they receive; how that TA affects their performance reporting and evaluations; and how those activities, in turn, affect the way they manage their programs.

We are interviewing up to 20 grantees from the CSP SEA and VPSC programs, and we will be aggregating the responses within each program, in order to understand what seems most effective in supporting grantees. No individuals or grantee organizations will be identified by name in our reports.

We would like to audiotape our conversation so that we can listen at a later time for points we might miss during the conversation. Is that okay with you? While we cannot assure confidentiality, every attempt will be made to not provide information that identifies a subject or district to anyone outside the study team, except as required by law.

Before we begin, do you have any questions?

[If respondent has questions that interviewer is unable to answer, offer to pass the inquiry to Teresa Doksum, Abt Associates’ Institutional Review Board Administrator, or Ellen Bobronnikov, Project Director, or offer their contact information if respondent prefers.]

I. Respondent Background

Let’s start with a few questions about your involvement in the [CSP SEA OR VPSC] grant. To be clear, we are talking about the grant that started on [DATE] and extends to [DATE].

I.1. What is your role in the grant? [Probe: What are your major responsibilities?]

1a. What is your role with respect to performance reporting and/or evaluation activities specifically? [How much of your time is spent on them? [Average percent of time is ok. Ascertain if individual is central or peripheral to performance reporting/evaluation efforts.] From our document review, we understand that you conducted the following types of performance reporting and evaluation activities? [Recap the type of evaluation and performance reporting activities conducted]. Does this sound accurate to you? Is there anything you would like to add?

I.2. Who is the most appropriate person or persons for me to talk to about ….

2a. any TA that your organization received since the grant began on the subject of conducting evaluations and performance reporting? [These are respondents for Section II. If there is an independent, external evaluator, interview separately from other respondents.]

2b. data collection, data reporting, and evaluation (if appropriate)? [These are respondents for Section III. If there is an independent, external evaluator, interview separately from other respondents.]

2c. how the information from performance reports and evaluations is used to manage and plan for programs? [These are respondents for Section IV. If there is an independent, external evaluator, interview separately from other respondents.]

2d. Is there anyone else involved in performance reporting or evaluation activities for this grant?

IF YES: What is his/her role?

II. Technical Assistance Received

[If appropriate, repeat introduction and question I-1]

I am interested in learning about any technical assistance your organization has received regarding performance reporting or evaluation. We are interested in TA from any source, not just the Department of Education, in the period since your [CSP SEA OR VPSC] grant began.

II1. Since your grant started in [YEAR], did your organization receive any TA from any source? [Probe for all types of TA, including workshops and seminars but also conference presentations, evaluation handbooks, one-on-one consultations, etc.]

IF NO, SKIP TO SECTION IV.

II.2. What TA did you receive since [DATE], when your Department of Education grant was awarded?

II.3. (RQ1) Let’s talk a little about each of those. For [LIST EACH TA RECEIVED], could you please tell me:

3a. What was it? [Probe for workshop, handbook, individualized consultation]

3b. (RQ1a) Who funded it? (Department of Education, foundation, other)

3c. (RQ1a) Who provided it?

3d. Who from your organization received it?

3e. What topics did it address?

3f. (RQ1b) How was it delivered? [Probe for, e.g., in-person group setting, one-on-one consultation, webinar, phone conference call, on-demand service, written materials.]

II.4. Looking back over all of the TA you have received in this period, did you learn anything about conducting assessment/evaluations that was new to you?

IF YES: What stands out for you as most helpful or important that you learned?

II.5. (RQ2d) Do you feel the TA has changed how you think about the performance reporting/evaluation activities you conduct?

IF YES: How? In what way? [Probe for perceived quality improvements in things such as:

• Conceptualizing the intervention

• Defining objectives

• Defining measures that speak to the objectives

• Study design (method/approach)

• Data collection]

5a. (RQ2d) Can you point to specific TA activities that you feel were responsible for this ?

IF YES: Which one(s)? [We aim to (1) identify whether specific TA activities were particularly useful, rather than the cumulative effect of multiple TA interventions, and (2) whether the helpful ones were those sponsored by the Department of Education or by some other source.]

II.6. (RQ2d) Did the TA lead you to make any changes in any of your specific performance reporting or evaluation activities?

IF YES: What did it lead you to change? [Probe for changes such as those below, and for specific examples:

• different way of conceptualizing the intervention (logic model)

• different program goals or objectives

• different measures are being tracked (e.g., student achievement or school quality)

• different data items are being tracked (e.g., math proficiency scores, percentage of schools making AYP)

• different way of collecting data (e.g., survey, existing database)

• different way of analyzing the data (changes in evaluation design, e.g., switch from simple monitoring to comparison group.]

6a. [RQ2d) Can you point to specific TA activities that were responsible for this?

IF SO: Which ones? [We aim to (1) identify whether specific TA activities were particularly useful, rather than the cumulative effect of multiple TA interventions, and (2) whether the helpful ones were the ones sponsored by the Department of Education or by some other source.]

II.7. Did any changes in your performance reporting or evaluation—including, but not only, the ones we were just talking about—happen for reasons other than TA?

IF YES: Please tell me about that: how important were those other reasons compared to TA? [Probe for changes in budget, gain/loss of staff expertise or time, program changes, etc.]

II.8. (RQ2d) How useful was the TA you received from any source? What worked well? Less well? Do you have any suggestions for improvement? [If necessary, clarify that this includes TA offered by sources other than the Department of Education, so long as it concerned performance reporting and/or evaluation, and occurred in the period of the Department of Education grant.]

II.9. (RQ2d) Please think about the kind of TA that would be most useful to you to improve the quality of your performance measurement/evaluation? Specifically …

9a. What kinds of topics would be most useful to cover? Why is that? What topics are least helpful? Why?

9b. What methods of delivering TA are most useful to you? (webinars, group in-person gatherings, one-on-one, etc.). Why is that? What methods are least useful? Why?

II.10. IF RESPONDENT HAS SUBGRANTEES: Based on the TA you have received so far, have you made any changes, or do you plan to make any changes, to your evaluation requirements for sub-grantees?

III. Challenges and Successes Related to Data Collection and Evaluation

[If appropriate, repeat introduction and question I-1]

We have talked a bit about general TA you received. Now, I would like to talk about the TA you received in the context of specific aspects of your performance reporting or evaluation.

III.1. What challenges do you face in performance measurement or evaluation? [Probe for specifics.]

III.2. What aspects of performance measurement or evaluation proceed smoothly/are not challenging? [Probe for specifics.]

We have carefully reviewed all the documents available to us about your grant. I would like to ask you about some aspects of your strategy, specifically about whether they were STRAIGHTFORWARD or challenging, and about any TA you might have received with that aspect of your strategy.

[For questions III.3–9 below, only ask about those items in which grantees had particular challenges or in which they had more success than other grantees.]

III.3. Let’s talk about the performance measures you decided to examine in order to track whether you are meeting your objectives. [If necessary, give examples of grantee’s own objectives and measures.]

Did you receive any TA from the Department of Education on this topic? Did you receive any TA from any other source?

IF YES:

3a. Did you receive the TA before or after defining your measures for this grant?

3b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

3c. Did it lead you to make any changes in what you plan to do?

IF YES: What were the changes?

3d. How straightforward or challenging was it to define the measures?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

III.4. Let’s talk about providing data for GPRA reporting. [If necessary, review what these are].

Did you receive any TA from the Department of Education on GPRA reporting requirements and how to comply with them? From any other source?

IF YES:

4a. Did you receive the TA before or after defining your measures for this grant?

4b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

4c. Did it lead you to make any changes in what you plan to do?

IF YES: What were the changes?

4d. How straightforward or challenging was it to comply with the GPRA reporting requirements?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

III.5. Let’s talk about selecting data sources or instruments you are using to collect data for each measure. [If necessary, give examples from grantee’s own documents, for example, using math proficiency scores as an indicator of student achievement in math.]

Did you receive any TA from the Department of Education on how to select appropriate data sources or instruments? From any other source?

IF YES:

5a. Did you receive the TA before or after defining your measures for this grant?

5b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

5c. Did it lead you to make any changes in the instruments or data sources you chose?

IF YES: What were the changes?

5d. How straightforward or challenging was it to identify appropriate instrument or data sources for each of your measures?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

III.6. Let’s talk about how you assure the completeness of the data that you are collecting. [If necessary, give examples from grantee’s own document about how they will assure high response rate for surveys, or low number of missing data for administrative data, etc.]

Did you receive any TA from the Department of Education on how to assure completeness of data? From any other source?

IF YES:

6a. Did you receive the TA before or after defining your measures for this grant?

6b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

6c. Did it lead you to make any changes in what you plan to do?

IF YES: What were the changes?

6d. How straightforward or challenging was it to ensure data completeness?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

Is grantee conducting an evaluation of any kind? If no, skip to Section IV. Else, ask questions III.7–9.

III.7. Let’s talk about your choice of study design for the evaluation you are conducting. Your grant documents say that you planned to … [Interviewer paraphrases study design, e.g., a pre/post analysis, a comparison group approach, a random-assignment approach, etc.]

How did you decide to use that approach? (Probe for the process and factors that resulted in the study design chosen, for example: grantee duplicated what had been done before, grantee wanted to but could not do more because of resource constraints, grantee consulted a professional evaluator, grantee decided to use this approach based on experience or on TA received.]

7a. Did you receive any TA from the Department of Education on how to select the best research design for your purposes? From any other source?

IF YES:

7b. Did you receive the TA before or after defining your research design?

7c. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

7d. Did it lead you to make any changes in your research design?

IF YES: What were the changes?

7e. How straightforward or challenging was it to decide on your research design?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

7f. Did you consider conducting an impact evaluation [DEFINE IF NECESSARY].  Why/ why not?  

IF YES:

7g. Did you encounter any challenges or barriers to implementing this type of study? [Probe]

7h. What strategies did you find useful in implementing this type of study? 

7i. What advice would you give others attempting an impact evaluation of the type you conducted?

III.8. Let’s talk about the timing of your data collection compared to when the program changes, or “intervention,” is happening. [If necessary, give examples from grantee’s own documents about when they propose to collect data.]

Did you receive any TA from the Department of Education on what to consider in timing data collection? From any other source?

IF YES:

8a. Did you receive the TA before or after defining your measures for this grant?

8b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

8c. Did it lead you to make any changes in the timing of your data collection?

IF YES: What were the changes?

8d. How straightforward or challenging was it to decide on the timing of data collection?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

III.9. Let’s talk about how you selected the comparison group. [If necessary, give examples from grantee’s own documents, for example, recap what the comparison group is.]

Did you receive any TA from the Department of Education on how to select an appropriate comparison group and test for baseline equivalence for the study design you chose? From any other source?

IF YES:

9a. Did you receive the TA before or after selecting your comparison group?

9b. Was the TA you received from the Department of Education helpful in this regard? Why/why not?

9c. Did it lead you to make any changes in the selection of your comparison group?

IF YES: What were the changes?

9d. How straightforward or challenging was it to select your comparison group?

IF CHALLENGING: What was challenging about it? Why?

IF STRAIGHTFORWARD: What made it straightforward? [Probe for reasons such as in-house expertise, had done it before, etc.]

IV. How Grantee Uses Data/Findings

These next questions have to do with the “big picture” as regards performance reporting and evaluations. Specifically, I am interested in learning about how useful, or not, this kind of information is in managing your programs, and how the Department could offer support in this regard.

IV.1. (RQ3a) Let’s talk a bit about the [DATA/FINDINGS] you have, and what you do with them. How useful are the [DATA/FINDINGS] to you in managing your current programs?

IF NOT USEFUL: Why is that? What would have made it more useful?

IF USEFUL: (RQ3a) How do you actually use the [DATA/FINDINGS] with respect to managing your current programs? How does the information shape your thinking? [Probe for specific examples e.g., helps define a target subgroup for more/less attention, prompts a change in curriculum, identifies a need for different teacher training, etc.]

1a. What is most helpful to you about the [DATA/FINDINGS] in this regard? What is less helpful?

IV.2. (RQ3a) How useful are the [DATA/FINDINGS] to you in planning future programs?

IF NOT USEFUL: Why is that? What would have made it more useful?

IF USEFUL: (RQ3a) How do you actually use the [DATA/FINDINGS] with respect to managing your current programs? How does the information shape your thinking? [Probe for specific examples per question IV.1 above.]

2a. What is most helpful to you about the [DATA/FINDINGS] in this regard? What is less helpful?

IV.3. (RQ3b) Are there any obstacles or barriers that keep you from using the [DATA/FINDINGS] more, or more effectively? [Probe for issues such as lack of staff expertise, lack of time to assimilate the findings, lack of budget to implement the findings, etc.]

IV.4. (RQ2c) In an ideal world, would you be [MONITORING YOUR PERFORMANCE/CONDUCTING YOUR EVALUATION] differently from the way you are now, or basically the same?

IF DIFFERENTLY:

4a. What would you be doing differently? Could you give me some specific examples? [Probe for different study design, data collection method, time frame, sample size, analysis method, etc.]

4b. What are the main obstacles or barriers that keep you from doing that? [Probe for lack of expertise, lack of resources, lack of time, etc.]

4c. Is there any TA support that the Department of Education could offer to help make that possible?

IF YES: What would that be?

IV.5. Are you doing any evaluations outside the scope of this grant?

IF YES: Could you send me a copy of that if findings are available?

If grantee is conducting an evaluation:

IV.6. Have you received any findings from your evaluation yet?

IF YES:

6a. When did you receive the findings?

6b. May we have a copy?

6c. Are these the final findings or interim findings?

IF INTERIM: When are the final findings due?

Those are all my questions. Thank you very much for your time!