Grantee, Contractor, and Local Evaluator Protocol

Grantee, Contractor and Local Evaluator Protocol

Discussion Guide

Administration and Areas of Assessment for the Grantee, Contractor, and Local Evaluator Protocol

Part 1 will identify your knowledge about and/or the role(s) you have played in seven tasks (listed below) associated with the data flow sequence of the 2008–2009 GPRA and non-GPRA performance measures, including evaluation data.

A. Derivation, analysis, and reporting of GPRA and non-GPRA data	E. Grantee collection and submission of data
B. Provision of guidance regarding submission of performance data	F. Data quality checks and validation of grantee data
C. Provision of training	G. Dissemination and use of program performance results
D. Provision of technical assistance

First, we will define each of the seven tasks and ask if you or some other person have had any knowledge about or role in these seven tasks. If you indicate that you did have knowledge about or a role in a particular task(s), we will ask you questions in a specific module(s) for that task(s) in Part 2. If you indicate that you did not have knowledge about or a role in a particular task(s), we will ask you: 1) who we should talk to in order to gain insight into how this task(s) was performed and 2) how we can contact this person.

In addition, we will ask if you had knowledge about or were involved with any other tasks that were not included in tasks A through G. If you indicate “yes”, we will ask you to describe this other task in detail so that we can ensure that it is not already covered by questions in one of the seven modules in Part 2. If it is determined that the other described task is legitimately different from tasks currently listed, then we will ask you to share your knowledge and role as it relates to this unlisted task.

Task Definitions and Areas of Assessment Regarding Task Roles and Knowledge

Below are the definitions of the seven tasks (listed above and defined below) associated with the data flow sequence of the 2008–2009 performance measures. Below these definitions are the topic areas that we will inquire about, if it is determined that you had knowledge about or a role in these tasks.

Task A. Derivation, Analysis, and Reporting of GPRA and Non-GPRA Data

Derivation, analysis, and reporting of GPRA and non-GPRA data, including evaluation data, are defined as the processes by which the data were selected, analyzed, and reported during the 2008–2009 reporting cycle. We would like to discuss:

A1. Description of each GPRA and non-GPRA measure your organization collected

A2. Process for determining what GPRA and non-GPRA performance data your organization collected

A3. Evaluation plan for GPRA and non-GPRA data

A4. Analysis of GPRA and non-GPRA data

A5. Use of GPRA and non-GPRA results

A6. Development of written report and who received it

A7. Changes in GPRA and non-GPRA data that your organization collected since the 2008–2009 reporting cycle

Task B. Provision of Guidance Regarding the Submission of Performance Data

The provision of guidance regarding submission of GPRA and non-GPRA performance measures, including evaluation measures, is defined as the process by which instructions, policy information, and definitions were provided to grantees on how to submit performance data for the 2008–2009 reporting cycle. We will discuss:

B1. Description of guidance

B2. Methods by which guidance was provided

B3. People within your organization who received the guidance

B4. When guidance was provided during the 2008–2009 program cycle

B5. Problems interpreting the 2008–2009 performance measure guidance provided by the program office

B6. Usefulness of guidance provided by program office

B7. Changes in the types of guidance provided by the program office or the methods by which they were provided since the 2008–2009 reporting cycle

Task C. Provision of Training

Provision of training is defined as training events, classes, or materials that were designed to help grantees with their reporting of GPRA and non-GPRA program performance data, including evaluation data, for the 2008–2009 reporting cycle. We will discuss:

C1. Description of trainings provided

C2. Primary mode used to provide the training (conferences, webinars, etc.)

C3. When trainings occurred during the 2008–2009 data reporting cycle

C4. Who within the grantee organizations received the training

C5. Usefulness of training

C6. Changes in the type of training(s) provided or the procedures by which training(s) were implemented or provided since the 2008–2009 reporting cycle

Task D. Provision of Technical Assistance

Provision of technical assistance (TA) is defined as any materials or activities that were used to assist grantees in their efforts to report GPRA and non-GPRA program performance data, including evaluation data, during the 2008–2009 reporting cycle. TA differs from training in that it occurs when grantees seek assistance about specific program reporting issues that they may be experiencing, and that it is usually provided to one grantee at a time. We will discuss:

D1. Description of TA provided by the program office

D2. Primary mode used to provide the TA

D3. When the TA occurred during the 2008–2009 data reporting cycle

D4. Who within the grantee organization received the TA

D5. Usefulness of TA provided by the program office

D6. Changes in the type of TA provided or the procedures by which TA was implemented or provided since the 2008–2009 data reporting cycle

Task E. Grantee Collection and Submission of Data

Grantee collection and submission of data is defined as the processes that grantees used to collect and report GPRA and non-GPRA program performance data, including evaluation data, to the program office during the 2008–2009 data reporting cycle. We will discuss:

E1. Processes grantees used to collect data to be used for the calculation and submission of 2008–2009 performance data

E2. Processes grantees used to submit performance data to the program office

E3. Availability—prior to report deadline—of data your organization needed to report on 2008–2009 performance

E4. Extent of time it took to review 2008–2009 performance results and to correct any problems before submitting them

E5. Whether your organization submitted performance results on time

E6. Problems encountered in collecting data used for the performance reports for the 2008–2009 reporting cycle

E7. Problems encountered in calculating data used for the performance reports for the 2008–2009 reporting cycle

E8. Problems encountered in reporting accurate performance results for the 2008–2009 reporting cycle

E9. Feedback provided by the program office with regard to 2008–2009 collection and submission process

E10. Changes in the procedures to collect and submit data since the 2008–2009 reporting cycle

Task F. Data Quality Checks and Validation of Grantee Data

Data quality checks and validation of grantee data are defined as any procedures—during the 2008–2009 reporting cycle—that the program office had in place to check that the data they received: 1) was of sufficient quality to calculate all the performance measures and 2) accurately captured the program performance measures required for grantees. We will discuss:

F1. Who assessed the quality and consistency of your 2008–2009 performance data

F2. Challenges with regard to data edits, data cleaning, or any other automated processes used to assess the accuracy of your 2008–2009 performance data

F3. Challenges with regard to the validation of participant level data that was used to calculate and report 2008–2009 performance data

F4. Changes in the procedures or activities used to assess the reliability or validity of the performance data since the 2008–2009 reporting cycle

Task G. Dissemination and Use of Program Performance Results

Dissemination and use of program performance results are defined as the process by which the program office distributed the performance results of the program for the 2008–2009 reporting cycle. This includes the policy context and background information that were provided along with program results. We will discuss:

G1. Program office contacts regarding your reported 2008–2009 performance results

G2. Receipt of aggregated 2008–2009 performance reports for your program

G3. Use of the analysis of the 2008–2009 performance data

G4. Changes in the use of the program office’s or your organization’s performance results since 2008–2009 reporting cycle

G5. Changes in the procedures or activities associated with generating performance results since the 2008–2009 reporting cycle

Task H. Other Tasks Associated with the Data Flow Sequence of the 2008–2009 Performance Measures

Other tasks are defined as tasks—associated with the 2008–2009 performance measures—that involved activities and processes not described in the seven data tasks we previously defined. We will discuss:

H1. Person(s) responsible for conducting task

H2. Description of activities to conduct this task

H3. When this task occurred during the 2008–2009 data reporting cycle

H4. Problems experienced with regard to conducting this task for the 2008–2009 performance reporting cycle

H5. Changes associated with conducting this task since the 2008–2009 performance reporting cycle

Wrap Up – Closing

1. Do you have any questions that you would like to ask us?

2. Do you have any other suggestions or ideas to add that may inform us or lead to improvements in the flow sequence of perform