Word version of Survey of Asthma Program Evaluators

National Asthma Control Program:

Evaluation Self-Assessment Instrument

Section II

ATTACHMENT – C: Data Collection Instrument Part 1 – Word version

SURVEY OF STATE and TERRITORIAL ASTHMA PROGRAM EVALUATORS

Form Approved

OMB No. 09200879

Expiration Date: 03/31/2014

Introduction

Program evaluation offers a set of tools and skills for ensuring that public health programs operate as effectively as possible. As evaluators, we can use a similar approach to ensure that our evaluation practice produces information that is useful and accurate and that our methods meet the standards for quality and propriety. We recognize that there are varying levels of evaluation experience among state grantee evaluators, but know that many of your public health skills are also common to evaluation.

Purpose

This self-assessment instrument is designed to support the professional development of evaluators by identifying our strengths and needs and by stimulating reflection, both individually and with our fellow evaluators. The self-assessment is intended for use by asthma program evaluators practicing at the state, territory, and national levels. Together we can identify opportunities for you to advance your evaluation skills, share them with others or recommend additional evaluation resources that could enhance your evaluation practice. Please be aware that this instrument is intended solely for the identification of your training, technical assistance and resource needs and will not to be used for personnel evaluation. We have organized the self-assessment in two parts. Section I provides an opportunity for you to assess your skills and experience carrying out a variety of evaluation related activities. This section is for your own personal use and professional reflection. Section I Summary Sheet asks you to reflect on what you learned from your self-assessment including: summarizing your strengths and needs, indicating evaluation methods or approaches you would like to learn more about, identifying skills you would like to build on and skills you would like to share with others. We ask that you share the Section I Summary Sheet with your Evaluation Technical Advisor, as it will help us tailor the types of resources and support we provide you, including planning for future trainings and materials development. Section II has questions about your organization and asks that you assess its experience with and attitudes toward evaluation. We ask that you fill this section out and return it to your ETA. We will use this information to understand your needs and the context in which you work. We may also be able to assist you in building evaluation capacity in your organization.

Follow-Up

Once you have completed Section I Summary Sheet and Section II, please contact your ETA to schedule a confidential discussion of your assessment. Together we can identify opportunities for you to advance your evaluation skills and share them with others. Your ETA can also recommend additional evaluation resources that can assist you in becoming a more effective evaluator.

Public reporting burden of this collection of information is estimated to average 20 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect

of this collection of information

Additional Resources:

A resource document entitled “Self-Assessment Follow Up” is available from your ETA upon completion of this instrument. The Self-Assessment Follow Up provides details about where to obtain information on a number of topics presented in this self-assessment instrument—including resource documents available in print and online, as well as professional development training opportunities.

2. Background information

Please tell us a little bit about yourself.

1. For what state/territory do you serve as evaluator? Please enter two character abbreviation.

2. What is your position in the asthma control program (check all that apply)?

Evaluator – state employee

Evaluator – contractor/consultant

Program Manager

Principal Investigator

Other (please specify):

3. How long have you been working in the asthma or respiratory health field?

< 1 year

1 to <2 years

2 to 5 years

>5 years

3.

4. Which of the following best describes your level of experience in program evaluation?

Relatively no experience, need comprehensive training

Little experience or background, need training on most topics

Moderate experience, need refresher courses and more advanced skill training

Advanced experience, need training or information on emerging topics

4. Evaluator Competencies

For each of the items below, mark the box that best represents your personal opinion. Please note, this list represents a fairly comprehensive set of evaluation skills and knowledge. It is not expected that you will be experienced or skilled in all of them. At the end of this survey, you will be asked to identify the areas in which you have the greatest need.

Knowledge I have a working knowledge of…
	Not at all	I have only a general knowledge	I have used or could use these	I have taught or could teach these
a. American Evaluation Association Guiding Principles for Evaluators				
b. Joint Committee Program Standards (developed by the Joint Committee on Standards for Educational Evaluation)				
c. CDC Framework for Program Evaluation in Public Health				
d. Multiple approaches to evaluation (e.g., participatory, theory-driven, empowerment)				

2. Please indicate you experience with the following evaluation activity:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
a. Involving stakeholders in the evaluation planning process				

5.
1. Please indicate your experience facilitating constructive interpersonal interactions, including:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this			Have taught or could teach this
a. establishing and maintaining teamwork…						
b. facilitating groups…………………...........						
c. demonstrating cross-cultural competence						
d. resolving conflict………………….............						

2. Please indicate your experience with carrying out the following evaluation activities:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
a. writing evaluation plans……………………				
b. designing data collection…………………				
c. budgeting for evaluations…………………				
d. contracting for evaluations…………………………..........				
e.. staffing evaluations………………………..				
f. establishing and maintaining databases…				

6.
1. Please indicate your experience with managing evaluations, especially:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this	Have taught or could teach this
a. ethical issues that arise .						
b. political issues that arise………………………………
c. general management of evaluations ..						

2. Please indicate your experience with carrying out the following evaluation activities:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
a. preparing IRB packages……………………………				
b. creating a program logic model……………………
c. using logic models to guide the development of evaluation questions …..				

7.

1. Please indicate to what extent you agree with the following statements about your program, unit or department within your organization.
	Strongly Disagree	Somewhat Disagree	Neutral	Somewhat Agree	Strongly Agree
Currently available information tells us what we need to know about the effectiveness of our program…………………………………					
Currently available information tells us what we need to know about the efficiency of our processes…………………………………					
Existing systems manage and disseminate information for those who need and can use it……………………					
Employees have access to the information they need to make decisions regarding their work………………………..					
My organization welcomes new information………………………………..					
There would be support among employees if we tried to do more (some) evaluation work…………………					
Doing (more) evaluation would make it easier to convince managers of needed changes…………………………………..					

8.
Please indicate your experience implementing various evaluation designs, specifically:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
experimental							
quasi-experimental….……………………							
non-experimental							

Please indicate your experience analyzing qualitative data, including:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
designing evaluations using mixed methods evaluations				
developing data collection instruments….……………………………...				
adapting existing data collection instruments for evaluation purposes				

9.
Please indicate your experience analyzing qualitative data, including:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
developing coding systems (e.g. inter- and intra-rater reliability checks, developing codebooks)							
applying appropriate analysis approaches to qualitative data (e.g. grounded theory, comparative analysis)….…………………………..							
coding for themes and patterns							

Please indicate your experience using data from a variety of sources for evaluation purposes:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
insurance/hospital records				
program documents (policies, meeting minutes, promotional materials)….………………………….				
surveillance systems				

Please indicate your experience using data from a variety of sources for evaluation purposes:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
synthesizing evaluation findings with stakeholders				
writing user-friendly evaluation reports…				

10.
Please indicate your experience tailoring evaluation findings to the following audiences:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
media							
state partners….…………………………							
program staff							
community groups							
legislators/advocacy groups/policy makers….…………………………………							

Please indicate your experience using the following techniques to communicate evaluation findings:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
posters				
presentations……………………………...				
reports				
newsletters				
developing action plans based on evaluation recommendations				

11.
Please indicate your experience with:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
developing action plans based on evaluation recommendations							

2. In addition to the skills listed previously, do you possess any other skills that you find useful in evaluating programs? If so please list these below:

12.
Please indicate your experience with collecting quantitative data using:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
observations							
record abstractions….……………………							
program staff							
surveys/questionnaires							

Please indicate your experience analyzing quantitative data using:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
inferential statistics				
descriptive statistics….…………………..				

Please indicate your experience analyzing quantitative data using:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Have taught or could teach this
observations				
interviews				
focus groups				
case studies				
documents				

13. Professional Practice and Experience
Please indicate your experience with the following professional development activities:
	Would like technical assistance on this		Have general knowledge about this		Have done or used this		Have taught or could teach this
building relationships or networks to enhance evaluation practices							
participating in evaluation trainings							
accessing professional development opportunities in evaluation in my community							
identifying appropriate evaluation tools and materials							

Please indicate your level of evaluation experience:
	Would like technical assistance on this	Have general knowledge about this	Have done or used this	Could teach or have taught this
a. using evaluation findings to plan new programs…				
b. using evaluation findings modify programs……….				
c. using evaluation findings to demonstrate program impacts				
using evaluation findings to:				

14. Organizational Practices and Readiness for Learning
In the following section please reflect on the asthma program with which you work. When considering these questions, think about how open the program is to supporting evaluation in general, to conducting evaluations, and to using evaluation findings in making decisions. Please answer the following questions in terms of the program, unit, or department with which you work and that you know best, not the larger organization.

Please indicate to what extent you agree with the following statements about your program, unit, or department within your organization:
	Strongly Disagree	Somewhat Disagree	Neutral	Somewhat Agree	Agree
My organization has evaluator positions or position descriptions that include evaluation as an essential job function……………….........................					
Organization leaders support and value program evaluation…………….					
Organization leaders integrate evaluation findings into the strategic planning process……………………..					
Organization leaders motivate staff to integrate evaluation into their work…………………………………...					
Program evaluation routinely drives continuous improvement of programs……………………………….					
Organizational funds are routinely dedicated to evaluation (beyond my salary)………………….					
Organization leaders support training to improve evaluation skills………………………...................					
Our performance review system values evaluation and offers promotions or rewards for evaluation work…………………………………….					
My organization has systems in place to change policies and procedures based on evaluation findings and recommendations..........					
Staff with evaluation expertise are available within my organization to assist me …..					
Organizational support is available for technology systems, including software that can be used to manage and analyze evaluation data (e.g., Excel, SPSS) ……					

15. Technical Assistance Needs
Please identify your greatest needs for technical assistance, whether in developing your own evaluation competencies or in building support for evaluation in your organization or building evaluation capacity.

ATTACHMENT – C: Data Collection Instrument Part 2

1. Assessment of NACP Evaluation Technical Assistance

This portion of the survey assesses the evaluation technical assistance provided by the National Asthma Control Program (NACP) to evaluators working with state asthma programs. Questions cover both the one-on-one technical services offered by your ETA as well as other evaluation resources provided by the NACP.

The information that you provide will be used to improve evaluation technical assistance and to make internal decisions about the extent to which various services are needed. We ask that you provide honest feedback about the individualized support you have received to date and the type of assistance you think is most helpful to the work you do.

For this section, we have taken every effort to create a survey that will not allow us to identify you or the state asthma program for which you provide evaluation services

1. Over the past year, check the level of interaction you have typically had with your ETA.

Daily

Weekly

Biweekly

Monthly

Quarterly

No interaction in past year

Other (please specify):________________________

1. Which of the following statements best represents your views regarding the level of interaction you have typically had with your ETA?

This is too much interaction.

This is just the right amount of interaction.

This is not enough interaction.

Other (please specify): _______________________________

2. Please describe the ideal level of interaction with you ETA.

3. Has there been anything that prevented you from interacting with your ETA as frequently as you would like?

No

Yes (describe below)

For the following topics, please indicate whether your ETA provided you with adequate advice and resources, or if topic not discussed, mark “N/A”:
	Adequate Advice/Resources Provided	Inadequate Advice/Resources Provided	Not applicable (N/A)
a. Evaluation planning			
b. Evaluation implementation			
c. General Problem Solving			
d. Evaluation planning			
e. Emerging Stakeholders			
f. Describing the Program			
g. Focusing the Design			
h. Gathering Credible Evidence			
i. Interpreting Findings/Justifying Conclusions			
j. Ensuring Use/Disseminating Findings			

1. Please rate the general quality of the interactions you have had with your assigned ETA.

Poor

Fair

Good

Excellent

Explain rating below: (optional)

In general, my ETA has been able to:
	Yes	No	Not applicable (N/A)
a. Help me better understand evaluation			
b. Direct me towards helpful resources			
c. Help me solve difficult technical problems			
Provide helpful insights about how to address evaluation issues as they have emerged in my practice			
e. Other (please specify)

Which of the following general resources have you used? Please rate their utility.
	Very Helpful	Somewhat Helpful		Not Helpful	N/A
a. Asthma Evaluation Listserv					
b. Using Evaluation to Reduce the Burden of Asthma Webinars					
c. Quarterly Evaluation Conference Calls					
d. Learning and Growing through Evaluation Manual					

Any additional comments?

1. Please provide any additional details about how we can improve our provision of evaluation technical assistance.