Guidelines for IMLS Grants to State Five-Year Evaluation- Form 1

GUIDELINES FOR IMLS GRANTS TO STATES FIVE-YEAR EVALUATION

Purpose of the Evaluation

Section 9134 (c) of IMLS’ authorizing legislation directs State Library Administrative Agencies (SLAAs) to “independently evaluate, and report to the Director regarding, the activities assisted under this subchapter, prior to the end of the 5-year plan.” This evaluation provides SLAAs an opportunity to measure progress in meeting the goals set in their approved Five-Year Plans with a framework to synthesize information across all state reports in telling a national story.

This guidance identifies a core set of questions for the SLAAs to use in conducting the 2013-2017 Five-Year Evaluations that:

• Highlight effective past practices;

• Assess the efficacy in implementing the activities used in advancing state goals; and

• Develop key findings and recommendations from evaluating the past five years for use in organizing the next Five-Year Plan.

There are three sets of questions for each SLAA. The guidance contains possible methodological choices to help each SLAA best work with an independent evaluator.

Format and Questions

IMLS analyzes and makes public all SLAA Five-Year Evaluations. In order to do this effectively, certain information needs to be included in all evaluation reports. This is particularly important in enabling IMLS to tell federal policy makers and practitioners about what has happened at a national level. The specified format is intended to ease the burden for any party to review across multiple reports.

Documents required for the Five-Year Evaluation include a cover page (1 page), evaluation summary (2-5 pages), evaluation report (25 pages, max.), and appendices. Please follow the format specified below:

Cover Page (1 page)

• State Library Administrative Agency

• Title of the evaluation

• Evaluator(s) name and organizational affiliation

• Date

• Name of the team, branch, unit, or person commissioning the evaluation

Evaluation Summary (2-5 pages)

• Summarize key findings for the three retrospective and three process questions below

• Briefly describe the evaluation methodology, referencing the four methodology questions below

Evaluation Report (25 pages, max.)

• Answer the first six questions under A. Retrospective and B. Process in order, and numbered as they are below.

• Describe the methodology employed, responding to the four questions under C. Methodology, below.

A. Retrospective Questions

A-1. To what extent did your Five-Year Plan activities make progress towards each goal?  Where progress was not achieved as anticipated, discuss what factors (e.g., staffing, budget, over-ambitious goals, partners) contributed? 

• Organize findings around each goal of the state’s 2013-2017 Five-Year Plan

• Categorize each goal as either 1) achieved, 2) partly achieved, or 3) not achieved

A-2. To what extent did your Five-Year Plan activities achieve results that address national priorities associated with the Measuring Success focal areas and their corresponding intents? 

See Appendix 1 for a list of focal areas and their intents

A-3. Did any of the following groups represent a substantial focus for your Five-Year Plan activities?  (Yes/No)

• Library workforce (current and future)

• Individuals living below the poverty line

• Individuals that are unemployed/underemployed

• Ethnic or minority populations

• Immigrants/refugees

• Individuals with disabilities

• Individuals with limited functional literacy or information skills

• Families

• Children (aged 0-5)

• School-aged youth (aged 6-17)

For the purposes of this question, a substantial focus would represent at least ten percent of the total amount of resources committed by the overall plan across multiple years.

For those who answer Yes to any of the above groups, please discuss to what extent each group was reached.

B. Process Questions

B-1. How have you used data from the old and new State Program Report (SPR) and elsewhere to guide activities included in the Five-Year Plan?

B-2. Specify any changes you made to the Five-Year Plan, and why this occurred.

B-3. How and with whom have you shared data from the old and new SPR and from other evaluation resources?

C. Methodology Questions

C-1. Identify how you implemented an independent Five-Year Evaluation using the criteria described in the section of this guidance document called Selection of Evaluators. 

C-2. Describe the types of statistical and qualitative methods (including administrative records) used in conducting the Five-Year Evaluation. Assess their validity and reliability.

C-3. Describe the stakeholders involved in the various stages of the Five-Year Evaluation and how you engaged them. 

C-4. Discuss how you will share the key findings and recommendations with others.

Appendices

• List of acronyms

• List of people interviewed

• Bibliography of all documents reviewed

• Copies of any research instruments used for surveying, interviewing, and/or use of focus groups

• Optional output of statistical findings

• Optional summaries of coding used in any qualitative analyses

Evaluation Strategies

Retrospective and Process Questions

• Make use of administrative data on program performance. This information can be data that is reported to IMLS on the SPR or other programmatic data collected by the SLAA.

• The administrative data will likely need to be supplemented with information collected from interviews, surveys, and/or focus groups.

• Data also may be available from secondary documents, including contracted third-party program evaluations, studies from non-partisan entities, and any SLAA reports submitted to IMLS and state policy makers.

• Other sources of information, such as Census data, state education data, and surveys conducted by the SLAA may be used to describe broad changes in communities or in the state. While these, for the most part, cannot be used for making direct attributions of outcomes from LSTA programming efforts, they can effectively describe the context of activities undertaken.

• Descriptive statistics should suffice in conducting any quantitative analysis. The mixing of summary tables and/or figures summarizing the results in the narrative is customary in this type of research. Presentation of extensive statistical output is generally reserved for appendices.

• A content analysis (with potential descriptive statistics for summarizing codes) is probably an acceptable method for conducting qualitative analysis. There are various types of sampling and coding strategies that will precede selecting a content analysis or other analytical choice; the independent evaluator should make these transparent in allowing you and other readers to assess the credibility of the evidence. (See below for more details on evaluation methodology and using an independent evaluator.)

Methodology Questions

• The independent evaluator should clearly address these questions to your satisfaction before proceeding to collect and analyze data.

• The independent evaluator will need to carefully document project records used in the study. Professional guidelines for this type of research require protocols in place to ensure confidentiality and consent.

• In working with the independent evaluator, other stakeholders reviewing the document should have set aside appropriate time to assure that they have enough knowledge of the scientific techniques that the evaluators will be using in collecting and analyzing data, including tradeoffs that they are making given limited resources and time.

• You should include a section that summarizes the methods used in any statistical and qualitative research. For qualitative research, many types of sampling and coding strategies may be appropriate; whatever gets selected should be made transparent in this section.

• The appendices should contain copies of any instruments used for data collection as well as those used in coding.

Selection of an Independent Evaluator

An independent evaluation is rigorous and objective (carried out free from outside influence). If a State decides to carry out the evaluation within the government, the evaluator should be able to demonstrate that it does not have a role in carrying out LSTA-funded activities and is independent of those who are being evaluated or who might be favorably or adversely affected by the evaluation results. If the evaluation is carried out within the State entity that includes the SLAA, the line of hierarchy within the entity should demonstrate that the evaluator is not accountable to those responsible for oversight of the LSTA program within the State. Regardless of whether the evaluation is done in-house or through a third party, the evaluator must be able to demonstrate professional competency to rigorously conduct the evaluation, including requisite expertise in statistical and qualitative research methods.

Submitting Your Report

Please send an electronic version of your Five-Year Evaluation by March 30, 2017 to: [email protected]

Please send one paper copy of your Five-Year Evaluation by March 30, 2017 to:

Robin Dale

Associate Deputy Director for Library Services

Institute of Museum and Library Services

955 L'Enfant Plaza SW, Suite 4000

Washington DC 20024

Appendix 1: Measuring Success Focal Areas and Intents

• Lifelong Learning

  • Improve users’ formal education

  • Improve users’ general knowledge and skills

• Information Access

  • Improve users’ ability to discover information resources

  • Improve users’ ability to obtain and/or use information resources

• Institutional Capacity

  • Improve the library workforce

  • Improve the library’s physical and technological infrastructure

  • Improve library operations

• Economic & Employment Development

  • Improve users’ ability to use resources and apply information for employment support

  • Improve users’ ability to use and apply business resources

• Human Services

  • Improve users’ ability to apply information that furthers their personal, family, or household finances

  • Improve users’ ability to apply information that furthers their personal or family health & wellness

  • Improve users’ ability to apply information that furthers their parenting and family skills

• Civic Engagement

  • Improve users’ ability to participate in their community

  • Improve users’ ability to participate in community conversations around topics of concern.