APP Appendices - Previous Version

Evaluation and Learning for IMLS's Applying Promising Practices for Small and Rural Libraries (APP) Program

Appendices

Contents

Appendices 25

Appendix A – Respondent Contact Letters 27

Appendix B – Grantee Survey Instruments & Interview Guides 28

Tool #1 - APP Program Cohorts 1 & 2 Baseline Capacity Survey 28

Tool #2 - APP Program Grantee Mid-Point Survey (Cohorts 1 & 2) 38

Tool #3 - APP Program End-point Capacity Survey (Cohorts 1 & 2). 49

Tool #4 - Grantee Interview Guide Cohort 1 58

Tool #5 - Grantee Interview Guide Cohort 2 62

Appendix C – Mentor Organizations Interview Guides 66

Tool #1 - Mentor Organization – Quarterly Interview Guide 66

Tool #2 - Mentor Organization – Cohort 1 End-point and Cohort 2 Mid-point Mentor Organization Interview Guide 68

Tool #3 - Mentor Organization – Cohort 1 Midpoint Interview Guide 74

Tool #4 - Mentor Organization – Cohort 2 Endpoint Interview Guide 77

Appendix D – Non-Returning Applicants Survey 81

Appendix E – IMLS Staff Interview Guide 85

Appendix F – Rubrics for Grantee & Observations Analysis of Administrative Records 89

Tool #1 - Document Review Rubric – Mentor Records 89

Tool #2 - Document Review Rubric – Grantee Reports 91

Tool #3 - Document Review Rubric for APP Recruitment & Selection 96

Tool #4 – Observation Rubric - Online Gathering/In-Person Convening 105

Tool #5 - Rubric Portal Evaluation Protocol 4.27_for IMLS review 109

Appendix F – APP Program Logic Model 111

Appendix G – Cost Burden Estimates 115

Estimates of Cost to Respondents 115

Estimates of Cost to the Federal Government 115

Appendix H – Cross Walk of Research Questions to Data Sources 116

Table H1: Alignment of Evaluation Goals and Questions 116

Table H2: Evaluation questions and associated tool items for data collection 118

Table H3: Anticipated data sources, method of analysis, and evaluative information 132

Appendix A – Respondent Contact Letters

This letter will be sent by IMLS on behalf of the evaluation to respondents, informing them of the purpose of the evaluation, the PPG’s role, and the forthcoming request for data collection (survey or survey and interview) from PPG. This letter will be customized to the respondent type (mentor organizations, grantee organizations, or applicant organizations). This letter will be sent approximately 1 week before PPG plans to contact the stakeholder for data collection (survey and interviews for applicants from round 1 of grantmaking that did not re-apply).

Letters to applicants

To: [Email address for identified contact]

From: IMLS

Subject: Requesting your assistance in a review of the Accelerating Promising Practices for Small and Rural Libraries Grant Program

Dear {First Name},

As an applicant of the Institute for Museum and Library Services (IMLS)’s Accelerating Promising Practices for Small and Rural Libraries (APP) program, I’m inviting you to participate in an evaluation of the APP program’s performance.

First, I want to assure you that this is not an evaluation of your organization or institution, nor of your past performance with IMLS grants. We are only interested in learning about your experience as an applicant to the APP program.

The goals of this evaluation are to better understand your perspectives on the APP program, XYZ, XYZ.

IMLS has contracted with the Partnership for Public Good (PPG), a national nonprofit research organization, to conduct this evaluation. In the coming weeks, researchers from PPG will be reaching out to you to collect your views. Your participation is entirely voluntary and will have no impact on your relationship with the APP program, IMLS, or the Federal government. Any information you provide to PPG will be reported to IMLS in aggregate form only. No individual responses will be reported.

We hope you are willing to participate in this effort as it will enable us to better understand the performance of the APP program and help improve this program for future grantees.

If you have any questions or concerns about this evaluation, please contact XXX at PPG (XXXX) or Marvin Carr at IMLS ([email protected]).

On behalf of IMLS & the PPG team, thank you in advance for taking the time to share your knowledge and insights with us.

Best,

Appendix B – Grantee Survey Instruments & Interview Guides

This appendix includes the following data collection tools for use with the grantees/Community of Practice participants:

1. Baseline capacity survey (Cohorts 1 & 2)

2. Midpoint survey (Cohorts 1 & 2)

3. Endpoint capacity survey (Cohorts 1 & 2)

4. Grantee interview guide (Cohort 1)

5. Grantee interview guide (Cohort 2)

Grantee Survey Instruments

Tool #1 - APP Program Cohorts 1 & 2 Baseline Capacity Survey

The Institute of Museum and Library Services (IMLS) has contracted with Partners for Public Good (PPG) to conduct an evaluation of its Accelerating Promising Practices for Small and Rural Libraries (APP) program. This survey asks questions about your participation in the Community Memory, Digital Inclusion, or Transforming School Library Practice Community of Practice. The information you share with us will contribute to our understanding of the experiences and expectations of participants in the program and will only be presented in aggregate form (i.e., not individually identifiable).

The survey takes approximately 15-20 minutes to complete. If you have any questions about the survey, please contact Lisa Frantzen, Senior Evaluation Consultant with PPG, at [email protected].

1. Of which Community of Practice are you a member?

   1. Community Memory

   2. Digital Inclusion

   3. Transforming School Library Practice

Personalized Training & Technical Assistance

2. Before staring work with the APP Community of Practice …

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library had implemented projects focused on [Community Memory/Digital Inclusion/Transforming School Library Practice].
Our library had people with skills and experience in project planning.
Our library had people with skills and experience in project implementation.
Our library had people with skills and experience in managing a grant-funded project.
I felt confident in implementing projects about [Community Memory/Digital Inclusion/Transforming School Library Practice].
I had the skills and knowledge needed to implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
Our library created programs and services to reflect the new and emerging needs of our community.
Our library had [Community Memory/Digital Inclusion/Transforming School Library Practice] project(s) that incorporated the (school) community’s priorities and perspectives.
Our library had a clear plan for how we could continually implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects that met the (school) community’s needs.

3. [Cohort 1] Before joining the Community of Practice, what was your library’s role in the community/in the school?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

4. [Cohort 2] Before the COVID-19 crisis, what was your library’s role in the community/in the school?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

5. [Cohort 2] Since the onset of the COVDID-19 crisis, what has been your library’s role in the community/in the school?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

6. [Cohort 2] Looking forward into the next couple of years, how do you see your library’s role in the community/in the school evolving?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

7. Before joining the APP Community Memory Community of Practice…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library had people with the right skills and experience in Community Memory projects to implement our desired Community Memory projects.
Our library had enough people with Community Memory skills and experience to implement our desired Community Memory projects.
Our library had people with skills and experience in conducting community needs assessments.
Our library had people with skills and experience in community engagement.
Engaging the community in defining what is needed for Community Memory projects was a high priority for our library.
Engaging the community in defining what is needed for Community Memory projects was an integral part of how our library works.
Our library had people with the right digital skills and experience (e.g., scanning, oral history recording) to implement our desired Community Memory projects.
Our library had people with digital preservation plan and legacy plan development skills.
Community Memory projects were a high priority for our library.
Community Memory projects were an integral part of our library’s work.

8. [Cohort 2] How has the COVID-19 crisis affected your thinking about Community Memory work?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

9. Before joining the APP Digital Inclusion Community of Practice…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library had the physical infrastructure necessary to meet the community’s internet access needs.
Our library had people with skills and experience needed to help community members access and appropriately use digital content.
Our library had people with the skills and experience needed to help community members create digital content.
Our library had the materials necessary to help community members develop appropriate levels of digital literacy.
Our library had connections with local organizations and businesses (e.g., computer repair shops) that provide ongoing support for the digital needs of our community.
Our library had people with the right digital skills and experience (e.g., knowledge of hardware and software) to implement our desired Digital Inclusion projects.
Our library had enough people with digital skills and experience our desired Digital Inclusion projects.
Digital Inclusion projects were a high priority for our library.
Digital Inclusion projects were an integral part of our library’s work.
I felt connected to the larger Digital Inclusion community (e.g., software developers, schools, training programs).
I actively engaged with the larger Digital Inclusion community.
I shared learnings from my work with the larger Digital Inclusion community.

10. [Cohort 2] How has the COVID-19 crisis affected your thinking on Digital Inclusion work?

____________________________________________________________________________________________________________________________________________________________

11. Before joining the APP Transforming School Library Practice Community of Practice…

TRANSFORMING SCHOOL LIBRARY PRACTICE COHORT	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
My school library (or school libraries) was seen by many others in the school as integral to the school.
Projects focused on transforming our school library (or libraries) were a high priority for our school.
Projects focused on transforming our school library (or libraries) were an integral part of our library’s work.
I knew how to engage stakeholders in my school community to think about and plan for the school library (or libraries).
My school library (or libraries) was already a dynamic hub for students to pursue self-directed, inquiry-based learning.
My school library had enough people with skills and experience in transforming school libraries.
I felt confident that I could adapt my school library (or school libraries) to the school’s evolving organizational and community needs.
I had the skills and knowledge to adapt the school library (or school libraries) to meet the evolving needs of our school community.
I felt confident that I could manage a grant-funded project.
I had the skills and knowledge to manage a grant-funded project.
I recognized areas where I could benefit from additional training to help me implement my grant-funded project more effectively.
I already had a thought partner at my school that I consistently talked to about how to improve our school library (school libraries).

12. [Cohort 2] How has the COVID-19 crisis impacted your thinking around your Transforming School Library Practice work?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

Community of Practice

13. Before joining the APP Community of Practice…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I thought of myself as part of a larger community of school librarians.
I thought of myself as part of a larger learning community.
I imagined that my cohort would become a professional peer learning network that I could draw on for professional support.
I regularly interacted with other libraries working in [Community Memory/Digital Inclusion/Transforming School Library Practice].
I felt I had other libraries with whom I could consult with on [Community Memory/Digital Inclusion/Transforming School Library Practice].
I regularly learned from other libraries working on [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I knew about available resources and tools for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.

14. Before joining the APP Community of Practice, to what extent did you have the organizational support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff or funding to support this work, or general interest in this work across your library or school.) Please explain how you did or did not have the support you needed.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

_____________________________________________________________________________

15. Before joining the APP Community of Practice, to what extent did you have the community support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include buy-in to this work by community members or school community members through their interest in the work, volunteerism to help in the work, etc.) Please explain how you did or did not have the support you needed.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

16. [Cohort 2] How has the COVID-19 crisis shifted your ability to implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects?

____________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________

Interaction with Libraries Field

17. [Cohort 1] Before joining the APP Community of Practice /[Cohort 2] Prior to the COVID-19 crisis unfolding…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I felt my library was a part of a larger network/community of libraries implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I had contributed learnings from my library’s work to the broader library field.
I had contributed learnings from my library’s [Community Memory/Digital Inclusion/Transforming School Library Practice] work to the broader library field.
I believed an integral part of my role as a librarian was to be a facilitator of community knowledge.
I believed part of the role of my library was to be a facilitator of community knowledge.
I believed an integral part of my role as a librarian was to be a provider of public/the school community’s access to information, ideas, and networks.
I believed part of the role of my library was to be a provider of public/the school community’s access to information, ideas, and networks.
Our library participated in key associations, listservs, and events that were relevant to our library.

Expectations and Library Demographics

18. As you entered the APP Community of Practice, what expectations did you have about what [Community Memory/Digital Inclusion/Transforming School Library Practice] capacities would be built for your library?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

19. As you entered the APP Community of Practice, what one word or phrase would you use to describe how you felt about beginning work in a cohort? __________________________________________

20. As you entered the APP Community of Practice, what one word or phrase would you use to describe how you felt about beginning your IMLS-funded [Community Memory/Digital Inclusion/Transforming School Library Practice] project? _________________________________________________

21. Briefly, what was your original motivation for applying to be a part of the APP program?

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

22. Tell us a little about the staffing structure in your library.

	# of Full-time staff members	# of Part-time staff members	# of Volunteers
Across your whole library
For your IMLS-funded Community Memory/Digital Inclusion/Transforming School Library Practice] project

23. Approximately how many people does your library serve each year? _______________________

24. In addition to the funds provided in this grant, approximately, how much money is earmarked for your [Community Memory/Digital Inclusion/Transforming School Library Practice] projects this year? ________________________

Tool #2 - APP Program Grantee Mid-Point Survey (Cohorts 1 & 2)

The Institute of Museum and Library Services (IMLS) has contracted with Partners for Public Good (PPG) to conduct an evaluation of its Accelerating Promising Practices for Small and Rural Libraries (APP) program. This survey asks questions about your participation in the Community Memory, Digital Inclusion, or Transforming School Library Practice Community of Practice. The information you share with us will contribute to our understanding of the experiences and expectations of participants in the program and will only be presented in aggregate form (i.e., not individually identifiable).

The survey takes approximately 15-20 minutes to complete. If you have any questions about the survey, please contact Lisa Frantzen, Senior Evaluation Consultant with PPG, at [email protected].

1. Of which cohort are you a member?

   1. Community Memory

   2. Digital Inclusion

   3. Transforming School Library Practice

Personalized Training & Technical Assistance

2. Please rate your library for the following statements as of today…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has implemented projects focused on [Community Memory/Digital Inclusion/Transforming School Library Practice].
Our library has people with skills and experience in project planning.
Our library has people with skills and experience in project implementation.
Our library has people with skills and experience in managing a grant-funded project.
I felt confident in implementing projects about [Community Memory/Digital Inclusion/Transforming School Library Practice].
I have the skills and knowledge needed to implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
Our library creates programs and services to reflect the new and emerging needs of our community.
Our library has [Community Memory/Digital Inclusion/Transforming School Library Practice] project(s) that incorporate the (school) community’s priorities and perspectives.
Our library has a clear plan for how we could continually implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects that meet the (school) community’s needs.

3. Thinking of the time since the COVID-19 crisis, how has your library’s role changed in the community/in the school?

____________________________________________________________________________________________________________________________________________________________

_____________________________________________________________________________

4. Please rate your library for the following statements as of today… [Community Memory COP]

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has people with the right skills and experience in Community Memory projects to implement our desired Community Memory projects.
Our library has enough people with Community Memory skills and experience to implement our desired Community Memory projects.
Our library has people with skills and experience in conducting community needs assessments.
Our library has people with skills and experience in community engagement.
Engaging the community in defining what is needed for Community Memory projects is a high priority for our library.
Engaging the community in defining what is needed for Community Memory projects is an integral part of how our library works.
Our library has people with the right digital skills and experience (e.g., scanning, oral history recording) to implement our desired Community Memory projects.
Our library has people with digital preservation plan and legacy plan development skills.
Community Memory projects are a high priority for our library.
Community Memory projects are an integral part of our library’s work.

5. How has the COVID-19 crisis affected the community’s perception of your Community Memory work?

____________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________

6. Please rate your library for the following statements as of today… [Digital Inclusion COP]

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has the physical infrastructure necessary to meet the community’s internet access needs.
Our library has people with skills and experience needed to help community members access and appropriately use digital content.
Our library has people with the skills and experience needed to help community members create digital content.
Our library has the materials necessary to help community members develop appropriate levels of digital literacy.
Our library has connections with local organizations and businesses (e.g., computer repair shops) that provide ongoing support for the digital needs of our community.
Our library has people with the right digital skills and experience (e.g., knowledge of hardware and software) to implement our desired Digital Inclusion projects.
Our library has enough people with digital skills and experience our desired Digital Inclusion projects.
Digital Inclusion projects are a high priority for our library.
Digital Inclusion projects are an integral part of our library’s work.
I feel connected to the larger Digital Inclusion community (e.g., software developers, schools, training programs).
I actively engage with the larger Digital Inclusion community.
I share learnings from my work with the larger Digital Inclusion community.

7. How has the COVID-19 crisis affected the community’s perception of your Digital Inclusion work?

____________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________

8. Please rate your library for the following statements as of today…[Transforming School Library Practice COP]

TRANSFORMING SCHOOL LIBRARY PRACTICE COHORT	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
My school library (or school libraries) is seen by many others in the school as integral to the school.
Projects focused on transforming our school library (or libraries) are a high priority for our school.
Projects focused on transforming our school library (or libraries) are an integral part of our library’s work.
I know how to engage stakeholders in my school community to think about and plan for the school library (or libraries).
My school library (or libraries) is already a dynamic hub for students to pursue self-directed, inquiry-based learning.
My school library has enough people with skills and experience in transforming school libraries.
I feel confident that I could adapt my school library (or school libraries) to the school’s evolving organizational and community needs.
I have the skills and knowledge to adapt the school library (or school libraries) to meet the evolving needs of our school community.
I feel confident that I could manage a grant-funded project.
I have the skills and knowledge to manage a grant-funded project.
I recognize areas where I could benefit from additional training to help me implement my grant-funded project more effectively.
I have a thought partner at my school that I consistently talked to about how to improve our school library (school libraries).

9. How has the COVID-19 crisis affected the school’s perception of your Transforming School Library Practice work?

____________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________

Community of Practice

10. Please rate your library for the following statements as of today…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I think of myself as part of a larger community of school librarians.
I think of myself as part of a larger learning community.
My cohort is becoming a professional peer learning network that I could draw on for professional support.
I regularly interact with other libraries working in [Community Memory/Digital Inclusion/Transforming School Library Practice].
I feel I have other libraries with whom I could consult with on [Community Memory/Digital Inclusion/Transforming School Library Practice].
I regularly learn from other libraries working on [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I know about available resources and tools for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.

11. To what extent do you have the organizational support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff or funding to support this work, or general interest in this work across your library or school.) Please explain how you do or do not have the support you need.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

_____________________________________________________________________________

12. To what extent do you have the community support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include buy-in to this work by community members or school community members through their interest in the work, volunteerism to help in the work, etc.) Please explain how you do or do not have the support you need.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

13. How has the COVID-19 crisis shifted your ability to implement your IMLS-funded [Community Memory/Digital Inclusion/Transforming School Library Practice] project?

____________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________

Interaction with Libraries Field

14. Please rate your library for the following statements.

Over the last year, and prior to the COVID-19 crisis unfolding…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I felt my library was a part of a larger network/community of libraries implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I had contributed learnings from my library’s work to the broader library field.
I had contributed learnings from my library’s [Community Memory/Digital Inclusion/Transforming School Library Practice] work to the broader library field.
I believed an integral part of my role as a librarian was to be a facilitator of community knowledge.
I believed part of the role of my library was to be a facilitator of community knowledge.
I believed an integral part of my role as a librarian was to be a provider of public/the school community’s access to information, ideas, and networks.
I believed part of the role of my library was to be a provider of public/the school community’s access to information, ideas, and networks.
Our library participated in key associations, listservs, and events that were relevant to our library.

15. Please rate your library for the following statements.

Since the COVID-19 crisis has unfolded…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I feel my library is a part of a larger network/community of libraries implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I am contributing learnings from my library’s work to the broader library field.
I am contributing learnings from my library’s [Community Memory/Digital Inclusion/Transforming School Library Practice] work to the broader library field.
I believe an integral part of my role as a librarian is to be a facilitator of community knowledge.
I believe part of the role of my library is to be a facilitator of community knowledge.
I believe an integral part of my role as a librarian is to be a provider of public/the school community’s access to information, ideas, and networks.
I believe part of the role of my library is to be a provider of public/the school community’s access to information, ideas, and networks.
Our library participates in key associations, listservs, and events that were relevant to our library.

16. What role, if any did your mentor organization have in helping you change your program due to shifts in needs as a result of the impact of COVID-19 on your community/school?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

17. Were you/have you been able to remain connected to your cohort during the COVID-19 pandemic? If so, in what ways have you interacted with your cohort?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

18. What role, if any did your cohort have in helping you change your program due to shifts in needs as a result of the impact of COVID-19 on your community/school?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

Expectations and Library Demographics

19. Now that you have had time and experience with the APP cohort, what have you learned about building [Community Memory/Digital Inclusion/Transforming School Library Practice] capacities for your library?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

20. What has surprised you about building [Community Memory/Digital Inclusion/Transforming School Library Practice] capacities for your library?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

21. Participating in the APP cohort, what one word or phrase would you use to describe how you feel about working in a COP? __________________________________________

22. In a sentence, what is the one takeaway from the experience you would share with an incoming cohort?

23. Participating in the APP cohort, what one word or phrase would you use to describe how you feel about your IMLS-funded [Community Memory/Digital Inclusion/Transforming School Library Practice] project? _________________________________________________

24. Briefly, what motivates you to keep participating in your APP cohort?

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

25. Tell us a little about the current staffing structure in your library.

	# of Full-time staff members	# of Part-time staff members	# of Volunteers
Across your whole library
For your IMLS-funded Community Memory/Digital Inclusion/Transforming School Library Practice] project

26. Has the number of people your library serves changed as a result of COVID-19? If yes, how many people does your library currently serve? ____________

27. Is this more or less than the number you previously served? (more/less)

28. In addition to the funds provided in this grant, approximately, how much money is currently earmarked for your [Community Memory/Digital Inclusion/Transforming School Library Practice] projects this year? ________________________

29. Has this amount changed as a result of the COVID-19 crisis? (yes/no)

30. (If yes) Has more or less money been earmarked for [Community Memory/Digital Inclusion/Transforming School Library Practice] projects this year?

Tool #3 - APP Program End-point Capacity Survey (Cohorts 1 & 2).

The Institute of Museum and Library Services (IMLS) has contracted with Partners for Public Good (PPG) to conduct an evaluation of its Accelerating Promising Practices for Small and Rural Libraries (APP) program. The information you share with us will contribute to our understanding of the experiences and expectations of participants in the program and will only be presented in aggregate form (i.e., not individually identifiable).

The survey takes approximately 15-20 minutes to complete. If you have any questions about the survey, please contact Lisa Frantzen, Senior Evaluation Consultant with PPG, at [email protected].

1. Of which COP are you a member?

   1. Community Memory

   2. Digital Inclusion

   3. Transforming School Library Practice

Personalized Training & Technical Assistance

1. Please rate your library for the following statements.

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has implemented projects focused on [Community Memory/Digital Inclusion/Transforming School Library Practice].
Our library has people with skills and experience in project planning.
Our library has people with skills and experience in project implementation.
Our library has people with skills and experience in managing a grant-funded project.
I felt confident in implementing projects about [Community Memory/Digital Inclusion/Transforming School Library Practice].
I have the skills and knowledge needed to implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
Our library creates programs and services to reflect the new and emerging needs of our community.
Our library has [Community Memory/Digital Inclusion/Transforming School Library Practice] project(s) that incorporate the (school) community’s priorities and perspectives.
Our library has a clear plan for how we could continually implement [Community Memory/Digital Inclusion/Transforming School Library Practice] projects that meet the (school) community’s needs.

2. Please rate your library for the following statements… [Community Memory COP]

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has people with the right skills and experience in Community Memory projects to implement our desired Community Memory projects.
Our library has enough people with Community Memory skills and experience to implement our desired Community Memory projects.
Our library has people with skills and experience in conducting community needs assessments.
Our library has people with skills and experience in community engagement.
Engaging the community in defining what is needed for Community Memory projects is a high priority for our library.
Engaging the community in defining what is needed for Community Memory projects is an integral part of how our library works.
Our library has people with the right digital skills and experience (e.g., scanning, oral history recording) to implement our desired Community Memory projects.
Our library has people with digital preservation plan and legacy plan development skills.
Community Memory projects are a high priority for our library.
Community Memory projects are an integral part of our library’s work.

3. Please rate your library for the following statements… [Digital Inclusion COP]

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
Our library has the physical infrastructure necessary to meet the community’s internet access needs.
Our library has people with skills and experience needed to help community members access and appropriately use digital content.
Our library has people with the skills and experience needed to help community members create digital content.
Our library has the materials necessary to help community members develop appropriate levels of digital literacy.
Our library has connections with local organizations and businesses (e.g., computer repair shops) that provide ongoing support for the digital needs of our community.
Our library has people with the right digital skills and experience (e.g., knowledge of hardware and software) to implement our desired Digital Inclusion projects.
Our library has enough people with digital skills and experience our desired Digital Inclusion projects.
Digital Inclusion projects are a high priority for our library.
Digital Inclusion projects are an integral part of our library’s work.
I feel connected to the larger Digital Inclusion community (e.g., software developers, schools, training programs).
I actively engage with the larger Digital Inclusion community.
I share learnings from my work with the larger Digital Inclusion community.

4. Please rate your library for the following statements…[Transforming School Library Practice COP]

TRANSFORMING SCHOOL LIBRARY PRACTICE COHORT	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
My school library (or school libraries) is seen by many others in the school as integral to the school.
Projects focused on transforming our school library (or libraries) are a high priority for our school.
Projects focused on transforming our school library (or libraries) are an integral part of our library’s work.
I know how to engage stakeholders in my school community to think about and plan for the school library (or libraries).
My school library (or libraries) is already a dynamic hub for students to pursue self-directed, inquiry-based learning.
My school library has enough people with skills and experience in transforming school libraries.
I feel confident that I could adapt my school library (or school libraries) to the school’s evolving organizational and community needs.
I have the skills and knowledge to adapt the school library (or school libraries) to meet the evolving needs of our school community.
I feel confident that I could manage a grant-funded project.
I have the skills and knowledge to manage a grant-funded project.
I recognize areas where I could benefit from additional training to help me implement my grant-funded project more effectively.
I have a thought partner at my school that I consistently talked to about how to improve our school library (school libraries).

Community of Practice

5. Please rate your library for the following statements…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I think of myself as part of a larger community of school librarians.
I think of myself as part of a larger learning community.
My cohort is becoming a professional peer learning network that I could draw on for professional support.
I regularly interact with other libraries working in [Community Memory/Digital Inclusion/Transforming School Library Practice].
I feel I have other libraries with whom I could consult with on [Community Memory/Digital Inclusion/Transforming School Library Practice].
I regularly learn from other libraries working on [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I know about available resources and tools for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.

6. To what extent do you have the organizational support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff or funding to support this work, or general interest in this work across your library or school.) Please explain how you do or do not have the support you need.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

_____________________________________________________________________________

7. To what extent do you have the community support needed for implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects? (Support could include buy-in to this work by community members or school community members through their interest in the work, volunteerism to help in the work, etc.) Please explain how you do or do not have the support you need.

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

Interaction with Library Field

8. Please rate your library for the following statements…

	Strongly Disagree	Disagree	Agree	Strongly Agree	Not Applicable
I feel my library is a part of a larger network/community of libraries implementing [Community Memory/Digital Inclusion/Transforming School Library Practice] projects.
I am contributing learnings from my library’s work to the broader library field.
I am contributing learnings from my library’s [Community Memory/Digital Inclusion/Transforming School Library Practice] work to the broader library field.
I believe an integral part of my role as a librarian is to be a facilitator of community knowledge.
I believe part of the role of my library is to be a facilitator of community knowledge.
I believe an integral part of my role as a librarian i to be a provider of public/the school community’s access to information, ideas, and networks.
I believe part of the role of my library is to be a provider of public/the school community’s access to information, ideas, and networks.
Our library participates in key associations, listservs, and events that were relevant to our library.

Expectations and Library Demographics

9. Reflecting back on your experience with the APP Community of Practice, what have you learned about building [Community Memory/Digital Inclusion/Transforming School Library Practice] capacities for your library?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

10. What has surprised you about building [Community Memory/Digital Inclusion/Transforming School Library Practice] capacities for your library?

________________________________________________________________________________

________________________________________________________________________________

________________________________________________________________________________

11. Having participated in the APP Community of Practice, what one word or phrase would you use to describe how you feel about working in a cohort? __________________________________________

12. In a sentence, what is the one takeaway from the experience you would share with an incoming cohort? ________________________________________________________________________

13. Participating in the APP Community of Practice, what one word or phrase would you use to describe how you feel about your IMLS-funded [Community Memory/Digital Inclusion/Transforming School Library Practice] project? _________________________________________________

14. Looking back over the least two years, please tell us briefly, what kept you motivated to keep participating in your APP Community of Practice?

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

15. Tell us a little about the current staffing structure in your library.

	# of Full-time staff members	# of Part-time staff members	# of Volunteers
Across your whole library
For your IMLS-funded Community Memory/Digital Inclusion/Transforming School Library Practice] project

16. In addition to the funds provided in this grant, approximately, how much money is currently earmarked for your [Community Memory/Digital Inclusion/Transforming School Library Practice] projects this year? ________________________

17. How much money was earmarked for [Community Memory/Digital Inclusion/Transforming School Library Practice] projects this year? (yes/no)

18. Is this a change over last year’s funding (increase/ no change/ decrease)

19. (If increase or decrease) Was this change attributable directly or indirectly to the COVID-19 crisis?

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

20. (If yes) How was the funding affected by the crisis?

21. (If no) What caused the change in funding?

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

Tool #4 - Grantee Interview Guide Cohort 1

Interview Objectives:

• Gather grantees’ reflections on programmatic effectiveness and outcomes over the course of the 2-year capacity-building program

• Understand how grantees organizations’ experience can inform IMLS strategy

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

This conversation is confidential, and all responses will be reported back in aggregate. During this interview I will ask you to look back over the last year, as well as over the full two-year cycle. You can expect this interview to last about an hour. Do you have any questions for me before we get started?

Background

1. To start, please tell me a bit about your role in your library.

2. Thinking back to your initial interactions with your mentor organization and your cohort, how prepared do you think you were to take advantage of the training and resources that a mentor organization could provide?

   1. How did you feel about you in participating in this program as part of a cohort with other libraries?

Effectiveness of Capacity-Building Activities

3. How effective do you think the in-person kick off meeting was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

4. How effective do you think the Year 2 virtual convening was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

5. How effective do you think the online cohort meetings were? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

6. How effective do you think the curriculum for cohort training was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

   1. Were there any topics that stood out to you because you found them to be very useful? Were there any topics that you did not think were very helpful?

7. How effective do you think the individual learning plans were? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

8. How effective do you think providing the online portal and resource set (i.e., archived webinars, resource lists, checklists and other tools) was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

9. [For TSLP only] How effective do you think the pairing of grantees with school partners was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

10. [For DI only] How effective do you think the one-on-one coaching was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

Effectiveness of Mentorship & Cohort-Based Model

11. What were your overall impressions of your mentor organization? (Possible Probes: mentor organization’s training skills, level of organization, communication and responsiveness, interpersonal style, adaptability to grantees’ needs)

12. How would you rate your mentor organization’s responsiveness and adaptability to your cohort’s evolving needs and circumstances as COVID-19 unfolded? (Probe for specific examples of what the mentor organization did well or could have done better.)

13. What value, if any, did you gain from working with a mentor organization, as opposed to, for example, only receiving the grant support for your project?

    1. To what extent do you think the absence of a mentor organization would have impacted your capacity-building and progress on their grant-supported activities?

14. What were your overall impressions of your cohort? (Possible Probes: To what extent did they see their cohort as a source of support? As a community of shared learners? As a resource for advice and encouragement as they worked on their grant-supported project?)

15. What value, if any, did you gain from participating in the capacity-building as part of a cohort, as opposed to, for example, only being paired with a mentor organization?

    1. How do you think your experience of the mentor organization would have been different if you were not part of a cohort?

    2. To what extent do you think the absence of a cohort would have impacted your capacity building and progress on their grant-supported activities?

Grantee Outcomes

16. Looking back over the two years of the initiative, what would you say were your biggest changes? (Probe for achievements and challenges). What do you think led to those changes?

17. Given that COVID-19 occurred in the middle of your cohort’s capacity-building program, what effects do you think this had on your ability to achieve your desired outcomes?

18. Over the course of the program, to what extent were you able to incorporate or integrate what you learned through the program into your grant-supported projects and other library activities?

    1. What factors do you think contributed to your success in integrating what you learned?

    2. What barriers to integration, if any, did you experience?

    3. Did you notice a difference between your first year and your second year in your ability to act on and integrate what you were learning in the program?

19. How did your participation in the program contribute to your perception of yourself as part of a larger community of (school) librarians? How did it contribute to your perception of yourself as part of a larger community of learners?

20. To what extent do you feel connected to other libraries pursuing similar efforts to improve the value and use of their small and rural libraries?

21. How well do you feel you have been able to engage people in your organization and in the local community to join you in thinking about and planning for library programs, services, or space?

22. Compared to when you started the program, to what extent—more, less, or about the same—have you interacted with members or partners within the (alt: school) community on your grant-sponsored library efforts?

    1. What do you think has contributed to this level of interaction?

    2. What effect do you think this level of interaction is having on building up support for the library’s role in the community?

    3. Have you been able to leverage your interactions with these community partners into other programming opportunities for the library?

23. How much progress did you make towards implementing your project?

24. [Only ask if this grantee will continue on a No-Cost Extension] How do you think the experience of completing your work without the formal mentorship and cohort infrastructure will affect your progress?

Looking Forward

25. How confident do you feel about your ability to continue your efforts in the absence of the grant funding, mentor organization, and formal cohort?

26. To what extent do you think you have the organizational support needed to sustain your projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff, or funding to support this work, or general interest in this work across their library or school).

27. To what extent do you think you have the community support needed to sustain your projects? (Support could include buy-in for this work from school or community members through demonstrations of their interest for their work, volunteers from the community to assist then in their work).

28. To what extent do you think you are better prepared to conceptualize and implement new, future activities or projects in response to your organization’s and community’s evolving needs?

Feedback to IMLS

29. Given what you now know after completing two full years of this program, do you have any insights you might offer IMLS around how to reach new libraries for the APP program? (Possible probes: What would you suggest they do differently? What, if anything, do you think they could make more explicit about the goals of this program?)

30. To what extent did you feel like you received the support you needed from IMLS to be as successful as possible as a grantee organization? (Possible examples of support include funding, program officer engagement and sharing, communication and responsiveness from IMLS) (DI’s program officer is active on the portal)

Wrap up

31. Is there anything else you think is important for us to know?

Tool #5 - Grantee Interview Guide Cohort 2

Interview Objectives:

• Gather grantees’ reflections on programmatic effectiveness and outcomes over the course of the 2-year capacity-building program

• Understand how grantees organizations’ experience can inform IMLS strategy

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

This conversation is confidential, and all responses will be reported back in aggregate. During this interview I will ask you to look back over the last year, as well as, over the full two-year cycle. You can expect this interview to last about an hour. Do you have any questions for me before we get started?

Background

1. To start, please tell me a bit about your role in your library.

2. Thinking back to your initial interactions with your mentor organization and your cohort, how prepared do you think you were to take advantage of the training and resources that a mentor organization could provide?

   1. How did you feel about you in participating in this program as part of a cohort with other libraries?

Effectiveness of Capacity-Building Activities

1. How effective do you think the in-person kick off meeting was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

2. How effective do you think the Year 2 virtual convening was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

3. How effective do you think the online cohort meetings were? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

4. How effective do you think the curriculum for cohort training was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

   1. Were there any topics that stood out to you because you found them to be very useful? Were there any topics that you did not think were very helpful?

5. How effective do you think the individual learning plans were? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

6. How effective do you think providing the online portal and resource set (i.e., archived webinars, resource lists, checklists and other tools) was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

7. [For TSLP only] How effective do you think the pairing of grantees with school partners was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

8. [For DI only] How effective do you think the one-on-one coaching was? What worked well, what didn’t, and what, if anything, would you suggest be changed about this component next time?

Effectiveness of Mentorship & Cohort-Based Model

9. What were your overall impressions of your mentor organization? (Possible Probes: mentor organization’s training skills, level of organization, communication and responsiveness, interpersonal style, adaptability to grantees’ needs)

10. How would you rate your mentor organization’s responsiveness and adaptability to changes in your cohort’s needs and circumstances due to the fallout of COVID-19? (Probe for specific examples of what the mentor organization did well or could have done better.)

11. What value, if any, did you gain from working with a mentor organization, as opposed to, for example, only receiving the grant support for your project?

    1. To what extent do you think the absence of a mentor organization would have impacted your capacity-building and progress on their grant-supported activities?

12. What were your overall impressions of your cohort? (Possible Probes: To what extent did they see their cohort as a source of support? As a community of shared learners? As a resource for advice and encouragement as they worked on their grant-supported project?)

13. What value, if any, did you gain from participating in the capacity-building as part of a cohort, as opposed to, for example, only being paired with a mentor organization?

    1. How do you think your experience of the mentor organization would have been different if you were not part of a cohort?

    2. To what extent do you think the absence of a cohort would have impacted your capacity building and progress on their grant-supported activities?

Grantee Outcomes

14. Looking back over the two years of the initiative, what would you say were your biggest changes? (Probe for achievements and challenges). What do you think led to those changes?

15. Given that COVID-19 occurred just before your cohort’s capacity-building program began, what effects do you think this had on your ability to achieve your desired outcomes?

16. Over the course of the program, to what extent were you able to incorporate or integrate what you learned through the program into your grant-supported projects and other library activities?

    1. What factors do you think contributed to your success in integrating what you learned?

    2. What barriers to integration, if any, did you experience?

    3. Did you notice a difference between your first year and your second year in your ability to act on and integrate what you were learning in the program?

17. How did your participation in the program contribute to your perception of yourself as part of a larger community of (school) librarians? How did it contribute to your perception of yourself as part of a larger community of learners?

18. To what extent do you feel connected to other libraries pursuing similar efforts to improve the value and use of their small and rural libraries?

19. How well do you feel you have been able to engage people in your organization and in the local community to join you in thinking about and planning for library programs, services, or space?

20. Compared to when you started the program, to what extent—more, less, or about the same—have you interacted with members or partners within the (alt: school) community on your grant-sponsored library efforts?

    1. What do you think has contributed to this level of interaction?

    2. What effect do you think this level of interaction is having on building up support for the library’s role in the community?

    3. Have you been able to leverage your interactions with these community partners into other programming opportunities for the library?

21. How much progress did you make towards implementing your project?

22. [Only ask if this grantee will continue on a No-Cost Extension] How do you think the experience of completing your work without the formal mentorship and cohort infrastructure will affect your progress?

Looking Forward

23. How confident do you feel about your ability to continue your efforts in the absence of the grant funding, mentor organization, and formal cohort?

24. To what extent do you think you have the organizational support needed to sustain your projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff, or funding to support this work, or general interest in this work across their library or school).

25. To what extent do you think you have the community support needed to sustain your projects? (Support could include buy-in for this work from school or community members through demonstrations of their interest for their work, volunteers from the community to assist then in their work).

26. To what extent do you think you are better prepared to conceptualize and implement new, future activities or projects in response to your organization’s and community’s evolving needs?

Feedback to IMLS

27. Given what you now know after completing two full years of this program, do you have any insights you might offer IMLS around how to reach new libraries for the APP program? (Possible probes: What would you suggest they do differently? What, if anything, do you think they could make more explicit about the goals of this program?)

28. To what extent did you feel like you received the support you needed from IMLS to be as successful as possible as a grantee organization? (Possible examples of support include funding, program officer engagement and sharing, communication and responsiveness from IMLS) (DI’s program officer is active on the portal)

Wrap up

29. Is there anything else you think is important for us to know?

Appendix C – Mentor Organizations Interview Guides

This appendix includes the following data collection tools for use with the mentor organizations:

1. Quarterly interview guide

2. Cohort 1 End-point and Cohort 2 Mid-point Mentor Organization interview guide

3. Cohort 1 Mid-point interview guide

4. Cohort 2 End-point interview guide

Tool #1 - Mentor Organization – Quarterly Interview Guide

Mentor Organization:

Date:

Interviewee Name:

Interview conducted by:

Interview Objectives:

• Understand ongoing changes and challenges mentor organizations are experiencing due to COVID-19

• Learn how mentor organizations are adapting their curriculum and mentorship activities to accommodate grantees’ evolving circumstances

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

The purpose of this interview is to understand how your mentor organization is adapting your curriculum and programming in response to changes necessitated by COVID-19. We will share insights from this interview with IMLS and the other mentor organizations so that all stakeholders can learn from your experiences. During this interview I will ask you to reflect on the last few months, as well as think forward to the months to come. You can expect this interview to last about 30 minutes. Do you have any questions for me before we get started?

Questions:

1. In the last quarter, what aspects of the capacity-building program have you been able to implement effectively?

2. What aspects of the capacity-building program have you been unable to implement?

3. How have you adapted your programming to address ongoing changes and/or challenges for grantees?

4. How has the current climate (e.g., the social and economic fallout of COVID-19) shifted how your organization interacts with your grantees? (For example: Are they having more frequent communications with grantees? Are grantees relating to the mentors differently? Are grantees becoming less engaged? Or are grantees reaching out to them more? Are mentors initiating more informal, ad hoc contact with grantees?)

5. How has the current climate (e.g., the social and economic fallout of COVID-19) changed or affected how cohort members interact with each other?

   1. And with the broader libraries field?

6. How has the current climate (e.g., the social and economic fallout of COVID-19) changed or affected how cohort members are able to implement their projects?

   1. What, if any, adaptations have you seen them making?

7. As time progresses, are you seeing the effects of COVID-19 lessen? To what extent have grantees been able to resume their previous working conditions?

8. At this point, what plans, if any, do you have to change your remaining activities and curriculum for the current cohorts? (Probe for changes related to COVID-19, as well as changes they planned to make irrespective of COVID-19, based on what they’ve learned about the cohorts to date)

9. Is there anything else you think is important for us to know?

Tool #2 - Mentor Organization – Cohort 1 End-point and Cohort 2 Mid-point Mentor Organization Interview Guide

Mentor Organization:

Date:

Interviewee Name:

Interview conducted by:

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

Insights from this interview will be shared with IMLS to assist them in refining their future APP program strategy. During this interview I will ask you to look back over the last year, as well as think forward to the year to come. You can expect this interview to last about an hour and a half. [Note to Interviewer: Alternatively, this interview may be divided into two shorter interviews if more convenient for the mentor organizations.] Do you have any questions for me before we get started?

Cohort 2 Mid-point Interview

Interview Objectives:

• Gather mentor organization’s reflections on programmatic effectiveness and outcomes with Cohort 2 and learn about mentor organization’s plans for the next year

• Understand the extent to which planned programming for the cohorts has been cancelled or adapted due to COVID-19

• Understand how mentor organizations’ experience can inform IMLS strategy

For this first set of questions, I only want you to think about your experience with Cohort 2.

Effectiveness of the Capacity-Building Program

1. To what extent have you had to cancel or adapt your programming due to COVID-19?

   1. In what ways did you engage with cohort members differently than you had originally planned?

2. What aspects of the capacity-building program have been particularly successful with grantees? Why have these aspects of the model worked well?

3. With which aspects of the capacity-building program, did you find that cohort members were most engaged? Why do you think they were most engaged those aspects?

4. With which aspects of the capacity-building program have you had more challenges in your work with grantees? What do you think is the reason for this?

5. Reflecting on your own capacities as a mentor organization, where do you think you have been particularly successful with grantees? What do you think has contributed to this success?

6. Where do you think your mentor organization has had more challenges in your work with grantees? What do you think is the reason for this?

Observed Grantee Outcomes

7. Thinking back to your first interactions with your cohort, how would you describe their readiness and interest in having a mentor organization and being part of a cohort?

8. Looking back over the first year of the initiative, what would you say have been the biggest changes for grantees? (Probe for achievements and challenges).

1. What do you think led to those changes?

9. To what extent have you observed grantees incorporating or integrating what they are learning through the program into their grant-supported projects and other library activities?

1. What factors do you think continued to their success in integrating what they learned?

2. What barriers to integration, if any, have you observed?

10. Over the past year, what is the level of interaction you have observed between libraries in the cohort?

1. Does this level of interaction match with what you had expected?

2. Have there been any particular challenges to facilitating their interaction?

11. To what extent have you observed interactions between grantees and other members of the library (alt: school) community? Does this level of interaction match with what you had expected?

1. What do you think has contributed to this level of interaction?

12. What effect do you think this level of interaction is having on building up support for the library’s role in the community?

13. In general, how are grantees progressing with implementing their projects?

14. What changes have you seen related to COVID-19?

Planned Changes

15. What plans, if any, do you have to change your remaining activities and curriculum for this cohort?

    1. Why do you feel those changes are needed?

Cohort 1 End-point Interview

Interview Objectives:

• Gather mentor organization’s reflections on programmatic effectiveness and outcomes from the most recent year and the two years in total for Cohort 1

• Understand how mentor organizations’ experience can inform IMLS strategy

For the following set of questions, I want you to think back over your experience during this last year of working with Cohort 1.

Effectiveness of Mentorship Model & Capacity-Building Activities

1. Looking back over the last year, in what ways did you adapt your programming for Cohort 1?

   1. In what ways do you think you as a mentor organization was successful in adapting to grantees’ needs as they may have changed during/after COVID-19? Where do you think you could have improved in your responsiveness to grantees’ changing circumstances?

   2. In what ways do you think IMLS as the funder organization was successful in adapting to grantees’ needs as they may have changed during/after COVID-19? Where do you think IMLS could have improved in its responsiveness to grantees’ changing circumstances?

Now, I’d like to ask you about the specific components of your capacity-building model, one-by-one.

2. How effective do you think the cohort kick off meeting was? What worked well, what didn’t, and what would you change about this component next time?

3. How effective do you think the use of convenings (in-person or virtual) was? What worked well, what didn’t, and what would you change about this component next time?

4. How effective do you think the online cohort meetings/webinars/virtual meetings were? What worked well, what didn’t, and what would you change about this component next time?

5. How effective do you think the curriculum for cohort training was? What worked well, what didn’t, and what would you change about this component next time?

6. How effective do you think the online portal for cohort interactions was? What worked well, what didn’t, and what would you change about this component next time?

7. [If applicable] How effective do you think the individual learning plans were? What worked well, what didn’t, and what would you change about this component next time?

8. How effective do you think providing the online resource set (i.e., archived webinars, resource lists, checklists and other tools) was? What worked well, what didn’t, and what would you change about this component next time?

9. [For TSLP only] How effective do you think the pairing of grantees with school partners was? What worked well, what didn’t, and what would you change next time?

10. [For DI & CM only] How effective do you think the one-on-one coaching/check-ins were? What worked well, what didn’t, and what would you change next time?

11. Were there any other aspects of capacity-building that your organization provided? If so, how would you rate their effectiveness?

12. Now that you have completed a full grant cycle with this cohort, how would you compare what you had hoped to accomplish as a mentor organization with what you were able to accomplish?

    1. (Probe): What would you say contributed to the difference between what you had hoped and what actually occurred?

13. How do you think the absence of a mentor organization would have impacted grantees’ capacity building and progress on their grant-supported activities?

14. How do you think the absence of being a part of a cohort would have impacted grantees’ capacity building and progress on their grant-supported activities?

Observed Grantee Outcomes

15. Looking back over the last year of the initiative, what would you say were the biggest changes for grantees? (Probe for achievements and challenges). What do you think led to those changes?

16. Given that COVID-19 occurred in the middle of this cohort’s capacity-building program, what effects do you think this had on their ability to achieve the desired outcomes?

17. In the last year, to what extent did you observe grantees incorporating or integrating what they learned through the program into their grant-supported projects and other library activities?

    1. What factors do you think continued to their success in integrating what they learned?

    2. What barriers to integration, if any, did you observe?

    3. Did you notice a difference between the cohort’s first year and the second year in their ability to act on and integrate what they were learning in the program?

18. Over the past year, what is the level of interaction you have observed between libraries in the cohort? Have you seen interactions change between year 1 and year 2?

    1. What do you think has contributed to this level of interaction? (Possible prompts: perceived knowledge from other grantees, comfort with other grantees, encouragement from mentor organization to engage)

19. From your perspective, how well have grantees been able to engage people in their organization and local community in visioning and planning for library programs, services, or space?

    1. What particular successes or barriers to success have you seen?

20. In general, how much progress did grantees make towards implementing their projects?

21. From your perspective, how well have grantees been able to engage with other members of the library field (outside of their cohort)?

    1. What do you think has contributed to this level of interaction?

22. [For TSLP only] How successful do you think grantees were in increasing their knowledge, skills, and confidence needed to…

    1. …develop their libraries as dynamic hubs for self-directed, inquiry-based learning?

    2. …position themselves as school library professionals who are integral instructional partners to classroom teachers?

    3. …administer and manage a federal grant-funded project

    4. …implement, evaluate, and communicate learning about a multi-year project?

23. To what extent do you think grantees are better prepared to approach other activities or projects in response to evolving organizational and community needs?

Looking Forward

24. How well-prepared do you think Cohort 1 grantees are to continue their efforts in [Community Memory/Digital Inclusion/Transforming School Library Practice] in the absence of the grant funding, mentor organization, and formal cohort structure?

    1. To what extent do you think grantees will be able to maintain and integrate what they have learned over the last two years?

25. To what extent do you think grantees have the organizational support needed to sustain their projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff, or funding to support this work, or general interest in this work across their library or school).

26. To what extent do you think grantees have the community support needed to sustain their projects? (Support could include buy-in for this work from school or community members through demonstrations of their interest for their work, volunteers from the community to assist then in their work).

27. Now that you have completed your first round of the two-year grant cycle, are there areas of impact that you realize a mentor organization is best positioned to influence that you didn’t anticipate when you first started this work?

28. What did you learn about this experience that you will apply to working with small and rural libraries in the future?

Feedback to IMLS

29. Given what you now know after two full years of delivering this intervention, do you have any new insights you might offer IMLS around how to reach new libraries for the APP program?

30. To what extent do you feel like you have received the support you need from IMLS to be an effective mentor organization?

    1. Are there any other recommendations you have for IMLS to strengthen their partnership with your organization?

Wrap up

Your responses to the remaining questions can be based on your experiences working with either or both cohorts.

31. Is there anything else you think is important for us to know?

Tool #3 - Mentor Organization – Cohort 1 Mid-point Interview Guide

Mentor Organization:

Date:

Interviewee Name:

Interview conducted by:

Interview Objectives:

• Gather mentor organization’s reflections on programmatic effectiveness and outcomes for the last year

• Understand the extent to which planned programming for the cohorts has been cancelled or adapted due to COVID-19

• Understand how mentor organizations’ experience can inform IMLS strategy

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

Insights from this interview will be shared with IMLS to assist them in refining their future APP program strategy. During this interview I will ask you to look back over the last year, as well as think forward to the year to come. You can expect this interview to last about an hour. Do you have any questions for me before we get started?

Effectiveness of the Capacity-Building Program

1. To what extent have you had to cancel or adapt your programming due to COVID-19?

   1. In what ways did you engage with cohort members differently than you had originally planned?

2. What aspects of the capacity-building program have been particularly successful with grantees? Why have these aspects of the model worked well?

3. With which aspects of the capacity-building program, did you find that cohort members were most engaged? Why do you think they were most engaged those aspects?

4. With which aspects of the capacity-building program have you had more challenges in your work with grantees? What do you think is the reason for this?

5. Reflecting on your own capacities as a mentor organization, where do you think you have been particularly successful with grantees? What do you think has contributed to this success?

6. Where do you think your mentor organization has had more challenges in your work with grantees? What do you think is the reason for this?

Observed Grantee Outcomes

7. Thinking back to your first interactions with your cohort, how would you describe their readiness and interest in having a mentor organization and being part of a cohort?

8. Looking back over the first year of the initiative, what would you say have been the biggest changes for grantees? (Probe for achievements and challenges).

1. What do you think led to those changes?

9. To what extent have you observed grantees incorporating or integrating what they are learning through the program into their grant-supported projects and other library activities?

3. What factors do you think continued to their success in integrating what they learned?

4. What barriers to integration, if any, have you observed?

10. Over the past year, what is the level of interaction you have observed between libraries in the cohort?

3. Does this level of interaction match with what you had expected?

4. Have there been any particular challenges to facilitating their interaction?

11. To what extent have you observed interactions between grantees and other members of the library (alt: school) community? Does this level of interaction match with what you had expected?

2. What do you think has contributed to this level of interaction?

12. What effect do you think this level of interaction is having on building up support for the library’s role in the community?

13. In general, how are grantees progressing with implementing their projects?

1. What changes have you seen related to COVID-19?

Planned Changes

14. What plans, if any, do you have to change your remaining activities and curriculum for the current cohort? (Probe for changes related to COVID-19, as well as changes they planned to make irrespective of COVID-19, based on what they’ve learned about the cohort)

15. What plans, if any, do you have to change your activities or curriculum for the new cohort? (Probe for changes related to COVID-19, as well as changes they planned to make irrespective of COVID-19)

1. Why do you feel those changes are needed?

16. Over the next year, do you have plans for Cohort 1 and Cohort 2 to participate in any activities together? Is there any specific intent to have them interact or build relationships?

Comparison of Cohort 1 and Cohort 2 Applications

17. As you looked through the participants and their projects for Cohort 1 vs. Cohort 2, what differences did you observe?

1. What do you think contributed to those differences?

Potential Role in Library Outreach

18. Were you asked by IMLS to do any outreach to libraries for their Cohort 2 Notice of Funding Opportunity?

1. If so, what, if any, outreach did you do?

19. Given what you now know after a year of delivering this intervention, what insights might you offer to IMLS around how to reach new libraries for the APP program?

1. Is there any particular role that you would like to play related to this?

Wrap up

20. To what extent do you feel like you have received the support you need from IMLS to be an effective mentor organization?

1. Are there any other recommendations you have for IMLS to strengthen their partnership with your organization?

21. Is there anything else you think is important for us to know?

Tool #4 - Mentor Organization – Cohort 2 End-point Interview Guide

Mentor Organization:

Date:

Interviewee Name:

Interview conducted by:

Interview Objectives:

• Gather mentor organization’s reflections on programmatic effectiveness and outcomes from the most recent year and the two years in total for Cohort 2

• Understand how mentor organizations’ experience can inform IMLS strategy

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate their Accelerating Promising Practices for Small and Rural Libraries (APP) Program. The aim of the APP evaluation is to better understand the methods most effective in building the capacity of small and rural libraries to serve their communities in the areas of Digital Inclusion, Community Memory, and Transforming School Library Practice.

Insights from this interview will be shared with IMLS to assist them in refining their future APP program strategy. During this interview I will ask you to look back over the last year, as well as think forward to the year to come. You can expect this interview to last about an hour. Do you have any questions for me before we get started?

Effectiveness of Mentorship Model & Capacity-Building Activities

1. Looking back over the last year, in what ways did you adapt your programming for Cohort 2?

   1. In what ways do you think you as a mentor organization was successful in adapting to grantees’ needs as they may have changed during/after COVID-19? Where do you think you could have improved in your responsiveness to grantees’ changing circumstances?

   2. In what ways do you think IMLS as the funder organization was successful in adapting to grantees’ needs as they may have changed during/after COVID-19? Where do you think IMLS could have improved in its responsiveness to grantees’ changing circumstances?

Now, I’d like to ask you about the specific components of your capacity-building model, one-by-one.

2. How effective do you think the cohort kick off meeting was? What worked well, what didn’t, and what would you change about this component next time?

3. How effective do you think the use of convenings (in-person or virtual) was? What worked well, what didn’t, and what would you change about this component next time?

4. How effective do you think the online cohort meetings/webinars/virtual meetings were? What worked well, what didn’t, and what would you change about this component next time?

5. How effective do you think the curriculum for cohort training was? What worked well, what didn’t, and what would you change about this component next time?

6. How effective do you think the online portal for cohort interactions was? What worked well, what didn’t, and what would you change about this component next time?

7. [If applicable] How effective do you think the individual learning plans were? What worked well, what didn’t, and what would you change about this component next time?

8. How effective do you think providing the online resource set (i.e., archived webinars, resource lists, checklists and other tools) was? What worked well, what didn’t, and what would you change about this component next time?

9. [For TSLP only] How effective do you think the pairing of grantees with school partners was? What worked well, what didn’t, and what would you change next time?

10. [For DI & CM only] How effective do you think the one-on-one coaching/check-ins was? What worked well, what didn’t, and what would you change next time?

11. Were there any other aspects of capacity building that your organization provided? If so, how would you rate their effectiveness?

12. Now that you have completed a full grant cycle with this cohort, how would you compare what you had hoped to accomplish as a mentor organization with what you actually accomplished?

    1. (Probe): What would you say contributed to the difference between what you had hoped and what actually occurred?

13. How do you think the absence of a mentor organization would have impacted grantees’ capacity building and progress on their grant-supported activities?

14. How do you think the absence of being a part of a cohort would have impacted grantees’ capacity building and progress on their grant-supported activities?

Observed Grantee Outcomes

15. Looking back over the last year of the initiative, what would you say were the biggest changes for grantees? (Probe for achievements and challenges). What do you think led to those changes?

16. Given that COVID-19 occurred just before this cohort’s capacity-building program began, what effects do you think this had on their ability to achieve the desired outcomes?

17. In the last year, to what extent did you observe grantees incorporating or integrating what they learned through the program into their grant-supported projects and other library activities?

    1. What factors do you think contributed to their success in integrating what they learned?

    2. What barriers to integration, if any, did you observe?

    3. Did you notice a difference between the cohort’s first year and the second year in their ability to act on and integrate what they were learning in the program?

18. Over the past year, what is the extent of interaction you have observed between libraries in the cohort? For example, are you seeing more, less, or the same level of interaction?

    1. What do you think has contributed to this level of interaction? (Possible prompts: perceived knowledge from other grantees, comfort with other grantees, encouragement from mentor organization to engage)

19. From your perspective, how well have grantees been able to engage people in their organization and local community in visioning and planning for library programs, services, or space?

    1. What particular successes or barriers to success have you seen?

20. In general, how much progress did grantees make towards implementing their projects?

21. From your perspective, how well have grantees been able to engage with other members of the library field (outside of their cohort)?

    1. What do you think has contributed to this level of interaction?

22. [For TSLP only] How successful do you think grantees were in increasing their knowledge, skills, and confidence needed to…

    1. …develop their libraries as dynamic hubs for self-directed, inquiry-based learning?

    2. …position themselves as school library professionals who are integral instructional partners to classroom teachers?

    3. …administer and manage a federal grant-funded project

    4. …implement, evaluate, and communicate learning about a multi-year project?

23. To what extent do you think grantees are better prepared to approach other activities or projects in response to evolving organizational and community needs?

Looking Forward

24. How well-prepared do you think Cohort 2 grantees are to continue their efforts in the absence of the grant funding, mentor organization, and formal cohort structure?

    1. To what extent do you think grantees will be able to maintain and integrate what they have learned over the last two years?

25. To what extent do you think grantees have the organizational support needed to sustain their projects? (Support could include leadership buy-in, organizational policies that support this type of work, staff, or funding to support this work, or general interest in this work across their library or school).

26. To what extent do you think grantees have the community support needed to sustain their projects? (Support could include buy-in for this work from school or community members through demonstrations of their interest for their work, volunteers from the community to assist then in their work).

27. Now that you have completed your second round of the two-year grant cycle, what have you learned about areas in which a mentor organization is best positioned to provide support that you didn’t anticipate when you first started this work?

28. What did you learn about this experience that you will apply to working with small and rural libraries in the future?

Feedback to IMLS

29. Given what you now know after three full years of delivering this intervention, do you have any new insights you might offer IMLS around how to reach new libraries for the APP program?

30. To what extent do you feel like you have received the support you need from IMLS to be an effective mentor organization?

    1. Are there any other recommendations you have for IMLS to strengthen their partnership with your organization?

Wrap up

31. Is there anything else you think is important for us to know?

Appendix D – Non-Returning Applicants Survey

The Institute of Museum and Library Services (IMLS) has contracted with Partners for Public Good (PPG) to conduct an evaluation of its Accelerating Promising Practices for Small and Rural Libraries (APP) program. We are seeking your feedback about your experience as an applicant to the first round of this program.

The information you share with us is confidential and will only be presented in aggregate form (i.e., not individually identifiable). Your feedback will contribute to our understanding of the experiences and expectations of participants applying to the APP program so that IMLS can continue to learn and improve its services for small and rural libraries.

The survey takes approximately 10 minutes to complete. If you have any questions about the survey, please contact Lisa Frantzen, Senior Evaluation Consultant with PPG, at [email protected].

1. For which cohort did you apply?

1. Community Memory

2. Digital Inclusion

3. Transforming School Library Practice

Application Process for Cohort 1

2. Where did you originally find out about IMLS’ APP program? Please check all that apply.

1. IMLS website

2. IMLS social media

3. IMLS newsletter or email

4. Another organization’s website:_[please list]_______

5. Another organization’s social media:_[please list]_______

6. Another organization’s newsletter or email:_[please list]_______

7. Other: _____________________________________________

3. What excited you about this particular program and motivated you to apply?

____________________________________________________________

4. Did you have any reservations around applying? If so, what were they?

__________________________________________________________

5. Which resources did you use to learn about the program before applying?

1. IMLS webinar

2. IMLS FAQs

3. TBD. – These will be completed once we obtain the specific options from IMLS as part of our document review.

6. Please indicate the extent to which you agree with the following statements.

	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree	Not Applicable/ I did not use
The goals of the APP program were clear.	☐	☐	☐	☐	☐	☐
IMLS communications about the application process and deadlines were clear.	☐	☐	☐	☐	☐	☐
The review criteria for the APP program were clear.	☐	☐	☐	☐	☐	☐
If I had a question about the application process or materials, I knew where to go to get answers.	☐	☐	☐	☐	☐	☐
The support resources provided by IMLS were helpful for completing the application.	☐	☐	☐	☐	☐	☐
Completing the application was a fairly easy process for my library.	☐	☐	☐	☐	☐	☐
The amount of time it took my library to complete the application was reasonable.	☐	☐	☐	☐	☐	☐
The timeline for completing the application was reasonable.	☐	☐	☐	☐	☐	☐
I had all or almost all of the resources I needed to prepare and submit the proposal.	☐	☐	☐	☐	☐	☐
The reviewer comments I received on my application were helpful.

7. Did you contact your Program Officer to discuss your application or the application process at any time? If so, did you find that conversation helpful? Why or why not?

__________________________________________________________________________

8. Overall, what worked well about the application process IMLS used for the APP program?

_____________________________________________________________________

9. What suggestions for improvement do you have for future application rounds for the APP program?

_________________________________________________________________________

Application Process for Cohort 2

10. Are you aware that IMLS opened applications for a second round of the APP program?

1. Yes

2. No

3. Unsure

11. If so, how did you find out?

1. IMLS website

2. IMLS social media

3. IMLS newsletter or email

4. Another organization’s website:_[please list]_______

5. Another organization’s social media:_[please list]_______

6. Another organization’s newsletter or email:_[please list]_______

7. Other: _____________________________________________

12. Did you consider applying for the second round of the APP program? Why or why not?

________________________________________________________________________

13. If you did not end up applying for the second cohort of the APP program, please let us know why.

__________________________________________________________________________

Wrap up

14. Is there anything else you think is important for us to know?

_________________________________________________________________________

Library Demographics

15. To gather a basic understanding of the library that you represent, can you share…

1. How many staff members your library has? (separate by full-time vs. part-time) _______________

2. How many annual visitors your library has? ______________________

3. How would you characterize the population that your library serves? (rural/urban, particular demographics of the population such as race/ethnicity, ages, etc.) __________________

4. Has your library applied previously for IMLS grants?

• Yes

• No

• Unsure

Appendix E – IMLS Staff Interview Guide

Date:

Interviewee Name:

Interview conducted by:

Interview Objectives:

• Gather IMLS staff reflections on the recruitment and selection processes used for Cohorts 1 and 2.

• Gather IMLS staff reflections on the effectiveness of those processes.

Introduction:

Thank you for taking the time to speak with me today. As you know, IMLS is partnering with my organization, Partners for Public Good (PPG), to evaluate the Accelerating Promising Practices for Small and Rural Libraries (APP) Program. Our conversation today will focus on the recruitment processes for APP Cohorts 1 and 2 with the intent of understanding any recruitment differences that may have led to differences in the characteristics of libraries now forming Cohorts 1 and Cohort2.

This conversation is confidential, and all responses will be reported back in aggregate. During this interview I will ask you to look back over the two years during the APP cohort recruitment processes. You can expect this interview to last about an hour. Do you have any questions for me before we get started?

Background

1. What role did you play during the recruitment process (Processes used to outreach to potential applicants and the application processes experienced by APP applicants) for Cohort 1? What were your responsibilities?

   1. And for Cohort 2?

   2. And for which cohort(s): [Community Memory/Transforming School Libraries/Digital Inclusion]?

2. What role did you play during the selection process (Processes used to review the received applications and determine selection for APP Cohorts) process for Cohort 1? What were your responsibilities?

   1. And for Cohort 2?

   2. And for which cohort(s): [Community Memory/Transforming School Libraries/Digital Inclusion]?

[Note, the remaining questions for this interview may be selected from the list below depending on the interviewees’ specific responsibilities during recruitment & selection.]

Recruitment Processes (Processes used to outreach to potential applicants and the application processes experienced by APP applicants)

APP Cohort:  Community Memory  Digital Inclusion  Transforming School Library Practice

(Note to Interviewer: please designate which cohort and if interviewee participated in the recruitment process by more than one cohort, please ask the questions for each cohort process separately.)

3. How would you describe the target audience IMLS was aiming to reach for APP Cohort 1?

   1. And for Cohort 2? Did anything change? If so, what and why?

Note: The Q4 and Q5 series of questions are follow-up questions to allow us to clarify what was uncovered in the document review. These questions may be slightly adapted based on the document review.

4. What channels of communication (website, social media, library associations, grantee listserv, IMLS newsletter, etc.) did IMLS promote its Notice of Funding Opportunity for APP Cohort 1?

   1. To what extent did IMLS promote the program with its existing grantees? How was it promoted with them?

   2. Was any individual outreach conducted with any other audiences for reaching libraries to apply for the APP program?

   3. Did anything change for Cohort 2? If so, what were the reasons for those changes?

   4. To what extent were these channels of communication effective in reaching the target audiences for Cohort 1 and 2?

5. In what ways were prospective participants given opportunities to learn about the program before applying? [Answer for both Cohort 1 and 2.]

   1. And how were these opportunities promoted to prospective applicants? [Answer for both Cohort 1 and 2.]

   2. Did you or any other IMLS staff field questions directly from potential applicants?

      1. If so, what was the process/expectation for responding to these inquiries? (listen for promptness in response, thoroughness of response, referral to other resources, etc.)

      2. How frequently were inquiries received?

6. From your perspective, to what extent were the external communications about the application period deadlines clear? [Answer for both Cohort 1 and 2.]

7. What, if any, changes were made to the application materials, process, or requirements between Cohorts 1 and 2?

   1. In your opinion, were these changes appropriate? Why or why not?

8. From your perspective, to what extent were the application materials, process and requirements equitable? Were there any groups for which the application process may have been too high a barrier?

9. To what extent was the overall application timeline appropriate? [Answer for both Cohort 1 and 2.]

   1. To your knowledge, were there other big events or other factors relevant to small and rural libraries occurring at the same time that may have prevented libraries from applying?

10. What worked well about the application process IMLS has used for the APP cohorts?

11. What suggestions for improvement do you have for future application rounds for the APP program?

Selection Processes (Processes used to review the received applications and determine selection for APP Cohorts)

APP Cohort:  Community Memory  Digital Inclusion  Transforming School Library Practice

(Note to Interviewer: please designate which cohort and if interviewee participated in the selection process for more than one cohort, please ask the questions for each cohort process separately.)

Note: These questions will accompany PPG’s review of both NOFO review criteria and the instructions that are given to reviewers.

12. Were there any groups that may have been good candidates for the program, but which did not match the criteria?

13. As you reviewed applications for Cohort 1, what was your impression of the quality of applications received?

    1. And for Cohort 2?

    2. Did you perceive any differences in quality across the two sets of applications? If so, what differences did you notice?

    3. From your perspective, what may have caused the differences between the two set of applications?

14. What processes were used for determining the final selection? [Answer for both Cohort 1 and 2.]

15. What, if any, changes were made to the selection process between Cohorts 1 and 2?

16. What worked well about the selection process IMLS has used for the APP cohorts?

17. What suggestions for improvement do you have for future selection rounds for the APP program?

Looking Forward

18. What role, if any did the Cohort 1 mentor organizations play in recruiting libraries for Cohort 2?

19. Looking forward, what role do you think the mentor organizations could play in helping IMLS reach new libraries?

    1. What, if any, recommendations do you have for incorporating them into the process?

Wrap up

20. Is there anything else you think is important for us to know?

Appendix F – Rubrics for Grantee & Observations Analysis of Administrative Records

This appendix includes the following data collection tools for use in grantee observations and administrative records review:

1. Document review rubric for mentor records

2. Document review rubric for grantee reports

3. Document review rubric for APP Recruitment & Selection

4. Observation rubric for Online Gathering/In-Person Convenings

5. Observation rubric for web portal activity

Tool #1 - Document Review Rubric – Mentor Records

Please use the following rubric to complete the following questions. Please identify the data source

Mentor Organization: ___________________________________________________________

Cohort:  Community Memory  Digital Inclusion  Transforming School Library Practice

Objective: To understand the dosage of the capacity-building program received by members of each APP cohort.

Review of Mentor Organization Records

Capacity-Building Component		Cohort 1	Cohort 2
Cohort Kick-off Meeting	Amount of time:
	Number of grantees that participated:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Year 1 convening	Amount of time:
	Number of grantees that participated:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Year 2 convening	Amount of time:
	Number of grantees that participated:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Final convening	Amount of time:
	Number of grantees that participated:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Virtual meetings/webinars	Total number of webinars offered:
	Average amount of time per webinar:
	Average number of grantees that participated per webinar:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Online portal	Dates that portal was open to this cohort:
	Number of grantees that used the portal:
	Average frequency of activity (to be pulled from evaluators review of portal):
	Number of mentor organization personnel (or partners) responsible for delivering this component:
One-on-one coaching/check-ins [if applicable]	Number of sessions available to grantees:
	Average amount of time per session:
	Average number of sessions received by grantees:
	Number of mentor organization personnel (or partners) responsible for delivering this component:
Other? __________________

Tool #2 - Document Review Rubric – Grantee Reports

Please use the following rubric to complete the following questions. Please identify the data source

Grantee: ___________________________________________________________________

Cohort 1 or 2: ___________________

Cohort:  Community Memory  Digital Inclusion  Transforming School Library Practice

Note, the below may be updated once a copy of the IMLS report formats has been shared.

Review of Mentor Organization Records

Please check off all of the capacity-building activities that this grantee has participated in.

• Kick off meeting

• Year 1 convening

• Year 2 convening

• Final convening

• Virtual meetings/webinars. How many? _______________

• Online portal [Note, more details gathered by evaluators in the portal observation rubric.]

• One-on-one coaching or check-ins [if applicable] How many? _______________

• Other?______________________________________________

Review of Grant Midpoint Grant Report

1. Grantee report of activities planned for their IMLS project.

2. Grantee report of activities implemented for their IMLS project.

3. How many different organizational practices did this grantee implement in their library? What types of organizational practices did grantees implement in their library?

4. How many and what different types of communications efforts have grantees undertaken to raise awareness of their project to their community?

5. What evidence does grantee provide to indicate that the mentor activities have impacted the quality and effectiveness of their project?

6. What evidence does grantee provide to indicate participating in a cohort has impacted the quality and effectiveness of their project?

7. What evidence does grantee provide to indicate that they have incorporated community partners into their grant-sponsored activities and library planning?

8. Any other outcomes (positive or negative) reported by grantee

Review of Grant Final Grant Report

9. Grantee report of activities planned for their IMLS project.

10. Grantee report of activities implemented for their IMLS project.

11. How many different organizational practices did this grantee implement in their library? What types of organizational practices did grantees implement in their library?

12. How many and what different types of communications efforts have grantees undertaken to raise awareness of their project to their community?

13. What evidence does grantee provide to indicate that the mentor activities have impacted the quality and effectiveness of their project?

14. What evidence does grantee provide to indicate participating in a cohort has impacted the quality and effectiveness of their project?

15. What evidence does grantee provide to indicate that they have incorporated community partners into their grant-sponsored activities and library planning?

16. Any other outcomes (positive or negative) reported by grantee

Tool #3 - Document Review Rubric for APP Recruitment & Selection

Date:

Review conducted by:

Review Objectives:

• Review specific details of the recruitment and selection processes used by IMLS for Cohorts 1 and 2. Analyze differences between processes used for Cohort 1 vs. Cohort 2.

APP Cohort:  Community Memory  Digital Inclusion  Transforming School Library Practice

Recruitment Processes (Processes used to outreach to potential applicants and the application processes experienced by APP applicants)

Marketing & Outreach Materials

1. How was the target audience that IMLS was aiming to reach for APP Cohorts 1 & 2 described and in what materials?

Cohort	Target Audience Description	Materials
Cohort 1
Cohort 2

2. What specific marketing materials were used to promote the APP Notice of Funding Opportunity? What content did they contain and for what target audiences?

Marketing Materials (e.g., Application Intro sheet, Social media summary post) Please list all.		Cohort 1	Cohort 2
Resource 1:_____________	Specific location(s) used:
	Frequency of use:
	Target audience:
	Content contained (instructions, directions to website, deadline, etc.):
	PPG reviewer observations (on presentation language, etc.):
Resource 2:_____________	Specific location(s) used:
	Frequency of use:
	Target audience:
	Content contained (instructions, directions to website, deadline, etc.):
	PPG reviewer observations (on presentation language, etc.):
Resource 3:_____________	Specific location(s) used:
	Frequency of use:
	Target audience:
	Content contained (instructions, directions to website, deadline, etc.):
	PPG reviewer observations (on presentation language, etc.):
Resource 4:_____________ (Add more rows as needed)	Specific location(s) used:
	Frequency of use:
	Target audience:
	Content contained (instructions, directions to website, deadline, etc.):
	PPG reviewer observations (on presentation language, etc.):

3. What channels of communication were used by IMLS in promoting its Notice of Funding Opportunity for the APP Cohorts? How frequently and effectively were these channels used?

Channel (i.e., Where marketing resources were shared)		Cohort 1	Cohort 2
IMLS Website	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
IMLS Social Media - Twitter	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
IMLS Social Media - Facebook	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
Library Associations	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
Grantee listserv	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
Other ways of promoting with current grantees (please list):____________________	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
Any ways of promoting with for Cohort 2 with current mentor organizations: ___________	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
IMLS newsletter	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):
Other (please list all other channels, insert rows as necessary):____________	Specific location used:
	Dates info was disseminated (list all instances):
	Marketing materials shared:
	PPG reviewer observations (on presentation language, etc.):

4. Was any individual outreach conducted by IMLS staff to reach libraries to apply for the APP program?

Cohort	How/where Communicated	Frequency	PPG Reviewer Notes
Cohort 1
Cohort 2

5. How were application deadlines communicated?

Cohort	How/where Communicated	Frequency	PPG Reviewer Notes on Clarity, etc.
Cohort 1
Cohort 2

6. How were application requirements communicated to applicants?

Cohort	How/where Communicated	Frequency	PPG Reviewer Notes
Cohort 1
Cohort 2

Application Materials, Supports & Process

7. In what ways were prospective participants given opportunities to learn about the program before applying?

Support Resources (please list all)		Cohort 1	Cohort 2
Resource 1:_____________	Specific location(s) used:
	Dates info was disseminated (list all instances):
	How promoted (after original posting – retweets, reposts, emails, etc.):
	PPG reviewer observations:
Resource 2:_____________	Specific location(s) used:
	Dates info was disseminated (list all instances):
	How promoted (after original posting – retweets, reposts, emails, etc.):
	PPG reviewer observations:
Resource 3:_____________	Specific location(s) used:
	Dates info was disseminated (list all instances):
	How promoted (after original posting – retweets, reposts, emails, etc.):
	PPG reviewer observations:
Resource 4:_____________ (Add more rows as needed)	Specific location(s) used:
	Dates info was disseminated (list all instances):
	How promoted (after original posting – retweets, reposts, emails, etc.):
	PPG reviewer observations:

8. What was the application process?

Cohort	Application Process	PPG Reviewer Notes
Cohort 1
Cohort 2

9. What application materials were used?

Cohort	Application Materials	PPG Reviewer Notes
Cohort 1
Cohort 2

10. What were the application requirements?

Cohort	Requirements	PPG Reviewer Notes
Cohort 1
Cohort 2

11. What was the application timeline?

Cohort	Timeline	PPG Reviewer Notes
Cohort 1
Cohort 2

12. PPG reviewer notes on the extent to which the application materials, process and requirements were equitable? Were there any groups for which the application process may have been too high a barrier?

    Cohort	PPG Reviewer Notes
    Cohort 1
    Cohort 2

13. PPG reviewer notes on what worked well about the application process IMLS has used for the APP cohorts:

Cohort	PPG Reviewer Notes
Cohort 1
Cohort 2

14. PPG reviewer notes on what could be improved about the application process IMLS has used for the APP cohorts:

Cohort	PPG Reviewer Notes
Cohort 1
Cohort 2

Selection Processes (Processes used to review the received applications and determine selection for APP Cohorts)

Selection Materials & Process

15. Applications received and advanced

Cohort	# of applications received	# of applications advanced to review/considered for funding
Cohort 1
Cohort 2

16. Composition of selection committee

Cohort	Names of those on selection committee (list all)	Position/role	IMLS Program Officer, Director or Field Reviewer
Cohort 1
Cohort 2

17. Review of internal communications for selection process, including guidelines for IMLS and/or field reviewers

Cohort	How/where Communicated	PPG Reviewer Notes on Process Clarity	PPG Reviewer Notes on Role Clarity	PPG Reviewer Notes on Timeline Clarity	Other PPG Reviewer Notes
Cohort 1
Cohort 2

18. What was the review process?

Cohort	Review Process	% of applications that scored on each level of the reviewers’ rating rubric	PPG Reviewer Notes
Cohort 1		Exceptional (7): Excellent (6): Very Good (5): Good (4) Some Merit (3): Poor (2): Inadequate/Insufficient) (1):
Cohort 2		Exceptional (7): Excellent (6): Very Good (5): Good (4) Some Merit (3): Poor (2): Inadequate/Insufficient) (1):

19. What was the final selection process?

Cohort	Final Selection Process	PPG Reviewer Notes
Cohort 1
Cohort 2

20. What selection materials were used? (including selection instructions/guidelines and scoring materials)

Cohort	Application Materials	PPG Reviewer Notes
Cohort 1
Cohort 2

21. What were the selection criteria? (Review in NOFO review criteria and instructions given to reviewers.)

Cohort	Requirements	PPG Reviewer Notes
Cohort 1
Cohort 2

22. What was the selection timeline?

Cohort	Timeline	PPG Reviewer Notes
Cohort 1
Cohort 2

23. PPG reviewer notes on the extent to which the selection process was equitable. Were there any groups that may have been good candidates for the program, but which did not match the criteria?

Cohort	PPG Reviewer Notes
Cohort 1
Cohort 2

Application Review

24. Review of the 12 repeat applications from year 1; extent to which applications were updated

Application ID#	IMLS feedback to this applicant (if applicable)	PPG Reviewer Notes on any updates made to applications [Alignment of these updates to Field Reviewer feedback, if applicable]	PPG Reviewer notes on demographics of repeat applicants

Wrap up

25. PPG reviewer summary notes

Cohort	PPG Reviewer Notes
Cohort 1
Cohort 2

Tool #4 – Observation Rubric - Online Gathering/In-Person Convening

Instructions: Check the box that best fits the level of development you observe and use the adjoining space to provide any additional details related to your observations:

	No Evidence There is no observable evidence of this	Emerging Occasional observations of this but they are rare to infrequent.	Developing This is observed somewhat frequently. There appears to be a commitment to this practice but the changes are still not institutionalized.	Deepening/Sustaining Evidence of this is frequently observed. The practice appears to be deeply embedded in how grantees are operating and where appropriate, a degree of mastery is observed.
Grantee and Mentor Organization Interactions
Mentors clearly communicate the rationale for the online gathering/convening to grantees				
Grantees are actively engaged in the online gathering/convening (paying attention, asking questions, taking notes, etc.)				
Grantees reference materials and resources provided by the mentor organizations during the online gathering/convening				
Grantees express appreciation to the mentor organization during the online gathering/convening				
Grantees express appreciation for something that the mentor org has done or shared during the online gathering/convening				
Use the space below to provide additional details on grantee and mentor organization interactions.
Grantee Peer Interactions
Grantees provide social support to their peers during the online gathering/convening (e.g., encouragement, validation)				
Grantees offer advice and/or resources to their peers (e.g., offer advice on projects and/or grants management, share resources)				
Use the space below to provide additional details on grantee peer interactions.
Grantee Awareness, Knowledge, and Skills Development
Grantees demonstrate new or increased knowledge in a capacity-building target area				
Grantees’ knowledge in the target areas appear to be linked to content presented during the online gathering/convening				
Grantees discuss how they have applied a new skill or knowledge directly to their IMLS-funded project				
Grantees are asking questions or talking about topics that move beyond the “how-to’s” of project implementation (e.g., evaluation, knowledge sharing, the adaptability of the library)				
Grantees discuss opportunities during the online gathering/convening to apply new learning and insights gained from mentor-led capacity-building activities				
Grantees report sharing or interacting with the larger library field related to their projects.				
Grantees report ways that they have gotten their communities involved in their projects.				
Use the space below to provide additional details on grantee awareness, knowledge, and skills development.

Instructions: Use the space below to provide additional details on the following observations:

Note if there is a high level of attendance at the intervention.
Note if grantees report any feedback related to length/timing/effort level of the intervention.
Note if grantees report any things they particularly liked or any recommendations for improvement for the online gathering/convening
Note type and duration of the intervention observed during the online gathering/convening and rationale/objectives if stated
Note if grantees report any organizational barriers or facilitators to implementing new knowledge or skills or their projects
Note if grantees report any community barriers or facilitators to implementing new knowledge or skills or their projects

Tool #5 - Rubric Portal Evaluation Protocol

Instructions and Definitions

Posting Scoring Sheet

Appendix F – APP Program Logic Model

Appendix G – Cost Burden Estimates

Estimates of Cost to Respondents

		INTERVIEW Cost Burden – Non Returning Applicants
	Mean hourly wage	Interview length (in hrs)	Interview cost per respondent	# interview respondents	Estimated burden of interview
Year 1	$25.61	0	$29.58	0	$0.00
Year 2	$25.61	0.166 (10 minutes)	$29.58	20	$591.60
Year 3	$25.61	0	$29.58	0	$0.00
All Years		12	$29.58	20	$591.60

Estimates of Cost to the Federal Government

Estimates of Cost to the Federal Government
	Mean Hourly Wage	Length (in hrs)	Cost per respondent	# of staff members	Estimated Burden
Year 1 OLS Staff	$60.00	12	$720.00	1	$720.00
Year 1 ODIS Staff	$59.00	50	$2,950.00	1	$2,950.00
Year 2 OLS Staff	$60.00	12	$720.00	1	$720.00
Year 2 ODIS Staff	$59.00	50	$2,950.00	1	$2,950.00
Year 3 OLS Staff	$60.00	12	$720.00	1	$720.00
Year 3 ODIS Staff	$59.00	50	$2,950.00	1	$2,950.00
3RD Party Evaluator	-	-	-	-	$616,028.00
All Years		186	$10,974.00	5	$627,038.00

Appendix H – Cross Walk of Research Questions to Data Sources

Table H1: Alignment of Evaluation Goals and Questions

The evaluation design seeks to test the theory of change by pursuing these primary evaluation goals:

1. Understand the extent to which a third-party mentor organization model is effective in building the capacity of small and rural libraries to engage with their communities; identify best practices in using a mentor organization model.

2. Understand the extent to which a cohort-based model is effective in building the capacity and connecting small and rural libraries and identify best practices in using a cohort model.

3. Understand the extent to which COVID-19 disrupted the planned project and capacity-building activities of the APP program, and how program activities were adapted.

4. Understand the extent to which capacity-building gains are sustained and identify any additional supports needed by small and rural libraries in supporting their communities in the areas of Community Memory, Digital Inclusion, and Transforming School Library Practice.

5. Identify any aspects of the recruitment and selection process that may have influenced different incoming characteristics between libraries in Cohorts 1 and 2.

The table below provides the questions to be answered by this evaluation and maps them to the overarching goals of the evaluation.

Goal #	EQ #	Evaluation Question (EQ)
1,2	1	What core competencies did each cohort grantee already possess upon their arrival into APP?
1,2,3	2	What was the variation in interventions (mentor organizations’ activities, coaching, tools, etc.) provided and the rationale for those variations? Were capacity-building plans changed for Cohort 2?
1,2	3	What capacity-building activities were most effective for building grantees’ knowledge in the target areas?
1	4	What capacity-building activities could be improved upon? What d mentor organizations learn about their own organizational capacity as mentors? What will they carry forward in their work with other communities, regions?
1,2	5	Did the cohort grantees acquire the core competencies that were conveyed through the mentor organization and capacity-building activities?
2	6	To what extent did cohort participation improve the quality and effectiveness of each grantee’s project development processes and outcomes?
3	7	To what extent where grantee project plans altered due to COVID-19? To what extent were cohort capacity-building plans altered?
4	8	To what extent did grantees implement what they learned from APP capacity-building (core competencies and peer learning) to enhance their overall institutional programmatic strategies? What, if any, barriers exist to integrating these learnings (including organizational factors)?
1	9	To what extent did grantees implement their core competencies within their projects from the mentor organization assistance and activities? What, if any, barriers exist to implementing these competencies?
4	10	To what extent did participation in APP lead to enhanced training and talent development of grantees’ staff members or in any participating community partner organizations?
1,2	11	To what extent did the variety of approaches and outcomes of the capacity-building and mentor organization assistance/ activities produce different or similar outcomes across cohorts?
1	12	What is the value in investing in mentor organizations and coaching? (i.e., What benefits are derived in comparison to costs incurred?)
2	13	What is the value of peer cohort learning opportunities? (i.e., What benefits are derived in comparison to costs incurred?)
4	14	Is there alignment between the grantee team’s self-reporting of changed practices with what the grantee’s institution is reporting?
2	15	To what extent did the cohort-based approach enhance the project’s programmatic reach/impact?
4	16	What variables in each grantee’s community, partnerships, and institution affected the success of APP’s capacity-building efforts and likelihood of adoption of these new practices? E.g., Did organizational practices, policies (or the lack thereof), or culture present barriers to the grantee adopting practices they learned in their cohort? Was there a lack of buy-in by key stakeholders to the capacity-building efforts?
4	17	What communications efforts were undertaken by each grantee to raise awareness and frame the intent of their projects to their communities?
4	18	To what extent were these communications efforts effective?
5	19	Were there differences in the recruitment and selection processes used for APP Cohorts 1 and 2? If so, in what ways could these differences have contributed to changes in the applications received for each cohort?
5	20	What role can mentor organizations play in helping IMLS reach new libraries?
5	21	What was the experience of applicants to Cohort 1 that were not selected? What influenced their decision not to apply to Cohort 2?

Table H2: Evaluation questions and associated tool items for data collection

EQ#	Evaluation Question	Indicators	Grantee APP Core Capacity Survey	Grantee Midpoint Survey	Grantee Endpoint Survey	Grantee Endpoint Interviews	Mentor Midpoint Interviews	Mentor Quarterly Interviews	Mentor Endpoint Interviews	Mentor Records Review	Grantee Reports Review	Observations (COP, Virtual Mtgs.)	IMLS Staff Interviews	IMLS Document Review	Non-returning Cohort 1 Applicants Survey
1	What core competencies did each cohort grantee already possess upon their arrival into APP?	• Baseline competencies in core competency areas for each cohort (project planning, project implementation, managing grant funds, cohort-specific competencies)	Q2.b-f, h &i, Q7.a-d, g-h, Q9.a-g, Q11.c-j, l, Q14
2	What was the variation in interventions (mentor organizations’ activities, coaching, tools, etc.) provided and the rationale for those variations? Were capacity-building plans changed for Cohort 2?	• Type and duration of capacity-building activities/support provided (Cohorts 1 & 2) • Rationale for activities/support provided • Level of engagement by cohort members in the activities/support provided • Type of mentor roles and capacity-building plans in responding to COVID-19 •Adaptations made by mentor organizations for Cohort 2 • Level and type of cross-cohort capacity-building activities		Q3, Q5, Q7, Q9, Q11-Q13, Q14-Q17	Q6, Q7 Q8		Q1-Q6, Q14-Q16	Q1-Q4, Q8		All	All	X
3	What capacity-building activities were most effective for building grantees’ knowledge in the target areas?	• Activities of maximum value for cohort members				Q3-Q10	Q2-Q3				Q5-Q6, Q13-Q14	X
4	What capacity-building activities could be improved upon? What did mentor organizations learn about their own organizational capacity as mentors? What will they carry forward in their work with other communities, regions?	• Activities with limited/less value to cohort members • Cohort members' recommendations for improvement • Mentor org reflections on own role as capacity builder • Effectiveness of mentor roles in responding to COVID-19		Q16 & Q17		Q3-Q10	Q4-Q6		Q1-Q14		Q8, Q16	X
5	Did the cohort grantees acquire the core competencies that were conveyed through the mentor organization and capacity-building activities?	• Endline competencies in core competency areas for each cohort	Q2.b-f, h &i, Q7.a-d, g-h, Q9.a-g, Q11.c-j, l, Q14	Q2.b-f, h &i, Q4.a-d, g-h, Q6.a-g, Q8.c-j, l, Q18 & Q19	Q1.b-f, h &i, Q2.a-d, g-h, Q3.a-g, Q4.c-j, l, Q9 & Q10	Q16-Q22					X
6	To what extent did cohort participation improve the quality and effectiveness of each grantee’s project development processes and outcomes?	• Quantity and quality of interactions between cohort members • Application of cohort learnings to grantee project	Q13.d-g, Q17-Q21	Q10.d-g, Q14, Q15, Q20-Q23	Q5.d-g, Q8, Q11-Q14	Q14, Q15	Q10		Q2-Q4, Q6, Q14		Q6, Q14	X
7	To what extent where grantee project plans altered due to COVID-19? To what extent were cohort capacity-building plans altered?	• Project plans changes due to COVID-19 • Capacity-building changes due to COVID-19		Q3, Q5, Q7, Q9, Q13-Q15, Q17, Q20-Q23, Q28 & Q29			Q13	All	Q16
8	To what extent did grantees implement what they learned from APP capacity-building (core competencies and peer learning) to enhance their overall institutional programmatic strategies? What, if any, barriers exist to integrating these learnings (including organizational factors)?	• Application of cohort core competencies to other programs or initiatives in their library	Q2.g, Q17.d & h	Q2.g, Q14.d&h, Q15 d&h	Q1.g, Q8.d&h	Q19-Q22Q26-Q28	Q8, Q10-Q12		Q17, Q19, Q21-Q22		Q3, Q7, Q11, Q15
9	To what extent did grantees implement their core competencies within their projects from the mentor organization assistance and activities? What, if any, barriers exist to implementing these competencies?	• Application of cohort core competencies to grantee projects	Q2.a, Q3-Q6, Q8, Q9a-e , Q10, Q11.a-l, Q12, Q13.a-g, Q14-16, Q17.a-c, Q20, Q22-24	Q2.a, Q3, Q5, Q6.a-e, Q7-Q13, Q14.a-c, Q15.a-c, Q24-Q29	Q1.a, Q3.a-e, Q4, Q5-Q7, Q8.a-c, Q15-Q21	Q18	Q9		Q17, Q20		Q5, Q13	X
10	To what extent did participation in APP lead to enhanced training and talent development of grantees’ staff members or in any participating community partner organizations?	• Number and type of personnel working on CM/DI/TSLP projects • Confidence levels of cohort members in implementing CM/DI/TSL projects • Changes in competency levels of cohort members in implementing CM/DI/TSL projects	Q2.e,f,h &l, Q7.a-d, g, &h, Q9b,c,f, &g, Q11.f-h, Q22	Q2.e,f,h &l, Q4.a-d,g,h, Q6b,c,f,&g, Q8.f-h, Q24	Q1.e,f,h &l, Q2.a-d,g,h, Q3b,c,f,&g, Q4.f-h, Q15	Q19, Q21, Q25, Q28					Q7, Q15
11	To what extent did the variety of approaches and outcomes of the capacity-building and mentor organization assistance/ activities produce different or similar outcomes across cohorts?	• Comparison of mentor intervention models and grantee types with cohort member outcomes [Analysis question]	All	All	All	All	All	All	All		All	X
12	What is the value in investing in mentor organizations and coaching? (i.e., What benefits are derived in comparison to costs incurred?)	• Cohort members' report of value of the mentor organizations and coaching received as compared to time invested		Q16		Q11-Q13	Q5					X
13	What is the value of peer cohort learning opportunities? (i.e., What benefits are derived in comparison to costs incurred?)	• Experience level of grantees' libraries in participating in a larger learning community of librarians • Depth and breadth of interactions between cohort members • Cohort members' report of value of the peer cohort to their own learning • Cohort members' report of value of the peer cohort to their own confidence building	Q13.c	Q10.c, Q17	Q4.c	Q3-Q5, Q14-Q15	Q2, Q3, Q10, Q16		Q2-Q4, Q6, Q14			X
14	Is there alignment between the grantee team’s self-reporting of changed practices with what the grantee’s institution is reporting?	• Comparison of cohort members' beliefs in changed culture to number of changed organizational practices implemented by their libraries	Q2.a, Q3-Q6, Q7.i&j, Q9.h&I, Q11.e, Q14-Q16, Q21,Q22,Q24	Q2.a, Q4.i&j, Q6.h&I, Q8.e, Q14-Q17, Q23,Q24,Q27	Q1.a, Q2.i&j, Q3.h&I, Q4.e, Q14-Q17	Q16-Q23					Q3, Q11	X
15	To what extent did the cohort-based approach enhance the project’s programmatic reach/impact?	• Cohort members' report of value of the peer cohort to their CM/DI/TSL projects	Q18	Q18	Q9	Q14-Q15			Q19, Q21-Q22		Q6, Q14
16	What variables in each grantee’s community, partnerships, and institution affected the success of APP’s capacity-building efforts and likelihood of adoption of these new practices? E.g., Did organizational practices, policies (or the lack thereof), or culture present barriers to the grantee adopting practices they learned in their cohort? Was there a lack of buy-in by key stakeholders to the capacity-building efforts?	• Experience level of grantees' libraries with implementing CM/DI/TSL projects • Experience level of grantees' libraries with aligning library services to the needs of their communities • Existence of library-wide support and resource allocation for implementing CM/DI/TSL projects • Existence of community support and resource allocation for implementing CM/DI/TSL projects • Experience level of grantees' libraries with interaction with the larger library field • Librarians' view of their role in contributing knowledge to the broader field • Librarians' view of their role in facilitating community knowledge • Librarians' view of their library's role in providing access to information, ideas, and networks	Q3 - Q6, Q13.a&b, Q17, Q21	Q10.a&b, Q14, Q15, Q23	Q5.a&b, Q8, Q14	Q16-Q28	Q11, Q12		Q23-Q26			X
17	What communications efforts were undertaken by each grantee to raise awareness and frame the intent of their projects to their communities?	• Quantity, quality, and frequency of communication efforts undertaken by grantees regarding raising awareness of their projects with their communities				Q21-Q22					Q4, Q12
18	To what extent were these communications efforts effective?	• Self-report of engagement by the communities with these communications efforts				Q22					Q3-Q4, Q7, Q11-Q12, Q15
19	Were there differences in the recruitment and selection processes used for APP Cohorts 1 and 2? If so, in what ways could these differences have contributed to changes in the applications received for each cohort?	• Outreach methods used and frequency • Identification of target audiences • Application materials and requirements used • Review and selection criteria used • Application timelines • # of repeat applicants and applications • Characteristics of applicant organizations											All	All
20	What role can mentor organizations play in helping IMLS reach new libraries?	• Mentor organization interest level, capacity, network, and vision for the partnership					Q18-Q19						Q22-23
21	What was the experience of applicants to Cohort 1 that were not selected? What influenced their decision not to apply to Cohort 2?	•Self-report of experience of Cohort 1 application process •Self-report of considerations and reasons for not applying to Cohort 2 •Characteristics of applicant organizations													All

Table H3: Anticipated data sources, method of analysis, and evaluative information

Data Source	Data Collection Method	Analysis Methods	Primary Areas to be Addressed
Grantees/ Cohort Members	APP Core Capacity Survey	Descriptive analysis of survey data Thematic analysis of open-ended questions	Grantee organization demographics Baseline and post-program core competencies Most effective mentor activities Impacts on project quality and effectiveness Existence of enhanced internal training	Community partner integration Mentoring, peer networks value Awareness-raising communications Implementation and integration of learnings; existence of any barriers to implementation Organizational factors that support or hinder adoption and sustaining of new practices
	Cohort 1 Midpoint survey	Descriptive analysis of survey data Thematic analysis of open-ended questions Comparative analysis with baseline data	Experience of being a cohort member including thoughts, feelings, perceptions Comparison of cohort member experiences by different demographics (extent to which experience was equitable)	Role, if any, that mentor organizations played in any changes made due to COVID-19 Thoughts, feelings, perceptions of cohort members’ use of new skills within their communities New knowledge gained from mentoring and peer network activities Implementation and integration of learnings
	Endpoint Grantee Interviews	Descriptive analysis of survey data Thematic analysis of open-ended questions Trend analysis with baseline and mid-point data	Most/least effective mentor activities Cohort participation’s impact on project quality and effectiveness Experience of being a cohort member including thoughts, feelings, perceptions Comparison of cohort member experiences by different demographics (extent to which experience was equitable)	Implementation and integration of learnings; existence of any barriers to implementation Community partner integration Mentoring, peer networks value Awareness-raising communications Thoughts, feelings, perceptions of cohort members’ use of new skills within their communities New knowledge gained from mentoring and peer network activities Implementation and integration of learnings
Mentor Organizations	Midpoint Mentor Organization Interviews	Thematic analysis of interview responses	Plans for cross-cohort collaboration Reflections on past year and any planned changes for capacity-building support for the upcoming year	Adaptations made to Cohort 1 and/or 2 programming due to COVID-19 Impression of quality of Cohort 2 applications in comparison to Cohort 1 Potential role in helping IMLS reach new libraries
	Endpoint Mentor Organization Interviews	Thematic analysis of interview responses Comparisons with mid-point data	Mentor activities provided and rationale Observed effectiveness of the capacity-building model including areas for improvement Receptivity of grantee organizations to mentoring	Cross-organizational interactions Integration of community partners; existence of any barriers to integration Reflection of mentor organizations on what learned about their own organizational capacity as mentors and how they would apply to future work with small libraries
Cohort Document Review & Observations by PPG	Document Review	Descriptive analysis of data (e.g., number of participants in programming) Thematic analysis	Mentor activities provided/variations Cohort member engagement in capacity-building activities Cohort participation’s impact on project quality and effectiveness	Integration of core capacities Implementation and integration of learnings; existence of any barriers to implementation Community partner integration Mentoring, peer networks value Awareness-raising communications
	Observations of Community of Practice and Virtual Meetings	Observational analysis Trend analysis of engagement over time	Mentor activities provided/variations Cohort member engagement in capacity-building activities Integration of core capacities	Cross-organizational interactions Community partner integration; existence of any barriers to integration Peer network engagement Thoughts, feelings, perceptions of cohort members’ use of new skills within their communities New knowledge gained from mentoring and peer network activities Implementation and integration of learnings
IMLS staff	Interviews	Thematic analysis	Outreach strategy for recruiting applicants for both APP cohorts Presentation language and modes of marketing APP program to applicants Application timeline	Review and selection process
Non-returning Round 1 applicants survey	Survey	Thematic analysis	Why round 1 applicants that were not accepted to the program did not apply a second time
IMLS Document review by PPG	Document Review	Descriptive analysis of cohort applications Comparative analysis between cohort application sets	Number of repeat applications from year 1; extent to which these applications were updated Outreach strategy for recruiting applicants for both APP cohorts	Presentation language and modes of marketing APP program to applicants Application timeline review and selection process Criteria for assessing quality of applications

44