Evaluation of 21st Century Community Learning Centers State Competitions interview protocols

Evaluation of the 21st Century Community Learning Centers State Competitions

Att_1875-NEW 4807 protocols--5-4-12

Evaluation of 21st Century Community Learning Centers State Competitions interview protocols

OMB: 1875-0267

Document [docx]

Download: docx | pdf

Evaluation of 21^st Century Community Learning Centers State Competitions

____________________________________________________

21^st CCLC State Program Staff

Interview Protocol

May 4, 2012

Prepared By: Prepared For:

Policy Studies Associates and U.S. Department of Education

The American Institutes for Research Contract No. ED-PEP-11-O-0089

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number XXXX-XXXX. Public reporting burden for this collection of information is estimated to average approximately 60 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Education, 400 Maryland Ave., SW, Washington, DC 20210-4537 or email [email protected] and reference the OMB Control Number XXXX-XXXX. Note: Please do not return the completed XXXX (cite form or other applicable reporting mechanism) application to this address.

State Administrators of the 21^st CCLC Program

Interview Protocol

Draft

Respondent’s name and title:

Interviewer:

State:

Date/Time:

Introduction to the Interview

Introduce the interviewer(s)
Explain the purpose of the study and topics to be covered in the interview. Interviewers will read the following statement to interview respondents at the beginning of each interview:

The purpose of this study is to examine how states administer the 21^st CCLC program grant competitions. Specifically, the study will identify lessons learned from SEA experiences in an effort to: 1) inform state administration of this and other Federal programs; and 2) provide Federal officials with information needed to craft guidance and technical assistance to states in this crucial area of state responsibility.

Explain the provisions for protecting respondent’s privacy. Interviewers will read the following statement to interview respondents at the beginning of each interview:

No individuals or schools will be identified by name in any reports or other communications about the study. Key study findings will be reported in the aggregate across all case study sites. Case study data will be maintained in secure files and will be accessible only to members of the research team. Information that could be used to identify individuals will not be released except as may be required by law.

Advise the respondent that the interview will last approximately 45-60 minutes.
Invite questions from the respondent
Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of the 21^st CCLC Program

Interview Questions

Note: Interviewers will ask all follow-up questions as specified in the protocol. In addition, interviewers will be directed to seek clarification of any response they judge to be vague or incomplete. The primary strategy for seeking clarification will be to ask respondents to provide specific examples to illustrate their responses. A second strategy will be to ask respondents to “say more” or to “expand on” or “explain the meaning of” a particular comment.

Evaluation of 21^st Century Community Learning Centers State Competitions Informed Consent

Purpose

The Policy and Program Studies Service (PPSS) in the Office of Planning, Evaluation and Policy Development, U.S. Department of Education (ED) requests clearance for the data collection for the Evaluation of 21^st CCLC State Competitions. The purpose of the study is to provide the U.S. Department of Education with information regarding the administrative conditions and capacities that may enhance implementation of the 21^st CCLC program, improve program outcomes, and offer guidance on program approaches and strategies that create the conditions for success. To this end, the evaluation will conduct case studies of nine states, and nine districts (or other sub-grantee organizations) within those states, to identify the patterns and complexities of the state competitions at the state and local levels. To assist with the evaluation, we are asking state and district staff to participate in personal interviews. You will be interviewed about issues related to: program staffing and administration; the planning and design of discretionary grant competitions; outreach efforts to eligible applicants; the application review and selection process; program monitoring and evaluation activities; technical assistance to grantees; state administrative capacities; and federal support. The interviews are designed to last no more than one hour.

Risks and Discomfort

There are few anticipated or known risks in participating in this study. Risks may include possibly feeling coerced to take part in the interviews, however you should know that you may discontinue your participation at any time or refuse to answer any questions you do not feel comfortable answering without penalty or loss of benefits to which you are otherwise entitled.

Benefits

Your participation in the evaluation will contribute to an understanding of how states administer the 21^st CCLC grant competitions and provide Federal officials with information needed to craft guidance and technical assistance to states in this crucial area of state responsibility.

Confidentiality

We will treat the information that you supply in a confidential manner. Only selected research staff will have access to data. We will NOT present results in any way that would permit them to be identified with you or any other specific individual. No personally identifiable information, such as name or state/district affiliation, will be disclosed to anyone outside the project. All data files will be destroyed once the study is completed.

More Information

If you would like more information about this study, you may contact the Project Director, Leslie Anderson, at Policy Studies Associates at (202) 939-5327 or at [email protected].

Informed Consent

I have read the above information. I have asked questions and received answers. I consent to participate in the study and to be tape recorded.

Signature: _____________________________________ Date: _______________________________

Print Name: ___________________________________ Position: _____________________________

My signature below indicates permission to audio-tape this interview.

Signature: _____________________________________ Date: ______________________________

Program Staffing and Administration

What is your professional background?
1. How long have you worked for the SEA?
2. How long have you worked with the 21^st CCLC Program?
3. What other responsibilities, if any, have you held within the SEA?

What are your current professional responsibilities related to the 21^st CCLC program?
1. To what areas of program administration would you say you devoted the most time in the past year? That is, what percentage of your time, if any, would you estimate that you devoted to each of the following activities in the past year and why:
  1. Agenda/priority setting;
  2. Designing/managing grant competitions;
  3. Providing outreach and applicant support;
  4. Monitoring and evaluating grantees;
  5. Providing capacity building, training, and technical assistance to grantees,
  6. Other

What are your administrative responsibilities for and time committed to other federal or state programs, if any?

Please describe SEA staffing for the 21^st CCLC program:
1. How many SEA staff (in FTEs as well as individuals) are assigned to the 21^st CCLC program?
2. What are the administrative responsibilities and time commitments of SEA staff assigned to the 21^st CCLC program?
3. Do SEA staff assigned to the 21^st CCLC program have administrative responsibilities and time commitments to other programs within the SEA?
4. Do other SEA staff not assigned to 21^st CCLC assist with 21^st CCLC competitions and/or other activities when there are significant demands on 21^st CCLC staff time?
5. Which SEA staff positions are supported—either fully or partially—by 21^st CCLC funding?

oes the state contract with any outside consultants for services related to the 21^st CCLC program?

No  Continue to the following question
Yes  Ask the follow-up probes

Follow up questions for respondents who say they contract with outside consultants:

How many outside contractors/consultants does your program work with?
What services do they provide?
What percentage of your administrative set-aside under the 21^st CCLC program is allocated to outside contractors/consultants?
How are administrative responsibilities for the 21^st CCLC program divided among SEA program staff and outside contractors/consultants?

Grant Competitions: Design

How does the state determine its competitive priorities for the 21^st CCLC grant applications?
1. What factors (e.g., state policies, 21^st CCLC funding allocations, state administrative capacity, community needs, grantee monitoring and evaluation data, federal monitoring visits, federal guidance or technical assistance etc.), if any, play a role in determining the competitive priorities for each competition?
2. Who is involved in the decision making process (e.g., other offices within the SEA, other state agencies, 21^st CCLC grantees, after-school networks, state policymakers etc.) for purposes of determining which priorities to mandate and which to make optional?
3. Do you ever consult with other federally supported SEA staff on the competitive priorities? From the federal 21^st CCLC program office?

[Review the state’s competitive priorities] Why did the state choose these priorities?

What priorities, if any, have been set aside and why? What priorities are considered essential and why?

What priorities are valued but considered unachievable among prospective grantees, and therefore not included among the competitive priorities? Why?

What do you believe to be essential features or strategies related to designing an effective grants competition?
1. Has your state faced significant challenges in soliciting the most promising applications possible? If so, what are they?
2. What federal or state policies, practices, or supports, if any, would address these challenges?

[For states that applied for and received the ESEA Flexibility waiver related to 21^st CCLC funds] In your next competition, do you intend to invite applicants to propose to use 21^st CCLC funds to extend or expand the regular school day, week, or year?

No  Continue to the following question
Yes  Ask the follow-up probes

Follow up questions for respondents who will be seeking applicants that intend to use 21^st CCLC funds to extend or expand the regular school day, week, or year:

If so, when will that competition take place?
How will you judge the applications?
Will there be any priority given to applicants that offer extended or expanded programming?
What consultation, if any, did you seek in designing the application process to include extended- or expanded-day applicants? If you did seek consultation, how did that help inform the revision to the application?
How will you ensure that the extended or expanded day applicants are supplementing, not supplanting the activities currently taking place in the schools that the applicant proposes to serve?

Grant Competitions: Outreach

Please describe the state’s current 21^st CCLC program application process.
1. Does the application process vary by type of award solicitation, such as continuation awards or awards for summer programs? If so, how?

What strategies, if any, has your program used to inform eligible entities about the grant competition and to provide support to grant applicants?
1. Does the state prioritize or target outreach activities toward certain types of applicants? If so, which applicants have been targeted and why? How has the program reached out to these applicants?
2. What guidance and/or support does your program offer to applicants? What type of guidance and/or support has been most and least effective? Why?
3. What role do external service providers or contractors, such as technical assistance, professional development providers or statewide evaluators play in the outreach process?

Are there particular types of applicants that are eligible to receive 21^st CCLC grants but do not apply (e.g., institutions of higher education)? Why? How do you know?

To what extent do you believe your program has reached the most promising programs? How do you know?

What do you believe to be essential features or strategies related to effective outreach and technical assistance to eligible applicants?
1. Has your state experienced any challenges in conducting outreach and/or providing technical assistance to eligible applicants? If so, what are the most significant challenges your state has experienced?
2. What federal or state policies, practices, or supports would address these challenges?

Grant Competitions: Application Review and Selection

In your last competition, how many applications were there in total? How many peer reviewers were involved?

How many reviewers reviewed each application?
How were applications assigned to individual reviewers? Did assignments address potential reviewer conflicts of interest? If so, how?

When and where did the reviews take place?

How are 21^st CCLC applications reviewed and assessed in your state?
1. Who is involved in the application review process, and how were they identified as qualified?
2. What processes and/or tools do reviewers use to evaluate applications and how were they developed?
3. Are reviewers trained prior to conducting application reviews? If so, how? How are variations in reviewer perspectives addressed?

How were the application rating criteria developed and why? To what extent were program data, state priorities, and/or national trends considered?

How does the state/peer review panel determine the likelihood that applicants will comply with such grant requirements as (1) providing a safe and easily accessible facility; (2) ensuring that students travel safely to and from centers; and (3) disseminating information about 21^st CCLC services to the community, etc.?

What do you believe to be essential features or strategies related to the application review process?
1. Are there any significant challenges associated with the application review process? If so, what are they?
2. What federal or state policies, practices, or supports would address these challenges?

To what extent has the state been successful in ensuring that the selected 21^st CCLC grantees serve students who “primarily attend schools eligible for schoolwide programs or schools that serve a high percentage of students from low-income families”?
1. What are the practical considerations in attempting to ensure that grantees serve a targeted population?

What do you believe to be the essential features or strategies related to the grants distribution process?
1. Are there any significant challenges associated with grants distribution? If so, what are they?
2. What federal or state policies, practices, or supports would address these challenges?

Program Monitoring and Evaluation

How does the state monitor 21^st CCLC grantees?
1. What specific strategies does the state use to monitor 21^st CCLC grantees and why?
2. Does SEA staff conduct in-person visits with 21^st CCLC grantees? Follow up questions if applicable. If so, please describe who is visited and how often visits take place. Please also describe what documents are reviewed and/or what program observations are made during the visit.
3. Does SEA staff conduct desk (phone) monitoring visits? If so, how often? What information is reviewed (e.g., fiscal, performance data, evaluation) during the desk monitoring?
4. What grant monitoring strategies have been most and least effective and why?

Has the state comprehensively evaluated (directly or through a grant or contract with an external evaluator) the effectiveness of its 21^st CCLC program? If so, how many evaluations have been conducted since 2002?
1. To what extent, if at all, have evaluation findings informed state efforts to guide and assist local programs?
2. What have been the challenges, if any, associated with conducting a comprehensive evaluation of the 21^st CCLC program?

What do you believe to be essential features or strategies related to program monitoring and evaluation?
1. Are there any significant challenges associated with the program monitoring and evaluation process? If so, what are they?
2. What federal or state policies, practices, or supports would address these challenges?

Technical Assistance to Grantees

Does the state provide targeted technical assistance to grantees? If so, how does the state identify which subgrantees need assistance and what assistance will be provided? Does the type of technical assistance provided vary for administrative issues (e.g., fiscal management, evaluation) versus programming/performance issues (e.g., low attendance, poor performance)? Please describe how targeted technical assistance is provided in your state.

Please describe any ways in which monitoring or evaluation findings inform state efforts to guide and assist local program implementation.

What do you believe to be essential features or strategies related to providing high quality technical assistance and support to 21^st CCLC grantees?
1. Are there any significant challenges associated with providing technical assistance and support to grantees? If so, what are they?
2. What federal or state policies, practices, or supports would address these challenges?

State Administrative Capacities

In your opinion, is the number of staff and the amount of the administrative set-aside (i.e., two percent for administration and three percent for monitoring, technical assistance, and evaluation) for your program sufficient to administer the grant competition effectively?

Are all of the SEA staff positions that administer the 21^st CCLC program funded by the 21^st CCLC program? If not, are those positions funded by other federally supported programs? What other funding sources support staff who administer the 21^st CCLC program?

In your opinion, does the 21^st CCLC program office have sufficient staff expertise to administer the 21^st CCLC grant competition—and program—effectively? If not, what staffing areas would benefit from additional preparation?

Does the state 21^st CCLC program coordinate its administrative responsibilities/activities with other SEA programs? If so, why and how? If not, why not?

Follow up questions when applicable:

Does your program draw upon the expertise of staff associated with other state-administered programs? Why?
What other factors (e.g., staff time and availability) motivate the 21^st CCLC program to coordinate with other SEA-administered programs?
Are there SEA policies or structures that encourage or necessitate program coordination? If so, what are they and why?

What are the factors that support or impede program implementation in your state? What factors have helped or hindered your efforts to achieve your program goals?

Federal Support

What resources and technical assistance have you received from the federal 21^st CCLC program office at ED and what has been most and least helpful? Why? What additional resources or assistance, if any, would be helpful to your program, particularly in regard to running the grants competition? Why?

How has your state responded to monitoring feedback from the federal program office? Which, if any, compliance requirements are most challenging and why?

What federal monitoring, technical assistance, and reporting requirements do you believe are most and least effective for administering a federal discretionary grants program and what, if anything, do you suggest the federal program office do differently? Why?

Conclusion

What are the greatest challenges related to the state administration of a 21^st CCLC program and grants competition?

What would you recommend that peers in other states might consider if they are looking for ways to improve administration of their 21^st CCLC program and, more specifically, their grant competitions? Why?

What recommendations do you have for federal officials regarding the design of the 21^st CCLC program as expressed through authorizing laws, regulations, and monitoring activities?

Evaluation of 21^st Century Community Learning Centers State Competitions

____________________________________________________

State Administrators of Other

Discretionary Grants Programs

Interview Protocol

May 3, 2012

Prepared By: Prepared For:

Policy Studies Associates and U.S. Department of Education

The American Institutes for Research Contract No. ED-PEP-11-O-0089

State Administrators of Other Discretionary Grants Programs

Interview Protocol

Draft

Respondent’s name and title:

Interviewer:

State:

Date/Time:

Introduction to the Interview

Introduce the interviewer(s)
Explain the purpose of the study and topics to be covered in the interview. Interviewers will read the following statement to interview respondents at the beginning of each interview:

Explain the provisions for protecting respondent’s privacy. Interviewers will read the following statement to interview respondents at the beginning of each interview:

No individuals or schools will be identified by name in any reports or other communications about the study. Key study findings will be reported in the aggregate across case study sites. Case study data will be maintained in secure files and will be accessible only to members of the research team. Information that could be used to identify individuals will not be released except as may be required by law.

Advise the respondent that the interview will last approximately 45-60 minutes.
Invite questions from the respondent
Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of the 21^st CCLC Program

Interview Questions

Evaluation of 21^st Century Community Learning Centers State Competitions Informed Consent

Purpose

Risks and Discomfort

Benefits

Confidentiality

More Information

If you would like more information about this study, you may contact the Project Director, Leslie Anderson, at Policy Studies Associates at (202) 939-5327 or at [email protected].

Informed Consent

I have read the above information. I have asked questions and received answers. I consent to participate in the study and to be tape recorded.

Signature: _____________________________________ Date: _______________________________

Print Name: ___________________________________ Position: _____________________________

My signature below indicates permission to audio-tape this interview.

Signature: _____________________________________ Date: ______________________________

Program Staffing and Administration

What position do you hold within the SEA and what are your primary responsibilities with respect to your program’s grant competition?
1. What other responsibilities, if any, do you have within the SEA?

Please describe SEA staffing for your program.

How many SEA staff are assigned to your program and what are their administrative responsibilities and time commitments to your program?
Do other SEA staff not assigned to your program assist with competitions related to your program and/or at other times when there are significant demands on program staff time?

Does your state contract with any outside consultants for services related to the administration and operation of your program?

No  Continue to the following question
Yes  Ask the follow-up probes

Follow up questions for respondents who say they contract with outside consultants:

How many outside contractors/consultants does your program work with?
What services do they provide?
What percentage of your program’s administrative set-aside is allocated to outside contractors/consultants?
How are administrative responsibilities for your program divided among SEA program staff and outside contractors/consultants?

Grant Competitions: Design

How does the state determine priorities and application requirements for your program’s grant competition?

What factors (e.g., state policies, funding allocations, state administrative capacity, community needs, grantee monitoring and evaluation data, federal monitoring visits, needs assessments, etc.), if any, play a role in determining the design of each competition? [If applicable] The frequency of grant competitions?
Who is involved in the decision making process in determining competition priorities (e.g., other offices within the SEA, other state agencies, grantees, etc.)?

What do you believe to be essential features or strategies related to designing an effective grants competition?

Does your state experience significant challenges in soliciting the most promising applications possible? If so, what are they?
What federal or state policies, practices, or supports would address these challenges?

Grant Competitions: Outreach

How does your program inform eligible entities about your program’s grant competition?

Are there specific goals in terms of outreach and support to potential applicants? For example, does the state prioritize or target outreach activities toward certain types of applicants? If so, please describe your state’s approach for reaching these goals.

What guidance and/or support, if any, does your program offer to applicants? What type of guidance and/or support to applicants has been most and least effective? Why?

Are there particular types of applicants that are eligible to receive grants but do not apply? If so, what factors do you believe influence a program’s decision not to apply for the grant?

To what extent do you believe your program has reached the most promising programs? How do you know?

What do you believe to be essential features or strategies related to effective outreach and technical assistance to eligible applicants?

Does your state experience significant challenges associated with outreach and/or providing technical assistance to eligible applicants? If so, what are they?
What federal or state policies, practices, or supports would address these challenges?

Grant Competitions: Application Review and Selection

How are grant applications reviewed and assessed in your state?

Who is involved in the application review process, and how were they identified as qualified?
Are reviewers trained prior to conducting application reviews? If so, please describe the training reviewers receive.

What do you believe to be essential features or strategies related to the application review process?

Are there any significant challenges associated with the application review process? If so, what are they?
What federal or state policies, practices, or supports would address these challenges?

Program Monitoring and Evaluation

To the extent that your program monitors its grantees, what grant monitoring strategies do you use? Do you conduct in-person visits with your grantees? Follow up questions if applicable. If so, please describe who is visited and how often visits take place. Please also describe what documents are reviewed and/or what program observations are made during the visit.

Do you or other SEA staff conduct desk (phone) monitoring visits? If so, how often? What information is reviewed (e.g., fiscal, performance data, evaluation) during the desk monitoring?

In your opinion, what grant monitoring strategies have been most and least effective? Why?

Has the state comprehensively evaluated (directly or through a grant or contract with an external evaluator) the effectiveness of this program?

What do you believe to be essential features or strategies related to program monitoring and evaluation?

Are there any significant challenges associated with the program monitoring and evaluation process? If so, what are they?
What federal or state policies, practices, or supports would address these challenges?

Technical Assistance to Grantees

What written guidance or technical assistance is offered to grantees to help them design and implement their programs? Why were these approaches chosen?

Does the state provide targeted assistance to grantees? If so, how does the state identify which subgrantees need assistance and how the state can best help?

What do you believe to be essential features or strategies related to providing high quality technical assistance and support to grantees?

Are there any significant challenges associated with providing technical assistance and support to grantees? If so, what are they?
What federal or state policies, practices, or supports would address these challenges?

State Administrative Capacities

In your opinion, is the number of staff for your program sufficient to administer the grant competition—and program—effectively? Why or why not?

In your opinion, does your program office have sufficient staff expertise to administer your grant competition effectively? If not, what staffing areas would benefit from additional preparation?

Does the state program coordinate its administrative responsibilities/activities with other SEA programs? If so, why? If not, why not?

Follow up questions when applicable:

Does your program draw upon the expertise of staff associated with other state-administered programs? What particular expertise does your program draw upon?
What other factors—such as staff time and availability—motivate your program to coordinate with other SEA-administered programs?
Are there policies, priorities or structures within the SEA that encourage or necessitate program coordination? If so, why? What are the policies and/or structures?

What are the factors that support or impede program implementation in your state? What factors have helped or hindered your efforts to achieve your program goals?

What would you recommend that other discretionary grants programs might consider if they are looking for ways to improve administration of their competitions? Why?

Federal Support

What resources and technical assistance have you received from the federal program office at the U.S. Department of Education and what has been most and least helpful? Why? What additional resources or assistance, if any, would be helpful to your program? Why?

Has your program been monitored by the federal program office at the U.S. Department of Education? Describe how your state responded to monitoring feedback from the federal program office. To what extent are compliance requirements challenging to implement and why?

What aspects of federal monitoring, technical assistance, and reporting requirements, if any, do you believe are most and least effective for administering a federal discretionary grants program? What, if anything, do you suggest the federal program office do differently with respect to monitoring, technical assistance, and reporting requirements, and why?

What recommendations, if any, do you have for federal officials regarding the design of discretionary grant competitions as expressed through authorizing laws, regulations, and monitoring activities?

Evaluation of 21^st Century Community Learning Centers State Competitions

____________________________________________________

21^st CCLC State Peer Reviewer

Interview Protocol

May 3, 2012

Prepared By: Prepared For:

Policy Studies Associates and U.S. Department of Education

The American Institutes for Research Contract No. ED-PEP-11-O-0089

21^st CCLC State Peer Reviewer Interview Protocol

Draft

Respondent’s name and title:

Interviewer:

State:

Date/Time:

Introduction to the Interview

Introduce the interviewer(s)
Explain the purpose of the study and topics to be covered in the interview. Interviewers will read the following statement to interview respondents at the beginning of each interview:

This study seeks to examine how states administer the 21^st CCLC program grant competitions. Specifically, the study will identify lessons learned from SEA experiences in an effort to: 1) inform state administration of this and other Federal programs; and 2) provide Federal officials with information needed to craft guidance and technical assistance to states in this crucial area of state responsibility.

Explain the provisions for protecting respondent’s privacy. Interviewers will read the following statement to interview respondents at the beginning of each interview:

Advise the respondent that the interview will last approximately 45-60 minutes.
Invite questions from the respondent
Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of the 21^st CCLC Program

Interview Questions

Evaluation of 21^st Century Community Learning Centers State Competitions Informed Consent

Purpose

Risks and Discomfort

Benefits

Confidentiality

More Information

If you would like more information about this study, you may contact the Project Director, Leslie Anderson, at Policy Studies Associates at (202) 939-5327 or at [email protected].

Informed Consent

I have read the above information. I have asked questions and received answers. I consent to participate in the study and to be tape recorded.

Signature: _____________________________________ Date: _______________________________

Print Name: ___________________________________ Position: _____________________________

My signature below indicates permission to audio-tape this interview.

Signature: _____________________________________ Date: ______________________________

Background and Overview

What are your current professional responsibilities?
1. What responsibilities, if any, do you have related to a 21^st CCLC program at either the state or local level?
2. What responsibilities, if any, do you have related to other state-administered programs?

What is your professional background?
1. On what topics do you have professional experience or expertise?
2. Have you worked for any organizations that deliver before-school, after-school, or summer learning programming, including any 21^st CCLC programs?
3. Have you ever applied for or been a part of a team that applied for a 21^st CCLC grant? If so, what was the result?
4. Have you worked for a State Education Agency or for any organizations hired by the state to administer a state or federal discretionary grants programs, including 21^st CCLC? If so, please identify the programs and your responsibilities/titles.

We are interested in knowing a little about all the grant competitions for which you reviewed applications or proposals. For each, please briefly describe the grant’s purpose, the granting agency, and timeframe.

Before going deeper into your peer review experiences related to the 21^st CCLC program, we would like to know your views on what characterizes effective and ineffective peer review processes. What are essential features or strategies for an effective peer review process for federal or state discretionary grants programs? What are features of an ineffective peer review process? Why?

Who managed and supported the peer review process for the 21^st CCLC grant competition(s) in which you reviewed applications or proposals?

Who was involved from the SEA, and in which divisions or offices did they usually work?
What were the SEA staff members’ roles and responsibilities related to the peer review process(es)?
Were there others, such as outside consultants, who were not formally employed by the SEA but who had a role in managing or supporting the peer review process(es)?
Do you know if the 21^st CCLC peer review process(es) aligned with or was informed by peer reviews for any other state-administered grants competitions?

Recruitment, Selection, and Training

How did you learn about the opportunity to serve as a peer reviewer?
1. What outreach and recruiting strategies do you believe are most effective for getting qualified reviewers?
2. Were you compensated for being a peer reviewer for the 21^st CCLC program, and if so, what was the compensation?

Why did you decide to become a peer reviewer?

Do you know how the state selected peer reviewers for the 21^st CCLC grant competition(s)?

Follow up questions for respondents who say they know how the state selected peer reviewers:

Please describe the peer reviewer application requirements and process.
What qualifications or professional roles did the SEA want peer reviewers to possess? How do you know the SEA wanted them?
Who were the other peer reviewers for the 21^st CCLC grant competitions? [If applicable] Do you believe the SEA chose reviewers who were well qualified to conduct effective peer reviews for the 21^st CCLC competition? Why do you think so? If not, why not?

What do you believe are the appropriate reviewer qualifications or professional roles for an effective 21^st CCLC peer review process? Why?

What, if any, training did you receive to serve as a peer reviewer for the 21^st CCLC competition(s)?
1. Please describe any training on the rating criteria and priorities for the grant competition. What about the rating tools, such as rubrics?
2. Did the training seek to ensure that each peer reviewer would assign ratings consistently and in the intended manner? How was this accomplished?
3. What about the training do you believe was effective for preparing you to conduct peer reviews? What was lacking in the training, and how could it be improved?
4. What do you believe are essential features or strategies related to effective training of peer reviewers?

Review and Rating Process

Please describe the application review and rating process for the 21^st CCLC competitions.
1. How many applications were there in total? How many peer reviewers were involved? How many applications did you review?
2. How many reviewers reviewed each application?
3. How were applications assigned to individual reviewers? Did assignments address potential reviewer conflicts of interest?
4. When and where did the reviews take place?
5. [If multiple peer reviewers reviewed each application:] How, if at all, did you interact with other reviewers that were assigned to the same applications as you were? What did you communicate about and through what means?
6. Did you or other peer reviewers participate in applicant interviews? If so, please describe.

What type of information, such as scores and comments, were you required to provide for each application you reviewed?
1. Did you use a rating tool, such as a rubric, that enabled you to assign standardized scores? If so, please describe the structure of the tool and the process by which you were to use it.
2. Please describe the rating criteria, including their organization and specificity.
3. What were the strengths and weaknesses of the rating criteria?
4. Were you required to provide detailed narrative comments that included strengths and weaknesses of the application? If so, please describe.

[If multiple peer reviewers reviewed each application:] How, if at all, were each peer reviewer’s ratings factored into an overall rating of an application? That is, were peer reviewers expected to come to consensus on the ratings they gave to an application? If so, please explain.
1. Was there a process to resolve significant discrepancies among peer reviewers? If so, please describe.

Did the opinions of 21^st CCLC program staff figure into the rating of applications? For example, could the opinions of 21^st CCLC program staff override or trump peer reviewer ratings? Are you aware of circumstances in which this happened? What was the fallout, if any?

What do you believe are essential features or strategies of an effective rating system to be used by peer reviewers?

Conclusion

Did the SEA optimally manage and support an effective peer review process? Please explain.
1. To what extent, if at all, could the peer review process have been more effective with respect to the selection of reviewers, their training, and the application rating process? How? What would improve SEA capacity to manage the peer review process?

Reflecting on our conversation and all your experiences with peer reviews, what are the most prominent challenges to having an effective peer review process for grant competitions?

Are you aware of and/or have you had experience with effective peer review processes used in other grants competitions that might be applied to the 21^st CCLC competition(s)? If so, what?

Evaluation of 21^st Century Community Learning Centers State Competitions

____________________________________________________

21^st CCLC Sub-Grantee Interview Protocol

Interview Protocol

May 3, 2012

Prepared By: Prepared For:

Policy Studies Associates and U.S. Department of Education

The American Institutes for Research Contract No. ED-PEP-11-O-0089

21^st CCLC Sub-Grantee Interview Protocol

Draft

Respondent’s name and title:

Interviewer:

State:

Date/Time:

Introduction to the Interview

Introduce the interviewer(s)
Explain the purpose of the study and topics to be covered in the interview. Interviewers will read the following statement to interview respondents at the beginning of each interview:

Explain the provisions for protecting respondent’s privacy. Interviewers will read the following statement to interview respondents at the beginning of each interview:

Advise the respondent that the interview will last approximately 45-60 minutes.
Invite questions from the respondent
Confirm respondent’s current roles and responsibilities within the agency/organization and in terms of the 21^st CCLC Program

Interview Questions

Evaluation of 21^st Century Community Learning Centers State Competitions Informed Consent

Purpose

Risks and Discomfort

Benefits

Confidentiality

More Information

If you would like more information about this study, you may contact the Project Director, Leslie Anderson, at Policy Studies Associates at (202) 939-5327 or at [email protected].

Informed Consent

I have read the above information. I have asked questions and received answers. I consent to participate in the study and to be tape recorded.

Signature: _____________________________________ Date: _______________________________

Print Name: ___________________________________ Position: _____________________________

My signature below indicates permission to audio-tape this interview.

Signature: _____________________________________ Date: ______________________________

21^st CCLC Program Staffing and Administration

What is your professional background?

How long have you worked for the LEA (or other organization with a 21^st CCLC grant) and with the 21^st CCLC program?
What other responsibilities, if any, have you held within the LEA/other organization?
Prior to your involvement with the 21^st CCLC program, to what extent had you worked with other Out-of-School Time (OST)/before-school, after-school, or summer learning programs?

Please describe your current administrative responsibilities related to the 21^st CCLC program. To what areas of program administration would you say you devote the most time? Why? What are your administrative responsibilities for and time committed to other programs in the district (or other organization)?

Please describe the current staffing for the 21^st CCLC program:

How many staff within the LEA and partner organizations have an administrative or management role in the 21^st CCLC program?
What are the responsibilities and time commitments of these individuals assigned to manage the 21^st CCLC program?
Do these individuals have administrative or management responsibilities and time commitments to other programs?
Which positions are supported—either fully or partially—by 21^st CCLC funding?

Does the 21^st CCLC program coordinate its administrative responsibilities/activities with other programs administered in the LEA/organization, such as Title I programs, Supplemental Educational Services or literacy improvement programs?

Follow up questions for respondents who say they coordinate with other programs administered in the LEA:

Why does your program coordinate with other LEA-administered programs? For example, does the 21^st CCLC program draw upon the expertise of staff associated with other state or federal programs? What particular expertise does the 21^st CCLC program seek?
What other factors—such as staff time and availability—motivate your program to coordinate with other LEA-administered programs?
Are there policies or structures within the LEA that encourage or necessitate program coordination? If so, what are they?
Around what specific program-related issues/activities does the 21^st CCLC program coordinate with other federal or state programs?

Grant Competition Structure and Process

Please describe your knowledge of and experience with the 21^st CCLC or other after-school programs prior to applying for the grant. In particular:

How did you learn about the 21^st CCLC program?

Were you familiar with other grantees or applicants?

Why did you choose to apply?

Please describe your experience with the application process.

How much time did you have to complete the application?
What were the grant requirements?
Was there an interview or follow-up process before the final selection?
Did state administrators share reviewer ratings and comments with you?
Were you given feedback on strengths and weaknesses of your application or other guidance upon receiving an award?
Once selected, was there an orientation for or convening of new grantees?

Did the state education agency provide adequate support throughout the application process? For example:

Were the application requirements clear?
Did the RFP clearly convey how your application would be rated?
Was technical assistance during the application process offered (e.g., a bidders’ conference, webinar)? If so, did you participate? Why or why not?
Were SEA staff available to answer questions?
To what extent was the assistance helpful? Why or why not?

To what extent did you, your LEA, and program partners have the staff experience, time, and resources to meet the competition requirements (e.g., development of an evaluation plan, development of a sustainability plan, assessment of need for the program in the community)? If not, why not?

To what extent did the competition priorities influence: (a) the design of your program, (b) the selection of your program partners, and (c) the selection of program staff? Why?
1. In your opinion, were the competition priorities aligned with the needs of your program community?
2. Are there any priorities that you would consider unachievable? Why?

What written guidance, support, or technical assistance was most helpful? What was least helpful? Why?

Were there any significant challenges associated with the grant competition process? If so, what were they?
What federal or state policies, practices, or supports would address these challenges?

Program Monitoring and Evaluation

How does the state monitor your 21^st CCLC program and ensure implementation of effective strategies for serving the academic needs of participating students?

What specific strategies does the state use to monitor the 21^st CCLC program (e.g., a state designed process and/or tool for self-monitoring)? How often are these strategies employed?

[If a self monitoring tool is used] Have staff been trained on how to use the self-monitoring tool? How are results used by your program and by the state?

Does your program receive feedback from the state regarding monitoring visits and reports?

To what extent does your program use the state monitoring reports to inform program design or program improvement? [If applicable]What monitoring data have been most useful for purposes of informing program improvement? Why?
Does the state look for benchmarks of progress? If so, what are the benchmarks?
Does the state provide assistance, issue warnings, and/or impose penalties if certain benchmarks are not met? If so, what are they?

Does your program face significant challenges associated with the program monitoring process? If so, what are they? Do you have any recommendations on how to improve the process, regulations, or guidance?

How does your program meet its state data collection requirements?

What data are collected (e.g., program enrollment and attendance, teacher surveys, youth surveys, etc.)?
What data tracking systems, if any, are used by your program (e.g., YouthServices, CitySpan, PPICS, etc.)?
How do you report your data to the state?
What use do you make, if any, of the data you report?
Does the state provide feedback to you regarding the data you have submitted?
Is additional data collected and/or analyzed for your program evaluation? If so, what data are collected?
To what extent are evaluation data used to inform program design or program improvement? Has your evaluation data been used to measure program performance and/or impact? [If applicable]What data have been most useful for purposes of informing program improvement? Impact? Why?
Are outside consultants used to help with data collection, evaluation, analysis, and/or reporting?
Does your program receive feedback from the state regarding evaluation reports? If so, to what extent does your program use evaluation feedback to inform program design or program improvement?

Does your program face significant challenges associated with the program evaluation process? If so, what are they? How does your program meet its state evaluation requirements? Do you have any recommendations on how to improve the process, regulations, or guidance?

Technical Assistance and Support

To what extent have you and other 21^st CCLC program staff received technical assistance from the state 21^st CCLC program office—or from a vendor that has a contract with the state 21^st CCLC office—in the past 12 months in each of the following areas:

Program design
Program improvement
Program evaluation
Sustainability
Other

How would you rate the quality of the technical assistance you and other 21^st CCLC program staff received from the 21^st CCLC program office, or from a vendor has a contract with the state 21^st CCLC office?

To what extent were the duration, frequency, and content of the assistance adequate?
To what extent and in what ways was the assistance helpful?

What has been most and least helpful about the technical assistance your office received from the state 21^st CCLC program office in the last 12 months? What additional resources or assistance, if any, would be helpful to your program? Why?

In your opinion, does your LEA/provider organization have the staff expertise and training needed to effectively administer your 21^st CCLC grant? If not, what staffing areas would benefit from additional preparation, and what preparation would be most useful?

What else, if anything, is needed to enable you and other 21^st CCLC program staff to manage the program in the best possible way?

State and Local Context and Afterschool Infrastructure

What are the factors that support or impede implementation of your program? What among the following, if anything, has helped or hindered your program’s efforts to achieve its goals?

History of out-of-school time (OST) programs in the state, including before-,after-school and summer programs and extended learning programs , including the presence of OST/after-school networks and community-based organizations
Access to resources and technical assistance from state or local OST/after-school networks or intermediaries
State or local funding and policy related to OST/after-school programming Connections/conflicts with other state or federal programs
Connections/conflicts with other private or nonprofit after-school programs
Availability of resources and funding from local businesses, philanthropic organizations, and community partners

Conclusion

Reflecting on our conversation, what are the biggest challenges related to the state grant competition and implementation of the 21^st CCLC program?

Again reflecting on our conversation, what would you recommend that state administrators might consider if they are looking for ways to improve administration of the 21^st CCLC program and, more specifically, the grants competition? Based on your experience, what do you think are the most important competition priorities? What elements should be mandatory? What aspects of the competition need more flexibility? Why?

What recommendations do you have for state officials regarding the design of the 21^st CCLC program as expressed through authorizing laws, regulations, and monitoring and evaluation activities?

B-45

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Kristy Krivickas
File Modified	0000-00-00
File Created	2021-01-31