Chapter 2: Protocols to Follow during the Impact Visit
Impact Visit and Data Collection Methods
Overview
This chapter focuses on ensuring that ANA staff and technical assistance providers utilize a systematic approach to collect data during impact visits.
The Impact Team’s Role
During each visit, the impact team’s role is to facilitate a positive experience between ANA as an institution and the grantee while collecting and verifying pertinent project data. Team members should be respectful in all situations and pay particular attention to the cultural norms of each grantee before, during, and after each visit.
Interview Methods during the Visit
The impact visit interviews should be conducted in a manner that allows the grantee, project staff and community members to report their accomplishments. For the most part, the data collected on the Project Impact Assessment Survey (hereafter referred to as ‘the tool’) should be recorded as reported by the project staff and community members. The impact evaluation team is encouraged to facilitate, when necessary, the visit through useful examples and clarifying statements. The team should not alter the data being reported by grantees unless it is found to be materially flawed during follow-up questions and documentation review.
Overall, the interview process during impact visits should be led by open-ended questions wherever possible. Avoid giving a list of possible answers for the grantee to choose from. As the tool is a survey instrument, record the data as stated by the grantee. Data analysis occurs at ANA after all data is collected from impact visits.
Documentation Review
(This data is included in Project Snapshot section of Congressional Report)
Whenever possible, verify all reported accomplishments, including:
key benchmarks
deliverables
created jobs
businesses created
Native American consultant hired
elders involved
youth involved
revenue generated
resources leveraged
people trained
partnerships formed
codes and ordinances developed by the project
project sustainability documentation
non-federal share match
teaching certificates, language surveys
people who increased their ability to speak a native language
IMPORTANT: The tool is housed in a macros-based PDF file format, and the document is therefore locked from further editing.
At the end of every impact season, once all impact reports have been completed, information from the tools will be downloaded into a database system from which DPPE will run data reports. For this reason, it is crucial that evaluators completely fill out the tool, without leaving checkboxes or data fields blank unless they are not applicable to the project. Furthermore, each impact evaluator must confirm final data results with each grantee’s OER, which will be uploaded in desktop 90 days after the project’s end date. Only when the tool has been updated with information from the OER can the tool be considered completed.
*Special note: qualitative data, or text narrative, does not need to be updated (e.g. OWP section, project beneficiaries section, etc.) but all quantitative data must be updated.
Evaluators are encouraged to use this section to record any background information they find relevant to the project.
Baseline Data:
List any specific data that describes the grantee’s situation prior to receiving ANA funding. This information is generally found in the grantee’s original application. Items can include, but are not limited to: the number of people who speak a native language, the unemployment rate and the number of codes and regulations implemented.
Background Information:
The evaluator can place any pertinent information they would like to have available during the visit in this space. ANA staff may want to include information gathered from the program specialist.
Evaluators may use this table to record any outstanding documents or information needing to be sent by the grantee following the site visit. This is meant to serve as a checklist to provide evaluators with easy access to information requiring further follow-up.
Section 1 - Objective Work Plan
Question 1.1
The paragraph of instructions below is included in this section of the tool (T/TA providers will have a preloaded tool sent to them by the ANA Evaluator prior to the visit):
Using the Objective Work Plan (OWP), list the grantee’s objectives, the results and benefits, the criteria for evaluating the results and benefits and the deliverables for each objective below. This should be done in advance of the impact visit. During the visit, confirm the status of these objectives with the grantee and note any comments below. Include the status of incomplete objectives with their projected completion date.
What to record:
ANA collects data on the progress made on each objective, as outlined by the activities in the grantee’s OWP and the results and benefits associated with them. For the purpose of this section, it is useful to have a copy of the grantee’s current approved OWP available during the visit, in order to view individual activities if necessary.
NOTE: It is not necessary to discuss and verify the completion of each individual project activity.
How to record:
Discuss all objectives with the grantee in order to determine the degree to which they were completed. Record this information in the comments section of the corresponding objective. Incorporate the two results and benefits sections in order to determine the extent to which the results, benefits and deliverables expected from project implementation have been realized. Using this information, record your comments on the extent to which the objectives were achieved.
In the first column of the deliverables table under each objective, please list the corresponding products as outlined in the “results and benefits expected” section of the grantee’s OWP. Record if relevant documentation was reviewed in the center column. Add any applicable comments in the third column. If there are products developed in addition to those outlined in the OWP, please include them in the comments section directly below the deliverables table in the corresponding objective (if relevant).
Required Documentation:
ANA incorporates the completion of outlined and expected deliverables into the determination of project completion. Therefore, deliverables and products included in the OWP should be available for review at the time of the impact visit.
Impact Indicators
ANA definition:
Measurement descriptions used to verify the impact or the achievement of the project goal. Indicators must be quantifiable and documented. Impact indicators include target numbers and tracking systems. ANA requires three impact indicators per project. In addition to the two standard indicators: Leveraged Resources and Partnerships Developed (please see corresponding section of manual for definitions), ANA requires the project to have at least one project specific impact indicator. Impact indicators are separate from the results and benefits section of the OWP.
Question 2.1 – Impact Indicators
What to record:
ANA collects data on the progress made towards achieving impact indicator target numbers as stated in the project proposal.
How to record:
(T/TA providers will receive a frontloaded version of the tool prior to the impact visit that will include all impact indicators and their target numbers).
Partnerships
ANA definition:
Agreements between two or more parties that will support the development and implementation of the proposed project. Partnerships include other community-based organizations or associations, tribes, federal and state agencies, and private or non-profit organizations.
What to record:
ANA collects data on two types of partnerships: those newly formed during the course of a project and strengthened existing partnerships. ANA does not collect data on any partnership not directly related to, and beneficial for, the funded project. Keep in mind that partnerships should be beneficial for both parties involved, rather than cases where benefits flow only from one party to another. ANA accepts formal and informal partnerships.
How to record partnerships:
Question 2.1.1:
In the first column on the table, record the names of the reported partnerships. Then check one of the boxes (“pre-existing partnership” or “new partnership”) to clarify if the partnership was strengthened or newly formed and tally the totals at the bottom of the table.
Question 2.1.2:
Ask the grantee to look at the list of partnerships and pick the three that were most important, in order of importance from one to three, to their project’s implementation. Record these three partnerships in the first column of the table. In the second column record the location/service area of the partner (local, regional or national). In the third column record the type of partnership (Government, Inter-tribal, Intra-tribal, Consortium, For-profit Company, Non-profit, Individual Investor /Foundation, or a School/University) the type of partnership (Non-profit, Tribal Department, State Agency etc.). Have the grantee briefly describe why or how each of the three partnerships was crucial to the project.
Required Documentation:
ANA does not require grantees to provide letters of support, memoranda of understanding or letters of commitment to support reported partnerships. Instead, the impact team is responsible for determining which of the reported partnerships were new or existing based on conversations with project staff during the impact visit.
Leveraged Resources
ANA definition:
The non-ANA resources, as expressed as a dollar figure, acquired during the project period that support the project and exceed the 20 percent match required for ANA grants. Such resources may include any natural, financial and physical resources available within the tribe, organization, or community to assist in the successful completion of the project. An example would be an organization that agrees to provide a supportive action, product, service, human or financial contribution that will add to the potential success of the project.
What to record:
ANA collects data on the resources attracted to the project during implementation above the 20% non-federal share match required by the grantee. If in doubt as to whether to include a leveraged resource, ask the following question:
Is the implementation of this ANA project the sole reason why this resource was leveraged?
If the answer is “yes”, include the leveraged resource.
Some examples:
Resource |
What to Include |
Grantee is using space (donated or reduced rental rate) in a newly constructed building. The construction was not funded by ANA and occurred prior to the project. |
Include cost of renting specific space, not cost of entire construction of building. |
Local day care provides services free of charge for project staff’s children. |
Include what it would cost per child to receive the same services. |
How to record leveraged resources:
In the table provided in question 2.1.3 of the tool, record the source of the leveraged resource and whether the resource was from a federal or non-federal source. In the second column, note the documentation reviewed during the visit. At the bottom of the table, list the total leveraged resources from federal and non-federal sources. In the final column, provide a brief description of how the resource leveraged contributed to meeting the project goal and objectives. The leveraged resource must be directly related to the ANA-funded project.
Required Documentation:
ANA does not require grantees to provide supporting documentation for their reported leveraged resources. However, ANA does encourage the impact team to verify reported leveraged resources at every opportunity.
Impact Indicator Targets
Prior to the impact trip, list all indicators in the first column of the table (there may be up to three additional indicators). Determine their original target numbers, either from the grantee’s application or revision document, and enter them into the second column.
During the impact visit, determine what progress was made towards achieving each indicator’s target number, and record it in the third column labeled “Actual Achieved.”
Required Documentation:
ANA does not require specific documentation to verify the achievement of impact indicator target numbers. However, ANA encourages the impact team to verify these numbers to the greatest extent possible.
Questions 2.2 – Statement of Need
What to record:
ANA is interested in hearing the grantee’s account of the extent to which the problems outlined in the grantee’s statement of need were met.
Statement of Need:
This area should be preloaded by the ANA evaluator prior to the visit. Record the statement of need provided by the grantee in the project proposal. Throughout the evaluation, think about the extent to which the project has addressed or overcome the problems originally identified.
How to record:
Please read the preloaded statement of need to the grantee or have them review it. Then ask the grantee to describe the extent to which they feel project activities have addressed the problems detailed in the statement of need section of their original proposal.
Question 2.3 - Impact Summary
What to record:
ANA is interested in hearing from the beneficiaries of, or specific groups affected by and/or involved in, its projects. ANA seeks information on how these groups benefited from the project, specifically during the implementation period. The evaluator will ask the grantee to provide information for this section, but will also utilize the impact summary when interviewing community members. Beneficiaries are defined by the grantee or interviewee (not the evaluator) and can include, but are not limited to: tribal youth, elders, staff, tribal government, and the community.
How to record:
Please ask the grantee or community member to identify project beneficiaries. Provide an introductory statement explaining what kind of information ANA is seeking. If necessary, provide examples of some groups, but be sure to allow the grantee or interviewee to determine which groups were affected by the project.
Enter one group in each beneficiary box and then record comments made by the grantee or interviewee. This is usually an opportune time during which to gather direct quotes from community members and project staff regarding the impact of ANA projects. These quotes can be included in the Congressional Impact Report.
Required Documentation:
ANA does not require grantees to provide supporting documentation for their reported project beneficiaries.
Question 3.1 – Who developed the project proposal?
What to record:
Record the names and titles of individuals significantly involved in the development of the ANA project application. Those involved do not necessarily need to be the authors, but can include individuals who contributed significant input during the process. Do not include individuals who may have worked on related documents but not the actual ANA application. These documents could include business and/or strategic plans developed prior to the ANA project proposal.
How to record people who developed the project proposal:
Record the names and titles of the individuals involved in the proposal process. Also note their relationship to the community targeted by the ANA project.
Documentation required:
ANA does not require any documentation to verify the proposal developers. This information should be collected by the impact team through the interview process.
Question 3.2 – Community Involvement
What to record:
Record the extent of community involvement during the planning phase and the implementation phase of the project.
How to record:
Ask the grantee if they feel the community was not involved, involved, or very involved during the two project phases. Record the answer by checking the appropriate box in each row of the table. If this question is not applicable to the project, for example a code and ordinance project that did not call for community involvement, check the N/A column.
Question 3.3 - Elders
ANA definition:
ANA respects and accepts the definition of “elder” set forth by each tribe. If the grantee does not have a formal definition, ANA defines elders as tribal members over 55 years of age.
What to record:
ANA is interested to learn whether elders were involved in the implementation phase of the ANA-funded project. ANA records the number of elders actively involved during project implementation. When recording the number of elders involved, only include those who actively participated. For example, elders contributing to language or cultural preservation activities, on committees, staff members, etc. Do not count elders who attended a conference or watched a performance as their involvement is considered passive.
How to record elders involved:
In section 3.3, enter the number of reported elders actively involved in project implementation in the appropriate space.
Documentation required:
ANA does not require specific documentation to verify the number of elders involved. However, ANA encourages the impact team to verify reported elder involvement to the extent possible. Examples of documentation include, but are not limited to, copies of meeting minutes and volunteer records.
Question 3.4 - Youth
ANA definition:
Any person less than 18 years of age.
What to record:
ANA is interested to learn whether youth were involved in the implementation phase of the ANA-funded project. ANA records the number of youth actively involved during project implementation. When recording the number of youth involved, only include those who actively participated. For example, children attending a day camp, language class, or training should be counted. Do not count children watching a performance or movie, as their involvement is considered passive.
How to record youth involved:
In section 3.4, enter the number of reported youth actively involved in project implementation in the appropriate space.
Documentation required:
ANA does not require specific documentation to verify the number of youth involved. However, ANA encourages the impact team to verify reported youth to the extent possible. Examples of documentation include, but are not limited to, classroom attendance records, camp sign-in sheets, and volunteer records.
Question 3.7 – Intergenerational Exchange
Determine if the project promoted intergenerational exchange and check the appropriate box. If yes, then ask the grantee to specify whether the exchange was between elders and youth, and/or grandparents and grandchildren, and check the appropriate box.
Question 3.8 – Outreach activities and marketing materials
Determine if any marketing or outreach activities or marketing materials were developed during the project, and ask the grantee to identify the three most useful. List these three in the space provided with any relevant comments.
Question 3.9 – Did any members of the community express doubts or misgivings about your project?
ANA definition of doubts and misgivings:
Any concerns, worries, fears, suspicions or other reservations from the community about the ANA project, or its actual or potential effects.
What to record:
Any such feelings from one or more people, either inside or outside the community.
How to record community concerns:
In section 3.8, mark “yes” if the grantee states that one or more people expressed doubts or misgivings about the project. If you mark “yes,” elaborate on the types of doubts or misgivings expressed and the number of people expressing them in the box below.
Question 3.10 – Opposition
ANA definition:
Any vocal or active opposition to project implementation activities.
What to record:
Any opposition from one or more people, either inside or outside the community.
How to record:
In section 3.9, mark “yes” if the grantee states they faced opposition while implementing the project. If you mark “yes,” elaborate on what type of opposition occurred.
Question 4.1 - Challenges
What to record:
ANA collects information on the challenges encountered by grantees during project implementation, whether the grantee anticipated the challenges and if it was able to overcome the challenge and how. This information is used by ANA to improve service delivery and to ensure that projects are successfully implemented.
How to record:
First, please ask the grantee to discuss the challenges encountered during the project’s implementation that seriously impeded the project’s progress. Record these challenges and explanatory or supporting comments in the table provided directly under question 4.1. The table has six blank spaces for challenges at the bottom and is pre-filled with the most common challenges reported by grantees. NOTE: Do not simply ask if the project encountered the pre-filled challenges. This is an open ended question for the project to self identify their challenges and how they overcame them. Fill in the table according to responses provided by the grantee. If applicable, comment on how the challenge was overcome.
After the grantee has finished describing their challenges, please ask them to rank the challenges according to their severity (1 = most severe).
Required Documentation:
ANA does not require grantees to provide supporting documentation for their reported challenges.
Question 4.2 – Additional comments for challenges.
Record any additional comments related to severe project challenges here.
Question 3.3 – Is there anything you would have done differently implementing the project?
Record responses to this question in the space provided.
Question 5.1 - Personnel
For section 5.1.1, determine if the project director at the time of your visit is the original project director and check the appropriate box. Regardless of the answer, ask the question that follows regarding ANA Post-Award Training and check the appropriate box. If the grantee stated the current project director is not the original director, please ask the grantee to provide the names, start dates, and end dates of all project directors, regardless of the amount of time they actually worked on the project.
For section 5.1.2, determine if all the positions proposed in the application were filled and check the appropriate box. Determine if all budgeted staff was hired within the planned timeframe in the OWP. In the comments section, provide information on the positions not filled and/or not hired within the OWP timeframe.
For section 5.1.3, determine if any consultants were hired for the project and check the appropriate box. If the grantee answers “yes,” record the total number hired and the total number who were Native American in the spaces provided. If numerous consultants were hired from the same consulting firm, count each individual person. If there was turnover in a consultant position, also count each person who worked. If a consultant contributes his/her labor without receiving income, do not count the consultant in this section. Instead, record this in-kind service in the leveraged resources table.
Question 5.2 - Full-Time Job Equivalents (FTEs) Created
ANA definition:
Any non-consultant position paid for through ANA grant funding, non-federal matching funds, or leveraged resources. This only includes positions directly related to the project and its goal(s) and objectives. For all positions sustained for 6 or more months, ANA assigns one FTE, or percentage of an FTE, based on the number of hours worked in a 40 hour workweek. For positions lasting less than 6 months, FTEs are calculated at the end of the year and are based upon the total hours worked.
What to record:
ANA collects data on three types of jobs created:
Maintained Jobs: positions that existed in some form prior to the project’s implementation (e.g., a program manager working .20 FTEs for the ANA project);
Project-Specific Jobs: positions created specifically for the project; and
Post-Project Jobs: positions created by the project that exist by the end of the project, and will continue into the future. These include, but are not limited to, positions created through workforce or business development projects.
How to record FTEs created:
First, determine what type of job was created: Maintained Jobs, Project-Specific Jobs, or Post-Project Jobs. When inputting information for each job, make sure that it is organized within the correct space within the jobs table in section 5.2 of the tool.
Next, following these steps:
First column: input the job title
Second column: input the total number of positions. This will capture information when the project creates several of the same position (e.g. 7 surveyors)
Third column: input the total number of people to fill the position(s) during the life of the project. This will capture staff turnover within the position
Fourth column: input the number of indigenous / native people that filled the position(s)
Fifth column: input the number of hours worked per week for the position
Sixth column:
If the position lasted more than 6 months, input “52”
If the position lasted less than 6 months, input the total number of weeks worked
Seventh column:
If the position was fulltime and lasted more than 6 months, input “2080”
If the position lasted more than 6 months but was less than fulltime, input the total from the following equation:
# of hours worked per week X 52
If the position was fulltime and lasted less than 6 months, input the total from the following equation:
40 X # of weeks worked
If the position was part time and lasted less than 6 months, input the total from the following equation:
# of hours worked per week X # of weeks worked
Source of job funding: Determine the source of funding for each position and allocated the total hours funded by each source in the next four columns (i.e. ANA grant funding, Project-Generated funding, Leveraged Resources, or Non-Federal Share)
Final columns:
For jobs that existed prior or were created during/for the project: check the box if the position will continue after the project ends
For jobs that were created for post-project activities: check the box if this job was created due to a business that was created as a result of the project
Documentation required:
ANA does not require a specific type of documentation to verify the number of jobs created. However, ANA encourages each impact team to verify jobs created when possible by corroborating the reported jobs with documentation reviewed in the Financial section (Section 7).
Question 6.1 – Income Generated
ANA definition:
Any income generated as a result of project activities.
What to record:
Record gross income created by project activities such as the sale of products or services. Only record income generated during the project timeframe (do not include project income or income generated after ANA funding has ceased). Do not record donations or free products or services given to the grantee in relation to the project. Items such as these should be included in the leveraged resource total.
Keep in mind that project-generated income should be reinvested back into the project during the life of ANA funding. Furthermore, project-generated income should be used before ANA project funding is spent down.
How to record income generated:
In the table in section 6.1.2, list each activity that generated project income (e.g., sale of project-developed curriculum). Next, note the type of documentation reviewed by the impact team and enter in the total amount of income generated to date.
Documentation required:
Any project reporting income should produce hardcopy documentation regarding revenue expended. Documentation can include, but is not limited to, sales receipts, bank reconciliations, profit-loss statement, etc. If no documentation was provided during the impact visit, write “none” and add any pertinent follow-up comments underneath the table in the “comments” section.
Question 6.1.3 – Was (or will) project income be used to sustain the work of the project after the grant ends?
What to record:
Record whether or not the project generated any income and specifically explain how it was used.
How to record:
Ask the grantee if project-generated income was, or will be, used to sustain the work of the project after the grant ends. Check the appropriate box and ask the grantee to explain in what way the income was utilized.
Documentation required:
ANA does not require any documentation to verify grantee statements for this question.
Question 6.2 – Businesses Created
What to record:
Record the businesses created as a direct result of the ANA-funded project.
How to record businesses created:
Ask the grantee if any businesses were created as a direct result of the ANA project. If the answer is “yes,” fill in the name, whether this business was created or expanded as a result of the project, the purpose and the type of business (cottage, micro or licensed) for each created or expanded business.
Required Documentation:
ANA does not require any documentation to verify the creation of a business. However, ANA welcomes supporting documentation when available.
Question 6.3 – Supporting Native Owned Businesses
What to record:
Record if the project supported native-owned businesses (e.g., paying for copies at a native-owned copy center).
How to record if the project supported native owned businesses:
Check “yes” or “no” in appropriate box.
Required Documentation:
None required.
Question 6.4 – Training
ANA definition:
A session, seminar, class, or similar learning activity facilitating the acquisition of practical, job-specific knowledge, skills, and abilities. To be considered a training, the session should facilitate learning geared towards advancing project goals and/or build organizational capacity to sustain project results and benefits.
What to record:
Record project-funded trainings (funded or supported through ANA dollars, non-federal share, or leveraged resources) conducted or attended by individuals involved in the ANA-funded project. Do not include attendance at ANA’s mandatory post-award training or other ANA-mandated trainings or attendance at informational conferences or workshops. If in doubt as to whether to include a training, ask yourself the following question:
Would this training have occurred without the ANA project?
If the answer is “no”, include the training.
How to record training:
In the table in section 6.4, list the title or primary objective of the training. In the second column, list the total number of people who successfully completed each training. In the third column list only the unduplicated number of people who completed the training, as some individuals may have participated in more than one training session. If, for example, one person participated in three trainings during the course of the project, count him/her only once, included in the first-listed training they completed, rather than three times (one for each training). At the bottom of each column, list the totals.
For family preservation projects only: Complete the final column of the table by recording the number of hours needed to complete each training.
Documentation required:
Any grantee reporting project-related training should produce hardcopy documentation during the impact visit. Documentation can include, but is not limited to, certificates of completion and training payment receipts.
Question 6.5 – Project Sustainability
ANA definition:
A sustainable project is an ongoing program or service that can be maintained without ANA funding.
What to record:
The questions in this section are designed to elicit information on what steps the grantee has taken to ensure project sustainability after ANA funding terminates, and to assess the extent to which these actions have laid the groundwork for the effective continuation of the project’s services and/or benefits. .
How to record project sustainability:
In section 6.5.1, determine if the grantee developed a formal sustainability plan for the project and check the appropriate box. Ask the grantee to provide a brief synopsis of how the results/benefits/services of this project will be sustained. Record this information in the space provided.
In section 6.5.2, ask the grantee what steps were taken to finance the project’s sustainability. Ask this question regardless of the answer to 6.5.1, meaning even if the grantee did not develop a formal sustainability plan. The evaluator may check as many boxes as apply depending on the grantee’s answer.
In section 6.5.3, ask the grantee what funding has been secured for the next 12-month period. If the grantee finishes discussing the sustainability section and the impact team still does not know which of the five choices in 6.5.3 to select, the team should ask clarifying questions to make the proper determination. The box ‘Do not need to sustain’ should be checked only if specifically stated by the grantee.
In section 6.5.4, ask the grantee to share some of the project’s best practices. If there is confusion on this question, ask the grantee what advice they would give to someone conducting a similar project. Record the information in the space provided.
Documentation required:
ANA does not require specific documentation to verify sustainability reporting. However, each impact team is encouraged to solicit supporting documentation to the extent possible.
Question 7.1 – How did you track the non-federal contribution to the project?
Definition of non-federal share: The total dollar value of all non-ANA funded project costs. Grantees must provide at least 20 percent of the total approved cost of the project. The non-federal share total is listed on the Federal Assistance Award (FAA).
What to record:
In this section, please collect data on the project’s non-federal share (NFS). The purpose of question 7.1 is to elicit information on how the grantee tracked the non-federal contribution to the project.
How to record:
In question 7.1, record the applicant’s response and note the documentation reviewed during the visit.
Documentation required:
Grantees should have an in-kind documentation sheet, invoices, timesheets, etc. for each type of NFS match claimed for the project.
Question 7.2 – Did you make any changes to your budget?
What to record:
In this section, please collect data on any modifications to the approved budget. If you feel the budget changes are significant, make note of it in the space provided. Significant changes can include, but are not limited to, moving funds between line items, creating new line items, moving funds from direct costs to indirect costs or vice versa, or allocating funds to new budget categories.
How to record budget changes:
Fill in the appropriate box based on grantee’s response and note specific budget changes in the space provided.
Required Documentation:
ANA does not require specific documentation for this question.
Question 7.3 – Were there, or do you expect there to be, any ANA funds left over at the end of the ANA project?
What to record:
For this question, please determine whether there were, or will be, any unobligated funds after ANA funding has ceased, to ensure that budget spending aligns with the project timeframe.
How to record budget changes:
Check the appropriate box and note amount and comments in spaces provided.
Documentation required:
ANA does not require specific documentation for this question.
Question 7.4 – Did a financial staff member attend the ANA Post Award Training?
How to record:
Check the appropriate box and list position title in space provided.
Question 7.5 – How often does someone from your staff meet with program staff with regard to this project?
How to record:
Record grantee’s answer to the question in the space provided.
Question 7.6 - Audit
What to record:
Write the date for the grantee’s most recent .
Question 7.7 – Financial Procedures
What to record:
Ask grantee questions 7.8.1-7.8.3 and fill in answers in spaces provided.
For question 7.8.4, ask to view sample documentation for the three most expensive budget line items (as directed in the tool) and record comments on findings in the space provided.
Financial Management Documentation:
Question 7.8
What to record:
Please discuss the financial management systems with the grantee. To complete section 7.6, ask which of the documents the organization has (from a-d) and check the appropriate box. Follow the directions provided on the tool and move on to the appropriate section.
SEDS Grants
What to record:
ANA compiles data on the number of codes and ordinances developed and/or implemented for use in the project snapshot section of the congressional report and for data analysis.
How to record:
Determine if any codes or ordinances were developed as a result of the project and check the appropriate box. If yes, record each code or ordinance according to type in the first column of the table. Include the number of codes or ordinances if multiple for one type. Determine if the code or ordinance was adopted or enacted and check the appropriate box in the center column. Determine if the code or ordinance was implemented and check the appropriate box in the last column. If the code or ordinance was not implemented, ask the grantee to explain why not and record the information in the space provided in the last column.
Family Preservation Grants
What to record:
ANA compiles data on the types of curricula utilized, number of people and couples served, number of foster children placed in native homes, and number of foster parents trained during family preservation projects.
How to record:
Go through questions 8.2 – 8.7 and their sub questions depending on the type of family preservation project. Some questions may not apply to the specific project being visited.
For question 8.7.1 please note that if a child is placed with multiple native families throughout the course of the project, this is counted as only 1 child.
Language Grants
Immersion
ANA definition: Language instruction in which the regular academic curriculum is taught in the native language.
What to record:
ANA compiles data on the number of language surveys developed, the number of people completing surveys, the number of native language classes held, the number of days immersion classes were taught, the number of teachers trained, and the number of people who either became fluent or increased their ability to speak a native language. Counting the individual language classes taught during a single day of immersion programming will skew the total number of classes taught. It is therefore important to distinguish between language immersion classes and standard language instruction. The number of language classes held refers to classes focused on the teaching of the language (e.g. A high school student taking French classes), whereas immersion classes refer to subjects taught in the native language and are therefore recorded by counting the number of days taught. For example, if a project teaches level I, level II and level III classes, these will be recorded in section 8.7.1. Language immersion can include math, science and history classes taught in the native language at an immersion school, and will be recorded in section 8.8.1.
How to record:
Go through questions 8.8 - 8.12 and their sub questions depending on the type of language project. Some of the questions may not apply to the specific project visited. For example if the project did not involve training teachers, simply skip question 8.9.
Environmental Grants
What to record:
ANA compiles data on the number of environmental regulations, codes or ordinances created during a project, as well information regarding environmental data collection and project sustainability.
How to record:
Determine what the focus of the project was by asking question 8.13 and checking the appropriate box. Then ask the subsequent questions associated with that specific project focus. For example, if the focus is identified as “develop regulations,” skip questions 8.14 and 8.15 and only ask question 8.16.
Note: If the evaluator determines that other questions in this section apply to the project being visited, or if there are multiple project foci, please ask all relevant questions!
If the project focus falls into the “other” category, please look through the follow-up questions (8.13 – 8.17) and determine if any apply to the project.
If the evaluator feels any of the sustainability or challenge questions have already been addressed earlier in the visit, there is no need to have the grantee repeat the information and these questions can be skipped.
For question 8.16 determine if any environmental regulations, codes or ordinances were created as a result of the project and check the appropriate box. If yes, then record each regulation/code/ordinance according to type in the first column of the table. Include the number of regulations, codes or ordinances if multiple for one type. Determine if the regulation, code or ordinance was adopted, enacted or passed and check the appropriate box in the center column. Determine if the regulation, code or ordinance was implemented and check the appropriate box in the last column. If the regulation, code or ordinance was not implemented, ask the grantee to explain why not and record the information in the space provided in the last column.
Impact Evaluation Manual
Chapter 2: Data Collection Protocol
Page
| File Type | application/msword |
| File Title | Chapter 2: Protocols to Follow during the Impact Visit |
| Author | mgallagher |
| Last Modified By | mgallagher |
| File Modified | 2010-03-30 |
| File Created | 2010-03-30 |