Organization Capacity Assessment Justification

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSIONS

 

A. Justification

A1. Need for Information Collection

The purpose of this request is to pilot test and validate the Organizational Capacity Assessment tool and solicit grantee feedback.

The Corporation for National and Community Service (CNCS) is working with ICF International (ICF) to develop a reliable and validated organizational capacity assessment tool tailored for CNCS grantees. The purpose of this tool is to enable CNCS grantees to identify strengths and areas for potential capacity building, and to measure changes in organizational capacity over time.
As part of the process, ICF is examining the clarity and construction of the assessment tool’s statements and questions and the overall burden of completing the assessment tool on the organization.

A2. Indicate how, by whom, and for what purpose the information is to be used.

Grantees will respond to the questions included in these instructions in order to assess the clarity and construction of the statements and questions and the overall burden of completing the assessment tool. CNCS will use the information to validate, revise and improve the Organizational Capacity Assessment tool.

 

 A3. Minimize Burden: Use of Improved Technology to Reduce Burden

CNCS will be eliciting and accepting grantees’ responses via telephone, mail or email using the attached forms and instructions.

A4. Non-Duplication

There are no other sources of information by which CNCS can meet the purposes described in A2 (above).

 

A5. Minimizing for economic burden for small businesses or other small entities.

This collection of information does not impact small businesses because they are not eligible for CNCS grants and would not be selected to participate. There is no economic burden to any other small entities beyond the cost of staff time to collect and report the data. This is minimized to the degree possible by only asking for the information absolutely necessary to assess an organization’s ability to understand the questions to complete the tool.

A6. Consequences of the collection if not conducted, conducted less frequently, as well as any technical or legal obstacles to reducing burden.

 

CNCS will be unable to request the necessary information to validate the tool and ensure it is comprehensive for CNCS grantees and meets their needs to assess their organizational capacity.

A7. Special circumstances that would cause information collection to be collected in a manner requiring respondents to report more often than quarterly; report in fewer than 30 days after receipt of the request; submit more than an original and two copies; retain records for more than three years; and other ways specified in the Instructions focused on statistical methods, confidentially, and proprietary trade secrets.

This pilot and cognitive interviews will ask participants to schedule interviews or responses at grantees’ convenience, though request will be suggested within a 30 day timeline. Participation is completely voluntary, however, and grantees are able to decline participating if the timeline creates an unnecessary burden.

A8. Provide copy and identify the date and page number of publication in the Federal Register of the Agency’s notice. Summarize comments received and actions taken in response to comments. Specifically address comments received on cost and hour burden.

Not applicable.

 

 A9. Payment to Respondents

There are no payments or gifts to respondents

A10. Assurance of Confidentiality and its basis in statute, regulation, or agency policy.

Participants’ responses to this information collection will be disclosed to the contractor as appropriate for analysis purposes. No additional information will be shared unless required by law.

A11. Sensitive Questions

 

The information collected does not include questions of a sensitive nature.

 

A12. Hour burden of the collection

We expect approximately 1.5 hours of effort for the cognitive interviews and 1 hour of effort for the pilot testing. We expect 6 respondents to participate in cognitive interviews and 150 respondents to participate in pilot testing.

 

A13. Cost burden to the respondent

There is no cost to the respondent.

 

A14. Cost to Government

$7,695.68 paid to the contractor to conduct the testing and analysis. This cost to government is for a contract with ICF international to conduct the cognitive interviews, conduct the pilot tests, and analyze the results for CNCS to protect CNCS grantees anonymity

A15. Reasons for program changes or adjustments in burden or cost.

  Not applicable.

 A16. Publication of results

Not applicable. 

 

A17. Explain the reason for seeking approval to not display the expiration date for OMB approval of the information collection.

Not applicable.

 

 A18. Exceptions to the certification statement

There are no exceptions to the certification statement in the submitted ROCIS form.

B1. Describe the potential respondent universe and any sampling or other respondent selection methods to be used.

Since we are interested in understanding the organizational capacity of funded organizations, the sampling will focus on identifying a sample that is representative of each organization type. We divided the organizations into strata with each strata representing an organization type. Table 1 shows the characteristics of the estimated final sample.

Each organization selected for pilot testing will be asked to select three individuals (e.g. board members and/or senior or mid-level staff including the director or lead of the organization) to complete the assessment tool. Table 1 shows the characteristics of the estimated final sample for pilot testing and Table 2 shows the estimated final sample for cognitive interviews.

 

Table 1: Selected Sample of Organizations for Pilot Testing

Organization Type	Full Sample N=2,028		Selected Sample (Estimated)
			n=50	n = 150
	Organizations	Percent	Organizations	Respondents
Higher Education	169	8%	4	12
Local Government	251	12%	6	18
National Non Profit	53	3%	1	3
Non-Profit	1,418	70%	35	105
Other	28	1%	1	3
State Government	109	5%	3	9

Table 2. Cognitive Interviews

Organization Type	Respondents
Higher Education	1
Local Government	1
National Non Profit	1
Non-Profit	1
Other	1
State Government	1

B2. Describe the procedures for collection of information.

Cognitive Interviews:

The purpose of the cognitive interviewing is to ensure that the respondents understand the questions in the way intended by the team. Respondents will be asked to “think aloud” – that is, to share verbally with the interviewer the process by which they come to their answers in the tool. There are a series of repeated interviewer probes used throughout the protocol to assist and guide the respondent through the process; examples are detailed in Table 3. Each probe is designed to gather information about a statement or a series of statements and explore the extent to which its meaning is clear while soliciting input about ways to increase the clarity and simplicity of statements and topics.

Table 3: Cognitive Interviewing Probes

Probe	Example
Response Specificity	What were you thinking as you gave your answer?
Statement Intention	Can you repeat the statement in your own words?
Response Options	Are there reasons why you do not have a strategic plan that are not listed here?
Time Frame Anchors and Comparisons	How sure are you that you can remember how long it takes for new hires to buy into your mission?
Response Confidence	How confident are you in your response?
Instructions and Examples	How well did the categories fit your mission?
Word-Specific	What do you think we mean by “mission?”

Pilot Interviews:

Once the sample has been identified, CNCS will send a pre-introduction email to the grantees, as well as program officers. ICF will then email each organization the Introduction Email (Appendix F) along with a copy of the assessment tool to request their participation in the pilot test. The Introduction Email will be sent to the Director or lead of the organization. Each organization will be asked to identify at least 3 board members and/or senior or mid-level staff including the Director or lead of the organization to complete the assessment tool. Since potential respondents will be emailed the assessment tool, the pilot test participants will be instructed to complete the assessment tool and email or mail it back to ICF. The assessment tool will be in Microsoft Word and Acrobat Portable Document Format (PDF) to accommodate the different capacities of the organizations.

B3. Describe methods to maximize response rates and to deal with issues of non-response.

Maximizing the response rate is a key priority for this study. The researchers will employ a variety of strategies to maximize the response rates and to address issues of nonresponse. This is compatible with research suggesting that the use of several different strategies can improve increase the success of retention efforts (Robinson, Dennison, Wayman, Pronovost, & Needham, 2007).

Maximizing the Response Rate

The specific strategies that will be used to maximize the response rate include:

1. Targeted communications via the tool introduction and email message;

2. Using multiple reminders to contact and follow-up with respondents

Targeted Communication:

The evaluation team will use multiple communication approaches including personalized, pre-notification via email to grantees from CNCS program officers.

Using Multiple Contacts

The tool will be distributed via email and participants will be able to return it by mail, fax or email to improve the overall response rate and reduce attrition, based on respondent preference. Non-respondents will be contacted up to three times with reminder emails or phone calls. All respondents will be informed of the significance of the tool, and the importance of their feedback to help improve it, to encourage their participation. In addition, processes to increase the efficiency of the tool (e.g., ensuring that the tool is as short as possible through consultation with and pre-testing through a technical working group) and the assurance of confidentiality will encourage participation.

B4. Describe any tests or procedures or methods to be undertaken.

Cognitive Interviews:

The interviews will be conducted by an ICF interviewer and a note taker. This interview will also be audio recorded for note taking purposes. During the interview, the interviewer will ask probing questions such as, “What were you thinking about as you gave your answer? Can you repeat the question in your own words?” to determine the respondent’s understanding of the statements, and responses (See Appendix D: Interview Protocol). Following the interviews, the data collected will be analyzed for refining the assessment tool prior to the pilot test.

Analysis

The analysis will use an investigative qualitative approach that will focus on identifying themes and patterns of interpretation which will either highlight any weaknesses in or confirm the formulation of the assessment tool statements. Further, the analysis of the cognitive testing results will be guided by the following questions:

• How well do the respondents understand the assessment tool statements?

• Do the respondents understand the intended meaning of the assessment tool statements?

Following the analysis, the ICF team will summarize the cognitive interviewing process, detailing results from the analyses, and listing any suggested changes that should be incorporated into the tool prior to pilot testing. The recommendations will be to either keep the statement as is, revise the statement based on feedback, develop a new statement, or delete the statement entirely from the tool. Following discussion with CNCS, ICF will update the organizational capacity tool to incorporate the suggested changes identified as a result of the cognitive interviewing process.

Pilot Testing:

The analysis of the pilot test will begin by producing frequency distributions and cross tabulations of all variables. We will target any items that have low item-total correlations to identify poorly performing statements.

The goal of the pilot test analysis is to determine:

• Effectiveness of the tool administration process

• Factors affecting tool implementation and response rates

• Significant differences in response rates, and response options by subgroup

• Burden of the assessment tool on the respondent (e.g. assessment tool response time)

• Non-response rate for each item

Validity: Each of the five domains will be analyzed independently to evaluate the individual item fit. This analysis allows us to identify problematic items, and those that are being interpreted differently by participants than intended.

Reliability: Depending on the response rate, ICF proposes the use of interrater reliability to determine the degree of agreement within subgroups and across the subgroups. ICF will examine the internal consistency of the five domains using Cronbach’s alpha. ICF will also evaluate the consistency of responses across participant groups. This is useful as it allows us to determine if AmeriCorps grantees perceive the assessment tool items in the same way as Senior Corps grantees.

After completing the analysis, the ICF team will summarize the pilot testing process, detailing results from the analyses, and listing any suggested changes that should be incorporated into the tool. The memo recommendations will be to either keep the statement as is, revise the statement based on feedback, develop a new statement, or delete the statement and not replace. Accompanying this report will be an updated, final organizational capacity tool that incorporates all changes identified during the piloting processes.

B5: Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor who will collect and/or analyze the information for the agency.

Contact for data collection and analysis:

Venessa Marks
Senior Manager
ICF International
646.895.9714
[email protected]

Contact for Statistical aspects and analysis:
Nanette Antwi-Donkor
Senior Technical Specialist
ICF International
703.225.2249
[email protected]

Appendices

Appendix A: Cognitive Interview Introduction Email

Dear <Name>,

The Corporation for National and Community Service (CNCS) is working with ICF to develop a reliable and validated organizational capacity assessment tool tailored for CNCS grantees. The purpose of this tool is to enable CNCS grantees to identify strengths and areas for potential capacity building, and to measure changes in organizational capacity over time.

As part of the process, ICF is examining the clarity and construction of the assessment tool’s statements and questions and the overall burden of completing the assessment tool on the organization. We would like to invite you to evaluate the assessment tool.

If you agree, we will forward you a copy of the latest version of the assessment tool along with instructions. After receiving the assessment tool, we would schedule a 90 minute interview at your convenience to discuss the assessment tool. This interview is confidential and we will not identify you or your organization in any reports by name or any other personal identifiers in any analysis or reports resulting from this interview.

Thank you for your time and assistance in this project. Your feedback will help CNCS and ICF ensure the effectiveness of the assessment tool for organizations like yours. Should you have any questions about the cognitive interview, you may contact Nanette Antwi-Donkor, [email protected].

Again, thank you for your consideration,

<Sender’s Name>

Appendix B: Cognitive Interview Follow Up Email

Dear <Name>,

Thank you very much for agreeing to review the Corporation for National and Community Service’s (CNCS) Organizational Capacity Assessment tool.

Attached is the CNCS Organizational Capacity Assessment tool. This tool will be administered via paper and pencil so please print the assessment tool for your review. If possible complete the assessment tool in one sitting and document your reactions using the following guiding questions:

• How long did it take to complete the study?

• Were any questions unclear? Please give specific examples.

• Is the format and layout of the tool pleasing?

• Do you have any suggestions for improving the tool or instructions?

Following your review, we would like to conduct a 90 minute interview with you to discuss your reactions and feedback to the assessment tool. Below are a few dates and times for this call. Please let me know if any of these times work for your schedule, or if not, when you would prefer to meet.

<Date Option>

<Date Option>

<Date Option>

The interview will be recorded and transcribed for precision. This interview is confidential and we will not identify you nor your organization in any reports by name or any other personal identifiers in any analysis or reports resulting from this interview. Please review and sign the attached Informed Consent Form and return prior to our call.

Should you have any questions, you may contact the Nanette Antwi-Donkor, [email protected].

Again, thank you for your assistance.

<Name>

Appendix C: Cognitive Interview Informed Consent Form

Background

The success of an organization is dependent on how well it knows and understands its capacity to manage and deliver effective programming. Organizational capacity assessment tools empower organizations to better measure their capacity to effect change in their communities and measure their performance. However, many organizational capacity assessment tools are expensive, generic or not publically available.

The Corporation for National and Community Service (CNCS) has partnered with ICF to develop a reliable and validated organizational capacity assessment tool tailored for CNCS grantees. This assessment tool will allow CNCS grantees to identify strengths and areas for potential capacity building, and to measure changes in organizational capacity over time.

Process

To ensure the credibility of the organizational capacity assessment tool, ICF will be conducting a series of cognitive interviews with potential respondents. The cognitive interviews will examine the construction and respondent burden of the assessment tool. The interviews will seek to gain insights into how the respondents understand, process and respond to the assessment tool statements and questions. The interview will be recorded for note taking purposes. However, your name and organization will not be associated with the recording.

Why Participate

Your participation and responses will help CNCS and ICF ensure the effectiveness of the assessment tool for organizations like yours.

Respondent’s Role

You will participate in a 90 minute interview scheduled at your convenience to discuss your feedback on the proposed assessment tool. Prior to the interview, you will be asked to review and respond to the assessment tool and document your feedback using the accompanying guiding questions.

Risks and Benefits

The risks and benefits associated with this pilot test are minimal. Your participation is completely voluntary. You will not be forced to answer any questions that you do not wish to. All information we collect from or about you will be protected to the extent allowed by law. We will not identify you nor your organization in any reports by name or any other personal identifiers in any analysis or reports resulting from this interview. Any data collected will be protected and kept confidential through the use of IT security procedures and management policies.

Questions

Should you have any questions about the cognitive interview, you may contact the Nanette Antwi-Donkor, [email protected].

Consent Statement

• I understand my participation in this pilot test is voluntary and that I can stop responding to the questions at any time.

• I understand that my answers to these questions will only be seen by the ICF team.

• I understand that my name will not be included in any reports or presentations of findings and that my data will be securely handled.

• I understand that the risks and benefits of participating in this study are minimal.

• I agree to participate in this pilot test.

Name (Please Print):
Signature:
Date:

Appendix D: Cognitive Interview Protocol – Note Taking Template

Introduction to the Assessment Tool

We are interested in finding out how organizations like yours understand and interpret questions and statements in the Organization Capacity Assessment Tool.

I am going to take you through parts of the assessment tool, reading all instructions, and statements. We are going to be using a technique called “think aloud” – all this means is that as you answer the assessment tool questions, please tell me anything you may be thinking to get to your answers. I might ask you to help me think of ways to improve the wording of the tool. If at any point, you have something to say about a question or the answer choices please do not hesitate to tell me.

I want to remind you that your participation in this interview is completely voluntary and we will not be using any of your answers to the tool statements, nor will we show them to CNCS. If at any point during the interview you wish to stop, you may do so. I would like to record the interview for note taking purposes. However, your name and organization will not be associated with the recording.

Do I have your consent to record?

Do you have any questions?

Are you ready to begin the interview?

Great! Let’s get started with a warm-up question to practice thinking out loud. First, can you tell me:

How many shoes do you own? <<Prompt: don’t forget to think out loud!>>

1. How did you come to that answer?

2. What were you thinking about?

3. Did you count ice skates? Did you count slippers? Did you count kids’ shoes?

Great work! Now, let’s start the assessment tool!

INTERVIEWER: Start the Interview

<Tool Section>

Let’s get started. I will read each question/statement to you. I would like you to listen to the entire question/statement and then answer it as you understand it. Please choose the statements that best reflect your organization. Remember that we are not going to use your answers to the tool’s questions/statements.

The next few questions/statements are about your <section topic>. Please tell us what you really think. Your honest answers will help us improve the effectiveness of the tool.

1. <Question/Statement>

INTERVIEWER: Listen and record respondent answer and verbal think aloud stream.

1. Please repeat the question using your own words. What do you think this question is asking you?

2. What were you thinking about as you gave your answer? A specific incident, or an overall rating?

3. Did anything about this question make it difficult to answer? Please explain.

4. What do you think we mean by “concept?” Who did you think of?

DEBRIEF

That is all the tool questions/statements I have for you today. Just a few wrap up questions, and then you are all done!

1. First, was there anything that I asked you about today that you found to be offensive or upsetting?

2. Was there anything that you were surprised I asked you about? Anything you expected, but I did not ask you about?

3. How easy or difficult was it to answer the questions, on the whole?

d. Are there any other comments, questions, or concerns you would like to share?

Thank you very much for taking the time to talk to us today!

Appendix E: Pilot Test Introduction Email

Dear <Name>,

The Corporation for National and Community Service (CNCS) is working with ICF to develop a reliable and validated organizational capacity assessment tool tailored for CNCS grantees. This assessment tool will allow the CNCS grantees to identify strengths and areas for potential capacity building, and to measure changes in organizational capacity over time.

As part of the process, ICF is pilot testing the assessment tool to ensure its clarity and effectiveness at capturing useful information. We would like to invite your organization to participate in the pilot test. If you agree, please have 3-4 key staff members including yourself complete the attached assessment tool and email it to <email> by <Date>.

Your participation is voluntary and confidential. We will not identify you or your organization in any reports by name or any other personal identifiers in any analysis or reports resulting from this pilot test. Your answers will not be seen by and/or used in any grant making decisions by CNCS.

Thank you for your time and assistance in this project. Your feedback will help CNCS and ICF ensure the effectiveness of the assessment tool for organizations like yours.

Should you have any questions about the cognitive interview, you may contact Nanette Antwi-Donkor, [email protected]

Again, thank you for your consideration,

<Sender’s Name>

Appendix F: Pilot Test Informed Consent Form

Background

The success of an organization is dependent on how well it knows and understands its capacity to manage and deliver effective programming. Organizational Capacity Assessment tools empower organizations to better measure their capacity to effect change in their communities and measure their performance. However, these organizational capacity assessment tools may be expensive, generic and not publically available.

Corporation for National and Community Service (CNCS) is working with ICF to develop a reliable and validated organizational capacity assessment tool tailored for CNCS grantees. This assessment tool will allow CNCS grantees to identify the core elements of their organizational capacity that performance, and to identify gaps in their organization and programs.

Why Participate

To ensure the credibility of the organizational capacity assessment tool, ICF is pilot testing the tool with potential respondents. The pilot test will examine the construction and respondent burden of the assessment tool. Your participation and responses will help CNCS and ICF ensure the effectiveness of the assessment tool for determining the capacity strengths and areas for potential capacity building of organizations like yours.

Respondent’s Role

Please complete the attached assessment tool and email to <email> by <Date>. Any information we collect will be protected in accordance with ICF’s security procedures and management policies. Your participation is voluntary and confidential. We will not identify you or your organization in any reports by name or any other personal identifiers in any analysis or reports resulting from this pilot test. Your individual answers will not be seen by and/or used in any grant making decisions by CNCS. However, we will share the aggregate results with CNCS.

Questions

Should you have any questions about the pilot test, you may contact Nanette Antwi-Donkor, [email protected]

Consent Statement

• I understand my participation in this pilot test is voluntary and that I can stop responding to the tool at any time.

• I understand that my answers to these questions will only be seen by the ICF team.

• I understand that my name will not be included in any reports or presentations of findings and that my data will be securely handled.

• I understand that the risks and benefits of participating in this study are minimal.

• I agree to participate in this pilot test.

Name (Please Print):
Signature:
Date:

18