Attachment J2. NHSPS and CAN Pilot Test Results
J2. NHSPS and CAN Pilot Test Results.docx
National Healthy Start Evaluation and Quality Assurance
Attachment J2. NHSPS and CAN Pilot Test Results
OMB: 0915-0338
Pilot test Results
National Healthy Start Program Survey
Healthy Start Community Action Network (CAN) Survey
II.
National Healthy Start Program Survey:
Pretest Report and
Recommendations
A. Overview of National Healthy Start Program Survey
The National Healthy Start Program Survey (NHSPS) will collect data from Healthy Start grantees to be used for the multilevel implementation and network components of the Healthy Start evaluation. The survey is designed to promote consistent collection of information about implementation, systems collaboration and coordination, and activities related to quality improvement. Project directors and other Healthy Start staff will complete the survey. Healthy Start grantees will complete the survey three times—once as a baseline at the start of the grant, once in the middle of the grant, and finally at the end of the grant.
Pretest purposes. By pretesting the NHSPS, we hoped to gain information on the average time it takes to complete the survey; grantees’ understanding of the survey questions and ability to provide empirical responses; and any questions that could be deleted or revised to improve clarity.
Pretest sample. MCHB provided contact information for project directors at four Healthy Start grantee sites. Two grantees completed the survey, one grantee partially completed the survey, and one grantee was unable to participate in the pretest due to competing priorities during the pretest period.
Adapting the survey for the pretest. We modified the survey in the following ways to fit the constraints of the pretest:
Mode. The NHSPS is designed as a web survey with skip patterns and fill text programmed into the instrument. Due to the small scale of the pretest and the high costs associated with programming a survey, we administered the survey to grantees as a paper-and-pencil survey.
Changing text from CAN to consortium. For the network questions in Section 5 regarding grantees’ collaborative efforts, we changed the wording from Community Action Network to consortium for the purpose of the pretest so that the wording reflected the current grantee terminology.
Administering Question 5.9 in advance. Given that we were administering the survey as a hard copy instrument, we could not rely on a computer to fill the items for us. As a result, we asked question 5.9 (equivalent to C1 in the Healthy Start Community Action Network Survey) in advance of other survey items and used the responses to 5.9 to fill 5.10, 5.11, 5.12, 5.14 and 5.15. This enabled us to get a better sense of respondents’ comprehension of the subsequent items.
5.9. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal. From the list of CAN members below, please select up to 10 organizations with which your Healthy Start project collaborated during the past 12 months. If your Healthy Start project collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. PROGRAMMER: INSERT DROP DOWN FIELDS LISTING ALL CAN MEMBERS NEXT TO CAN MEMBER 1–10 
▼ CAN Member 1
▼ CAN Member 2
▼ CAN Member 3
▼ CAN Member 4
▼ CAN Member 5
▼ CAN Member 6
▼ CAN Member 7
▼ CAN Member 8
▼ CAN Member 9
▼ CAN Member 10 |
Conducting the pretest. Grantees received electronic copies of the instrument via email in early January 2014. They printed the survey and responded to the items using pen. We asked pretest participants to record their start and finish time for each section on a time sheet, including any start and stop times for breaks within sessions. Grantees were instructed to mark confusing and/or problematic items with a star while they completed the survey and then to refer back to these items to add a description of the issue after they completed the section and recorded the total time for the section. Each of the three grantees returned the survey through different means, including scanning/ emailing, faxing, and regular mail. One grantee only submitted Sections 1 through 6 and was unable to provide insight into the length of time spent completing these sections.
Implications of pretest timing. One of the goals of the pretest was to assess the length of the survey. We asked programs to self-report their time by survey section using a timing sheet that we had compiled for the purpose of the pretest. Of the three programs that returned the survey, one took 144 minutes to complete the survey, one took 234 minutes, and the program that partially completed the survey did not provide time estimates. A number of factors likely influenced the amount of time it took program staff to complete the survey, including accessibility of statistics on outcomes and participation, knowledge of their program (which could vary based on the type of staff/ individual completing a given section), and number of distractions while completing the survey (for example, whether the respondent was multi-tasking or was fully focused on the survey). Additionally, we need to account for the pretest survey mode being paper and pencil instead of web based. This adds time to the survey because respondents need to pay attention to notes about skip patterns rather than automatically being routed to the correct questions as they would be for the web-based survey. Regardless, pretest results suggest that we need to decrease survey length by roughly 25 percent to get the total length down to an average of 120 minutes.
B. Recommended Changes to the National Healthy Start Program Survey
Based on feedback from the three pretest participants, we suggest making the modifications listed below to decrease the amount of time it takes grantees to complete the survey and to improve question clarity.
Revise Response Options for Questions About Outcomes and Participation to Be Multiple Choice Instead of Open Ended
The two Healthy Start grantees that participated in the pretest and reported timing information noted that Sections 2 and 4 took longer to complete than other sections. These sections contain a series of questions that ask about the total number of participants receiving a particular type of service, screening positive for a subset of conditions, or engaging in certain behaviors. For items about health conditions and health behaviors, one pretest participant noted that it was arduous to look up these statistics and suggested we request a percentage instead of a total. For questions about total number of participants receiving a given service (for example, socio-emotional screenings for children or parenting support groups), grantees tended to write in the total number of participants, suggesting 100 percent participation rate. Although it is possible that both pretest participants have 100 percent participation rate for all of the services of interest, it is more likely that this was simply an estimate rather than a number they retrieved from program data.
In attempt to reduce the burden on grantees and ensure that we collect useful data, we suggest revising these questions such that we ask grantees to select a percentage range (0 percent, 1–24 percent, 25–49 percent, 50–74 percent, 75–99 percent, 100 percent) rather than provide a raw number of participants. Below are a few examples:
Examples of Proposed Revisions to Response Options
Original Question Text |
Revised Question Text |
||||||
2.2 In [GRANT YEAR], how many participating women received a comprehensive needs/risk assessment? Your best estimate is fine.
NO RESPONSE M |
2.2 In [GRANT YEAR], what percentage of participating women received a comprehensive needs/risk assessment? Select one only 0% 1-24% 25-49% 50-74% 75-99% 100% None NO RESPONSE M |
We suggest implementing this change for the following questions:
2.2: Participants receiving comprehensive assessment
2.23a-d: Participants with Medicaid, free care, private insurance and other insurance
2.37: Participants with reproductive life plans
2.39: Participants with primary care provider
2.41a-d: Participants use of and attempts to quit using tobacco
2.42a-c: Participants abuse/ use of alcohol and other substances and attempts to quit
2.44a-c: Participants breastfeeding at discharge, at six months (partially or exclusively), and at six months (exclusively)
2.45a-c: Participants screening positive for HIV, Chlamydia, and STDs
4.3: Participants (children) receiving socio-emotional screening
4.19: Male participants who are partners/fathers
4.32: Participants who attended support groups
4.33: Participants who received one-on-one parenting services
For 2.43a-c: Participants who are overweight, obese, and underweight, we suggest changing to asking for percentages of overweight, obese, underweight, and normal weight. This is recommended in this case as overweight, obese, underweight, and normal weight are mutually exclusive and should total to 100 percent.
C. Deletions
In addition to minimizing burden on grantees by modifying the type of data we collect, we also recommend deleting items from the survey. We used the pretest to identify potential items for deletion. Specifically, we paid particular attention to items that respondents seemed to just circle “yes” for all of the options in the series; items that respondents noted were very confusing; and items that respondents perceived as duplicates or questions that we had already asked earlier in the survey. Additionally, we identified a few items that did not create confusion for respondents or yield questionable data, but they were not essential, so they seem like good candidates for deletion given the time constraints of the survey. Below is a list of deleted items:
1.2, 1.2a: Local name of Healthy Start program
1.17: Retention strategy by life-course stage. We deleted 1.17 and changed the text of 1.16 so that we ask more generally about retention strategies for all participants.
2.4: Whether or not project assigns risk categories. We will get this from the site visits.
2.5: Description of risk categories. This was an open-ended question that yielded a lot of text from respondents (and likely took more time to respond). Given that we will get this information from the site visits and need to make cuts, we suggest deleting it from the survey.
2.6: Risk factors that program screens for by life-course stage. We do not suggest deleting this item in full but rather, deleting the “Other Specify” text where respondents can enter additional risk factors for which they screen participants. Text entered at 2.6 carries over to other items in the screening series so it will save time at this item and in future items.
2.7: Percent of participants screened by risk factor for each life-course stage. Pretest respondents wrote in “all participants” for each risk factor and indicated that this was their best estimate. Given that the question was difficult for pretest respondents to answer and does not seem to yield useful data, we suggest deleting it.
2.10: Whether or not program offers different case management models by risk level. We will get this from the site visits and focus groups.
2.11: Description of levels of case management. Like 2.5, this is an open-ended response that potentially takes a lot of time to answer. We will get this information from the site visits to reduce burden during the survey.
2.14: Average case load and range of cases. We suggest deleting range of cases per case manager and ask only for average case load because the question about range seemed to confuse pretest respondents.
2.30, 2.34: Activities covered under formal partnerships (2.30) and informal partnerships (2.34) with certified application counselor (CAC) organizations. Pretest respondents indicated confusion and wrote “yes” for all activities. We will get similar information from the site visits.
2.6 b and c: Number of formal partnerships with primary care providers broken down by memorandum of understanding (MOU) and contract. Rather than ask about number of MOUs and contracts, we suggest simply asking about the total number of formal partnerships.
3.24: Age of participants served through home visiting. We collect more information about home visiting and who it serves later in the survey and during the site visits.
3.36 (Parts a, c, d, I, o, p, r, s, t, u, v, w, y, z, aa, bb, cc, dd): Health education topics addressed. Pretest participants noted that the list of health education topics is very lengthy and they address most of them in some way. We suggest deleting the health education topics for which we collect data through the Healthy Start Participant Form and retain the topics we do not inquire about on the form. These deletions cut the question by more than 50 percent.
3.38: Collaboration with community organizations that offer “quit lines” for smoking cessation. We collect more information about the nature of Healthy Start and partner smoking cessation services in the following questions.
3.43: Topics covered in group tobacco-cessation counseling. We suggest rolling this in to item 3.42 (topics covered in one-on-one tobacco cessation counseling) and simply asking about topics covered in tobacco cessation counseling.
3.46: Tobacco-use cessation opportunities for other (non-partner) family members. We suggest rolling this into 3.45 (tobacco-use cessation opportunities for partners) and asking about opportunities for partners and other family members in one question.
3.50: Drug/alcohol counseling opportunities for other (non-partner) family members. We suggest rolling this into 3.49 (opportunities for partners), as we did for the tobacco cessation question 3.46.
3.52: Healthy weight services provided through referral. We suggest deleting this item and, in 3.51, ask about healthy weight services provided on-site and through referral.
3.54: Healthy Start refers to nutritionists at partner agencies. We suggest combining this with 3.53 (nutritionist on-site) and asking in general whether Healthy Start or partner agencies employ a nutritionist.
4.25: Health education topics covered with male participants. Both pretest participants selected “yes” for all items in this list. We suggest dropping the item because it is difficult and time consuming and seems to encourage respondents to simply check off “yes.”
4.31: Duration of parenting education. Parenting education services are most likely part of other services offered that we already ask about in the survey. We suggest deleting this item.
5.3: Does the CAN have a smaller leading body? We will be able to get information about the composition and structure of the CAN from grantee reports and the site visit. We suggest dropping this item.
5.4: How many individuals make up the smaller leading body? We will be able to get information about the composition and structure of the CAN from grantee reports and the site visit. We suggest dropping this item.
5.7: Main areas of CAN activity. We ask a similar question at 5.16a. We suggest deleting this item.
5.22, 5.23, 5.24: Change in community since beginning of funding cycle. This is a very subjective question. We will use secondary data on the community if available and qualitative data collection activities to capture this in the evaluation.
5.30: Overall impact of Healthy Start on community. This is another subjective question that will be better answered through qualitative data collection efforts.
6.26: University-affiliated evaluator (if external). This is not an essential question and we can capture this through grantee reports.
Section 7: Healthy Start Achievements (with the exception of 7.4, 7.5 and 7.7). This section contains subjective questions about the respondent’s perception of Healthy Start’s impact, including multi-part items about each specific Healthy Start goal that is achieved. Given that we will capture this through administrative data analysis during the evaluation, we suggest dropping these burdensome questions from the survey.
D. Minor Revisions Based on Pretest Feedback
Minor revisions were also identified during the pretest based on feedback from respondents.
Adding “During [GRANT YEAR]” to certain items that do not specify time period. Pretest respondents pointed out that certain questionnaire items do not specify whether we are asking about the present (current grant year) or during the previous grant year. We suggest adding clarification text to these ambiguous items to make it clear we are talking about the grant year of interest (for example, in 3.21).
Adding Intimate Partner Violence and Immigration to types of referrals (1.11) and topics addressed through class and counseling (3.36). Pretest respondents added a few items to the “other-specify” categories, including immigration and intimate partner violence. We imagine that numerous grantees might write in these topics and suggest adding them to the survey.
Revise percentage ranges to match other items with percentage ranges (3.9, 3.13, 3.16). Three of the items in the survey already use percentage ranges as response options, however the percentage ranges do not align with the new response options. We suggest revising them to match the following:
Select one only
100%
75 – 99%
50 – 74%
25 – 49%
1 – 24%
0%
NO RESPONSE M
Add clarifying text about the Certified Application Counselor (CAC) organizations (2.28). During the pretest, organizations that are CAC organizations thought the CAC questions were not applicable to their project. We added in text to clarify the question.
Proposed Revisions to 2.28
Original Question Text |
Revised Question Text |
2.28. A formal partnership can be defined as a written agreement (usually involving a subcontract or memorandum of understanding [MOU]) with providers to provide care to Healthy Start participants. A subcontract is a legally binding document with an organization that states that the organization will provide services for Healthy Start. An MOU is a written agreement between entities that formalizes a relationship, but it is not legally binding as a contract. Does your project have a formal partnership with any certified application counselor (CAC) organizations in the community? Yes 1 No 0 NO RESPONSE M |
2.28. A formal partnership can be defined as a written agreement (usually involving a subcontract or memorandum of understanding [MOU]) with providers to offer care to Healthy Start participants. A subcontract is a legally binding document with an organization stating that the organization will provide services for Healthy Start. An MOU is a written agreement between entities that formalizes a relationship, but it is not legally binding like a contract. IF 2.27 = 1(ORG IS CAC ORG): Even if your organization is a certified application counselor (CAC) organization, does your project have a formal partnership with any certified application counselor (CAC) organizations in the community? IF 2.26 = 0 or 2.27 = 0 (ORG IS NOT CAC ORG): Does your project have a formal partnership with any certified application counselor (CAC) organizations in the community? Yes 1 No 0 NO RESPONSE M |
Add response option “Healthy Start staff make appointments for participants” (3.17—assistance offered to participants to complete primary care referrals). Pretest participants noted that one of the main ways they assist participants with their referrals to primary care participants is to make the appointment for the participant. Given that both pretest participants noted this, we suggest adding it as a response option.
Add field for pregnant women in perinatal depression screening question (4.1). Pretest participants asked why we do not inquire about the number of pregnant women receiving perinatal depression screenings. We suggest revising the question to include both pregnant and post-partum participants.
Proposed Revisions to 4.1
Original Question Text |
Revised Question Text |
||||||||||||||||||||||||||||||||
4.1. During [GRANT YEAR], how many participating women with a live birth received a perinatal depression screening on site or at a partner site? Your best estimate is fine. ________ NUMBER OF PARTICIPATING WOMEN RECEIVING PERINATAL DEPRESSION SCREENING ________ TOTAL NUMBER OF PARTICIPANTS WITH A LIVE BIRTH |
4.1. During [GRANT YEAR], what percentage of participating pregnant and postpartum/interconceptional women received a perinatal depression screening on site or at a partner site? Your best estimate is fine. (NHSPS 2.24 modified)
|
||||||||||||||||||||||||||||||||
Other revisions and clarifications. We made a number of non-substantive revisions to grammar and to make questions more consistent with other items in the same section. Examples include modifying the multiple-choice response options in 3.27 (On average, what is the duration of a scheduled home visit?) to account for different lengths of time. Previously, the response options did not include a choice for 30–45 minutes. Another example is item 3.22 (Are home visits conducted by Healthy Start staff, contract staff, or both?). During the pretest, the question read “Are these conducted by Healthy Start staff, contract staff, or both”, as a follow-up to a question about whether they provide home visits in general.
E. Conclusion
In summary, the pretest provided important feedback about the clarity, flow, and timing of the questions on the NHSPS. We suggest the following:
Deleting 35 questions out of 238 that were included in the pretest version of the instrument, including some of the lengthiest items with multiple sub-sections (for example. 2.6, 2.7, 3.36, and 4.5) and open-ended questions asking respondents to provide short response answers (for example 2.5, 2.7)
Asking respondents to select percentage ranges instead of filling in total number of participants in open-ended fields for 13 of the most time-consuming questions
Improving question clarity by revising question text
Implementing these suggestions would substantially reduce the amount of time grantees take to complete the survey. It would also make the survey questions clearer to respondents and the response options to multiple-choice items more robust.
III. Healthy Start Community Action Network (CAN) Survey: Pretest Report and Recommendations
A. Overview of Healthy Start Community Action Network (CAN Survey) Pretest
Funded by the Health Resources and Services Administration’s (HRSA), Maternal and Child Health Bureau (MCHB), Healthy Start aims to reduce disparities in infant mortality and improve perinatal outcomes in the United States. Under its next funding cycle, beginning in September 2014, the program will transform its framework from nine service and systems core components to the following approaches:
Improving women’s health
Promoting quality services
Strengthening family resilience
Achieving collective impact
Increasing accountability through quality improvement, performance monitoring, and evaluation
MCHB requires an accompanying evaluation to describe the program’s implementation; provide insights into its characteristics, system features, and activities that affect outcomes; and assess the program’s overall success. One of the new components of the evaluation is a network study that will measure the level of collaboration among the Healthy Start grantees and other local health and social sector programs. The Healthy Start Community Action Network (CAN) Survey is designed to collect data needed for the network study and provide important information on the extent to which Healthy Start and other community organizations are working together to achieve common goals.
Pretest purposes. From the respondent perspective, we wanted to determine whether the question language was clear and understandable, the instrument flow was smooth and made sense, and the questions were not too difficult to answer. For multiple-choice items, we wanted to confirm that the answer choices reflected the main ideas that respondents wanted to convey. Finally, we wanted to test the average length of the survey.
Pretest sample. To identify respondents for the CAN survey pretest, we contacted four Healthy Start program grantees and asked them to provide a list of all organizations that participate in their Healthy Start consortium, and the contact information for two or three consortium members (excluding Healthy Start clients) who would be willing to assist with the pretest effort. Specifically, we requested that each grantee provide contact information for at least one health care provider. Two of the four grantees provided contact information for respondents that completed the survey. By the end of the pretest, we had tested the instrument with a total of five individuals: two health care providers, two individuals working at community organizations and one individual community member. The community member was not a Healthy Start client and did not work for or represent a community organization. This posed some problems for this respondent regarding the applicability of the questions, which we discuss in the next section under “Defining the Healthy Start CAN Survey Sample.” Our pretest sample outcomes are shown in Table A.3.a.
Table A.3.a. Healthy Start CAN Survey Participants
Pretest Sample |
Completed Survey by Participant Type |
Health care provider |
2 |
Other community service provider |
2 |
Community member involved in the consortium (not a Healthy Start client) |
1 |
Adapting the survey for the pretest. A few modifications were made to the survey to fit the constraints of the pretest.
Mode. The survey is designed to be a web survey but we administered it as a paper instrument. We removed programming text to make it more visually appealing as a paper survey.
Administering C1 separately. Given that we were administering the survey as a hard copy instrument, we could not rely on a computer to fill the items for us. As a result, we asked C1 in advance of other survey items and used the responses to C1 to fill C2, C3, C4, D3 and D4. This enabled us to get a better sense of respondents’ comprehension of the subsequent items.
C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal.
From the list of CAN members below, please select up to 10 organizations with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New)
Select 10 organizations.
[FILL CAN MEMBER #1] 1
[FILL CAN MEMBER #2] 2
[FILL CAN MEMBER #3] 3
[FILL CAN MEMBER #4] 4
[FILL CAN MEMBER #5] 5
[FILL CAN MEMBER #6] 6
[FILL CAN MEMBER #7] 7
[FILL CAN MEMBER #8] 8
[FILL CAN MEMBER #9] 9
FILL CAN MEMBER #10] 10
Changing text from CAN to consortium. The 2014 Funding Opportunity Announcement for Healthy Start requires that grant applicants describe plans for developing a cross-sector CAN. According to the FOA, the CAN is intended to “increase trust among community partners/ members, assess and “map” the community using data, encourage effective and equitable allocation of limited resources, ensure that the contributions of community partners/members are valued and respected, and use varied communication modalities and technologies to provide community partners/members with full and timely access to information.” The CAN is similar to the Healthy Start consortium developed under previous rounds of funding, which requires grantees to develop a group of community members, clients, medical providers, social service agencies, and members of the faith and business communities. Given the similarities between the CAN and the consortium, we decided to modify the wording in the pretest version of the Healthy Start CAN Survey to refer to “the consortium” instead of “the community action network (CAN)” to ensure that respondents understood the terminology in the questions.
Conducting the pretest. Five individuals completed the survey. The surveys were distributed to participants between December 19, 2013 and January 6, 2014 and were returned between December 31, 2014 and January 15, 2014. All surveys were formatted as a paper-and-pencil survey and were distributed to respondents via email as a PDF. All questions requiring pre-filled text (for example, [ORG NAME] or [Healthy Start Grantee]) were populated prior to being sent to the respondents. Respondents could choose to return the survey through one of three options: 1) scanning and emailing the completed paper survey, 2) faxing the completed paper survey, or 3) returning the completed paper survey by mail. We reviewed the surveys as they were returned to us. After completion of the pretest, we held a final debriefing and identified the minor changes outlined below.
Implications of pretest timing. One of the goals of the pretest was to assess the length of the questionnaire. We instructed respondents to self-report how long it took them to complete the survey by noting the time they started and stopped working on the questionnaire, including start and stop times for any breaks. The self-reported survey lengths ranged from 23 minutes to 85 minutes, with an average of 55.8 minutes per complete. Because the times were self-reported, we have no way of ascertaining the accuracy of the reported times with precision. We have reason to believe that on average, the reported durations overestimated the length of the survey—for example, all but one of the respondents indicated that they completed the survey while at work, and potentially multi-tasking. Differences resulting from the pretest environment also impacted the length of the survey. It takes more time to complete the survey as a paper instrument because the respondent had to navigate the logistics of skips, a process normally handled by the web instrument.
Based on these factors, we estimate that the pretest conditions added between 5 and 30 minutes to the reported pretest survey length for any given respondent. In the next section, we recommend making a few deletions that would likely reduce the total survey time by 5 minutes. Taking all of this into account, we predict that the survey takes between 30–45 minutes to complete with the recommended deletions.
Interestingly, completion time did not appear to be associated with the number of organizations selected at C1—the question that asks respondents to choose up to 10 organizations part of the CAN with whom they collaborate with the most. We would expect that respondents selecting more organizations would take longer because the subsequent network questions ask about the respondents’ collaboration with each organization selected in C1. However, the number of organizations did not appear to have an impact on length.
B. Recommended Changes to the Healthy Start Community Action Network (CAN) Survey
1. Defining the Healthy Start CAN Survey Sample
The Healthy Start CAN Survey asks respondents how their organization interacts with Healthy Start, the CAN and the community as the purpose of the network study to assess organizational ties and networks in the community. During the pretest, most of the respondents did not have problems responding on behalf of their organization, with the exception of the one respondent who was an individual community member who did not represent an organization on the consortium. Many of the questions were irrelevant to this respondent as questions were framed to be from an organizational perspective (that is, “When did [FILL ORG NAME] start collaborating with Healthy Start?” or “During the past 12 months, in which of the following ways did [FILL ORG NAME] collaborate with [Healthy Start Grantee] and the other CAN members regarding services for women, children, and their families?”).
As a result, we recommend defining the sample such that it only includes CAN members who represent an organization that is formally or informally partnered with Healthy Start. Given that the purpose of the survey is to assess organizational networks in the community, this modification will help align the sample with the goals of the survey.
2. Addition of “Don’t Know” Response in Section A and B (A3a, A3b, A3c, A4, B1, B2 and B3)
Sections A and B include open-ended questions about organizational background and history working with Healthy Start. While most respondents did not have trouble answering these questions, there were a few instances (one at A3b, one at A3c, one at B2) where respondents wrote in “I don’t know.” We decided to add the text, “Your best estimate is fine,” to encourage respondents to provide input, but added a “don’t know” response option for cases where they truly do not know the answer. This will help us distinguish between “don’t know” responses from missing data.
Proposed Revisions to the Open-Ended Organization Background (Section A) and Collaborative History (Section B) Questions
Revised Question Text |
||||||||||||
A3a. How many clients did [ORG NAME] serve in the past year? Your best estimate is fine. Total: ______________________ Don’t know A3b. How many of the total served in the past year are women of reproductive age (15–44 years old)? Your best estimate is fine. If your organization does not serve women, enter 0. Women (reproductive age): ______________________ Don’t know A3c. How many of the total served in the past year are children under the age of 2? Your best estimate is fine. If your organization does not serve children under the age of 2, enter 0. Children:____________________ Don’t know
A4. How many staff members are employed by [ORG NAME]? Include staff members who interact directly with clients and administrative staff. Your best estimate is fine.
B1. When did you first learn about [Healthy Start Grantee]? Your best estimate is fine. ____/____ (MM/YYYY) Don’t know B2. Collaborating can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which [ORG NAME] and [Healthy Start Grantee] worked toward a common goal. When did [ORG NAME] begin collaborating with [Healthy Start Grantee]? Your best estimate is fine. (New) ____/____ (MM/YYYY) Don’t know B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. Please include a date after June 2014 to reflect the newest cycle of the Healthy Start project. If your organizations participated in the CAN prior to June 2014, please enter 06/2014. (New) ____/____ (MM/YYYY) Don’t know
|
3. Deletion of A6 (Organization’s Budget)
Respondents are asked about their organization’s annual budget in Item A6.
A6. What is [ORG NAME]’s annual budget?
Your best estimate is fine.
$______________________
We found that only one out of five respondents answered the question with confidence—three respondents were unable to answer this item and one respondent answered the question but indicated lack of confidence in the response provided. Given our doubts that the question will yield useful data, we recommend deleting the question to reduce overall length of the survey and burden on the respondent.
4. B3 Clarification
In Item B3, respondents are asked about the length of participation in the CAN. For the pretest, we revised the wording to ask how long the respondents’ organizations have participated in the consortium instead of asking about the CAN. All but one respondent were able to identify the month and year that their organization began participating. Although respondents seem to understand the text during the pretest, we are concerned that the question might be interpreted differently once we change the text back to “CAN.. Specifically, for Healthy Start grantees that are funded again under the new funding opportunity and simply convert the consortium to a CAN, we are unsure whether CAN survey respondents would be able to distinguish activities between the two funding cycles (and thus provide the date that they began participating in the consortium). In order to eliminate potential confusion, we recommend adding the instructions below. This additional text would be accompanied with a range check that would generate an error message if the respondent tried to enter a date prior to 09/2014.
Proposed Revisions to B3
Original Question Text |
Revised Question Text |
B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. ____/____ (MM/YYYY) |
B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. Please include a date on or after June 2014 to reflect the your participation in the CAN under the newest cycle of the Healthy Start project. ____/____ (MM/YYYY) Don’t know |
5. C1 Clarification
Respondents found the wording of item C1 ambiguous. In this question, respondents are asked to select up to 10 organizations with which their organization collaborates. Respondents who work for very large organizations with multiple departments (like a County Health Department) said they were not sure which organizations on the list worked most closely with their organization. In attempt to reduce ambiguity, we added a note advising respondents to only take into consideration their knowledge of their organization.
Proposed Revisions to C1
Original Question Text |
Revised Question Text |
C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal. From the list of CAN members below, please select up to 10 organizations with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New) Select 10 organizations. [FILL FROM LIST PROVIDED BY HEALTH START] [FILL CAN MEMBER #1] [FILL CAN MEMBER #2] [FILL CAN MEMBER #3] [FILL CAN MEMBER #4] [FILL CAN MEMBER #5] [FILL CAN MEMBER #6] [FILL CAN MEMBER #7] [FILL CAN MEMBER #8] [FILL CAN MEMBER #9] [FILL CAN MEMBER #10] [FILL CAN MEMBER #n] |
C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal. Based on your knowledge of your organization, please select up to 10 organizations from the list of CAN members below with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New) Select 10 organizations. [FILL FROM LIST PROVIDED BY HEALTH START] [FILL CAN MEMBER #1] [FILL CAN MEMBER #2] [FILL CAN MEMBER #3] [FILL CAN MEMBER #4] [FILL CAN MEMBER #5] [FILL CAN MEMBER #6] [FILL CAN MEMBER #7] [FILL CAN MEMBER #8] [FILL CAN MEMBER #9] [FILL CAN MEMBER #10] [FILL CAN MEMBER #n] |
6. Deletion of “Add-In” Responses in Section C (C2, C3, C4)
Items C2, C3 and C4 currently allow respondents to list names of organizations not on CAN with whom they partner. Below is the unrevised wording of C2:
C2. During the past 12 months, in which of the following ways did [ORG NAME] formally and/or informally partner with [Healthy Start Grantee] and the other CAN members?
If [ORG NAME] was involved in these activities with additional agencies in the community who are not CAN members, please list the names of those organizations in the spaces provided at the end and indicate which ways your organization partnered with them.
|
SELECT ALL THAT APPLY |
||||||||||||
Community Action Network Members |
Signed formal memorandum of understanding with organization |
Met with organization for joint planning outside of CAN meetings |
Participated in collaborative group or working group with organization in addition to the CAN |
Submitted joint grant proposal |
|||||||||
|
[LIST OF CAN MEMBERS FROM C1] |
|
|
|
|
|
|
|
|
|
|
|
|
|
[Healthy Start Grantee] |
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
Other community agencies (Please specify) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Of the five respondents, only one of the respondents used the additional lines to enter other organization names and the respondent did this only for item C2 because she found it to be burdensome to fill in the other organizations for other questions. Another respondent was confused by the additional lines. A third respondent mentioned that the organizations in C1 were fairly comprehensive and did not feel the need to use the additional space. Given respondents’ feedback, we recommend dropping the open-ended spaces where respondents can list additional organizations, as very few respondents would likely use these lines. In addition, deleting the item will decrease the length of the survey and the burden on the respondent.
Revised Question Text |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
C2. During the past 12 months, in which of the following ways did [ORG NAME] formally and/or informally partner with [Healthy Start Grantee] and the other CAN members
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
C3. During the past 12 months, in which of the following ways did [ORG NAME] collaborate with [Healthy Start Grantee] and the other CAN members regarding services for women, children, and their families?
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
C4. During the past 12 months, which of the following activities did [ORG NAME] engage in with [Healthy Start Grantee] and the other CAN members?
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
C. Conclusion
In summary, the pretest provided important feedback about the clarity of the Healthy Start CAN Survey questions, which is essential to accurately measuring the size and strength of the organizational networks in the Healthy Start community. The pretest also suggests that the interview length was within a reasonable range for this type of survey. Based on the pretest, we recommend several changes to the instrument that will improve the questionnaire clarity and brevity and eliminate potential sources of ambiguity in questionnaire wording.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Katie Morrison |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |
© 2024 OMB.report | Privacy Policy