Appendix H: HBCC-NSAC Toolkit English version pre-test summary

Appendix H

HBCC-NSAC Toolkit English version pre-test summary

A. Introduction

In July 2022, the study team conducted a pre-test of the English version of the home-based child care (HBCC) provider questionnaire in the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) as part of the Home-Based Child Care Supply and Quality (HBCCSQ) project. The goals of this pre-test were to: 1) ensure providers with different characteristics interpret items in the provider questionnaire in the same way, 2) test how long it took providers to complete the provider questionnaire, and 3) collect information about provider specific practices that could be used as examples in the provider questionnaire items.

This appendix describes the respondents and procedures for conducting the pre-test and summarizes recommendations for refining specific items and subsequent updates.

B. Respondent recruitment

The study team worked with staff from five community organizations to identify and recruit nine HBCC providers to participate in the pre-test. These organizations shared contact information for potential participants with recruiters who contacted them using a recruitment script. The study team attempted to recruit HBCC providers who were a mix of: licensed and license-exempt, in both urban and rural areas, from different racial and ethnic backgrounds, and Spanish-English bilingual. The study team aimed to have three providers with a given characteristic.

The study team encountered a few challenges recruiting providers across all of the target characteristics. Exhibit 1 describes each challenge and lessons learned that the study team will apply to the pilot study recruitment approach. Exhibit 1 references relevant attachments from the request for Office of Management and Budget approval under the umbrella generic, Pre-testing of Evaluation Data Collection Activities.

Exhibit 1. Pre-test recruitment challenges and lessons learned for the pilot study

Pre-test recruitment challenge	Lessons learned for pilot study recruitment
Most community organizations said it would likely be too difficult or time consuming for their Spanish-speaking providers to fill out the English version of the provider questionnaire. We ultimately recruited one Hispanic or Latino provider who is Spanish-English bilingual and one Black provider who is Spanish-English bilingual.	The study team will return to these community organizations during the pilot study to recruit primarily Spanish-speaking providers to complete the Spanish instruments. Recruiters will be trained to understand that bilingual providers may be comfortable reading and answering questions in one language more than the other and to confirm providers’ preferred language for data collection during the recruitment calls (Instruments 1 and 2).
Since the initial four community organizations did not recommend any rural providers, the study team added a fifth organization to specifically recruit providers who live in rural communities. We ultimately recruited three rural providers.	During site selection, the study team will consider the types of providers community organizations primarily work with including the geographic areas they serve in order to purposively include one or more with rural providers. During the initial calls with community organizations (Appendix A), the study team will ask about the characteristics of providers they work with, including whether they work with providers in urban or rural areas.
None of the community organizations were able to recommend providers who are Asian or Pacific Islander for the pre-test.	During site selection, the study team will consider the types of providers community organizations primarily work with including racial or ethnic groups in order to purposively include one or more with Asian or Pacific Islander providers. During the initial calls with community organizations (Appendix A), the study team will ask about the characteristics of providers they work with, including whether they work with any particular racial or ethnic groups.

Exhibit 2 shows the pre-test respondents by these characteristics.

Exhibit 2. July 2022 English version pre-test respondent characteristics

Data collection	Urban		Rural
	English-speaking providers	Spanish-English bilingual providers	English-speaking providers	Spanish-English bilingual providers
Black, non-Hispanic	3a	1	0	0
Hispanic or Latino	0	1	0	0
White, non-Hispanic	1	0	3	0
Asian or Pacific Islander, non-Hispanic	0	0	0	0

^a Two of the four respondents in this category were Family, Friend, and Neighbor (FFN) providers. All other providers were Family Child Care (FCC) providers.

C. Pre-test procedures

Respondents had the option to receive an electronic, Word version of the provider questionnaire via email, or paper copies through the mail. Respondents were asked to complete the English version of the provider questionnaire, track how long it took them, and return it to the study team a few days before a cognitive interview.

Two study team members, with one serving as lead interviewer and the second as note taker, conducted 30-minute cognitive interviews via a virtual meeting platform with each pre-test respondent. The purpose of the cognitive interviews was to identify questions that were confusing or difficult for respondents to answer and get respondents’ recommendations for changes. With respondents’ consent, the study team recorded the interviews. The team members used concurrent probing techniques to assess item-specific issues and then used a retrospective approach for more general discussion (for example, asking if there were any questions that were difficult to respond to). During the interview, respondents were asked to have a copy of the provider questionnaire in front of them to refer to specific items and interviewers read items aloud as needed.

D. Pre-test findings and subsequent updates

Based on issues identified in the pre-test, the study team made some item-level changes to the provider questionnaire. Exhibit 3 describes overall changes made to the provider questionnaire instructions, format, or scales. Exhibit 4 describes item-level changes made to the provider questionnaire items, organized by domain.

Exhibit 3. Overall feedback and subsequent changes

Topic	Feedback	Changes made
Instructions	One respondent said that she did not think the HBCC-NSAC Toolkit was for FFN respondents, as it seemed more geared toward center-based care.	Added more inclusive language to the instructions specifying who the HBCC-NSAC Toolkit is for.
Instructions	One respondent explained that she was not sure how to answer some of the statements because there were things she might do a lot with some kids, but not as much with others.	Added specific language to instructions for how to handle this.
Format	At least two respondents explicitly stated that they would like to see a section for next steps and resources to make the HBCC-NSAC Toolkit actionable and to warrant taking the time to fill it out. Similarly, another respondent said she had to shift her mindset as she was completing the provider questionnaire. In the past, when she has seen a rating scale, there is a tendency to want to rate yourself high, but she realized that in order to truly use this as a “tool”, she needed to be reflective and answer honestly.	Added a “What Do I Do Now?” section to the end with more specific instructions for next steps. This section also includes recommended reflection questions to guide the provider in creating a professional development plan based on their responses. The reflection questions will prompt providers to think about this as a tool for self-reflection rather than a rating scale or monitoring tool.
Scale	The consensus among FCC respondents was that: “a lot” = multiple times/day, “rarely” = I have done it before but not often, and “sometimes” = somewhere in between. However, one FFN respondent commented that she resorted to the “not sure how to answer” option because she wasn’t sure how to define “a lot.” She wasn’t sure whether to quantify it as “every day” versus “every time it comes up” (which might not necessarily be every day).	Added more in-depth explanations of the response scales to the instructions.

Exhibit 4. Item-level reflection statement feedback and subsequent changes, by domain

Item	Feedback	Changes made
Support for Social Development
I.57	This item asks about the provider’s community. One respondent was unsure how “community” was defined and wanted clarification between their HBCC community and the outside, surrounding community.	Changed “community” to “wider community” when applicable, so that respondents understand we are not asking about the community within their HBCC and added examples.
I.63	A few respondents mentioned that it is difficult to find “positive news stories” or discuss news/current events with young children in a way that makes sense to the children.	Dropped this item, but incorporated news stories as an example in another item related to supporting a positive self-identity.
I.64-I.65	One respondent stated that they don’t necessarily talk about “famous” people from the same or different racial and ethnic backgrounds, but they do talk about everyday people in the children’s lives.	Removed “famous” from the statements, so the item is also inclusive of everyday people. The purpose of the item is for providers to talk about people from the same or different backgrounds who exhibit positive values, those people do not need to be “famous”.
I.61-I.76	White respondents who care for all or nearly all White children did not know how to answer the items under supporting a positive self-identity that had to do with relating to children’s own racial or ethnic backgrounds. All of these respondents mentioned using resources (dolls, books, TV shows, etc.) as launch points for age-appropriate conversations about different races and cultures but weren’t sure how to respond to items referring to White children’s own backgrounds (for example, unsure if it’s appropriate to emphasize White children’s own background). One respondent of color said one of her families explicitly told her that they did not want race discussed because they did not want their child to feel singled out.	Items related to a child’s own racial or ethnic background were grouped together into a separate section titled “Supporting a positive racial and ethnic identity”, rather than dispersed throughout the other Supporting a positive self-identity dimension. There is individualized introductory information for these items that describe why the items are important and relevant for children of color and White children. Providers may also respond to items related to support for diversity among varied racial, ethnic, and cultural backgrounds in the support for antibullying and antibias section. We also reached back out to our academic expert panelists, Iheoma Iruka and Stephanie Curenton, for their thoughts and guidance on this approach. We are waiting for their response.
I.76	This item asks about opportunities for children to volunteer in the community. A few respondents, including both FFN respondents, mentioned that they don’t always have the opportunity or feel safe to bring children outside of their setting and into the wider community.	Added examples of ways providers can provide opportunities for children to volunteer and support others in their wider community without necessarily leaving their home (for example, writing letters to mayors and state representatives about important neighborhood issues).
Support for Emotional Development
II.20, II.21, II.23	These items ask about incorporating children’s home language(s). A few respondents were confused about what to put when children’s home language is English.	Reworded items to clarify what is meant by home language to make it clear providers who only speak English with children who only speak English should check Not Applicable.
II.22	One respondent was not sure if “pictures of children and their families” referred to her actual children and families, or just children and families in general. In addition, this respondent said some families would not want their pictures on display in someone else’s home.	Dropped this item but added photos of children’s family members as an example to another item about materials in the HBCC setting.
II.28	One respondent noted that “sharing favorite objects from home” was not permitted because it became an opportunity to “show off” and created issues between children.	Removed this item.
Positive and Proactive Behavior Management
III.11	This item asks about incorporating routines from families’ homes. One respondent explained that while she does ask families about home routines, she won’t necessarily use them if they are not feasible in the HBCC setting.	Added “when possible” to the end of the statement.
III.25-.26	One respondent suggested we replace the word “misbehave” to something else more neutral because “younger children do not understand they are misbehaving” since they are just learning what is and is not appropriate behavior.	Replaced “misbehavior” with “behavior” in the phrase “When one child’s behavior harms another child…”.
III.29	This item asks about adapting rules. One respondent interpreted this as different rules for different age groups.	Added examples to show various ways providers can adapt rules for children beyond age group.
Support for Learning
IV.36, IV.45, IV.48	A few respondents said “homework” is not relevant for 5- & 6-year-olds who are not in kindergarten yet, or do not have homework even if they are. In addition, one respondent said students who attend schools in low-income areas are rarely given homework since expectations for completion in those schools is so low.	Removed items specific to homework completion and used “schoolwork” as examples for some items in this domain.
IV.46	This item asks about incorporating families’ preferences. One respondent said the family’s preferences might not always be reasonable or feasible for the HBCC setting (for example, if the family wants an older and younger sibling to spend the entire day together or, alternatively, stay completely apart from one another).	Changed “I honor families’ preferences…” to “I try to honor…”
Support for Physical Development
V.13	Both rural respondents explained that “taking” kids somewhere is not really an option, but they have access to plenty of land and provide opportunities for children to access outdoor activities.	Changed “I take children to the park or other space…” to “I provide children with opportunities to explore…”
V.23	One respondent did not understand including children “from different backgrounds” in physical activities because all of her children come from very similar backgrounds.	Removed “backgrounds” and included specific examples.
V.34	One respondent explained that she talks a lot to parents about the importance of sleep, but not necessarily to the children.	Changed “I talk about the importance of getting enough sleep…” to “I talk to children and families about the importance of getting enough sleep…”
V.41	This item is about including roles for children during meal or snack times. One respondent said there are strict rules around food preparation for licensed respondents, especially since COVID (children have to be spaced apart, children cannot touch each other’s food, etc.)	Added COVID-safe examples of food preparation and planning.
V.46	One respondent marked “Never” because none of her children have food allergies but asked if a child who refuses to eat eggs counts as a “nutritional alternative” to talk about with families.	Added examples to item to be inclusive of other ways a provider may need to ask about nutritional alternatives from families.

The referenced collection of information is voluntary. Information will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and the expiration date is 8/31/2024.

The referenced collection of information is voluntary. Information will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and the expiration date is 8/31/2024.