The Home-Based Child Care Toolkit for Nurturing School-Age Children Pilot Study
Pre-testing of Evaluation Data Collection Activities
0970 – 0355
Supporting Statement
Part B
December 2022
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Ann Rivera and Bonnie Mackintosh
Study Objectives
This pilot study will test the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit). As noted in Supporting Statement Part A Section A1, the pilot is part of the Home-Based Child Care Supply and Quality (HBCCSQ) project. The objective of the pilot study is to administer the HBCC-NSAC Toolkit to a diverse group of HBCC providers and the families they care for and elicit enough responses to examine the psychometric properties of the items in the HBCC-NSAC Toolkit instruments. The study team aims to identify changes to the wording or ordering of statements and questions in the instruments that will enable future participants to understand them more accurately, improving the overall quality and usefulness of the HBCC-NSAC Toolkit. The study team will use the pilot study to refine and improve the quality of the HBCC-NSAC Toolkit and the recruitment and data collection procedures in preparation for a future large-scale validation study (full Information Collection Request (ICR) to be submitted).
Generalizability of Results
The pilot study is intended to present an internally-valid description of HBCC-NSAC Toolkit instruments when used by home-based providers who care for school-age children and the families whose children receive care from those providers to inform instrument development, not to promote statistical generalization to other sites or service populations. Findings will not be representative of the experiences of HBCC providers and families, and any written products resulting from the study will acknowledge this limitation.
Appropriateness of Study Design and Methods for Planned Uses
The pilot study is designed to collect information efficiently from a diverse group of providers and the families they care for to test the HBCC-NSAC Toolkit and recruitment and administration procedures. To adequately test the HBCC-NSAC Toolkit requires that providers have a range of characteristics that may be associated with their responses, including variation in geographic area (for example, urban or rural), licensed and exempt license status, race and ethnicity, and ages of children served. The HBCC-NSAC Toolkit is offered in the English and Spanish languages and recruitment efforts will target providers and families who speak these languages. Working with staff from community organizations that have access to a wide range of providers allows for efficient recruitment of a diverse testing pool. Participants will be selected to vary by the above characteristics and the study team will closely monitor the respondent pool to ensure an adequate number of respondents across the characteristics are selected to participate. The details of the methodological approach are described in Section B2.
As discussed in Supporting Statement Part A Sections A1 and A2, the instruments in the HBCC-NSAC Toolkit include a provider questionnaire (Instrument 3) and a family questionnaire (Instrument 4) that will be available in English and Spanish. This proposed pilot study will be conducted in two phases.
Phase one. The study team will pre-test the Spanish version of the provider questionnaire. Although the English version was pretested before this ICR to identify potential problems with interpretation of items, the Spanish version was completed recently and needs cognitive testing. Similarly, the family questionnaire has not been pretested (in either English or Spanish) and needs cognitive testing.
The pre-test will allow the study team to assess whether Spanish-speaking participants can easily use and understand the provider questionnaire and family questionnaire (Instrument 3 and Instrument 4), learn about any differences in understanding the Spanish versions of the HBCC-NSAC Toolkit instruments by dialect or cultural background, and obtain information about completion time. Similarly, pre-testing the family questionnaire (Instrument 4) with English-speaking participants will allow the study team to learn whether the items are easy or difficult to understand, whether these items are meaningful for the families (would the families recommend the HBCC providers use this with families), and the time it takes to complete. All pre-test participants will participate in cognitive interviews (Instrument 5 and Instrument 6) so the study team can understand how participants interpret items. Lessons learned will help inform the study team’s understanding of potential differences in responses and potential additions or revisions for the larger validation study (full ICR to be submitted). The study team could also provide guidance in interpreting responses after examining the data.
Phase two. Phase two of the pilot is designed to test the HBCC-NSAC Toolkit across a diverse group of HBCC providers and the families they serve. In order to minimize burden to providers, the study team will use a matrix sample approach. The full list of items in the provider questionnaire will be divided to create several complementary versions, that would vary by proposed dimensions. Each version will include different combinations of domains and dimensions that overlap. The study team will use a matrix sampling of dimensions and assign HBCC providers to versions with different combinations of items. The versions have been designed to have overlap of each dimension across at least two versions (see Appendix E for matrix sampling details). This approach has been used in national studies to develop assessments (for example, the mathematics and reading assessments for grades 3-5 of the Early Childhood longitudinal Study (1998 cohort), and the mathematics assessment for the Middle Grades Longitudinal Study 2017-2018) and will allow the study team to obtain information on a larger set of items without overwhelming or fatiguing participants.1 The study team will be able to examine responses in relation to each combination of dimensions.
As described in Supporting Statement Part A, Section A3, for phase two of the pilot, the study team will program the provider questionnaire (Instrument 3) into a web survey platform. This approach mirrors the intended mode for the full validation study (full ICR to be submitted) and allows for flexible participation because providers can complete it at any time during the data collection period using a computer or smart phone. The option of telephone interviewing and paper-based instruments accommodates providers who may not have access to or be comfortable with technology. Given the families will largely be a high poverty and low education population, the paper-based family questionnaire will ensure all have access to the questions. Additionally, the paper mode allows providers to easily distribute the instrument to and collect it from participating family members and mirrors the approach intended for the ultimate use of the HBCC-NSAC Toolkit. The study team will document mode of participation and examine the data for potential mode effects in analysis.
As noted in Supporting Statement Part A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. The data collected are not intended to be representative, and findings may not apply to all providers and families. This study does not include an impact evaluation and will not be used to assess participants’ outcomes. Pilot progress and results will be reported as internal memorandum to the Administration for Children and Families (ACF) and will clearly describe key limitations. No products will be made publicly available from the pilot study.
Target Population
Number and characteristics of respondents for phase one of the pilot study. In phase one, up to 9 Spanish-speaking HBCC providers who regularly care for at least one school-age child (age 5 and in kindergarten, or ages 6 through 12), meaning they care for the school-age child(ren) at least 10 hours per week in a home (in the past year), will complete the Spanish versions of the provider questionnaire. The study team will seek providers who may speak different Spanish dialects by including providers from both urban and rural locations and from different geographic areas of the United States (such as the Northeastern, Southeastern, and Southwestern regions). Up to 9 Spanish-speaking and 9 English-speaking family members will complete the family questionnaire. These family members must have at least one school-age child in HBCC and are most responsible for the care of that child when they are outside of the HBCC setting. All participants must be at least 18 years old.
Number and characteristics of respondents for phase two of the pilot study. In phase two, the study team will collect information from 150 HBCC providers who regularly care for at least one school-age child (age 5 and in kindergarten, or ages 6 through 12), meaning they care for the school-age child(ren) at least 10 hours per week in a home (in the past year). The study team will also collect information from up to 150 family members with at least one school-age child in HBCC and who are most responsible for the care of the child when they are outside of the HBCC setting. All participants must be at least 18 years old. The recruitment goal is 90 English-speaking providers and 60 Spanish-speaking providers. The proposed approach balances cost and burden with ensuring enough information to meet the study’s analytic needs.
The study team will seek providers:
In both urban and rural settings,
In states that vary on HBCC licensing rules,
From varied racial and ethnic backgrounds,
With varied licensed and licensed exempt status
As well as:
Providers who care for only school-age children and some who care for a school-age child in a mixed-age HBCC, and
Providers in areas of high-poverty concentration.
Table B.1 is an example of the HBCC provider characteristic combinations expected in phase two. As shown on the first row of Table B.1, of the 150 total providers, the study team expects to recruit more HBCC providers who are licensed or regulated by states to provide child care and early education (CCEE), commonly referred to as family child care (FCC) providers (n = 85) than those who are unlicensed, or legally exempt from state licensing or other state regulations for CCEE, commonly referred to as family, friend, and neighbor (FFN) providers (n = 65),. The study team expects to recruit more FCC providers than FFN providers because FFN providers usually care for only 1 to 2 children and are less likely to care for school-age children in mixed age groups.2 FFN providers are also typically more difficult to recruit.
As noted in Section B1, each provider will receive only one of the four versions of the provider questionnaire (Appendix E). As shown in the second row of Table B.1, each version will be assigned to between 17 and 22 providers. Each dimension is in at least two versions of the provider questionnaire. The proposal below, as shown in the last row of Table B.1, would allow for a minimum of 30 respondents per dimension in the Spanish and English versions, and among both FCC and FFN providers. (The study team expects fewer Spanish-speaking providers in FCCs and will likely recruit more Spanish-speaking FFN providers). At least thirty respondents in each of these groups will allow for comparisons of variation in responses across groups for each of the dimensions. The study team also plans to use item response theory to examine how well items contribute to the proposed dimension. The fit of the items to the theoretical dimension and the item-total correlations will provide initial evidence of construct validity. A factor analysis of the residuals from a Rasch rating scale model will provide information about potential differences in dimensionality. Analyses will also be conducted to look for potential differential item functioning (DIF) among different subgroups. DIF analyses examine whether respondents with the same level of the trait (for example, support for learning) have different probabilities of endorsing specific items within that scale. Though not all DIF indicates bias, it does indicate a need to consider whether there is a balanced representation of items and whether each item indicates a behavior that is important for measuring the level of the trait (e.g., Support for physical development) and has examples that hold the same meaning across groups (e.g., does not suggest an activity that is unfamiliar to a particular group).
A sample size of 30 to 35 is sufficient for estimating a Rasch rating scale model if the group of respondents is diverse enough that the item responses represent a range of levels of the trait (for example, the responses are not strongly skewed for some or all items). The number of items being piloted for each dimension will allow some items to be dropped if needed to improve the fit. Should some items have poor fit or detect DIF, the study team will examine them carefully and consult with experts to decide if they fit better with a different dimension or should be dropped or revised.
Table B.1. Example HBCC provider characteristic combinations for phase two of the pilot study
|
Licensed providers (FCCs) |
Exempt providers (FFNs) |
Pilot study respondents (including English- and Spanish-speaking providers across rural and urban geographic areas) |
85 |
65 |
Respondents for each of the four versions of the provider questionnaire |
21-22 |
17 |
Respondents for each dimension (each dimension is in at least two versions of the provider questionnaire) |
42-43 |
33 |
aProviders will complete one of the four versions of the provider questionnaire.
bEach dimension is in at least two versions of the provider questionnaire.
Respondent recruitment and site selection
Site selection. To recruit providers, the study team will work with about 10 community organizations that offer support or quality improvement opportunities to HBCC providers and are willing to work with the study team to recruit these providers. The HBCCSQ study team has prior relationships with many organizations offering support to HBCC providers including organizations operating in multiple states. 3 When considering organizations to potentially partner with, the team will prioritize the following:
Organizations that have the capacity to work with the study team to recruit enough providers.
Organizations that work with providers from a range of backgrounds, including those who live in under-resourced communities.
Respondent recruitment for phase one of the pilot. The study team will work with community organizations that provide supports to HBCC providers in order to recruit 9 total providers to participate in phase one. Once recruited, the study team will ask participating providers if they have Spanish-speaking families of school-age children who attend their HBCC who would be willing to complete the Spanish version of the family questionnaire. The study team will also ask the project’s experts with lived experience providing HBCC (Supporting Statement Part A Table A.2) and providers who participated in the summer 2022 pre-test (section B.2) to invite English-speaking families to complete the English version of the family questionnaire, and if needed, Spanish-speaking families to complete the Spanish version of the family questionnaire.
Phase two. The study team will use non-probability, purposive recruitment to identify and recruit HBCC providers from 5 to 10 states in different regions with the characteristics outlined above on page 4. The study team will partner with organizations that provide support to HBCC providers in order to recruit a varied group of HBCC providers.
As much as possible, the study team will draw from available lists of licensed FCC providers and subsidized FFN providers. The quality of the data obtained from the state lists may be inconsistent and some states may not allow access to contact information. Some of the community organizations that we will partner with may serve a small number of FCCs and be able to help supplement state lists. The team will work closely with community organizations that support HBCCs to help with recruitment, particularly for recruiting FFN providers. FFN providers are typically not connected to funding and will not appear on the state lists. The study team will also ask for recommendations from other providers contacted by the study team during recruitment to participate in the pilot.
If the study team is still unable to reach the desired targets for each of our provider characteristics, then the study team will conduct in person recruitment visits in three purposively selected community organizations (sites). For each of the three sites selected, the study team will work with staff from these organizations to identify and recruit HBCC providers to complete the HBCC provider questionnaire (for example, after a scheduled professional development session) or receive the electronic or paper versions to complete later during the data collection period. Table B.2 is an example of the number of providers the study team expects to recruit across different recruitment approaches.
Table B.2. Proposed HBCC provider targets by recruitment approach
Sample source |
Provider type(s) targeted |
Target number of providers |
State lists |
Licensed providers (FCC) |
85 |
Subsidized exempt providers (FFN) |
25 |
|
Community organizations providing lists; in-person recruiting as needed |
Exempt, unlisted providers (FFN) (possibly some FCCs) |
25 |
Recommendations of providers from other providers during recruitment |
Exempt, unlisted providers (FFN)
|
15 |
Total |
|
150 |
The study team will ask participating providers to recruit one or more families of school-age children who attend their HBCC to complete the family questionnaire, based on the number of families served and their characteristics. The study team will use non-probability, purposive recruitment to identify potential respondents for the family questionnaire.
Because respondents will be purposively selected, they will not be representative of their populations. Instead, the study team aims to obtain variation in HBCC providers’ characteristics to ensure a diverse group of respondents test the instruments in the HBCC-NSAC Toolkit. All providers who care for school-age children will be asked to invite at least one family families to complete the family questionnaire. The study team anticipates that most providers will invite families and that there will be a sufficient mix of cared for families with different characteristics who complete the family questionnaire, for example, both English-speaking and Spanish-speaking families, and families with different racial and ethnic backgrounds.
Development of Data Collection Instruments
The telephone script and recruitment information collection instruments (Instrument 1 and Instrument 2) were developed by the study team to include the key questions necessary to determine whether providers will be invited to participate in the pilot and whether providers will be asked to recruit families to participate in the pilot. Recruitment criteria are described in section B4.
To develop the instruments in the HBCC-NSAC Toolkit (Instrument 3 and Instrument 4), the study team selected domain topics based on the larger HBCCSQ project’s conceptual framework, literature review, and measures review to highlight features of quality that are typically implemented differently or more likely to occur in HBCC than in other CCEE settings, and that might support positive outcomes for children and families. In some cases, the study team examined and adapted items from existing measures of quality used in HBCC or used in school-age youth program settings (see Appendix G). However, the HBCC-NSAC Toolkit is designed to measure dimensions that were identified as inadequately measured or not measured in other existing instruments. When needed, the study team developed items specific to the intended constructs and population. Most HBCC measures are designed for settings with children birth to age 5, so the typical dimensions, for example for social-emotional development, differ somewhat for school-age children. Thus, most items in the HBCC-NSAC Toolkit are new and developed by the study team based on literature about the most promotive and protective practices for school-age children in each domain. Academic experts and experts with lived experience providing HBCC provided input to measures development from selection of the constructs through review of final items in the draft provider questionnaire. These experts provided written and/or verbal feedback. Additionally, staff from community organizations reviewed the draft provider questionnaire and provided feedback (see Supporting Statement Part A Section A.8).
Table B.3 describes the key dimensions within each domain of the HBCC-NSAC Toolkit. Items were developed around each key dimension.
Table B.3 Domains and key dimensions in the HBCC-NSAC Toolkit instruments
Domain |
Key dimensions |
Support for social development |
|
Support for emotional development |
|
Positive and proactive behavior management |
|
Support for learning |
|
Support for health and physical development |
|
During the pilot study, the study team will use matrix sampling of dimensions to administer all provider questionnaire items across the domains to achieve the study’s objective to obtain enough responses to examine the initial psychometric properties of the provider questionnaire and identify items that may be adding measurement error. As noted earlier in Section B2, the study team will use both descriptive statistics and item response theory to analyze the items to inform any changes to the provider questionnaire before the validation study (full ICR to be submitted). All of the provider questionnaire items have been reviewed by multiple academic research and measurement experts, experts with lived experience providing HBCC, staff from community organizations who support providers, and HBCC providers (both licensed FCC and exempt FFN providers).
The family questionnaire (Instrument 4) was developed as a short and easy to use communication tool for the HBCC provider to learn about family preferences. Providers and families can use families’ responses to identify topics, aligned with the domains and dimensions in the provider questionnaire, that are important to families regarding their school-age children’s care and education in HBCC.
The cognitive interview guide for the Spanish version of the provider questionnaire (Instrument 5) was developed to ensure Spanish speaking HBCC providers who speak different Spanish dialects interpret items in the provider questionnaire the same way and test how long it took providers to complete the questionnaire.
The cognitive interview guide for the family questionnaire (Instrument 6) was developed to ensure families with different characteristics interpret items in the same way and test how long it takes families to complete the questionnaire.
The family data collection logistics call (Instrument 7) was developed by the study team to describe the family data collection to providers and determine their willingness to distribute and collect (once completed and sealed) the family questionnaire from families of school-age children in their care. For providers who agree to invite families to participate, the family data collection logistics call guides the discussion of the provider’s responsibilities when sharing the family questionnaire with families, collecting it, and returning it to the study team (Appendix J).
Pre-testing of Data Collection Instruments-English version of the HBCC-NSAC Toolkit provider questionnaire
In July 2022, the study team pre-tested the English version of the HBCC-NSAC Toolkit provider questionnaire with 9 providers that represented a mix of characteristics (see Table B.4).
Table B.4. Characteristics of HBCC providers for the pre-test of the English version of the HBCC provider questionnaire
Data collection |
Urban |
Rural |
|
English-speaking providers |
Spanish-English bilingual providers |
English-speaking providers |
|
Black, non-Hispanic |
4 (2 exempt FFN, 2 licensed FCC) |
|
|
Hispanic or Latino |
|
1 (licensed FCC) |
|
White, non-Hispanic |
1 (licensed FCC) |
|
3 (licensed FCC) |
The study team purposively selected providers from urban and rural areas, across different races, including some providers who are bilingual in Spanish and English. Recruiters emailed interested providers to describe the study and then called them to collect provider characteristics to confirm eligibility and schedule their participation.
While completing the provider questionnaire, providers documented start and stop time, marked statements that described activities they wanted to learn more about and statements they were not sure how to answer, and noted any other questions or comments in an open text field. Providers then participated in a 30-minute cognitive interview to share their thoughts about the items and dimensions in the provider questionnaire and their experience using them.
The study team used participant feedback to revise and finalize the introduction, instructions, and items in the provider questionnaire (See Appendix H for the pre-test summary and Instrument 3 for the provider questionnaires).
Additional Pre-testing of Data Collection Instruments, phase one of pilot study
As described in Supporting Statement Part A Section A2, and above in Sections B1 and B2, after receiving approval from the Office of Management and Budget (OMB), the study team will conduct phase one of the pilot study to pre-test the Spanish version of the provider questionnaire and the English and Spanish versions of the family questionnaire.
ACF has contracted with Mathematica for this pilot study data collection. Study team members at Mathematica have experience recruiting respondents for studies that focus on CCEE, respondents who are experiencing high levels of poverty, are members of historically marginalized communities, and who may speak languages other than English.
The study team have developed flyers and other recruitment materials to facilitate outreach to community organizations and potential participants. All materials explain the study in English and Spanish and will be written in plain language for respondent readability (Appendices A, B, C and D).
For phase one of the pilot study, the study team will work with community organizations to identify Spanish-speaking providers (Appendix A). The study team will ask staff from these community organizations to share the study flyer with providers in their network. With the provider’s permission, staff from community organizations will share the contact information for interested providers with the study team using a secure website. The study team will work with community organizations to contact and recruit Spanish-speaking providers, and work with participating providers to recruit Spanish-speaking family members to complete the Spanish version of the family questionnaire. The study team will work with the project’s experts with lived experience providing HBCC and providers who participated in the summer 2022 pre-test to obtain English-speaking families and Spanish-speaking families if needed.
For phase two of the pilot study, the study team will obtain lists of providers from state and local agencies. The study team will also use the same recruitment procedures as phase one to work with community organizations to assist with provider recruitment and use other recommendations from recruited providers. The team will work with providers to recruit families.
For phase one, trained recruiters will send an email, flyer, and the provider consent form (Appendix F) to introduce the study to providers identified from state lists, community organizations, and other providers. Recruiters will also call providers to discuss the study and learn about the providers’ characteristics and eligibility using a call script (Instrument 1), and to obtain verbal consent from providers who are eligible and interested in participating. At the end of the recruitment call, providers who meet study criteria and are interested in participating will be read the consent letter and form over the phone. Providers will be asked to verbally indicate whether they consent to participate. The study team will offer to mail a copy of the completed consent form to providers.
For phase two, trained recruiters will send an email and flyer to introduce the study to providers identified from state lists, community organizations, or other providers. Recruiters will also call providers to discuss the study and learn about the providers’ characteristics and eligibility using a call script (Instrument 2).
To generate provider awareness of and interest in the study during phase two, the study team will work with community organizations to arrange for recruiters to speak about the study at the organizations’ gatherings. The study team will also ask participating providers to recommend other providers in the area who provide care to school-age children. If the study team does not reach the desired number of respondents for each characteristic (See Section B.2 Target Population), study team members will conduct on-site recruiting during weeklong site visits with up to three community organizations. During on-site recruiting, study team members will ask local schools if they are willing to post or distribute flyers about the study. The study team will continue recruitment until the desired number of respondents for each type of provider are reached.
Family recruitment
During the initial recruitment call with providers in both phases of the pilot study (Instrument 1 and Instrument 2), recruiters will explain that providers will be asked to recruit one or more families to respond to the family questionnaire.
During phase one, recruiters will conduct a follow up call with Spanish-speaking providers, who agree to participate, to discuss the family data collection and their willingness to share the family questionnaire with Spanish-speaking families (Instrument 7). The study team will also contact the project’s experts with lived experience providing HBCC and providers who participated in the summer 2022 pre-test to discuss the family data collection and their willingness to invite English-speaking families to complete the English version of the family questionnaire and if needed, Spanish-speaking families to complete the Spanish version of the family questionnaire (Instrument 7).
During phase two, recruiters will call providers who agree to participate to discuss the family data collection and their willingness to share the family questionnaire with English-speaking families and Spanish-speaking families (Instrument 7). In both phases of the pilot study, providers will remain eligible to participate in the study, even if they prefer not to recruit families for the study.4 If providers agree to recruit families, the study team will give providers all materials needed to recruit and support the participation of families in the study (Appendix J). Designated study team members will be available to answer study related questions from providers or family members as needed to support family recruitment.
For both phases of the pilot study, all participants will complete a consent form before completing their applicable instrument(s) in the HBCC-NSAC Toolkit (Appendix F). The provider consent form will be available electronically and on paper while the family questionnaire consent form will be available only on paper. Each participating family will complete a paper consent form prior to completing the family questionnaire.
For phase one, verbal consent will be obtained from providers at the end of the recruitment call (Instrument 1). The Spanish version of the provider questionnaire will be available electronically in a Word document or paper. The questionnaire will be formatted to include clear labels and directions for providers to easily identify the items most relevant to their setting characteristics. Family questionnaires will be completed on paper.
Based on provider preference, the study team will email HBCC providers the provider questionnaire in a Word document or mail the paper copies (Appendix I). Phase one providers, experts with lived experience providing HBCC, and providers who participated in the summer 2022 pre-test who agree to recruit a family member to participate in phase one will receive a packet containing a family consent form, paper-based family questionnaire, easy-to-follow instructions for distributing it to and collecting it from a family member of a school-aged child (Appendix J), a self-seal envelope to give to families, and prepaid shipping materials to return completed paper instruments to Mathematica.
In phase one, providers and family members who return completed instruments will each be scheduled for a 30-minute cognitive interview to share their thoughts about the instruments and their experiences using them. Cognitive interviews with providers and family members will be conducted using a secure virtual meeting platform.
For phase two, providers will complete the consent form using the same mode they will use to complete their HBCC-NSAC Toolkit instrument(s). For example, providers who complete the web-based provider questionnaire (Instruments 3) will first complete a web-based consent form.
The provider questionnaire will be available as web-based instruments. The web-based instruments will be optimized so providers can respond on smartphones. Providers may also complete via paper, or the study team’s trained telephone interviewers can also administer the provider questionnaire to those who request it. Following consent, providers will receive a randomly assigned version of the provider questionnaire (Instrument 3) to complete using their preferred data collection mode.
Based on provider preference, the study team will email HBCC providers a link to the web-based provider questionnaire, mail the paper-based versions, or call the provider to administer them by phone. Providers recruited during on-site recruitment could complete the provider questionnaire on a study team member’s tablet or cell phone, or on paper. If they prefer to complete it later, the study team will obtain contact information so those providers receive an emailed link or telephone call from a trained interviewer.
The study team will ship each participating provider who will invite families of school-age children to participate in the study a packet of family consent forms (Appendix F), the paper-based family questionnaires with self-seal envelopes (Instrument 4), and easy-to-follow instructions for distributing them to and collecting them from families with a school-aged child (Appendix J). Families will place their completed consent forms and questionnaires into the provided self-seal envelopes and seal them before returning to providers. Providers will receive prepaid shipping materials to return completed paper instruments to the study team.
The study team will debrief recruiters and phone interviewers on their experiences collecting data to learn about the effectiveness of the data collection procedures and any common questions providers ask about completing the HBCC-NSAC Toolkit instruments.
The study team will monitor data quality and consistency throughout the data collection. Before data collection begins, recruiters and telephone interviewers will participate in training to discuss the purpose of the HBCC-NSAC Toolkit and the pilot study. Recruiters will learn the importance of testing the HBCC-NSAC Toolkit with a diverse group of HBCC providers and the families they care for, as well as strategies to build rapport and execute the recruitment and data collection protocol. Also, during training, recruiters will practice using the recruitment script to ensure provider characteristics are collected accurately and consistently across respondents. Telephone interviewers will review the provider questionnaire and best practices for administration and entry into the web-based version.
During data collection, the study lead will hold weekly check-ins with recruiters so staff can discuss progress, ask questions, troubleshoot problems, and share successful strategies. In addition, study team members will discuss any questions or comments received from participants to ensure data collectors accurately respond to questions or issues reported by respondents. The team will monitor completion of the HBCC-NSAC Toolkit instruments, including conducting initial data checks as the data are received, to identify high missingness or if multiple respondents indicate they are unsure how to answer particular items. The team will follow up with providers and families as needed to address any issues. The web-based instruments will include checks for completeness to minimize missing data, for example, the programmed provider questionnaire may notify respondents if they skipped a question and ask them to provide a response. Every interviewer will be monitored at least twice during data collection.
Respondent selection
The pilot study instruments are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection. Because the study team will screen potential respondents for eligibility and interest in participating during the recruitment call, the study team will collect information about how many potential respondents screened into the study (1) participated, (2) were not eligible, or (3) declined to participate, and (4) the number of respondents who did not complete all study activities. The study team will collect key characteristics about providers and the families they care for during recruitment calls (Instrument 1, Instrument 2, and Instrument 7), such as license status and race/ethnicity. The study team will assess whether non-response was more prevalent among particular types of providers, for example, licensed and exempt providers. The study team will use that information to inform the larger validation study (full ICR to be submitted). When item level nonresponse is greater than ten percent, the study team will examine the associations between nonresponse and HBCC provider characteristics, such as race/ethnicity of providers, race/ethnicity and language of children in care, provider experience levels, or the number of children cared for and their ages. Examination of item level nonresponse will be used to inform the analyses of the items and dimensions and inform potential revisions. This information will be used in revisions of questions after the pilot, with a goal of minimizing item non-response in the final instruments.
The purpose of this request is for pilot testing data collection instruments and procedures to evaluate and improve the instruments’ quality for use in the forthcoming validation study that will include a larger number of respondents (full ICR to be submitted). The data will be for internal use only. Information reported to ACF will clearly state that results are not meant to be generalizable.
Data Handling
Data collected during the recruitment call will be entered into a secure electronic database. The original recruitment call script used to collect data about provider characteristics and eligibility will be saved to a secure network folder. Data retrieved from the web-based instruments (including data collected by phone and paper that is entered into the web-based instruments) will be saved on a secure drive accessible only to Mathematica study team members. Direct export of the electronic data to the secure drive will result in minimal processing. Providers will receive prepaid mailing materials to return completed paper instruments to Mathematica. Trained study staff will enter data from returned paper family questionnaires into a secure database.
Data Analysis
Phase one. The study team will examine the frequencies for each of the item responses looking for variation and missingness. Variation will signal that this may be a useful tool for the providers. Missingness suggests that items may not be understood by families or are not relevant for families. The study team will use the cognitive interview responses to seek additional information about appropriateness for their setting, and any need for clarification, or to revise wording. Examination of responses across the providers may point to potential patterns to look for in the phase two data and help to guide any revisions before the larger validation study (full ICR to be submitted).
Phase two. Analysis of phase two pilot test data will evaluate psychometric properties of items with the overall sample (N ≥ 100) and for select subsamples of sufficient size (N ≥ 30). For exploratory work, a sample size of 30 items and 30 respondents is usually adequate for estimating the difficulty of items and the underlying trait level within one logit with 95 percent confidence and for indicating meaningful and statistically significant differences.5,6 Fewer items are needed for ratings than dichotomies; however, at least 10 observations are needed for each category to estimate the step parameters with confidence. The analysis of the pilot study will be exploratory in order to inform the selection of items and identify problematic items and categories. The analyses will be iterative, identifying the items and categories that provide the strongest measurement of each construct. That is, based on the results of one analysis, the study team may combine categories or remove items for the next analysis. The study team will repeat the process until the optimal set of items are identified.
The study team will also use item response theory (IRT) to examine item properties and initial evidence of the validity of dimensions for HBCC providers from different subgroups (licensed/exempt, English/Spanish language). Unlike classical test theory, IRT uses probablistic models that can account for missing data. The overlap in assignment of the four versions of the provider questionnaire (see Appendix E) and IRT will allow the study team to include items from related dimensions even if data on those items are not available from a subset of the respondents.
The study team plans to use a one-parameter Rasch rating scale or partial credit model using Winsteps. Because Rasch models assume unidimensionality, the study team will conduct most of these analyses by dimension. When dimensions show high interfactor correlations, the study team will estimate the model at the domain level. If there is a high level of misfit in items, the study team will examine factor analysis of the residuals and also examine potential differential item functioning (DIF) by the provider characteristics (beginning with those characteristics identified as having different descriptive statistics or item-total correlations).
The family questionnaire is intended to be used in the final HBCC-NSAC Toolkit as a communication tool between parents and HBCC providers about parent preferences in relation to the topics covered in the provider questionnaire. The study team will examine the descriptive statistics for each of the item responses looking for variation and missingness. The variation will signal that this may be a useful tool for the providers. Missingness suggests, for example, that these may not be understood or are not relevant for the families. The study team will also examine item means and missingness by the match of child and provider’s race/ethnicity and by the HBCC provider’s race/ethnicity. The study team will not create scales for this survey.
Data Use
The contractor study team will use the pilot study results to revise and finalize the HBCC-NSAC Toolkit instruments for the validation study. Instrument revisions and an accompanying final study memo and data tables will be shared with ACF at the conclusion of this pilot study. The final instruments will be included in a full ICR to OMB for the forthcoming larger validation study.
Table B.5 lists the federal and contract staff responsible for the study, along with each individual’s affiliation and email address.
Table B.5. Individuals responsible for study
Name |
Affiliation |
Email address |
Ann Rivera |
Office of Planning, Research, and Evaluation Administration for Children and Families |
|
Bonnie Mackintosh |
Office of Planning, Research, and Evaluation Administration for Children and Families |
|
Patricia Del Grosso |
Mathematica |
|
Ashley Kopack Klein |
Mathematica |
|
Sally Atkins-Burnett |
Mathematica |
|
Annie Li |
Mathematica |
|
Laura Kalb |
Mathematica |
|
Myah Scott |
Mathematica |
Instrument 1: Phase one provider telephone script and recruitment information collection
Instrument 2: Phase two provider telephone script and recruitment information collection
Instrument 3: HBCC-NSAC Toolkit provider questionnaire
Instrument 4: HBCC-NSAC Toolkit family questionnaire
Instrument 5: Phase one provider cognitive interview guide
Instrument 6: Phase one family cognitive interview guide
Instrument 7: Family data collection logistics call
Appendix A: Community organization outreach materials
Appendix B: Phase one provider and family recruitment materials
Appendix C: Phase two provider and family recruitment materials
Appendix D: Supplemental provider and family recruitment materials
Appendix E: HBCC-NSAC Toolkit provider questionnaire matrix sampling
Appendix F: Consent
Appendix G: Sources for HBCC-NSAC Toolkit development
Appendix H: HBCC-NSAC Toolkit English version pre-test summary
Appendix I: Provider and family invitations
Appendix J: Family data collection instructions
Appendix K: Frequently asked questions
Appendix L: Provider and family thank you letters
1 Pollack, J., Atkins-Burnett, S., Rock, D., and Weiss, M. (2005). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), Psychometric Report for the Third Grade (NCES 2005–062). U.S. Department of Education. Washington, DC: National Center for Education Statistics.
National Center for Education Statistics. “Middle Grades Longitudinal Study of 2017–18 (MGLS:2017)
2016 Item Validation Field Test (IVFT) Data Collection.” Supporting Statement Part A. Submitted to the Office of Management and Budget, 2015.
2 A. R. Datta, C. Milesi, S. Srivastava, and C. Zapata-Gietl. “NSECE Chartbook – Home-Based Early Care and Education Providers in 2012 and 2019: Counts and Characteristics.” OPRE Report No. 2021-85, Washington DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021.
3 In July 2022, during the initial development of the HBCC-NSAC Toolkit, the study team worked with five community organizations to recruit providers to pre-test the English version of the HBCC provider questionnaire. The study team has maintained a relationship with these organizations and may partner with a number of them again to recruit providers for this pilot study, particularly when seeking Spanish-speaking providers for phase one.
4 During phase one, if providers who are pilot study respondents decline to invite families to complete the family questionnaire, the study team will ask experts with lived experience providing HBCC and providers who participated in the summer 2022 pre-test to invite additional families to complete the family questionnaire. During phase two, the study team will ask providers who agree to share the family questionnaire to invite additional families if possible, to make up for providers who decline to invite families to complete the family questionnaire.
5 Linacre, J.M. “Sample Size and Item Calibration Stability.” Rasch Measurement Transactions, vol. 7, no. 4, 1994, p. 328.
6 For each dimension the minimum sample size is 33.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Mathematica Report |
| Subject | report |
| Author | Sharon Clark |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |