Appendix E: HBCC-NSAC Toolkit provider questionnaire matrix sampling

Appendix E

HBCC-NSAC Toolkit provider questionnaire matrix sampling

HBCC-NSAC Toolkit provider questionnaire matrix sampling

Purpose

The initial version of the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) provider questionnaire, administered in the July 2022 pre-test, had 237 items and took participants between 40 minutes and almost 2 hours to complete (participants who took the longest also left open-ended comments, which will not be included in the pilot). The study team removed duplicative items after the pre-test, but the provider questionnaire is still too long to be administered to pilot study participants. In later use, we expect that providers will select specific domains – meaningful sections – that align with the provider’s interests, needs, and professional development goals. Once a provider has completed one section, they may choose to move on to another at their own pace. This may be decided individually or with the support of a coach or mentor.

To reduce burden on pilot study participants, replicate a more realistic administration of the provider questionnaire, and obtain information about more items, we will use matrix sampling and randomly assign HBCC providers to one of four shorter versions of the provider questionnaire. This approach has been used in national studies (for example, the mathematics and reading assessments for grades 3-5 of the Early Childhood longitudinal Study (1998 cohort), and the mathematics assessment for the Middle Grades Longitudinal Study 2017-2018) to develop assessments and will allow us to obtain information on a larger set of items without overwhelming participants. ¹ This information on the larger set of items may allow us to create a briefer version of the instrument. This matrix sampling design will allow us to examine the dimensionality of the proposed sections and domains.

Four versions of the HBCC-NSAC Toolkit provider questionnaire for the pilot study

Exhibit 1 shows the domains and dimensions and number of items in each of the proposed four versions of the provider questionnaire. The added color-coding of the dimensions illustrates the domains being measured:

Color: Domain

Blue: I.A-I.D Support for Social Development dimensions

Green: II.A-II.C Support for Emotional Development dimensions

Yellow: III.A-III.B Positive and Proactive Behavior Management

Red: IV.A-IV.D Support for Learning dimensions

Purple: V.A-V.D Support for Health and Physical Development dimensions

To create each of the four versions, the study team divided the full list of items that make up the provider questionnaire to create shorter versions. Each version includes different combinations of domains and dimensions and no version exceeds 120 items, which will take approximately 25 minutes to complete. Based on responses to each of the four versions, the study team will be able to examine responses in relation to each combination of dimensions. Further, each dimension overlaps across at least two versions in order to obtain enough responses for items within a given dimension for analysis purposes and allow examination of each dimension with the other dimensions.

Exhibit 1. Domains and key dimensions in each provider questionnaire version for pilot study matrix sampling

Version 1: domains & dimensions	N	Version 2: domains & dimensions	N	Version 3: domains & dimensions	N	Version 4: domains & dimensions	N
I.A. Support for Social Development: Builds and strengthens a positive relationship with children	11	I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication	14	I.D. Support for Social Development: Supports antibullying and antibias practices	19	I.A. Support for Social Development: Builds and strengthens a positive relationship with children	11
I.C. Support for Social Development: Supports children’s social skills	15	I.C. Support for Social Development: Supports children’s social skills	15	II.A. Support for Emotional Development: Helps children understand and regulate emotions	12	I.B. Support for Social Development: Supports children’s perspective-taking and nonverbal communication	14
I.D. Support for Social Development: Supports antibullying and antibias practices	19	II.B. Support for Emotional Development: Supports a positive sense of belonging	15	II.B. Support for Emotional Development: Supports a positive sense of belonging	15	II.A. Support for Emotional Development: Helps children understand and regulate emotions	12
III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines	15	III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices	15	II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity)	15	II.C. Support for Emotional Development: Supports a positive self-identity (including racial and ethnic identity)	15
IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset	10	IV.A. Support for Learning: Provides learning opportunities	15	III.A. Positive and Proactive Behavior Management: Uses predictable and responsive routines	15	IV.A. Support for Learning: Provides learning opportunities	15
IV.C. Support for Learning: Scaffolds problem-solving	9	IV.C. Support for Learning: Scaffolds problem-solving	9	III.B. Positive and Proactive Behavior Management: Uses proactive and positive disciplinary practices	15	V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being	13
IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home	8	IV.D. Support for Learning: Collaborates with families to strengthen learning connections with home	8	IV.B. Support for Learning: Supports positive approaches to learning and a growth mindset	10	V.B. Support for Health and Physical Development: Communicates with families about health and physical development	7
V.A. Support for Health and Physical Development: Provides a variety of activities to support physical well-being	13	V.B. Support for Health and Physical Development: Communicates with families about health and physical development	7	V.C. Support for Health and Physical Development: Scaffolds physical activity	11	V.C. Support for Health and Physical Development: Scaffolds physical activity	11
V.C. Support for Health and Physical Development: Scaffolds physical activity	11	V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition	19			V.D. Support for Health and Physical Development: Promotes health, safety, and nutrition	19
Total number of items:	111		117		112		117

The referenced collection of information is voluntary. Information will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and the expiration date is 8/31/2024.

1 Pollack, J., Atkins-Burnett, S., Rock, D., and Weiss, M. (2005). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), Psychometric Report for the Third Grade (NCES 2005–062). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

National Center for Education Statistics. “Middle Grades Longitudinal Study of 2017–18 (MGLS:2017)

2016 Item Validation Field Test (IVFT) Data Collection.” Supporting Statement Part A. Submitted to the Office of Management and Budget, 2015.