Part A QCCITT revised 11-15-11_clean

Measurement Development: Quality of Caregiver-Child Interactions for Infants and Toddlers

Supporting Statement, Part A

For OMB Approval

November 15, 2011

A. JUSTIFICATION

The Administration for Children and Families (ACF) of the Department of Health and Human Services is requesting Office of Management and Budget (OMB) clearance for data collection associated with the Measurement Development: Quality of Caregiver-Child Interactions for Infants and Toddlers (Q-CCIIT) project, a three-year project that began in 2010. The current submission requests clearance for the two-year period of data collection and analysis (FY 2011–2013). This section provides supporting statements for each of the points outlined in Part A of the OMB guidelines.

A.1. Circumstances Making the Collection of Information Necessary

a. Overview of Request

ACF has contracted with Mathematica Policy Research and its subcontractors, Child Trends, FPG Child Development Institute at the University of North Carolina, and WestEd, under contract number HHSP23320095642WC/HHSP23337016T, to develop a new measure to assess the quality of caregiver-child interaction for infants and toddlers in nonparental care.

The legislative base for the Q-CCIIT project is the Improving Head Start for School Readiness Act of 2007 (P.L. 110-134), outlining requirements on monitoring, research, and standards for Head Start and Early Head Start (see Appendix A). The Q-CCIIT measure to be developed and evaluated will provide the Office of Head Start with a reliable and valid observation tool to document caregiver-child interactions for infants and toddlers in nonparental care. This information will satisfy the accountability and program improvement goals mandated by the Act. Further, the Q-CCIIT project includes both Early Head Start settings and community-based settings to address the crucial need of measuring quality in infant and toddler classrooms across diverse settings. The potential audiences include state and county Quality Rating and Improvement Systems (QRIS) that monitor quality, often providing tiered reimbursement for care settings serving children with subsidies. A new tool may then help support the quality assessment process and possibly inform the quality improvement efforts of providers receiving Child Care Development Funds.

b. Project Context and Rationale

The new Q-CCIIT observational measure will assess the quality of child care settings for children from birth through 36 months, focusing specifically on the quality of caregiver‐child interactions.³ The measure will be appropriate for use across child care settings, center‐based and family child care (FCC), and single- and mixed-age classrooms.⁴ Specifically, the Q-CCIIT project is designed:

• To assess the state of the measurement field related to child-adult interactions and quality of child care settings for infants and toddlers

• To develop a measure for assessing the quality of child-caregiver interactions efficiently, yielding reliable and valid data

• To collect data to demonstrate the psychometric soundness of the new measure

• To outline a plan for the sustainability of the measure beyond its development and psychometric testing

After several decades of research demonstrating the generally low quality of infant/toddler child care (Boller et al. 2010; Helburn, 1995), the field is now focused on how best to provide nurturing, safe child care environments for infants and toddlers. Central to this focus is determining how child care quality is defined and measured. With the rise in high-stakes uses of observational measures of child care in state QRIS and in Head Start performance measurement, the availability of reliable and valid measures of infant/toddler care is critical for a range of stakeholders, including policymakers, program developers, child care providers, parents, and researchers. There is a dearth of measures appropriate for assessing the quality of infant/toddler care settings; those that do exist have limitations, ranging from being too specific to a certain type of setting to lacking reliability and both concurrent and predictive validity (Zaslow et al., 2011). Many measures currently available were developed as downward extensions of observations for older preschool children’s settings, which differ in structure and content from classrooms for infants and toddlers.

ACF has proposed that a new measure to assess the quality of infant/toddler care should focus on caregiver-child interactions, because relationships with primary caregivers are the “active ingredients” by which children’s early communication, learning, and competence unfold. The significant role of early relational influences on children’s emerging abilities extends beyond those encountered in the immediate home setting; interactions between young children and their nonparental caregivers have likewise been shown to be important contributors to children’s well-being (Shonkoff and Phillips, 2000). As illustrated in the Q-CCIIT research-based conceptual model (Figure A.1), indicators of process quality, such as sensitive and stimulating caregiver-child interactions, directly influence children’s cognitive, language, and social-emotional development. Structural features of care, such as caregiver professional development and child-to-adult ratio, likewise affect children’s development, albeit indirectly, through influences on process quality (NICHD ECCRN, 2002; National Research Council, 2003; Phillipsen et al., 1997; Vandell & Wolfe, 2000). The quality of early care is multifaceted, and encompasses child-peer relations and effective communication between caregivers and parents (for example, consistency in childrearing philosophy and practices). In recognition of the transactional nature of caregiver-child, child-peer, and caregiver-parent relationships, individuals’ own behavioral styles also contribute to and shape the quality of interactions with others (Sameroff, 2009; Sameroff & Chandler 1975).

Figure A.1.
Q-CCIIT Research-Based Conceptual Model for Infant/Toddler Quality of Care

Several activities will address the goals of the Q-CCIIT project. A literature review has been conducted (1) to provide a summary of the extant measures of quality appropriate for use in nonparental care environments that serve infants and toddlers, and (2) to evaluate the degree to which these measures adequately gauge the features of quality that are important to capture. Drawing on recommendations from the literature review and the relationship-focused conceptual model (Figure A.1), we have outlined a measurement framework for the Q-CCIIT observational measure (Appendix B). Through an iterative development process, the Q-CCIIT measure will be created, tested, and refined to observe child care settings with levels of reliability and validity that meet or exceed the standards in the field. Item development will include further consultation of the literature as well as video review. As part of the iterative process, pretesting of observation components was conducted and the observers and measure development team debriefed on items and procedures. Once an almost-final measure is prepared, we will conduct a pilot test, to be followed by a psychometric field test with the final measure. With the completion of the data collection activities, the Q-CCIIT project will develop a detailed plan to ensure the sustainability of the new infant/toddler child care quality measure.

A.2. Purpose and Use of the Information Collection

The Q-CCIIT project will address a crucial need to identify and measure quality in infant and toddler nonparental care settings. Building on a strong, research-informed conceptual framework, the Q-CCIIT measure will have the capacity to provide information about the quality of caregiver-child interaction across diverse settings, caregivers, and children and will measure constructs that relate to child development. The information collected will be used to understand the psychometric soundness of the new measure—its reliability and validity to measure what it aims to measure across people and time. We will examine the Q-CCIIT measure’s characteristics by subgroups of setting type (center-based versus FCC) and by age category (infant, toddler, and potentially mixed-age groups) to ensure its utility across diverse settings and ages. With a literature review of the existing and emerging research base as a foundation, the Q-CCIIT project will begin developing and testing a child-caregiver measure for infants and toddlers. Once the development of the Q-CCIIT measure is complete (see section A.1), data collection to refine and test the psychometric properties of the measure will involve multiple phases, which include a pilot test and a psychometric field test (Table A.1).

Pilot test. A pilot test in early 2012, involving 60 classrooms⁵ from two geographic locations, will allow us to operationalize the almost-final measure and examine the psychometric properties of the measure (including its reliability and validity). As part of conducting the 3-hour observation, we will request to observe a short (fewer than 10 minutes) small-group activity; such as shared book time, during the full observation (for caregivers already do such activities). After the observation, caregivers who spend more than 4 hours a day in the classroom will complete a 20-minute background questionnaire (on topics such as education, experience, child care practices, and depressive symptoms) that could account for variation in observed interactions. The questionnaire also includes brief follow up questions to the observation about certain key events (for example, communication among caregivers transitioning into the classroom).

Psychometric field test. The psychometric field test, to be conducted in fall 2012, aims to document the psychometric properties of the final Q-CCIIT measure. It will involve 400 classrooms from 10 geographic locations. Its components include the Q-CCIIT observations, validation observations with a second measure of quality (Appendix C), and collection of caregiver background questionnaires. We will collect reliability evidence for both test-retest reliability (with repeat observations at two time points for 60 classrooms) and inter-rater reliability (across paired observations). We will collect validity evidence through concurrent validation observations (in 200 classrooms).

Setting recruitment activities. For both the pilot test and the psychometric field test, Mathematica site coordinators will recruit settings into the Q-CCIIT project. We expect to obtain the names and contact information for a group of potentially eligible settings from public sources such as state and local child care resource and referral agencies. Site coordinators will also attempt to identify the name of the proprietor (that is, the center director or FCC owner) by searching databases and internet sites. Advance letters will be sent to settings, followed by a call to answer any questions the proprietors might have, to determine their eligibility to participate,⁶ and to enroll their setting into the project. Specifically, the Mathematica site coordinator will follow the basic topics discussed in the advance letter, including reviewing the data collection activities, giving tokens of appreciation, and responding to questions. As part of this initial call, the Mathematica site coordinator will collect information on the number of classrooms and the number of infants and toddlers in them, as well identify a setting point person (SPP) who will assist with recruiting families and scheduling the observation visits.

About a month before the start of data collection, Mathematica site coordinators will contact the designated SPPs at each setting to collect more specific information on eligible classrooms affiliated with the center or FCC setting. SPPs will gather classroom child rosters⁷ and update classroom information to assist in completion of a child care setting recruitment form. Site coordinators will collect the age ranges of the classroom and in some cases the birthdates of the children from birth to 36 months of age in each of the eligible classrooms. We will also collect the languages spoken in the classroom, as well as staffing and classroom schedule information. We will use this information to determine when data collectors will visit the classroom and whether data collectors need to be bilingual. We estimate that the gathering of this information prior to the visit will take 30 minutes per setting.

Table A.1
Q-CCIIT Data Collection Plan

	How Instrument Is Collected	Pilot Test (Early 2012)	Psychometric Field Test (Fall 2012)
Child care setting recruitment form	Paper/Phone	80	300
Q-CCIIT observation	Paper	60	400
Infant classrooms		15	100
Toddler classrooms		15	100
Family child care		30	200
Test-retest		n.a.	60
Caregiver background questionnairea	Paper SAQ	120	800
Validation observation	Paper	n.a.	200

^a All caregivers in an observed classroom will complete a questionnaire. On average, we expect two caregivers per classroom.

n.a. = not applicable; SAQ = self-administered questionnaire.

A.3. Use of Improved Information Technology and Burden Reduction

Overall, the proposed data collection does not lend itself to methods involving information technology. We will use paper-and-pencil SAQs to collect caregiver background characteristics (a 20-minute, one-time questionnaire). Access to the internet can be limited in child care settings; given the time available and the anticipated technology resources of respondents, online or web versions are not feasible.

A.4. Efforts to Identify Duplication and Use of Similar Information

From our review of the measures available in the field and expert consultation, we found no evidence of any other assessments that offer information as comprehensive as the Q-CCIIT measure. Since the main goal of this project is measurement development, existing administrative data sets are also not appropriate.

No existing measures or data provide the in-depth assessment of child-caregiver interactions that are the focus of the new Q-CCIIT measure. Existing observational measures of infant and toddler interactions have focused primarily on parent and home settings rather than nonparental child care settings. There is no evidence of any existing measures that fulfill the same role and have the same psychometric validation as the planned Q-CCIIT measure.

A.5. Impact on Small Businesses or Other Small Entities

Small businesses might be affected, since they will likely make up a portion of the FCC and community-based child care centers. Data collection for all settings includes a child care setting recruitment form and for all caregivers includes a background questionnaire (on topics such as education and experience). So that burden on setting staff is minimized, the recruitment form will take no more than 30 minutes and the caregiver instrument will take no more than 20 minutes.

A.6. Consequences of Collecting Information Less Frequently

The data collection includes a pilot test and a psychometric field test. We will recruit different child care settings for the two phases. In each phase, information is collected at one time point. The data collection cannot occur less frequently, as the two phases support the development and refinement of the Q-CCIIT measure.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances requiring deviation from these guidelines.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (P.L. 104-13) and OMB regulations at 5 CFR Part 1320 (FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection. The first notice for the Q-CCIIT data collection was published in the Federal Register, volume 46, no. 76, page 12966 on March 9, 2011. No public comments were received during the 60 days following that announcement.

Members of the Q-CCIIT expert panel have been contacted for advice on various aspects of the study design and the data collection instruments. Their feedback was obtained through in-person meetings, telephone conversations, and written comments. Members of the Q-CCIIT expert panel are listed in Table A.2.

TABLE A.2

Q-CCIIT MeaSUREMENT DEVELOPMENT Expert Panel Membership

Name	Affiliation
Robert Bradley	Arizona State University
Judith Carta	University of Kansas
Martha Edwards	Ackerman Institute for the Family
Karen Heying	Zero to Three
Judith Jerald	Save the Children
Ronald Lally	WestEd
Tammy Mann	Campagna Center
Lori Roggman	Utah State University
Susan Sandall	University of Washington
Kathy Thornburg	University of Missouri, Center for Family Policy and Research
Deborah Lowe Vandell	University of California, Irvine

A.9. Explanation of Any Payment or Gift to Respondents

We recognize that participation in the Q-CCIIT project will place some burden on the child care settings and caregivers. We have attempted to minimize this burden through our data collection procedures and our use of carefully constructed instruments and assessments. Nevertheless, at each phase of the project we should acknowledge the burden that participation entails. Our plan to provide tokens of appreciation is based on those used effectively in previous projects and attempts to acknowledge respondents’ efforts in a respectful way. As described below, ACF has structured the incentives to be provided at each level of data collection—the setting level and the caregiver level:

• For his/her assistance in organizing data collection, we will provide each SPP a $25 gift card to purchase materials to use with the children at the center or FCC home.

• For each classroom observation coupled with completion of the caregiver background questionnaire, we will provide each caregiver with a $25 gift card.

• During the psychometric field test, we will also offer an additional $25 gift card for classrooms selected for a second observation to assess test-retest reliability.

A.10. Assurance of Privacy Provided to Respondents

This project is being conducted in accordance with all relevant regulations and requirements, including the Privacy Act of 1974 (5USC 552a), the Privacy Act Regulations (34 CFR Part 5b), and the Freedom of Information Act (5 CFR 552) and related regulations (41 CFR Part 1-1, 45 CFR Part 5b, and 40 CFR 44502). All data collectors will be knowledgeable about privacy procedures and prepared to describe them in detail or to answer any related questions raised by respondents.

We have crafted consent forms and advance materials (Appendix E) that explain in simple, direct language the steps we will take to protect the privacy of the information about each participant. During the Q-CCIIT observations, caregivers and children will engage in typical classroom activities. All caregivers will receive documents explaining the project, how they were selected to participate, what their participation entails, and how the data will be used. They will also receive information about privacy protections, which will be repeated in the introductory comments of observation visits. For self-administered questionnaires (SAQ), caregivers will be asked to read and sign a consent form informing them of their rights as a study participant. Caregivers will also be read an introduction to the SAQ that explains their right to refuse to participate altogether, and tells them they may choose not to answer any individual question or set of questions if they do participate. In all cases, assurances of privacy will be given to each participant as he or she is recruited for the project. Individuals will be assured (1) that their responses will not be shared with the child care setting, (2) that all responses will be reported in aggregate form only, and (3) that the data collected are for research and education purposes only and will never be used for monitoring or accountability purposes. The Q-CCIIT fact sheet makes it clear that participation is voluntary (Appendix E). We have received IRB clearance based on these procedures which assures privacy and protection of participants’ rights.

As a further guarantee of privacy, identifying information will be maintained in separate tables in the database, which will be linked to the data entry screens only by a sample identification number. Personal identifiers that could be used to link individuals with their responses will be removed from all completed questionnaires and stored under lock and key at the research team offices. Access to the file linking sample identification numbers with identifying information will be limited to a small number of people with a “need to know.” In addition, all Mathematica staff are required to sign a confidentiality statement (see Appendix F).

Access to hard-copy documents will be strictly limited. Completed paper instruments are stored in secure facilities. Mathematica’s data systems and their associated databases are secured behind a firewall between the local area network and any external internet connection; all data stored on Mathematica’s servers are encrypted with a FIPS 140-2 complaint encryption program validated under the Crypotgraphic Module Validation Program, and data are available only to staff associated with the project through password protection and encryption keys.

A.11. Justification for Sensitive Questions

To assess the psychometric properties of the new Q-CCIIT measure in distinguishing levels of quality, we will be asking some sensitive questions as part of the caregiver background questionnaire in the pilot and psychometric field tests, including some aimed at assessing feelings of depression. Caregivers’ mental health can affect their ability to provide supportive and enriching care, which would be assessed in an observation of caregiver-child interactions. The questions are from standardized measures or have been used extensively with nonparental caregivers or teachers in prior studies with no evidence of harm (for example, in the Early Head Start Family and Child Experiences Survey and the Head Start Family and Child Experiences Survey). All respondents will also be assured that their identity will be kept private. Data from these questionnaires will be reported in aggregate. None of the responses will be shared at the individual level, so there will be no risk of disclosure. Each caregiver will receive an envelope to conceal the completed SAQ for privacy.

A.12. Estimates of Annualized Burden Hours and Costs

The proposed data collection does not impose a financial burden on respondents, nor will respondents incur any expense other than the time spent participating. The estimated annual burden for project respondents—caregivers—is listed in Table A.3. Response times are derived from previous studies using the same instruments with a similar population. The total annual burden is expected to be 283 hours for all the instruments.

Estimates of annualized costs

To compute the total estimated annual cost, we multiplied total burden hours by the average hourly wage for each adult participant, based on median weekly wages from the Bureau of Labor Statistics, Current Population Survey estimates (2009 annual estimates by occupation). The results are in Table A.3 below. For caregivers, we used the median salary for child care providers ($9.18 per hour).

TABLE A.3
ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST

Instrument	Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours	Average Hourly Wage	Total Annual Cost
Child care setting recruitment form	173	1	0.5	87	$9.18	$798.66
Q-CCIIT measure-group activity	260	1	0.17	44	$9.18	$403.92
Caregiver background questionnaire	460	1	0.33	152	$9.18	$1,395.36
Estimated Annual Total				283		$2,597.94

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

Not applicable.

A.14. Annualized Cost to Federal Government

The total cost to the federal government for Q-CCIIT under the terms of the contract to Mathematica is $4,061,923.00. The cost for the data collection elements is $2,574,614.00. These costs include the sampling, data collection, data processing, and data coding. Respondent tokens of appreciation are also included in the costs.

A.15. Explanations for Program Changes or Adjustments

None; this is a new collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Analysis plans

The pilot phase of Q-CCIIT (as described in A.2) serve to inform the psychometric field test for the final Q-CCIIT measure. Quantitative analyses will use the data collected in the pilot test to conduct preliminary psychometric analyses and make any adjustments needed to the measure. We will use the data from the psychometric field test to demonstrate the new measure’s reliability and validity. The analyses will be completed in the following ways:

Reliability estimates. To ensure that the Q-CCIIT observation provides reliable measurement of the quality of caregiver‐child interaction for infants and toddlers, we will calculate both item- and scale-level descriptive statistics, estimates of reliability as measured by the coefficient alpha (internal consistency), and item-total correlations. We will also estimate (1) test-retest reliability by calculating Pearson correlations and (2) inter-observer reliability by calculating Pearson correlation, percent agreement, and Kappa coefficient.

IRT analysis. To examine item functioning within subscales and evaluate the reliability and validity, we will conduct one-parameter IRT analysis (Rasch model) to generate interval-level scores. The reliability estimates, ordering of item difficulties, factor analysis of residuals, and fit statistics produced by Rasch models can provide evidence for evaluating the validity and reliability of the quality measure and identifying optimal rating scale categories. We will conduct this analysis in the pilot test and make any needed revisions before the psychometric field test. With data from the psychometric field test, we will examine the model’s replicability for specified subgroups of interest (for example, infant or toddler classrooms, center or family child care).

Confirmatory factor analysis (CFA). We will perform CFA to test hypotheses corresponding to the theoretical notions of the quality of caregiver‐child interaction for infants and toddlers. We will use Mplus (Muthén and Muthén, 2007) to examine the relationships between observed indicators and primary latent factors and the correlations between latent factors. In addition to CFA for the overall sample of settings, we will perform subgroup analysis to test whether the factor structure of the new measure is the same in specified subgroups such as setting (center-based or FCC) or by age (infant or toddler classrooms).

Convergent validity. We will examine evidence of convergent validity of the Q-CCIIT measure with the validation observation tool.

Time schedule and publications

A sustainability plan (slated for summer of 2013) will detail guidelines on training and materials and processes for producing future editions of the measure and conducting ongoing, appropriate psychometric testing to ensure distribution of the measure for widespread and appropriate use. The psychometric analysis report (planned for fall of 2013) will present analyses and results from the psychometric field test.

A.17. Reason(s) Display of OMB Expiration Date Is Inappropriate

Approval not to display the expiration date for OMB approval is not requested.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

REFERENCES

American Academy of Pediatrics, American Public Health Association, and National Resource Center for Health and Safety in Child Care and Early Education. (2002). Caring for Our Children: National Health and Safety Performance Standards: Guidelines for Out-of-Home Child Care Programs, 2nd edition. Elk Grove Village, IL: American Academy of Pediatrics, and Washington, DC: American Public Health Association.

Boller, K., Del Grosso, P., Blair, R., Jolly, Y., Fortson, K., Paulsell, D., Lundquist, E., Hallgren, K., & Kovac, M. (2010). The seeds to success modified field test: Findings from the impact and implementation studies. Princeton, NJ: Mathematica Policy Research.

Muthén, L.K., & B.O. Muthén. (2007). Mplus User’s Guide, 5th ed. Los Angeles: Muthén & Muthén.

National Research Council, Committee on Family and Work Policies. (2003). Working Families and Growing Kids: Caring for Children and Adolescents, Eds. Eugene Smolensky and Jennifer Appleton Gootman. Washington, DC: National Academies Press.

NICHD Early Child Care Research Network. (2002). Child-Care Structure  Process  Outcome: Direct and indirect effects of child-care quality on young children’s development. Psychological Science, vol. 13, no. 3, pp. 199-206.

Phillipsen, L. C., M. R. Burchinal, C. Howes, & D. Cryer. (1997). The prediction of process quality from structural features of child care. Early Childhood Research Quarterly, 12, pp. 281-303.

Sameroff, A. (2009). The Transactional Model of Development: How Children and Contexts Shape Each Other. Washington, DC: American Psychological Association.

Sameroff, A. J., & M. J. Chandler. (1975). Reproductive risk and the continuum of caretaking casualty. In Review of Child Development Research, vol. 4, Eds., F.D. Horowitz, M. Hetherington, S. Scarr-Salapatek, and G. Sigel. Chicago, IL: University of Chicago Press.

Shonkoff, J. P., & D. A. Phillips. (2000). From Neurons to Neighborhoods. Washington, DC: National Academies Press.

Vandell, D. L., & Wolfe, B. (2000). Child care quality: Does it matter and does it need to be improved? Madison, WI: University of Wisconsin-Madison, Institute for Research on Poverty.

Zaslow, M., I. Martinez-Beck, K. Tout, & T. Halle. (2011). Differing purposes for the measurement of quality: Implications for selection of measures, for carrying out the measurement process, and communicating results. In Quality Measurement in Early Childhood Settings, Eds., M. Zaslow, I. Martinez-Beck, K. Tout, and T. Halle. Baltimore, MD: Brookes Publishing.

3 We use the term caregiver throughout this document to signify teachers or providers in center-based and family child care settings.

4 In this document, the term classrooms signifies groupings in center-based and family child care settings.

5

6 Eligible settings must have a classroom with at least two children from birth to 36 months of age at the time of data collection.

7 Because some settings could have concerns about providing personal identifying information, Mathematica site coordinators will be prepared to work with SPPs to use alternate means of creating identifiers, such as recording children’s initials rather than their names on rosters.