Pretesting Data Collection (OMB 0970-0355)
Supporting Statement Part B for OMB Approval
Measurement Development: Quality of Family-Provider Relationships in Early Care and Education
A total of three iterative rounds of cognitive interviews will be conducted with 102 English- and Spanish-speaking early care and education providers and parents of children ages birth through five years. Currently, there is no consensus in the literature on sample size requirements for cognitive interviews. It is clear that as sample size increases so does the likelihood of detecting problems in questions (see Blair & Conrad, 2011; Blair, Conrad, Ackermann & Claxton, 2006). Blair and colleagues (2006) found that the mean number of problems found in survey items significantly increased when the sample size was increased from 5 respondents to 50 respondents, but that the rate for identifying unique problems tapered off at a sample size of 20. From our experience with diverse samples, saturation (the point in which the addition of interviews yields redundant data and thus, no new information is learned) is reached at a sample size of at least 12 respondents per round. Given our experience with cognitive testing and that we want to stratify the samples on key variables (e.g., teacher assistants/aids for the provider group), for the present study we plan to conduct 36 interviews with parents and 55 with early care and education (ECE) providers. We plan to conduct only 11 interviews with ECE center providers because we are not stratifying them on key variables and they will only be given a small portion of the survey.
After the cognitive interviews are completed, one pilot test will be conducted. The pilot test will consist of approximately 45 directors, 90 providers/teachers, and 312 parents for a combined total of 447 respondents. The sample size for the pilot test was decided after discussion with a Westat psychometrician, who determined that this sample of providers and parents is of sufficient size to conduct item analysis of each question asked of a varied set of providers and parents in both center and home-based settings. The questions asked of the directors are to provide background and control information for the item analyses.
For the cognitive interviews, Child Trends will recruit participants, determine participants’ study eligibility and schedule a cognitive interview with eligible parents and childcare providers. Participants will be recruited from different communities, programs and care facilities in the Washington, D.C., metropolitan area as well as other large metropolitan areas and rural areas in the United States.
Early care and education providers will be recruited in several ways including: (1) from website (e.g., Craigslist) and newspaper ads (Appendix N); (2) from study fliers posted in strategic places in the community (Appendix O); and (3) from centers, programs, and home-based providers identified from lists compiled by localities and child care referral programs. Programs that are identified from provider lists will be contacted via phone to ascertain their interest in assisting with the study. The study will be briefly described and programs’ will be asked for permission to recruit parents and providers from their center/program (Appendix P). If permission is given, study fliers (Appendix O) will be posted and announcements will be conducted to inform potential participants about the study and how to contact the study center to obtain more information about the study as well as to establish study eligibility.
Parents will be recruited from community-based centers, early education and Head Start programs, from study fliers posted in the community (Appendix O), from website (e.g., Craigslist) and newspaper ads (Appendix N), and, in DC, from on-the-ground recruitment (handing out brochures and describing the study to potential participants) (Appendix T). If potential participants express interest during a recruitment announcement at a center/program or in the community, they will be informed that they may also indicate their interest by noting their names and phone numbers on individual sign-up sheets (Appendix Q) to obtain more information about the study. By using individual sign-up sheets, potential participants will not be able to see the names and phone numbers of the others who have expressed interest.
Sign-up sheets will be transported back to Child Trends offices in a sealed envelope and placed in a locked file cabinet. Child Trends study staff will then call interested individuals to establish eligibility using screeners (see Part A Appendices A, C and E) (see below for screening process). Contact sheets for those who are ineligible or not interested will be shredded on a weekly basis. Additional information from the sign-up sheet, such as e-mail addresses will only be used to contact participants to give them our contact information, if we are unable to reach them by phone. Child Trends has successfully used sign-up sheets in past recruitment efforts of hard-to-reach populations, such as Spanish speaking participants.
A brief screener interview will be conducted with those who express an interest in participating in the cognitive interviews to determine their eligibility for the study. The parent screener (see Part A, Appendix A) will identify those who have at least one child aged birth to five years old in a non-parental early care and education arrangement. The early care and education (ECE) provider screener (see Part A, Appendices C and E) will identify those who provide early care and education services in a home-based, center-based, Head Start, or pre-K program to children ages birth to five years. Additionally, other questions are included in the ECE screener instrument, such as job title and length of tenure, in order to ensure that center directors, teachers, aides and assistants are included as well as those with differing levels of experience. Questions about the level and frequency of communication between parents and providers are also included to ensure that the study includes parents and providers with varying levels of engagement.
If the potential participant qualifies for the study, a cognitive interview will be scheduled at a time that is convenient for him or her. The cognitive interviews (see Part A Appendices B, D, and F) will be conducted in-person and over-the-phone. Recruited participants will receive an email or letter (via air mail) thanking them for agreeing to be part of the study along with the date, time and location of the interview (Appendices R and S). We will also include directions and a map of the area for in-person interviews. Recruited participants will also receive a reminder call and/or text the day before the cognitive interview (see Part A Appendix M). These procedures have been found to minimize the number of cancellations and no-shows.
Each cognitive interview will be approximately 2 hours long; the screener interviews will average 5 minutes for ineligible participants and 12 minutes for eligible participants. Reminder calls for eligible, scheduled participants will average 3 minutes (see Supporting Statement Part A). In person, cognitive interviews will be conducted at Child Trends when possible or at meeting rooms in community centers; interviews will also be conducted via the telephone.
Westat will recruit for the pilot test, which will be conducted in the Atlanta and Seattle metropolitan areas, and possibly in California. Recruiters will begin with a listing of licensed center and home-based care settings. Centers will be recruited first and then providers within centers and then parents within the classrooms of the providers. The process will be the same for home-based providers. However, if we encounter resistance from home-based providers because we have not spoken to the parents first, we may seek parent volunteers by posting ads on public bulletin boards or on Craigslist and then ask parents to identify their home-based providers (Appendix V).
The general procedures for identifying centers and home-based providers are similar to those described above for the cognitive interviews. We will compile a list of licensed providers from child care referral websites in the local communities. For Head Start centers, the Program Information Report (PIR) can be used. The Office of Head Start will also provide us with names of individuals who can help us identify Head Start programs to invite. In addition, the project team has contacts in the communities that we can speak with for suggestions of centers and home-based providers that we might want to invite to participate.
Westat field staff will call listed centers and home-based providers to determine their interest in participating in the study. Once a center or home-based provider expresses an interest in participating, field staff will mail a welcome package to the director/home-based provider and will schedule a visit to answer any questions. The Welcome Package will contain a letter on OHS/OPRE letterhead that welcomes the center/home-based provider to the study, explains the purpose of the study, and explains the importance of participation not only of the center and providers, but also of families (Appendix U). The letter will contain the project director’s name (Christine Nord’s) telephone number and address, as well as an 800 number and email address so that potential respondents can call or send emails if they have any questions or concerns. The package will also include the frequently asked questions (FAQs) and the study brochure (Appendix T).
During the visit, field staff will ask the center director/home-based provider for permission to post the flyers seeking volunteers and will leave study brochures and FAQs for parents and providers (Appendices V, W and X). If the center or home-based provider has mailboxes for the providers or parents, field staff will ask permission to put the flyers, the study brochure, and the FAQs in parents’ and providers’ mailboxes. To the extent that centers/home-based providers allow them to, field staff will also try to recruit both parent and provider volunteers while at the care facility.
As with the cognitive interviews, parent respondents will be asked to complete a short screener to ensure that we obtain a diverse sample of respondents (see Part A Appendix G).
To maximize response rates, Child Trends project staff will conduct cognitive interviews during hours most convenient for parents and early care and education providers (e.g., evenings, weekends). Additionally, we will conduct phone interviews when possible in order to decrease travel burden.
Westat will distribute paper copies of the questionnaires (see Surveys in Part A Appendices H, J, and L) that respondents can complete at times convenient to them. Respondents will be given postage-paid envelopes so that they can return the questionnaires directly to Westat. Westat field staff will ask participants for telephone numbers and email addresses so that the staff can follow-up with participants if the questionnaires are not returned within 10 days. During follow-up calls, Westat field staff will offer participants the opportunity to complete the questionnaires over the telephone. Spanish versions of the parent questionnaires will be available.
The proposed procedures for the cognitive interviews have been successfully applied in other similar studies conducted by Child Trends, and there are no plans to test the procedures. The procedures used in the pilot test have been used by Child Trends and by similar studies at Westat. The pilot test will serve as the test of the procedures for the main field test to be conducted in 2013. A separate OMB package will be submitted for the field test.
The cognitive interview team is led by Drs. Lina Guzman and Nikki Forry, co-principal investigators; Drs. Toni Porter and Juliet Bromer, senior substantive consultants; Dr. Selma Caal, research scientist; Manica Ramos, qualitative analyst; and Katherine Rodrigues, research assistant, Kristen Peterson, Research Assistant, and Ali Chrisler, project manager. Additional staff consulted at Westat (prime contractor) include Dr. Christine Nord, project director.
The pilot test is led by Dr. Christine Nord, project director. Experienced Westat field staff and Westat telephone interviewers will perform the data collection. Dr. Frank Jenkins, a senior statistician, will analyze the results.
References
Blair, J., Conrad, F. (2011). Sample Size for Cognitive Interview Pretesting. Public Opinion
Quarterly, 75 (4), 636-658.
Blair, J., Conrad, F., Ackerman, A.C. & Claxton, G. (May, 2006). The Effect of Sample Size on
Cognitive Interview Findings. Poceedings of the American Statistical Association, Section on Survey Research Methods. Alexandria, VA: American Statistical Association.
Appendices
Appendix N: FPRQ Cognitive Interviews – Website and Newpaper Advertisements
Appendix O: FPRQ Cognitive Interviews – Study Fliers
Appendix P: FPRQ Cognitive Interviews – Script for Program Recruitment
Appendix Q: FPRQ Cognitive Interviews – Individual Sign-Up Sheet
Appendix R: FPRQ Cognitive Interviews – Thank you letter for telephone interviews
Appendix S: FPRQ Cognitive Interviews – Thank you letter for in-person interviews
Appendix T: FPRQ Study Brochure
Appendix U: FPRQ Pilot Test – Center Director Cover Letter and FAQs
Appendix V: FPRQ Pilot Test - Study Advertisements
Appendix W: FPRQ Pilot Test – Parent Cover Letter and FAQs
Appendix X: FPRQ Pilot Test – ECE Provider Letter and FAQs
B-
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | jwest |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |