CCAMPIS Phase I Memo

MEMO TO: Patricia Butler

FROM: Wendy Mansfield, Kirsten Barrett

DATE: April 27, 2007

PAGE: 1

APPENDIX G

CCAMPIS Phase I MEMO

MEMORANDUM

TO: Patricia A. Butler, Ph.D.

FROM: Wendy Mansfield, Ph.D. DATE: 4/26/07

Kirsten Barrett, Ph.D. 6208-720-43

SUBJECT: Results of Phase I Survey

Mathematica Policy Research (MPR) conducted the Phase I data collection for the Child Care Survey of Postsecondary Institutions (referred to hereafter as the Child Care Survey) between January 16, 2007, and April 13, 2007. Seventy-five institutions were randomly selected from the full study sample to participate in Phase I; 38 were Child Care Access Means Parents in Schools (CCAMPIS) grantees (24 from the 2001 cohort and 14 from the 2002 cohort), and 37 were nongrantees.

This memo describes the Phase I results in three parts. Part I provides an overview of the data collection, including response rates, procedures implemented to encourage responses, respondents’ use of the worksheets designed to facilitate completion of the survey, and respondents’ experiences completing the survey. Part II examines data quality by reviewing the completeness of responses to the three survey sections. (Appendix A summarizes the responses to each survey question.) Part III presents considerations for the Phase II data collection based on the Phase I experience.

A. OVERVIEW

1. Response Rates

Fifty-seven of the 74 eligible institutions completed the survey. This represents an overall response rate of 77 percent, including 84 percent of CCAMPIS institutions and 69 percent of non-CCAMPIS institutions (Table 1).¹

TABLE 1: RESPONSE RATES

Status	Total		CCAMPIS		Non-CCAMPIS
	Number	Percent	Number	Percent	Number	Percent
Completes	57	77.0	32	84.2	25	69.4
Partials	5	6.8	2	5.3	3	8.3
Nonrespondents	10	13.5	4	10.5	6	16.7
Refusals	2	2.7	0	0.0	2	5.6
Total	74	100.0	38	100.0	36	100.0

Note: There are 36 non-CCAMPIS institutions. Although there were 37 in the initial sample, one institution was found to be ineligible. The determination was made too late in the data collection period to replace the institution in Phase I.

2. Procedures Implemented to Encourage Responses

Table 2 lists the various activities MPR implemented to encourage respondents to complete the Child Care Survey. Data collection began January 16, and respondents were asked to complete the survey by March 16. Completed surveys trickled in slowly during the first seven to eight weeks of the 12.5-week data collection period and arrived more rapidly during the last five weeks (Figure 1). The March 7 email prompt, which reminded respondents of the March 16 deadline, may have been a catalyst for the late surge.

Table 2: Activities to Encourage Survey Completions

Activity	Date
U.S. Department of Education (ED) sent advance letters with personalized salutations to all Phase I sample members.	1/10/07
National Coalition for Campus Children's Centers (NCCCC) sent an email promoting the study to its listserv.	1/10/07
MPR emailed login IDs and passwords to sample members.	1/16/07
MPR emailed a reminder prompt.	1/30/07
MPR began placing reminder telephone calls.	2/13/07
MPR emailed a second reminder prompt.	3/07/07
MPR began calling respondents who submitted incomplete surveys.	3/07/07
ED emailed CCAMPIS nonresponders to encourage them to complete the survey.	3/16/07

Figure 1: Cumulative Number of Completions, by Week

3. Respondents’ Use of the Survey Worksheets

MPR created two worksheets to help sample members complete the survey. The Student List Worksheet was designed to help child care directors determine the number of students using the institution’s child care services each year (from 2001 to 2006), and the Data List Worksheet was created to help them collect data needed from another institutional office. The former was accessed by 23 percent of respondents and the latter by 17 percent.²

4. Respondents’ Experience Completing the Survey

We telephoned respondents to determine what difficulties they encountered obtaining the requested data and how much time it required to complete the survey; respondents also telephoned our help desk to ask questions. This section examines what we learned from those calls.

Barriers to Data Access. We received feedback on challenges to obtaining Pell Grant data from 19 institutions (63 percent of the 30 respondents who submitted surveys without such data). The most prevalent impediment (cited by 47 percent) was the inability or unwillingness of the financial aid office to release data (Table 3). Another common obstacle (cited by 37 percent) was the amount of time needed to collect the data (discussed below).

Table 3: Barriers to Data Access

Barrier	Percent	Number
Financial aid office did not release data about students	47	9
Unwilling to release data due to confidentiality concerns, particularly when the on-campus center was a noninstitutional entity under contract to the institution Unable to release data (for example, if the child care center did not collect Social Security numbers, the financial aid office could not generate a student report) Data not gathered or stored by the financial aid office		5 2 2
Respondent too busy	37	7
Child care director/staff too busy Annual reports consuming time Records available only in hardcopy Busy providing child care Financial Aid Office Too busy		5 2
Center does not serve Pell Grant students	16	3
Total	100	19

Respondent Burden. One-quarter of the respondents (13) provided estimates of the time required to complete the survey (Table 4). For those not submitting Pell Grant data (for questions C2, E1, F1, and K1), estimates ranged from 20 to 45 minutes (for four respondents) to 2 hours (for two other respondents). Those reporting Pell Grant data took from 45 minutes to 4.5 hours (for five respondents) to 10 to 20 hours (for two other respondents).

Table 4: Estimated Completion Times

Completion Times
Respondents Not Providing Pell Grant Data	Respondents Providing Pell Grant Data
20 minutes	45 minutes
30 minutes	2.5 hours
30 minutes	3 hours
45 minutes	3.5 hours
2 hours	4.5 hours
2 hours	10 hours
	20 hours
Median: 0.6 hours	Median: 3.5 hours
Mean: 1.3 hours	Mean: 6.3 hours

Respondent Questions. MPR’s toll-free help desk received 32 calls from sample members. The largest proportion of calls (28 percent) concerned access to the website, lost usernames and passwords, and survey navigation difficulties (Table 5). The next most frequent reasons for calling were to name an alternative respondent at the institution (16 percent), learn more about the survey and the intended use of its results (16 percent), and clarify error messages (13 percent). All issues were resolved quickly.

Table 5: Summary of Help Desk Calls

Reason for Call	Percent	Number of Calls Received
Seek help with accessing website, recovering lost username/password, and/or navigating through web survey	28	9
Identify alternative person to complete the survey	16	5
Request additional information about the survey and intended use of the findings	16	5
Clarify error messages	13	4
Clarify data requested (e.g., what centers to report on, what time period to report)	9	3
Request due date extension	6	2
Indicate not currently providing child care	6	2
Provide general feedback about survey content and length	6	2
Total	100	32

B. DATA QUALITY

This section discusses the completeness of the data gathered in each of the three sections of the Child Care Survey. (See Appendix A for question-by-question results.) Respondents reported almost all of the data for 7 of the 12 survey subsections (Table 6). These covered two of the three institutional-level sections (Subsection A. Funding for Child Care Services and Subsection B. Institutional Resources and Referrals); four of the six sections concerning on-campus centers (Subsection D. Operations and Accreditation, Subsection G. Access and Services, Subsection H. Staffing, and Subsection I. Fees and Subsidies); and one of the three sections on off-campus centers (Subsection J. Operations and Accreditation). Respondents typically provided half or less of the data in four other survey subsections that requested Pell Grant information (Subsections C, E, F, and K). This was attributed to respondents’ inability to obtain data from the financial aid office (or other institutional office), time constraints, or lack of records. The remaining subsection on fees and subsidies for off-campus centers (Subsection L) also had few respondents reporting data. Below we review the item response rates by subsection, and then we summarize respondents’ sources of information for reporting Pell Grant data.

Table 6: Range in Item Response Rates

Survey Section	Range in Item Response Rates
	Highest Item Response Rates (percent)	Lowest Item Response Rates (percent)
Section I – Child Care Services
Funding for Child Care Services	100.0	90.7
Institutional Resources and Referrals	100.0	82.4
Pell Grant Recipients Persistence and Graduation	100.0	13.5
Section II – On-Campus Child Care Centers
Operations and Accreditation	100.0	90.0
Children Using On-Campus Child Care Centers	90.5	20.0
Postsecondary Students Using On-Campus Child Care Centers	82.4	17.6
Access and Services	100.0	71.1
Staffing	97.7	88.9
Fees and Subsidies	100.0	31.2
Section III – Off-Campus Child Care Centers
Operations and Accreditation	100.0	36.4
Postsecondary Students Using Off-Campus Child Care Centers	20.0	**
Fees and Subsidies	72.7	**

Note: Percentages were based on the number of respondents eligible to answer the question. For purposes of this table, missing data and “don’t know” or “data not available” responses were treated as having not provided data. Partial responses to questions with multiple items were treated as having provided data.

**N < 5

1. Section I: Child Care Service

Section I asked about the types of child care services provided at the postsecondary institution and the funding for those services, about institutional resources and referrals, and about Pell Grant recipients’ persistence and graduation. Nearly all respondents provided answers to almost all questions in Subsection A. Funding for Child Care Services and Subsection B. Institutional Resources and Referrals, while about half of respondents reported a lack of available data in Subsection C. Pell Grant Recipients: Persistence and Graduation Information.

In Subsection C, respondents were asked to specify the number of Pell Grant recipients using child care services in each academic year from 2001-2002 to 2006-2007 and the percentage of those who persisted and graduated (question C2). For each year, approximately one-half of respondents (43 to 64 percent, depending on the year) indicated that data were unavailable. Of the remaining half, 18 to 75 percent (depending on the item) simply left missing responses. Thus, 13 to 46 percent of all respondents (depending on the item) reported persistence or graduation data. Similar findings were realized when respondents were asked to report the unduplicated count of Pell Grant students since 2001-2002 and their graduation and retention rates (questions C4 and C5).

Subsection C also asked respondents if child care assistance for Pell Grant recipients had improved persistence and graduation rates among these students (question C7). More than half of respondents (54 percent) selected the “don’t know” response. In later telephone discussions, respondents indicated that they lacked data to answer this item and that they felt uncomfortable basing a response on their own observations and experience. Although respondents were next asked to report their information source—with “expert opinion” offered as an option—this follow-up item (question C8) was hidden unless and until respondents indicated student rates had improved. Thus, respondents who specified “don’t know” in question C7 may not have realized that drawing solely from their experience was appropriate.

2. Section II: On-Campus Child Care Centers

Section II contained questions about on-campus child care centers: children using the centers, students using the centers, center access and services, and fees and subsidies. Response rates varied greatly for these questions, with lower rates found for items requesting Pell Grant data from earlier academic years. Item response rates were high for Subsection D. Operations and Accreditation. Of the 83 percent of respondents who reported that they currently have one or more on-campus child care centers (question D1), 90 percent or more answered the remaining questions in this subsection.

Item response rates were mixed in Subsection E. Children Using On-Campus Child Care Centers. Respondents were asked to report the number of children served at on-campus child care centers in each year from 2001-2002 to 2006-2007 and the number of these children with a parent who was a Pell Grant recipient (question E1). Depending upon the year, between 18 and 40 percent of respondents indicated that they lacked data to answer the question, with the 2006-2007 academic year having the least missing data. Of the remaining respondents, 38 to 59 percent (depending on the item) simply left missing responses. Thus, 24 to 80 percent of all respondents (depending on the item) with at least one on-campus center reported data on children with a parent who was a Pell Grant recipient. Among respondents who reported one or more children with parents who were Pell Grant recipients in 2006-2007, 43 to 91 percent answered the remaining questions (E2 to E6) in this subsection on the children’s characteristics.

In Subsection F. Postsecondary Students Using On-Campus Child Care Centers, item response rates were similarly mixed, but somewhat lower. The lower rates are consistent with respondent reports that information about children served is more readily available than information about their parents. In this subsection, respondents were asked to report the number of Pell Grant recipients who enrolled children in on-campus child care centers in each year from 2001-2002 to 2006-2007 (question F1). Depending on the year, between 33 and 56 percent of respondents indicated that they lacked data to answer this question, with the 2006-2007 academic year having the least missing data. Of the remaining respondents, 32 to 45 percent (depending on the year) simply left missing items. Thus, 31 to 60 percent of all respondents (depending on the item) with at least one on-campus center reported data on Pell Grant recipients who enrolled children in on-campus centers. Among respondents who reported one or more Pell Grant recipients with children in on-campus child care in 2006-2007, item response rates ranged from 24 to 82 percent for the remaining questions (F2 to F8) in this subsection on the students’ characteristics.

Item response rates were relatively high for items in Subsection G. Access and Services and Subsection H. Staff at On-Campus Centers. Close to 90 percent of respondents reporting one or more on-campus centers answered questions in these subsections.

Item response rates varied greatly in Subsection I. Fees and Subsidies at On-Campus Centers. Between 8 and 39 percent of respondents were unable to indicate the highest and lowest fees paid by postsecondary students (questions I1 and I2). Among those who could report fees, all specified the factors influencing fees (question I3). About two-thirds of respondents (66 percent) reported providing subsidies at some point in time since 2001-2002 (question I4). Of these, 95 percent were able to provide data about the eligibility criteria for subsidies (question I6), while 31 percent to 43 percent (depending on the year) specified the subsidy amounts by year (question I7).

3. Section III: Off-Campus Child Care Centers

Section III asked about off-campus child care centers. Topics covered include operations and accreditation, characteristics of postsecondary students using off-campus child care, and fees and subsidies for off-campus centers. Item nonresponse (including “data not available” responses) occurred more frequently in Section III than in the comparable subsections in Section II, which concerned on-campus child care centers. Since just two-fifths of respondents reported in Subsection J. Operations and Accreditation having formal or informal working relationships with one or more off-campus child care centers (question J1), the remaining Section III questions applied only to these 22 respondents. Between 18 and 64 percent of them lacked data to answer follow-up items on licensure and accreditation (questions J2 to J6).

Missing data were even more problematic in Subsection K. Postsecondary Students Using Off-Campus Child Care Centers. At least three-fourths of respondents were unable to provide data about Pell Grant recipients using off-campus child care centers in years 2001-2002 through 2006-2007 (question K1). Of the three respondents reporting one or more Pell Grant recipients using off-campus child care in 2006-2007, for any given item, one to two specified that there were no data available to answer the remaining questions in this subsection on students’ characteristics (questions K2 to K8).

Item nonresponse was similarly high in Subsection L. Fees and Subsidies. Respondents were asked to report the highest and lowest fees paid by postsecondary students for off-campus child care and reasons for differences in fees. Three-fourths or more of the respondents (75 percent to 85 percent) selected the “don’t know” option or skipped an item entirely (questions L1 and L2). Of the six respondents who specified that subsidies had been provided since 2001-2002 for off-campus child care to Pell Grant recipients, only one or two (depending on the year) were able to report which years the subsidies were provided (question L4). Of the two to three respondents reporting years in which subsidies were provided, none was able to provide detailed subsidy data for 2001-2002, and only one was able to provide such data for each academic year from 2002-2003 to 2006-2007 (question L6).

4. Information Sources for Pell Grant Data Questions

Half or less of the respondents provided Pell Grant data, and those who did used a variety of sources (Table 7). The largest proportion of respondents consulted either institutional records and/or child care records to determine the number of Pell Grant recipients using child care services and recipients’ persistence or graduation status (questions C3 and C6). When reporting on whether providing child care assistance to Pell Grant recipients resulted in improved persistence and graduation rates and on which factors played a role in this improvement (questions C8 and C10), respondents drew from all the sources listed; the most frequently cited sources were students themselves, the respondent’s experience, and child program records.

Table 7: Sources for Responding to Pell Grant Data Questions

Data Source	Question C3 (N=51) (percent)	Question C6 (N=19) (percent)	Question C8 (N=23) (percent)	Question C10 (N=23) (percent)
Not applicable – no data available	47.1	15.8	--	0.0
Consulted institutional records	31.4	63.2	34.7	17.4
Consulted records from the child care program	45.1	63.2	52.2	52.2
Asked child care center staff	5.9	10.5	30.4	47.8
Asked students	7.8	5.3	56.5	56.5
Relied on your experience	15.7	26.3	52.2	60.9
Other	3.9	0.0	4.4	8.7

Note: Percentages do not sum to 100, as respondents could select multiple sources per question.

C. IMPLICATIONS FOR PHASE II DATA COLLECTION

The Phase I data collection allowed MPR to test the survey procedures, gain feedback from survey participants, and assess the quality of data provided by respondents. Drawing on what we learned from Phase I, we discuss potential modifications to Phase II for ED’s consideration. The most significant matter is whether to retain Pell Grant data questions. We also note a few non-Pell Grant data items with low response rates. Lastly, we propose a change to clarify a question and a procedural change that applies only if ED decides to retain the Pell Grant questions.

1. Pell Grant Data

For the Pell Grant data questions, roughly half or less of the respondents provided data. Thus, a key question for Phase II is whether to retain these questions. There are a few reasons for eliminating these questions:

• The low item response rates may limit the ability to draw conclusions from the data.

• Pell Grant data questions were responsible for the largest part of the respondent burden. As noted above, the estimated completion times ranged from 20 minutes to 2 hours when respondents did not provide Pell Grant data and from 45 minutes to 20 hours when they did provide such data. On average, the latter group of respondents required 5 hours more to complete the survey than the former group.

• The perceived respondent burden is magnified by the Pell Grant data questions. Their inclusion was the catalyst for providing the two worksheets and in turn is likely the impetus for child care directors having to request data from another institutional office. The inclusion of these questions may discourage some respondents from completing the survey.

The importance of the Pell Grant information, however, may argue that it is useful to obtain such data from any respondents who can provide them. One possibility would be to retain a couple of the critical Pell Grant questions that had the highest response rates (Table 8 presents the item response rate for each Pell-Grant-related question in Subsections C, E, F, and K). We recognize, though, that ED may have a greater substantive interest in some of the Pell Grant questions with lower response rates (for example, those providing counts for students by race or ethnicity, or those from earlier years). We would like to discuss this issue further with ED.

Table 8: Response Rates for Items Requesting Pell Grant Data

Section	Item	Response Rate (percent)
C. Pell Grant Recipients’ Persistence and Graduation (N=54)	C2 C4 C5 C7*	13.5 - 46.0 37.0 18.5 - 22.2 46.3
E. Children Using On-Campus Child Care Centers (N=45)	E1d (2006-2007 only) E1d (2001-2002 to 2005-2006) E2 E3 E4 E5 E6	51.1 24.4 - 37.8 35.6 - 37.8 35.6 - 42.4 20.0 - 40.0 42.2 20.0
F. Postsecondary Students Using On-Campus Child Care Centers (N=45)	F1b (2006-2007 only) F1b (2001-2002 to 2005-2006) F2 F3 F4 F5 F6 F7 F8	42.2 24.4 - 38.0 17.8 - 22.2 28.9 - 31.2 20.0 17.8 - 22.2 11.2 - 15.6 6.7 - 17.8 13.3 - 24.4
K. Postsecondary Students Using Off-Campus Child Care Centers (N=20)	K1 K2 K3 K4 K5 K6 K7 K8	0-20 10.0 15.0 15.0 5.0 - 10.0 5.0 5.0 10.0

Note: Percentages were based on the number of respondents eligible to answer the question. For purposes of this table, missing data and “don’t know” or “data not available” responses were treated as having not responded to the item.

2. Other Items with Low Response Rates

There are a few other survey items that ED may want to consider dropping because of low response rates. These concern subsidies and off-campus centers.

Subsidies. Several questions about subsidies obtained low response rates, though only one in the on-campus centers section did so. Of the respondents with at least one on-campus center, less than one-third (31 percent) reported 2006-2007 subsidy amounts for those centers (question I7). Of the respondents with a relationship with at least one off-campus center, most did not provide data for the fees and subsidies questions (Table 9).

Table 9: Item Response Rates for Fees and Subsidies Questions for Off-Campus Centers

Section	Item	Response Rate (percent)
L. Fees and Subsidies (N=20 for L1-L2; N=22 for L3; N<=6 for L4-L6)	L1 L2 L3 L4 L5 L6	25.0 15.0 – 20.0 72.7 0.0 - 33.3 66.7 0.0 – 50.0

Off-campus centers. Another off-campus center question achieved a low response rate: 36 percent specified how long those centers had been operating (question J2).

3. Clarification of a Question

While item nonresponse is often due to a respondent’s inability to readily obtain the requested information, it can also be affected by aspects of the survey question, such as its placement or wording. This may be a factor in the low response rate for question C7, which asked whether child care assistance for Pell Grant recipients had improved persistence and graduation rates among those students. More than half of child care directors (54 percent) selected the “don’t know” response for this item. A follow-up item (question C8) asked for the information source used to respond to question C7, and it included “relied on your experience” as a response option. However, respondents did not see question C8 unless they selected “yes” or “no” for question C7 and thus were unaware of the option to draw from their experience if they did not have relevant data. We suggest showing on the same screen both question C7 and follow-up question C8, as well as adding a prompt to question C7 to indicate that respondents who lack relevant records data should rely on their experience to answer the question. Although some respondents might still be unable or unwilling to answer question C7, others might be encouraged to do so.

4. Procedural Change

If ED decides to retain any of the Pell Grant data questions—which often required the child care director to request information from the financial aid office or another institutional office—we recommend beginning respondent calls shortly after the initial email is sent to respondents. As noted above, most respondents began to submit surveys about one week before their original due date. Often the surveys were submitted without data that the respondents would have had to acquire from the other institutional office. This necessitated follow-up calls to encourage respondents to request the data from that office. If we begin follow-up calls shortly after the start of data collection, we could ensure that child care directors are aware of the potential need to request data from another institutional office, confirm that such requests (if needed) have been made, and refer the directors to the worksheets that will facilitate their making the request (if they have not yet done so).

If ED decides to exclude the Pell Grant data questions from the Phase II survey, the earlier onset of telephone prompts is not necessary.

1 These percentages reflect the final Phase I response rate. The item response rates reported in the remainder of this memo, including Appendix A, are based on the 73 percent of respondents who completed the survey by April 6, 2007.

2 These percentages are based on the full sample of 74 respondents, not just on those who completed the survey or who provided Pell Grant data.

G-1