NonSub Change Memo - HPOG Grant Extensions, Long-term FU updates, Pilot Test results, COVID-19 updates

DATE: June 24, 2020

TO: Josh Brammer

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

FROM: Nicole Constance, Hilary Bruck, and Amelia Popham

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

SUBJECT: Request for Non-Substantive Change to the National and Tribal Evaluation of the 2^nd Generation of the Health Profession Opportunity Grants (HPOG 2.0) (OMB #0970-0462)

This memo requests approval of non-substantive changes to several of the approved information collections for the National and Tribal Evaluation of the 2^nd Generation of the Health Profession Opportunity Grants (HPOG 2.0) (OMB # 0970-0462).

Background

Between August 2015 and July 2019, OMB has approved several information collection instruments, procedures and supporting materials in support HPOG 2.0. These include baseline data collection from HPOG 2.0 National and Tribal Evaluation study participants, data collection protocols and procedures for the HPOG 2.0 Tribal Evaluation, a Short-term Follow-up Survey as part of the National Evaluation’s impact study, and several instruments in support of the National Evaluation, including a phone-based skills assessment pilot instrument and an Intermediate Follow-up Survey to support the HPOG 2.0 National Evaluation impact study.

ACF requests approval of non-substantive changes to several of the previously approved information collections and supporting materials. These requested changes are based primarily on three events: (1) ACF’s decision to extend the HPOG Program grants by twelve months, ending in September 2021 instead of September 2020; (2) the National Evaluation team’s experiences completing data collection for the Short-Term Follow-up Survey (Instrument 12) and the results of the Phone-based Skills Assessment Pilot (Instrument 19); and (3) the COVID-19 pandemic and associated travel restrictions.

Overview of Requested Changes

The factors cited above have led ACF and the National and Tribal Evaluation teams to identify a number of changes to the remaining information collections. Specifically, ACF seeks approval of five non-substantive changes to several previously approved instruments and procedures. There are four changes associated with the National Evaluation and one change associated with the Tribal Evaluation. This section summarizes the specific changes for which ACF seeks approval and the expected benefits.

#1— Revision to Informed Consent Forms A and C

The previously approved versions of the informed consent forms indicated that the evaluation would collect data from administrative sources for one year prior to enrollment and up to five years after enrollment. ACF’s plans for the evaluation have evolved since the informed consent forms were initially approved in 2016, and now plans a longer evaluation period for the National Evaluation. As a result, the informed consent forms need minor revisions to communicate clearly to participants how their data will be collected from administrative sources and used in the research study.¹ ACF seeks approval for minor wording changes in two of the previously approved National Evaluation Informed Consent Forms (Attachment B)—Form A (written consent for those subject to the lottery; approved in 2016) and Form C (verbal consent for those subject to the lottery, approved in April 2020).

#2— Revisions to Instrument 18: Intermediate Follow-up Survey

ACF requests approval for several changes to Instrument 18, which fall into four categories: selecting the final questions for Section J of the survey following completion of the assessment module piloted from September 2019-January 2020²; adding several new questions to correct an omission in the original version and improve data quality, as well as capture information about the effects of COVID-19 on study participants; dropping several questions to reduce the overall instrument length; and adding a new, shortened version of the instrument to capture critical items only (Instrument 18a) for participants who appear to be on the verge of becoming a final refusal. The requested changes to the full instrument are included in revised Instrument 18 HPOG 2.0 Intermediate Follow-up Survey_ REV_June2020, and the shortened version of the instrument is included in Instrument 18a HPOG 2.0 Intermediate Follow-up Survey_Critical Items Only.

• Final selection of questions for Section J: As noted in Supporting Statements A and B approved in July 2019, since telephone interviews with study subjects almost always play an important role in experimental evaluations of these programs, ACF determined it would be useful to have a module that measures program participants’ basic skills over the phone. The Phone-based Skills Assessment Pilot (previously approved Instrument 19) contained a series of 45 literacy and numeracy based assessment questions, to be reduced to 24 items after analysis of the pilot test results. The evaluation contractor completed the pilot data collection and analysis efforts and determined which items to retain in Section J of the Intermediate Follow-up Survey and which items to cut.³ The skills assessment module in the revised Intermediate Follow-up Survey (Section J) includes 11 verbal and 11 math skill questions: three “easy” questions, five “medium” difficulty questions, and three “hard” questions. Students who correctly answer three or more “medium” difficulty questions will then receive the “hard” questions; those who answer two or fewer questions correctly will receive the “easy” questions; no student will answer more than eight questions in each subject. The remaining 23 items conditionally approved in July 2019 will be dropped: J1, J5-J10, J13-J16, J20, J22, J25-J28, J31, J35-J36, J40-J41, and J44.

• Adding New Questions: ACF is proposing to add several questions to the survey. Two of the new questions were unintentionally omitted from Section A of the previously approved version. These items (A24c and A24d) are needed for the Cost-Benefit Analysis and to improve the accuracy and data quality of outcomes related to time spent in training or at work and the child care needs of participants. The other new items will collect information on the effects of COVID-19 on study participants’ education, training, and employment patterns since the beginning of the outbreak in early 2020. The COVID-19 module will provide important context on how COVID-19 affected the key outcomes of interest to the HPOG 2.0 impact evaluation and allow us to look at the effect of COVID-19 on the intersection of work, education, and well-being of study participants and their families. These new questions do not cover any new topics or domains. The proposed COVID-19 module is estimated to be 5 minutes, which will be offset by the proposed cuts described below.

• Dropping Questions to Reduce Administration Time: After conducting the short term survey,⁴ the evaluation contractor revisited the version of the intermediate survey previously approved by OMB. Since many of the items are the same on both instruments, the contractor was better able to estimate that the previously approved version of Instrument 18 would take 67 instead of 60 minutes to administer, inclusive of the final skills assessment module. Dropping some items in Instrument 18 would reduce administration time and participant burden without compromising the ability to answer the evaluation’s key research questions. ACF expects these changes will reduce the overall length of the interview by nearly 25 percent; from 67 minutes to 50 minutes. Even with the addition of the new questions above, we estimate the revised length will be 55 minutes, an 18 percent reduction in the overall length of the survey. The dropped items are listed as “OMITTED” in the revised version of Instrument 18.

• Adding Instrument 18a-Critical Items Only: ACF is seeking approval to add a new version of Instrument 18 (Instrument 18a) that is shorter to administer and includes only the items most critical to the study. ACF requests approval to administer this “critical items” instrument to reluctant respondents as a way to test the viability of this approach for maximizing response rates. Doing so would help maximize the overall completion rate and minimize nonresponse for the key outcomes of interest. This information will also be used to improve imputation procedures for outcomes not collected in the shortened version of the survey.

Version 18a of Instrument 18 can be completed in just 20 minutes, and will be administered to individuals who refuse to respond to the full Intermediate Follow-up Survey. The shortened version of Instrument 18 will be offered as the last refusal conversion effort. Supporting Statement B, Section B2 provides more detail on the critical items survey administration procedures.

#3: Revisions to the Intermediate Follow-up Survey Supporting Materials

ACF reviewed the supporting materials for the Intermediate Survey, specifically Attachment P (the HPOG 2.0 Intermediate Survey Advance Letter) and Attachment S (email reminder text), to determine if there were changes that should be made to those materials to help improve respondent cooperation. The proposed revisions to the advance letter are shown in Attachment P_HPOG 2.0 Intermediate Survey Advance Letter_REV, and intended to streamline the letter. A more streamlined advance letter will allow the interviewers better opportunity to explain the study to the respondents and address questions during the introductory interaction. The revised advance letter will alert participants about the study so they are more receptive to the interviewer outreach, without providing so much detail that the survey may appear overwhelming.

The proposed email reminder text revisions are found in Attachment S_HPOG 2.0 Intermediate Survey Email Reminder Text_ REV. These revisions streamline the language and emphasize the ability to complete by telephone. The revisions also offer greater flexibility in how the text can be used. The revised email text can be used by interviewers as a direct email to study participants or as a letter, sent from the survey director to all survey non-responders.

#4: Minor Increase in Burden Estimate for Instrument

ACF decided to extend the HPOG Program grants by twelve months, ending in September 2021 instead of September 2020. As a result, ACF requests approval for an increase in the number of respondents to Instrument 1 PAGES Participant-Level Baseline Data Collection (for participants at non-tribal grantees), for which OMB most recently renewed approval in July 2019. This instrument collects the baseline information for all incoming HPOG 2.0 study participants. ACF expects that up to 3,500 additional non-tribal participants will enroll in the study during the extended enrollment period.⁵

#5: Revisions to Tribal Evaluation Procedures

Under the previously approved Tribal Evaluation study procedures, qualitative interviews and focus groups occur in person at each of the five Tribal HPOG 2.0 grantee sites. The fourth and final round of site visits was planned for spring 2020. The tribal evaluation team conducted a site visit to one grantee in February 2020. Site visits and associated data collection were postponed to summer 2020 for the other four grantees. It is not likely that the remaining site visits can occur in-person as a result of social distancing guidelines and travel restrictions implemented nationwide to stop the spread of COVID-19. However, Tribal HPOG 2.0 grantee staff and partners are continuing to implement their programs remotely, and it is important that the tribal evaluation team capture for the HPOG 2.0 Tribal Evaluation information about this period of program implementation. In the event that in person data collection in summer 2020 is not feasible, ACF requests approval to conduct remote data collection by phone or virtual meeting in addition to in person. No changes to the previously approved HPOG 2.0 Tribal Evaluation data collection instruments (Instruments 6-11) are needed to accommodate remote data collection. In order to facilitate remote data collection and analysis, the tribal evaluation team will need to obtain consent for participation in interviews and focus groups verbally and would like to record the interviews and focus groups. Therefore, ACF requests approval to add additional versions of the previously approved focus group and interview consent forms (see Attachment I_Focus Group Informed Consent Form_Remote and Attachment J Interview Verbal Informed Consent Form_Remote). The remote version of the informed consent form in Attachment I allows for focus group participants to provide verbal consent to participate and requests permission from participants to record the focus group. The remote version of the consent form in Attachment J requests permission to record a participant’s interview.

The contact person for participants enrolled in the Tribal Evaluation that is listed on the consent form will be changed from Michael Meit (former Tribal Evaluation Project Director) to Carol Hafford (new Tribal Evaluation Project Director). These changes have been made across all consent forms for the tribal evaluation (Attachment I, Attachment J, Attachment B2, and Attachment B3).

Time Sensitivities

ACF requests a response about these changes as quickly as possible in order to allow the Tribal Evaluation data collection to stay on schedule. We also need ample time for the evaluation contractor to program the Intermediate Follow-up Survey instrument and test the programming logic before survey administration begins in September 2020.

1 Study participants that already signed consent forms with the five year restriction will receive a letter noting that ACF’s plans for the evaluation have evolved over time to include a longer data collection period.

2 OMB approved the Intermediate Follow-up Survey in July 2019, with a term of clearance that the Department resubmit the instrument to reflect the final items for inclusion in the assessment module, once the phone-based pilot assessment was complete. The pilot assessment data collection and analysis is now complete. Section J of the Intermediate Follow-up Survey reflects the final item selection. A summary of the analysis methods used is provided in the revised Supporting Statement A, Section A16. ACF can provide a copy of the internal memo describing the methods and results upon request.

3 As noted above, details on the data collection and analysis can be found in Supporting Statement A Section A16.

4 The Short-term Follow-up Survey data collection took place between October 2018 and November 2019.

5 It is unclear whether the grantees will maintain the same pace of enrollment during this extended period, thus the 3,500 additional non-tribal participants is likely the upward limit. There is sufficient burden remaining for the Tribal grantees.

2