Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Feedback After the Office of Planning, Research, and Evaluation’s 2024 Research and Evaluation Conference on Self-Sufficiency (RECS)
Formative Data Collections for Program Support
0970 - 0531
Supporting Statement
Part B
September 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Marie Lawrence and Emily Ross
Part B
B1. Objectives
Study Objectives
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) plans to conduct a survey and convene focus groups to assess the use of conference content following the Research and Evaluation Conference on Self-Sufficiency (RECS) 2024. The study’s objective is to build understanding of specific attendee groups’ goals for attending the conference, experiences at the conference, and use of information they learned at the conference. OPRE will use the information gathered from the survey and focus groups to inform the planning of future conferences.
Generalizability of Results
This study is intended to present internally valid descriptions of the experiences of specific groups of conference attendees, not to promote statistical generalization to other settings or service populations.
Appropriateness of Study Design and Methods for Planned Uses
This data collection includes a survey that will be sent to all conference registrants and focus groups involving a convenience sample of attendees that match the criteria of attendee segments (i.e., groups of attendees with shared characteristics, such as their professional affiliation, mode of RECS attendance, and/or their years of work experience) of interest to OPRE. The survey items will also allow us to assess and report on the extent to which survey respondents are representative of all conference registrants. This approach is appropriate to produce a broad understanding of all conference attendees’ goals, experiences, and uses of conference content, and a deeper understanding of the goals, experiences, and use of conference information by attendees in prioritized segments. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
The target population for this study is RECS 2024 attendees. Attendees generally include researchers, state and local administrators, practitioners, and federal officials and policymakers. We estimate that there will be about 700 in-person attendees and 500 virtual attendees.
Sampling:
For the survey, we will use the nonprobability sampling method of a convenience sample. The contract team will share the survey via email to all RECS attendees and encourage survey participation with reminder emails. This approach will allow the contracted study team to collect survey data efficiently. We expect about 150 responses to the survey, as described in section B5.
Respondent Recruitment: Before beginning recruitment for focus groups, the study team will work to identify four prioritized attendee segments based on the professional affiliations of the attendees (e.g., government staff, academics, non-academic researchers, and program practitioners) and mode of attendance (i.e., those who attended in RECS in-person will be invited to participate in the focus groups).
The contracted study will adopt the following three-pronged approach to recruiting participants for the focus groups:
Information about the proposed focus groups will be shared as part of conference registration and RECS registrants will be provided with an option to express interest in participating.
At RECS, a one-page flyer about the proposed study was shared with in-person and virtual attendees. The flyer included a QR code that attendees could use to express interest in participating in the proposed focus groups (note that only contact information was requested for this initial step and no other information was requested).
Attendees who had not expressed an interest in participating in the focus groups via registration or one-page flyer were emailed an invitation after the conference to express interest in participating.
Using contact information provided by attendees who expressed an interest in participating in the focus groups, the contracted study team will match this data to the RECS registration data to identify the professional affiliations and mode of RECS attendance of each interested attendee.
The contracted study team will email focus group invitations to individuals who (1) expressed an interest in participating in the focus group and (2) match the prioritized attendee segment(s). Participants in each focus group will reflect each prioritized attendee segment. The contracted study will conduct four focus groups, each attended by an estimated six participants.
While the focus groups will only include approximately 24 total RECS attendees who represent the prioritized attendee segment, we will gather information on all RECS attendees’ interest in participating in the focus groups to (1) ensure the contracted study team has enough potential participants per prioritized attendee segment, from which focus group participants can be chosen, and (2) inform OPRE of how many RECS attendees expressed an interest in participating, as this may be valuable information for future iterations of this study.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
In collaboration with the OPRE team, the contracted study team identified research domains for this study, which have guided the development of the survey items and focus group protocol. Some items in the survey have been adapted from the Survey of Practitioners' Use of Research (SPUR) – specifically the items related to conceptual use and instrumental use of research.
To minimize measurement error:
The survey will undergo quality assurance review by the contracted study team prior to being administered.
The survey will include language about privacy.
Most survey items will be multiple choice, with two open-ended items.
The open-ended questions will allow the contracted study team to capture information that may not be otherwise captured in the survey.
The survey tool and focus group guide will undergo testing to ensure the data collection tools are effective and clear.
B4. Collection of Data and Quality Control
Collection of Data
The contracted study team from the American Institutes for Research (AIR) will conduct this information collection, including the collection of data. The following includes the data collection processes.
Survey
All RECS 2024 registrants will be sent a link to the survey for participation. The survey will be administered online via Vovici. The survey will remain open for 2 weeks. During this 2-week period, survey recipients will receive two reminders – one week after the first survey invitation is sent and another two days before the survey is closed.
Focus Groups
We will adopt a three-pronged approach to recruiting participants for the focus groups:
As part of conference registration, RECS registrants were asked if they might be interested in participating in a proposed 90-minute focus group following the conference.
A one-page flyer about the study was shared with attendees at RECS. The flyer included a QR code that attendees could use to express interest in participating in the proposed focus groups.
Attendees who have not expressed an interest in participating in the focus groups via registration or at the conference were emailed an invitation after the conference to express interest in participating.
The contracted study team will match contact information provided by attendees who expressed interest in this proposed data collection to the RECS registration data to identify the professional affiliations and mode of RECS attendance of each interested attendee.
The contracted study team will email focus group invitations to individuals who (1) expressed an interest in participating in the focus group and (2) match the prioritized attendee segment(s). Participants in each focus group will reflect each prioritized attendee segment.
The focus groups will be conducted virtually via Zoom.
Quality Control
The study team will take the following steps to monitor data for quality and consistency:
The contracted study team will review survey progress in Vovici two days after administration to ensure surveys are being completed.
The contracted study team will monitor the project inbox for survey participant concerns and/or questions about the survey.
Focus groups will be conducted by an experienced facilitator who will use a detailed protocol to conduct these focus groups.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The survey will be sent to all RECS registrants, and we will aim to gather responses from in-person and virtual conference attendees. Based on previous RECS surveys conducted following the conference, we expect to receive a response rate of approximately 10–15% for the survey. While this survey sample will likely not be representative of the RECS attendees, we will identify how representative the sample is relative to the full list of RECS attendees and contextualize the findings accordingly.
The focus groups are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported. We will report the total number of focus group participants.
NonResponse
Surveys
Nonresponse bias analyses plans: survey analysis will include analysis to ascertain if specific attendees with specific attributes systematically disregard particular survey items.
Focus groups
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent characteristics will be documented and reported in written materials associated with the data collection.
B6. Production of Estimates and Projections
The estimates produced are for future conference planning and reporting basic descriptive statistics at the aggregate level only.
B7. Data Handling and Analysis
Data Handling
Survey
Most survey items will be developed as multiple-choice– minimizing the possibility of errors due to data entry.
Data will be downloaded as one CSV file.
When developing a new variable based on survey responses (e.g., to identify attendee segments), we will develop code in Excel, which be quality checked to ensure accuracy.
Focus Group
With participant permission, focus groups will be recorded on Zoom. The contracted study team will use autogenerated transcripts and team notes for coding. We will undertake a two-step approach to our data analysis (described below).
Data Analysis
After collecting the data, the contracted study team will analyze the data collected through the survey and focus groups to identify patterns and glean insights on how attendees experience and use information learned at the conference afterward – and identify differences and similarities across attendee segments.
After the survey is closed for participation, the contracted study team will clean the survey data and develop a new variable to indicate attendee segments. The contracted study team will conduct descriptive analyses. For survey items that collect qualitative information, the contracted study team will conduct qualitative analysis to identify patterns and glean insights. When sample sizes are sufficient, we will examine the data by attendee segments (i.e., at least 30 responses representing an attendee segment).
To analyze information shared during the focus groups, we will begin by developing a preliminary coding scheme that maps closely to the key topics and subtopics included in the research focus group protocols guide, as well as codes for attendee segments. The coding scheme will be continuously refined as themes emerge from the data. For each topic, we will document what was learned, indicating where respondents reached similar conclusions or where there were discrepancies from the different attendee segments.
Data Use
No potentially identifying information will be released to the public. All information will be reported at the aggregate level. This study will inform future conference planning and inform a public product(s) (e.g., blog post or brief) sharing the lessons learned on knowledge retention and research use. We do not intend for this information to be used as the principal basis for public policy decisions.
Staff name |
Email Address |
Organization |
Gee Rege |
American Institutes for Research |
|
Amanda Ahlstrand |
American Institutes for Research |
|
Yuliya Cheban |
American Institutes for Research |
Attachments
Appendix A: OPRE Post-RECS Study Recruitment & Data Collection Communication Materials
Appendix B: Data Sources for Instrument 1
Appendix C: Informed Consent Form Virtual Focus Groups
Instrument 1 – OPRE Post-RECS Data Collection Survey Questions
Instrument 2 – OPRE Post-RECS Focus Group Facilitation Guide
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jessica Cruttenden |
| File Modified | 0000-00-00 |
| File Created | 2024-10-07 |