GOALS OMB PART A-FINAL_9-10-15_ revised January 2016

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for the collection of information for the Goal-Oriented Adult Learning in Self-Sufficiency study (hereinafter, GOALS study), funded by the Office of Planning, Research and Evaluation (OPRE) within ACF.

The GOALS study addresses the nexus between the growing knowledge base in the psychological sciences and longstanding approaches to self-sufficiency programs targeted at adults and young adults. The project will explore the programmatic implications of existing research on psychological processes associated with goal-directed behaviors, including socio-emotional regulation and cognitive skills, executive functioning, and related areas. The study will synthesize current research on these topics; address how insights gained from research can be used to promote economic advancement among low-income populations; identify promising strategies to strengthen underlying skills in these areas; and inform measurement of changes and developments in skill acquisition. The study is being conducted by Mathematica Policy Research.

This submission seeks OMB approval for three data collection instruments (Program Director/Administrator Exploratory Telephone Interview Guide, Site Visit Master Interview Guide and Topics by Respondent, and the Site Visit Participant Interview Guide) to be used as part of the GOALS study. The instruments cover two data collection activities:

Exploratory telephone calls. This activity involves conducting telephone calls with program directors/administrators for the purpose of documenting and collecting initial data about program objectives, populations served, how interventions are structured to strengthen psychological processes associated with goal-directed behaviors, and to assess progress toward targeted outcomes. During the calls, a senior member of the study team will conduct a semi-structured phone interview with program directors (Attachment A).

Site visits. This activity involves conducting site visits for the purpose of documenting and collecting detailed data from programs integrating psychology-informed frameworks into their adult self-sufficiency services. During the visits, site teams will conduct semi-structured interviews with key individuals involved with the program including updates from the telephone discussion with program directors/administrators, individual or small group discussions with program staff (case managers, coaches, other specialized staff, and representatives of community partner organizations), and group discussions with program participants (Attachments B and C).

Study Background

With almost 16 million children and 47 million total individuals categorized as living in poverty in the U.S., improving self-sufficiency outcomes among low-income families is a primary goal of policymakers and program administrators. Interventions targeted at improving employment and earnings among young, disadvantaged adults with children have demonstrated little effect over the long term (Hamilton, 2012; Butler et al., 2012). One promising area of recent research and programming focuses on the role psychological processes, particularly cognitive and socio-emotional skills, may play in strengthening goal-directed behaviors. Working memory, impulse control, and flexibility – skills commonly referred to as “executive functions” – may be important in helping individuals manage day-to-day tasks and develop the skills necessary to achieve longer-term goals. In addition to executive functioning, socio-emotional skills, emotion regulation, and non-cognitive skills like perseverance, may play a key role in guiding goal-directedness.

There is evidence that adults who engage in goal-directed activities are more likely to be productive and successful in the workplace and at home, while adults who struggle in these areas tend to face challenges in these roles. Furthermore, evidence from employer surveys indicates that workforce development programs intended to improve goal-directed behaviors among employees are highly valued by employers. In particular, analysts and researchers have noted the importance of socio-emotional (“soft”) skills in employment, training, and human capital development programs.

Current and past circumstances and individual barriers such as exposure to poverty, lack of in-demand skills, parenting responsibilities, and other factors may challenge an individual’s ability to operationalize psychological processes that support goal-directed behaviors and advance self-sufficiency. There has been substantial research on the impact of exposure to adverse experiences in early childhood and adolescence; however, there has been less research on psychological processes that could support adults and young adults in overcoming these contextual challenges. ACF sponsored this study to explore how psychology-informed approaches, frameworks, and interventions can usefully be incorporated into programs aimed at improving job entry, retention, and advancement.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The primary purpose of the exploratory telephone calls is to document fundamental details about programs that include a focus on goal-directed adult behavior to support self-sufficiency. The exploratory calls will inform selection of locations for the site visits. The purpose of the site visits is to obtain in-depth information about the selected programs.

Research Questions

The GOALS study will address the following principal research questions, with the last three being the primary focus of this information collection request (ICR):

How does existing research describe psychological processes associated with goal-directed behaviors?
What does existing research say about improving goal-directed behavior in adults?
What are we learning from existing programs using psychology-informed frameworks?
To what extent have existing programs using psychology-informed frameworks been evaluated?
What are the options for strengthening and measuring goal-directed behaviors in adults and for evaluating these interventions?

Study Design

This study will document the context, features, and implementation of programs using goal-oriented strategies to support adult self-sufficiency. This descriptive, qualitative study is designed to generate information and conceptual frameworks that can be used to inform further development and enhancements of programs, as well as a short- and long-term research agenda designed to inform improvement and development of the adult self-sufficiency intervention evidence base. Program selection for the exploratory telephone calls and for the site visits is discussed briefly below (see Supporting Statement B for more information).

The exploratory telephone interviews will assist the researchers to document program objectives and populations served, interventions to strengthen psychological processes associated with goal-directed behaviors, and the extent to which participants are making progress toward targeted outcomes. During the interviews, the research team will learn about the history and implementation of the program and program context, and will begin discussing the program’s explicit or implicit theory of change (implementation inputs, outputs, and progress toward outcomes). The interviews will also explore how programs have developed new staff skills and ways of doing business that support participants in strengthening goal-oriented skills development. The research team will also explore the degree of standardization of program processes and learn about any evaluations or data-driven decision-making activities in which they have participated. Based on the interviews, the study team will create brief profiles of each program and use them to inform recommendations about which programs to visit.

The research team will conduct site visits to the programs selected for further study. The site visits aim to document evidence on research-based interventions or approaches already being undertaken or being developed by program administrators and will gather more detailed and rich program information and further explore topics that arose in the exploratory calls. During site visits, the research team will document program operations and implementation as well as features and dimensions of the approaches used by program staff to strengthen goal-directed behaviors for adults. The visits will include gathering information from a range of program staff, partners, and participants to support triangulation of the information across respondents. A primary aim of the visits is to carefully document the implicit or explicit theory of change and identify aspects of the program that are innovative or appear promising for further development and evaluation.

Universe of Data Collection Efforts

Addressing the research questions adequately requires collecting detailed data about program implementation, the program logic model, and program context. Data collection will include interviews that gather information on: (1) system inputs, (2) implementation inputs, (3) implementation outputs, (4) short-term outcomes, (5) long-term outcomes, (6) the implementation context, and (7) replicability and sustainability.

ACF is requesting approval for two data collection activities that assess these aspects of the selected programs at different levels of intensity and for different purposes:

GOALS Program Director/Administrator Exploratory Telephone Interview Guide (Attachment A). The primary purpose of the exploratory calls is to document fundamental details about the programs that will inform selection of locations for the site visits and the field about the range of programs working to support adult goal-directed behavior.
GOALS Site Visit Master Interview Guide and Topics by Respondent (Attachment B). The purpose of the site visits is to obtain in-depth information about the selected programs through discussions with program staff and representatives of community partner organizations.
GOALS Site Visit Participant Interview Guide (Attachment C). The purpose of the site visits is to obtain information about how programs provide services designed to strengthen goal-oriented skills through discussions with program participants.

Both data collection activities will gather information on similar topics but the exploratory calls are meant to provide a snapshot of what programs are doing while the site visits are designed to provide an in-depth view of the program and how it operates. Each of these activities is described in more detail below.

Exploratory calls

The exploratory calls will provide a snapshot of each program that the research team will use to recommend site visit locations. The calls will gather information about community context; program philosophy, purpose, and goals; funding and program costs; administrative and staffing infrastructure; target population and client characteristics; core program components and services; coordination and linkages with other services within the community; outcome measurement and evaluation; and program successes, challenges, and lessons. Long-term progress in goal-directed behaviors will be addressed. The information obtained from programs during the exploratory calls will allow the team to describe the programs’ theories of change and articulate how they do or do not conform to various conceptualizations of psychology-informed services. Further, the information will allow the team to make sound recommendations of programs for further study.

Calls will be conducted with up to 24 programs. The research team will call one program director or program administrator from each program for up to 24 total interviews of one hour each. The research team will select programs for the exploratory calls based on the literature review findings (i.e., those with a strong or promising evidence base), nominations from the study’s expert advisory group, and nominations from knowledgeable ACF staff members and partners. Senior members of the research team will invite directors of selected programs to participate in the exploratory calls (Attachment E). Each call will be led by a senior member of the research team.

Site visits

A study exploring how programs can incorporate psychology-informed frameworks into their approaches requires a clear understanding of the services provided by programs currently integrating psychology-informed frameworks in their models. The site visits will collect data about program implementation and operations. The visits will be used to produce a narrative description of the key inputs, outputs, outcomes, and implementation contexts for each program and detailed case studies of the programs visited, including the ways that programs do or do not conform to various conceptualizations of psychology-informed interventions identified in the literature review. They will further enable the development of a conceptual framework to inform future program development and evaluation.

At each site, the team will conduct semi-structured interviews with program administrators, direct service staff, other appropriate staff, community partners, and program participants. The research team will also review documents obtained in advance and on site. The team will tailor site visit activities based on the nature of the intervention, administrative and staffing structure, and involvement of community partners. Contractor staff will tailor discussions based on the individual program structure and allocation of roles and responsibilities.

Below is a list of activities the research team will conduct during each site visit.

Updates from the telephone discussion with program directors/administrators. During the site visits, the team will talk with program directors/administrators to update the information collected during the exploratory calls.
Individual and small group discussions with program staff (case managers, coaches, other specialized staff, and representatives of community partner organizations). The team will conduct individual or small group discussions (of no more than 2-3 respondents per group) with direct service staff and/or community partners. To learn more about the program, the study will examine the program origin and evolution, policies, administrative structure, staffing (for example, types of staff, roles and responsibilities), assessment procedures, services and supports, use of incentives, specialized services (for example, mental health treatment), reporting and data management systems, and the program challenges, successes, and outcomes.
Discussions with program participants. Site visitors will hold small group discussions with between six and eight program participants to learn about their experiences with the program.

Site visits will be conducted in up to 12 programs. An average of up to 15 staff per site will be interviewed during these site visits, for approximately 180 total staff interviews. Site visitors will conduct one small group discussion with approximately 7 program participants at each site, for a total of 84 total discussant participants. (Attachment B).

Based on the exploratory call results and expert input we will invite selected programs (to participate in the site visits. Site visitors will work with program directors/administrators and those they designate to schedule the visits and plan for the group discussions. Program directors will help with recruitment of community partners and with program participants. To augment this data collection, the team will also explore extant data by conducting program observations and reviewing program documents such as performance outcomes and program costs.

A3. Improved Information Technology to Reduce Burden

Due to the nature of the semi-structured interviews and the small sample sizes of both the exploratory calls and the site visits, it is not appropriate to use information technology such as computerized interviewing.

A4. Efforts to Identify Duplication

The information collection will not duplicate information that is already available. The data to be collected during the exploratory calls and site visits are not available from any other source. There is no other data source providing detailed information on the nature and content of programs currently integrating psychology-informed frameworks in their models including the history and implementation status of the program, the program context, staff roles and responsibilities, services offered, implementation fidelity, outcomes, or impacts.

Publicly available documents, as well as documents provided by the programs, will be reviewed by the team to help guide the discussions and potentially shorten them by enabling information to be filled in pre- or post-discussion.

A5. Involvement of Small Organizations

Exploratory calls

It is possible that some of the organizations involved in the exploratory calls will be small. However, the research team expects the impact on these programs to be limited given that the calls will only be one hour long and scheduled at a time convenient to the respondent.

Site visits

It is possible that some of the organizations involved in the site visits will be small. However, the research team expects the impact on these programs to be limited. The field visit and interviews will be scheduled in collaboration with the program staff to minimize disruption on daily activities. The field visit team will conduct group discussions to the extent that it is feasible to do so.

A6. Consequences of Less Frequent Data Collection

All of the data to be collected associated with this specific burden request are one-time in nature. The data collection effort described in this document is designed to provide unique information to answer questions of interest to policymakers.

Exploratory calls

The information collected through the exploratory calls will enable the team to assess the program objectives and populations served and to find out whether and how interventions are structured to strengthen psychological processes associated with goal-directed behaviors. Without this information the study team will not be able to identify programs worthy of further study.

Site visits

The information collected through the site visits will enable the team to describe the program design and operations in each site, produce a narrative description of the key inputs, outputs, outcomes, and implementations context for each program, and create detailed case studies of the programs, including the ways that programs do or do not conform to various conceptualizations of goal-oriented skills interventions from the literature review. Furthermore, the information about program implementation and operations will be used to develop a conceptual framework to inform future program development and evaluation. The consequences of not collecting this information would be a lack of in-depth information about the nature of current programs and strategies developed to build goal-directed behaviors in adults and young adults.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on July 1, 2015, Volume 80, Number 126, page 37621, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment D. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

Experts in their respective fields from outside of the study were consulted in developing the design, the data collection plan, and the materials for which clearance is requested.

Expert Panel Members

Beth Babcock, President and CEO, Crittenton Women’s Union

Marilyn Fox, Extension Educator, University of Nebraska Cooperative Extension; Building Nebraska Families

Crystal Hall, Assistant Professor of Public Affairs, University of Washington

Tim Kautz, doctoral student, University of Chicago

Donna Pavetti, Vice President for Family Income Support Policy, Center on Budget and Policy Priorities

A9. Incentives for Respondents

Site visits

It has been our experience that small tokens of appreciation are useful when interviewing low-income, hard-to-reach populations (such as those targeted for this information collection) to acknowledge the burden placed on participants. Research has shown that such tokens of appreciation are effective at increasing response rates for populations with lower educational level (Berlin et al. 1992) and low-income and nonwhite populations (James and Bolstein 1990). To help offset this burden and increase response rates, a $25 token of appreciation will be offered to each program participant who serves as a discussion group respondent. The token of appreciation will be provided at the time of the group discussion, after careful explanation of the procedures for the discussion. Any individual who chooses not to participate after receiving the detailed explanation will be provided the token of appreciation, so that it is not perceived as a coercion to participate. Tokens of appreciation will not be offered to any program staff. We will offer the tokens of appreciation to all program participants who agree to participate in the focus groups, not just to recipients of ACF services, because the study is targeting low-income adults generally and we do not want to distinguish between ACF and non-ACF service recipients during the group discussions.

Having used this amount for other federal projects of similar size and scale (see next paragraph for examples) targeting low-income individuals and families, including TANF recipients, we have found that $25 is not enough to unduly influence a participant’s decision to participate in the discussion group, reducing the possibility of non-response bias. However, the amount of the token of appreciation is enough to honor and respect the time participants are spending with us to talk about their experience with the program. The $25 token of appreciation takes into consideration the 75 minute focus group discussions with program participants and up to 40 minutes of travel (20 minutes in each direction to and from the group discussions) for a total of 115 minutes of the participant’s time. The token of appreciation also includes incidental expenses incurred during their transportation to and from the group discussion (approximately $5 for transportation costs, a figure that is derived from the GSA contractor rates for daily per diems). Offsetting such costs will help ensure that we get an adequate number of these hard-to-reach individuals to participate in the study.

Offering lower or no tokens of appreciation for focus group participants may increase the costs associated with recruiting the needed number of participants. Additionally, the amount is sufficient to encourage participation in the discussion group, but is not overly generous. The amount is based on what was previously approved by OMB in past studies for focus groups with similar low-income, hard-to-reach populations, such as the Descriptive Study of Tribal TANF Programs ($25, OMB # 0970-0411, expiration date October 31, 2013), the Fatherhood and Marriage Local Evaluation and Cross-Site Data Collection ($25, OMB # 0970-007, expiration date July 31, 2018), and the Temporary Assistance for Needy Families Study ($40, OMB # 0970-004, expiration date June 30, 2016).

A10. Privacy of Respondents

The Contractor is very cognizant of federal, state, and HHS data security requirements. All Mathematica research staff and consultants will comply with relevant policies related to secure data collection, data storage and access, and data dissemination and analysis.

Every effort will be made to maintain the privacy of respondents to the extent permitted by law. All respondents included in the study will be told that information they provide will be used only for the purpose of this research. Individuals will not be cited by name (or other identifying information) as sources of information in prepared reports. All papers that contain participant names or other identifying information will be kept in locked areas and any computer documents containing identifying information will be protected with a password. Any data collected that contains personal information for the GOALS study will be housed on secure servers.

The site visit interviews are purely voluntary. Respondents will be told that all of their responses will be seen only by members of the study team, their names will not appear in any written reports, and that responses to the questions are voluntary.

In addition to these procedures, Mathematica has extensive corporate administrative and security systems to prevent the unauthorized release of personal records, including state-of-the-art hardware and software for encryption that meets federal standards and other methods of data protection (for example, requirements for regular password updating), as well as physical security that includes limited key card access and locked data storage areas.

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

Exploratory calls

During the exploratory calls, program directors will participate in semi-structured interviews. The research team plans to conduct interviews with a single individual in up to 24 sites for a total of up to 24 interviews across all sites. The time per response is estimated at 60 minutes (1 hour).

Site visits

During site visits, staff will participate in semi-structured individual and small group discussions. The research team plans to gather updates from the exploratory calls with program directors/administrators, and conduct individual or small group discussions (with no more than 2-3 respondents per group) with program direct service staff and community partners for an average of 15 program staff in up to 12 sites, for a total of 180 interviews with program staff across all sites. The research team plans to conduct group discussions with an average of seven program participants at each site for a total of 84 discussants across all sites. The research team estimates the average time per response will be 75 minutes (1.25 hours). Exhibit A-1 presents the reporting burden on study respondents and the total cost.

The average hourly wage for each respondent group is calculated based on information from the Bureau of Labor Statistics.^¹ The average hourly rate^² is calculated for each respondent group using the following categories:

For program directors (Social and Community Service Manager Occupations SOC 11-9151): wage rate of $31.61, plus a 40 percent adjustment for benefits, or $44.25.
For program staff and community partners (Community and Social Service Occupations SOC 21-0000): wage rate of $21.50 plus a 40 percent adjustment for benefits, or $30.01.
For program participants: the minimum hourly wage ($7.25) plus a 40 percent adjustment to account for benefits, or $10.15 per hour.

Because members of the respondent group will come from multiple job categories, Exhibit A-1 provides an average across the relevant categories.

Exhibit A-1: Total Burden Requested Under this Information Collection

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Exploratory telephone call semi-structured interview – program director	24	12	1	1	12	$44.25	$531.00
Site visit semi-structured discussions (2-3 respondents per group) – program staff and community partner organization staff^a	180	90	1	1.25	113	$37.13³	$4,195.70
Site visit group discussion – program participants	84	42	1	1.25	53	$10.15	$537.95
Estimated Annual Burden Total					178		$5,265

^a This includes updates program administrators/directors from the exploratory calls.

Total Annual Cost

The estimated total annual cost of the time it will take program staff and participants to complete study tasks is $5,265.

A13. Cost Burden to Respondents or Record Keepers

This data collection effort involves no recordkeeping or reporting costs for respondents other than those described in Exhibit A-1 above.

A14. Estimate of Cost to the Federal Government

The total cost for these data collection activities to the federal government will be $401,484. The total annual cost is $200,742. This includes the cost of identifying programs, obtaining initial program and model information (including conducting exploratory calls), conducting the site visits (including administering the interviews and providing incentive payments), analyzing study data, and developing program summaries.

A15. Change in Burden

This is a new data collection.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

All data collected will be qualitative and descriptive. Data will be analyzed using a systematic process for qualitative information, using a qualitative analysis software package to store, organize, and code information. Programs will be analyzed to create individual summaries, and to detect cross-site themes.

Time Schedule and Publication

Exploratory calls

All calls will occur between Winter 2016 to Spring 2016

Site visits

All site visits will occur between Spring 2016 to Spring 2017.

Exhibit A-2 presents an overview of the project schedule for information collection. It also identifies publications associated with each major data collection activity. Findings from the exploratory calls and site visits will contribute to the Summer 2017 final report.

Exhibit A-2: Overview of Data Collection Schedule

Data Collection Activity	Timing	Associated Publications
Exploratory calls with program directors	Winter – Spring 2016	Memorandum identifying recommended programs for further study (Spring 2016) Summaries of exploratory call programs (Fall 2016)
Site visits and semi-structured interviews with program staff and participants	Spring 2016- Spring 2017	Summaries of site visit programs (Spring 2017)

A17. Reasons Not to Display OMB Expiration Date

All instruments created for the GOALS study will display the OMB approval number and the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto 1992 An experiment in monetary incentives. Pp. 393-398 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

James, Jeannine M., and Richard Bolstein.1990 The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly 54:346-361.

1 http://www.bls.gov/oes/current/oes_nat.htm.

2 Assuming 2080 FTE hours worked.

3 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage of Community and Social Service Occupations and Social and Community Service Manager Occupations.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-24