Justification

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0970-0401)

T ITLE OF INFORMATION COLLECTION: Course Participant Knowledge Gain Assessment and Instructor and Course Participant Experiential Feedback for the Development of an Online Course on Evidence Building Strategies

PURPOSE:

As part of the Children’s Bureau (CB) efforts to improve the capacity of state child welfare agencies to meet established standards, achieve results, and improve outcomes, the Building Evidence Training (BET) Project has engaged an accredited school of social work and a state child welfare training academy to test the usability of a new, online, self-paced 6-unit curriculum to build the knowledge and skills of current child welfare frontline staff and future child welfare frontline staff (Bachelor of Social Work [BSW] and Master of Social Work [MSW] students).

There are three types of instruments course participants (i.e. child welfare frontline workers and supervisors and BSW and MSW students) will complete. These instruments are self-administered within the web-based course. Course participants will complete the Evidence-Based Practice Attitude Scale (EBPAS) at the beginning of the course and again after completing the course. The EBPAS will be used to assess course participants’ shifts in attitudes related to evidence building.

Course participants will complete pre- and post-tests for each unit; specifically, respondents will complete 11 pre- and post-tests – a unique pre- and post-test for each of five units plus a post-test only for one unit. These tests will be used to assess course participant’s knowledge gain. Lastly, course participants will complete one experiential survey to identify potential improvements for the content and method of delivery of the course and course materials to ensure it meets course participants’ educational needs.

Instructors (i.e. faculty at the accredited school of social work and trainers at the child welfare training academy) are asked to complete seven experiential surveys (1 survey for each of the six units, plus 1 overall survey). The instructor experiential feedback surveys will be used to identify strengths of and potential improvements for the content and method of delivery of the course and course materials to ensure it meets the instructors’ educational needs.

The EBPAS will be used internally by the project team to assess how effective the course material is at shifting course participants’ attitudes to be more favorable to evidence building activities. The pre- and post-tests will be used internally by the project team to assess how effective the course material is at achieving the core competencies. Instructors—faculty at the accredited school of social work and trainers at the child welfare training academy—will also have access to the individual course participants’ pre- and post-test responses for their specific cohort of course participants. This will allow instructors to assess how well their students are progressing and adjust their instruction to address challenges students have with the content.

The experiential feedback surveys will be used internally by the project team to make conclusions about how best to revise the method of delivery. The aggregate results of analyses of the EBPAS, pre- and post-tests, and the experiential feedback surveys will be shared with an advisory committee that is providing feedback on the course development; no individual level responses will be shared with the advisory committee

Instructors’ and course participants’ names and email addresses will be collected only to provide instructors and course participants with access to the website; this is the extent of personally identifiable information (PII) that is necessary. No PII will be retained. Without this feedback from usability testing we have no other way to accurately gauge if the course meets the goals. 

DESCRIPTION OF RESPONDENTS:

Course participants: Course participants are child welfare frontline workers and supervisors and BSW and MSW students. As they move through the self-paced course, they will complete the EBPAS upon entering the course, complete the Unit 1 post-test and pre- and post-tests for the remaining units, complete the EBPAS again after completing all the units, and complete one experiential survey.

Instructors: Instructors are faculty at the accredited school of social work and trainers at the child welfare training academy. They are asked to complete one experiential survey for each of the six units and one overall experiential survey for the entire course.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[X] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [ ] Other: ______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Matthew McGuire & Serena L. Williams

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ X ] Yes [ ] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ X ] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ X ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [ X ] No

BURDEN HOURS

Category of Respondent	No. of Respondents	No. of Responses per Respondent	Time per Response (in hours)	Total Burden Hours
Individual (BSW and MSW students) – EBPAS	40	2	10 minutes	13
Individual (BSW and MSW students) – Pre- and Post-Tests	40	11	7 minutes	51
Individual (BSW and MSW students) – Experiential Survey	40	1	10 minutes	7
Individual (child welfare frontline workers and supervisors) – EBPAS	40	2	10 minutes	13
Individual (child welfare frontline workers and supervisors) – Pre- and Post-Tests	40	11	7 minutes	51
Individual (child welfare frontline workers and supervisors) – Experiential Survey	40	1	10 minutes	7
Individual (faculty at the accredited school of social work) – Experiential Survey	2	1	10 minutes	.3
Individual (faculty at the accredited school of social work) – Unit Survey	2	6	7 minutes	1
Individual (trainers at the child welfare training academy) – Experiential Survey	2	1	10 minutes	.3
Individual (trainers at the child welfare training academy – Unit Survey)	2	6	7 minutes	1
Totals	84			145 hours

Category of Respondent	Total Burden Hours	Hourly Wage Rate	Total Burden Cost
Individual (BSW and MSW students)	71	$7.251	$514.75
Individual (child welfare frontline workers and supervisors)	71	$22.842	$1621.64
Individual (faculty at the accredited school of social work)	1.3	$35.713	$46.42
Individual (trainers at the child welfare training academy)	1.3	$24.544	$31.90
Totals	145 hours		$2214.71

FEDERAL COST: The estimated annual cost to the Federal government is: $9052

Staff	Grade/Step	Salary	% of Effort	Fringe (if applicable)	Total Cost to Gov’t
Contractor Cost		$123,594	2%	$904	$3376
		$63,835	4%	$934	$3487
		$80,184	2%	$586	$2189
Total					$9052

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [X ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

The target audience for this course is child welfare frontline workers and supervisors and students engaged in BSW and MSW programs. We engaged the Indiana University (IU) School of Social Work and the Indiana Department of Child Services (DCS) to participate in usability testing.

IU will include all students enrolled in a BSW-level course on trauma and all students enrolled in an MSW-level course on children, youth, and families as respondents. The IU faculty for these two courses will serve as instructor respondents.

DCS will usability test the curriculum as part of in-service training for frontline workers and supervisors. Thirty-four frontline workers and six supervisors across the state have been asked to participate. Two DCS trainers will serve as instructor respondents.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[X ] Web-based or other forms of Social Media

[ ] Telephone

[ ] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [ ] Yes [ X ] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of Respondents.

Participation Time: Provide an estimate of the amount of time (in minutes) required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of Respondents and the Participation Time then divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

1 Used current reported minimum wage to calculate the total burden cost for BSW and MSW students: http://www.dol.gov/whd/minimumwage.htm

2 Used current reported mean hourly wage for “Child, Family, and School Social Workers” to calculate total burden cost for child welfare frontline workers and supervisors: https://www.bls.gov/oes/current/oes_nat.htm#29-0000

3 Used current reported annual mean wage for “Social Work Teachers, Postsecondary” ($74,280) divided by 2,080 hours to calculate mean hourly wage for faculty at the accredited school of social work: https://www.bls.gov/oes/current/oes251113.htm

4 Used current reported mean hourly wage for “State Government Social Workers” to calculate total burden cost for trainers at the child welfare training academy: https://www.bls.gov/oes/current/oes211029.htm

7