Various Governments Pretesting Activites

ASPEP Cognitive Testing Protocol

E-6 Form Splitting

8/8/2012 -- Draft

Research Questions to Address:

• Do the changes to the form adversely affect the response process, and the responses provided?

• Are there any items that cause the respondents problems?

Introduction:

• Explain purpose of meeting: to understand the process used to answer questions, to get feedback on proposed changes

• Meeting with entities in different states, different types and of different sizes. Not everybody keeps their records the same way, so some questions will seem odd and/or obvious.

• Explain that we are not testing the respondent – we only want information

• Structure of meeting: general and specific questions about the form itself, the reporting organization and also how R reported data

• Permission to record discussion? Have R sign consent form.

Before the Questionnaire:

• What types of services does this government entity provide?

• What is your role in this government entity? What kind of responsibilities do you have?

• What was your role in the process for responding to this questionnaire? (Gathered data? Compiled data? Checked data? Authorized release?)

General Probes

• Reflect back R’s specific answer: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question? / Tell me how you arrived at these answers. What did you include in this number? What did you specifically exclude?

• What records (if any) did you look at? Were any other people involved in the process of coming up with this number? [If Yes] What departments are they in?

• How well does this data request match with your records?

Item 2 - FT Hours

Warm-up probes:

• What is this question asking you?

• How did you arrive at your answers?

• What does the term “full time employees” mean to you?

Part 2: Employees, Payroll, & PT Hours

Part 2 Instructions and Example:

The example and instructions bridge two pages. Make a note of how the R grapples with the layout.

• Did you notice the example?

  • What did you think it was telling you?

  • Was it clear? Did it help you figure out what you needed to know to fill out the form? If not, what could we do to make it more helpful?

• Did you notice the instructions?

  • What did you think they were telling you?

  • Were they clear? Did they help you figure out what you needed to know to fill out the form? If not, what could we do to make them more helpful?

  • Anything missing? Anything that shouldn’t be there?

• Did you notice the payroll frequency codes? [And at the top of the pages]

  • What did you think it was telling you?

  • Was it clear? Did it help you figure out what you needed to know to fill out the form?

  • Are they easy or difficult to find?

  • Was it helpful having them at the top of the page?

Part 2 Reporting:

Make careful note how respondents grapple with the new layout.

• How did you arrive at your answer to this question?

• Did you have any difficulty figuring out how to report information in this question? Make sure that respondents do not have more than two pay frequencies. (Why?)

FOR ELEMENTARY/SECONDARY SCHOOLS

• Do you offer any post-secondary education? If yes, how would you report those employees on this form?

FOR POST-SECONDARY SCHOOLS

• Do you offer any elementary or secondary education? If yes, how would you report those employees on this form?

FOR SCHOOLS WITH BOTH TYPES OF PROGRAMS

• If respondent only reports one type of education: I noticed that you only filled out the [elem/secondary / post-secondary]. Do you have the other type of education programs?

• What made you decide not to report those employees and payroll amounts? Find out why the respondent appeared to report only one type of educational program. Did the respondent leave them off or report them elsewhere?

Wrap-up Questions

• What, if anything do you like about the form that we showed you today?

• What if anything do you not like about the form we showed you today?

• (Show them the other version of the form.) Which version of the form do you like better? Why?

• About how long would it take you to complete this form (estimate in hours)?

• (If R hasn’t indicated reporting online) Do you think you would fill this form out online?

• If you respond online, would you like to receive a paper form in the mail, or would you like to receive a notification to report online without receiving a paper form?

Thank you for your help today.