Questionnaire Development Process

Attachment 12

Behavioral Risk Factor Surveillance System (BRFSS) Questionnaire Development Process

Purpose

Throughout 2013 and 2014, an evaluation of the BRFSS questionnaire development process was undertaken. It indicated that most stakeholders, including CDC programs and state coordinators, felt that the process for changing the questionnaire was unclear and/or inconsistent. The BRFSS responded to this evaluation through a process improvement plan. The BRFSS enhanced workflow presented in steps below aims to streamline the BRFSS process. Also new is a revised application for proposing a new question(s), the re-establishment of an Advisory council, and a dedicated person to assist programs with questionnaire development from application to approval.

Enhanced Workflow Description

1. The first step of the BRFSS Question approval process is initiated by CDC programs or other entities wishing to add or revise a question to the BRFSS questionnaire through a New Question Application. This revised application should include the source of the question(s), performance (if known—e.g., evidence of validity, reliability, cognitive testing), public health importance, and analytic plan.

2. The completed application is then submitted to the BRFSS Technical Assistance (TA) Team. This team is responsible for progressing questions through the questionnaire development process. It will act as a liaison between stakeholders and the BRFSS Advisory Council for process adherence. The TA Team will receive applications and if necessary recommend to the applicant any preliminary changes. The questions are also forwarded to the Advisory Council for a preliminary vote.

3. The Council convenes to determine whether new question applications adhere to BRFSS’s purpose and if necessary makes recommendations for changes to the questions. It also reviews any available cognitive testing results provided by the applicant and address any issues. Finally, the council recommends (or denies) the application’s advancement to the next step in the process through a preliminary vote.

4. Next, the council’s preliminary approved questions are forwarded for cognitive testing. The cognitive testing usually includes a private focus group during a first round and a small sample of telephone respondents in a second round. The purpose of cognitive testing is to identify question delivery issues that may come up during an actual interview. Based upon the results of cognitive testing, changes may be recommended to the applicant. The applicant is then asked to revise and resubmit.

5. The questions are then forward back to the Advisory Committee for a final review and vote.

6. Approved questions are sent to states for review. The questions are then presented by the applicant at the BRFSS Annual Conference/Meeting. BRFSS state coordinators or their designees vote to include or exclude questions from the BRFSS questionnaire. A minimum of a 70% approval rate must be achieved for inclusion on the questionnaire.

7. Once questions have achieve the require 70% vote of approval from state coordinators, they are field tested by a host state. This field test is conducted on a sample size of 300 telephone respondents in order to identify any remaining issues with the delivery of new questions. The BRFSS TA Team conducts briefings with BRFSS interviewers before and after their shifts inquiring about issues that may have arisen during the interviews. The interviews are also monitored by the BRFSS TA Team for quality assurance. The compiled data is then sent back to the TA Team and analyzed for any issues. Any remaining issues are again communicated to the applicant and necessary revisions are made to the questions. The questions are also sent for Spanish translation.

8. The survey containing the new and existing questions is sent through CDC clearance. Beginning in 2015 the questionnaire will also be submitted annually for OMB review and approval.

9. Upon OMB approval, the final BRFSS questionnaire is sent to the coordinators for implementation.

10. The Survey Operations Team at CDC BRFSS then completes the CATI programming.

11. The survey is then administered at the state-level.