The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The objective of this research is to conduct cognitive and usability testing of the survey content for the two Scientists and Engineers Statistical Data System(SESTAT) surveys sponsored by the National Science Foundation. The two SESTAT surveys are the National Survey of College Graduates (NSCG) and the Survey of Doctorate Recipients (SDR). While the NSCG instrument will be the basis of the cognitive and usability testing, the overlap in survey content between the NSCG and SDR will allow the results from this testing to be applicable to both surveys.
The purpose of this testing is twofold. The first objective is to test questionnaire concepts added in 2013 and proposed for 2015 and understand how they work. These include the following:
Questions added to the NSCG in 2013:
- Financial support for undergraduate or graduate degrees (question D12)
- Student loan amount and balance (question D13)
Questions being considered for inclusion in the 2015 survey cycle:
- Certifications and licenses (questions A39-A49)
- Educational certificates (questions D14-D21)
Questions on occupational history are also being considered for the 2015 survey cycle. However, these have not yet been developed. They will be tested under a submission that will be made later this year, which will also include probing of the entire questionnaire. Changes to the questionnaire may be made based on an expert review that has not yet been completed.
The second objective is to understand how well these new questions work within the current questionnaires. Thus, the testing will include probing of questions throughout the questionnaire. The attached protocol includes probes for questions throughout the questionnaire.
Between June and September, 2013, staff from the Center for Survey Measurement (CSM) will conduct two rounds of testing in each of the three modes used to conduct the NSCG: self-administered paper, CATI, and Internet. A maximum of 15 interviews will be conducted per round per mode, for a total of 90 interviews. We will be targeting respondents who have at least a bachelor’s degree; respondents whose job is related to computer science, math, life sciences, physical sciences, social sciences, engineering or health sciences; respondents who are self-employed, employed in the private sector and retired; respondents who received scholarships or loans for college or graduate studies. To the maximum extent possible, within these specific criteria we will attempt to interview respondents of varying ages, genders and races. We will use a multi-pronged strategy to recruit the respondents needed for this study, such as posting ads in free local newspapers, distributing flyers, contacting local associations, setting up recruiting tables at local establishments, and using personal networks.
All interviews will be conducted in the Washington DC metropolitan area. The cognitive interviews will be conducted at the Census Bureau’s cognitive laboratory, offices of community associations, and locations convenient to respondents. A copy of the self-administered paper questionnaire is attached. Screen shots of the CATI instrument are not available yet because the programming has not been finalized, but the content is the same as the paper questionnaire.
The usability interviews will be conducted at the Census Bureau’s usability lab. Screen shots of the Internet instrument are not available because programming of that instrument is not complete. The content is the same as the other two questionnaires. Copies of the demographic questionnaire, the background questionnaire, and the satisfaction questionnaire used in usability studies are attached.
The same protocol will be used for all the interviews. It is also attached. Interviews using the paper questionnaire will be conducted using respective think aloud techniques, with probes being asked at the end of each section to avoid respondents’ navigation through the section. For the CATI and web interviews, the concurrent think-aloud interviews will be conducted. This will avoid the necessity to scroll back through the questionnaire to see the questions and responses.
The end of the usability protocol includes questions for a research study on factual questions that OMB approved separately on September 9, 2012.
Cognitive interviews will be tape-recorded, with the participants' permission, to facilitate a summary of the results. All participants will be informed that their response is voluntary and that the information they provide is confidential. Respondents will receive $40 for their participation.
The estimated time for completion of each cognitive interview is one and one-half hours. Thus, the estimated burden for this research is 135 hours.
The contact person for questions regarding data collection and study design is:
Terry DeMaio
Center for Survey Measurement
U.S. Census Bureau
Room 5K319
Washington, D.C. 20233
(301) 763-4894
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Rodney L Terry |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |