New Questions Related to the 2009 American Recovery and Reinvestment Act for various surveys covered by SRS

ATTACHMENT

Revised December 15, 2009

Proposed Procedures for Conducting Cognitive Interviews on

ARRA (Stimulus Funds) Legislation

Purpose

Our purpose is to evaluate questions to be included in 2010 National Science Foundation (NSF) Surveys with regard to their potential for obtaining needed information from college graduates in science and engineering about the effect of the 2009 American Recovery and Reinvestment Act (i.e. Stimulus Bill) (ARRA) on their current employment.

Procedures

We will identify individuals who are college graduates and who are employed in science/engineering related positions. We will ask them to volunteer for a one hour interview to be conducted in the Cognitive Interviewing Laboratory at Washington State University (WSU).

Individuals we identify will be asked to come to Wilson-Short Hall (where the cognitive interviewing laboratory is located) for an interview that will be up to one hour in length. They will be told that the purpose of the interview will be “to help the SESRC evaluate questions to be included in a survey of U.S. college/university graduates, which will be conducted by the National Science Foundation.” They will be informed that they will be reimbursed $35 for volunteering to help us with this research project.

We will make an explicit effort to recruit respondents from people who we think are likely to be employed on grants or contracts that are supported by stimulus funds, but will not be limited to such individuals. They will be identified through the help of the WSU Office of Grants and Research Development, which has knowledge of grants received that are considered to be the result of stimulus funding.

We plan to interview some individuals who we do not believe are directly employed on stimulus funds as well as some who are. The number of interviews of each type will be decided in cooperation with the National Science Foundation.

A draft of the questionnaire and protocol is included in this document.

Timing

Up to 20 interviews will be conducted. Ten interviews will be with individuals we believe to be employed in projects supported by stimulus funds. We expect to complete all of the interviews by January 29, 2010.

Written Report

A written report containing the main findings and interpretations from the interviews will be drafted for the National Science Foundation and provided to them no later than February 19, 2010. The report will be discussed with NSF staff prior to February 12th A final version of the report will be prepared after comments are received back from NSF. We expect that a final report will be available by February 26, 2010.

Protocol

When the volunteer arrives for the interview the procedures we use will follow closely those used in previous cognitive interviews conducted for the National Science Foundation by Don Dillman, Nicholas Parsons and Taj Mahon-Haft in reports on the effects of community college on becoming a scientist, available at “http://www.sesrc.wsu.edu/dillman/”.

Specifically, respondents will be interviewed in the presence of two individuals: Don Dillman and either Benjamin Messer and/or Morgan Millar, graduate research assistants in the SESRC. One person will conduct the interviews while the other takes notes. This is the procedure followed for previous work of this nature, with Mahon-Haft and Parsons. Detailed notes will be written-up at the end of each interview. The interviews will not be recorded.

Respondents will be told that the interviews are confidential. They will be asked to sign a consent form for conducting the interview that indicates that the cognitive interviews are being conducted on behalf of the National Science Foundation and that the procedures and content of this research have been approved by the U.S. Office of Management and Budget. The OMB control number for this collection is 3145-0174.

The consent form will include the following:

“This information is solicited under the authority of the National Science Foundation Act of 1950, as amended. All information collected will be confidential and used only for research purposes. Your identifying information will be removed from any results or reports of data. We want to assure you that anything you tell us or any information that you give us will not be attributed to you individually. The information will only be shared with the research team. We consider the information you choose to share with us personal and confidential. You have the right to refuse to answer any question, and to refuse to participate. “

Each respondent will be informed at the beginning the interview that the purpose of the interview is to ask them to answer several questions proposed for a National Survey of College Graduates. We will explain that a “think-aloud” procedure is being used for the interview, and the respondent will be asked to respond to the questions as if he or she were at home completing the questionnaire, with one difference. We will ask the respondent to talk out-loud as the interview proceeds. It’s up to him/her what is read, but whatever is read should be read out-loud, and we would like to know what is being thought about as it is decided how to answer each question.

The respondent will then be told that because it may be unusual for him/her to talk out-loud while answering a questionnaire, we will first provide a practice questionnaire. This brief questionnaire will consist of one or two very brief questions, in order to “teach” the response procedures. This general training procedures is describe on pages 221-227 of Internet, Mail and Mixed-Mode Surveys by Dillman, Smyth and Christian (Wiley, 2009).

The respondent will then be given selected questions proposed for the 2010 version of the National Survey of College Graduates. A draft of those questions is attached.

Light probing will occur for the questions preceding the two ARRA questions. After respondents answer the ARRA questions, we expect to ask additional probing questions about their source of employment, knowledge of ARRA, and other effects of the stimulus program.

Q 1. Were you working for pay or profit during the week of October 1, 2009?

Working includes being a student on paid work-study, self employed, or on any type of paid or unpaid leave, including vacation.

Use an X to mark your answer.

Yes

No

Q2. Who was your principal employer during the week of October 1, 2009?

If you had more than one job, report the one for which you worked the most hours that week.

If your employer had more than one location, report the location that employed you.

If you worked for a contracting or consulting company, report the name of that company, not the client organization.

Employer Name

Department/Division

City/Town

State

ZIP Code

*Q3. Was your principal employer an educational institution?

Yes

No

*Q4. During the week of October 1, 2009, what type of academic position(s) did you hold at this institution?

Mark Yes or No for each item.

Yes No

1 President, Provost, or Chancellor (any level)

2 Dean (any level), department head or chair

3 Research faculty, scientist, associate or fellow

4 Teaching faculty

5 Adjunct faculty

6 Postdoc (e.g., postdoctoral fellow or associate)

7 Research assistant

8 Teaching assistant

9 Other position – Specify

Q 5. What was the title of the principal job you held during the week of October 1, 2009?

Example: Financial analyst

Q6. Did your duties on this job require the technical expertise of a bachelor’s degree or higher in…

Mark Yes or No for each item.

Yes No

1 Engineering, computer science, math, or the

natural sciences

2 The social sciences

3 Some other field (e.g., health, business, or

education) – Specify

Q7. During what month and year did you start this job (that is, the principal job you held during the week of October 1, 2009)?

Month Year

PRINCIPAL JOB STARTED

*Q8. To what extent was your work on your principal job related to your highest degree? Was it…

Mark one answer.

Closely related

Somewhat related

Not related

*Q9. Thinking about your principal job held during the week of October 1, please rate your satisfaction with that job’s…

Mark one answer for each item.

Very Somewhat Somewhat Very

Satisfied Satisfied Dissatisfied Dissatisfied

1 Salary

2 Benefits

3 Job security

4 Job location

5 Opportunities for

advancement

6 Intellectual challenge

7 Level of responsibility

8 Degree of independence

Q 10. How would you rate your overall job satisfaction with the principal job you held during the week of October 1, 2009?

Mark one answer.

Very satisfied

Somewhat satisfied

Somewhat dissatisfied

Very dissatisfied

Q11. During a typical week on your principal job, how many hours did you work?

NUMBER OF HOURS WORKED PER WEEK

Q12. Concerning your principal job during the week of October 1, 2009, were any of the following benefits available to you, even if you chose not to take them?

Mark Yes or No for each item.

Yes No

1 Health insurance that was at least partially paid

by your employer?

2 A pension plan or a retirement plan to which your

employer contributed?

3 A profit-sharing plan?

4 Paid vacation, sick or personal days?

*Q13. Thinking back now to 2008, was any of your work during 2008 supported by contracts or grants from the U.S. government?

FEDERAL EMPLOYEES: Please answer “No.”

Mark one answer.

Did not work in 2008

Yes

No

Don’t know

Q14. During 2009, was any of your work supported by the Federal stimulus bill (American Recovery and Reinvestment Act of 2009)?

FEDERAL EMPLOYEES: Please answer “No.”

Mark one answer.

**Did not work in 2009

Yes

No

Q15. (If yes) please explain in your own words how the stimulus bill (ARRA) has supported your work. This question is very important, so please tell us all you can.

Examples of probes that may be used, particularly when answering the ARRA related questions.

“Tell me what you’re thinking as you answer this question.”

“Are you familiar with the ARRA or the Federal Stimulus Bill? What do you know about the ARRA or Federal stimulus bill?”

“You have indicated yes/no. Can you tell me why you chose this answer?”

“How certain are you about your answer? How do you know that your work has/has not been supported by ARRA?”

“What aspects of your work are you thinking about when coming up with your answer?”

“What do you think the question refers to when asking about ‘your work’?”

Are you aware of whether some other employees in the University are now being paid at least in part by ARRA (stimulus funds)?”

“To the best of your knowledge, are you paid by one source of funds, i.e. one grant or contract, or multiple sources of funds? “

“Has your supervisor, told you that some or all of your funding comes from ARRA? (And, is it some or all?)”

If yes to receiving ARRA funds “What proportion of your appointment comes from ARRA?”

“We are wondering if when we ask that question, “During 2009, was any of your work supported by the Federal Stimulus bill?” that a respondent might have said yes, because of money allocated to the University by the State of Washington to support the University budget, even though they were not employed specifically on ARRA funds through a specific grant or contract?   Do you think it’s likely a person would have said yes, even though they personally, are not being paid specifically from stimulus act (ARRA) funds?”

To the best of your knowledge, has the stimulus bill (ARRA) provided you with: a tax refund or benefit; s upport for home purchase or improvements; support for buying an automobile, including ‘Cash for Clunkers’; i mproved public services; health care services; c hild care services; support for education.

Probes for Specific Answers

Q15: Could you explain a little more about the reason you answered _______ (Y/N)?

If respondent answered “No”:

NOQ15: The last question is a particularly important question in the survey. Is it possible that part of your work may be supported in some way by the ARRA stimulus bill, but that you are not aware of that support?

Yes

No

If respondent answered “Yes” on Q15:

YESQ15a: Could you explain a little more about the reason you answered ________ (Y/N)?

YESQ15a: The last two questions are particularly important questions in this survey. Is it at all possible that your work is NOT supported in any way by the ARRA stimulus bill?

Yes

No

We will probe further as needed on these questions to see if respondents are interpreting the question appropriately, and particularly if they appear to have interpreted the question as asking for indirect effects.

9