Attachment 7_SGI pilot survey analysis

NINR Summer Genetics Institute Evaluation Project Discovery Logic

Attachment 7: NINR SGI – Pilot Survey Analysis

Outline

1. Survey Tool Findings

2. Survey Question Analysis

3. Feedback Sheet Analysis

4. Survey Recommendations

Note: NINR = National Institute of Nursing Research; SGI = Summer Genetics Institute; DL = Discovery logic.

1. Survey Tool Findings

Below are listed several overall measures regarding the NINR SGI survey tool.

1. Time to complete survey: The five pilot participants took between 16 and 85 minutes to complete the pilot survey (16, 21, 24, 73, and 85 minutes) including any breaks taken. This duration was derived from a SharePoint log that indicates when they survey was first accessed and when survey was submitted. The average time to completion was 44 minutes. They all completed the survey the same day they started it. One participant completed the survey the day it opened, two completed it after the day 6 “email reminder”, one completed it after the day 14 “phone call reminder”, and one completed it after the day 16 “second email reminder”. The survey was open for 20 days and a marketing email was sent out 2 weeks prior to survey start.

2. Occurrence of survey refusals or illogical question responses: No participant refused to participate in the survey and no illogical responses were observed.

3. Point at which drop-outs leave survey: No pilot participant left the survey before completion. As described above, it appears that if the participants chose to start the survey, they tend to complete it.

4. Use of CV email option: Three participants elected inside the survey to send their CV to NINR, but only one of them sent it.

2. Survey Question Analysis

The NINR SGI Alumni Survey consisted of 40 questions of which several branched into one or two sub questions. Below are listed a summary of the survey findings (bullets a-dd in section B1 below) and a section identifying any responses or issues needing further discussion or investigation (bullets I-X in section B2 below).

B1. Summary of survey findings:

1. The five participants attended the NINR SGI program in 2000, 2003, 2004, 2007, and 2009.

2. A participant was located in each of these age ranges 41– 45, 46 – 50, 51 - 55, 56 – 60, and 61 – 65.

3. There were four females and one male, and they all selected white as their racial category.

4. All participants has Masters and PhDs (see table below) while none had DrPH, DNP, MD, DDS, JD or other degrees.

P#	BA/BS	Masters	PhD
174	1974-78	1980-82	1995-1998
176	1985-1989	1992-1993	1995-2000
177	1982-1987	1993-1995	2007-2011
178	1976-1980	1987-1989	2002-2006
179	1966-1970 BS microbiology, 1995-1997 BS Nursing	1999-2002 Masters degree nursing	2007-2011 PhD nursing

5. Two participants held NP certificates. None had obtained any of the other certificates (see 2.I in section B2 below). All were RNs.

6. Four of the five participants held two current positions while one held one position (see table below). Three participants held faculty appointments, two were clinicians, two were government scientists, one was a research institute scientist and one was postdoc.

Current Position
Unemployed
Scientist at research institute	1
Scientist for federal, state or local government	2
US military or Commissioned Corps
Faculty appointment	3
Clinician (provide direct patient care such as RN, NP, CNS, MD, DDS, Psychologist)	2
Pre-doctoral position
Post-doctoral position	1
Self-employed
Student
Other (Please specify)

7. Three participants worked at nursing schools, one at a federal agency and one at the U.S. Army (see table below).

Principal Employer
(Unemployed)
Dental school
Engineering school
Medical school
Nursing school	3
U.S. Army	1
federal agency	1
School of public health
Other university school or department
University-affiliated research center or research institute
Non-profit research center or institute not affiliated with a university
For-profit research center/institute or organization not affiliated with a university
Other employer (Please specify type)

8. All participants worked in research ranging from a 100% to minimum 25%. Three participants teach and/or are involved with direct patient care. Two work in health care administration.

	P#	Research	Teaching	Direct Patient Care	Health Care Admiration
	176	100
	174	26-50	1-25		26-50
	177	76-99		1-25
	179	26-50	26-50	1-25
	178	51-75	1-25	1-25	1-25

9. Two participants were accepted into a fellowship/postdoctoral/traineeship program after attending the SGI. One of them completed a “RWJ Executive Nurse Fellowship in Leadership” in 2008 while the other expect to complete “genomics” in 2013, please see item 2.II in section 2B below.

10. All participants applied for research grants (range from “1” to “5 or more”) since attending the SGI. Two participants who applied for “5 or more” grants stated that all their grant applications had a genetics component. One participant applied for 1 grant with no genetics component. Two participants applied for grants (2 and 1 respectively) but left out, despite being asked, if the grants had a genetics component, please see item 2.III section 2B below.

11. Four participants were awarded research grants (5 or more; 4; 2; 2). The participant with 5 or more grants elected to email her CV, but never did. The other three participants entered grant information but one of the entered only one of two grants awarded and did not include award amount and length. Grant provided by the three participants:

• P1: 04/08-12/08 Rasmussen, N. (P.I.) and Chaperon, C. Variation in Clock and Pain Gene Expression in Inbred Strains of Mice, NASA Nebraska Space Grant/UNO, $7,500.

• P1: 07-09 Chaperon, C. (P.I.) and Rasmussen, N. (Co-I.). Blood Pressure Rhythm Pre- Interventions Study in VCD-Induced Perimenopausal Mice. Gerontological Nursing Interventions Research Center and the Hartford Center of Geriatric Nursing Excellence at Iowa, $15,000.

• P1: 07-08 Rasmussen, N. (P.I.) and Chaperon, C. Variability in In-bred Strains of Mice, Dean's Grant, $5,000.

• P1: 5/1/06-4/30/07 Rasmussen, N. (P.I.). Circadian Nociception and Gene Expression in Mice, NASA Nebraska Space Grant/UNO, $7,500.

• P2: National Institute of Nursing Research, National Institute of Health, National Research Service Award, (F31 NR011265), April 2009

• P3: A Coaching Intervention to Promote Nutrition and Bone Health in Deployed Soldiers. HU0001-10-1-TS15, N10-C02 TriService Nursing Research Program, Amount $439,253; Period of award: August 2010-July 2012.

• P3: An RCT of Nurse Coaching vs. Herbal CAM for Soldier Weight Reduction. HT9404-12-1-TS03, N12-007. TriService Nursing Research Program, Amount: $447,517; Period of award: March 2012-February 2015.

12. Two participants participated as co-PIs on research grants (5 or more; 1).

13. One participant participated as a project manager on one research grant.

14. One participant participated as a research assistant on two research grants.

15. No participant participated as clinical nurses on research grants.

16. All five participants had authored or co-authored peer-reviewed research publications. Two participants elected to email their CV but never sent them. The other three participants provided these publications:

• P1: Rasmussen, N.A. & Farr, L.A. (2009). Beta-endorphin response to an acute pain stimulus. Journal of Neuroscience Methods, 177, 285-288.

• P2: Wysocki, K. & Ritter, L. (in press). Diseasome: an approach to understanding gene-disease interactions. Annual Review of Nursing Research (Vol. 29). New York: Springer Publishing Co.

• P2: Wysocki, K. (2010). Giving through teaching: How nurse educators are changing the world. National League of Nursing. Chapter 12 contribution, Nurse educators who work in other countries. New York: Springer Publishing Co.

• P3: Hickey KT, Sciacca RR, and McCarthy MS. A Descriptive Survey of Summer Genetics Institute Nurse Graduates in the United States. (In press).

• P3: Loan LA, Patrician PA, McCarthy M. (2011). Participation in a national nursing outcomes database: Monitoring outcomes over time. Nursing Administration Quarterly 35(1):72-81.

• P3: Patrician P, Loan LA, McCarthy MS, Fridman,M, Donaldson N, Bingham M, & Brosch L. (2011). Nurse staffing and adverse events. Journal of Nursing Administration, 41(2):64-70.

• P3: Martindale RG, McCarthy MS, McClave SA. (2011). Critical care nutrition guidelines: Are not they all the same? Minerva Anestesiologica, 77.

• P3: Patrician PA, Loan LA, McCarthy M, Brosch LR, and Davey KS. (2010). Towards evidence-based management: Creating an informative database of nursing-sensitive indicators. Journal of Nursing Scholarship, 42(4):358-366.

• P3: McCarthy MS, Loan LA, Azuero A, and Hobbs C. (2010) Consequences of modern military deployment on calcium status and bone health. Nursing Clinics of North America, 45:109-122.

17. Three participants had authored or co-authored non-peer reviewed research publications. Two participants elected to email their CV but never sent them. One participant provided five publications:

• P1: McCarthy, MS, Ramme, S, and Cummings, SL. (2011). A Review of Telehealth Applications in the Military Community. Nova Science Publishers, New York.

• P1: Dickens, SD, Gibson, J, and McCarthy, MS. (2011). Adapting an Army Professional Nursing Course to the Attitudes of Adult Learners. Nova Science Publishers, New York.

• P1: McCarthy, MS. (2011). Protocol for continuous aspiration of subglottic secretions. In SR Villar (Ed.) Protocolos en Cuidados Criticos (Protocols in Critical Care). Editorial Marban S. L., Spain.

• P1: McCarthy, MS and McGiffin, K. Health and Nutrition Status During Deployment. Northwest Guardian. (In press.)

• P1: McCarthy, MS. The Wear and Tear of War: A Program of Research Dedicated to Bone Health in Young Warriors. Army Nurse Corps Association (ANCA) Newsletter, Vol 36 No 1, March 2011.

18. Three participants had authored or co-authored published books or monographs. Two participants elected to email their CV but never sent them. One participant provided this book:

• P1: Jacob, E., & Rasmussen, N. (2006). Interindividual variations in response to pain and analgesics. In A. Lucas (Ed.), Frontiers in Pain Research (pp. 49-71). Hauppauge, NY: Nova Science Publishers, Inc.

19. All participants had presented research papers or posters at conferences. Two participants elected to email their CV but never sent them. Three participants provided 11, 13 and 7 papers and/or posters.

20. No participant received any patents that are a result of their research

21. One participant has released a copyrighted product (e.g., educational videos, teaching tools) as a result of her research:

• P1: Ward, Linda (2011). Genomic Nursing Concept Inventory. U. S. Copyright #TX6732396

22. Three participants have received science, teaching or health care awards or honors. One participant elected to her CV but never sent it. Two participants provided these science awards and or honors:

• P1: AZNP 21st Annual Southwest Regional Nurse Practitioner Symposium, Flagstaff, Arizona, July 2009. Leukotriene Receptor Gene Variation and Atopic Asthma, poster session winner

• P2: 2010 Phyllis J. Verhonick Nursing Research Course – 1st place poster : Most Innovative Science award

• P2: 2009 Association of Military Surgeons of the United States Conference, Karen Reider Nursing Research Section, 2nd place poster: Best in Science award

One participant provided this healthcare award and/or honor:

• Fellow of the American Academy of Nurse Practitioners (FAANP), June 20, 2009

23. Two participants had been elected to science, teaching or healthcare leadership positions. They both elected to email their CVs but one participant never sent it although she did include the title of a science leadership position to which she was elected “ISONG President.” For the participant who did email his CV, it was unclear whether he was elected or appointed to the listed positions, please see item 2.VII section 2B below

24. Four participants had been appointed to science, teaching or healthcare leadership positions. Two participants elected to email their CVs but one participant never sent it. A third participant provided an appointed science position and an appointed teaching position:

• Appointed member, CLIAC (Clinical Laboratory Improvement Amendment Committee), Centers for Disease Control and Prevention. 2011

• Appointed member, G2C2 editorial board, National Human Genome Research Institute, 2011.

A forth participant provided an appointed health care position

• Appointed as Chief, Center for Nursing Science & Clinical Inquiry, Madigan Healthcare System, Tacoma, WA (2 years)

25. Two participants had intentionally incorporated knowledge of genetic testing and/or genetic diseases into their clinical practice while three participants stated they did not have a clinical practice.

26. Two participants had used knowledge gained from the Summer Genetics Institute in their clinical practice in the following areas:

• P1: Patient/family education; Patient/family counseling; Ordering genetic tests

• P2: Patient/family education; Patient/family counseling; Incorporating family health history into clinic records

27. The participants were asked to indicate the extent to which they agree or disagree with each of the following statements pertaining to your participation in the Summer Genetics Institute:

Question	Strongly agree	Agree	Neither agree nor disagree	Disagree	Not Applicable
The Summer Genetics Institute has been valuable to my career	5
The Summer Genetics Institute has made me more likely to engage in genetics research activities.	4	1
The Summer Genetics Institute has made me more likely to incorporate genetics into my clinical activities.	1	2			2
The Summer Genetics Institute has made me more likely to incorporate genetics into my teaching activities.	3	1			1
I have received a good return on investment by attending the Summer Genetics Institute. Institute.	5

28. The participants indicated the biggest SGI strengths:

• The laboratory experiences and the networking with other nurses interested in genetics.

• It introduces to SGI participants, methods in genetics that can be used in their future research.

• Building a foundation of my genomic research career. Fostering collegial professional relationships that have helped me with my professional growth over the years.

• 1. Greater appreciation for a much broader scope of physiology, biology, biochemistry, all sciences, and the impact on health and disease.2. Greater depth and understanding of research design and methodologies involving genetics, and the analytic techniques for studying phenomenon from a genetics perspective.3. Laboratory skills I learned during SGI have come in very handy in my current bench work."

• 1) Networking. I established relationships with 17 genetic nurses across the country. Over the years, which now number 9, I continue to work with many of those individuals in presentations, educational endeavors, and research. 2) Genetic and genomic competency and a sense of urgency in integrating genome science into nursing education. Since the SGI, I have spearheaded faculty efforts to incorporate genomics into our RN-to-BSN, basic BSN, MN, DNP, and PhD curricula. All all BSN students, 260 each year, complete a 2-credit course that is built on the nursing genetic and genomic essentials. This work would not have been accomplished if I hadn't attended the SGI.

29. The participants indicated aspects of SGI that could be improved:

• No applicable

• I was lucky to have had the full 2 month program. I think those trying to do in one month what we did in 2 months are missing out greatly.

• Voluminous amount of paper to deal with--perhaps CDs or a common internet location for course materials, to download or not, would be a reasonable option.

• 1) I have often thought it would be useful to have an electronic resource, like a Facebook page I suppose, to link SGI graduates who are interested in being linked. ISONG is the closest thing we have, but only a handful of my SGI cohort are ISONG members. Such a resource would be useful in genomic education, practice and research. 2) I have no suggestions for improving the SGI experience itself. I attended when it was 8 weeks, and it was outstanding.

30. The participants added “any other comments about SGI that would like to share” in a free form text box:

• This is an excellent opportunity that has been a leap-frog initiative for many nurses interested in incorporating genetics into their research.

• I was more excited about being accepted into the SGI than I was when I received notice of my acceptance into a PhD program. This has been a huge paradigm shift in my professional career development and research trajectory.

• Some kind or 2-3 day refresher course or update would be awesome!

B2. Identifying any responses or issues needing further discussion or investigation:

1. Question 7: Few participants provided answers, should the certificate names be spelled out to make sure they understand the acronyms? [Decision on 4/9/2012: leave unchanged]

2. Question 15a: Two participants provided the answers “RWJ Executive Nurse Fellowship in Leadership” and “genomics”; do we need to change the question to specify the program type since the answer genomics do not indicate which program? [Decision on 4/9/2012: leave unchanged]

3. Question 16a: Two participants with 2 and 1 research grant applications respectively did not answer the genetics component question. Should this question be mandatory to answer if 16 is answered yes? [Decision on 4/9/2012: yes, make mandatory]

4. Question 17b: One participant did not provide grant amount or length. Two provided all the requested information. Should we adjust the question language? [Decision on 4/9/2012: leave unchanged]

5. Question 21: One participant response is missing. An option could be to make answering all questions mandatory. [Decision on 4/9/2012: leave unchanged]

6. Question 21: [Decision on 4/9/2012: Add definition or example of a clinical research nurse to “Since attending the Summer Genetics Institute on how many funded research grants have you been a clinical research nurse (i.e. a nurse who supports study implementation within the context of a care delivery setting; not the PI, co-PI, or research project manager)?

7. Question 22: The instructions for where and when to send the CV needs to be made very clear. [Decision on 4/9/2012: yes, update survey emails and SharePoint, see separate documents]

8. Question 29a-b: One participant chose to email their CV, yet they have a response to question 29b. Should the survey skip them over 29b to 30? [Decision on 4/9/2012: leave unchanged]

9. Question 30: We will combine questions 29 and 30 (elected and appointed positions) into one. Will this language work: “Since attending the Summer Genetics Institute, have you been elected or appointed to any science, teaching or healthcare leadership positions?” [Decision on 4/9/2012: yes, combine]

10. Question 32: If the participant answers “I do not have a clinical practice” in 31, should the survey skip them over 32 to 33? [Decision on 4/9/2012: yes, skip over 32]

3. Feedback Sheet Analysis

A total of five alumni participated in the pilot test of the NINR SGI Alumni Survey. These individuals completed the web survey, provided their feedback on the survey instructions and questions, and participated in an interview with the Discovery Logic team to provide further feedback on the survey instrument.

Overall, all participants felt that the survey was very easy to navigate and complete, the questions and response options were clear and easy to understand, and that no important questions were missing from the survey. Sample comments from pilot test participants were:

• “The survey was very straightforward and logically put together.”

• “The survey was very thorough.”

• “My overall impression was the survey was seamless.”

Following, we provide more detailed feedback from the pilot test participants about the survey instrument.

C1. NINR SGI Alumni 2012 Pilot Survey:

1. Understanding of introduction, confidentiality statement, technical instructions, and questions

All five pilot test participants indicated that the survey introduction and technical instructions were clear and easy to understand. None of the pilot test participants had any problems with the introduction, technical instructions, and questions.

Participants felt very comfortable in the confidentiality of their survey responses and that their survey responses would only be reported in aggregate. Participants were not worried about the information they were sharing; the survey did not ask for sensitive information.

2. Clarity of questionnaire wording

All pilot test participants indicated that the wording of the survey questions was clear and easy to understand. None of the pilot test participants identified any survey questions that were difficult to understand.

3. ease of login, navigation, saving

Pilot test participants felt that the survey was very easy to navigate. None of the pilot test participants had any issues with logging in to the survey, navigating through the survey, or saving their survey responses.

One pilot test participant indicated that she would have liked to have been able to go back to previous survey pages to review/change her responses [SharePoint does not support this feature].

4. branching of questions

Pilot test participants had no issues with the branching of questions.

5. response options

Pilot test participants had no issues with the response options of the survey questions.

6. whether any important issues are missing from the survey

None of the pilot test participants felt that any important issues were missing from the survey.

7. Sending the CV versus answering survey questions

In general, pilot test participants liked the idea of forwarding their CVs to NINR versus filling in detailed information in the survey that could be found in their CV (e.g., titles of grants and publications). They felt that the ability to send their CVs to NINR (versus completing all the detailed survey questions that asked for information contained in their CVs) made the survey easier and less time consuming to complete.

Some pilot test participants started to cut and paste aspects of their CV into the survey web site and then decided to send their CV to NINR.

Two pilot test participants felt that the survey instructions (e.g., prior to the questions that require survey respondents to provide detailed information that is included in their CVs) should more clearly state that sending the CV to NINR will reduce the time it takes to complete the survey.

A few pilot test participants indicated that they chose to email their CV but there were not clear instructions in the survey that explained how to send their CVs (e.g., were to email the CV).

8. Likelihood of completing the survey

All pilot test participants indicated that they would have completed the survey if they had not been asked to participate in the pilot test.

9. Other feedback

Some pilot test participants were interested in receiving feedback about the results of the survey (once it is administered to all alumni) and the outcomes of the survey.

4. Survey Recommendations

1. Include a specific deadline or a time range (from 4/1-4/16 for example) for submitting the survey. [Decision on 4/9/2012: yes, update survey emails, see separate documents]

2. More clearly state in the instructions that sending the CV to NINR will increase the ease of completing the survey and reduce the time to complete the survey. [Decision on 4/9/2012: leave unchanged]

3. Provide clear instructions for how and where to email the CVs. [Decision on 4/9/2012: yes, update survey emails and SharePoint, see separate documents]

4. Change “I chose to email my CV to NINR” to “I chose to email my CV, resume, or biosketch to NINR.” [Decision on 4/9/2012: yes, change to I chose to email my CV or resume to NINR.” Also change “CV” to “CV or resume” through survey and emails.]

5. Make changes to the questions based on the question analysis on page 9 above. [Decision on 4/9/2012: yes, as agreed upon]

6. Instructions for full survey data analysis:

   1. One of the business rules for deciding what data to extract from the CV could be “If in doubt, extract the data.” That way we can sort through the data when it has been combined and have a better “big picture” feel for it.

   2. The data collected this summer from the entire survey for question 29/30, will be sorted into meaningful bins. Here are 4 bins to start with and more can be added as the data is analyzed: Obvious paid full time positions (i.e. professor, administrator); professional activity positions (i.e. AEA board director, state official); Conference activities (i.e. abstract reviewer); Journal activities (i.e. Journal editor).

[Decision on 4/9/2012: agreed.]

5/21/2012 v5 Page 10 of 10