Appendix A - Evaluation Results

APPENDIX A: Evaluation Recommendations (excerpt from report)

August 2010

Process and Needs Assessment Evaluation of the HINTS Program

Final Report

Prepared for

National Cancer Institute

6130 Executive Blvd.

Bethesda, MD 20892-7338

Prepared by

Bridget Kelly, Ph.D., M.P.H.

Linda Squiers, Ph.D.

Marjorie Margolis, B.A.

Julia Kish Doto, Ph.D.

Elizabeth Adams, M.S.

Molly Lynch, M.P.H.

Paula Eguino-Medina, B.S.

Debra Pinkney, M.S., M.P.H.

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

RTI Project Number 0212257.001.001

Recommendations and Summary

Overall, comments regarding the HINTS survey and website were predominantly positive. Respondents provided interesting feedback and important suggestions. Asked whether they would be likely to use the HINTS data in the future, most users said “yes.”

4.1 Recommendations

Based on the findings, RTI makes the following recommendations to NCI for improvement to HINTS and the HINTS website:

4.1.1 Recommendations Overall

• Increase outreach about HINTS by contacting graduate schools in communication, public health, and statistics about the availability of this free public-use dataset. Also consider adding links to HINTS from other DHHS websites or advertising at public health conferences and on popular health websites (e.g., WebMD.com). Another option is to create web widgets, that users could place on their own websites to keep updated on HINTS findings or news. The HINTS program might also work to create partnerships to promote HINTS on the sites we know users visit, such as cancercontrolplanet.cancer.gov or the ACS website.

• Consider reorganizing the HINTS website and inserting tabs to make specific sections easier to find. One option would be to organize content according to the target audience. For example, a section for “data users” could include the datasets and codebooks, while a section “for cancer center communication directors” might include the brochure, briefs, and recent findings. Other tabs could include “FAQs” or “Reports.”

• Use plain language principles for those web pages most relevant to results users in order to ensure the literacy level is not too high. Some pages might include subheads with less text. Users could then click on the subhead in which they are interested to get more information on a particular topic. Briefs and other products most likely to be used by results users should be kept at a lower literacy level.

• Add a blog to the website where users could share information about data analysis methods and other topics. This blog could be monitored periodically by an NCI staff member who could intervene to correct any misinformation.

• Consider adding a feature to engage users more in content generation. This might involve a submission system, with which users could submit ideas for new topic modules or questions for the next HINTS survey.

• Add dynamic features to the website to give it a more updated feel. (Consider a running ticker across the top with new findings or news).

• Provide a subscription-based e-mail newsletter with updates or new results from HINTS analyses. This tool could also be used to more actively target specific types of users with the HINTS products most useful to them (e.g., the Cancer Trends Report for cancer center communication directors; updated documentation on weighting to the data users). When they subscribe they could answer questions about type of data user and stakeholder group (e.g., cancer control planner vs. academic) so that a newsletter could be tailored accordingly.

• Continue developing HINTS Briefs and consider other ways to use the Quick Facts feature elsewhere on the website or in other HINTS products or publications.

• Rename the My Toolkit to make it more clear what this feature is for (e.g., “my documents” or “my items for download”) or add to the website the capability of directly downloading documents without needing to use My Toolkit.

• Consider adding questions to HINTS on topics such as additional cancer types, use of more specific Internet sites, use of social media and topics specific to cancer patients (e.g., survivorship issues, end-of-life issues).

4.1.2 Recommendations Specific to Data Users

• Provide more documentation on weighting of the sample and the sampling methods more generally or where documentation already exists, simply make it more prominent. Documentation on sampling should specifically detail how the cluster sampling was dealt with (multiple respondents in one household).

• Consider providing more documentation on reliability and validity of HINTS items.

• Strive for consistency in question wording on future iterations of HINTS and across modes, when possible.

• Update the trend report to include methods for analysis across three time periods. Continue to include code for SAS and STATA for analyzing across all three time periods.

• As response rates are of concern, add information about trends in response rates for different survey methods and other national surveys to provide context to HINTS response rates.

• Make the list of publications and presentations searchable by key word or HINTS question. Make it more obvious that clicking on the links will take the user to the to the citation on PubMed. It may also be useful to add works in progress.

4.1.3 Recommendations Specific to Results Users

• Make clear the reasons why collecting local data is not feasible given the sample of the HINTS survey.

• Consider placing some results, quick facts, or highlights on the home page of the HINTS website. Topics could rotate to keep results users interested in returning to the site.

• Consider the wide range of audiences listed by results users when deciding upon new topics for HINTS briefs.

• Revise briefs so that “quick facts” are closer to the top in the HTML versions and add footnotes to clarify charts and graphs. Make it more obvious that briefs are printable in PDF form.

• Develop a communication plan for each of the different target audiences which would include specific dissemination channels and strategies relevant for the audience.

• Consider using current target audiences in promotional strategies and efforts. For example, the HINTS program could disseminate copies of HINTS brochures or briefs to members of the PAN steering committee and other results users, who can pass them out to others in their cancer centers and other work places. Graduate students could become HINTS ambassadors, charged with disseminating information about HINTS to classmates via social media and word of mouth.

4.2 Summary

The focus groups and interviews provided valuable data on current and future uses of HINTS. Overall, impressions of HINTS were very positive. However, both current and potential users had numerous ideas for improvements to the data, the documentation, the website, and presentation of results.

As changes are made to the HINTS website in the future, it will be important for NCI to continue soliciting feedback from all of these types of users. Some ways HINTS could make this feasible would be to add a link on the website on which users could click to send an e-mail providing comments or suggestions. Alternatively, NCI might initiate periodic web surveys to solicit suggestions (these could pop up when a user visits the site). An additional idea might be to recruit a panel of users that could meet annually to provide feedback or pretest new ideas. Ideally, such a panel would include both current and potential users of both data and results and would include representation from many of the different subsegments (e.g., academics, cancer control planners, cancer center communication directors, students, federal employees).

Final Report 3