Economic Impacts of the Text REtrieval Conferences (TREC)

NOTE TO REVIEWERS: This survey will be used to collect data for RTI’s economic impact study of the NIST Text REtrieval Conferences (TREC). The survey questions below will be uploaded to RTI’s Web site after OMB approval. Please note that the text colored RED indicates instructions for RTI’s survey development team that will program the Web version of this survey. Please read this text because it may help you understand the flow of the survey. Of note, in several places, dollar figures and years are shown in YELLOW highlighting. These numbers are calculated based on responses to earlier questions. In the Web survey, the calculations will appear automatically to participants.

Economic Impact of the
Text REtrieval Conferences (TREC)

A Survey Sponsored by the National Institute of Standards and Technology (NIST)

The National Institute for Standards & Technology (NIST) has commissioned RTI International, a not-for-profit research institute, to conduct a retrospective impact assessment of the Text REtrieval Conference (TREC) Program.

Since the TREC Program was launched in 1992, NIST has provided a multitude of supportive activities for the IR research field, such as the development of measurement methods and evaluation infrastructure; the coordination of annual test evaluation exercises; and the hosting of annual workshops. The goal of this study is to better understand the economic and research contributions of these activities.

The following survey is an important effort that seeks to provide insights into the effectiveness of the TREC Program. The survey is voluntary and is estimated to take approximately 20 minutes to complete. It requests:

• information on your use of TREC resources,

• estimates on the value of TREC resources to the research conducted by your organization, and

• data on the impact that using TREC resources has had on the performance of IR systems.

Participants in this survey will be entered into a raffle drawing for an Apple iTouch (retail value $400). Only participants who complete the survey will be entered into the drawing. Please provide contact information in the final section so that we may contact you if you are selected.

Responses to this survey will be kept strictly confidential. At no time will any individual’s name, any company or university name, their participation, or identifiable response be released by RTI to any third party, including NIST. The data that survey respondents provide will only be used to present aggregate analytical findings to NIST in the form of a final report that will be publicly released by the end of this calendar year following a peer review process.

Questions about the survey should be directed to Dallas Wood at (919) 541-8743 or [email protected] [US Eastern Time], or Brent Rowe at (415) 848-1317 or [email protected] [US Pacific Time].

Note: This questionnaire contains collection of information requirements subject to the Paperwork Reduction Act (PRA). Notwithstanding any other provisions of the law, no person is required to respond to, nor shall any person be subject to penalty for failure to comply with, a collection of information subject to the requirements of the PRA, unless that collection of information displays a currently valid OMB Control Number. The estimated response time for this questionnaire is 20 minutes. The response time includes the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this estimate or any other aspects of this collection of information, including suggestions for reducing the length of this questionnaire, to the National Institute of Standards and Technology, Attn., Gregory Tassey, [email protected], (301) 975-2663. The OMB Control No. is 0693-0033, which expires on 10/31/2012.

Before beginning the survey, please check the box that best characterizes your organization.

Organization Description
Information Retrieval Service or Software Company
Other Software Company
University or Academic Research Laboratory
Government or Institutional Research Laboratory
Other (please specify):	     

(NOTE: The exact questions shown throughout this survey depend on the type of organization currently employing you to conduct IR research.)

1. Basic Information on the Use of TREC Products and Services

The following questions ask about how TREC has benefited your research.

1a

Since 1992, TREC has hosted annual workshops, developed test collections, developed evaluation methods, and published research papers in order to foster information retrieval (IR) research. Please indicate how important each TREC product or service is for your IR research.

	Product or Service Not Used	Product or Service Used and Not Very Important for IR Research	Product or Service Used and Somewhat Important for IR Research	Product or Service Used and Very Important for IR Research
Workshops
Test collection(s)
Evaluation method(s)*
Research papers

* Evaluation methods refer to the performance metrics (such as Mean Average Precision) and experimental designs developed through TREC to evaluate the performance of IR systems.

<<If respondent answers NOT USED for each aspect of TREC, direct them to goodbye screen. >>
Comments:      

1b

TREC has pursued a large number of tracks aimed at fostering research in specific IR areas. Typically, each track is associated with its own set of resources (workshops, test collections, evaluation methods, or research papers). In the table below, for those tracks associated with the resources you have used, please indicate how important those resources have been for your IR research. If you have not used the product or service, please check the appropriate box.

Track	Years Held	Product or Service Not Used	Product or Service Used and Not Very Important for IR Research	Product or Service Used and Somewhat Important for IR Research	Product or Service Used and Very Important for IR Research
Ad hoc	1992–1999
Blog	2006–present
Cross-language	1997–2002
Enterprise	2005–2008
Filtering/Routing	1992–2002
Genome	2002–2007
Interactive	1994–2002
Legal	2006–present
Question answering	1999–2007
Spam	2005–2007
Speech	1997–2000
Video/TRECVid	2001–present
Web	1999–2004, 2009
Other (___________)

Comments:      

2. Detailed Information on Personal Use of TREC Products and Services

The following questions ask about how often and in what ways you use different aspects of TREC.

2a

Participation in TREC Workshops and Use of TREC Resources/Output

‘92

‘93

‘94

‘95

‘96

‘97

‘98

‘99

‘00

‘01

‘02

‘03

‘04

‘05

‘06

‘07

‘08

Participated in TREC workshop

Used additional TREC resources (test collection, evaluation methods*, or research papers)

In the table below, please identify the years in which you used the products and services associated with TREC. Specifically, in what years did you participate in a TREC workshop and/or use the test collection(s), evaluation method(s), or papers generated by TREC. For example, if you used the test collection from the 1994 Ad Hoc track in 2004, then you would check ‘04 for “Used additional TREC resources.”

*Evaluation methods refer to the performance metrics (such as Mean Average Precision) and experimental designs developed through TREC to evaluate the performance of IR systems.

Comments:      

2b

Use of TREC Test Collections
The questions in the table below ask about how you have used the test collections associated with TREC. In some cases, you may not be able to provide an exact response. In those instances, please answer by providing your best estimate.

#	Question	Answer
I.	Have you published any research that uses the test collections from TREC?
Ia.	If yes, how many papers have you published in peer-reviewed journals?	     
Ib.	If yes, how many papers have you presented at conferences?	     
II.	Have you used test collection data from TREC for tasks not studied at TREC?
IIa.	If yes, please elaborate on how you used these data?      

Comments:      

2c

Use of TREC Evaluation Methods

The question in the table below asks about how you have used evaluation methods associated with TREC.

#	Question	Answer
I.	Have you used TREC evaluation methods to study the performance of an IR system using non-TREC test collections?

Comments:      

2d

Use of TREC Research Papers

The questions in the table below ask about how you have used research papers associated with TREC. In some cases, you may not be able to provide an exact response. In those instances, please answer by providing your best estimate.

#	Question	Answer
I.	Approximately how many papers have you read that were generated by TREC?
II.	Have you ever referenced a paper from TREC in a peer-reviewed journal article?
III.	Have you ever referenced a paper from TREC in a patent filing?

Comments:      

3. The Value of TREC Products and Services to Your Organization’s IR Research Activities

The purpose of the following questions is to better understand the value your organization has received from TREC. If you have worked at multiple organizations that have benefited from TREC, please focus only on how TREC resources are used and valued at your current employer. If you do not engage in IR activities at your current employer or would prefer to provide information on the value of TREC to another organization at which you have worked, please CLICK HERE to go back to the first question on the survey and change your response, if necessary.

3a

How many FTEs does your organization currently employ in IR research and development?
     FTEs

3b

The following questions ask about the IR research activities pursued by the individuals employed by your organization. Given that you may not be familiar with the research activities of all these individuals, what percentage of the total FTEs you indicated in Q3a do you believe you would be able to speak for in your responses?
     %

3c

Do these researchers use TREC resources when conducting their research?

If yes, how many years have these researchers been using TREC resources?      

In what year did you begin using these TREC resources?      

3d

What percentage of your organization’s research activities are devoted to the following research areas?

Research Area	Percentage of Research Activity
Desktop Search	     %
Enterprise Search	     %
Database Search	     %
Web Search	     %
Other IR Research	     %
Non-IR Research	     %
Total	100%

Comments:      

3e

In the previous questions, we asked you to consider your organization’s IR system development and evaluation activities. These are activities that TREC was designed to foster by offering a variety of resources (such as test collections and workshops).

Suppose that NIST was considering discontinuing TREC next year. However, a different organization offered to perform the exact same services and the determination of research areas for specific tracks would be determined by an external panel of academic and private IR researchers. Would your organization be interested in purchasing these services?

Would the value of the services offered by the TREC-like program be worth at least $2,000 to your research over the next year?

<<The figure above is calculated differently for non-profit and for-profit organizations. Non-profit organizations will see an initial value that equals $2,000 (the cost of membership for Non-Profit Organizations in the LDC) plus 50% of the value of the FTEs they are responding for (calculated using responses above and BLS data). Private sector organizations will see an initial value that equals $20,000 (the cost of membership for For Profit Organizations in the LDC) plus 50% of the value of the FTEs they are responding for (calculated using responses above and BLS data).

If the person answers that “YES” they value TREC this much, they will be presented with increasing values (in increments of .7, 1, 2, and 10x their FTEs) until they answer “NO”. If they answer “NO”, they will be presented with decreasing values (in increments of .25., .1, and .01x their FTEs) until they answer “YES” or reach $0. In the example below, we present increasing values (that only show up in Microsoft Word if you have completed questions 3a, 3b, and 3c above) on the assumption that the answer to this question will be “YES”. The web version of this survey will adapt up or down and show respondents a new number on a new screen each time they say “YES” or “NO”.>>

Would the value of the services offered by the TREC-like program be worth at least $2,000 to your research over the next year?

Would the value of the services offered by the TREC-like program be worth at least $2,000 to your research over the next year?

Would the value of the services offered by the TREC-like program be worth at least $2,000 to your research over the next year?

Would the value of the services offered by the TREC-like program be worth at least $2,000 to your research over the next year?

To better understand your responses, please elaborate on the factors you considered in making this estimate?      

3f

Over the past 16 years, TREC has provided a number of resources (test collections, evaluation methods, workshops, and research papers) to help foster IR research. Without the use of TREC resources, your organization might have

• spent more time creating new data sets (or using smaller or less robust data sets),

• spent more time evaluating IR systems and been less confident in evaluation results,

• spent more time going down faulty paths, and/or

• created IR systems of lower quality.

Earlier in this survey you indicated that researchers in your organization have used TREC resources for  0 years. Suppose that you were asked by a project manager or your organization’s financial accountant to justify the use of these resources by specifying the value they contributed to your research.

Do you believe these resources contributed at least $2,000 of value to your research?

NOTE: We are not asking how much your organization would have or could have paid, but rather, what you should have paid, looking back. Consider the full impact that TREC has had on your research (including all the applicable benefits listed above)?

<< The figure above is calculated differently for non-profit and for-profit organizations. Non-profit organizations will see an initial value that equals $2,000 (the cost of membership for Non-Profit Organizations in the LDC) plus 50% of the value of the FTEs they are responding for (calculated using responses above and BLS data), multiplied by the number of years they have been utilizing TREC resources. Private sector organizations will see an initial value that equals $20,000 (the cost of membership for For Profit Organizations in the LDC) plus 50% of the value of the FTEs they are responding for (calculated using responses above and BLS data), multiplied by the number of years they have been utilizing TREC resources.

If the person answers that “YES” they value TREC this much, they will be presented with increasing values (in increments of .7, 1, 2, and 10x their FTEs) until they answer “NO”. If they answer “NO”, they will be presented with decreasing values (in increments of .25., .1, and .01x their FTEs) until they answer “YES” or reach $0. In the example below, we present increasing values (that only show up in Microsoft Word if you have completed questions 3a, 3b, and 3c above) on the assumption that the answer to this question will be “YES”. The web version of this survey will adapt up or down and show respondents a new number on a new screen each time they say “YES” or “NO”.>>

Do you believe these resources contributed at least $2,000 of value to your research?

Do you believe these resources contributed at least $2,000 of value to your research?

Do you believe these resources contributed at least $2,000 of value to your research?

Do you believe these resources contributed at least $2,000 of value to your research?

To better understand your responses, please elaborate on the factors you considered in making this estimate?      

4. The Impact of TREC on IR System Quality

Questions in this section ask about how these benefits have translated into improvements in the quality of the IR systems your group has developed. Please answer these questions to the best of your ability.

In some cases, you may not be able to provide an exact response. In those instances, please answer by providing your best estimate.

4a.

There are many ways to value the improvement in the quality of IR systems. In order to focus on the impact that TREC has had specifically, we are particularly interested in the improvement you have seen in your IR systems' Mean Average Precision (MAP).

Keeping this mind, did the quality of the IR system(s) developed by the group of researchers you are responding for improve during the time your organization used TREC products and services (on average as measured by Mean Average Precision)?

If yes, by how much (on average as measured by Mean Average Precision)?

      % improvement in IR system performance

How much of this change would you attribute to TREC?

      % of the total change

Comments:      

4b.

The improvement in web search over the past 17 years has resulted from a variety of changes including improvements in computer and networking hardware as well as in software, including IR systems, Specific to web search, we are interested in the impact of TREC.

Improvement in web search can be described by improvement in relevance, satisfaction, or many other factors; we are interested in the impact on the time needed to find the desired information. For example, if you used a web search engine today to look for a new, yellow stripped shirt, it might take you 5 minutes to find what you were looking for; if you used a web search engine from 1993 (even with today’s computer hardware software and the multitude of web sites available), it would likely take you much longer.

If it were possible to hold improvements in hardware and non-IR software constant, in your opinion, did the time it takes to fill an information need (equally well) using web search engines improve (decrease) between 1992 and 2009 as a result of improvements in IR systems?

If yes, by how much?

      % reduction in time required to fill information need

How much of this improvement would you attribute to TREC?

      % of total improvement

Comments:      

5. The Impact of TREC on Academic Research Activity (for Academic Respondents only)

5a

Do you believe the number of individuals pursuing doctoral and Masters degrees has increased as a result of TREC?

5b

How many doctoral candidates in your department have written dissertations/theses that used TREC resources (such as test collections, research papers, etc.)?

     

5c

How many Masters students in your department have earned degrees that depended on theses or projects using TREC resources (such as test collections, research papers, etc.)?

     

5d

Do you believe that competition in TREC workshops improves the employment prospects of graduate students in your department?

5e

In the table below, please rank the following organizations in order of where your graduate students typically find employment after graduation. Mark the organization that employed the most of your students (=1) and so on to the organization that employed the fewest (=6).

Type of Organization	Ranking
Information Retrieval Service/Software Company
Other Software Company
University or Academic Research Laboratory
Government or Institutional Research Laboratory
Other (please specify):

Comments:      

6. Thank you for participating. Finally, please fill out some information about yourself and your employer.

This survey is confidential, and at no time will any individual’s name, any organization’s name, participation, or identifiable response be released by RTI to any third party, including the National Institute for Standards & Technology (NIST). This information will only be used to characterize the size of organizations benefiting from TREC’s activities. All data will be held strictly confidential and only reported when aggregated with data provided by all respondents and reported in that form in a final report to NIST.

6a

Please provide the following information about yourself:

Name:

     

Title:

     

Employer:

     

E-mail:

     

Geographic location:

6b

<<For respondents that indicated they were in the private sector>> Please provide any of the following information that you are able to:

Stock trading symbol (ticker), if publicly traded:

     

2008 approx. company annual sales (if no ticker):

Million US$

2008 approx. company total annual R&D budget (if no ticker):

Million US$

2008 approx. annual R&D budget for IR

Million US$

2008 approx. company employment

FTEs

2008 approx. number of IR researchers

FTEs

6b

<<For respondents that indicated they were in academia, government, or a government lab>> Please provide any of the following information that you are able to:

2008 approximate funding for your research group

Million US$

Percentage of funding for research group devoted to IR research

%

2008 approximate number of full-time IR researchers/professors

FTEs

2008 approximate number of student IR researchers

FTEs

6c

How can TREC best provide benefits to the IR research community and IR-relevant industry in the future? (e.g., new tracks, new evaluation procedures, etc.)

     

6d

Would you be willing to participate in a 15-minute interview to discuss your responses to this survey?