Part B Supporting Statement Realtor and appraiser

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each stratum. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

As noted in part A, this information collection comprises a suite of customer satisfaction surveys of the VBA LGY Service. The respective target populations for these surveys are as follows:

• Survey #1 – Survey of Realtor Satisfaction with the VA Home Loan Guaranty Process (i.e. Realtor Survey): gathers satisfaction data from realtors nationwide.

• Survey #2 – Survey of Appraiser Satisfaction with the VA Home Loan Guaranty Process (i.e. Appraiser Survey): gathers satisfaction data from appraisers currently serving on the VA Fee Panel.

This section describes the universe of respondents for each survey, and the sampling plan that will be employed to achieve a representative sample for each survey.

Realtor Survey

For the Realtor Survey, useful data can be obtained from realtors since they have direct contact with the veterans and are often involved in the veterans’ financing aspect of the transaction. To ensure useful data, the target audience will be mainly derived from “veteran rich” areas. The survey will include realtors who are both familiar and unfamiliar with the VA loan program. The purpose of the survey will be multi-faceted. First, it will serve as an awareness tool by revealing what percentage of participants has knowledge of the VA loan program. Second, it will also help gather information about realtors’ perceptions of the program. Lastly, it will reveal the reason(s) why the realtors promote or don’t promote the Loan Guaranty program.

Table 5 displays the number of realtors to be surveyed, the expected response rate, and the expected yield of completed Realtor Surveys. VA anticipates a response rate of 30%.

TABLE 5: REALTOR SURVEY, EXPECTED RESPONSE RATE AND SURVEY YIELD
Number of Realtors	Expected Response Rate	Completed Surveys Expected
6,000	30%	1,800

Appraiser Survey

For the Appraiser Survey, the size of the eligible population will be equal to the number of appraisers currently serving on the VA Fee Panel. There are an estimated five thousand appraisers on the current panel and VA expects a response rate of 40%.

Table 6 displays the sampling frame and the expected response rates and the expected yield.

Table 6: APPRAISER Survey, EXPECTED RESPONSE RATE AND SURVEY YEILD
Number of Appraisers	Expected Response Rate	Completed Surveys Expected
5,000	40%	2,000

2. Describe the procedures for the collection of information, including: Statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose in the proposed justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The Realtor Survey will entail stratified random sampling of 6,000 realtors geographically located in veteran rich areas. VA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.

The Appraiser Survey will be sent to all of the appraisers currently on the Fee Panel. Again, 95% confidence intervals are desired for the statistical estimates of customer satisfaction that are produced.

3. Describe methods used to maximize the response rate and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

LGY will obtain the services of a contractor to develop, administer, and analyze this set of surveys.

Strategies to Maximize Response Rates

LGY will employ methods to minimize respondent burden and to maximize survey response rates. This section identified the strategies to be employed to reach these objectives. Each strategy is outlined below.

• Strategy # 1 to Maximize Response Rates: Using Web Technologies for Ease of Response

Both the Realtor and Appraiser Surveys will be web-based. The web address that the surveys will be posted on will be included in all of the mailing notifications as indicated below. This initial notification will include a cover letter and an URL and password. As all credible realtors and appraisers will have computers and internet connections, the surveys will be administered online to maximize the timeliness, efficiency, and response rate of data collection.

The web-based surveys will be developed with the end user in mind; VA will provide a user-friendly website for the realtors and appraisers to complete the survey.

The on-line survey technology will incorporate several features to maximize response rates and respondent usability. These include a password system, which prevents any one person from completing more than one survey and allows respondents to begin the survey, then come back at a later point to finish it (i.e., this is particularly useful for long surveys). Other features include user-friendly drop-down boxes, internal links to the directions throughout the survey, and internal links to key terms and definitions.

• Strategy # 2 to Maximize Response Rates: Using Advance and Follow-Up Mailings to Publicize the Surveys and Encourage Response

LGY will use a 5-step survey and follow-up process to administer the surveys (see Table 7 below). An increase in the overall response rate is the major advantage of using this process. The use of a letter as a follow-up tends to increase the response rate by between 5 and 8 percentage points.

Mailing	Mailing Material	REALTOR Survey	APPRAISER Survey
#1	Pre-notification/cover letter		
#2	Notification/cover letter Notification/cover letter w/ URL & password Paper survey	 	 
#3	1st letter w/ URL & password		
#4	2nd letter w/ URL & password	 	 
#5	3rd letter w/ URL and password		

• Strategy # 3 to Maximize Response Rates: Conduct Cognitive Labs/Pre-testing of Surveys

The contractor will conduct cognitive labs with three or more test users for each survey to determine whether respondents understand the survey questions and answer choices, as intended. In conjunction with LGY, the contractor will draw a small pool of names from potential participants in each of the surveys for inclusion in the cognitive labs. Cognitive lab participants will be drawn from the same population that will be used for the main study. The contractor will submit the list of potential participants to VBA for review and approval. Once identified, the contractor will contact potential participants by telephone and ask them to participate. Cognitive lab sessions will take place in the metropolitan Washington, DC area.

Once the participants have been selected, VA will conduct cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Cognitive labs are one-on-one sessions with potential survey participants, in which respondents are asked to complete the questionnaire while thinking aloud. The primary purpose of these sessions is to gather feedback on survey questions and answer choices to ensure they are easily understood and correctly interpreted. Outcomes of cognitive labs include, but are not limited to: addition or omission of specific questions, changes to wording of questions, clarification of question response options, addition of response options, and changes to ordering of questions.

The contractor will prepare a summary report of the cognitive testing session for paper and web versions of the customer satisfaction surveys. The results of the cognitive labs will be taken into account when revising and finalizing the survey questionnaires.

• Strategy # 4 to Maximize Response Rates: Maintaining a Toll-Free Survey Hotline

During the period that the surveys are in the field, the contractor will provide and maintain a toll-free telephone line to answer any questions respondents and regional office points of contact may have about the survey (e.g., how to interpret questions and response items, the purpose of the survey, how to get another survey if their copy has been lost/damaged). Project staff will be available to answer telephone calls during regular business hours (8:30 a.m.-6 p.m. ET). A voice messaging system will be available to receive messages after regular business hours so after-hours calls can be responded to within 24 hours.

Strategy # 5 to Maximize Response Rates: Excluding Questions of a “Sensitive” Nature

None of the questions included in the surveys are sensitive, or private in nature, which will encourage compliance.

Strategy # 6 to Maximize Response Rates: Assuring and Maintaining Confidentiality

Survey respondents for all surveys will be assured that their personal anonymity will be maintained. All hard copy questionnaires will be scannable, and consist of approximately eight printed pages, printed back to back with a numeric Litho-Code on the front and back cover. Both realtors and appraisers will be provided unique passwords that will allow the contractor to identify when a respondent has completed the survey and exclude them from further reminder letter or postcards.

Strategy # 7 to Maximize Response Rates: Secure Networks and Systems

The contractor will have a secure network infrastructure that will protect the integrity of the databases, the survey application, and all associated server resources. The servers must be protected by a strong firewall system and the operations center must be in a secure temperature-controlled environment with video surveillance, where network services are continually monitored by automated real-time programs to ensure the integrity and availability of all critical components. All key servers will be supported by a backup power supply that can continue to run the systems in the event of a power outage. Additionally, the contractors must be immediately alerted if critical monitor thresholds are exceeded, so that they can proactively respond before outages occur.

Approach to Examine Non-Response Bias

Non-response bias refers to the error expected in estimating a population characteristic based on a sample of survey data that under-represents certain types of respondents. Stated more technically, non-response bias is the difference between a survey estimate and the actual population value. Non-response bias associated with an estimate consists of two components – the amount of non-response and the difference in the estimate between the respondents and non-respondents. While high response rates are always desirable in surveys, they do not guarantee low response bias in cases where the respondents and non-respondents are very different. Two types of non-response can affect the interpretation and generalization of survey data: item non-response and unit non-response. Item non-response occurs when one or more survey items are left blank in an otherwise completed, returned questionnaire. Unit non-response is non-participation by an individual that was intended to be included in the survey sample. Unit non-response – the failure to return a questionnaire – is what is generally recognized as survey non-response bias.

There are two approaches to tackling the effects of non-response. One is to minimize the chances of non-response at the data collection stage. This may involve introducing measures which aim to maximize the response rate. The other approach is to make statistical adjustments at a survey follow-up stage when all the data is collected. Both approaches are described in the next paragraphs of this section.

Since it is not always possible to measure the actual bias due to unit non-response, there are strategies for reducing non-response bias by maximizing response rates across all types of respondents. In the face of a long-standing trend of declining response rates in survey research (Steeh, 1981; Smith, 1995; Bradburn, 1992; De Leeuw & Heer, 2002; Curtin & Presser, 2005), these strategies include:

• Use of notification letters, duplicate survey mailings, reminder letters and postcards.

• Use of novelty in correspondence such as reminder postcards designed in eye-catching colors.

• Use of an extended survey field-period to afford opportunities to respond for subgroups having a propensity to respond late (e.g., males, young, full-time employed).

• Use of well-designed questionnaires and the promise of confidentiality.

• Providing a contact name and telephone number for inquiries.

Employing these strategies to the administration of these surveys will be crucial for maximizing high response rates across all respondent types (see section on maximizing response rates above).

Non-response follow-up analyses can help identify potential sources of bias and can help reassure data users, as well as the agency collecting and releasing the data, of the quality of the data collected. The approach to examining the presence of non-response bias will be conducted as follows:

• For the Realtor and Appraiser Surveys, Compare the Demographics of Respondents from the VA LGY Surveys to the Demographics of Non-Respondents from the VA LGY Surveys. To examine the presence of non-response bias, VA will compare the demographics of responders (i.e., those who responded to the VA LGY Surveys) to the non-responders (i.e., those who did not respond to the VA LGY Surveys).

• Realtor Survey. The comparison between responders and non-responders will be made on the following variables for this survey:

• Region – it is possible that participants from a certain part of the country (i.e., region) may respond to the survey at a higher rate than those who are from another part of the country.

• Gender – it is possible that participants from a certain gender (i.e., male) may respond at a higher rate than their counterpart.

• Experience in Real Estate – it is possible that respondents and non-respondents may differ with respect to their years of experience as realtors and familiarity with the VA home loan program.

• Appraiser Survey. The comparison between responders and non-responders will be made on the following variables for this survey:

• Region – it is possible that participants from a certain part of the country (i.e., region) may respond to the survey at a higher rate than those who are from another part of the country.

• Gender – it is possible that participants from a certain gender (i.e., male) may respond at a higher rate than their counterpart.

• Experience in Appraisals – it is possible that respondents and non-respondents may differ with respect to their years of experience as fee panel appraisals and familiarity with the VA home loan program.

• Volume of VA Appraisals Performed – it is possible that respondents and non-respondents may differ with respect to the number of appraisals they perform annually.

In addition, because the survey of appraisers will be a conducted on the census of VA fee panel appraisers, VA will be able to generalize these findings to the entire frame.

Based on the steps discussed above, VA will identify issues with respect to non-response bias for both surveys.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions of 10 or more individuals.

The contractor will conduct cognitive labs with three or more test users for each survey, to determine whether respondents understand the survey questions and answer choices, as intended. Working closely with VBA, the contractor will draw a small pool of names from potential participants in each of the surveys for inclusion in the cognitive labs. Cognitive lab participants will be drawn from the same population that will be used for the main study. The contractor will submit the list of potential participants to VBA for review and approval. Once identified, the contractor will contact potential participants by telephone and ask them to participate. Cognitive lab sessions will take place in the metropolitan Washington, DC area.

Once the participants have been selected, VA will conduct cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Cognitive labs are one-on-one sessions with potential survey participants, in which respondents are asked to complete the questionnaire while thinking aloud. The primary purpose of these sessions is to gather feedback on survey questions and answer choices to ensure they are easily understood and correctly interpreted. Outcomes of cognitive labs include, but are not limited to: addition or omission of specific questions, changes to wording of questions, clarification of question response options, addition of response options, and changes to ordering of questions.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The LGY contact person is Katherine Faliski, 202-461-9527.

For the last set of surveys, LGY contracted the services of ICF International to administer the survey. The following is a list of the persons involved in the survey.

• Dr. Christopher Spera, ICF International 703-934-3446

• Mr. John Kunz, ICF International, 703-934-3627

LGY plans to contract the services of a contractor for these surveys.