Attachment A - 2022 FDIC SBLS Protocol 11-5-2020_final

2022 FDIC Survey of Small Business Lending

Cognitive Testing Protocol

Aryn Hernandez

Jessica Nickel

Research Questions:

• Do respondents interpret the questions and instructions in a manner that meets the questionnaire’s intent?

• Do respondents comprehend the terminology used throughout the questionnaire? (If not, what terminology are they using?)

• Will respondents need to go to other people or departments within their organization to report the data requested on the questionnaire?

• Overall, is the requested information available in respondent’s records?

Expected Length of Interview: 60 minutes

Informed Consent: Respondents will be asked to complete a consent form electronically before the time of the interview.

Introduction:

• Introductions from everyone on the call

• Explain purpose of meeting: to understand the process used to answer questions

• Explain that we are not testing the respondent – we only want information to help us improve the survey

• Structure of meeting: general and specific questions about the form itself, as well as the organization

Before discussing the questionnaire:

• To begin, can you please tell me a little about your bank?

• What is your role at this bank?

• Have you filled out the SBLS in the past?

General probes to use throughout the interview:

• How would you go about answering this question?

• Reflect back R’s specific answer: “you said…”

• How did you arrive at this number/answer this question? / Can you tell me more about this?

  • How accurate is this estimate?

• What records (if any) would you look at? What line or line items would be of interest?

• Are you the best person to answer this question? Would you consult with anyone else?

  • What department or position is that person in?

• Specifically, what would you include in this number? What would you exclude?

In round 1, we intend to show each participating bank only one of the sets of questions from the sections below.

Question Specific Probes for Set 1:

• IB.13

  • Does your bank create an internal score for loan applicants?

    • If yes:

      • Is it numeric?

      • What does you bank call this score?

      • What kind of information goes into this score? Note that this info might be a trade secret, note if respondent is hesitant to give details.

        • Hard (numeric/objective) data?

        • Soft (qualitative/subjective) data?

      • Is generating this score an automated process?

        • If yes: Please tell me about that process.

    • If no:

      • Does your bank generate any other different kind of score for applicants that you would not describe as a credit score?

      • What type of information does your bank use when deciding whether to approve a loan application or not?

• IB.14:

  • If yes: What kind of factors might loan officers use to adjust this score after it is created?

    • Is this a common practice at your bank?

• IB.15 – IB.16:

  • In your own words, what is this question asking you to do?

  • Do all of these pieces of information make sense, are any of them confusing?

  • Are there other pieces of information that your bank collects about loan applicants that are not listed here?

    • If yes: what are they and do you use them in generating the score?

  • Is there any information that does not belong on this list?

  • Note if R is hesitant to share what goes into score.

• IB.17 – IB.19:

  • In your own words, what is this question asking you to do?

  • Does your bank use different standards of evaluation for these types of loan applicants?

    • Are you able to pull records based on these categories?

  • Are there other pieces of information, including informal information, that your bank uses to evaluate any of these types of loan applicants that are not listed here?

  • Is there any information that does not belong on this list?

• IB.20 – IB.22:

  • In your own words, what is this question asking you to do?

  • Does your bank have different requirements for these different types of loan applicants?

  • For items that are not always (or never) required: why might you require this in some cases, but not others? Can you give me any examples?

  • Is there other information that your bank needs from applicants that are not listed here?

• IC.4 – IC.6:

  • Before we move on to the next questions, can you describe your bank’s typical loan approval process?

  • Now let’s look at the questions, how would you answer this question?

    • In your own words, what does ‘levels of approval’ mean?

    • What levels can approve for each question?

      • If they don’t know: Who would know?

  • What are your banks levels of approval, how many do you have total?

  • Are there situations in which the number of levels of approval would vary within one of the categories of borrower?

  • For a given loan applicant, is it usually clear who should or will have final approval authority?

• IC.7 – IC.9:

  • Are there situations in which the number of levels of approval would vary within one of the categories of borrower?

    • If yes: How would you answer this question?

    • What situations could cause this level to vary within a category of borrowers?

• IC.10 – IC.12:

  • How did you come to that number?

    • Are the levels written into your loan policies?

• IC.13:

  • In your own words, what is this question asking you to do?

• IC.14:

  • In your own words, what is this question asking you?

  • How would you define ‘geographic purview?’

    • Is there another term or phrase you would use to describe this?

    • Is there a different categorization other than geography that would be more appropriate to explain what this level of approval covers?

  • Does the geographic purview vary by level?

• IC.15 – IC.16:

  • In your own words, what is this question asking you?

  • How would you define ‘signature authority?’

    • Is there another term or phrase you would use to describe this?

  • Generally speaking, what are the typical circumstances where the lower levels make the approval?

  • Are there situations in which this level has signature authority, but would choose to defer to a higher level?

• IC.17:

  • Do all of these options make sense, are any of them confusing?

  • Is there anything missing from this list?

• IC.18:

  • How would you define ‘Not very influential?’ ‘Somewhat influential?’ ‘Very influential?’

• ID.1 – ID.9:

  • How would you define ‘FinTech’ as it relates to small business loans?

  • FinTech may be a trade secret: How comfortable are you with discussing FinTech?

  • Do you feel like the paragraph at the beginning of this section accurately describes FinTech?

  • What is this question asking you to do?

  • Can you give me specific examples of how your bank would use FinTech for a small business loan?

  • Do these levels accurately reflect your loan process? Are any levels missing?

  • [For each level they say yes:] can you give me an example of that technology?

• ID10:

  • In your own words, what is this question asking you?

  • Do all of these categories make sense, are any of them confusing?

  • Are there any categories missing?

  • Having gone through these FinTech questions, how much does your bank use FinTech for small business lending overall?

    • Is your bank interested in using more Fintech in the future?

Question Specific Probes for Set 2:

• Screeners:

  • Can you describe your bank’s core data systems?

    • Do you have different systems for different loan purposes?

      • For different loan products?

  • [For #2, If yes:] How are the systems separated out?

  • [For #7, If yes:] What are these different divisions?

    • Are there other features that would be considered?

• IIIA.1 – IIIA.2:

  • [If yes:] How familiar are you with your bank’s SBA lending program and loan approval process?

    • [If not very familiar:] Who would be? What position/department are they in?

  • Are all of the reasons clear or are any unclear?

  • Are there any reasons missing from the list?

• IIIA.3:

  • Are all of the programs clear or are any unclear?

  • Are there any programs missing from the list?

  • Is the list or your bank’s SBA programs documented somewhere?

• IIIA.4:

  • Are all of the options clear, or are any of them unclear?

  • Are any of these statements accurate to your bank?

    • Is there a different way you would describe your bank’s participation?

• IIIB.1:

  • What term or phrase would you use to describe getting loans off of your books?

  • When answering this question, were you only considering loans that were sold?

• IVA.1:

  • Are the includes helpful or unhelpful?

• IVA.2:

  • Note that this number should be subset of question IVA.1.

  • Are the includes and excludes helpful or unhelpful?

• IVA.3:

  • Note that this number should be subset of question IVA.2.

• IVA.4 – IVA.5:

  • Note that these numbers should sum to question IVA.3.

  • Are you able to retrieve this from your records?

• IVA.6:

  • Respondent should be excluding credit card products.

  • In your own words, what is this question asking?

  • Are any of the standardized loan products you thought about when answering this question created specifically by and for your bank?

    • Is there another term or phrase you would use other that ‘standardized loan products’ to describe these?

  • Are any of the standardized loan products you were thinking of automated, or in other words is the approval determined by an algorithm?

• IVA.7:

  • Are the examples helpful or unhelpful?

  • Typically, are loan originations categorized by the purpose or collateral?

    • Would a loan like this ever get categorized as a residential loan in your non-Call Report systems?

• IVA.8:

  • Are you able to retrieve this from your records?

• IVA.9:

  • Are you able to retrieve this from your records?

• IVA.10 – IVA.13:

  • Would you be able to provide this data?

    • If no: Would you be able to provide any of these cells?

      • Could you provide this information at a more aggregated level?

      • If yes: What level could you provide/how would those levels be categorized? Loan size? Borrower size?

  • Are you able to retrieve this from your records?

• IVA.14:

  • Would you be able to provide this data?

  • Are these SBA C&I loans housed in separate systems from your non-SBA C&I loans?

  • Are you able to retrieve this from your records?

  • Would you include loans that have already been sold or securitized?

    • Could you provide this information from loans that have already been sold or securitized if necessary?

• IVB.1 – IVB.4:

  • Would you be able to provide this data?

    • If no: Would you be able to provide any of these cells?

      • Could you provide this information at a more aggregated level?

      • If yes: What level could you provide/how would those levels be categorized? Loan size? Borrower size?

  • Are you able to retrieve this from your records?

  • Is it easier to provide this data for originations or for outstanding loans asked about here?

Question Specific Probes for Set 3:

• IIA:

  • Do you find the includes and excludes to be helpful or unhelpful?

  • How would you define ‘geographic market’ in your own words?

    • Is there another term or phrase you would use for this?

    • Would a borrower at the edge of your geographic market need to be more credit-worthy than average?

  • What does your bank consider to be its market or physical footprint?

  • How do you judge the primary business locations of potential small business borrowers?

    • At what point is a location typically considered too far?

• IIA.1:

  • Alternate between the 3 options.

  • Is your geographic market a number of miles or does your bank use a different physical boundary?

• IIA.2:

  • Are all of these reasons clear, or are any unclear?

  • Are there any reasons missing from this list?

• IIA.3:

  • If none of the above: can you tell me more about that?

    • Are there other types of boundaries your bank considers?

• IIA.4:

  • How would you answer this question?

• IIA.5 – IIA.9:

  • Are there any steps missing?

  • Are there any other means a business could complete an application step at your bank?

• IIA.10 – IIA.11:

  • Are all of these practices clear, or are any unclear?

  • Are there any practices missing from this list?

  • Are there any practices that don’t belong on this list?

  • IIA.11: How does this question differ from the previous question?

• IIB.1 – IIB.2:

  • Would you be able to answer this question?

• IIB.3:

  • In your own words, how would you define ‘competitive advantage?’

  • Are there any other categories you would consider that are not listed here?

• IIB.4 – IIB.9:

  • Are you comfortable providing this information?

    • Would you need permission from anyone to provide this information?

• IIB.10:

  • In your own words, can you tell me what this question is asking?

    • What comparison are you being asked to make here?

  • Is there any advantage you would list for other?

• IIB.11:

  • How does this question differ from 10?

• IIB.12:

  • How would you determine what is important?

    • Does your bank do any research?

Question Specific Probes for Coronavirus Pandemic questions:

• IIC.1:

  • Are all of these problems clear, or are any unclear?

  • Are there any problems missing from this list?

• IIC.2:

  • Are all of these actions clear, or are any unclear?

  • Are there any actions missing from this list?

• IIC3:

  • Are all of these government responses clear, or are any unclear?

  • Are there any government responses missing from this list?

Wrapping Up:

• Do you think you could complete this survey by yourself? Or would you need help from someone else? If the latter, who, what is their position, and what department are they from? Which sections of the survey would they answer?

• How long do you think it would take your bank to complete this survey? Include the time spent by anyone who worked on it.

• Overall, do you have any suggestions for us on how we can improve this survey?

Thank you for your time!