TEGL36-10ATT-B.pdf

ETA Advisory File Text

ATTACHMENT B GUIDANCE ON CAPTURING CUSTOMER SATISFACTION RESULTS Using the American Customer Satisfaction Index to Measure Customer Satisfaction A. Overview of Measurement Approach The Employment and Training Administration ETA will use the American Customer Satisfaction Index ACSI to meet the customer satisfaction measurement needs of the Workforce Investment Act WIA . The survey approach that will be utilized allows grantees flexibility and at the same time captures common customer satisfaction information that can be aggregated and compared at a grant recipient state regional and national levels. This will be done through the use of a small set of required questions that will form a customer satisfaction index. ETA will use the ACSI which is created by combining scores from three specific questions that address different dimensions of customers experience. The ACSI is a widely used customer satisfaction measurement approach. It is used extensively in the business community including at over 150 Fortune 500 companies and in many European countries. Twenty-nine agencies of the Federal government have used the ACSI. The ACSI will allow the workforce investment system to not only look at performance within the system but also be able to gain perspective on the workforce system s performance by benchmarking against organizations and industries outside of the system. The ACSI also has a history of being useful in tracking change in customer satisfaction over time making it an ideal way to gauge grantees progress toward continuously improving in performance. Since the ACSI trademark is proprietary property of the University of Michigan and the Claes Fornell International Group CFI ETA established a license agreement with the University of Michigan that will allow grant recipients the use of the ACSI for a statewide sample of participants and employers. It is worthwhile to note that WIA state grantees may use the ACSI questions and approach to measure satisfaction with WIA services at the local board level if individual license agreements are purchased from CFI and the ACSI questions and measurement approach are uniformly administered throughout the state. States using the ACSI methodology to measure local board performance are not required to conduct separate statewide customer satisfaction surveys to obtain State level WIA customer satisfaction results as long as the surveys are administered by telephone respondents are contacted within the window of opportunity outlined for participants and employers at least 500 surveys are completed statewide for each group -- participants and employers the surveys comply with the methodology spelled out in this guidance and the Local Board results can be weighted to account for sampling differences if appropriate and aggregated to obtain state level outcomes. B. WIA Participant Customer Satisfaction Measures The weighted average of participant ratings on each of the three questions regarding overall satisfaction is reported on a 0-100 point scale. The score is a weighted average not a percentage. 1. Who Will Be Surveyed WIA Title I-B participants with a last expected service date are eligible to be chosen for inclusion in the random sample. WIA Title I-B participants include individuals who are either adults dislocated workers older youth ages 19-21 and younger youth ages 14-18 . 2. How Many Number to be Surveyed States may randomly sample individuals to obtain the desired number of completed surveys. States are required to determine the appropriate sample sizes and sampling percentages using the required response rates and the required number of interviews. At least 500 completed participant surveys must be obtained each year for calculation of the measures. A completed survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than 5 points from the score obtained from surveying the whole population. 3. How methodology and When to Conduct Surveys To ensure ACSI results for individuals are collected in a consistent and uniform manner procedures used by states to obtain participant customer satisfaction information must satisfy the following criteria Participant customer satisfaction responses must be collected by way of telephone interviews. In-person interviews and mail questionnaires may be used only in situations where the individual does not have a telephone Participants should be contacted as soon as possible on or after the last expected service date and no later than 60 days after the closure date. Closure date is another term for the last expected service date States must complete a minimum of 500 surveys during the program year to accurately assess performance on the ACSI If sampling is used states must randomly select participants to obtain the desired number of completed surveys. States are required to determine the appropriate sample sizes and sampling percentages using the required response rate and the required number of completed interviews The sampling methodology used to select potential respondents for the surveys must ensure the consistent random selection of a sample of participants eligible for the surveys. With this in mind no individual in the participant group eligible for the survey may be arbitrarily excluded from a sample. Procedures used to draw a sample must conform to accepted statistical practices such as using a table of random numbers. Participant samples must be drawn on at least a monthly basis The minimum response rate is 70 percent. The response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number in the sampling frame. A survey is considered complete where valid answers are provided by respondents for each of the core questions outlined by ETA It is very important that all records for eligible participants be entered in a timely fashion into the databases. The information in these databases will be used to develop lists of participants who received services that are the subject of the survey. Every precaution must be taken to prevent a response bias States must develop a process for creating an up-to-date customer list capturing information on each customer s address and telephone number. Local programs should collect alternate contact information from a person known by the customer who would know the whereabouts of the customer in the event the customer cannot be reached at the address and telephone number recorded for the customer A letter in advance of the survey should be sent out informing the customer that he or she can expect to receive a telephone call about his or her satisfaction with the services. States should place a phone number on the letter suggesting that customers call if they need help or call if their phone numbers changed Attempts must be made to contact all eligible participants in the sample to avoid introducing bias. A minimum of five follow-up attempts is required involving various times of the day before closing the record Respondents must be told that responding to the survey is voluntary and that the information they provide will be kept confidential. However states must not make any assurances regarding confidentiality without ensuring that they have the legal authority to make such assurances. Such authority can be found in state privacy laws for example States must use the minimum core customer satisfaction questions specified by ETA in collecting and reporting satisfaction outcomes. These mandatory items should be located at the beginning of the questionnaire. States may include other questions about the service experience and The interview should be limited to 15 minutes or less. Shorter questionnaires produce better response rates when compared to longer questionnaires. 4. What are the Core Questions The following introductory statement and core ACSI questions are to be read to the respondent at the beginning of the interview 1. The introductory script sets the context for the interview by focusing the interview on the service experience date s of service and the firm s or organization s delivering the service s . In addition to contact information the survey administrators will need to know the service s provided to each customer the date s of service and the firm s providing the service s in order to complete the interviews. The service s should be worded in terms recognizable to the customer. Introductory Script My name is interviewer with firm or agency and I am conducting a survey for the program name . I would like to speak to Ms. Mr. name of participant . Are you the Ms. Mr. name of participant who received a name of service s from theon date or date range I F THE RESPONDENT CONFIRMS PARTICIPATION PROCEED WITH THE INTERVIEW . IF NO END THE INTERVIEW AND THANK THE RESPONDENT FOR HIS OR HER TIME . I would like to ask you some questions about your recent experience with the services provided to you. Our purpose is to learn from you how to improve programs and services offered to people. The survey was approved by the Office of Management and Budget under OMB Control No. 1205-0420 expiring 02 28 2009 to collect information on your experience with services we provide to individuals. Without this approval we would not be able to conduct this survey. The questionnaire is voluntary and should take no longer than 15 minutes to complete. Please know the answers you provide to my questions will be kept confidential. If you have any questions regarding this estimate or any other aspect of this data collection including suggestions for reducing this burden send them to the U.S. Department of Labor Office of Workforce Development Room C-4526 Washington D.C. 20210 Paperwork Reduction Project 1205-0420 . Do you have any questions for me before we start the interview I F NO CONTINUE . IF YES PROBE TO UNDERSTAND THE 1 Note The first question can be modified to suit the individual needs of the State and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance. QUESTIONS PROVIDE ANSWERS AND RESTATE THE IMPORTANCE OF THE RESPONDENT S PARTICIPATION IN THE INTERVIEW . The Questions First I want you to rate your overall experience with the services. I will read to you three statements and ask you to rate your experience. Utilizing a scale of 1 to 10 where a 1 means Very Dissatisfied and 10 means Very Satisfied what is your overall satisfaction with the services provided from Very Very DK 2 REF 3 Dissatisfied Satisfied 1 2 3 4 5 6 7 8 9 10 11 12 Considering all of the expectations you may have had about the services to what extent have the services met your expectations A 1 now means Falls Short of Your Expectations and 10 means Exceeds Your Expectations. Falls Short of Exceeds DK REF Expectations Expectations 1 2 3 4 5 6 7 8 9 10 11 12 Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services A 1 now means Not very close to the Ideal and 10 means Very Close to the Ideal. Not Close Very Close DK REF To Ideal To Ideal 1 2 3 4 5 6 7 8 9 10 11 12 C. Measuring One-Stop Employer Customer Satisfaction The weighted average of employer ratings on each of the three questions regarding overall satisfaction is reported on a 0-100 point scale. The score is a weighted average not a percentage. 1. Who Will Be Surveyed One-Stop Employers Those eligible for surveying include employers who received a substantial service where the service has been completed or if it is an ongoing service when a full 2 DK Don t Know 3 REF Refused to Answer segment of service has been provided. All employers who received a substantial service involving personal contact with WIA staff are eligible to be chosen for inclusion in the random sample. Examples of substantial services include staff facilitated job orders customized job training customized labor market information requests and on-the-job training activities. When an employer has received multiple services priority should be given to the service that required the greatest expenditure of financial assistance or staff time 4 and the survey conducted regarding their satisfaction with that service. An employer should only be surveyed one time during the course of the program year. 2. How Many number obtained Except in small states a sample will be taken from these employers. At least 500 completed interviews are required for the employer survey for calculation of the indicator. States are required to determine the appropriate sample sizes and sampling percentages using the required response rate and the required number of completed interviews. The sampling percentage should be constant over time to avoid under- representing or over-representing time intervals. A completed employer survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than 5 points from the score obtained from surveying the whole population. 3. How methodology and When to Conduct Surveys To ensure ACSI results for individuals are collected in a consistent and uniform manner procedures used by states to obtain employer customer satisfaction information must satisfy the following criteria Employer customer satisfaction responses must be collected by way of telephone interviews. In-person interviews and mail questionnaires may be used only in situations where the individual does not have a telephone Employers should be contacted as soon as possible after the completion of the service and no later than 60 days after the completion of the service. For employers who listed a job order where no referrals were made contact should occur 30 to 60 days after a job order was listed States must complete a minimum of 500 employer surveys during the program year to accurately assess performance on the ACSI If sampling is used states must randomly sample employers to obtain the desired number of completed surveys. States are required to determine the 4 Where an employer has received multiple services in a given time period and there is separate contact information for each service the contact information for the priority service should be used for surveying. appropriate sample sizes and sampling percentages using the required response rate and the required number of completed interviews The sampling methodology used to select potential respondents for the surveys must ensure the consistent random selection of a sample of employers eligible for the surveys. With this in mind no individual in the employer group eligible for the survey may be arbitrarily excluded from a sample. Procedures used to draw a sample must conform to accepted statistical practices such as using a table of random numbers Employer samples must be drawn on at least a monthly basis. To obtain sufficient numbers smaller states will need to survey on an ongoing basis The minimum response rate for the employer survey is 70 percent. The response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number in the sampling frame. A survey is considered complete where valid answers are provided by respondents for each of the core questions outlined by ETA It is very important that all records for eligible employers be entered in a timely fashion into the databases. The information in these databases will be used to develop lists of employers who received services that are the subject of the surveys. Every precaution must be taken to prevent a response bias. States must develop a process for creating an up-to-date customer list capturing information on each customer s address and telephone number. Local programs should collect alternate contact information from a person known by the customer who would know the whereabouts of the customer in the event the customer cannot be reached at the address and telephone number recorded for the customer. A letter in advance of the survey should be sent out informing the customer that he or she can expect to receive a telephone call about his or her satisfaction with the services. States should place a phone number on the letter suggesting that customers call if they need help or call if their phone numbers changed. Attempts must be made to contact all eligible employers in the samples to avoid introducing bias. A minimum of five follow-up attempts is required involving various times of the day before closing the record. Respondents must be told that responding to the survey is voluntary and that the information they provide will be kept confidential. However states must not make any assurances regarding confidentiality without ensuring that they have the legal authority to make such assurances. Such authority can be found in state privacy laws for example. States must use the minimum core customer satisfaction questions specified by ETA in collecting and reporting satisfaction outcomes. These mandatory items should be located at the beginning of the questionnaire. States may include other questions about the service experience. The interview should be limited to 15 minutes or less. Shorter questionnaires produce better response rates when compared to longer questionnaires. 4. What are the Core Questions The following introductory statement and core ACSI questions are to be read to the respondent at the beginning of the interview 5. The introductory script sets the context for the interview by focusing the interview on the service experience date s of service and the firm s or organization s delivering the service s . In addition to contact information the survey administrators will need to know the service s provided to each customer the date s of service and the firm s providing the service s in order to complete the interviews. The service s should be worded in terms recognizable to the customer. Introductory Script My name is interviewer with firm or agency and I am conducting a survey for the program name . I would like to speak to Ms. Mr. name of employer contact . Are you the Ms. Mr. name of contact who received a name of service s from the on date or date range I F THE RESPONDENT CONFIRMS PARTICIPATION PROCEED WITH THE INTERVIEW . IF NO END THE INTERVIEW AND THANK THE RESPONDENT . I would like to ask you some questions about your recent experience with the services provided to your business. Our purpose is to learn from you how to improve programs and services offered to area businesses. The survey was approved by the Office of Management and Budget under OMB Control No. 1205-0420 expiring 02 28 2009 to collect information on your experience with services we provide to employers. Without this approval we would not be able to conduct this survey. The questionnaire is voluntary and should take no longer than 15 minutes to complete. Please know the answers you provide to my questions will be kept confidential. If you have any questions regarding this estimate or any other aspects of this collection including suggestions for reducing this burden send them to the U.S. Department of Labor Office of Room S-4231 Washington D.C. 20210 Paperwork Reduction Project 1205-0420 . Do you have any questions for me before we start the interview I F NO CONTINUE . IF YES PROBE TO UNDERSTAND THE QUESTIONS PROVIDE ANSWERS AND RESTATE THE IMPORTANCE OF THE RESPONDENT S PARTICIPATION IN THE INTERVIEW . 5 Note The first question can be modified to suit the individual needs of the State and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance. The Questions First I want you to rate your overall experience with the services. I will read to you three statements and ask you to rate your experience. Utilizing a scale of 1 to 10 where 1 means Very Dissatisfied and 10 means Very Satisfied what is your overall satisfaction with the services provided from Very Very Dissatisfied Satisfied DK 6 REF 7 1 2 3 4 5 6 7 8 9 10 11 12 Considering all of the expectations you may have had about the services to what extent have the services met your expectations A 1 now means Falls Short of Your Expectations and 10 means Exceeds Your Expectations. Falls Short of Exceeds DK REF Expectations Expectations 1 2 3 4 5 6 7 8 9 10 11 12 Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services A 1 now means Not very close to the Ideal and 10 means Very Close to the Ideal. Not Close Very Close DK REF To Ideal To Ideal 1 2 3 4 5 6 7 8 9 10 11 12 D. Definition of Terms Sample - A group of cases selected from a population by a random process where everyone has an equal probability of being selected. Last Expected Service Occurs when the participant completes the activities outlined in his or her service strategy or service plan and there are no additional services expected other than supportive or follow-up services. Last expected service may also occur in situations where the participant voluntarily or involuntarily discontinues his or her participation in services outlined in the service plan. 6 DK Don t Know 7 REF Refused to Answer Last Expected Service Date This date is used to determine when a customer becomes a part of the sampling frame for the participant customer satisfaction survey. In many instances this date will be the same as the exit date. In situations where a case was ended reopened within 90 days of the original closure date and then ended again the date used to determine inclusion in the sampling frame is the initial last expected service date. Response rate - The percentage of people in the sample who are contacted and respond to the core questions on the survey. E. Calculation of the ACSI The ACSI scores represent the weighted sum of the three ACSI questions values which are transformed into 0 to 100 scale values. The weights are applied to each of the three questions to account for differences in the characteristics of the State s customer groups. For example assume the mean values of three ACSI questions for a state are 1. Overall Satisfaction 8.3 2. Met Expectations 7.9 3. Compared to Ideal 7.0 Then these mean values from raw data must first be transformed to the value on a 0 to 100 scale. This is done by subtracting 1 from these mean values dividing the results by 9 which is the value of range of a 1 to 10 raw data scale and multiplying the whole by 100 1. Overall Satisfaction 8.3 - 1 9 x 100 81.1 2. Met Expectations 7.9 - 1 9 x 100 76.7 3. Compared to Ideal 7.0 - 1 9 x 100 66.7 The ACSI score is calculated as the weighted averages of these values. Assuming the weights for the example state are 0.3804 0.3247 and 0.2949 for questions 1 2 and 3 respectively the ACSI score for the state would be calculated as follows 0.3804 x 81.1 0.3247 x 76.7 0.2949 x 66.7 75.4 Weights were calculated by a statistical algorithm to minimize measurement error or random survey noise that exists in all survey data. State-specific weights were calculated using the relative distribution of ACSI respondent data for non-regulatory Federal agencies previously collected and analyzed by CFI and the University of Michigan. Specific weighting factors have been developed for each State. New weighting factors will be published annually.