TEGL7-99.pdf

ETA Advisory File Text

-1-U. S. Department of Labor Employment and Training Administration Washington D.C. 20210CLASSIFICATION WIA Performance CORRESPONDENCE SYMBOL OWS DATE March 3 2000 TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 7-99 TO ALL STATE WORKFORCE LIAISONS ALL STATE WORKER ADJUSTMENT LIAISONS ALL STATE EMPLOYMENT SECURITY LIAISONS ALL ONE-STOP CAREER CENTER SYSTEM LEADS FROM LENITA JACOBS-SIMMONS Deputy Assistant Secretary SUBJECT Core and Customer Satisfaction Performance Measures for the Workforce Investment System 1. Purpose. To provide technical guidance for States to implement the core and customer satisfaction performance measures and calculate performance levels required under the Workforce Investment Act WIA . 2. References. Workforce Investment Act of 1998 Pub. L. 105-220 Section 136 Workforce Investment Act Interim Final Rule 20 CFR Part 666 published at 64 Federal Register 18662 April 15 1999 . 3. Background. Section 136 of the WIA specifies core indicators of performance for workforce investment activities in adult dislocated worker and youth programs. Fifteen core measures apply to the adult dislocated worker and youth programs and two measures of customer satisfaction apply across these three funding streams for a total of 17 required measures. The measures specified in the Act are as follows -2-RESCISSIONSEXPIRATION DATE Continuing Adult Program1.Entry into unsubsidized employment 2.Retention in unsubsidized employment six months after entry into the employment 3.Earnings received in unsubsidized employment six months after entry into the employment and 4.Attainment of a recognized credential relating to achievement of educational skills which may include attainment of a secondary school diploma or its recognized equivalent or occupational skills by participants who enter unsubsidized employment. Dislocated Worker Program5.Entry into unsubsidized employment 6.Retention in unsubsidized employment six months after entry into the employment 7.Earnings received in unsubsidized employment six months after entry into the employment relative to earnings of job of dislocation and 8.Attainment of a recognized credential relating to achievement of educational skills which may include attainment of a secondary school diploma or its recognized equivalent or occupational skills by participants who enter unsubsidized employment. Youth ProgramOlder Youth aged 19-21 9.Entry into unsubsidized employment 10.Retention in unsubsidized employment six months after entry into the employment 11.Earnings received in unsubsidized employment six months after entry into the employment and 12.Attainment of a recognized credential relating to achievement of educational skills which may include attainment of a secondary school diploma or its recognized equivalent or occupational skills by participants who enter unsubsidized employment or who enter postsecondary education advanced training or unsubsidized employment. Younger Youth aged 14-18 13.Attainment of basic skills and as appropriate work readiness or occupational skills 14.Attainment of secondary school diplomas and their recognized equivalents and 15.Placement and retention in postsecondary education advanced training military service employment or qualified apprenticeships. -3- Across Funding Streams16.Customer satisfaction for participants and 17.Customer satisfaction for employers. The core performance measures are the key measures of success in achieving the legislative goals of WIA. The measures are used to set agreed upon performance goals on a State and local level ensure comparability of state performance results to maintain objectivity in measuring results for incentive and sanction determinations and provide information for system wide reporting and evaluation for program improvement. States and localities may also need other measures for ongoing oversight program management and continuous improvement purposes. It should be noted that some measures will be used by other related programs. For example the Trade Adjustment Assistance and NAFTA - Transitional Assistance programs mandatory partners of WIA will use dislocated worker measures discussed herein. Section 136 of the Act also calls for the use of quarterly Unemployment Insurance UI wage records to measure performance. There is a significant time delay in availability of the UI wage record for use as a data source. To address the time-lags associated with the UI wage records States will be required to go back in time prior to WIA implementation to both simulate WIA performance levels for developing baseline data for negotiations and calculating performance measures for WIA implementation. For most of the core measures the performance levels reported will also lag behind. This means that outcomes for most participants will not be reported during the same year that individuals participate in services. This guidance includes the following sections a discussion of the performance and accountability framework which identifies key concepts that apply across all of the measures. the core measures that apply to the adult dislocated worker and youth funding streams. This includes a discussion of operational definitions of the measures and data sources and methods to collect data on outcomes. the customer satisfaction measures that apply across funding streams to participants and employers. It presents a detailed description of the methodology for implementing the customer satisfaction measures. the attachments which include A WIA Core and Customer Satisfaction Measures-at-a- Glance B Definitions of Key Terms and C Guidance for Calculating WIA Core Measures. -4- 4. Performance and Accountability Framework. A. Applying the WIA Core Measures Across Partner Programs. A major emphasis of the legislation is improving coordination between the workforce investment system and adult education literacy and vocational rehabilitation programs that were re-authorized in the bill and vocational education programs addressed in separate legislation. Incentive grants will be awarded based on exceeding State adjusted levels of performance for WIA Titles I and II Adult Education and Literacy and the Carl D. Perkins Vocational and Applied Technology Education Act. The Department is working with the U.S. Department of Education to develop a coordinated performance accountability framework across labor adult education vocational education and vocational rehabilitation programs. Initially these efforts are focusing on the common components of the core measures. These efforts will also include developing common terms and additional measures which States and localities may choose to draw upon. To encourage integration of services and recognize shared contributions toward outcomes the Department will use the following strategy for tracking and reporting across WIA Title I funded WIA funded programs and other workforce investment partners. Once an individual has registered for WIA funded services programs can claim credit for outcomes on core measures attained by participants who receive non-WIA funded services such as those offered by One-Stop and School-to-Work partner programs . The non-WIA-funded partner programs that are included in the performance measurement system must be party to local Memorandums of Understanding MOU . Each partner program must have the capacity to track registered WIA participants until the individual exits the definition of exit is in Section 4.D all WIA and non-WIA-funded partner services. Some methods for tracking participants across programs include specifying the non-WIA funded services in the individuals service plans coordinating services across WIA and non-WIA funded services and providing follow-up services to individuals. B. Applying Measures to Funding Streams and Population Groups. The WIA core measures other than the customer satisfaction measures are applied separately to the three funding streams adult dislocated workers and youth. The youth funding stream is further divided into two categories older youth aged 19-21 and younger youth aged 14-18 . This means that 15 of the measures are divided among the four different populations groups. The two customer satisfaction measures are applied across all population groups. Since the outcomes vary across the four population groups the Department maintains that it is important to measure performance separately for each group. The following rules apply to determine how to count participants in the funding streams CIf a participant is served by a specific funding stream he she will be counted in that funding stream s set of measures e.g. a participant served by adult funds will be in the adult measures . -5- CIf a participant is served by more than one funding stream he she will be counted in more than one set of measures e.g. a participant served by youth funds and adult funds will be counted in the youth measures and adult measures . CSince there are two sets of measures within the youth funding stream the following rule applies for those served by the youth funding stream a youth must be included only in the set of youth measures that applies based on the person s age at registration i.e. youth between the ages of 14 and 18 at registration will be included in the younger youth measures and youth between the ages of 19-21 at registration will be included in the older youth measures regardless of how old the participant is at exit this is a different rule for counting youth in the measures than the rule for counting youth used in PY 1999 by the early implementing States . CAll exit-based measures which includes all the core measures except the younger youth skill attainment rate and the employer customer satisfaction measure will be effective when the participant has exited all WIA Title I and partner services definition of exit is provided in Section D . In the case of a youth participant who becomes an adult participant the individual will be counted in both the youth measures and adult measures. Neither set of exit-based measures will be applied until that participant has exited the adult program services as well as all other WIA Title I or partner services . This is because exit-based measures assess a participant s outcome following their full range of services even when those services are from different program areas. C. Point of Registration for Counting Customers in Measures. For accountability purposes WIA established core measures of performance for customers who receive workforce investment activities beyond self-service and informational services. The WIA Interim Final Rule Section 666.140 stated that the point of registration determines who is counted in the measures and who is excluded from the measures. All youth who receive WIA Title I services will be registered for services and counted in the measures. For performance measurement of the adult and dislocated worker programs WIA distinguishes self- service and informational activities as separate from the other activities within the WIA service categories core intensive and training . However there are informational activities within the core services category described in the Act. There are two main factors to consider when determining which core services require adults and dislocated workers to be registered and counted in the measures 1.Level of staff involvement with the customer. When there is significant staff involvement in -6- terms of resources or time individuals receiving the staff-intensive core services are required to be registered for the adult and dislocated worker programs all youth customers are required to register . 2.Purpose of the service. The Act specifically excludes those individuals who participate in self- service activities only such as browsing the Internet . For staff-assisted activities the purpose of the service should be examined to determine if registration is required for the service. Services that are designed to inform and educate individuals about the labor market and their employment strengths weaknesses and the range of services appropriate to their situations should be considered informational in nature. Staff-assisted services that are designed to impart job seeking and or occupational skills should require registration. For example individuals receiving the following categories of core services should be registered. CStaff assisted job search and placement assistance including career counseling CStaff assisted job referrals such as testing and background checks CStaff assisted job development working with employer and job-seeker CStaff assisted workshops and job clubs Using these criteria Table 1 presents the WIA core intensive and training services that require registration on a Federal level and the core informational self-service activities that do not require registration. This table includes the required WIA services specified in the Act for all of the categories and identifies some finer distinctions for service categories such as job search and placement assistance. Required services specified in the Act are represented in Table 1 in italics. It is important to keep in mind that an individual must receive a WIA funded staff-assisted core intensive or training service to trigger registration and include the person in the core measures. For example in a One-Stop environment if an individual received only Wagner Peyser funded core services or TANF funded case management and did not receive any WIA funded staff-assisted core intensive or training service that individual would not be registered for WIA and counted in the core measures. However as long an individual has received WIA funded staff-assisted core intensive or training and goes on to receive non-WIA funded partner services that person would be registered for WIA and counted in the core measures. 1 The individual would already be registered to be receiving follow-up services. -7-Table 1 Proposed Registration for WIA Services Core Services - Self- Service Informational no registration required WIA Core Services registration required WIA Intensive Services registration required WIA Training Services registration required Determination of eligibility to receive assistance under Title IBStaff assisted job search placement assistance including career counselingComprehensive specialized assessment such as diagnostic testing interviewingOccupational skills training Outreach intake which may include WPRS referrals orientation to the One-Stop centerFollow-up services including counseling regarding the workplace 1Full development of individual employment planOn the job training Initial assessment of skill levels aptitudes abilities need for supportive servicesStaff assisted job referrals such as testing background checks Group counselingWorkplace training cooperative education programs Employment statistics information including job vacancy listings job skill requirements for job listings info. on demand occupations Staff assisted job development working with employer jobseeker Individual counseling career planningPrivate sector training programs Performance info. on eligible training providersStaff assisted workshops and job clubsCase managementSkill upgrading retraining Performance info. on the local One-Stop delivery systemShort-term pre- vocational servicesEntrepreneurial training Core Services - Self- Service Informational no registration required WIA Core Services registration required WIA Intensive Services registration required WIA Training Services registration required -8-Information on supportive services and referral to supportive servicesFollow-up services including counseling for registrants those previously receiving intensive training services after entering employmentJob readiness training Information regarding filing for Unemployment compensationAdult education and literacy activities in combination with training Assistance in establishing eligibility for welfare-to-work activities and for other training and education programsCustomized training Resource room usage Internet browsing job information and training searches Internet accounts Career Kit Personnel Kit Initial development of employment plan Talent referrals informational e.g. talent scouts labor exchange referrals of resumes without further screening Workshops and job clubs -9- D. Point of Exit for Counting Outcomes. For all of the core measures except the younger youth skill attainment rate and employer customer satisfaction measure comparability across States is only possible if a single point in time is used to begin measurement. The term exit is being used to determine when to count an individual in a specified reporting period. Each individual becomes part of an exit cohort a group who are determined to be exiters within a particular quarter and are looked at together for measurement purposes. There are two ways to determine exit during a quarter 1. a participant who has a date of case closure completion or known exit from WIA-funded or non-WIA funded partner service within the quarter hard exit or 2. a participant who does not receive any WIA-funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services soft exit . Participants who have a planned gap in service of greater than 90 days should not be considered as exited if the gap in service is due to a delay before the beginning of training or a health medical condition that prevents an individual from participating in services. Service providers should document any gap in service that occurs with a reason for such a gap in service. Participants who exit from services because they are incarcerated deceased or have a health medical condition that prevents the individual from participating in services should be excluded from the measures. Once a participant has not received any WIA funded or partner services for 90 days except follow-up services and there is no planned gap in service or the planned gap in service is for reasons other than those specified above that participant has exited WIA for the purposes of measurement in 15 of the 17 core measures the younger youth skill attainment rate and employer customer satisfaction measures are not based on exit . The exit date is the last date of WIA funded or partner service received except follow-up services . For a soft exit the exit date cannot be determined until 90 days has elapsed from the last date of service. At that point the exit date recorded is the last date of service. The exit quarter referred to throughout the definitions of the measures is the quarter in which the last date of service except follow-up services takes place. If a participant exits WIA and receives additional WIA services after exiting that participant is treated as a new participant for purposes of the core measures and will be included in the appropriate measures. The definition of exit applies to all of the core measures except the younger youth skill attainment and employer customer satisfaction measures. E. Time Period to be Used for Measurement. As far as time periods for measurement are concerned there are two groups of measures. The first group of measures is related to employment as the primary outcome or as a partial outcome and require the use of the UI wage records as a data source. These measures are the adult dislocated worker and older youth entered employment retention earnings and credential measures and the younger youth retention rate. The second group of measures are not employment based and are not tied to the UI wage records. These measures are -10- the younger youth skill attainment rate and diploma or equivalency rate and the two customer satisfaction measures. A major consideration for the first group of employment-related measures is the time lag associated with using the UI wage records as a data source. For the most part there is a time delay of at least two quarters after the quarter in which a person exits services and the time when all the UI wage record data for the quarter after exit are available for record matching. For measures that occur at six months after exit such as the employment retention and earnings measures the delay will be at least one year after the exit quarter. A detailed chart of the timing of activities and the measurement time frame is included in Attachment C Exhibit II.1. To address the time-lags in the use of the UI wage records States will need to go back in time prior to WIA implementation to calculate performance measures for actual PY 2000 WIA implementation. This will only need to be done for the group of employment-related measures that use the UI wage records. Since there is no need to go back in time for the second group of measures these non- employment related measures will use the actual program year time period beginning on July 1 2000. The two time periods for measurement in PY 2000 are as follows 1. The first time period is for all measures that utilize the UI wage record and will involve going back in time three quarters to account for the time-lag. These measures include the adult dislocated worker and older youth entered employment retention earnings and credential measures and the younger youth retention rate. For PY 2000 all States will use the last three quarters of PY 99 and the first quarter of PY 2000 for all the measures listed above. 2. The second time period is for the four measures that are not based on the UI wage records the younger youth skill attainment rate the younger youth diploma or equivalency rate and the two customer satisfaction measures. For PY 2000 these four measures will be real time measures in which States will use the four quarters in PY 2000 for purposes of calculating performance. This approach of having two time periods with one going back three quarters prior and the first quarter of the current program year and the second for the group of real-time measures being for the actual program year will continue for future years of WIA implementation. F. Participants to be Excluded from the Measures. Participants who exit from the program under the following conditions should be excluded from the measures CFor JTPA participant data exclude those who received objective assessment only and participants in Title II 8 funded programs. -11- CParticipants who exit from services because they are incarcerated institutionalized deceased or have a health medical condition that prevent them from participating in services should be excluded from the measures. CFor JTPA participant data exclude dislocated workers served exclusively with National Reserve Account NRA funds. For WIA participant data exclude dislocated workers served exclusively with National Emergency Grants. Additional exclusions may apply to certain measures. These exclusions are identified in the specific definitions of applicable measures in Section 5.H. G. Operational Definitions of the Core Performance Measures. All of the definitions of the measures are written as an equation clearly identifying who is in the numerator and who is in the denominator. In cases where there are conditions that apply to both the numerator and denominator the condition is represented in italics at the beginning of the measure. For example The adult entered employment rate is defined as Of those who are not employed at registration the number of adults who have entered employment by the end of the first quarter after exit divided by the number of adults who exit during the quarter. The condition of those not employed at registration applies to both the numerator and denominator as follows the number of adults who are not employed at registration and enter employment by the end of the first quarter after exit divided by the number of adults who are not employed at registration and exit during the quarter. In addition the phrase who exit during the quarter is used frequently in the denominators of the measures. This phrase refers to the exit quarter.Operational parameters and definitions of key terms are also provided to further clarify details needed to implement the measures. Since many of the measures are identical or similar across funding categories every attempt has been made to define the measures as consistently as possible. This means that identical or similar measures use the same time periods and consistent operational parameters to the extent possible. A two page summary of the measures in a calculation format is provided in Attachment A. Definitions of all the key terms identified in this document are included in Attachment B. Detailed guidelines for calculating the measures are provided in Attachment C. 5. Core Performance Measures that Apply to the Adult Dislocated Worker and Youth -12- Funding Streams H. Adult Measures Measure 1 Adult Entered Employment RateOf those who are not employed at registration Number of adults who have entered employment by the end of the first quarter after exit divided by the number of adults who exit during the quarter. Operational Parameters CIndividuals who are employed at registration are excluded from this measure i.e. programs will not be held accountable for these individuals under this measure . CEmployment at registration is based on information collected from the registrant not on UI wage records. Rationale While the Department recognizes concerns that have been raised about excluding incumbent and underemployed workers from this measure the intent of the legislation clearly focuses on entering employment. It becomes very difficult to include individuals who are employed at registration without further complicating the measure and making it subject to misinterpretation. Individuals who are employed at registration can demonstrate positive outcomes in the retention earnings gain replacement and combination employment and credential measure. Definitions Employed at Registration - An individual employed at registration is one who during the 7 consecutive days prior to registration did any work at all as a paid employee in his or her own business profession or farm worked 15 hours or more as an unpaid worker in an enterprise operated by a member of the family or is one who was not working but has a job or business from which he or she was temporarily absent because of illness bad weather vacation labor- management dispute or personal reasons whether or not paid by the employer for time-off and whether or not seeking another job. Not Employed at Registration - An individual who does not meet the definition of employed at registration. Employed in Quarter After Exit Quarter - The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the -13- primary data source for tracking employment in the quarter after exit. When supplemental data sources are used individuals should be counted as employed if in the calendar quarter after exit they did any work at all as paid employees i.e. received at least some earnings worked in their own business profession or worked on their own farm. Measure 2 Adult Employment Retention Rate at Six Months Of those who are employed in the first quarter after exit Number of adults who are employed in the third quarter after exit divided by the number of adults who exit during the quarter Operational Parameters CThis measure includes only those who are employed in the first quarter following exit regardless of their employment status at registration . CIndividuals who are not found to be employed in the first quarter after exit are excludedfrom this measure i.e. programs will not be held accountable for these individuals under this measure . CEmployment in the first and third quarters following exit does not have to be with the same employer. Rationale While the proposed measure does not necessarily indicate continuous employment for six months it does meet the intent of the Act by showing that the person is employed six months after entering employment. This measure requires verifying employment in the quarter after exit rather than relying on employment status at registration to be in aligned with the U.S. Department of Education Vocational Rehabilitation services measure of employment retention. Measure 3 Adult Average Earnings Change in Six Months Of those who are employed in the first quarter after exit Total post-program earnings earnings in quarter 2 quarter 3 after exit minus pre-program earnings earnings in quarter 2 quarter 3 prior to registration divided by the number of adults who exit during the quarter. Operational Parameters CThis measure includes the same population as the adult employment retention measure those who are employed in the first quarter following exit regardless of their employment status at registration . CTo ensure comparability of this measure on a national level the UI wage records will be the -14- only data source for these measure. Therefore individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from the UI wage records are excluded from the measure i.e. programs will not be held accountable for these individuals under this measure . CIndividuals who are not found to be employed in the first quarter after exit are excludedfrom this measure. CStates should exclude from this measure any individuals whose entry registration date is so far back in time that accessing quarters 2 and 3 of pre-registration wage data is unfeasible or unreasonable. However participants excluded from this measure for this reason should still be included in any other applicable measures. For example the person should still be counted in the retention measure. CIf supplementary sources are the data source for a participant s employment in the 2nd and or 3rd quarter prior to registration that participant is excluded from this measure. Rationale This measure is designed as a pre- and post-program look at earnings change. Since the legislation specifies earnings at six months after employment a six month period was selected for the pre- program comparison. The 2nd and 3rd quarters prior to registration were selected as the pre- program reference period because trends have shown that many program participants experience intermittent or stop-gap employment immediately prior to participating in employment and training programs. Measure 4 Adult Employment and Credential Rate Of adults who received training services Number of adults who were employed in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of adults who exited services during the quarter. Operational Parameters The numerator of this measure includes those who were employed in the first quarter after exit regardless of whether they were employed at registration. Credentials can be obtained while a person is still participating in services and up to three quarters following exit. Definitions Training Services - include WIA-funded and non-WIA funded partner training services. These services include occupational skills training including training for nontraditional employment on- 2 It is planned that outcomes for participants in the Trade Adjustment Assistance and NAFTA Transitional Adjustment Assistance programs will be measured using the same measures described in Measures 5 6 and 7 of this section. -15- the-job training programs that combine workplace training with related instruction which may include cooperative education programs training programs operated by the private sector skill upgrading and retraining entrepreneurial training job readiness training adult education and literacy activities in combination with other training and customized training conducted with a commitment by an employer or group of employers to employ an individual upon successful completion of the training. Credential - nationally recognized degree or certificate or State locally recognized credential. Credentials include but are not limited to a high school diploma GED or other recognized equivalents post-secondary degrees certificates recognized skill standards and licensure or industry-recognized certificates. States should include all State Education Agency recognized credentials. In addition States should work with local Workforce Investment Boards to encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment retain employment or advance into better employment. Rationale This measure is limited to individuals who are in training because that is the only set of services that lead to attainment of a credential. To promote program integration and partner collaboration this measure does recognize joint participation in WIA services and non-WIA funded training programs. While there will be variation in the ability of local providers or state agencies to track participants who are enrolled in non-WIA funded training services it is considered a significant and necessary step toward building an integrated workforce investment accountability system. However it should be noted that this also means that WIA funded programs will be able to get credit for non-WIA funded training in addition to WIA funded training. This measure includes all individuals who received training regardless of whether they had jobs at registration. It recognizes not only individuals who enter employment but also incumbent workers students and welfare-to-work participants who have jobs or are placed in jobs but continue receiving training services to attain more skills and better jobs. A time frame of three quarters after exit to obtain a credential is proposed to allow time for individuals to take tests which may occur after training is completed and may only be offered once a year. I. Dislocated Worker Measures2 -16- Measure 5 Dislocated Worker Entered Employment Rate Number of dislocated workers who have entered employment by the end of the first quarter after exit divided by the number of dislocated workers who exit during the quarter Operational Parameters All dislocated workers are included because dislocated workers are either not employed or scheduled to lose their primary job for which eligibility for the dislocated worker program was based. Rationale This measure uses the same measurement quarters as the adult entered employment rate. The only distinction is that all dislocated workers are counted in this measure regardless of their employment status at registration. Measure 6 Dislocated Worker Employment Retention Rate at Six Months Of those who are employed in the first quarter after exit Number of dislocated workers who are employed in the third quarter after exit divided by the number of dislocated workers who exit during the quarter. Operational Parameters Employment in the first and third quarters following exit does not have to be with the same employer. Individuals who are not found to be employed in the first quarter after exit are excludedfrom this measure i.e. programs will not be held accountable for these individuals under this measure . Rationale This measure has the same rationale as the adult employment retention rate. Measure 7 Dislocated Worker Earnings Replacement Rate in Six Months Of those who are employed in the first quarter after exit Total post-program earnings earnings in quarter 2 quarter 3 after exit divided by the pre- dislocation earnings earnings in quarters 2 quarter 3 prior to dislocation Operational Parameters To ensure comparability of this measure on a national level the UI wage records will be the -17- only data source for this measure. Individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from the UI wage records are excluded from the measure i.e. programs will not be held accountable for these individuals under this measure . This measure includes the same population as the retention measure those who are employed in the first quarter following exit. Individuals who are not found to be employed in the first quarter after exit are excludedfrom this measure. If there is no date of dislocation or if the date of dislocation is after registration use the 2nd and 3rd quarters prior to registration. If a State is not systematically collecting the date of dislocation and uses the 2nd and 3rd quarters prior to registration date it should be noted that the registration date may not closely simulate the results from the 2nd and 3rd quarters prior to the dislocation quarter. States should exclude from the earnings replacement calculations any individuals whose entry registration date is so far back in time that accessing quarters 2 and 3 pre- dislocation pre-registration wage data is unfeasible or unreasonable. However participants excluded from this measure for this reason should still be included in any other applicable measures. For example these participants should still be counted in the retention measure. The calculation for this indicator will be done on an aggregate basis. The Department will continue to research the feasibility of calculating this measure on an individual basis using wage record data. Definitions Date of dislocation - The last day of employment at the dislocation job. If there is no date of dislocation date of registration will be used instead. Rationale As opposed to the adult earnings measure that calculates post-program earnings increases as compared to pre-program earnings the earnings replacement rate which computes the percentage of pre-program earnings being earned post-program is being used for dislocated workers. This is because it may be difficult to find dislocated workers with formerly high paying jobs a new job with equivalent or better wages therefore measuring earnings increases would result in negative numbers and would be not be representative of a dislocated worker s true success. Because a major goal of the dislocated worker program is to ameliorate earnings loss as well as to replace or increase earnings a better measure of the dislocated worker program s effectiveness is percentage of earnings of the new job in relation to the job of dislocation. It was further determined that the quarter prior to dislocation often includes severance pay or may reflect reduced hours. This is why quarters 2 and 3 prior to dislocation were chosen. -18- Measure 8 Dislocated Worker Employment and Credential Rate Of dislocated workers who received training services Number of dislocated workers who were employed in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of dislocated workers who exited services during the quarter Operational Parameters CThe numerator of this measure includes those who were employed in the first quarter after exit regardless of whether they were employed at registration. CCredentials can be obtained while a person is still participating in services. CSee adult employment and credential measure definitions for training services and credentials. Rationale The rationale for this measure is the same as for the adult employment and credential measure. J. Older Youth Age 19-21 Measures Measure 9 Older Youth Entered Employment Rate Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the first quarter after exit Number of older youth who have entered employment by the end of the first quarter after exit divided by the number of older youth who exit during the quarter Operational Parameters Individuals who are employed at registration are excluded from this measure i.e. programs will not be held accountable for these individuals under this measure . Employment at registration is based on information collected from the registrant not on UI wage records. Individuals in both employment and post-secondary education or advanced training in the first quarter after exit will be included in the denominator. Individuals who are not employed but are in only post-secondary education or advanced training in the first quarter after exit are excluded from this measure i.e. programs will not be held accountable for these individuals under this measure . -19- Definitions Post-Secondary Education - a program at an accredited degree-granting institution that leads to an academic degree e.g. AA AS BA BS . Does not include programs offered by degree- granting institutions that do not lead to an academic degree. Advanced Training - an occupational skills employment training program not funded under Title I of the WIA which does not duplicate training received under Title I. Includes only training outside of the One-Stop WIA and partner system i.e. training following exit . Rationale The exclusion of those individuals who move on to post-secondary education or advanced training and not employment avoids the disincentive of penalizing a program for placing an older youth in post-secondary education or advanced training. It better aligns the older youth measures with the younger youth measures which reward programs for moving youth into post-secondary education or advanced training. Those individuals who are excluded from this measure due to entry into post- secondary education or advanced training will be measured in the older youth credential rate. Measure 10 Older Youth Employment Retention Rate at Six Months Of those who are employed in the first quarter after exit and who are not enrolled in post- secondary education or advanced training in the third quarter after exit Number of older youth who are employed in third quarter after exit divided by the number of older youth who exit during the quarter. Operational Parameters This measure includes all individuals who are employed in the first quarter following exit except those individuals who are employed in the first quarter and not employed in the third quarter following exit but are in post-secondary education or advanced training third quarter following exit. These individuals are excluded from this measure i.e. programs will not be held accountable for these individuals under this measure . Employment in the first and third quarters following exit does not have to be with the same employer. Rationale As in the entered employment rate it would be unfair to penalize a participant who was employed in the quarter after exit but left employment to start post-secondary education in the third quarter after exit. Therefore those not employed in the third quarter after exit who are in post-secondary -20- education or advanced training in that third quarter following exit are excluded from the measure. Measure 11 Older Youth Average Earnings Change in Six Months Of those who are employed in the first quarter after exit and who are not enrolled in post- secondary education or advanced training in the third quarter after exit Total post-program earnings earnings in quarter 2 quarter 3 after exit minus pre-program earnings earnings in quarter 2 quarter 3 prior to registration divided by the number of older youth who exit during the quarter. Operational Parameters This measure includes the same population as the older youth employment retention measure regardless of their employment status at registration . To ensure comparability of this measure on a national level the UI wage records will be the only data source for this measure. Therefore individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from UI wage records are excluded from this measure i.e. programs will not be held accountable for these individuals under this measure . Individuals who are not found to be employed in the first quarter after exit are excludedfrom this measure. States should exclude from this measure any individuals whose entry registration date is so far back in time that accessing quarters 2 and 3 of pre-registration wage data is unfeasible or unreasonable. However participants excluded from this measure for this reason should still be included in any other applicable measures. For example the person should still be counted in the retention measure. CIf supplementary sources are the data source for a participant s employment in the 2nd and or 3rd quarter prior to registration that participant is excluded from this measure. Rationale See rationale for adult earnings change. Measure 12 Older Youth Credential Rate Number of older youth who are in employment post-secondary education or advanced training in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of older youth who exit during the quarter Operational Parameters CAs opposed to the adult and dislocated worker measures where a credential must be coupled with employment for older youth a credential can be coupled with employment -21- entry into post-secondary education or entry into advanced training. CAs opposed to the adult and dislocated worker measures where only those who received training services are included in the measure all older youth exiters will be included in this measure. Credentials can be obtained while a person is still participating in services. Rationale This denominator is different from the adult and dislocated worker credential measure in that those two measures are a subset of participants enrolled in training while this measure includes all older youth. The reason is that there is not a specific activity of training in the youth program as there is in the adult and dislocated worker programs through individual training accounts. Therefore it is not possible to limit the measure to those enrolled in training. K. Younger Youth Age 14-18 Measures Measure 13 Younger Youth Skill Attainment Rate Of all in-school youth and any out-of-school youth assessed to be in need of basic skills work readiness skills and or occupational skills Total number of basic skills goals attained by younger youth plus number of work readiness skills goals attained by younger youth plus number of occupational skills goals attained by younger youth divided by the total number of basic skills goals plus the number of work readiness skills plus the number of occupational skills goals set. Operational Parameters The measure should create an appropriate intermediate-type measure for youth who require more services such as academic and soft skills development prior to attaining a diploma or equivalency employment and post secondary education. If a participant is deficient in basic literacy skills the individual must set at a minimum one basic skills goal the participant may also set work readiness and or occupational skills goals if appropriate . WIA participants counted in this measure will be all in-school and any out-of-school youth assessed to be in need of basic skills work readiness skills and or occupational skills. All youth measured in this rate must have a minimum of one skill goal set per year and may have a maximum of three goals per year. The target date for accomplishing each skill goal must be set for no later than one year. The skill goal or the target date set can only be extended if the participant has a gap in service where they are placed in a hold status in which the participant is not receiving services but plans to return to the program. When they enter a hold status the one year clock for the goal target date stops. The clock begins again once the participant is no -22- longer in a hold status. Goals will fall into the category of basic skills work readiness skills or occupational skills. Participants may have any combination of the three types of skill goals three skill goals in the same category two skill goals in one category and one skill goal in another or one skill goal in each category etc. . Success of skill attainment goals will be recorded in the quarter of goal achievement while failure will be recorded in the quarter one year from the time the goal was set if not attained by such time. Definitions Basic literacy skills deficient - the individual computes or solves problems reads writes or speaks English at or below the 8th grade level or is unable to compute or solve problems read write or speak English at a level necessary to function on the job in the individual s family or in society. In addition States and locals have the option of establishing their own definition which must include the above language. In cases where States and or locals establish such a definition that definition will be used for basic literacy skills determination. Basic skills goal - measurable increase in basic education skills including reading comprehension math computation writing speaking listening problem solving reasoning and the capacity to use these skills. Occupational skills goal - primary occupational skills encompass the proficiency to perform actual tasks and technical functions required by certain occupational fields at entry intermediate or advanced levels. Secondary occupational skills entail familiarity with and use of set-up procedures safety measures work-related terminology record keeping and paperwork formats tools equipment and materials and breakdown and clean-up routines. Work readiness skills goal - work readiness skills include world of work awareness labor market knowledge occupational information values clarification and personal understanding career planning and decision making and job search techniques resumes interviews applications and follow-up letters . They also encompass survival daily living skills such as using the phone telling time shopping renting an apartment opening a bank account and using public transportation. They also include positive work habits attitudes and behaviors such as punctuality regular attendance presenting a neat appearance getting along and working well with others exhibiting good conduct following instructions and completing tasks accepting constructive criticism from supervisors and co-workers showing initiative and reliability and assuming the responsibilities involved in maintaining a job. This category also entails developing motivation and adaptability obtaining effective coping and problem-solving skills and acquiring an improved self image. -23- Rationale There is concern about the potential lack of comparability among States and local areas on this measure. At the same time local flexibility is important because of the individual service strategy philosophy. With these competing forces it is important to set some guidelines to ensure comparability while not being so proscriptive as to prevent participants from setting appropriate goals. Therefore a maximum of three goals per person in each year is allowable in order to prevent the setting of multiple minimum-level skills goals while still encouraging participants to set goals in any one of the skill categories. Measure 14. Younger Youth Diploma or Equivalent Attainment Of those who register without a diploma or equivalent Number of younger youth who attained secondary school diploma or equivalent by the end of the first quarter after exit divided by the number of younger youth who exit during the quarter except those still in secondary school at exit Operational Parameters If a younger youth exits WIA while still enrolled in secondary education the individual isexcluded from the measure i.e. programs will not be held accountable for these individuals under this measure . All younger youth except those still in secondary school at exit and those who have already attained their diploma or equivalent prior to registration will be assessed in this measure in the quarter after exit. Definitions High School Diploma Equivalent - a GED or high school equivalency diploma recognized by the State. Rationale Those participants who exit WIA while still enrolled in secondary education are excluded from the measure. This exclusion is because some participants may decide to leave the program before finishing high school. It would be impossible for those participants to have attained a diploma or equivalent at the time of exit if they are still enrolled in high school. This measure is intended to motivate program staff to continue services to participants until they attain a diploma or its equivalent. -24- Measure 15. Younger Youth Retention Rate Number of younger youth found in one of the following categories in the third quarter following exit post secondary education advanced training employment military service qualified apprenticeships divided by the number of younger youth who exit during the quarter except those still in secondary school at exit . Operational Parameters If the participant is in one of the placement activities listed above during the third quarter following exit the individual is counted as successfully retained the participant does not have to remain in the same activity for the entire retention period as long as the participant is found in one of the activities during the third quarter . If the participant exits WIA and does not enter into any of placement activities by the time retention is measured the participant is counted in the denominator of the measure and it is reflected as a negative outcome. If a younger youth exits WIA while still enrolled in secondary education the individual isexcluded from the measure i.e. programs will not be held accountable for these individuals under this measure . Definitions Qualified apprenticeship - a program approved and recorded by the ETA Bureau of Apprenticeship and Training BAT or by a recognized State Apprenticeship Agency State Apprenticeship Council . Approval is by certified registration or other appropriate written credential. Military service - reporting for active duty. Rationale This measure assesses retention in the third quarter following exit from the program. This measurement time period is consistent with the adult and dislocated worker retention period which also measure retention in the third quarter following exit. The WIA stipulated one measure for younger youth placement and retention. Because of the increasing emphasis on post-program measures retention was chosen as the focus of this measure. To be consistent with the -25- diploma equivalency attainment rate and because it would be unfair to penalize those participants who exit while still in secondary school the denominator of this measure does not include those who exit while still in secondary school. L. How Summer Youth Activities fit into Performance The Measures. Unlike the performance measurement system under the Job Training Partnership Act JTPA where participants in the separately funded summer youth program were not included in measuring JTPA youth outcomes all WIA youth will be measured as part of a comprehensive youth program assessed by the core performance measures for youth. The majority of youth who have participated in the summer program in the past have been between the ages of 14-17. These participants would fall under the three younger youth measures the skill attainment rate the diploma equivalency measure and the retention rate. An in-school youth who exits and returns to secondary school following participation in summer employment opportunities would not be included in the younger youth diploma or equivalency rate or the younger youth retention rate. Such a participant would only be included in the skill attainment rate. Within the skill attainment rate if the participant is basic literacy skills deficient the individual must have a basic skills goal. If the participant is not basic literacy skills deficient a work readiness skills goal would be appropriate for a youth in a summer employment opportunity. Therefore it is in the best interest of the local board to serve an in-school youth who is basic skills deficient all year long and not only in the summer employment opportunities component. Out-of-school youth would be included in all three younger youth measures because they would not be returning to secondary school following summer employment opportunities. Therefore it is in the best interest of participants and of the local board to serve out-of-school youth all year long and not only in summer employment opportunities. All older youth are included in the four older youth measures entered employment retention earnings change and credential attainment with exceptions in certain circumstances. All four of these measures are outcome measures assessed following exit. Older youth who participate in the summer portion of the youth program and exit will be held to the same rules as any other older youth who exits the program. Therefore it is in the interest of local areas to serve older youth who participate in summer employment opportunities in year-round services if the youth do not have a credential and or job placement. Youth who transition from JTPA to WIA during the summer. All JTPA youth who transition into WIA on July 1 will be included in the WIA youth performance measures. The following section applies to those in-school youth who were enrolled in JTPA and transition into WIA. Service providers may assess youth enrolled in JTPA Title II-B in the spring of 2000 as if they were WIA -26-participants in anticipation of them becoming WIA participants on July 1 2000. This means determining basic skills needs and setting skill attainment goals for these youth. Alternatively youth competencies established under JTPA may be converted into skill attainment goals so that they will count toward the skill attainment rate. Reassessment should only take place if the youth never received a basic skill level determination. Examples of how to convert youth competencies into skill attainment goals include the following basic education skills under youth competencies would be the equivalent of a basic skills goal under the skill attainment measure pre-employment skills and work maturity skills under youth competencies would be the equivalent of a work readiness skills goal under the skill attainment measure and job-specific skills under youth competencies would be the equivalent of an occupational skills goal under the skill attainment measure. M. Data Sources and Methods to Collect Data on Adult Dislocated Worker and Youth Measures. This section describes data sources and methods to collect data for all of the measures except the two customer satisfaction measures see Section 6. for description of customer satisfaction methodology and data sources . Since there are common outcomes across most of the adult dislocated worker and youth measures the data sources are described by outcome i.e. employment credential etc. . Table 2 is presented to help easily reference the measures by the outcome category and data sources. Table 2 Measures by Outcome Category and Data Sources MeasureOutcome CategoryData Source s Adult Entered Employment RateEmploymentUI wage records and supplemental data sources Adult Employment Retention Rate EmploymentUI wage records and supplemental data sources Adult Earnings ChangeEmploymentUI wage records only Adult Employment and Credential RateEmployment and Credential AttainmentUI wage records supplemental data sources and credential data sources Dislocated Worker Entered Employment RateEmploymentUI wage records and supplemental data sources MeasureOutcome CategoryData Source s -27-Dislocated Worker Employment Retention RateEmploymentUI wage records and supplemental data sources Dislocated Worker Earnings Replacement RateEmploymentUI wage records only Dislocated Worker Employment and Credential Employment and Credential AttainmentUI wage records supplemental data sources credential data sources Older Youth Entered Employment RateEmployment and Placement in Postsecondary Education or Advanced TrainingUI wage records supplemental data sources and postsecondary education or advanced training data sources Older Youth Employment Retention RateEmployment and Placement in Postsecondary Education or Advanced TrainingUI wage records supplemental data sources postsecondary education or advanced training data sources Older Youth Earnings ChangeEmployment and Placement in Postsecondary Education or Advanced TrainingUI wage records and placement in postsecondary education advanced training data sources Older Youth Credential RateEmployment Credential Attainment and Placement in Postsecondary Education or Advanced TrainingUI wage records supplemental data sources credential data sources and postsecondary education or advanced training data sources Younger Youth Skill AttainmentSkill AttainmentSkill attainment data sources Younger Youth H.S. Diploma or EquivalentCredential AttainmentCredential data sources Younger Youth Retention RateYounger Youth RetentionAll identified in the younger youth retention discussion -28- 1. EmploymentUnemployment Insurance Wage RecordsTo the extent it is consistent with State law the Unemployment Insurance UI wage records will be the primary data source for tracking entered employment employment retention earnings change replacement the employment portion of the credential rate for adults dislocated workers and older youth and the employment portion of the younger youth retention rate. In addition to UI wage records within a State the Wage Record Interchange System in the pilot stages at this time and other State Employment Security Offices to track UI wage records across States are considered acceptable wage record sources. If individuals are not found in the UI wage record sources States may use supplemental data sources for the entered employment retention credential and younger youth retention measures. However the only data source that can be used for the earnings change replacement measures is the UI wage records. Further guidance on the use of UI wage records is provided in Attachment C. Supplemental Data SourcesWhile the majority of employment in a State s workforce is covered and will be in the UI wage records certain types of employers and employees are excluded by Federal unemployment law standards or are not covered under a State s UI law. Uncovered employment typically includes Federal employment postal service military railroad out-of- state employment self employment some agricultural employment and some employment where earnings are primarily based on commission. States have flexibility in methods used to obtain information on participants in uncovered employment. Examples include 1 Case management follow-up services and surveys of participant to determine that the participant is employed and written documentation of that employment or 2 Record sharing and or automated record matching with other employment and administrative databases to determine and document employment. These databases may include Office of Personnel Management Federal Career Service United States Postal Service Railroad Retirement System U.S. Department of Defense State Department of Revenue or Tax State income tax for self-reported occupations and Government Employment Records State government local government judicial -29- employment public school employment etc. Most States will likely utilize case management follow-up services and surveys of participants with written documentation for supplemental data. Some States already have record sharing and or automated matching systems in place that they will be able to use to track uncovered employment. For States that do not have these systems in place developing these relationships with State and Local agencies is encouraged. However record sharing of Federal databases can best be coordinated on a national level rather than having each individual State contact these agencies. It is also possible for States to establish record sharing matching agreements with agencies such as the State Corrections Department to determine whether individuals are incarcerated and should be excluded from the measures. Even though record sharing is an efficient and reliable data source many of these agencies may only provide data annually so that it would not be available for quarterly reporting. All data and methods to supplement wage record data must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by a written document such as W2 form pay stub 1099 form or other written documentation. Telephone verification of employment with employers is acceptable but must also be documented. For self-employed individuals telephone verification with major clients contracting entities is also acceptable but must be documented. Supplemental data must be recorded within 30 days after the individual was found missing in the wage record. 2. Credential AttainmentStates and localities will have flexibility in methods used to collect data on credentials for the adult dislocated worker older youth credential rate and younger youth diploma equivalent attainment rate. Examples include 1 Case management follow-up services and surveys of participant to determine that the participant has received a credential and written documentation of that credential or 2 Record sharing agreements and or automated record matching with administrative other databases to determine and document that the participant has received a credential. These databases may include CState Board of Education CState Board Governing Community Colleges CState Board Governing Universities CState Licensing Boards for Private Schools -30- CState Education Associations Integrated Postesecondary Education Reporting Unit CHigher Education Planning Unit CState Department of Professional or Occupational Regulation possibly other units such as health care administration or specific boards like the Board of Nursing CEmployers CTraining Institutions Providers and CAdult Basic Education Providers GED Equivalent testing agencies Most States will likely utilize case management follow-up services and surveys of participants with written documentation of credentials. Some States already have record sharing and or automated matching systems in place that they will be able to use to track credentials. All data and methods to determine achievement of credentials must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written document such as a certificate degree or other written documentation. Telephone verification with the certification institution entity that a person has attained the credential is also acceptable but must also be documented. 3. Placement in Post-Secondary Education or Advanced TrainingThe older youth credential rate and the younger youth retention rate have multiple potential outcomes. In both cases one of the potential outcomes is placement in post-secondary education or advanced training. The younger youth retention rate also has other possible outcomes. Therefore the data sources and methods for data collection are described later in this section under Younger Youth Retention. In addition to the older youth credential rate the other three older youth employment-related measures also allow for individuals to be excluded from those measures if the youth is found in post-secondary education or advanced training outside of the WIA funded and partner system. The following data sources can be used to determine whether youth are placed in post-secondary education or advanced training. 1 Case management follow-up services and surveys of the participant to determine if the youth has been placed in post-secondary education or advanced training and written documentation of that placement or 2 Record sharing agreements and or automated record matching with administrative other databases to determine and document that the participant has been placed in post-secondary education or advanced training. These databases may include CState Board Governing Community Colleges -31- CState Board Governing Universities CState Education Associations CIntegrated Postsecondary Education Reporting Unit CHigher Education Planning Unit and CTraining Institutions Providers Most States will likely utilize case management follow-up services and surveys of participants with written documentation. Some States already have record sharing and or automated matching systems in place that they will be able to use to track outcomes. All data and methods must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written documentation. Telephone verification with the appropriate institution entity is also valid but must be documented. 4. Skill AttainmentTo measure skill attainment for the younger youth skill attainment measure local programs must assess youth at intake to determine whether they are in need of basic skills work readiness skills and or occupational skills. This assessment will include a review of basic skills occupational skills prior work experience employability interests aptitudes and supporting service needs. From this assessment skill attainment goals will be established see operational parameters under Section I skill attainment measure . To determine whether youth meet the skill attainment goals will require a pre-assessment and post-assessment of skill level. The use of a standardized assessment procedure such as a standardized test or a performance-based assessment with a standardized scoring method is encouraged. These tests may include Test of Adult Basic Education TABE DOL Workplace Literacy Test DOL-WLT Adult Measure of Educational Skills AMES Adult Basic Learning Examination ABLE Adult Literacy Test ALT Simon Schuster Armed Forces Qualifying Test AFQT Basic Occupational Literacy Test BOLT California Achievement Test CAT Career Ability Placement Survey CAPS Comprehensive Adult Student Assessment System CASAS General Aptitude Test Battery GATB Iowa Test of Basic Skills ITBS Metropolitan Achievement Test MAT Reading Job Corps Screening Test RJCST or -32- Wide Range Achievement Test WRAT In cases where a standardized test or a performance-based assessment is not available such as for work-readiness skills assessment techniques must be objective unbiased and conform to widely accepted clearly defined criteria be field tested for utility consistency and accuracy and provide for the training preparation of all raters scorers. Information on achievement of skill attainment goals should be derived from case management or follow-up services. All data and methods to determine achievement of skill attainment goals must be documented and are subject to audit. 5. Younger Youth RetentionSince the younger youth retention rate has five possible positive outcomes multiple data sources are needed to track participants. The data sources to determine whether a youth has been placed in employment or military service are addressed in the previous discussion on data sources for employment. The data sources to determine whether a youth has been placed in post-secondary education or advanced training are addressed in the previous discussion on placement in post- secondary education or advanced training. To determine whether a youth has been placed in a qualified apprenticeship the following methods can be used 1 Case management follow-up services and surveys of the participant and written documentation or 2 Record sharing agreements and or automated record matching with the U.S. Department of Labor Bureau of Apprenticeship and Training or recognized State Apprenticeship Agency . Most States will likely utilize case management follow-up services and surveys of participants with written documentation. Some States already have record sharing and or automated matching systems in place that they will be able to use to track outcomes. All data and methods to determine achievement of one of the five outcomes must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written documentation. Telephone verification with the appropriate institution entity is valid but must also be documented. 6. Customer Satisfaction Measures. A. Overview of Measurement Approach. To meet the customer satisfaction measurement requirements of WIA the Department will use customer satisfaction surveys. The survey approach that will be utilized allows State and local flexibility and at the same time captures common customer -33- satisfaction information that can be aggregated and compared at a State and national level. This will be done through the use of a small set of required questions that will form a customer satisfaction index. The Department will use the American Customer Satisfaction Index ACSI which is created by combining scores from three specific questions that address different dimensions of customers experience. For WIA application there will be one score for each of the two customer groups participants and employers. The ACSI is the most widely used index currently in practice. It is used extensively in the business community including over 150 Fortune 500 companies and in many European countries. Twenty-nine agencies of the Federal government are using the ACSI. In addition it has been used twice in the past four years to assess customer satisfaction for ETA s Quality Initiative the Enterprise. The ACSI will allow the workforce investment system to not only look at performance within the system but also be able to gain perspective on the workforce system s performance by benchmarking against organizations and industries outside of the system. The ACSI also has a history of being useful in tracking change in customer satisfaction over time making it an ideal way to gauge States progress toward continuously improving in performance. Since the ACSI trademark is proprietary property of the University of Michigan and its software is owned by Claes Fornell International CFI Group the Department has established a license agreement with the University of Michigan that will allow States the use of the ACSI for a Statewide sample of participants and employers. States that want to use the ACSI for measuring customer satisfaction for each local area will have to establish an independent contract with the University of Michigan. States may also contract with CFI Group for additional assistance in measuring analyzing and understanding ACSI data. B. Customer Satisfaction Measures Measure 16 Participant Satisfaction The weighted average of participant ratings on each of the three questions regarding overall satisfaction are reported on a 0-100 point scale. The score is a weighted average not a percentage. 1. Who Will Be Surveyed WIA Title I-B participants who are exiters as defined in the core measures who are either Adults Dislocated Workers Youth 19-21 or Youth 14-18 will be surveyed. All individuals from all funding streams in an exit cohort are eligible to be chosen for inclusion in the random sample. 3 Note The first question can be modified to suit the individual needs of the State and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance. -34- 2. How Many number obtained Except in small States a sample will be taken from these exiter groups in each quarter. Five hundred completed participant surveys must be obtained each year for calculation of the indicator. A completed participant survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. For small States those with less than 1000 exiters in a year the entire population must be surveyed. The response rate from the sample with valid contact information must be a minimum of 50 percent. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than 5 points from the score obtained from surveying the whole population. 3. How methodology The responses are obtained using a uniform telephone methodology. The rationale for only using telephone surveys include the comparability of the indicator for assessing performance levels is most reliably obtained with a telephone survey telephone surveys are easily and reliably administered and defining procedures for mailed surveys is more difficult than defining procedures for telephone surveys. Estimates of the cost of telephone surveys nationwide run an average of 15 per completed survey. Since it is being proposed that States complete 500 participant and 500 employer surveys the cost would be an estimated total of about 15 000 per State per year. 4. When to Conduct Surveys The surveys should be conducted on a rolling basis within the time frame for participants and employers indicated below. To obtain sufficient numbers smaller States will need to survey on an ongoing basis. Participants should be contacted within 60 days of the exit date or the date that an exit date has been determined. This means either 60 days after the date of an exit interview or 60 days after the 90 days have elapsed since the last service date. 5. What are the Questions A survey conducted by telephone in which the following questions are asked at the beginning of the interview3 My name is with XXXXX and I am conducting a survey for the XXXX XXXXX. 4 DK Don t Know 5 REF Refused to Answer -35- I would like to speak to Ms. Mr. . Are you the Ms. Mr. who was looking for a job a few months ago I would like to ask you some questions about your recent experience looking for a job. Our purpose is to learn from you how to improve programs and services offered to people in XXX. The survey should take about XX minutes to complete. First I am going to read a list of services you may have received. Indicate as I read them those you recall receiving during the period in which you were seeking employment and or training at the XX center. C A thorough assessment of your needs C Assistance about finding a job C Assistance to develop an individual employment plan C Assistance to decide about the best training to take C Assistance from someone to support you during your job search or training C Did you receive any training C Occupational training C Training to give you general skills for the workplace C Training in English or math Did you get any other help or services that I have not mentioned specify 1 Utilizing a scale of 1 to 10 where 1 means Very Dissatisfied and 10 means Very Satisfied what is your overall satisfaction with the services provided from VeryVery DK4 REF5 DissatisfiedSatisfied 12345678910 11 12 2 Considering all of the expectations you may have had about the services to what extent have the services met your expectations 1 now means Falls Short of Your Expectations and 10 means Exceeds Your Expectations. 6 This standard is similar to the participant standard that distinguishes core services information self- service from those services that warrant registration. 7 Where an employer has received multiple services in a given time period and there is separate contact information for each service the contact information for the priority service should be used for surveying. -36- Falls Short of ExceedsDKREF Expectations Expectations 123456789101112 3 Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services 1 now means Not very close to the Ideal and 10 means Very Close to the Ideal. Not Close Very Close To Ideal To IdealDK REF 1234567891011 12 Measure 17 Employer Satisfaction The weighted average of employer ratings on each of the three questions regarding overall satisfaction are reported on a 0-100 point scale. The score is a weighted average not a percentage. 1. Who Will Be Surveyed Those eligible for surveying include employers who have received service where the service has been completed or if it is an ongoing service when a full segment of service has been provided e.g. after listing an open job order the employer has received some referrals or if no service 30 days have elapsed after the initial request . All employers who have received a substantial service involving personal contact with One-Stop staff are eligible to be chosen for inclusion in the random sample this excludes those employers who request a brochure or standard mailing those who ask a question that is answered with little expenditure of staff time or those who use electronic self- services .6 Examples of services include staff facilitated job orders customized job training customized labor market information requests and on-the-job training activities. When an employer has received multiple services priority should be given to the service that required the greatest expenditure of funds or staff time7 and the survey conducted regarding their satisfaction with that service. -37- 2. How Many number obtained Except in small States a sample will be taken from these employers in each quarter. Five hundred completed participant surveys must be obtained each year for calculation of the indicator. A completed employer survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. For small States those with less than 1000 employers who received a substantial service in a year the entire population must be surveyed. The response rate from the sample with valid contact information must be a minimum of 50 percent. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than 5 points from the score obtained from surveying the whole population. 3. How methodology The responses are obtained using a uniform telephone methodology. The rationale for only using telephone surveys include the comparability of the indicator for assessing performance levels is most reliably obtained with a telephone survey telephone surveys are easily and reliably administered and defining procedures for mailed surveys is more difficult than defining procedures for telephone surveys. Estimates of the cost of telephone surveys nationwide run an average of 15 per completed survey. Since it is being proposed that States complete 500 participant and 500 employer surveys the cost would be an estimated total of about 15 000 per State per year. 4. When to Conduct Surveys The surveys should be conducted on a rolling basis within the time frame indicated. To obtain sufficient numbers smaller States will need to survey on an ongoing basis. Employers should be contacted within 60 days of the completion of the service or 30-60 days after a job order has been listed where no referrals have been made. 5. What are the Questions A survey will be conducted by telephone in which the following questions are asked at the beginning of the telephone interview the first question is a sample and can be modified to suit the needs of individual states A survey conducted by telephone in which the following questions are asked at the beginning of the 8 Note The first question can be modified to suit the individual needs of the state and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance. -38- interview8 My name is with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms. Mr. . Are you the Ms. Mr. who describe the service received . I would like to ask you some questions about your recent experience with . Our purpose is to learn from you how to improve programs and services offered to employers. The survey should take about XX minutes to complete. 1 Utilizing a scale of 1 to 10 where 1 means Very Dissatisfied and 10 means Very Satisfied what is your overall satisfaction with the service s provided from VeryVery DissatisfiedSatisfied DK REF 12345678910 11 12 2 Considering all of the expectations you may have had about the services to what extent have the service s met your expectations 1 now means Falls Short of Your Expectations and 10 means Exceeds Your Expectations. Falls Short of Exceeds DKREF Expectations Expectations 12345678910 11 12 3 Now think of the ideal service s for people in your circumstances. How well do you think the service s you received compare with the ideal service s 1 now means Not Very Close to Ideal and 10 now means Very Close to the Ideal. Not Close Very Close To Ideal To Ideal DK REF 12345678910 11 12 -39- C. Definition of Terms Sample. A group of cases selected from a population by a random process where everyone has an equal probability of being selected. Response rate. The percentage of people who have valid contact information who are contacted and respond to the questions on the survey. Valid contact information. Information that leads the interviewer or surveyor to the location where the contact individual is located whether or not the individual responds and answers the questions. D. The Calculation The calculation for the American Customer Satisfaction Indicator ACSI is accomplished by calculating the weighted average of the raw scores for each of the customer satisfaction questions given by each respondent. The weighted average score is then transformed to an index reported on a 0-100 scale. The aggregate index score is simply the weighted average of each case s index score. Notes CFI Group will provide the actual weights given for W1 W2 and W3 below. In calculating respondent level index scores round to two decimal points. When calculating the average index score round to the nearest whole number. For any case the general formula for calculating the index score is given as Index Score Q1 W1 Q2 W2 Q3 W3 -1 x 11.111 where Q1 raw score on question 1W1 weight for question 1 Q2 raw score on question 2W2 weight for question 2 Q3 raw score for question 3W3 weight for question 3 Example If the respondent answers were 5 8 and 9 respectively for each of the three customer satisfaction questions and the weights for each of the three questions were .4 .2 and .4 respectively the calculation for the respondent s index score would be as follows 5 .4 8 .2 9 .4 - 1 x 11.111 -40- 7.2 - 1 x 11.111 68.89 If two more respondents whose on the three questions were 6 10 and 6 and 9 6 and 7 respectively using the same weights listed above those two respondent s index scores would be 64.44 and 73.33. To calculate the aggregate index score simply average the individual respondent s index scores and round to the nearest whole number as follows 68.89 64.44 73.33 3 69 These weights are examples only CFI group will provide the actual weights. 7. Action Required. States shall distribute this Guidance Letter to all officials within the State who need such information to implement WIA core and customer satisfaction measures. 8. Inquiries. Questions concerning this issuance may be directed to your appropriate Regional Office. 9. Attachments. A WIA Core and Customer Satisfaction Measures-at-a-Glance B Definitions of Key Terms C Guidance for Calculating WIA Core Measures -41- Attachment A - WIA Core and Customer Satisfaction Measures-at-a-Glance Adult Measures1. Entered Employment Rate Of those who are not employed at registration of adults who have entered employment by the end of the 1st quarter Qtr. after exit of adults who exit during the Qtr. 2. Employment Retention Rate Of those who are employed in 1st Qtr. after exit of adults who are employed in 3rd Qtr. after exit of adults who exit during the Qtr. 3. Earnings Change in Six Months Of those who are employed in 1st Qtr. after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit - Pre-Program Earnings earnings in Qtr. 2 Qtr.3 prior to registration of adults who exit during the Qtr. 4. Employment and Credential Rate Of those who received training services of adults who were employed in the 1st Qtr. after exit and received a credential by the end of 3rd Qtr. after exit of adults who exit during the Qtr.Dislocated Worker Measures5. Entered Employment Rate of dislocated workers DW who have entered employment by the end of the 1st Qtr. after exit of dislocated workers who exit during the Qtr. 6. Employment Retention Rate Of those who are employed in the 1st Qtr. after exit of DW who are employed in 3rd Qtr. after exit of DW who exit during the Qtr. 7. Earnings Replacement Rate Of those who are employed in the 1st Qtr. after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit Pre-Dislocation Earnings earnings in Qtr. 2 Qtr.3 prior to dislocation 8. Employment and Credential Rate Of those who received training services of DW who were employed in the 1st Qtr. after exit and received a credential by the end of 3rd Qtr. after exit of DW who exit during theQtr. -42- Older Youth 19-21 years old Measures9. Entered Employment Rate Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the first quarter after exit of older youth OY who have entered employment by the end of the 1st Qtr. after exit of OY who exit during the Qtr. 10. Employment Retention Rate Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit of OY who are employed in 3rd Qtr. after exit of OY who exit during the Qtr. 11. Earnings Change in Six Months Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit - Pre-Program Earnings earnings in Qtrs 2 3 prior to registration of OY who exit during the quarter 12. Credential Rate of OY who were in employment post-secondary education or advanced training in the first Qtr. after exit and received a credential by the end of 3rd Qtr.after exit of OY who exit during the Qtr. Across Funding Streams16. Participant Customer Satisfaction The weighted average of participant ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale. Younger Youth 14-18 years old Measures13. Skill Attainment Rate Of all in-school youth and any out-of-school youth assessed to be in need of basic skills work readiness skills and or occupational skills Total of attained basic skills of attained WR skills of attained Occ. skills Total of basic skills goals of WR skills goals of Occ. skills goals 14. Diploma or Equivalent Attainment Rate Of those who register without a diploma or equivalent of younger youth YY who attained a secondary school diploma or equivalent by the end of the 1st Qtr. after exit of YY who exit during the Qtr. except those still in secondary school at exit 15. Retention Rate of younger youth found in one of the following categories in the 3rd Qtr. after exit post secondary education advanced training employment military service qualified apprenticeships -43- of YY who exit during the Qtr. except those still in secondary school at exit 17. Employer Customer Satisfaction The weighted average of employer ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale. Attachment B - Definitions of Key Terms Advanced Training - an occupational skills employment training program not funded under Title I of the WIA which does not duplicate training received under Title I. Includes only training outside of the one-stop WIA and partner system i.e. training following exit . Basic literacy skills deficient - the individual computes or solves problems reads writes or speaks English at or below the 8th grade level or is unable to compute or solve problems read write or speak English at a level necessary to function on the job in the individual s family or in society. In addition States and locals have the option of establishing their own definition which must include the above language. In cases where States and or locals establish such a definition that definition will be used for basic literacy skills determination. Basic Skills Goal - measurable increase in basic education skills include reading comprehension math computation writing speaking listening problem solving reasoning and the capacity to use these skills. Credential - nationally recognized degree or certificate or State locally recognized credential. Credentials include but are not limited to a high school diploma GED or other recognized equivalents post-secondary degrees certificates recognized skill standards and licensure or industry-recognized certificates. States should include all State Education Agency recognized credentials. In addition States should work with local Workforce Investment Boards to encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment retain employment or advance into better employment. Date of Dislocation - The last day of employment at the dislocation job. If there is no date of dislocation date of registration will be used instead. Employed at Registration - An employed individual is one who during the 7 consecutive days prior to -44- registration did any work at all as a paid employee in his or her own business profession or farm worked 15 hours or more as an unpaid worker in an enterprise operated by a member of the family or is one who was not working but has a job or business from which he or she was temporarily absent because of illness bad weather vacation labor-management dispute or personal reasons whether or not paid by the employer for time-off and whether or not seeking another job. Employed in Quarter After Exit Quarter - The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the primary data source for tracking employment in the quarter after exit. When supplemental data sources are used individuals should be counted as employed if in the calendar quarter after exit they did any work at all as paid employees i.e. received at least some earnings worked in their own business profession or worked on their own farm. Exit - Determined as follows 1. a participant who has a date of case closure completion or known exit from WIA-funded or non-WIA funded partner service within the quarter hard exit or 2. a participant who does not receive any WIA Title I funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services soft exit . Exit Date - the last date on which WIA Title I funded or partner services were received by the individual excluding follow-up services. Exit Quarter - quarter in which the last date of service except follow-up services takes place. High School Diploma Equivalent - a GED or high school equivalency diploma recognized by the State. Military service - reporting for active duty. Not Employed At Registration. An individual who does not meet the definition of employed at registration. Occupational Skills Goal - primary occupational skills encompass the proficiency to perform actual tasks and technical functions required by certain occupational fields at entry intermediate or advanced levels. Secondary occupational skills entail familiarity with and use of set-up procedures safety measures work-related terminology record keeping and paperwork formats tools equipment and materials and breakdown and clean-up routines. Planned Gap in Service - no participant activity of greater than 90 days due to a delay before the beginning of training or a health medical condition that prevents an individual from participating in services. Post-Secondary Education - a program at an accredited degree-granting institution that leads to an academic -45- degree e.g. AA AS BA BS . Does not include programs offered by degree-granting institutions that do not lead to an academic degree. Qualified Apprenticeship - a program approved and recorded by the ETA Bureau of Apprenticeship and Training BAT or by a recognized State Apprenticeship Agency i.e. State Apprenticeship Council . Approval is by certified registration or other appropriate written credential. Response Rate - The percentage of people who have valid contact information who are contacted and respond to the questions on the survey. Sample - A group of cases selected from a population by a random process where everyone has an equal probability of being selected. Training Services - Include WIA funded and non-WIA funded partner training services. These services include occupational skills training including training for nontraditional employment on-the-job training programs that combine workplace training with related instruction which may include cooperative education programs training programs operated by the private sector skill upgrading and retraining entrepreneurial training job readiness training adult education and literacy activities in combination with other training and customized training conducted with a commitment by an employer or group of employers to employ an individual upon successful completion of the training. Valid Contact Information - Information that leads the interviewer or surveyor to the location where the contact individual is located whether or not the individual responds and answers the questions. Work Readiness Skills Goal - Work readiness skills include world of work awareness labor market knowledge occupational information values clarification and personal understanding career planning and decision making and job search techniques resumes interviews applications and follow-up letters . They also encompass survival daily living skills such as using the phone telling time shopping renting an apartment opening a bank account and using public transportation. They also include positive work habits attitudes and behaviors such as punctuality regular attendance presenting a neat appearance getting along and working well with others exhibiting good conduct following instructions and completing tasks accepting constructive criticism from supervisors and co-workers showing initiative and reliability and assuming the responsibilities involved in maintaining a job. This category also entails developing motivation and adaptability obtaining effective coping and problem-solving skills and acquiring an improved self image. -46-