TEGL8-99.pdf

ETA Advisory File Text

1 U. S. Department of Labor Employment and Training Administration Washington D.C. 20210CLASSIFICATION WIA Performance CORRESPONDENCE SYMBOL OWS DATE March 3 2000 TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 8-99 TO ALL STATE WORKFORCE LIAISONS ALL STATE WORKER ADJUSTMENT LIAISONS ALL STATE EMPLOYMENT SECURITY LIAISONS ALL ONE-STOP CAREER CENTER SYSTEM LEADS FROM LENITA JACOBS-SIMMONS Deputy Assistant Secretary SUBJECT Negotiating Performance Goals and Incentives and Sanctions Process under Title I of the Workforce Investment Act WIA . 1. Purpose. To inform States of ETA s implementation guidelines for the Negotiation and Goal Setting Process as well as for the Incentive and Sanction Process. 2. References. Workforce Investment Act of 1998 Pub. L. 105-220 Section 136 Workforce Investment Act Interim Final Rule 20 CFR Part 666 published at 64 Federal Register 18662 April 15 1999 . 3. Background. Under the Workforce Investment Act of 1998 States submitting a State Plan under Title I subtitle B must propose expected levels of performance for each of the 17 indicators of performance consisting of the adult dislocated worker and youth programs respectively and the two customer satisfaction indicators. While States may develop additional performance measures for State use such measures will not be negotiated with the Department or included in the incentives and sanctions process. In order to ensure an optimal return on the investment of the activities authorized the Secretary and Governor must reach agreement in the State plan on the levels of performance for each of the first three program years. The requirements of the Government Performance and Results Act GPRA will also be a factor in negotiating the expected levels of performance. A critical component of annual reports to Congress will be representations of the extent to which States RESCISSIONSEXPIRATION DATE Continuing 1 In the Interim Final Rule 20 CFR part 666 the outcome of the reaching agreement on State expected levels of performance was termed adjusted levels of performance. In this document and in the Final Rule the outcome of reaching agreement of State expected levels of performance will be termed negotiated performance level. 2 DISTRIBUTION have met these negotiated performance levels. This process begins with each State negotiating performance levels with their Regional Office for all measures in each program area adults dislocated workers and youth and for customer satisfaction and contains a commitment that incentives and sanctions provided for under the law will be based on meeting those negotiated performance levels. This directive provides the requirements for that process from negotiation between States and ETA s Regional Offices to subsequent review of levels achieved to determine appropriate incentives and sanctions. 4. Reaching Agreement on State Performance Levels. With the statutory inclusion of performance negotiations in WIA comes an expectation that States will use this provision in a pro- active manner to support their efforts for continuous improvement and customer satisfaction. Therefore ETA has developed requirements that provide a uniform framework for negotiations and give States flexibility in pursuing continuous improvement for each program area. In order to obtain negotiated levels of performance for all 17 measures that demonstrate this commitment the following negotiation process will be utilized. Please note that early implementing States will be negotiating performance levels for their 2nd and 3rd years of performance. Early implementing States should conform such negotiations to the guidance in this directive. A. Process For Reaching Agreement on State Performance Levels The DOL process for reaching agreement with States on performance levels will include these steps 1 States will develop baseline data following the instructions within this directive. 2 States will analyze the baseline data and then offer expected State levels of performance submitted as part of their State Plan for each of the 17 indicators for each of the first three program years. States are encouraged to share this advance data and work with their Regional Office throughout this process. 3 The information will go to the ETA Regional Office serving the State to review and the Regional Office will negotiate with the State to obtain mutually agreed upon expected levels of performance. 4 The Regional Office will submit the negotiated performance levels1 to the National Office for ETA for inclusion in the State Plan approval process. 3 5 Actual performance for each program year will then be compared to these negotiated performance levels to determine whether States are eligible for incentives or subject to sanctions. 6 Prior to the fourth program year States will submit their expected performance levels for the fourth and fifth years of their program following this guidance in the form of plan modifications. 7 ETA will also follow this guidance in the review negotiation and submission of the performance levels for program years four and five although adjustments in the process may occur based upon experience. B. Baselines To the extent available baselines for each of the 17 measures will be developed by each State and will be a key factor used to determine the level of performance that is negotiated with ETA. Summary definitions of the measures are presented in Attachment A. Baselines are intended to give an indication of the past outcomes of a performance measure. For performance negotiations to be data-driven and reality based the development of baselines is a critical aspect of the negotiation process. Because WIA authorizes new programs there is not a perfect match of baseline data available to predict program performance under WIA. Instead baselines must be based on data from preexisting programs. This situation will change in the coming years once WIA data is accumulated. Initially developing baselines using JTPA data is more appropriate for some of the core performance measures than for others. Baselines can be developed for the entered employment employment retention and earnings gain measures based on wage records because the wage record data although not used under JTPA can be accessed retrospectively by most States. To assist States in the development of baseline data and expected performance levels ETA has developed parameters to be used by all States in their baseline calculations. The Department anticipates that States will submit expected levels of performance that are above the baseline figures computed for the State. The parameters are outlined in Attachment B. The matrix in Attachment B will give States general instructions for each measure and will provide for comparable data nationwide. However for specific operational definitions and calculation instructions on the core performance measures and the customer satisfaction measures States should reference the TEGL 7-99 Core and Customer Satisfaction Measures For the Workforce Investment System. C. State Expected Levels of Performance States will submit expected State levels of performance as part of the State Plan. States should be prepared to provide support for their expected performance levels by providing information on how 4 baseline performance levels were developed as well as by providing other information States believe may affect performance. States will include in their plan CBaseline performance data and expected levels of performance for each of the 17 measures. CThe extent to which the expected levels of performance will result in the State attaining a high level of customer satisfaction. CHow the expected levels of performance compare with projected national averages see Attachment B for projected national averages taking into account how factors such as economic conditions the characteristics of participants when the participants entered the program and the services to be provided contributed to the development of the expected performance levels CThe extent to which such expected levels of performance promote continuous improvement and ensure optimal return on investment of Federal funds. In addition the methodology for developing the baseline data and expected levels of performance should be included when possible. This should include a description of data sources calculations factors see below for further discussion on factors and values of factors used for expected levels of performance. For example if a factor such as unemployment rate is used the level of unemployment should also be included. ETA has developed projected national average performance levels for each performance measure which should be utilized in developing State expected levels of performance. The national average for each performance measure is included in the matrix in Attachment B. While most of these national averages are based on the experiences of a limited number of early implementation States ETA has determined these levels to be representative of estimated performance under WIA. Performance levels may vary up or down based on environmental factors beyond the control of the State. Examples of possible factors to consider in negotiating expected levels of performance are listed below. This list of factors is not intended to be prescriptive or exhaustive but to suggest the kinds of information that might be considered in the negotiation process. Differences in Economic Conditions Unemployment rate Rate of job creation loss New business start-ups Characteristics of Participants Indicator of welfare dependency Indicator of educational level 5 Indicator of poor work history Indicator of basic skills deficiency Indicator of disability Indicator of age Creation of hardest-to-serve index Services to be Provided WIA Title I to be spent on each category of service core intensive training Extent of follow-up services planned Extent of experimental pilot programs Extent of non-Title I training money available Other Factors That May Be Considered Community factors like availability of transportation and daycare Policy-objective factors like evidence of application of Malcomb Baldridge criteria pursuit of new or enhanced partnerships piloting of new programs D. Negotiation of Expected Levels of Performance The Regional Office will review the information contained in the State plan and will compare the expected performance levels with the national averages baseline information from other States and the negotiated levels of performance established for other States taking into account factors including differences in economic conditions the characteristics of participants when the participants entered the program and the services to be provided as discussed above. It is understood that comparative information on negotiated levels may not be fully available during the negotiation process . A discussion of the appropriateness of each performance level and the adequacy of any information States offer to substantiate each performance level will be the core of the negotiation process. The Regional Office will analyze the quality of the data presented by States including the relevance of the data the source of the data the time period from which the data was drawn and if the data is part of a trend or anomalous. When the Regional Office finalizes its analysis and determines that the State could increase their expected performance levels to support State continuous improvement and customer satisfaction strategies they will negotiate with the State to obtain mutually agreed upon expected levels of performance. E. Revisions WIA allows for the renegotiation of performance levels if circumstances arise that result in a significant change in the factors used to develop the original performance levels. Since States have limited baseline data to project performance levels for PY 2000 2001 and 2002 it is anticipated that States may want to renegotiate performance for PY 2001 and 2002 as more baseline data becomes available. DOL will collaborate with the system to develop methods for making such revisions. Such criteria 6 could include significant changes in economic conditions changes in service mix or changes in client characteristics. 5. Incentives and Sanctions Policies. Under 20 CFR 666.200 a State is eligible to apply for an incentive grant if its performance for the immediately preceding year exceeds CThe State s negotiated levels of performance for the following areas adult measures dislocated worker measures youth measures and customer satisfaction measures for WIA Title I CThe negotiated levels of performance included in plans submitted to the Department of Education for Title II Adult Education and Literacy programs and CThe negotiated levels of performance under Title I of the Carl D. Perkins Vocational and Technical Education Act. This guidance addresses the initial requirements for a State to be eligible for the WIA Title I incentives grant awards. Additional guidance will be provided at a future date about the relationship between the Department of Labor and Department of Education in the incentive process as well as a detailed description of the incentive award application process. A. Evaluating Performance for Incentives and Sanctions The extent to which each State exceeds meets or falls below its negotiated performance levels will be examined. For each core performance measure the percentage by which each State met its negotiated performance level will be calculated e.g. if a State had an 80 negotiated performance level for the adult entered employment rate and the State s actual performance was 80 they would have achieved 100 of their negotiated performance level and receive a score of 100 on that measure. If the State s actual performance was only 40 the would have achieved only 50 of their negotiated target level and receive a score of 50 on that measure . This analysis will be clustered by program area adults dislocated workers and youth - both older and younger youth measures and for customer satisfaction so that each of the four groups will receive an aggregate performance score calculated from their scores for each measure in each program area and in the customer satisfaction group. The exact aggregation approach to be used will be determined as actual performance information becomes available and will likely include simple or weighted averaging. The resulting score will be the cumulative program area score for each of the four areas. To be eligible to apply for an incentive grant a State must meet two criteria. A State must achieve at least a 100 cumulative program area score for each of the program areas and for the customer satisfaction group. In addition a State may not have any of their 17 measures fall below 80 of their negotiated performance levels in order to be eligible to apply for an incentive grant. 7It is understood that as part of a continuous improvement strategy States may wish to emphasize improvement on a particular measure e.g. entered employment which may have a slightly adverse effect on other measures in the same program area. To allow for these strategies and to avoid consideration of sanctions for only missing their negotiated goal by a few percentage points States will not trigger the sanctions process as long as they have achieved 80 of their negotiated performance levels for each measure. Example 1 shows the four performance measures for the adult area and the extent to which a state exceeded or fell below the negotiated performance levels on each of those measures. Note that the State did not achieve two of the targets. However since the performance on each of the measures was above the 80 threshold the State is within the acceptable performance range. This allows States the flexibility to target certain measures within a program area for improvement without concern for minor decreases in other measures within the program area. The following two examples shows how the cumulative program scores would be used to determine a State s eligibility for an incentive award. In each example the bars for each program area represent the cumulative program score. To encourage States to strive to achieve the negotiated level of performance rather than just to meet the lower acceptable level 80 allowed incentive grants will only be available for a State that achieves a cumulative score in each area adults youth dislocated worker customer satisfaction that is 100 or greater and where a State does not fall below 80 of 8 the negotiated level on any single measure. Example 2 shows the 100 incentive threshold and cumulative program scores for each program area. A State has a cumulative program score of 115 of the negotiated target for the adult program 110 for the dislocated workers 100 for the youth program and an 110 cumulative score for the two customer satisfaction measures and did not fall below 80 of negotiated target on any of the measures . The State would be eligible for an incentive award. In Example 3 A State has cumulative program scores of 100 or higher of negotiated target performance levels in all program areas but less than 100 cumulative score on customer satisfaction. The State would not be eligible for an incentive award. Cumulative scores will NOT be averaged across areas a State cannot make up a deficit in one area by exceeding performance in another. B. Qualifying for Incentive Grants. In order to promote equity and uniformity only the Federally required core and customer satisfaction measures will be considered. C. Amount of the Incentive Award. Awards will range from 750 000 to 3 000 000 with a proportionate reduction in the minimum and maximum when total available funds are insufficient. D. Incentive Application Process. The final approach to distributing incentive awards is still under development but may include the following process steps CPerformance reports from each State will be submitted by a date specified by the Secretary. CThe Department of Labor will partner with the Department of Education to review the performance of each State and determine which States are eligible for an incentive award. CFor those States that are determined eligible the following criteria may be used to determine the amount each State is eligible to receive CThe extent to which goals were exceeded CThe extent of improvements from previous year and CThe relationship of actual performance levels to actual performance levels in other states CThese adjusted award amounts will be published and applications for awards will be solicited. 9E. Failing to Meet the Negotiated Levels. Negotiated levels will be established for each of the 17 performance measures. The lowest acceptable level lower limit will be calculated as 80 of the established State negotiated level. States must attain 80 of the target performance level on each measure for performance to be determined acceptable. If a State falls below this threshold of 80 on any of the performance measures the State may be subject to sanction. Although WIA and this guidance provide some specific requirements for sanctions the Department will review possible sanctions on a case by case basis. Example 4 A State has exceeded its negotiated target performance on adult entered employment by performing 110 on employment retention by 105 on earnings change by 100 but only achieved 75 of the negotiated performance target for credential attainment. The State may be subject to sanction. Example 5 A State did not achieve 100 of the negotiated performance targets on any of the adult measures. However the State did achieve at least 80 of the negotiated performance target on each of these measures. The state would not be subject to sanction. F. Unacceptable Performance in Year One The first year a State experiences unacceptable performance ETA will make technical assistance available. This technical assistance will be provided upon request. One approach to this technical 10 assistance may be helping the State to develop a performance improvement plan with the following components CStatement regarding which performance indicators will be improved through implementation of plan CAnalysis of performance problem including a description of problem solving techniques used to determine the most likely causes of the problem and a description of the most likely causes CIdentification of possible actions to improve performance and a descriptions of what actions the State will take to improve and CA timetable for implementing the selected solutions. G. Unacceptable Performance in Two Consecutive Years. If performance is unacceptable for two consecutive years on the same performance indicator monetary sanctions may be imposed. H. Amount of Monetary Sanctions. Sanctions will be determined on a case by case basis and may range from 1 up to a maximum 5 reduction. I. Sanctions for Failure to Submit Annual Performance Progress Reports. Under 20 CFR 667.300 e States that are more than 45 days late may be sanctioned up to 1 plus up to an additional 1 for each additional 45 day period of lateness up to a total reduction of 5 . 6. Action. States are to provide these requirements to the appropriate State staff and use this guidance to a develop baseline data and expected performance levels PY2000 PY2001 and PY2002 and submit this information in the State s WIA State Plan. b negotiate performance levels to be used in the State Plan approval process. 7. Inquiries. Questions concerning this issuance should be directed to your appropriate Regional Office. 8. Attachments. Attachment A WIA Core and Customer Satisfaction Measures-at-a-Glance Attachment B Common Approach for Establishing Negotiated Levels of Performance Attachment C Summary of Important Terms. 11 Attachment A - WIA Core and Customer Satisfaction Measures-at-a-Glance Adult Measures1. Entered Employment Rate Of those who are not employed at registration of adults who have entered employment by the end of the 1st quarter Qtr. after exit of adults who exit during the Qtr. 2. Employment Retention Rate Of those who are employed in 1st Qtr. after exit of adults who are employed in 3rd Qtr. after exit of adults who exit during the Qtr. 3. Earnings Change in Six Months Of those who are employed in 1st Qtr. after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit - Pre-Program Earnings earnings in Qtr. 2 Qtr.3 prior to registration of adults who exit during the Qtr. 4. Employment and Credential Rate Of those who received training services of adults who were employed in the 1st Qtr. after exit and received a credential by the end of 3rd Qtr. after exit of adults who exit during the Qtr.Dislocated Worker Measures5. Entered Employment Rate of dislocated workers DW who have entered employment by the end of the 1st Qtr. after exit of dislocated workers who exit during the Qtr. 6. Employment Retention Rate Of those who are employed in the 1st Qtr. after exit of DW who are employed in 3rd Qtr. after exit of DW who exit during the Qtr. 7. Earnings Replacement Rate Of those who are employed in the 1st Qtr. after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit Pre-Dislocation Earnings earnings in Qtr. 2 Qtr.3 prior to dislocation 8. Employment and Credential Rate Of those who received training services of DW who were employed in the 1st Qtr. after exit and received a credential by the end of 3rd Qtr. after exit of DW who exit during theQtr. 12 Older Youth 19-21 years old Measures9. Entered Employment Rate Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the first quarter after exit of older youth OY who have entered employment by the end of the 1st Qtr. after exit of OY who exit during the Qtr. 10. Employment Retention Rate Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit of OY who are employed in 3rd Qtr. after exit of OY who exit during the Qtr. 11. Earnings Change in Six Months Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit Total Post-Program Earnings earnings in Qtr 2 Qtr 3 after exit - Pre-Program Earnings earnings in Qtrs 2 3 prior to registration of OY who exit during the quarter 12. Credential Rate of OY who were in employment post-secondary education or advanced training in the first Qtr. after exit and received a credential by the end of 3rd Qtr. after exit of OY who exit during the Qtr. Across Funding Streams16. Participant Customer Satisfaction The weighted average of participant ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale. Younger Youth 14-18 years old Measures13. Skill Attainment Rate Of all in-school youth and any out-of-school youth assessed to be in need of basic skills work readiness skills and or occupational skills Total of attained basic skills of attained WR skills of attained Occ. skills Total of basic skills goals of WR skills goals of Occ. skills goals 14. Diploma or Equivalent Attainment Rate Of those who register without a diploma or equivalent of younger youth YY who attained a secondary school diploma or equivalent by the end of the 1st Qtr. after exit of YY who exit during the Qtr. except those still in secondary school at exit 15. Retention Rate of younger youth found in one of the following categories in the 3rd Qtr. after exit post secondary education advanced training employment military service qualified apprenticeships of YY who exit during the Qtr. except those still in secondary school at exit 13 17. Employer Customer Satisfaction The weighted average of employer ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale. 14 Attachment B Common Approach for Establishing Negotiated Levels of Performance 1. General Instructions States are requested to calculate State averages of past outcomes on the performance indicators where possible. These averages are to be calculated according to the following specifications CSource data SJTPA data for individuals who terminated from JTPA between October 1 1997 to September 30 1998. This time period was chosen so that all states would have access to the necessary wage record data. Use of a common time period assures consistency among States. SAdult measures should be based on terminees from Title II-A. SDislocated worker measures should be based on terminees from Title III including substate and Governor s Reserve programs. SOlder youth measures should be based on terminees from Title II-C who were age 19 or older on the date of application. SYounger youth measures should be based on terminees from Title II-C who were under age 19 on the date of application. SWage records data will be required for most measures. Wage record data should be acquired for CThe second and third quarters before application for adults and older youth CThe second and third quarters before the date of dislocation for dislocated workers. NOTE if the dislocation date is not available use the application date instead and CThe first second and third quarters after termination for adults dislocated workers and older youth. CExclusions CIndividuals who received only objective assessment should be excluded from all calculations. CIndividuals terminated with other termination codes for health medical or institutionalized should be excluded from all calculations. This includes individuals whose other-termination code is institutionalized or health medical codes 1 or 2 in SPIR Item 39 Other terminations . CDislocated workers served exclusively with National Reserve Funds should be excluded from all calculations CEntered employment measures for adults and older youth exclude individuals employed at application to JTPA. This data should be based on information collected from the participant and normally included in the JTPA data system in the field labor force status. It should not be based on wage records. CInclusions CIndividuals served by 5 -incentive funded programs are to be included even though they were exempt from JTPA performance standards. Detailed specifications for calculating measures are to be found in TEGL 7-99 Core and Customer Satisfaction Measures for the Workforce Investment System. 152. Matrix showing how projected averages are derived The column labeled Projected Average presents projected average values for WIA performance indicators based on available data. The column labeled Explanation ... describes how the projected averages were calculated. The column labeled How to ... describes how to calculate a comparable past value for the State based on JTPA data or indicates other data that might be useful for the negotiation process. Indicators of PerformanceProjec ted Avera geExplanation of the method used for determining the average How to determine a State average using JTPA data A d u l t sEntered employment71 CBased on wage records for 7 States. Adjusted upward by 1 percentage point to account for differences between the 7 States and all States.CExclude individuals employed at application to JTPA. CCalculated as the percentage with positive earnings in the wage records in the quarter after termination. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Retention78 CBased on wage records for 7 States. Adjusted downward by 0.5 percentage point to account for differences between the 7 States and all States.CBased on individuals employed positive earnings in the wage records in the quarter after termination. CCalculated as the percentage with positive earnings in the wage records in the third quarter after termination. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Earnings change 3 700C Based on wage records for 7 States. CBased on individuals employed positive earnings in the wage records in the quarter after termination. CCalculated as the average of total earnings in the second and third quarters after termination minus total earnings in the second and third quarters before application . CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Credential60 CAssumes that the definition of credential will be somewhat flexible and that about 80 of those employed in the quarter after exit will obtain a credential. CCannot be calculated from JTPA data. CStates should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. 16D i s l o c a t e d W o r k e r sEntered employment77 CBased on wage records for 7 States. CAdjusted upward by 3 percentage points to account for differences between the 7 States and all States.CBased on all dislocated workers except NRA . CCalculated as the percentage with positive earnings in the wage records in the quarter after termination. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Retention85 CBased on wage records for 7 States. CAdjusted downward by 1 percentage point to account for differences between the 7 States and all States.CBased on individuals employed positive earnings in the wage records in the quarter after termination. Calculated as the percentage with positive earnings in the wage records in the third quarter after termination. See TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Earnings replacement92 CBased on wage records for 5 States. Two of the 7 available States had extremely high replacement rates and were excluded from the average.CBased on individuals employed positive earnings in the wage records in the quarter after termination. CA. Calculate the sum over all applicable dislocated workers of total earnings in the second and third quarters after termination CB. Calculate the sum of total earnings in the second and third quarters before application. CThe measure is A divided by B. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Credential60 CAssumes that the definition of credential will be somewhat flexible and that about 80 of those employed in the quarter after exit will obtain a credential. CCannot be calculated from JTPA data. CStates should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. 17O l d e r Y o u t h 1 9 - 2 1Entered employment63 CBased on wage records for 7 States. Adjusted downward by 3 percentage points to account for differences between the 7 States and all States.CExclude individuals employed at application to JTPA. CExclude individuals who entered non-Title-II training and do not have earnings in wage records for the quarter after termination. CCalculated as the percentage with positive earnings in the wage records in the quarter after termination. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Retention77 CBased on wage records for 7 StatesCBased on individuals employed positive earnings in the wage records in the quarter after termination. CExclude individuals who entered non-Title-II training and do not have earnings in wage records for the third quarter after termination. CCalculated as the percentage with positive earnings in the wage records in the third quarter after termination. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Earnings change 3 150CBased on wage records for 7 StatesCBased on individuals employed positive earnings in the wage records in the quarter after termination. CExclude individuals who entered non-Title-II training and do not have earnings in wage records for the third quarter after termination. CCalculated as the average of total earnings in the second and third quarters after termination minus total earnings in the second and third quarters before application . CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Credential50 CAssumes that the definition of credential will be somewhat flexible and that about 70 of those employed in the quarter after exit will obtain a credential. CValues are lower than for adults and dislocated workers because the employment rate is lower. CCannot be calculated from JTPA data. CStates should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. 18Y o u n g e r Y o u t h 1 8 o r l e s sGoal Attainment72 CCalculated based on PY 97 SPIR data on the attainment of youth employment competencies. It is the total number of competencies up to 3 per youth divided by the number of types of training received up to 3 per youth but at least equal to competencies attained . The value was lower in PY 98 66 . CBecause the values likely to be achieved by the States and locals depend on the difficulty of achieving the goals set under WIA it is not possible to calculate useful State values using JTPA data. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Diploma attainment55 CBased on separate estimates for in-school youth and for dropouts. CFor in-school youth a rate of 75 is used. This is the percentage of youth in low-income families bottom 20 of family incomes nationwide who complete high school which was derived from the Current Population Survey October 1996. CFor dropouts JTPA data on completion of major level of education was used approximately 21 for dropouts under 19 at entry but 17 or older at exit. CThese two numbers were combined using JTPA data on the split between in- school youth and dropouts.CCannot be calculated from JTPA data. CStates may provide data on SThe percentage of all low income youth who complete high school in the State the national average is about 75 . SThe percentage of JTPA youth who completed a major level of education among those who were school dropouts at application the national average is about 21 . Exclude youth under 17 at termination to approximate the exclusion of youth still attending school. SExpected relative levels of service to in- school youth and dropouts. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Retention54 CThe percentage of youth 18 and under at entry and 17 and older at exit who entered employment or non-Title-II training advanced training or postsecondary education derived from PY 97 SPIR data.A rough approximation can be calculated from JTPA data CInclude all youth under age 19 at application and age 17 or over at termination. CMeasure as the percentage who enter employment or enter non-Title-II training. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. 19C u s t o m e r S a t I s f a c t I o nParticipants68 CBased on an analysis of data from C Enterprise members 76 CA comparison group of non-Enterprise members 74 . CEarly data from early implementation States 65 - 70 that show that State level results tend to be lower than the SDA SSA-level enterprise results. CLower results than the Enterprise surveys are expected when youth are added to the sample.CResults of similar customer satisfaction surveys conducted by the State. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. Employers66 CEarly data from early implementation States suggest that employer results are likely to be 1 to 4 percentage points lower than for participants. CResults of similar customer satisfaction surveys conducted by the State. CSee TEGL 7-99 Core and Customer Satisfaction Performance Measures for the Workforce Investment System for instructions on how to calculate performance on the measure. CBeyond the State average States may provide additional rationale such as unique economic conditions program designs or participant characteristics along with valid data supporting the State s position of increasing or decreasing their level of performance. The specifications outlined in the matrix are based on the following correspondence between WIA and JTPA data items WIA ItemClosest Corresponding JTPA Item Registration dateApplication date AdultServed by Title II-A Dislocated workerServed by Title III Younger youthAge at application is under 19 and served by Title II-C Older youthAge at application is 19 or older and served by Title II-C Employed at registrationLabor force status at application employed Entered advanced training or entered postsecondary educationEntered non-Title-II training. 20 Attachment C Summary of Important Terms Performance Measures The 17 performance indicators required by the Workforce Investment Act of 1998 section 136 Interim Final Rule 20 CFR part 666 published at 64 Federal Register 18862 April 15 1999 to be used for determining return on investment and to be used in the incentives and sanctions process. Program Area A cluster of measures used in the evaluation of performance for incentive purposes. There are four program areas Adults Dislocated Workers Youth both older and younger youth and customer satisfaction even though it is not technically a program . Expected Performance Level The target performance level a State offers as the first step in the negotiation process. Negotiated Performance Level The numeric performance target agreed to by the State and the region for each of the 17 core performance indicators. Negotiation Factor Any information which may affect a State s expected performance on any of the core performance measures that can be used in the negotiation process. The relevance validity and source of this information will be assessed during the negotiation process. Cumulative Program Area Score The aggregate amount by which States exceed or fall below the negotiated performance levels in a particular program area. Incentive Threshold To be eligible for incentives a State must achieve at least 80 of the target performance levels on each performance measure and have cumulative program scores greater than 100 in each program area. Sanction Threshold States must achieve at least 80 of the target performance level on each of the 17 core indicators to avoid sanctions. To be subject to monetary sanctions a State s performance must fall below 80 of the target performance level on the same measure two consecutive years.