TEGL11-01Attach1.pdf

ETA Advisory File Text

Attachment I Relating the Three Conditions for Revising Levels of Negotiated Performance Background Section 136 b 3 A vi of the Workforce Investment Act WIA specifies that a Governor may request revisions to state negotiated levels of performance in the event unanticipated circumstances arise in a state resulting in a significant change in the factors that include factors described in section 136 b 3 A iv II of the Act. Additionally 20 CFR 666.130 a states the change in factors is to be determined from when the initial plan was submitted and approved. These factors are widely accepted variables known to impact resulting outcomes on one or more of the 17 WIA performance measures. These factors include differences in economic conditions the characteristics of participants and the services provided. There are three conditions for any state workforce agency request for revisions to state negotiated levels of performance to be approved by the ETA Regional Office. These conditions are graphically displayed in a logic model below. Relating the Three Conditions Condition 1 Unanticipated Circumstances Condition 3 Demonstrated Change in the Factor s The process for requesting revisions to earlier negotiated levels of performance is triggered by an unanticipated circumstance. The unanticipated circumstance must occur for a request to be considered by the Secretary. Condition 2 Unanticipated Circumstance Occurs Factor s Known to Impact Performance and Revised Levels can be Objectively Determined The unanticipated event must impact one or more of the factors taken into consideration in the earlier negotiations of performance levels. It must be reasonable to associate the event with a change in the factor s . The change in the factor s in turn must be related to performance on one or more of the measures. The degree of the impact on performance may be estimated using historical data. Condition 1 2 Actual performance on the 17 WIA title I measures of performance varies above or below expected levels of performance for a number of reasons. Variations from expected outcomes can also result from significant changes in factors that are beyond the control of the state workforce agency. In these situations state workforce agencies should consider submitting requests to revise negotiated levels of performance impacted by the significant change in factors. Variations from expected outcomes can occur from significant changes in administrative practices within the control of the state workforce agency. Changes in expected performance under these circumstances do not merit revisions to negotiated levels of performance. By themselves these changes would not be considered the result of unanticipated circumstances. Examples of unanticipated circumstances are listed below. Please note the list of unanticipated circumstances is not exhaustive and does not constrain a state workforce agency from documenting its own experiences that have prompted the need to seek a revision. Limitations in the baseline data used to project performance levels for program years 2001 and 2002. The baseline data used in the initial negotiations of these performance levels were based on actual and estimated outcomes attained by former JTPA participants. The data sources and collection methods may be very different than those found under WIA warranting possible corrections to baseline data. Changes in the characteristics of program participants such as service to hard-to-serve customer groups not taken into account in the earlier negotiation s on state levels of performance. Changes in economic conditions such as plant closings and mass layoffs. Changes in economic assumptions and outlooks. Disasters such as floods earthquakes hurricanes tornadoes and acts of war. Significant changes in federal funds allotted to the state to implement WIA. Unanticipated legal or legislative actions that increase the quality of participant outcomes and as a result increase 3 the level of effort needed by participants to attain these outcomes. For example a new state law may impose higher standards for high school graduation. This action would increase the standard needed to attain a diploma or equivalent and could decrease the numbers of graduates. Condition 2 Demonstrated Change in the Factor s The unanticipated circumstance by itself does not provide sufficient information to justify a revision in state negotiated levels of performance. There should be sufficient evidence to demonstrate change in one or more factors initially considered when reaching agreement on the state negotiated levels of performance. Those factors might have been an annual unemployment rate of 5.4 a baseline adult entered employment rate of 72 or 53 of the adult participants demonstrating one or more significant barriers to employment. It is worthwhile to keep in mind that an unanticipated circumstance may impact performance measures covering multiple program years. Condition 3 Factor s Known to Impact Performance and Revised Levels can be Objectively Determined Because unanticipated circumstances are by definition outside the normal operating conditions there can be no predetermined numerical guidelines within which negotiated levels of performance might be revised. However any justification for revising performance levels should be based on generally acceptable approaches and data sources. A. Guidelines for Approaches and Data Sources The methods used to determine revisions to negotiated levels of performance must - Adhere to widely accepted statistical practices including predictive or forecasting techniques where appropriate. - Demonstrate a reasonable cause and effect relationship between one or more factors and performance on a measure. - Be fair objective and yield quantifiable results. - Support and not undermine state efforts in achieving continuous improvement of workforce investment activities. The source data must be 4 - Developed by 1 a federal state or local governmental agency or 2 some other reputable source such as the state s MIS unit a university or a private research foundation. - Gathered according to acceptable data collection techniques. - Compiled according to widely accepted analytical procedures. The factors used to determine revisions to negotiated levels of performance include - Differences in economic conditions. - The characteristics of participants at the time of registration. - Services to be provided to participants. B. Using Predictive Statistical Models to Estimate Revised Levels State workforce agencies may develop their own statistical forecasting models using the multiple regression approach or use one or more other statistical methods used to predict selected outcomes. The multiple regression statistical method is a widely used technique that determines the relationship between a selected performance outcome and multiple explanatory factors or variables. While state workforce agencies may develop their own statistical models ETA recognizes that many states do not have the statistical expertise or the necessary resources to develop these models. In order to assist in determining the degree of relationship between the factors and each of the performance measures ETA will work with the states to develop technical guidance and optional predictive statistical models using multiple regression analysis for use in the future. C. Determining Revised Performance Levels in the Absence of Predictive Statistical Models While deriving revised levels of performance from predictive statistical models is a preferred approach by many state practitioners the use of these models may not be feasible because of a lack of national or state historical data on WIA participants. This is especially true in the first few years of the operation of a 5 new program such as those funded under WIA. In the absence of representative state and or national historical data the weighted average approach should be used to determine revisions to state negotiated levels of performance. The weighted average approach is described in Attachment II. D. The Importance of Negotiation in Revising State Levels of Performance Under certain circumstances the predictive statistical models may yield levels of performance that are unrealistic. Imprecision in the models extreme factor values and the occurrence of rare isolated economic changes such as those due to natural disasters are examples of conditions that may yield unrealistically high or low levels of expected performance. In these and similar situations the proposed revised levels of performance should be negotiated between the ETA Regional Office and state staff to ensure reasonable and appropriate levels are set. The expectation for performance is derived from state experience with similar unanticipated circumstances or suggested by research studies. However the revised level should not be based solely on past performance or experience under similar circumstances. The revised level of performance should reflect what is ideally attainable given the change in the environment and any needed changes to the delivery of services. Since environmental conditions are likely to be unique for each unanticipated circumstance arising in a state defining an appropriate performance level will likely be somewhat subjective. In reaching agreement on a definition for exemplary performance the ETA Regional Office and state staff should consider the following questions What is the unanticipated circumstance What are the changes in the factors considered in formulating the original state negotiated levels of performance Who are the participants impacted by the changes in the factors i.e. population group What is the estimated performance impact of the changes in the factors on the population group What changes can be feasibly made by the state to the design and delivery of services to address the impact of the unanticipated circumstance What programmatic changes are 6 being proposed by the state Given the changes in the factors what level of performance is estimated to occur - without changes to the design and delivery of services to the target group - with feasible changes to the design and delivery of services to address changes in the factors caused by the unanticipated circumstance What level of performance does the state cite for the balance of exiters not included in the population groups impacted by the unanticipated circumstance Is this level different than the initial state negotiated level of performance for the measure Why Is the level reasonable and appropriate What level of performance does the state suggest as appropriate for the measure Why What level of performance does the ETA Regional Office cite as ideally attainable Why