Supplemental Information for the 4th Quarter after Exit Indicators in PY 2020-2021
This supplement provides additional information to be used in the PY 2020-2021 negotiations for the 4th quarter after exit indicators.
At the time of PY 2020-2021 negotiations, there are three full years (PY 2016 -2018) of outcome data for the Measurable Skill Gains (MSG) indicator, two full years (PY 2017-2018) of outcome data for the 2nd Quarter after Exit indicators (i.e., Employment Rate 2nd Quarter after Exit and Median Earnings 2nd Quarter after Exit), and one full year (PY 2018) of outcome data for the 4th Quarter after Exit indicators (i.e., Employment Rate 4th Quarter after Exit and Credential Attainment Rate). Since there are at least two full years of data available for the MSG and 2nd Quarter after Exit indicators, a WIOA statistical adjustment model was developed for these indicators. Model estimates from these models will be fully implemented as a factor in PY 2020-2021 negotiations and in assessing PY 2020-2021 performance. WIOA statistical adjustment models could not be developed for the 4th Quarter after Exit indicators since there is only one full year of data available.
In negotiations for previous WIOA program years (i.e., PY 2017-2019), a proxy statistical adjustment model based on WIA data was used for all applicable indicators. The proxy models produced estimated performance outcomes to be used as a factor in negotiations and generated model estimates in order to assess performance after adjusting the negotiated levels of performance based on the actual characteristics of the participants served and economic conditions for the states during the program year. The proxy models used available WIA data from PY 2005-2015. Continued use of this proxy model was considered for the 4th Quarter after Exit indicators in PY2020-2021, however, it was determined that these proxy models were no longer reliable for use in informing negotiations and assessing performance.
Since there is not a usable statistical adjustment model for the 4th Quarter after Exit indicators, there will not be estimated outcomes produced by the statistical adjustment models to be used as a factor in negotiations for PY 2020-2021. Instead, negotiated levels of performance for these indicators should be reached using the available data on the other three factors of negotiations. These factors broadly account for how state performance levels compare across states, the degree in which there is continuous improvement, and how a state contributes to national performance goals. The details of all the negotiation factors are explained in Training and Employment Guidance Letter No. 11-19.
This supplement provides the following additional information which can be used in the PY 2020-2021 Negotiations for the 4th Quarter after Exit indicators:
- Q4 Indicator Data - this section has data on the 4th Quarter after Exit indicators. The data provided can inform the negotiation factors which will be used to reach a negotiated level of performance.
- WIA Proxy Model Analysis for Q4 Indicators - this section has additional details on the proxy statistical adjustment models for the 4th Quarter after Exit indicators. It explains where the most recent model estimates are located and provides information on the why model predictions are not being used as a factor for reaching negotiated levels of performance in PY 2020-2021.
PY 2020-2022 Performance Negotiations
PY 2020-2021 Negotiations Tool
b. This tool aims to aid in the negotiation process of setting negotiated levels of performance for program years 2020 and 2021. It provides users with benchmarks provided by the statistical adjustment model (SAM)--as required by WIOA--along with an analysis of historical performance that shows both actual levels of performance as well as historic WIA performance outcomes viewed through a WIOA lens.
PY 2020-2021 Model Selection Paper
a. Overview of the Development of the Statistical Adjustment Models
This document provides an overview of the statistical analysis that was conducted in developing and evaluating the statistical adjustment models that will be used for the 2020 and 2021 program years (PY). Beginning with PY 2020, the statistical models will be used as a factor in performance negotiations and to assess state performance after each program year in accordance with WIOA performance accountability provisions for the following performance indicators and WIOA title I and title III programs: Employment Rate 2nd Quarter after Exit (Q2ER), Median Earnings 2nd Quarter after Exit (ME), and Measurable Skill Gains Rate (MSG) for the Adult, Dislocated Worker, Youth, and Wagner-Peyser programs (note: the MSG indicator does not apply to the Wagner-Peyser program).
Previously, the statistical adjustment models underwent several rounds of development. The initial methodology of the statistical adjustment models was developed by the U.S. Department of Labor's Chief Evaluation Office and is explained in the WIOA Statistical Adjustment Model Methodology Report. The models recommended at that time were implemented as a preliminary test beginning with PY 2017 using WIA data and proxy variables where necessary. Subsequently, the model specifications (i.e., which variables were included in the models) changed slightly beginning in PY 2018 as a result of stakeholder feedback and ongoing statistical analysis by ETA. However, through PY 2019 the models had a limited role being used as a proxy method of assessing performance because of the structural differences in data collection between WIA and WIOA.
Beginning with PY 2020, there is sufficient WIOA data to generate usable model estimates for the Q2ER, ME, and MSG indicators. As a result, the Q2ER and ME models have been refined and the MSG models have been developed. The changes to the model were largely limited to the selection of explanatory variables, the logic regarding how some variables were calculated, and determinations on the data used. The MSG indicator is a new performance indicator under WIOA (collection and reporting of this indicator began in PY 2016), so the models for this indicator are newly developed yet they follow the same basic framework as the previous models.
The following analysis is centered on three main sections that are the focus for this model development period. The sections include:
- WIOA vs. WIA model performance - how model performance is affected by the data
- Refinement of current model specifications - assessing the Q2ER and ME models
- MSG model development
This report also lists the final model specifications, resulting model estimates, and model predictions that will be used as a factor in the negotiation of PY 2020-2021 performance levels.
b. This document provides an overview of the statistical analysis that was conducted in developing and evaluating the statistical adjustment models that will be used for the 2020 and 2021 program years (PY). Beginning with PY 2020, the statistical models will be used as a factor in performance negotiations and to assess state performance after each program year in accordance with WIOA performance accountability provisions for the following performance indicators and WIOA title I and title III programs: Employment Rate 2nd Quarter after Exit (Q2ER), Median Earnings 2nd Quarter after Exit (ME), and Measurable Skill Gains Rate (MSG) for the Adult, Dislocated Worker, Youth, and Wagner-Peyser programs (note: the MSG indicator does not apply to the Wagner-Peyser program).
PY 2020-2021 State Model Summaries (each region has a compressed folder with all of the state summaries inside)
a. These are state-specific summaries of the key elements derived from the statistical adjustment models developed for PY 2020-2021. For each individual performance indicator there are plots that show how the actual level of performance for the specific state in PY 2018 compared to all states and how the predicted level of performance (i.e., Estimate0) for the specific state in PY 2020 compares to the predicted levels for all states. There are also tables that give all the relevant model estimates and pre-PY 2020 data for all of the model variables. In addition, the last tab has a table that identifies all the variables included in each individual indicator model.
b. Region 1
c. Region 2
d. Region 3
e. Region 4
f. Region 5
g. Region 6
- February 6, 2020
TEGL 11-19 : Negotiations and Sanctions Guidance for the Workforce Innovation and Opportunity Act (WIOA) Core Programs. Through this guidance, the U.S. Departments of Labor and Education (Departments) delineate the process for negotiating levels of performance, as required by section 116(b)(3)(A)(iv) of the Workforce Innovation and Opportunity Act (WIOA). This guidance also provides explanation of the two instances in which a state may be sanctioned for performance failure or for failure to report (section 116(f) of WIOA). Section 116 of WIOA, its implementing regulations, and this guidance govern how the Departments will determine when it is necessary to sanction a state, which is statutorily required by WIOA.
- February 16, 2018
TEGL 9-17 : Negotiating Performance Goals for the Workforce Innovation and Opportunity Act (WIOA) Title I Programs and the Wagner-Peyser Act Employment Service as amended by Title III of WIOA, for Program Years (PYs) 2018 and PY 2019. This guidance informs States of the guidelines for the negotiation process for the WIOA title I programs (Adult, Dislocated Worker, and Youth) and the Wagner-Peyser Act Employment Service program, as amended by title III of WIOA, for PY 2018 and PY 2019. Once negotiated levels are agreed upon by each State and the U.S. Department of Labor (DOL), Employment and Training Administration (ETA), they will be incorporated into the State Unified or Combined Plan. This guidance also provides information on the negotiations process for the Adult Education and Family Literacy Act programs under title II of WIOA and the Vocational Rehabilitation program as amended by title IV of WIOA for PY 2018 and PY 2019.
- TEGL 26-15 : Negotiating Performance Goals for the Workforce Innovation and Opportunity Act (WIOA) Title I Programs and the Wagner-Peyser Employment Service as amended by Title III, for Program Year (PY) 2016 and PY 2017
- Attachment I: Recommended Timeline for the PY 2016 and PY 2017 Negotiations Process
- Attachment II: Executive Summary of the Statistical Adjustment Model
- Attachment III A: Note to Reviewer
- Attachment III B: Excel Spreadsheet Tool for the Negotiation | Federal and State Negotiations Tool
- Attachment IV: WIOA Operating Guidance TEGL References
- Attachment V: Data Tools for Identifying Characteristics of Participants and Economic Conditions
- Attachment VI: Negotiated Indicators of Performance for PY 2016 and PY 2017
- This document presents CEO's proposed methodology for setting WIOA performance targets. It begins with background information on WIOA and its requirements on the statistical adjustment model, followed by a brief description of previous methodologies used by The Department of Labor for establishing performance outcome goals.
- WIOA Statistical Model Methodology Report: Pdf | Docx
The following chart lists the specific primary indicators to be negotiated for PY 2016 and PY 2017.
|Department/Agency||Applicable Statutory Citations||Primary Indicators to be Negotiated for PY 2016 and PY 2017|
|DOL/ETA||Title I, 29 U.S.C. secs. 3101 et seq.||
|DOL/ETA||Wagner Peyser Employment Service program, as amended by title III, 29 U.S.C. secs. 49 et seq.||
|ED/OCTAE||Title II, 29 USC secs. 3271 et seq.||
|ED/RSA||Vocational Rehabilitation Program, as amended by title IV, 29 USC secs. 701 et seq.||
- Bureau of Labor Statistics
- U.S. Census Bureau
- An Environmental Scan - Workforce information sources to assist employment, economic development, education and workforce investment planning and decision making.
- Center for Economic Studies (CES)