Negotiations Supplement for the Q4 Indicators

Supplemental Information for the 4th Quarter after Exit Indicators in PY 2020-2021
This supplement provides additional information to be used in the PY 2020-2021 negotiations for the 4th quarter after exit indicators.

At the time of PY 2020-2021 negotiations, there are three full years (PY 2016 -2018) of outcome data for the Measurable Skill Gains (MSG) indicator, two full years (PY 2017-2018) of outcome data for the 2nd Quarter after Exit indicators (i.e., Employment Rate 2nd Quarter after Exit and Median Earnings 2nd Quarter after Exit), and one full year (PY 2018) of outcome data for the 4th Quarter after Exit indicators (i.e., Employment Rate 4th Quarter after Exit and Credential Attainment Rate). Since there are at least two full years of data available for the MSG and 2nd Quarter after Exit indicators, a WIOA statistical adjustment model was developed for these indicators. Model estimates from these models will be fully implemented as a factor in PY 2020-2021 negotiations and in assessing PY 2020-2021 performance. WIOA statistical adjustment models could not be developed for the 4th Quarter after Exit indicators since there is only one full year of data available.

In negotiations for previous WIOA program years (i.e., PY 2017-2019), a proxy statistical adjustment model based on WIA data was used for all applicable indicators. The proxy models produced estimated performance outcomes to be used as a factor in negotiations and generated model estimates in order to assess performance after adjusting the negotiated levels of performance based on the actual characteristics of the participants served and economic conditions for the states during the program year. The proxy models used available WIA data from PY 2005-2015. Continued use of this proxy model was considered for the 4th Quarter after Exit indicators in PY2020-2021, however, it was determined that these proxy models were no longer reliable for use in informing negotiations and assessing performance.

Since there is not a usable statistical adjustment model for the 4th Quarter after Exit indicators, there will not be estimated outcomes produced by the statistical adjustment models to be used as a factor in negotiations for PY 2020-2021. Instead, negotiated levels of performance for these indicators should be reached using the available data on the other three factors of negotiations. These factors broadly account for how state performance levels compare across states, the degree in which there is continuous improvement, and how a state contributes to national performance goals. The details of all the negotiation factors are explained in Training and Employment Guidance Letter No. 11-19.

This supplement provides the following additional information which can be used in the PY 2020-2021 Negotiations for the 4th Quarter after Exit indicators:

  • Q4 Indicator Data - this section has data on the 4th Quarter after Exit indicators. The data provided can inform the negotiation factors which will be used to reach a negotiated level of performance.
  • WIA Proxy Model Analysis for Q4 Indicators - this section has additional details on the proxy statistical adjustment models for the 4th Quarter after Exit indicators. It explains where the most recent model estimates are located and provides information on the why model predictions are not being used as a factor for reaching negotiated levels of performance in PY 2020-2021.

PY 2020-2022 Performance Negotiations

  1. PY 2020-2021 Negotiations Tool

    a. WIOA Negotiation Tool PY 2020-2021

    b. This tool aims to aid in the negotiation process of setting negotiated levels of performance for program years 2020 and 2021. It provides users with benchmarks provided by the statistical adjustment model (SAM)--as required by WIOA--along with an analysis of historical performance that shows both actual levels of performance as well as historic WIA performance outcomes viewed through a WIOA lens.

  2. PY 2020-2021 Model Selection Paper

    a. Overview of the Development of the Statistical Adjustment Models

    This document provides an overview of the statistical analysis that was conducted in developing and evaluating the statistical adjustment models that will be used for the 2020 and 2021 program years (PY). Beginning with PY 2020, the statistical models will be used as a factor in performance negotiations and to assess state performance after each program year in accordance with WIOA performance accountability provisions for the following performance indicators and WIOA title I and title III programs: Employment Rate 2nd Quarter after Exit (Q2ER), Median Earnings 2nd Quarter after Exit (ME), and Measurable Skill Gains Rate (MSG) for the Adult, Dislocated Worker, Youth, and Wagner-Peyser programs (note: the MSG indicator does not apply to the Wagner-Peyser program).

    Previously, the statistical adjustment models underwent several rounds of development. The initial methodology of the statistical adjustment models was developed by the U.S. Department of Labor's Chief Evaluation Office and is explained in the WIOA Statistical Adjustment Model Methodology Report. The models recommended at that time were implemented as a preliminary test beginning with PY 2017 using WIA data and proxy variables where necessary. Subsequently, the model specifications (i.e., which variables were included in the models) changed slightly beginning in PY 2018 as a result of stakeholder feedback and ongoing statistical analysis by ETA. However, through PY 2019 the models had a limited role being used as a proxy method of assessing performance because of the structural differences in data collection between WIA and WIOA.

    Beginning with PY 2020, there is sufficient WIOA data to generate usable model estimates for the Q2ER, ME, and MSG indicators. As a result, the Q2ER and ME models have been refined and the MSG models have been developed. The changes to the model were largely limited to the selection of explanatory variables, the logic regarding how some variables were calculated, and determinations on the data used. The MSG indicator is a new performance indicator under WIOA (collection and reporting of this indicator began in PY 2016), so the models for this indicator are newly developed yet they follow the same basic framework as the previous models.

    The following analysis is centered on three main sections that are the focus for this model development period. The sections include:

    • WIOA vs. WIA model performance - how model performance is affected by the data
    • Refinement of current model specifications - assessing the Q2ER and ME models
    • MSG model development

    This report also lists the final model specifications, resulting model estimates, and model predictions that will be used as a factor in the negotiation of PY 2020-2021 performance levels.

    b. This document provides an overview of the statistical analysis that was conducted in developing and evaluating the statistical adjustment models that will be used for the 2020 and 2021 program years (PY). Beginning with PY 2020, the statistical models will be used as a factor in performance negotiations and to assess state performance after each program year in accordance with WIOA performance accountability provisions for the following performance indicators and WIOA title I and title III programs: Employment Rate 2nd Quarter after Exit (Q2ER), Median Earnings 2nd Quarter after Exit (ME), and Measurable Skill Gains Rate (MSG) for the Adult, Dislocated Worker, Youth, and Wagner-Peyser programs (note: the MSG indicator does not apply to the Wagner-Peyser program).

  3. PY 2020-2021 State Model Summaries (each region has a compressed folder with all of the state summaries inside)

    a. These are state-specific summaries of the key elements derived from the statistical adjustment models developed for PY 2020-2021. For each individual performance indicator there are plots that show how the actual level of performance for the specific state in PY 2018 compared to all states and how the predicted level of performance (i.e., Estimate0) for the specific state in PY 2020 compares to the predicted levels for all states. There are also tables that give all the relevant model estimates and pre-PY 2020 data for all of the model variables. In addition, the last tab has a table that identifies all the variables included in each individual indicator model.

    b. Region 1

    c. Region 2

    d. Region 3

    e. Region 4

    f. Region 5

    g. Region 6

WIOA NEGOTIATION GUIDANCE

WIOA STATISTICAL MODEL METHODOLOGY REPORT

  • This document presents CEO's proposed methodology for setting WIOA performance targets. It begins with background information on WIOA and its requirements on the statistical adjustment model, followed by a brief description of previous methodologies used by The Department of Labor for establishing performance outcome goals.
  • WIOA Statistical Model Methodology Report: Pdf | Docx

WIOA NEGOTIATED INDICATORS OF PERFORMANCE

The following chart lists the specific primary indicators to be negotiated for PY 2016 and PY 2017.

Department/Agency Applicable Statutory Citations Primary Indicators to be Negotiated for PY 2016 and PY 2017
DOL/ETA Title I, 29 U.S.C. secs. 3101 et seq.
  • Employment in the 2nd Quarter After Exit (the title I Youth indicator is education or training activities or employment in the 2nd quarter after exit)
  • Employment in the 4th Quarter After Exit (the title I Youth indicator is education or training activities or employment in the 4th quarter after exit)
  • Median Earnings in the 2nd Quarter After Exit (For title I Adults and Dislocated Workers only)
  • Credential Attainment Rate
DOL/ETA Wagner Peyser Employment Service program, as amended by title III, 29 U.S.C. secs. 49 et seq.
  • Employment in the 2nd Quarter After Exit
  • Employment in the 4th Quarter After Exit
  • Median Earnings in the 2nd Quarter After Exit
ED/OCTAE Title II, 29 USC secs. 3271 et seq.
  • Measurable Skill Gains
ED/RSA Vocational Rehabilitation Program, as amended by title IV, 29 USC secs. 701 et seq.
  • N/A, RSA will not negotiate PY 2016 or PY 2017 State targets levels for primary performance indicators.

ECONOMIC DATA RESOURCES

WIOA State Negotiated Levels of Performance

Each year, states strive to achieve performance goals for each measure. WIOA Negotiated Indicators of Performance.