Supplemental Information for the 4th Quarter after Exit Indicators in PY 2020-2021

At the time of PY 2020-2021 negotiations, there are three full years (PY 2016 -2018) of outcome data for the Measurable Skill Gains (MSG) indicator, two full years (PY 2017-2018) of outcome data for the 2nd Quarter after Exit indicators (i.e., Employment Rate 2nd Quarter after Exit and Median Earnings 2nd Quarter after Exit), and one full year (PY 2018) of outcome data for the 4th Quarter after Exit indicators (i.e., Employment Rate 4th Quarter after Exit and Credential Attainment Rate). Since there are at least two full years of data available for the MSG and 2nd Quarter after Exit indicators, a WIOA statistical adjustment model was developed for these indicators. Model estimates from these models will be fully implemented as a factor in PY 2020-2021 negotiations and in assessing PY 2020-2021 performance. WIOA statistical adjustment models could not be developed for the 4th Quarter after Exit indicators since there is only one full year of data available.

In negotiations for previous WIOA program years (i.e., PY 2017-2019), a proxy statistical adjustment model based on WIA data was used for all applicable indicators. The proxy models produced estimated performance outcomes to be used as a factor in negotiations and generated model estimates in order to assess performance after adjusting the negotiated levels of performance based on the actual characteristics of the participants served and economic conditions for the states during the program year. The proxy models used available WIA data from PY 2005-2015. Continued use of this proxy model was considered for the 4th Quarter after Exit indicators in PY2020-2021, however, it was determined that these proxy models were no longer reliable for use in informing negotiations and assessing performance.

Since there is not a usable statistical adjustment model for the 4th Quarter after Exit indicators, there will not be estimated outcomes produced by the statistical adjustment models to be used as a factor in negotiations for PY 2020-2021. Instead, negotiated levels of performance for these indicators should be reached using the available data on the other three factors of negotiations. These factors broadly account for how state performance levels compare across states, the degree in which there is continuous improvement, and how a state contributes to national performance goals. The details of all the negotiation factors are explained in Training and Employment Guidance Letter No. 11-19.

This supplement provides the following additional information which can be used in the PY 2020-2021 Negotiations for the 4th Quarter after Exit indicators:

• Q4 Indicator Data - this section has data on the 4th Quarter after Exit indicators. The data provided can inform the negotiation factors which will be used to reach a negotiated level of performance.
• WIA Proxy Model Analysis for Q4 Indicators - this section has additional details on the proxy statistical adjustment models for the 4th Quarter after Exit indicators. It explains where the most recent model estimates are located and provides information on the why model predictions are not being used as a factor for reaching negotiated levels of performance in PY 2020-2021.

(click on the tabs in the top bar to view the other sections)

WIA Proxy Model Analysis for the 4th Quarter after Exit Indicators

As discussed in the Overview section, the WIA proxy models were considered for continued use for the 4th Quarter after Exit indicators in PY 2020-2021 negotiations. Doing so would carry forward past practices and provide an estimated target as one of the factors for negotiations. However, there were some concerns about using this approach. First, it would add complexity as the other indicators (i.e., Measurable Skill Gains, Employment Rate 2nd Quarter after Exit, and Median Earnings 2nd Quarter after Exit) will all be using new WIOA models in PY 2020-2021. Second, it wasn’t certain that a model developed using proxy WIA data from PY 2005-2015 would have validity in predicting PY 2020 performance levels. As result, the feasibility of continuing the use of the WIA proxy models was tested.

The predictions shown in the plot below are the predictions that the proxy WIA model would have produced for the PY 2020-2021 negotiations. They are estimated using the WIA proxy model parameter estimates and the most recent participant characteristic and economic conditions data (i.e., PY 2018). These predictions are compared against the actual performance of states in PY 2018. The black line is where the points would be if the models predicted perfectly (i.e., the prediction matched the actual result).

As the plot shows, the proxy WIA model predictions are not very accurate for PY 2018. Almost half of the Employment Rate 4th Quarter after Exit predictions missed by more than 5% and in some of the models the residuals (i.e., the difference between the prediction and actual results) are not evenly distributed. The Credential Attainment predictions are worse with almost half missing by more than 10%. Those models also resulted in unevenly distributed residuals and a significant number of predictions that missed by more than 20%. These data suggest that the proxy WIA models for the 4th Quarter after Exit indicators are not well specified and would not provide useful predictions for the PY 2020-2021 negotiations. In fact, the predictions for PY 2020-2021 would almost certainly be much worse since the PY 2018 predictions were conducted using actual state PY 2018 participant characteristics and economic conditions data. However, the proxy models for these indicators are still useful for assessing PY 2018-2019 performance since that process only accounts for the difference between the pre and post PY predictions (not the level of the prediction) and the purpose of those assessments are for identifying technical assistance needs and improving performance accountability processes.

While the proxy WIA models will not be carried forward for PY 2020-2021, those model estimates can be obtained via PY 2018-2019 performance products. The full WIA proxy model estimates (i.e., variable coefficients and state fixed effects) that have been used previously for all 4th Quarter after Exit indicators are in the PY 2018-2019 WIOA Performance Negotiation Tool. The proxy model estimates for these indicators are also in the PY 2018 Performance Assessment materials and will be in the PY 2019 Performance Assessment materials.

PY 2018 Predicted Peformance Compared to Actual Performance

The plot below shows show predicted performance in PY 2018 compared to the actual PY 2018 performance using the WIA proxy model. Select the program and indicator check-boxes to filter the plot to the specific individual indicator of interest. (note: hover over a point to see state specific data)