TEGL_26-15-Attachment-II_Acc.pdf

ETA Advisory File Text

1 Attachment II Executive Summary Statistical Adjustment Model Methodology The Workforce Innovation and Opportunity Act WIOA section 116 Performance Accountability System requires the use of a statistical adjustment model when setting level s of performance. WIOA requires that levels of performance be negotiated for each of the primary indicators of performance at the State level. State -level actual performance outcomes are a functio n of a the characteristics of the participants being served as well as b the labor market conditions in which those participants are being served . WIOA specifically requires that factors of both types be accounted for and t he use of a statistical mod el when negotiating levels of performance is intended to account for variation in factors of both types . A properly specified statistical model will appropriately adjust performance goals for States serving harder -to-serve populations and or in economies f acing more diff icult labor market conditions. The statistical model objectively quantifies how and to what extent each of these factors affects l evels of performance i.e. actual outcomes. The goal of the statistical approach is to account for these fact ors and separate them from those factors that program administrators are able to control. The Department of Labor s Chief Evaluation Office CEO in collaboration with the Department s Employment and Training Administration ETA as well as the Department of Education s Office of Career Technical and Adult Education OCTAE and the Rehabilitative Services Administration RSA conducted extensive research and statistical analysis regarding the development of an appropriate statistical adjustm ent model. Additionally the Chief Evaluation Office ETA OCTAE and RSA consulted with workforce system professionals and external experts in the statistical and economics fields about the approach taken to develop the statistical model. The Chief Ev aluation Office and ETA conducted analyses using data from individual records of participants served by the Workforce Investment Act WIA title I -B and Wagner -Peyser WP title III programs. These records contain detailed information about each program participant s characteristics pro gram activities and outcomes. States have submit ted these records quarterly and each quarterly submission file contains the ten most recent quarters of information on all participants who received funded services during that time span. WIA records from Program Year PY 2005 July 1 2005 through June 30 2006 to PY 2014 July 1 2014 through June 30 2015 and WP records from PY 2012 July 1 2012 through June 30 2013 to PY 2014 were us ed to calculate outcomes for the WIOA performance indicators Employment Rate 2 nd quarter after exit Employment Rate 4 th quarter after exit and Median Earnings in the 2 nd quarter after exit for each year from 2005 to 2014. The Employment Rate 4 th quarter after exit for WP was estimated using proxy data from the 3 rd quarter after exit. The Credential Attainment Rate within 4 quarters after exit was estimated using proxy WIA data extending only to the 3 rd quarter after exit. The Youth Employment o r Placement in Education indicators for the 2 nd and 4 th quarters after exit were estimated using WIA data for the 1 st and 3 rd quarters after exit respectively. 2 The Department of Labor s Chief Evaluation Office has recommended that the statistical adjust ment model include all of the variables expected to explain changes in the performance outcomes i.e . explanatory variables as required by WIOA sec. 116 and specified in Tables 1 and 2 with a few exceptions. Certain variables that do not apply to Youth programs those in Table 1 that are not marked with an x in the Youth column also were removed from the Youth specific target estimation models. The variables for male exiters exiters with education beyond a bachelor s degree an d the economic variable for trade transportation and utility related employment also were omitted to avoid the loss of model precision that can occur when two or more explanatory variables are highly correlated to one another . The variable representing exiters who received training was also removed from Credential Attainment models for Adult Dislocated Worker and Youth programs on account of correlation with other Credential Attainment variables. The individual -level data were also aggregated to the St ate level on a quarterly basis and each variable is presented as the percent of total exiter s except for those representing youth education level pre -test scores and post -test scores which were expressed as average s. To produce targets for each State CEO recommended estimating t he coefficients for the participant characteristics also known as the impact each individual characteristic imposes on a given performance outcome and economic conditions using a fixed effects model . This type of model will a llow the Departments to estimate the program effect of each State that does not change over time in other words this is the fixed effect estimator for each S tate. The average State fixed effect wil l be used when projecting targets based on the participan t characteristics and economic conditions. Under this approach the targets reflect the outcome the State should have achieved after adjusting only for the measureable changes in the characteristics of exiters actually served during the program year as ca ptured by the explanatory variables and the actual condition of the local economies as measu red by the economic variables. The State fixed effects are treated as program specific effects that program administrators can largely control. Initial WIOA perfor mance targets those targets set prior to the beginning of the program year must be negotiated with consideration of the most recent available data at the time of model estimation. At the end of the program year the data from the initial model will be upd ated with the most current data to reflect the actual participant characteristics and economic conditions during that program year. The mo del will then yield new targets based on the updated data. This current initial model will be used in the negotiation process between ETA s regional offices and States to negotiate levels of performance for WIOA title I Adult Dislocated Worker and Youth programs and the title III Wagner -Peyser Employment Service for the following performance indicators 1 employment in the second quarter after exit 2 employment in the fourth quarter after exit 3 median earnings in the second quarter and 4 credential attainment rate. This statistical model also must be used by States to negot iate levels of performance with the local areas. Once States and grantees begin reporting on the WIOA primary indicators of performance the Departments of Labor and Education will use those outcomes to begin building and refining the statistical models for the remaining indicators. The model will continue to be refined with each set of data that is reported in addition to factoring in the economic conditions. 3 The tables below provide a description of each explanatory variable. As discussed WIOA require s the statistical adjustment model to account for variation in participant characteristics as well as local labor market conditions. Table 1 contains the descriptions of the explanatory variables based o n participant characteristics. Table 2 contains the i nformation on the economic variab les including unemployment rate and industrial structures employment level . All statistical adjustment modeling used the economic varia bles as explanatory variables. The data described in Table 2 were obtained from the B ureau of Labor Statistics 1. It is important to note that because the performance measures derived from the WIA data were not adjusted for seasonal changes the unemployment rate used here also is not seasonally adjusted . The non - seasonally adjusted unemplo yment rate is used to maintain consistency with the outcome data. The economic data are aligned with the characteristic data elements by State and time period . For example the unemployment rate for Alabama in the 2 nd quarter of calendar year 2013 is aligned with the characteristics of Alabama s exiters in the 2 nd quarter of calendar year 2013. Table 1. Explanatory Variables on Participant Characteristics Variable Description Adult DW Youth WP Female x x x x 14 Age 15 x 16 Age 17 x Age 18 x 19 Age 20 x 26 Age 35 x x x 36 Age 45 x x x 46 Age 55 x x x 56 Age 65 x x x 66 Age x x x Hispanic ethnicity x x x x Race Asian not Hispanic x x x x Race Black not Hispanic x x x x Race Hawaiian Pacific Islander not Hispanic x x x x Race American Indian or Native Alaskan not Hispanic x x x x Race More than one not Hispanic x x x x Highest grade completed Less than High School graduate x x x x Highest grade completed High school equivalency x x x x Highest grade completed Some college x x x x Highest grade completed Certificate or Other Post -Secondary Degree x x x x Highest grade completed Associate degree x x x Highest grade completed Bachelor degree x x x Employed at participation x x x 1 Unemployment rate http www.bls.gov lau Employment http www.bls.gov cew datatoc.htm Seasonal adjustment http www.bls.gov cps seasfaq.htm . 4 Variable Description Adult DW Youth WP Individual with a disability x x x Veteran x x Had earnings in 2nd and 3rd preprogram quarters x x x Had earnings in 3rd preprogram quarter x x x Had earnings in 2 nd preprogram quarter x x x Received services financially assisted under the Wagner -Peyser Act x x x Limited English -language proficiency x x x Single parent x x Low income x x x TANF recipient x x x Other public assistance recipient x x x Homeless x x x Offender x x x Unemployment insurance claimant non -exhaustee x x x Unemployment insurance claimant exhaustee x x x Received supportive services x x Received needs -related payments x x Received intensive services x x Received training services x x Established Individual Training Account ITA x x Pell grant recipient x x x Received pre -vocational activity services x x Pregnant or parenting youth x Youth who needs additional assistance x Youth enrolled in education at or during program participation x Youth enrolled in education at exit x Youth enrolled in education at participation x Youth with basic literacy skills deficiency at or below 8th grade x Youth that is or was in foster care x Youth that received educational achievement services x Youth that received employment opportunities x Youth participated in an alternative school x Average educational functioning level for Youth participants x Average standardized pre -test score x Average standardized post -test score x 5 Table 2. Explanatory Variables on Economic Conditions Economic Variable Definition Unemp Rate Not seasonally adjusted quarterly unemployment rate NatResEmp Percentage of total employment in NAICS 1133 -Logging and Sector 21 -Mining ConstEmp Percentage of total employment in Sector 23 -Construction ManfEmp Percentage of total employment in Sectors 31 32 33 -Manufacturing TechEmp Percentage of total employment in Sector 51 -Information Sector 52 -Finance and Insurance Sector 53 -Real Estate and Rental and Leasing Sector 54 -Professional Scientific and Technical Services Sector 55 -Management of Companies and Enterprises and Secto r 56 -Administrative and Waste Services EdHealthEmp Percentage of total employment in Sector 61 -Eductaional Services and Sector 62 -Health Care and Social Assistance LeisHospEmp Percentage of total employment in Sector 71 -Arts Entertainment and Recreati on and Sector 71 -Accommodations and Food Services OtherServEmp Percentage of total employment in Sector 81 -Other Services PublicAdminEmp Percentage of total employment in Federal State and Local Government