About the Study
In 2017, the Chief Evaluation Office partnered with the Employment and Training Administration (ETA) to fund contractors Urban Institute, George Washington University, Capital Research Corporation, and the National Association of State Workforce Agencies to conduct an analysis of employer performance measurement approaches required by the Workforce Innovation and Opportunity Act (WIOA).
WIOA requires the public workforce system to report a primary performance indicator of its effectiveness in serving employers. The knowledge development and analysis examines the appropriateness, reliability, and validity of three proposed measures of effectiveness:
- Retention with the Same Employer: Percentage of participants who exit and are employed with the same employer in the second and fourth quarters after exit.
- Repeat Business Customers: Percentage of employers receiving services in a given year who also received services within the previous three years.
- Employer Penetration Rate: Percentage of employers using services out of all employers in the state.
The study also analyzed other measures used by states, local areas, and Federal agencies to assess effectiveness in serving employers.
This Department of Labor-funded study helps meet requirements of Section 116 of the Workforce Innovation and Opportunity Act (WIOA). It contributes to the growing labor evidence-base to inform performance measurement and employment and training programs and policies and addresses Department strategic goals and priorities.
- How do states, local areas and federal agencies define 'services to employers'? In what ways are the definitions similar and in what ways are they different?
- What measures are currently being used, or have been used in the past or in other contexts, to understand engagement with employers by states, local areas and Federal agencies? How are the data for these measures collected?
- What are options for evaluating the feasibility, validity and reliability of proposed measures?
- Performance data submitted by states in Program Year 2018 indicate that states were most commonly piloting the Retention with the Same Employer and Employer Penetration Rate measures: 42 states were piloting each of these measures.
- Interviewed state and local staff reported that the categories of employer services used for reporting on employer services during the pilot were adequate for performance reporting but had some limitations. The study found officials responsible for WIOA Title II and Title IV programs viewed their mission in serving employers very differently from the way Title I and Title III program officials viewed employer services. Further, interviewees reported that some of the categories were not specific enough to capture the intensity of services provided by Title II and Title IV programs, such as providing adult basic education to incumbent workers or training to employers on how to work with persons with disabilities.
- For each measure, the study team identified threats to validity, reliability, and practicality as well as potential unintended consequences as required as part of the scope of work to assess the pilot measures. For example:
- Researchers found that the Employer Penetration Rate measure emphasizes quantity rather than the quality or intensity of the employer service provided, does not reflect the workforce system’s efforts to target particular sectors, and poses reliability issues because of problems in how employer services data are entered and how the programs count establishments.
- Analysis revealed the Repeat Business Customers measure is useful but would be improved by the use of statistical adjustment.
- Researchers found the Retention with the Same Employer measure is easiest to implement but it does not account for whether employers retaining job seekers have received business services from any of the WIOA core programs.
- Despite the pros and cons for each measure, the study found few alternative measures that appear preferable.
- Though the authors did not find an overwhelming case for adopting either one measure or several measures, adopting more than one measure offers the advantage of capturing more aspects of performance and may make results more actionable for the different Title I, II, III, and IV programs. Alternatively, a single measure has the advantage of clarity on how state performance is assessed and fewer resources devoted to record keeping.
- Additional research could:
- Further analyze selected measure or measures to strengthen them, support their adoption, and establish satisfactory levels of performance.
- Continue to examine alternative measures and measurement approaches that could supplement adopted measures and improve performance measurement.
- Develop statistical adjustment models (as required by law) and baseline standards to meet federal requirements and improve usefulness of measures.
Project Duration: 36 Months
Contract End Date: September 2020
Contractor: The Urban Institute, with George Washington University, Capital Research Corporation, and the National Association of State Workforce Agencies
For More Information: ChiefEvaluationOffice@dol.gov
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.