Workforce Innovation Fund (WIF) Evaluations

• Workforce Innovation Fund (WIF) Evaluations: A Synthesis Report on Findings and Experiences (Synthesis Report, July 2022)
• Workforce Innovation Fund (WIF) Evaluations: A Synthesis Report on Findings and Experiences Appendices (Appendices, July 2022)
• Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Development of the Grant Applications and Initial Implementation (Implementation Report, July 2022)
• Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Final Report (Final Report, July 2022)

The Workforce Innovation Fund (WIF): A Synthesis Report on Evaluation Findings and Experiences

• The WIF evaluations show positive to mixed effects across a range of outcomes, although study limitations affected the reliability of some of the results. Given the varied range of interventions implemented and the different expectations about what outcomes they were designed to improve, it is difficult to draw firm conclusions about specific intervention types.
• The WIF structure was observed to involve a number of stakeholders in a complex array of relationships, which presented challenges to both grantees and evaluators. The various entities involved with the WIF project each had different primary concerns and interests. Evaluators were most interested in the feasibility of the evaluation design, while the grantee was concerned with fulfilling the funder’s grant agreement requirements. Communication among stakeholders was described as a key component to maintain effective working relationships and evaluator independence. When findings are not positive or do not demonstrate the expected outcomes, the evaluator needs to ensure that the grantee fully understands those findings.
• Evaluators and grantees experienced challenges in evaluation execution, many of which are common in the evaluation field but some of which were unique to WIF. Observed challenges included lower than expected levels of program participation, barriers to accessing data, and challenges with survey administration and non-response. An evaluation execution challenge unique to WIF was the limited period of performance of the grant, and short follow up period to assess outcomes. Lastly, while WIF had the goal of comparing outcomes across grantees, it was not considered possible to collect comparable data across interventions due to a lack of a standard set of data measures required of all interventions.
• The WIF grantees also imparted specific strategies that can be used to address the challenges encountered. Recommendations include: defining roles clearly and increasing communication and accountability across stakeholders; developing clear, consistent standards and lines of authority to enforce standards; and building capacity within grantee/project staff to ensure evaluation goals and methods are understood.

Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Final Report

• Project partners were optimistic about achieving outcome targets in the second phase, in spite of not achieving target outcomes in the first phase. Both projects brought on required partners, implemented the service model, and conducted rigorous evaluations. Neither project reached the pre-specified outcome targets, so they did not request payment from Department of Labor (DOL).
• Conducting a randomized control trial (RCT) was reported to be one of the most difficult aspects of the PFS project Each project experienced difficulties with some areas of implementing an RCT, including obtaining adequate sample size, high levels of participation in program services among the treatment group, a strong treatment-control contrast, and high quality and accessible data.
• The PFS sponsor was suggested to be specific about the required reporting on the RCT methodology and factors that may have affected the interpretation of results. Based on the experience the DOL pilot projects, it is important to provide detailed requirements related to how final results should be presented and what issues will be addressed. Other suggested considerations include being transparent about evaluation plans and progress through the life of the project, and having the sponsor clearly outline the responsibilities of and scope of work for the validator.
• The two PFS pilot projects required substantial time and resource investment by the key partners. Both pilots were able to successfully develop and manage the projects, but partners reiterated that doing so took a great deal of time. Staff changes were especially challenging over the course of the grants. Both states reported that they invested substantial staff time in developing, managing and overseeing the project.

Synthesis Report

Kean, E., Cook, R., De la Rosa, S. M., Williams, J. (2020). Abt Associates. The Workforce Innovation Fund (WIF): A Synthesis Report on Evaluation Findings and Experiences. Employment and Training Administration, U.S. Department of Labor.

Synthesis Report Appendices

Kean, E., Cook, R., De la Rosa, S. M., Williams, J. (2020). Abt Associates. The Workforce Innovation Fund (WIF): A Synthesis Report on Evaluation Findings and Experiences Appendices. Employment and Training Administration, U.S. Department of Labor.

Final Report

Kean, S., Schwartz, D., Sarna, M., Martinson, K. (2020). Abt Associates. Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Final Report”. Employment and Training Administration, U.S. Department of Labor.

Development of Grant Applications and Initial Implementation

Abt Associates. (2016). Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Development of the Grant Applications and Initial Implementation”. Employment and Training Administration, U.S. Department of Labor.