About the Study
In 2021, the Chief Evaluation Office (CEO) partnered with the Ohio Department of Job and Family Services and 11 participating counties in Ohio, funding contractor American Institutes for Research to conduct the Using Behavioral Insights to Increase Youth Use of Workforce Services in Virtual Contexts. The impact evaluation aims to test how behavioral insights can enhance service engagement and completion among young adults participating in Ohio’s Comprehensive Case Management and Employment Program (CCMEP).
Ohio’s Comprehensive Case Management and Employment Program (CCMEP) is a program serving young adults ages 14–24 by providing them with services that are tailored to each participant’s individual goals and intended to help them develop skills, find employment, and advance along career paths. Integrating funding from both the Workforce Innovation and Opportunity Act and the Temporary Assistance for Needy Families program, CCMEP provides a wide range of employment, training, and supportive services to participants, based on a comprehensive assessment of their needs. The program’s goal is to improve low-income individuals’ access to rewarding career pathways and to break cycles of poverty.
This Department of Labor-funded study was a result of behavioral insight efforts. It contributes to the labor evidence-base to inform behavioral interventions and employment and training programs and policies and addresses Departmental strategic goals and priorities.
- The text messages increased the number of services started by CCMEP participants within their first 60 days in the program. On average, every other person who received the text messages started one more service than they would have otherwise.
- The text messages substantially increased service completion rates. They increased the likelihood that a CCMEP participant would successfully complete at least one program service within their first 60 days in the program by 10 percentage points. This was a 46 percent improvement on the control group mean.
- The text messages had higher impacts on 60-day service completion rates for participants who were: (1) younger than 18, (2) basic skills deficient, and (3) not parents.
- The text messages had no impacts on outcomes measured within the first 90 days—neither the number of services started nor the likelihood of completing at least one service.
- The impacts of the text messages on the number of services started were no different for any of the four subgroups analyzed: (1) younger than 18, (2) basic skills deficient, (3) parenting, and (4) in school.
- County program administrators see clear benefits of sending behaviorally informed texts to encourage youth engagement but highlighted potential risks and adverse effects as well. There was consensus that these messages should be templates that CCMEP coaches can personalize and customize to reflect participants’ recent interactions with the program.
Report
Amin, S., Davis, S., Fatima, S. (2023). American Institutes for Research. Using Behavioral Insights to Increase Youth Use of Workforce Services in Virtual Contexts: Final Report. Chief Evaluation Office, U.S. Department of Labor.
Appendix
Amin, S., Davis, S., Fatima, S. (2023). American Institutes for Research. Using Behavioral Insights to Increase Youth Use of Workforce Services in Virtual Contexts: Appendices. Chief Evaluation Office, U.S. Department of Labor.
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.