About the Study
In 2017, the Chief Evaluation Office (CEO) funded contractor Mathematica and its partners, American Institutes for Research, the Urban Institute, the W.E. Upjohn Institute, and ideas42, to assist with CEO’s Behavioral Interventions (BI) work, including developing rigorous impact evaluation design options for studying innovative program improvement strategies. In 2019, in partnership with the Employment and Training Administration, the BI team began investigating behavioral barriers to successful online job search. The BI team partnered with West Michigan Works! (WMW), a local workforce agency, to conduct the study “Applying Behavioral Insights to Inform Job Search: Evaluating Effects of a Behaviorally Informed Intervention on Job Search Online in West Michigan” from August 7, 2021 to March 3, 2022.
This Department of Labor-funded study was a result of the learning agenda process. It contributes to the labor evidence-base to inform the use of Behavioral Interventions in Labor programs and policies and addresses Departmental strategic goals and priorities.
The randomized controlled trial aimed to answer:
- How might applying insights from behavioral science lead people to expand their job search?
- Does providing job seekers with salient information about job postings lead them to engage with a larger number of postings or wider range of industries?
- What were the key features of the context in which this intervention was implemented?
- How did website visitors interact with the new web page? What were their perceptions about benefits and drawbacks of the new design features?
Informed by existing evidence, the study team hypothesized that providing a short list of relevant occupations alongside each job listing on a webpage would lead job seekers/website visitors to broaden their search by viewing listings they would have otherwise passed over based on the job title.
- In contrast to researchers’ hypothesis, the study found that this behavioral intervention led website visitors to be 4 percentage points less likely to click on a job listing and to click on an average of 0.1 fewer listings.
- The intervention showed no meaningful impact on the number of industries a website visitor browsed in, nor did it affect their likelihood of making return visits to the site.
- Qualitative results from interviews with five website visitors suggest this small sample of West Michigan Works! customers found the new information helpful in searching efficiently and considering job descriptions they might have overlooked. They also reported using the new information to quickly focus their attention on what they believed to be the most promising openings. This suggests that job seekers may have used the new information to refine rather than expand their search.
- A small sample of four WMW staff felt that the research partnership complemented and supported their organizational culture of continuous improvement.
- This study demonstrates the potential of using web-based experiments to test behavioral interventions that could help job seekers, while the job market was recovering from unemployment spikes due to COVID-19. Further research could help distinguish between interventions that promote efficient search and those that narrow search without improving efficiency, in different labor markets and contexts.
Report
Spitzer, A., Welch, E., Chojnacki, G., Congdon, B., O’Leary, C. (2023). Mathematica. Applying Behavioral Insights to Inform Job Search: Evaluating Effects of a Behaviorally Informed Intervention on Job Search Online in West Michigan. Chief Evaluation Office, U.S. Department of Labor.
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.