TEGL_6-19_acc.pdf

ETA Advisory File Text

EMPLOYMENT AND TRAINING ADMINISTRATION ADVISORY SYSTEM U.S. DEPARTMENT OF LABOR Washington D.C. 20210 CLASSIFICATION Unemployment Insurance CORRESPONDENCE SYMBOL OUI DUIO DATE October 31 2019 RESCISSIONS None EXPIRATION DATE ContinuingADVISORY TO TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 6-19 STATE WORKFORCE AGENCIES FROM JOHN PALLASCH s Assistant Secretary SUBJECT Expectations for States Implementing the Reemployment Service and Eligibility Assessment RESEA Program Requirements for Conducting Evaluations and Building Program Evidence 1.Purpose. To provide States with guidance and expectations regarding the implementation of the RESEA evaluation and evidence requirements. 2.Action Requested. The Department of Labor s DOL s Employment and Training Administration ETA requests ptate Workforce Administrators to provide information contained in this Unemployment Insurance Program Letter UIPL to appropriate program and other staff in the state s workforce system. This information should be shared with staff in but not limited to the Unemployment Insurance UI program workforce programs administered under the Workforce Innovation and Opportunity Act WIOA I including the Wagner-Peyser Employment ServiceI and workforce information labor market information programs. 3.Summary and Background. Summary fn accordance with the statutory provisions for RESEA contained in the Social Security Act SSAFI states are expected to begin conducting evaluations of RESEA interventions and service delivery strategies no later than fiscal year EFYF 2020 to support building new evidence on effective RESEA interventions that all states can rely on in designing and delivering the RESEA program. Background On February 9 2018 the President signed the Bipartisan Budget Act of 2018 mublic Law No. 11R-123 BBA which amended the SSA and created a permanent authorization for the RESEA program. The RESEA provisions are contained in Section 30206 of the BBA enacting new Section 306 of the SSA. Section 306 of the SSA includes a tiered evidence approach for the RESEA program to encourage states to use evidence-based strategiesI where they existI and to conduct evaluations and build evidence for other interventions and service delivery strategies. The goal is to ensure that each state employs RESEA interventions and service delivery 2 strategies that are based on rigorous causal evidence from evaluations rated as high or moderate causal and are shown to reduce benefit duration as a result of improved employment outcomes. In addition states using interventions or service delivery strategies without such evidence must be under evaluation at the time of use to determine their effectiveness in achieving this goal. Over time as the RESEA program uses this tiered evidence approach 1 states will add to the evidence base grow the workforce system s understanding of what interventions work well for whom and in what contexts and expand the use of interventions with strong evidence of success. Similar tiered evidence models are used across federal government programs such as the Department of Health and Human Services Home Visiting Program. The statute in section 306 c SSA requires states to use RESEA grant funds for evidence-based interventions or service delivery strategies that reduce the average number of weeks participants receive benefits by improving employment outcomes including earnings. Specifically it requires the following with regard to evidence building and evaluations c EVIDENCE-BASED STANDARDS.-- 1 IN GENERAL.--In carrying out a State program of reemployment services and eligibility assessments using grant funds awarded to the State under this section a State shall use such funds only for interventions demonstrated to reduce the number of weeks for which program participants receive unemployment compensation by improving employment outcomes for program participants. 2 EXPANDING EVIDENCE-BASED INTERVENTIONS.--In addition to the requirement imposed by paragraph 1 a State shall A for fiscal years 2023 and 2024 use no less than 25 percent of the grant funds awarded to the State under this section for interventions with a high or moderate causal evidence rating that show a demonstrated capacity to improve employment and earnings outcomes for program participants. B for fiscal years 2025 and 2026 use no less than 40 percent of such grant funds for interventions described in sub-paragraph A and C for fiscal years beginning after fiscal year 2026 use no less than 50 percent of such grant funds for interventions described in sub- paragraph A . d EVALUATIONS.-- 1 Tiered evidence refers to a policy tool that allows federal agencies to tie federal funding to strategies with evidence to encourage the use of interventions that have strong evidence of success and test promising new ideas. With the RESEA program the legislation ties certain levels of future funding to interventions with moderate or high causal evidence ratings to encourage the use of those interventions that have stronger evidence that they work and requires interventions without those ratings to be to under evaluation at the time of use. 3 1 REQUIRED EVALUATIONS.--Any intervention without a high or moderate causal evidence rating used by a State in carrying out a State program or reemployment services and eligibility assessments under this section shall be under evaluation at the time of use. 2 FUNDING LIMITATION.--A State shall use not more than 10 percent of grant funds awarded to the State under this section to conduct or cause to be conducted evaluations of interventions used in carrying out a program under this section including evaluations conducted pursuant to paragraph 1 . ETA provided preliminary guidance with regard to these provisions for FY 2019 in UIPL No. 07-19 https wdr.doleta.gov directives corr doc.cfm DOCN 8397 . This guidance provides information on the new RESEA evidence-based requirements and provides definitions of high and moderate causal evidence. High or moderate causal intervention ratings are based on how many good quality studies show positive impacts of that intervention. To provide states a solid foundation on the meaning of good quality studies this guidance presents a description of how DOL rates studies quality of evidence through its Clearinghouse for Labor Evaluation and Research CLEAR see Section 6 below . The guidance also discusses the standards for rating intervention effectiveness and identifies relevant interventions that currently meet those standards. In addition to setting standards and intervention ratings this guidance also suggests RESEA components that are in need of expanded evidence and includes a discussion of evaluation approaches and strategies for carrying out evaluations. Finally the guidance points to resources that are available to states to better understand and use existing evidence and to help states initiate rigorous high-quality evaluations to build evidence on the effectiveness of interventions in their RESEA programs. While the intent is that states will implement interventions and service delivery strategies supported by rigorous evidence there is not yet a large body of such evidence related to the new parameters for the permanent RESEA program. States must begin conducting rigorous studies to produce new evidence that helps determine the success of the interventions and service delivery strategies that meet the goals of the RESEA program. 4. Expectation that States Begin RESEA Evaluations No Later than FY 2020. Reemployment evaluations to date have focused mainly on broad categories of services or services at a program level. These evaluated programs have similarities to RESEA but also many differences. A primary goal of the RESEA legal requirement for evaluations is to expand the evidence base by conducting new high-quality evaluations of states RESEA programs particularly to build evidence about specific program components or activities. Congress as reflected in the provisions of section 306 SSA intended the evidence base for RESEA to expand and to improve the program through state use of evidence-based interventions with high or moderate causal ratings. While there is a modest and growing evidence base from which to synthesize and draw conclusions about RESEA interventions 4 effectiveness there is an immediate need to grow and expand it to address new RESEA program components. Previous evaluations of the Reemployment and Eligibility Assessment REA program the predecessor to the RESEA program were based on the whole program and the need now is to develop and expand evidence on more well-defined activities program components and service delivery approaches that states use in operating the RESEA program. Development of a culture of continuous improvement and evidence building around the RESEA program will strengthen it over time and improve reemployment outcomes for unemployment compensation UC claimants. To meet Congressional intent with regard to causal evidence ratings in the tiered evidence approach and to ensure states ability to comply with the evidence and evaluation provisions in the statute states are expected to begin evaluating RESEA interventions and service delivery strategies as soon as feasible and no later than the end of FY 2020 for the following reasons The requirement that states use only interventions with high or moderate causal evidence ratings or have them under evaluation is in effect in FY 2020 RESEA while modeled in part after the former REA is a different program and includes the actual delivery of reemployment services in addition to the foundational elements of the REA program so evidence beyond evaluations of the REA program is needed Expanded evidence is needed to ensure that states have sufficient evidence to support program delivery when the minimum percentage requirements for use of interventions with high or moderate causal ratings begin in FY 2023 In the new RESEA state plan required in FY 2020 states must articulate a description of their evaluation structure for RESEA interventions without a high or moderate causal evidence rating and Rigorous impact evaluations sufficient to achieve a high or moderate causal rating are most often multi-year in length and states need to begin conducting evaluations now to obtain sufficient evidence to support delivery strategies and interventions of their RESEA programs in FY 2023 and beyond. 5. Evaluation Parameters. Given that a key goal of the RESEA program is reduced average duration of UC benefit receipt as a result of improved employment outcomes states RESEA impact evaluations must include duration of UC and employment as primary outcomes Unemployment Compensation Duration This outcome is measured as the number of weeks RESEA participants receive UC and Employment For RESEA participants employment and earnings outcomes can be measured in the second full calendar quarter following the start of a participant s UC claim similar to the WIOA measures or sooner in the claims cycle to the extent that data is available. States are also encouraged to propose additional outcomes that could provide early indications that the RESEA program is working as intended. Examples of outcomes that 5 states might consider include increased participation in or completion of the RESEA program activities or time to reemployment following the start of RESEA interventions. States should consider when feasible coordinating their RESEA evaluations with their WIOA-mandated evaluation projects which can create economies of scale and generate synergies across programs. States new evaluations must meet evidence standards for study quality and find favorable impacts with at least a reasonable degree of statistical confidence to allow the intervention under examination to potentially qualify for a high or moderate rating as defined in Section 8 below. The goal of this evidence-generating approach is to provide states operating RESEA programs with a sufficient number of new studies that meet these standards which can support along with current evidence the statutory requirement for states to use interventions demonstrated to be effective. DOL recognizes that all findings whether positive negative or null are important contributions to the evidence base and DOL is committed to learning from and using evaluations and data to inform program improvements. As such it is both critical and expected that all evaluations conducted of RESEA interventions be publicly available regardless of the outcomes. States are also encouraged to share links to their publicly posted completed evaluations with CLEAR to ensure their inclusion in future evidence reviews. RESEA evaluations will play an important role in building the reemployment evidence base and in helping states and other program decision-makers make more informed choices about how to bundle RESEA program components and strategies to best meet the needs of the people being served by them. 6. Clearinghouse for Labor and Evaluation Research CLEAR . A first step in identifying interventions with high and moderate causal ratings is determining which existing studies provide evidence about them that is relevant and credible. DOL will leverage CLEAR to identify evaluations in the evidence base that are relevant to the RESEA program and determine which impact studies have high moderate or low causal evidence ratings. DOL established CLEAR to make research on labor topics more accessible to practitioners policymakers researchers and the public so that evidence can inform policy and program decisions. To achieve this goal CLEAR conducts systematic evidence reviews of research and evaluation reports on labor topics and then reviews and summarizes those studies. CLEAR also rates studies that estimate causal impact. CLEAR currently has over 600 studies summarized across 18 labor-related topic areas including Reemployment and is continually growing. The Reemployment evidence review identifies summarizes and determines the quality of existing causal evidence on reemployment service delivery strategies intended to promote reemployment of UC claimants while also reducing UC receipt duration. Under the Reemployment topic area CLEAR has reviewed 45 publications published between 1978 and 2018 and has developed one-page summary profiles of and ratings for each of these studies. The reviewed studies use causal designs otherwise known as impact studies and assess the effectiveness or impact of an intervention. These studies identify how a particular intervention changes claimants outcomes relative to a comparison group such as those that receive a different intervention or those that did not receive the intervention. 6 Many impact studies use random assignment designs. Such designs randomly i.e. through the functional equivalent of a coin toss assign some eligible individuals to a treatment group or groups that may participate in the intervention and others to a control group that do not participate in the intervention. These designs use random assignment to prevent systematic pre-existing differences between the two groups from creating bias in an evaluation. Thus systematic differences in outcomes between the two groups can reasonably be attributed to the intervention. Other causal impact studies may use quasi-experimental designs that estimate impact but do not use random assignment. Instead quasi-experimental designs use administrative data and statistical techniques to identify a comparison group that is similar to the treatment group to act as a control group. The credibility of the evidence from an impact study depends on how it is designed and carried out. Currently CLEAR has established causal evidence guidelines 2 which identify the criteria CLEAR uses to assess the strength of a study s causal evidence. CLEAR s causal evidence ratings are an indicator of the quality of the study and the level of confidence you can have that the study s findings truly reflect the causal impact of the intervention studied and not some other factor. CLEAR also has guidelines for high-quality quantitative descriptive and implementation studies but does not currently assign evidence ratings to those types of studies.3 CLEAR currently assesses its causal evidence ratings based on the rigor of the study as follows. Studies receive a high rating for causal evidence if there is confidence that the study s estimated effects are solely attributable to the intervention being examined. Studies receive a moderate rating for study quality if there is some confidence that the estimated effects are attributable to the intervention studied but there might be other contributing factors that were not included in the analysis. Studies that do not meet the criteria for a high or moderate rating receive a low rating which indicates that it is not possible to be confident that the estimated effects are attributable to the intervention studied. In these instances other factors likely contributed to the estimated effects. Moving forward and as described below CLEAR will assign causal evidence ratings to new RESEA studies based on both study quality and effectiveness of the intervention examined in a study as appropriate. DOL will publicly and transparently post information about this process on the CLEAR website when future evidence reviews begin. 2 Find CLEAR s causal evidence guidelines here https clear.dol.gov sites default files CLEAR EvidenceGuidelines V2.1.pdf 3 Find CLEAR s quantitative descriptive guidelines and guidelines for reviewing implementation studies here under Reference Documents https clear.dol.gov about 7 7. The Need for Expanded Evaluations of Interventions. The statute in section 306 i 3 SSA defines an intervention as a service delivery strategy for the provision of State reemployment services and eligibility assessment activities. In operating RESEA programs states bundle or mix multiple components and activities together in ways that may vary in their details or emphasis. They may also vary in the strategies or approaches for carrying them out. For instance all RESEA programs include a claimant selection component but states may vary in how they select claimants for participation. For evidence rating purposes an evaluation intervention may be a whole program or any component of it. RESEA is different from any of the interventions for which evidence currently exists. It shares some elements with earlier programs particularly with REA but it also has new elements. For example it places a greater emphasis on more intensive reemployment services for claimants and states now have greater freedom in deciding how to select claimants. States need to develop a substantial body of high-quality evidence about the effectiveness of RESEA strategies and components. Exhibit 1 lists components for which evidence needs to be built in order to meet the basic requirement of demonstrating effectiveness and to provide meaningful findings to help states design and implement their RESEA programs. Other gaps in the evidence base are expected to emerge as more is learned about states current RESEA programs. As new evidence is produced the list of interventions that have been demonstrated to be effective will be refined. Exhibit 1. RESEA Components in Need of Expanded Evidence Component Sub-component Specific Evidence to Build Selecting Claimants and Scheduling Meetings Claimant Selection Mechanism What selection approach if any identifies claimants that will be most favorably affected by RESEA selection o Selecting those with high exhaustion risk o Selecting those with low exhaustion risk o Selecting based on other criteria Timing of Claimant Selection Is it better to select claimants as soon as possible i.e. after the first payment has been made or later in the life of the claim 8 Component Sub-component Specific Evidence to Build Scheduling RESEA Meetings How soon after claimant selection should RESEA meetings be required to occur What scheduling strategies are most likely to ensure the claimant fully participates in all RESEA services Does getting the claimant to initially show up increase the likelihood of full participation in RESEA services What is the effect of having multiple RESEA meetings rather than just one Reemployment Activities Reemployment services What strategies are most effective to support development of a reemployment plan that the claimant owns and implements What is the impact of more basic assistance e.g. American Job Center orientation self- directed use of online tools general labor market information vs. the impact of more intensive individualized career services What is the impact of more intensive one-on- one career counseling What are effective strategies for delivering job search assistance Are there types of training that can help get individuals to work quickly e.g. on-the-job training apprenticeship Are there particular assessments or other ways of identifying claimants needs that create a more effective reemployment plan better connects claimants with services and ultimately leads to more positive outcomes What is the effect of case management Is increased frequency of intensive case management e.g. more regular contact more effective Activities to support work search compliance What strategies that support a claimant s work search compliance impact employment outcomes What is the effect of review of continued eligibility for benefits on the claimant s employment outcomes How do the strength immediacy and surety of penalties for failure to attend FTA affect job search efforts and outcomes 9 8. Criteria for Causal Evidence Ratings. Section 306 c 2 SSA conditions funding for RESEA programs on states using interventions either demonstrated as effective with a high or moderate causal evidence rating or being under evaluation. Beginning in FY 2020 the definitions established below will be in effect and explain how an intervention can qualify for a high rating or a moderate rating.4 These ratings examine available evidence and determine whether the interventions have favorable impacts on both employment and benefit duration outcomes.5 The high and moderate causal evidence standards described below rely on evidence of impact exclusively from studies that received a high or moderate rating for study quality in CLEAR. These studies are identified in the definitions as credible studies. High For an intervention to qualify for a high causal evidence rating there must be at least two credible impact studies of the intervention as reviewed by CLEAR that have each found favorable impacts on employment and UC duration with a strong degree of statistical confidence.6 Moderate For an intervention to qualify for a moderate causal evidence rating there must be at least one credible impact study of the intervention that found a favorable impact on employment and one credible impact study of the intervention that found a favorable impact on UC duration. Again these ratings of the study or studies are as reviewed by CLEAR. Each study must have at least a modest degree of statistical confidence.7 The findings on employment and benefit duration may both come from the same study or from different studies. DOL also defines two additional categories potentially promising and no rating. Potentially Promising A potentially promising rating indicates that there is some suggestive evidence that an intervention may be effective. Such interventions are candidates for further evaluation that possibly would allow the intervention to qualify for a higher rating. For an intervention to qualify for a potentially promising causal evidence rating there must be one impact study reviewed by CLEAR irrespective of 4 As the evidence base grows more information will be available to help distinguish which approaches have the strongest evidence of effectiveness. At that time the standards for evidence of effectiveness may evolve as well in order to help better support those distinctions. 5 Specifically the ratings criteria are based on interventions estimated impacts on 1 reduced UC duration and 2 increases either employment rates or earnings as measured in the second full calendar quarter after the claim began. 6 A strong statistical confidence is defined as an estimated impact that is statistically significant different from zero at the 5 level. p .05 . This means that there is less than a 5 chance that the study s results are due to chance and not actually the intervention. Impact estimates must meet that threshold for both outcomes UC duration and employment. 7 A modest degree of statistical confidence is defined as an estimated impact that is statistically significant different from zero at the 10 level. p .10 . This means that there is less than a 10 chance that the study s results are due to chance and not actually the intervention. Impact estimates must meet that threshold for both outcomes UC duration and employment. 10 the causal evidence rating it received 8 that has found significant favorable impacts on either employment or U C duration with at least a mode rate degree of statistical confidence. 9 No Rating All interventions that do not qualify for any of the three ratings above will receive no rating regardless of the rating given by CLEAR for the quality of studies of that intervention . These may be interventions for which no impact studie s have been conducted interventions with an impact study that have not been reviewed by CLEAR yet or interventions whose studies have been reviewed by CLEAR but have not shown any favorable impacts . 9. High and Moderate Causal Ratings for Existing and Futu re Interventions Beginning in FY 2020 . Existing impact studies of approaches to speed the reemployment of U C claimants typically focus on broadly defined sets of services and activities . CLEAR s 2018 Reemployment Research Synthesis on reemployment interv entions 10 identified the following broad intervention categories from the existing evidence base that are relevant to RESEA 11 Reemployment and Eligibility Assistance REA The REA program the predecessor to RESEA provided claimants up to three mandator y in person sessions in which workforce staff assess ed their continued eligibility for UC provide d them with labor market information and support ed their development of a reemployment plan . In some cases they also provided referrals to reemployment ser vices. Fail ure to attend REA sessions without good cause affect s continued receipt of UC . Job Search Assistance JSA JSA interventions provide claimants assistance and training in job search techniques including job search workshops preparing a resum e and interview training. The JSA interventions that included strong linkages 8 CLEAR also rates some studies as low. These studies are not used when considering whether an intervention is eligible for a high or moderate effectiveness rating . However studies rated as low can contribute to a potentially promising rating. Th e potentially promising rating indicate s that some suggestive evidence exists that an intervention might be effective. While evidence from a low rated study is not a strong basis for concluding that an intervention is effective it can suggest that the intervention may be worth considering for more rigorous testing . Some studies that are rated as low may still be considered promising and thus a candidate for further evaluation. 9 As noted earlier for moderate effecti veness ratings a modest degree of statistical confidence is defined as an estimated impact that is statistically significant different from zero at the 10 level . 10 Find CLEAR s Reemployment Research Synthesis here https clear.dol.gov sites default files CLEAR 20Reemployment 20Synthesis 20November 202018.pdf 11 All descriptions are adapted from CLEAR s 2018 research synthesis What do we know about the effect of reemployment initiatives which can be found at https clear.dol.gov sites default files ResearchSynthesis Reemployment.pdf 11 between UI and workforce partners and required claimants at risk for benefit exhaustion to report for job search assistance demonstrated positive impacts. Profiling Profiling interventions identify claimants at higher risk of exhausting UC and offer or require enhanced employment services. These services may include an orientation providing labor market information and referrals to job search training or resume training work shops. Claimants that fail to participate in required services without good cause lose U C. More Stringent Employer Contact Requirements This type of intervention increase s the amount of work search effort required of claimants to continue receiving U C strengthen verification of work search effort s or both. Less Stringent Employer Contact Requirements This type of i ntervention reduce s the amount of work search effort required of claimants to continue receiving U C loosen verification of work search ef fort s or both. These broadly defined interventions often involve partially overlapping services and activities. Beginning in FY 2020 the se interventions will receive effectivene ss ratings using the definitions above . To the extent that the states pro grams use interventions that have not received high or moderate evidence ratings those states must be conducting high -quality impact evaluations using the CLEAR guidelines for study quality. Interventions Receiving a High Rating Of the interventions con sidered only REA currently receives a high causal evidence rating . If a state s RESEA program has components that are sufficiently similar to the evaluated REA program components a state can demonstrate that those pieces or components of its RESEA progra m are evidence -based by referring to this intervention and its rating. While no evaluation of REA components that are part of a state s RESEA program is required states are encouraged to continue to evaluate the se interventions in order to build rigorou s evidence in the context of t he new RESEA program . As noted above RESEA is not identical to the REA program and has different components so REA interventions alone w ill not be sufficient to meet RESEA requirements and states should continue to conside r implementing new interventions . If a state s RESEA program includes other components that are not evidence -based those components must be under evaluation at the time of use. Interventions Receiving a Moderate Rating Applying the criteria above JS A and profiling interventions receive a moderate causal evidence rating. If a state s RESEA program includes components like the JSA and profiling strategies described above in this guidance a state can demonstrate the corresponding components of its RES EA program are evidence -based by referring to these components and intervention s and their rating s. Again while no evaluation of JSA components of a state s RESEA p rogram are required states are encouraged to continue to evaluate their interventions in order to continue to grow the base of evidence regarding their use in the RESEA program . Additional evidence on these interventions is st ill valuable and could result in raising the causal evidence rating for the intervention s to the high category. If a state s RESEA program includes other components that are not evidence -based those components must be under evaluation at the time of use. 12 Interventions Receiving a Potentially Promising Rating Applying the criteria above the component of requiring mo re stringent employer contacts receives a potentially promising causal evidence rating. This rating indicates that the component or intervention may be of interest to consider adopting or testing as it might be effective. Additional evidence on these in terventions might also support a change of causal evidence rating for the intervention . States implementing i nterventions with only a potentially promising rating must be evaluated at the time of use . Interventions Receiving No Rating Applying the crite ria above the component of less stringent employer contacts receives no rating. Additionally t he more detailed components included in Exhibit 1 as well as any additional RESEA interventions or program components not identified here as being demonstrated effective by current evidence also currently receive no rating. Such interventions must be evaluated if states choose to implement them . 10. Evaluation Approaches . Section 306 c SSA gives states time to evaluate RESEA interventions before the percent age requirements for use of interventions with high o r moderate causal ratings begin in FY 2023 . States evaluations will need to meet the causal evidence standards described in Section 8 of this guidance. To help states impact evaluations have the best chance of meeting CLEAR s standards states are strongly encouraged to 1 choose an experienced evaluator 2 choose a simple impact study design the simplest being random assignment with administrative data follow -up and 3 take advantage of the evaluation technical assistance E val TA guidance being provided described in more detail in Section 13 below . Importantly there are multiple type s of evaluations and evaluation -related activities that ultimately support a strong impact evaluation tha t produces high or moderate causal evidence. For example it may be appropriate to conduct an evaluability assessment discussed in more detail below or feasibility study before embarking on an impact evaluation to identify any challenges or barriers su ch as data availability or limited sample size to conducting an evaluation of a specific intervention or service delivery design. Alternatively it may be desirable to pair both an implementation evaluation and an impact evaluation. DOL considers activi ties leading up to an impact evaluation that has the capability of producing a high or moderate causal rating to be interventions designated as under evaluation. Examples of these activities include an evaluability assessment as described below and an implementation study that helps refine the specific intervention and research questions to be addressed in the impact evaluation. Evaluation Design DOL encourages states evaluation designs to specify use of or be building evidence to move toward the us e of approaches capable of earning high or moderate quality ratings with 13 the goal of producing both the strongest possible evidence and the highest possible causal evidence rating for the intervention being studied. As noted previously impact evaluation s are necessary to achieve these ratings however there may be other types of evaluation s or pre -evaluation activities that should be conducted first or along with an impact study to maximize learning about the intervention . The most common types of evaluation designs are Impact Evaluation This type of evaluation assess es the impact of a program or component of a program on outcomes typically relative to a counterfactual situation. This evaluation provides some estimate of what would have happened in absence of the program or component of the program. Impact evaluation includes both experimental i.e. randomized controlled trials and quasi - experimental designs. These types of evaluations speak to the does it work question . Outcome Evaluation This type of evaluation measures the extent to which a program has achieved its intended outcome s and focuses on outputs and outcomes to assess effectiveness. Unlike an impact evaluation an outcome evaluation cannot show causal impacts. An outcome e valuation can help answer question s like Did the program policy or organization do what it intended to do Process or Implementation Evaluation These types of evaluations assess how the program or service is delivered relative to its intended theory of change and often include information on content quantity quality and structure of services provided. Process or implementation evaluation s can be condu cted on their own but are often conducted along with impact and or outcomes evaluation s. Proce ss or implementation evaluations can help answer questions like Was the p rogram or policy implemented as intended or How is the program policy or organization operating in practice Formative Evaluation This type of evaluation typically done bef ore full implementation of a program assesses whether a program or component of a program is feasible appropriate and acceptable before it is fully implemented. It may include some of the activities described above such as process evaluation or outcom e evaluation. However unlike summative evaluation designs like impact and outcome evaluations which seek to answer whether or not the program met its intended goal s or had the intended impact s a formative evaluation focuses solely on learning and i mprovement and does not answer questions of overall effectiveness . Selecting an Evaluator While DOL recognizes there is value in all types of evaluations the RESEA evidence - gene rating approach specifically requires impact evaluations of interventions to help determine causal evidence rat ings for those interventions. While states may have evaluation capacity within the agency operating the RESEA program DOL recommends that states use an experienced and independent evaluator that can identify the most ap propriate and rigorous 14 design to answer research questions and learn about the RESEA program and program components and interventions . Deciding What to Evaluate DOL recognizes that each state s RESEA program is a uniquely implemented bundle of differen t interventions and service delivery strategies or components. However this can make it difficult to know which program components and how these components are generating the observed outcomes of the intervention . Therefore states are strongly encou raged to work with their independent evaluators to develop evaluations that seek to estimate the imp act of individual RESEA program components and interventions or to develop evaluations of whole programs where the components are well defined and the effe ctiveness of which could be evaluated at a later time through meta -analyses . Building this type of evidence will further states understanding of the effectiveness of components that could be mixed and matched to develop a program that meets the needs of a specific state. However DOL recognizes that e valuating only a component of the program implies the need for the evaluation to detect smaller impacts which requires much larger samples. Again states are encouraged to work with experienced evaluators and explore partnerships with other states to develop the most rigorous and appropriate designs to determine the effectiveness of program components. See Section 11 below for more discussion on evaluation partnerships across states. Pre -evaluation act ivities states can begin doing now to support getting to a firm evaluat ion plan include the following activities States are encouraged to conduct evaluability assessments of their RESEA programs. Evaluabi lity assessments will help states define specif ic interventions that are evaluation -ready to test in a feasible measurable way ensure that the intervention and the component s to test are well -understood by all stakeholders confirm availability of data and other operational resourc es needed to cond uct an evaluation and build consensus on evaluation goals to ensure results are relevant to stakeholders. Evaluability assessments also are useful for identifying a program s potential strengths and challenges with planning and executing an evaluation . For example they may assess whether adequate access to information technology IT and data resources exist and are available to support the evaluation or if program staff has sufficient evaluation expertise. The results of an evaluability assessment refin e a state s broad learning goals with more narrowly -focused research questions that explore the RESEA program s influence on a particular population s outcomes of interest and identify the type of evaluation that can best answer these questions . Evaluabil ity assessments highlight evaluation feasibility issues such as operational gaps that must be addressed to succ essfully execute the evaluation such as availability of IT resources and data availability staff ing to increase evaluation capacity developin g partnerships with organizations that have a ppropriate evaluation expertise and creating evaluation procedures and training staff. Addressing these operational gaps identified through an evaluability assessment strengthen s a state s ability to produce a high -quality evaluation that meets CLEAR standards. States can 15 find an Evaluation Design Assessment Tool developed by IMPAQ International to support WIOA evaluations here https evalhub.workforcegps.org resources 2018 09 07 19 53 Evaluation -Design - Assessment -Tool States are also encouraged to develop logic models when formulating evaluation plans. Logic models are graphical representations of intervention s and how they operate. They are designed to show the following regarding a RESEA intervention Inputs such as staff time RESEA funds and other resources used to deliver the program Activities such as meeting with American Job Center AJ C staff to create an individual reemployment plan provision of reemployment services conducting the eligibility assessment and other activities the program regular ly operat es Outputs the immediate results of the program such as improved job readin ess skills or enhancing labor market knowledge and Outcomes the expected short -term and long -term goals of the program such as reduced UC duration faster return to employment and improved earnings. Logic models define the inputs activities or oth er tangible activities that lead to outputs and outcomes for the RESEA program . Logic models and other similar program mapping activities demonstrate how RESEA interventions drive the change in outcomes for claimants. States can find more information abo ut developing logic models for labor programs in a webinar titled Fully Articulating Your Vision Using Logic Models to Support Innovation https evalhub.workforcegps.org sitecore content global resources 2015 05 07 11 07 Fully Articulating Your Vision Using Logic Models to Support Innovation States evaluability assessments and logic m odels ultimately help states identify specific research questions that may be added to a multi -year learning strateg y or agenda . Organizing learning priorities is an approach that is gaining traction across the Federal government most recently supported in the Foundations for Evidence - Based Policymaking Act of 201 8 Public Law No. 115 -435 12 which requires Federal government agencies to produce evidence -building plans. Learning agendas can also serve as roadmap s to help states plan for immediate and fut ure evaluations by clarifying learning goals research questions the types of evaluations that would answer those questions and the states priorities in building evidence. 11. Strategies to Meet RESEA Evaluation Requirements . Some sta tes may be interes ted in conducting their own individual impact evaluations. As indicat ed previously in UIPL No. 7-19 DOL encourages states to consider evaluation 12 https www.congress.gov bill 115th -congress house -bill 4174 16 partnerships with other states so that states may consider conducting pooled evaluations of similar RESEA interventions. This approach has the benefit of potentially yielding sample sizes large enough to demonstrate effectiveness. Smaller states in particular might benefit from this strategy. It also has the benefit of allowing states to pool their limited evaluation funding . From previous research we know that sample sizes required to detect impacts on labor market outco mes of the kind required by section 306 SSA are large . DOL recognizes that many states do not have a sufficien tly large number of RESE A-eligible claimants in a single year and some states do not have that many RESEA -eligible claimants in several years. A pooling strategy including a well -defined intervention aligned across states can help overcome this challenge . States can pool the ir data and yield samples large enough to detect effects and potentially demonstrate effectiveness on a more reasonable timeframe. As mentioned previously states are encouraged to work with experienced evaluators and explore partnerships with other state s to develop the most rigorous and appropriate study designs to evaluate program components and interventions . 12. Evaluation Resources . In addition to the tools linked to above DOL s CLEAR also has several additional tools to help states better understa nd the evidence in the Reemployment topic area. CLEAR s Reemployment Synthesis is a short high -level plain -language report that summarizes studies of interventions that are relevant to RESEA . It describes key takeaways from the reemployment evidence b ase provides an overview of the interventions studied and identifies gaps in the research . This report may be useful as RESEA program managers begin to focus on conducting evaluability assessments and efforts to build the evidence base. As a companion to the Synthesis CLEAR s Reemployment Synthesis Supplement gives sta tes a more detailed look at the information provided in the synthesis. It provides brief description s of the findings for all the reports reviewed in the Reemployment topic area . This supplemental tool also includes links that lead directly to the study profiles in CLEAR where more information about the specific studies and interventions is available . It also is organized by sections that correspond to the intervention categories ident ified in the Reemployment Synthesis that will receive ratings beginning in FY 2020 as described above . CLEAR and its resources are at the links below CLEAR https clear.dol.gov Reemployment topic area https clear.dol.gov topic -area reemployment Reemployment Synthesis landing page https clear.dol.gov synthesis - report r eemployment -synthesis and a download able brief report https clear.dol.gov sites default files ResearchSynthesis Reemployment.pdf . Reemployment Synthesis Supp lement https clear.dol.gov sites default files ResearchSynthesis Reemploy Sup.pdf . 17 Another important resource for states will be DOL s RESEA Evidence Buildi ng and Implementation Study . In September 2018 DOL s Chief Evaluation Office awarded a three -year contract to Abt Associates the Urban Institute Capitol Research Corporation and National Association of State Workforce Agencies the RESEA study team to provide support on implementing the evaluation requirements in section 306 SSA . Among the tasks planned for the RESEA study team is an implementation study of states RESEA programs . DOL is conducting t his implementation study to examine how RESEA p rograms and strategies are operat ed understand how states are bundling various services to improve outcomes for RESEA participants and identify new innovative and potentially promising strategies being implemented . Findings from this implementation ev aluation will inform an evaluation report that will further describe research and evaluation options fo r DOL and states to consider and will contribute to the RESEA evidence base. Reports from the study will be publicly available when completed . A brief description of DOL s RESEA study on the Chief Evaluation Office s website is available here https www.dol.gov asp evaluation currentstudies Reemployment -Services -and -Eligibility - Assessments -Research.htm . Finally it is DOL s intent that states also leverage other available evaluation capacity - building resources. These include but are not limited to Evaluation and Research Hub A new community of practice created with input from state and local workforce agency representatives across the country. While it is available to address the evaluation requirements of the WIOA the resources included on the Hub can inform or supp ort the evaluation needs of all ETA -funded programs. You can find it here https evalhub.workforcegps.org about WIOA Evaluation Technical Assistance Tools State and local workforce agencies part icipated in ETA s peer learning effort to share and disseminate evaluation resources a s well as address questions such as Where and how do we start Key tools are included here o Evaluation Readiness Assessment Tool https evalhub.workforcegps.org resources 2018 09 07 19 45 Evaluation - Readiness -Assessment -Tool o Evaluation Design Assessment Tool https evalhub.workforcegps.org resources 2018 09 07 19 53 Evaluation - Design -Assessment -Tool o Evidence Says Work -based Learning https evalhub.workforcegps.org resources 2018 09 07 16 10 The -Evidence - Says -Work -Based -Learning Workforce System Strategies WSS A research clearinghouse that profiles evidence -based and emerging p ractices in workforce development to help the field make informed decisions about improving outcomes for job seekers and employers. Its resource library contains more than 1 200 profiles of evaluation reports policy and practice briefs and how -to guides . It is available here https strategies.workforcegps.org announcements 2018 05 04 20 17 Connect -Your - Peers -to-Workforce -System -Strategies . 18 13. Evaluation Technical Assistance EvalTA . As described above a critical piece of the DOL s RESEA project is to provide Eval TA to states. The E val TA will include a suite of tools and resources to help states meet evaluation and evi dence -building needs f or their RESEA programs. E xperienced staff from the RESEA study team will develop and deliver EvalTA . Beginning in summer 2019 the RESEA study team is offering generalized Eval TA which has been informed by state feedback from web inars clarifying calls and a review of available documents on states FY 2019 RESEA programs . The goal s of thi s generalized Eval TA is to help states with the following 1 gradually and continually increase their evaluation capacity so states are prepa red to begin ev aluation -related work by FY 2020 2 to describe evaluation activities in their FY 2020 RESEA state plans and 3 to meet evidence - related statutory requirements both now and moving forward . The Eval TA will be provided through resources DO L will make broadly available such as webinars toolkits briefs templates and videos to explain key topics to improve states understand ing of basic evaluation concepts and begin to plan and carry out evaluations. Th ese resources will build on existi ng DOL evaluation technical assistance resources e.g. for WIOA as described above and focus particularly on knowledge required for evaluations of RESEA program components and interventions that can meet evidence standards . When they are finalized a schedule of Eval TA activities as well as all resources developed through the E val TA will be available on the R eemployment Connections community of practice on WorkforceGPS www.workforcegps.org . In addition to this generalized Eval TA t he RESEA team will also offer customized Eval TA to individual states or small groups of states as appropriate that are planning or carrying out evaluations. Customized E val TA is anticipated t o begin in fall 2019 and is likely to include detailed verbal and written technical assistance to states at key points during individual evaluations . The RESEA study team may provide customized E val TA in areas such as procurement and selection of an independent evaluator selection of meth ods and development of evaluation design plans monitoring random assignment and dealing with unanticipated issues methods of analysis and reviews of analysis plans reporting and dissemination and other issues as appropriate and needed . Plans for custo mized E val TA will be updated by DOL s RESEA project yearly as states needs are better understood and as new RESEA interventions and evaluations are planned. While not every state is expected to need or participate in customized E val TA all states are encouraged to take advantage of the generalized E val TA being offered. Previous experience from other tiered evidence initiatives across the government suggest that an adequate planning period combined with a comprehensive E val TA strategy can improve evalu ation quality . Together these efforts can help meet one of the primary goals of t his early phase of RESEA implementation to expand the evidence base by supporting states in conducting new high -quality evaluations of RESEA program components and interven tions . 14. Inquiries . For further informati on p lease direct inquiries to the appropriate ETA Regional Office. 19 15. References . Section 306 Social Security Act 42 U.S.C. 506 The Bipartisan Budget Act of 2018 Public Law No. 115 -123 The Foundations f or Evidence -Based Policymaking Act of 2018 Public Law No. 115 -435 and Unemployment Insurance Program Letter No. 07 -19 Fiscal Year FY 2019 Funding Allotments and Operating Guidance for Unemployment Insurance UI Reemployment Services and Eligibility A ssessment RESEA Grants issued January 11 2019. 16. Attachment s . Not Applicable.