TEIN7-97_Ch1_Attach1.pdf

ETA Advisory File Text

1 EDWAA IMPACT EVALUATION SUMMARY OF REVISED DESIGN The net impact evaluation of the Economic Dislocation and Worker Adjustment Assistance EDWAA program is intended to provide policy-relevant information about the efficacy of the EDWAA program in improving the reemployment prospects and earnings of dislocated workers--that is workers permanently separated from their prior employers. The evaluation will provide information about whether retraining provided through EDWAA is effective in improving labor market outcomes as well as information about which groups of dislocated workers benefit most from retraining. This information will be generated through a rigorous random-assignment evaluation. In developing and refining the design for the evaluation the Employment and Training Administration and its contractor Mathematica Policy Research made several efforts to obtain suggestions from the EDWAA program s stakeholders Y Two meetings were held with members of the Enterprise Council -- a body formed to promote continuous improvement in EDWAA -- to discuss the study s proposed design and the use of random assignment. Y A Training and Employment Information Notice was issued describing the proposed design and requesting comments. Comments were received from 4 States Colorado New Jersey Illinois and New York and 1 PIC Philadelphia . Y The contractor visited or talked by telephone with six EDWAA substate areas to assess the acceptability of the design and determine the feasibility of its implementation. Y Meetings were held with Congressional staff and staff of the Office of Management and Budget to determine their views of the evaluation. Y A meeting was held with representatives of public interest groups -- the National Association of Counties the National Association of Private Industry Councils the National Governor s Association the National Association of Workforce Development Professionals and the U. S. Conference of Mayors -- to obtain additional feedback from representatives of EDWAA stakeholders. 2 This document describes major elements of the design for the evaluation as it has been revised based on information gathered through the efforts listed above. The document begins with a description of the objectives of the evaluation. It then describes the design components--the sample frame the treatment groups random assignment and the sampling plan. OBJECTIVES OF THE EVALUATION The main objective of the evaluation is to estimate the net impact of EDWAA retraining on the post-program employment and earnings of dislocated workers . Secondary objectives are to examine impacts of retraining on the characteristics of post-program jobs including wage rates and fringe benefits to describe the reemployment services and retraining received through EDWAA to measure client satisfaction with the services and to examine impacts on important subgroups. Important subgroups include individuals identified through rapid response activities individuals whose layoff was part of a mass layoff older individuals individuals with low education levels and individuals with income support for example from unemployment compensation or their spouse s earnings . DESIGN COMPONENTS Sample Frame All EDWAA applicants who are determined to be eligible for EDWAA and who apply for EDWAA services through substate area SSA program operators will be included in the sample frame. We will exclude individuals who receive services through the national reserve account. These grants are often provided to workers laid-off from specific plants and can begin and end at any time during the program year. These circumstances make it difficult to implement a design which offers groups of workers different levels of services and which enrolls a sample over a fixed period e.g. a program year . Treatment Group Design To focus the evaluation on the impact of EDWAA retraining we propose to define two treatment groups one group that is eligible to receive the full array of EDWAA services the full EDWAA group and one group eligible for EDWAA basic 3 readjustment services BRS but not EDWAA retraining the BRS-only group . New applicants determined to be eligible for EDWAA will be assigned randomly to one of these treatment groups. Under this design an estimate of the impact of EDWAA retraining can be obtained by comparing mean outcomes of the two treatment groups and dividing the difference in these outcomes by the retraining participation rate among the full EDWAA group. Since in this revised design we do not create a no- EDWAA control group we are not able to directly estimate either the separate impact of BRS or the impact of EDWAA overall. This design is a revision to the initially proposed design which included in addition to the same two treatment groups as in the current design a control group that would not be offered EDWAA services. The initial design would have provided an estimate of the overall impact of EDWAA. Estimates of the separate impacts of the major EDWAA services BRS and retraining could also have been obtained. The initial design was rejected for two reasons. First it required that eligible applicants randomly assigned to the control group be denied any EDWAA services. Local and state officials who reviewed the design expressed an aversion to an evaluation design that denied EDWAA assistance to eligible applicants based on random assignment. Second the design implicitly assumed that the control group could be denied EDWAA basic readjustment services. But comments from program operators and the contractor s observations of local programs suggest that in many sites individuals denied EDWAA basic readjustment services would be able to access similar services. Furthermore it might be difficult to deny EDWAA-sponsored basic readjustment services to control group members in sites where EDWAA funds are used to pay for services offered to a broad set of clients. This factor suggests that it would be difficult to isolate either the impact of EDWAA basic readjustment services or the impact of the EDWAA program overall. Hence the primary advantages of the alternative designs would be jeopardized by the way in which EDWAA is currently operating. 1Random assignment procedures will be adapted to each site s application process. 4 Random Assignment Random assignment will take place when individuals apply for and are determined to be eligible for EDWAA services. 1 We selected the application and eligibility determination point because it is the earliest feasible point to conduct random assignment. Selecting this early point for random assignment minimizes the likelihood that clients will develop an EDWAA service strategy that includes retraining and subsequently be denied retraining because they are assigned to the BRS-only group. This must be weighed against the possibility that some individuals assigned to the full EDWAA group may drop out of EDWAA before they ever participate in retraining. The earlier random assignment occurs the higher the dropout rate is for the full EDWAA group. A higher dropout rate will dilute the treatment by lowering the retraining rate which may reduce our ability to detect between-group differences for a given sample size. To address this issue we have increased the sample sizes to compensate for the potential of a high dropout rate when we have early random assignment. Sampling Plan To create a nationally representative sample of EDWAA eligibles we have chosen a two-stage sampling process. In the first stage we will randomly select 30 SSAs with probability of selection proportional to the caseload. In conducting the site selection we will stratify the sites by geography and size of caseload so that the sample represents the different operating environments that exist within the EDWAA program nationwide. In the second stage of sampling we will randomly assign all eligibles in each of the 30 SSAs to one of the treatment groups. We expect to use this procedure to create a national sample of approximately 9 000 EDWAA eligibles who enter EDWAA over an approximately one year period. We will randomly assign three quarters of them to the full EDWAA group and one quarter to the BRS-only group. We plan to collect baseline information on all of these individuals through the EDWAA application and a supplementary baseline 5 form and administrative records data SPIR wage records and possibly UI and ES data . We also plan to conduct follow-up interviews 30 months after random assignment with a subsample of about 4 300 sample members.