TEN 29-23-A.pdf

ETA Advisory File Text

TRAINING AND EMPLOYMENT NOTICE NO. 29-23 DATE May 6 2024 EMPLOYMENT AND TRAINING ADMINISTRATION U.S. DEPARTMENT OF LABOR WASHINGTON D.C. 20210 TO STATE WORKFORCE AGENCIES STATE WORKFORCE ADMINISTRATORS STATE WORKFORCE LIAISONS STATE AND LOCAL WORKFORCE BOARD CHAIRS AND DIRECTORS FROM JOS JAVIER RODR GUEZ Assistant Secretary SUBJECT Evaluation Requirements and Flexibilities f or States under Title I of the Workf orce Innovation and Opportunity Act WIOA 1.Purpose. To clarif y existing requirements and f lexibilities f or states in conducting evaluations as a f orm of research pursuant to the regulations at 20 CFR 682.220 consistent with WIOA Sections 116 e 129 134 168 169 29 U.S.C. 3141 e 3164 3174 3223 3224 . 2.Action Requested. The Employment and Training Administration ETA requests that state workf orce agencies review this notice and revise as needed planning and operations f or conducting and reporting on evaluations. 3.Summary and Background. a.Summary This notice af f irms and summarizes the f ollowing the role and purposes of evaluation under the regulations at 20 CFR 682.220 as a f orm of research requirements and options f or states in conducting evaluations reporting requirements expectations regarding state level coordination and cooperation in f ederal evaluations. The notice also provides inf ormation on technical assistance resources f or states regarding evaluation methods and completed studies. b.Background States are required to conduct evaluations under the regulations at 20 CFR 682.220 and to identif y evaluation activities in their WIOA State Plans and annual perf ormance reports. However the Employment and Training Administration ETA in the U.S. Department of Labor the Department f ound that in some state plans and annual perf ormance reports there appeared to be dif f erent interpretations as to what constitutes an evaluation thus indicating a need f or additional inf ormation and clarif ication. The need f or such clarif ication is also supported by f indings f rom a study of WIOA implementation commissioned by the Department s Chief Evaluation 2 Of f ice which documented that some state administrators were unclear as to evaluation requirements and were awaiting f urther inf ormation on them.1 This notice is thus intended to clarif y the existing requirements under the regulation but also to highlight options f or states to utilize when designing studies. States research could f or example identif y key aspects of various services such as those discussed in Training and Employment Guidance Letter TEGL 21-22 2 or operational f actors that lead to higher employment and earnings f or participants including those with signif icant barriers to employment or f rom marginalized or underserved groups. Further states might potentially conduct evaluations using new data sources such as those in longitudinal databases developed under Workforce Data Quality Initiative3 or use new analytical and statistical methods f or which training has been provided to states by ETA.4 Overall this notice is intended to support the engagement of the workf orce system in evaluations as a f orm of research related to ef f orts to build and nurture a culture of evidence 5 as articulated in Of f ice of Management and Budget OMB guidance f or f ederal agencies regarding the Foundations of Evidence in Policy-Making Act of 2018. In building a culture of evidence through evaluations States as well as local boards and practitioners may be able to identif y new and more ef f ective approaches to delivering services in a state as well as adding to the larger body of evidence on what works f or customers with dif f erent pref erences and needs. 4.Definition Requirements Flexibilities and Technical Assistance for State Evaluations under 20 CFR 682.220. Under the regulation states are required to conduct evaluations of activities under the WIOA title I core programs in order to promote continuous improvement research test innovative services and strategies and achieve high levels of perf ormance and outcomes 6 consistent with the f ollowing a.Definition State evaluations under 20 CFR 682.220 are a f orm of research like evaluations that the Department conducts under WIOA Section 169 7 and are 1 Mack M. and Dunham K. Implementation Study of the Workforce Innovation and Opportunity Act WIOA . Performance Accountability Eligible Training Providers Labor Market Information and Evaluation Requirements Mathematica Policy Research November 2020 pp. xvi and 32 found at Performance Accountability Eligible Training Providers Labor Market Information and Evaluation Requirements dol.gov 2 See TEGL 21-22 Increasing Equitable Service Access and Employment Outcomes for all Jobseekers in WIOA Adult and Dislocated Worker Programs dated June 9 2023 3 Funding for these grants has been authorized by Congress over multiple years since 2010. For more information see Workforce Data Quality Initiative WDQI U.S. Department of Labor dol.gov 4 Examples of such training include sessions in 2023 on Data Analytics and for the Aligned Case Management Institute offered by ETA in collaboration with the National Association of State Workforce Agency s Workforce Information Technology Support Center WITSC. 5 As required for DOL and other Federal evaluations per Office of Management and Budget OMB M-21-27 Memorandum on Evidence-Base d Policymaking Learning Agendas and Annual Evaluation Plans p. 2 https www.whitehouse.gov wp- content uploads 2021 06 M-21-27.pdf 6 20 CFR 682.220 a As required by 682.200 d States must use funds reserved by the Governor for statewide activities to conduct evaluations of activities under the WIOA title I core programs to promote continuous improvement research and test innovative services and strategies and achieve high levels of performance and outcomes. 7 20 CFR 682.220 b 4 To the extent feasible be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA sec. 169 regarding title I programs and other employment-relate d programs WIOA sec. 242 c 2 D regarding adult education sec. 12 a 5 14 and 107 of the Rehabilitation Act of 1973 29 U.S.C. 709 a 5 711 727 applied with respect to programs carried out under title I of that Act 29 U.S.C. 720 et seq. and the investigations by the Secretary of Labor under sec. 10 b of the Wagner-Peyser Act 29 U.S.C. 49i b . 3 characterized per OMB s guidance by the use of systematic data collection and analysis of one or more programs policies and organizations 8 to f ind answers to questions related to ef f ectiveness and ef f iciency. State evaluations as a f orm of research are distinct f rom other management oversight and data f unctions that are required under WIOA title I such as perf ormance assessment and accountability f unctions monitoring or certif ication of local programs and routine labor market inf ormation collection and reports under the Wagner-Peyser Act as amended by WIOA title III . However evaluation research can involve use of the data that arise f rom those other management responsibilities as demonstrated in several state studies.9 Also f indings and observations f rom such management f unctions may in turn stimulate questions and topics to explore in greater depth through evaluation research. b.Focus Evaluations conducted under the regulation must concern activities under WIOA title I core programs i.e. the Adult Dislocated Worker and Youth programs either alone or in conjunction with services under other programs.10 c.Requirements and Flexibilities for States Evaluations Evaluation as a f orm of research presents States with an opportunity f or f inding answers to critical questions of interest answers that are not available f rom other f orms of data collection and oversight to improve relevant outcomes f or the programs and customers experience and satisf action. Evaluation research related to title I core programs is driven by the concerns and interests of states and does not have to f ollow a pre-determined approach or specific topics except f or evaluations of Pay f or Perf ormance contract strategies11 . However the regulations identif y several data sources that states must use when appropriate such as analysis of customer f eedback and outcome and process measures in the statewide workf orce development system 12 as well as use of the most rigorous analytical and statistical methods that are reasonably feasible such as the use of control groups. 13 ETA expects states to consider use of such data sources and methods in their evaluation projects but if states f ind those data sources are not appropriate or the use of rigorous methods is not reasonably f easible states have a broad range of options as to topics covered relevant to title I core programs types of evaluations methods data sources phases or components and overall duration as discussed in the regulations and in the examples below. Information on evaluation types data sources and methods can be f ound in the Evaluation Toolkit Key Elements f or State Workf orce Agencies and in various resources identif ied in Attachment III . 8 Supra Note 5 quoting the definition of evaluation in 5 U.S.C. 311 3 as amended by sec. 101 of Public Law 115-435 the Foundations for Evidence-Base d Policymaking Act of 2018. 9 See examples A Process Evaluation of the Integration of Title I and Title II in New Jersey and Performance and Equity in Colorado s WIOA Programs A Sequential Mixed-Methods. Both studies used data collected under WIOA core programs. 10 20 CFR 682.220 a and 20 CFR 682.220 e 1 . 11 See page 14 of TEGL 8-20 Pay for Performance PFP Guidance for WIOA title I subtitle B dated December 7 2020. Evaluations of the design and performance of WIOA Pay-for-Perform an ce PFP contract strategies are required under 20 CFR 683.540 c 2 and must conform to requirements under 20 CFR 682.220 and this notice. 12 20 CFR 682.220 b 2 . 13 20 CFR 682.220 b 3 . 4 i. Types of Studies Data and Methods There are multiple types of evaluation studies that states may undertake including as per the regulation process and outcome studies studies of pilot and demonstration projects that have an evaluative component analyses of administrative and programmatic data impact and benef it-cost analyses and use of rigorous designs to test the ef f icacy of various interventions. 14 These types of evaluations are similar to those conducted by the Department under WIOA section 169 which also include other types of studies such as formative implementation and rapid cycle evaluations all of which can also be undertaken by states. The type of study and its design typically determine the types of data and methods used to analyze the data. States have many options as to data types which can be quantitative or qualitative arise f rom administrative and other existing sources or be collected specif ically f or an evaluation. These last might include f or example data f rom individual or f ocus group interviews surveys document reviews observations during site visits or f rom various reporting systems or databases including f or example economic or geo-spatial data. There are also multiple methods that can be used in analyzing quantitative and qualitative data including a broad array of statistical approaches or new f orms of analysis associated with processing qualitative data. ii.Topics States have wide latitude as to the topics and scope including geographical scope of their evaluations which can include but are not limited to the f ollowing Participants and participant groups and their characteristics needs and views Service receipt and outcomes as well as key f eatures of services individually or collectively regarding all types of interventions such as career guidance training supportive services or employer engagement Business services and employer perspectives Existing services and new or innovative services or program f eatures relevant to title I programs including those conducted with discretionary grant f unds Administrative or management f unctions Staf f ing f eatures practices or prof essional development Costs and cost ef f iciency and Programs or services at the local regional or state level. 14 20 CFR 682.220 e 2 . 5 iii.Phases Components and Duration State evaluations can involve multiple phases components and activities which can include but are not limited to a literature or evidence review f easibility study planning research coordination design data collection analysis and report preparation clearance and dissemination. 15 Research that addresses basic or f oundational questions can be conducted if it is part of a larger evaluation designed to promote program improvement test new approaches or lead to higher levels of perf ormance and outcomes. Further evaluation research projects can be conducted over multiple program years though ETA expects states to complete evaluations as expeditiously as possible. d.Evaluation Reports and their Dissemination States must annually prepare and submit reports to State and Local Workf orce Development Boards containing the results of their evaluation research as available and the State must make those reports available to the public including by electronic means such as state workf orce agency or state board websites.16 Evaluation projects sometimes involve creation of multiple products such as interim and f inal reports brief s or practice guides all of which should be provided to boards and the public. ETA expects states to disseminate all such products and reports in the year they are each completed over the course of all evaluation projects including those of long duration. States are also encouraged to disseminate f indings through multiple means including webinars social media and live presentations to promote awareness of the studies and lead to utilization of these f indings if warranted in state policy and practices at the local level. e.Identifying Evaluation Activities in the State Plan and the Annual Performance Report Narrative States must provide inf ormation regarding evaluation ef f orts f or title I core programs in two dif f erent WIOA-required documents as discussed below. i.Unified or Combined State Plan. Each State s plan as per TEGL 4-23 and the updated version of the elements required in the plans f ound at WIOA State Plan Requirements 17 must describe 15 20 CFR 682.220 e 3 . 16 20 CFR 682.220 c . 17 See Required Elements for Submission of the Unified or Combined State Plan and Plan Modifications under the Workforce Innovation and Opportunity Act Program Years 2024-2027 dated October 31 2023. This material can also be found at OMB Control 1205-0522 approved through March 31 2026 6 Under III b 4 C How the state will conduct evaluations and research projects of activities carried out in the State under WIOA core programs How such projects will be coordinated with and designed in conjunction with State and local boards and with State agencies responsible f or the administration of all respective core programs and f urther How the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA and Under VI a 2 B How the state intends to use Governor s set-aside f unding f or mandatory and discretionary activities including how the state will conduct evaluations of title I Adult Dislocated Worker and Youth activities. ii.Annual Performance Report Narrative. States must also include in the narrative section of the annual perf ormance report per TEGL 5-18 18 brief descriptions of Current or planned evaluation and related research projects including methodologies used Ef f orts to coordinate the development of such projects with WIOA core programs other state agencies and local boards A list of completed evaluations related reports and links to where they were made accessible to the public electronically State ef f orts to provide data survey responses and timely site visits f or f ederal evaluations and Any continuous improvement strategies utilizing results f rom studies and evidence-based practices evaluated. Note As per TEGL No. 08-2019 States are required to conduct evaluations of the design and perf ormance of Pay f or Perf ormance PFP contract strategies when such contract strategies exist in a state. Such evaluations must comply with the evaluation provisions in 20 CFR 682.220 as discussed in this notice and f or that reason inf ormation on the PFP evaluations should be included in the section of the narrative report on evaluations and cover the items listed above as relevant . States must also submit inf ormation on PFP projects f or the Annual Perf ormance Report on the ETA-9174 f orm which requires discussion of the research questions addressed by the evaluation s and the challenges in evaluating the PFP contract strategies.20 18 See TEGL 5-18 Workforce Innovation and Opportunity Act WIOA Annual Statewide Performance Report Narrative dated November 7 2018. 19 See page 14 of TEGL 8-20 Pay for Performance PFP Guidance for WIOA title I subtitle B dated December 7 2020 20 Ibid page 12. 7 f.Funding Requirements and Options Conducting evaluations of title I core program activities is a required statewide activity and must entail use of at least a small amount of f unds f rom the state set-aside. However ETA recognized that states must balance priorities in their spending of those f unds and identif ied areas f or f lexibility in 20 CFR 682.220 f 21 f or states to f ind or combine f unding f rom other sources. Options regarding f unding sources include the f ollowing i.Collaborating with state agencies administering other WIOA title II-IV core programs to participate in a joint evaluation as determined through the coordination processes required with those programs and ii.Using public or private sources consistent with f ederal and state law regulations and guidance f rom other grants cooperative agreements or contracts including f or evaluations whether f rom the Department other Federal or state agencies and philanthropic organizations or f oundations States may also seek to minimize costs related to evaluations. Examples of such cost- saving measures that states utilized in the past are discussed below though states may use other approaches i.If participating in a f ederal evaluation relating to a title I core program a state may engage with the f ederal contractor and under a separate contract provide small amounts of f unds to generate an analysis and report containing results related to the state ii.Partnering with other agencies in the state to create a centralized capacity to evaluate multiple programs and house various data sets an approach taken in several states as identif ied in a study on state evaluation capacity conducted by the National Association of State Workf orce Agency NASWA 22 or partnering with the relevant state agency in conducting an evaluation of title I services provided to individuals who were identif ied under Reemployment Services and Eligibility Assessments RESEA grants or iii.Partnering with workf orce agencies in other States to jointly f und a multistate study using a common design and data collection methods but generating separate analyses and a report f or each State. 21 20 CFR 682.220 f In funding evaluations conducted under paragraph a of this section States are permitted but not required to 1 Use funds from any WIOA title I IV core program to conduct evaluations as determined through the processes associated with paragraph b 1 of this section and 2 Use or combine funds consistent with Federal and State law regulation and guidance from other public or private sources to conduct evaluations relating to activities under the WIOA title I IV core programs. Such projects may include those funded by the Department of Labor and other Federal agencies among other sources. 22 See TEN No. 42-16 and ETA Research Reports Evidence-Building Capacity in State Workforce Agencies - 2017. 8 g.Coordination with Workforce Development Boards and Other State Agencies One of the major ref orms under WIOA is a strengthened emphasis on coordination and partnership. That emphasis extends to state evaluations which are required to be coordinated with and designed in conjunction with i.State and Local Workf orce Development Boards and ii.State agencies responsible f or the administration of the core programs under the other titles of WIOA i.e. the Employment Service Adult Education and Family Literacy and Vocational Rehabilitation programs .23 Although the nature of coordination regarding evaluations is not def ined in the regulations or statute ETA expects SWAs to undertake reasonable ef f orts to coordinate with boards and state agencies. These efforts might involve f or example inf orming State and local boards about plans to conduct evaluations and providing opportunities f or meaningf ul input into possible topics and the design of such studies. Local boards may also be of f ered opportunities to allow new approaches to be tested in their local area and asked to assure that data is properly collected and provided f or state or f ederal studies. SWAs are also permitted to coordinate evaluation activities with other agencies at the state or local level such as those administering Unemployment Insurance tribal governments and organizations serving American Indians Temporary Assistance f or Needy Families the Supplemental Nutrition Assistance Program and career and technical education programs. Such agencies might be partners in conducting studies of title I core programs in conjunction with other programs. h.Coordination with Federal Evaluations State agencies are required to the extent f easible to conduct evaluations in coordination with those conducted by the Department and the U.S. Department of Education ED regarding programs under WIOA.24 As with state-level coordination the meaning of coordination with those f ederal evaluations is not def ined in the statute or regulation but ETA expects states to make a reasonable ef f ort to coordinate with such evaluations in two ways 23 20 CFR 682.220 b 1 Be coordinated with and designed in conjunction with State and Local WDBs and with State agencies responsible for the administration of all core programs. 24 20 CFR 682.220 b 4 To the extent feasible be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary ofEducation under WIOA sec. 169 29 U.S.C. 3224 regarding title I programs and other employment-related programs WIOA sec. 242 c 2 D 29.U.S.C. 3332 c 2 D regarding adult education sec. 12 a 5 14 and 107 of the Rehabilitation Act of 1973 29 U.S.C. 709 a 5 711 727 applied with respect to programs carried out under title I of that Act 29 U.S.C. 720 et seq. and the investigations provided by the Secretary of Labor under sec. 10 b of the Wagner-Peyser Act 29 U.S.C. 49i b . 9 i.Exploring the active and historical evaluations conducted by the Department 25 and ED related to workf orce programs and services delivered under all titles of WIOA. States may also want to become f amiliar with relevant evaluations conducted by other f ederal agencies such the U.S. Department of Health and Human Services or by the ED s Institute of Education Sciences as noted in Attachment III and ii.Becoming f amiliar with the Department s Evaluation Policy see Attachment II and at U.S. Department of Labor Evaluation Policy . The policy identif ies the principles of rigor relevance transparency independence and ethics in human subject protections . States are strongly encouraged to f ollow similar principles to ensure their evaluations will have valid and reliable data are widely available and protect the privacy of individuals who are the subjects of research. i.Providing Data for Federal Evaluations States are also required to cooperate 26 to the extent practicable in evaluations and research conducted by the Department and ED related to employment and training programs adult education and literacy and Vocational Rehabilitation.27 Cooperation in such f ederal evaluations involves timely provision of i.Data in accordance with the Department s established privacy protections ii.Responses to surveys iii.Dates and scheduling of site visits and iv.Data and survey responses f rom local sub-grantees and state and local boards and assistance with conduct of site visits involving sub-grantees and local boards.28 States are required to assume responsibility f or promoting data provision by local subgrantees due to the States role in monitoring and oversight of local boards and local one-stop systems. State workf orce agencies may also work with the state agencies responsible f or local one-stop partner programs to promote those partners timely provision of data survey responses and site visits should such data be needed f or various f ederal evaluation and research projects. As noted in the regulations 29 if a state determines it is not f easible to cooperate in timely provision of data the Governor must inf orm the Secretary in writing and explain the reasons why it is not practicable. In such circumstances the state must cooperate with the Department in developing a plan or strategy to mitigate or overcome the problems preventing timely provision of data survey responses and site visits. 25 Examples of reports from the Department s evaluations can be found at https www.dol.gov agencies oasp evaluation completedstudies 26 20 CFR 682.220 d . 27 Sec. 12 a 5 14 and 107 of the Rehabilitation Act of 1973 29 U.S.C. 709 a 5 711 727 applied with respect to programs carried out under title I of that Act 29 U.S.C. 720 et seq. . The vast body of research related to vocational rehabilitation is now under the Administration of Community Living in the U.S. Department of Health and Human Services having been transfer r e d under WIOA from the Education Department 28 20 CFR 682.220 d 1 . 29 20 CFR 682.220 d 3 . 10 j.Technical Assistance and Dissemination State agencies ability to conduct evaluations depends on many f actorsI including their institutional capacity and available f unds. There is wide variation among States on these f actors as seen in the study conducted by NASWA cited above. A recent report f rom NASWA Evidence Building Capacity in State Workf orce Agencies A COVIa-19 Pulse Survey 30 identif ied technical assistance and training needs regarding advanced data analysis research design and question f ormulation report preparation and communicating f indings among other needs. To build the evaluation capacity of state agencies ETA currently provides technical assistance through 1 extensive on-line resources at the Evaluation and Research Hub in Workf orceGPS ETA s online technical assistance and training website and 2 a series of on-line training sessions f or state teams as part of the annual Evaluation Peer Learning Cohort EvalPLC project. The sessions cover dif ferent types of evaluations how to f ind existing research studies and utilize themI planning and management activities related to conducting evaluations and development of a capstone evaluation project. ETA also disseminates inf ormation on findings f rom evaluation reports via webinars and quarterly Research and Evaluation R E Notes as well as posting evaluation reports in two databases the ETA Research Publications Database and the Workf orce System Strategies database also in the WorkforceGPS website . The Department s Chief Evaluation Of f ice also posts on its website completed evaluation reports and resources to inf orm evaluation methods and reports. These resources along with a list of other f ederal agency databasesI which provide inf ormation on research studies related to employment and training programsI can be f ound in Attachment III. 5.Inquiries. For f urther inf ormation please direct inquiries to the appropriate ETA Regional Of f ice. 6.References. See Attachment I. 7.Attachment s . Attachment I Ref erences Attachment II U.S. Department of Labor Evaluation Policy Attachment III Selected Resources on Evaluation Methods and Evaluation Reports Relevant to WIOA title I Programs 30 NASWA Evidence Building Capacity in State Workforce Agencies A COVID-19 Pulse Survey https www.naswa.org system files 2021-03 evidencebuildingcapacityinstateworkforceagencies - acovid-19survey.pdf . Attachment I References Workforce Innovation and Opportunity Act WIOA Pub. L.113-128 enacted July 22 2014 including the f ollowing sections o116 e 129 134 168 169 242 c 2 D 29 U.S.C. 3141 e 3164 3174 3223 3224 . oSections 12 a 5 14 and 107 of the Rehabilitation Act of 1973 29 U.S.C. 709 a 5 711 727 applied with respect to programs carried out under title I of that Act 29 U.S.C. 720 et seq. oWagner-Peyser Act 29 U.S.C. 49i b WIOA Regulations 20 CFR Sections o603.6 b 8 o682.200 d o682.210 l o682.220 o683.540 a 2 and c 2 regarding Pay-f or-Perf ormance contracts Training and Employment Notice 42-16 Release and Availability of the Report Evidence-Building Capacity in State Workforce Agencies Insights from a National Scan and Two State Site Visits dated May 8 2017 Training and Employment Guidance Letter TEGL No. 5-18 Workforce Innovation and Opportunity Act WIOA Annual Statewide Perf ormance Report Narrative dated November 7 2018 TEGL No. 8-20 Pay f or Perf ormance PFP Guidance f or WIOA title I subtitle B dated December 7 2020 TEGL No. 22-12 Increasing Equitable Service Access and Employment Outcomes f or All Jobseekers in Workf orce Innovation and Opportunity Act Adult and Dislocated Worker Programs dated June 9 2023 Information Collection Request OMB Control No. 1205-0522 Required Elements for Submission of the Unified or Combined State Plan and Plan Modifications under the Workforce Innovation and Opportunity Act. Approved through March 31 2026 The Foundations for Evidence-Based Policymaking Act of 2018 Public Law No. 115-435 dated January 14 2019 Office of Management and Budget M-21-17 Memorandum Evidence-Based Policymaking Learning Agendas and Annual Evaluation Plans dated June 30 2021 https www.whitehouse.gov wp-content uploads 2021 06 M-21-27.pdf I-1 Attachment II U.S. Department of Labor Evaluation Policy This evaluation policy statement presents key principles that govern the Department of Labor s DOL planning conduct and use of program evaluations. The policy represents a commitment to conducting rigorous relevant evaluations and to using evidence f rom evaluations to inf orm policy and practice. DOL seeks to promote rigor relevance transparency independence and ethics in the conduct of evaluations. This policy addresses each of these principles. The mission of the Department of Labor is to ...f oster promote and develop the welf are of the wage earners job seekers and retirees of the United States improve working conditions advance opportunities f or profitable employment and assure work-related benef its and rights. The importance of these goals demands that we continually innovate and improve and that we evaluate the perf ormance and effectiveness of our programs and activities. Through evaluation DOL can learn systematically so that we can make our services as ef f ective as possible. Evaluation produces one type of evidence. A learning organization with a culture of continual improvement requires many types of evidence including not only evaluation but also descriptive research studies perf ormance measurement f inancial and cost data survey statistics and program administrative data. Although this policy f ocuses on evaluation the principles apply to the development and use of other types of evidence as well. The emphasis on evidence i s meant to support not inhibit innovation improvement and learning. The intent is to integrate the use of evidence and opportunities f or f urther learning into all activities. Where an evidence base is lacking evidence will be developed through strong evaluations and analysis. Where evidence exists it will be used to encourage replication studies. While much of DOL s evaluation activity is overseen by the Chief Evaluation Of f ice CEO in the Of f ice of the Assistant Secretary f or Policy OASP agencies and program of f ices within DOL also sponsor evaluations through dedicated contracts or as part of their grant-making typically in coordination with the CEO. Discretionary grant f unding opportunity announcements f rom DOL will include evidence provisions that specif y how activities will be evaluated. Successf ul applicants will be required to cooperate with and participate in evaluations. As legally allowed programs with waiver authorities will require evaluations as a condition of waivers. Rigor DOL is committed to using the most rigorous methods that are appropriate and f easible within statutory budget and other constraints. Rigor is required f or all types of evaluations including impact and outcome evaluations implementation and process evaluations descriptive studies and f ormative evaluations. Rigor requires ensuring that inf erences about cause and ef f ect are well f ounded internal validity requires clarity about the populations settings or circumstances to which results can be generalized external validity and requires the use of measures that accurately capture the intended inf ormation measurement reliability and validity . DOL maintains an evaluation workforce with training and experience appropriate f or planning and overseeing a rigorous evaluation portf olio recruiting staf f with advanced academic degrees and experience in disciplines such as public policy economics and sociology. DOL also II-1 provides prof essional development opportunities so staf f can keep their evaluation and methodological skills current. Relevance Evaluation priorities should take into account legislative requirements and the interests and needs of leadership specific agencies and programs program of f ice staf f and leadership and DOL partners such as states territories tribes and grantees the populations served researchers and other stakeholders. Evaluations should be designed to address DOL s diverse programs customers and stakeholders and DOL should encourage diversity among those carrying out the evaluations. Transparency DOL will make inf ormation about evaluations and f indings f rom evaluations broadly available and accessible typically on the Internet. This includes identif ying the evaluator releasing study plans and describing the evaluation methods. DOL will release results of all evaluations that are not specif ically f ocused on internal management legal or enf orcement procedures or that are not otherwise prohibited f rom disclosure. Evaluation reports will present all results including f avorable unf avorable and null f indings. DOL will release evaluation results timely usually within two months of a report s completion and will archive evaluation data f or secondary use by interested researchers e.g. public use f iles with appropriate data security protections . Independence Independence and objectivity are core principles of evaluation. Agency and program leadership program staf f stakeholders and others should participate in setting evaluation priorities identif ying evaluation questions and assessing the implications of f indings. However it is important to insulate evaluation f unctions f rom undue inf luence and f rom both the appearance and the reality of bias. To promote objectivity DOL protects independence in the design conduct and analysis of evaluations. After technical peer review the Chief Evaluation Of f icer has authority to approve release and disseminate evaluation reports. Ethics DOL-sponsored evaluations will be conducted in an ethical manner and saf eguard the dignity rights saf ety and privacy of participants. Evaluations will comply with both the spirit and the letter of relevant requirements such as regulations governing research involving human subjects. Effective November 2013 II-2 Attachment III Selected Resources on Evaluation Methods and Evaluation Reports Relevant to WIOA title I Programs Resources to Inform Evaluation and Research Methods 1 U.S. Department of Labor ETA Evaluation and Research Hub EvalHub in Workf orceGPS. 2 U.S. Department of Labor Chief Evaluation Of f ice https www.dol.gov asp evaluation Resources.htm. 3 U.S. Department of Education What Works Clearinghouse https ies.ed.gov ncee wwc PracticeGuides 4 U. S. General Accounting Of f ice Applied Methods and Research Designing Evaluations 2012 Revision GAO-12-208G Designing Evaluations 2012 Revision 5 U. S. Department of Health Human Services Of f ice of Policy Research and Evaluation Methods and Tools https www.acf .hhs.gov opre topic methods-and-tools Current and Past Evaluations Relevant to WIOA Title I Core and Other Workforce Development and Education Programs 1 Databases administered by the U.S. Department of Labor A.Of f ice of Policy Development and Research Division of Research Evaluation ETA Research Publications Workf orce System Strategies WSS B.Chief Evaluation Of f ice Clearinghouse f or Labor Evaluations and Research CLEAR Current studies https www.dol.gov asp evaluation CurrentStudies.htm Completed studies https www.dol.gov asp evaluation CompletedStudies.htm 2 Evaluations related to WIOA title II regarding Adult Education and Family Literacy under WIOA Section 242 c 2 D conducted by the U.S. Department of Education https www2.ed.gov about of f ices list ovae pi proginit.html and Institute of Education Science s educational interventions https ies.ed.gov ncee search q adult education and https ies.ed.gov ncee wwc FWW Results f ilters Postsecondary. 3 Evaluations and other research related to WIOA title IV Vocational Rehabilitation III-1 Services 1 now under the U.S. Department of Health Human Services https www.acl.gov node 455 and the databases f ound at https www.acl.gov aging-and-disability-in-america data-and-research nidilrr- publications-and-resources 4 Other Relevant Sources Databases A. U.S. Department of Health Human Services Pathways to Work Evidence Clearinghouse Self-Suff iciency Welf are Employment B. U.S. Department of Education Institute of Education Sciences National Center f or Education Research Research Networks C.The Of f ice of Management and Budget s Evaluation Site Evaluation.gov Home and U.S. Department of Labor s page Evaluation.gov Dept Of Labor. D.U.S. Department of Labor s tool f or looking at WIOA title I core program data WIOA by the Numbers interactive-data-analysis 1 Vocational Rehabilitation evaluations are conducted under sections 12 a 5 14 and 107 of the Rehabilitation Act of 1973 29 U.S.C. 709 a 5 711 727 applied with respect to programs carried out under title I of that Act 29 U.S.C. 720 et seq. . III-2