TEN_42-16_Acc.pdf

ETA Advisory File Text

TRAINING AND EMPLOYMENT NOTICE NO. 42-16 DATE May 8 2017 EMPLOYMENT AND TRAINING ADMINISTRATION U.S. DEPARTMENT OF LABOR WASHINGTON D.C. 20210 TO COMPREHENSIVE AMERICAN JOB CENTER DIRECTORS AFFILIATE AMERICAN JOB CENTER DIRECTORS DEPARTMENT OF LABOR EMPLOYMENT AND TRAINING ADMINISTRATION REGIONAL ADMINISTRATORS STATE WORKFORCE AGENCY ADMINISTRATORS STATE WORKFORCE ADMINISTRATORS WORKFORCE INNOVATION AND OPPORTUNITY ACT WIOA STATE WIOA LIAISONS WORKFORCE DEVELOPMENT BOARD STATE AND LOCAL EXECUTIVE DIRECTORS WORKFORCE DEVELOPMENT BOARD STATE AND LOCAL CHAIRS ALL EMPLOYMENT AND TRAINING ADMINISTRATION GRANTEES FROM BYRON ZUIDEMA s Deputy Assistant Secretary SUBJE CT Release and Availability of the Report Evidence-Building Capacity in State Wor kforce Agencies Insights from a National Scan and Two State Site Visits 1. Purpose. The Employment and Training Administration ETA announces the release and availability of th e report titled Evidence-Building Capacity in State Workforce Agencies Insights from a National Scan and Two State Site Visits prepared by the National Association of State Workforce Agencies NASWA and associated Training and Employment Notice TEN . 2. Background. The Workforce Innovation and Opportunity Act WIOA Pub. L. 113-128 was enacted in July 2014 to strengthen and improve the nation s public wor kforce system and help get Americans including youth and those with significant barriers to employment into high -quality jobs and careers and to help employers hire and retain skilled workers. WIOA also expands the role of evaluations within the context of evidence-based decision-making. The WIOA evaluation requirements and regulations emphasize the states responsibilities to conduct evaluations coordinate and consult with other programs and local workforce boards and cooperate with Federal evaluations and research projects . Accordingly in an effort to gain a better understanding of the capacity of State Workforce Agencies SWAs to meet these responsibilities ETA commissioned a scan by the Center for Employment Security at NASWA to provide a systemic assessment of the capacity of SWAs to conduct research and evaluation activities. 3. Assessment and Analytical Approach. To conduct the assessment of evidence-building capacity in SWAs NASWA developed a two-part study. The national scan in Part 1 of the study began with a basic question What qualifies as research and evaluations While ETA does not 2 define research and evaluation to frame the purpose of the scan and gather input from the SWAs NASWA provided the following definitions Pub lic workforce research is an empirical process by which data about workforce programs is used to develop descriptions measurements comparisons and tests of hypothesized relationships. Public workforce evaluations are empirica l analyses of program and ot her data to describe the operation of a program measure the program impacts on outcomes of policy and program interest and or determine the cost of effective ness. The national scan include d a series of forty questions to develop an understanding of stat e workforce agency research capacity. The initial set of questions were used to gather feedback on the interest or demand by SWAs governors and legislatures for the types of research and evaluations that can be produced and the kinds of state and or ou tside researcher partnerships related to funding conducting or participating in research and evaluation . The second set of questions focused on understanding current staff SWAs capacity staff levels experiences and skills for conducting research and evaluation the types and levels of funding including Workforce Data Qu ality Initiative WDQI grants available for research and evaluation a summary of research and evaluation studies produced with or without partners from calendar years CY 2011 to 2015 and plans to initiate new studies or evaluations with or with out outside contractor or partner support during calendar years 2016 through 2018. The third set of questions asked the states to identify individual stu dies and evaluations including the authors and partners research methods used data sets accessed central research question addressed and approximate cost of the study. Part 2 of the report includes summaries from in depth site visits to tw o states Washington and Ohio . The site vis its consisted of semi -structured interviews with state entities that facilitate workforce development research and utilize longitudinal administrative data sets. Both states provided extensive background and historical information related to the evolution of their longitudinal administrative data systems to support research studies and evaluations described the roles and functions of the different organizations within their respect ive states that conduct coordinate or support research and evaluation on workforce programs explained how data sharing requests are processed and data is confidentially secured and discussed specific studies assessments and surveys conducted on workforce programs. The states also shared a dditional information about compute r systems and software staffing program and budget environments and describe d relationships between research data centers state workforce investment boards research plans and management use of evidence -based policy -formation supported by the research and evaluation entities in each state. 4. Findings on Evidence -Building Capacity from the Scan and Site Visits . Insights from the national scan focused on pressing research questions within SWAs state workforce agency capacity to address the demand f or information current staff capacity levels and funding levels. More specifically the study found that many state agencies need assistance with funding and technical expertise to promote SWAs as learning organizations that use evidence to drive 3 decisi on making. A lmost all of the 41 states who participated in the scan report ed there is consistent demand for workforce research and evaluation . Analyses of the assessment found that Pressing research questions were heavily weighted toward 1 understan ding labor markets 2 measuring program performance and outcomes and 3 measuring program impacts and effectiveness. Three quarters of the SWAs reported the capacity to initiate and advance research and evaluation efforts. Eighty percent of the resp onding agencies report ed partnering with or relying on outside researchers to conduct at least one research or evaluation effort from CY 2011 through 2015. When asked to describe current internal research and evaluation staff capacity including experie nce and research skills o Twenty percent of the agencies report their staff capacity is inadequate o Forty -four percent report staff capacity is fair o Twenty -nine percent report their research staff capacity is adequate and o The remaining states were at either end of the spectrum with nonexistent 2 percent or very adequate five percent research staff capacity. When further asked about current staff capacity to assess program impacts very few of the state agencies report ed having sufficient capacity to evaluate long -term impacts. In fact about half report ed no capacity or no awareness of capacity to conduct these types of research. When asked about staff levels most agencies report ed estimates of full -time equivalent staff currently worki ng on research and evaluation projects. Furthermore three agencies report they have zero research staff a quarter of the agencies report less than one 1 full time employee FTE and half the agencies report two 2 or less FTE . The scan o f staff capa city did not account for agencies that rely on outside research partners or contractors that support workforce research activity. Twenty percent of the 41 agencies report ed they spent zero dollars on research and evaluations in calendar year 2015. Anothe r 20 percent report they spent less than 100 000 and 37 percent report spending more than 100 000. Agency funding sources varied and some agencies drew upon a range of funding sources . Notably SWAs reported that an important source of funding has bee n the U.S. Department of Labor s Workforce Data Quality Initiative WDQI grants to build staff capacity while also strengthening their research data infrastructure. Almost thirty percent of agencies report ed they consider their research and evaluation expenditures adequate. 4 In addition to a history and culture of using workforce research to inform policy and practice the common factors that contribute to the substantial workforce research activity evident in Ohio and Washington include o Development o f a cross -agency longitudinal administrative data set covering a range of public programs and including Unemployment Insurance wage record data o A long history of sharing data between the state workforce development and education agencies o A neutral thir d-party entity to collect data across agencies and govern the longitudinal administrative data set o The third -party entity governing the longitudinal data set employs staff with great knowledge of the individual agency data sets e.g. former agency staff who have worked with the data for a long time o Washington has enacted legislation to institutionalize its cross -agency longitudinal administrative data set and the key roles and responsibilities for the entities engaged in data and research efforts leg islation is being pursued in Ohio in order to help institutionalize its model o Data governance data access procedures and security standards have been addressed and maintained as a high priority o Buy -in leadership and support from the office of the go vernor and agency heads o Strategies to develop and maintain trust and information sharing among state agencies and their staff o Data and research staff work in environments that are mission -driven collegial and allow research staff room to innovate th us retaining talented staff and o Objective research products produced in a politically -neutral environment upon which policymakers can rely for information to inform decisions. 5. Additional Resources from the National Scan . The assessment provides qualitative and descriptive analyses from 41 states that participated in the national scan. In addition to the analyses the report includes a list of state workforce agency publication websites Appendix A a table of state workforce agency research stu dies and evaluations from CY 2011 -2015 Appendix B a summary of the five rounds of WDQ I grant funding Appendix C the evaluation regulations at 682.220 Appendix D the outreach communication about the national scan Appendix E the primary points of contacts by job position Appendix F a summary of pressing research questions Appendix G Ohio s Research Agenda Appendix H Mississippi s Research Agenda Appendix I a list of state workforce agency research units Appendix J a list of SWAs r esearch partner or contractors Appendix K and the draft OHIOANALYTICS GOVERNANCE MANUAL Appendix L . NASWA also plans to release state agency profiles of the workforce research studies and evaluations collected from the states that participated in the scan. 6. Inquiries . To view an abstract of th is publication as well as to download the executive summar y and full report in PDF versions visit the ETA Research Publication Database Web site at http wdr.doleta.gov research keyword.cfm .