ten_20_11_acc.pdf

ETA Advisory File Text

NO. 20-11 TRAINING AND EMPLOYMENT NOTICE DATE December 23 2011 TO ALL STATE WORKFORCE LIAISONS ALL STATE AND LOCAL W ORKFORCE AGENCIES ALL STATE ONE-STOP CAREER CENTER SYSTEM LEADS ALL STATE AND LOCAL WORKFORCE INVESTMENT BOARDS FROM JANE OATES s Assistant Secretary SUBJECT Release and Availability of Employment and Training Administration ETA Occasional Paper Evaluation of the Technology-Based Learning Grants Final Report 1. Purpose . To announce the release and availability of the ETA Occasional Paper Evaluation of the Technology-Based Learning Grants Final Report prepared by Social Policy Research SPR Associates. 2. Background . Technology-Based Learning TBL is usually defined as learning that takes place via some form of electronic technology typically a computer with materials accessed over the Internet or on a computer in a computer lab. TBL is essentially synonymous with several other terms in common usage including e-learning. Practically speaking TBL is becoming increasingly inseparable from the Internet but in a strict sense TBL is broader and more inclusive than terms referring to learning that occurs via the Internet such as online learning. In June 2008 ETA released a Solicitation for Grant Applications SGA to provide 10 million in funding for TBL projects throughout the country. Based on responses to this SGA ETA awarded funds in January 2009 to 20 grantees in 16 states to develop and implement TBL projects over a three-year period. These 20 grantees included nine community colleges five universities four private non-profit organizations one of which was affiliated with a university a state workforce agency and a local workforce investment board WIB . Each grantee planned a project focused on a particular high-growth industry the most common of these were health care and information technology. The evaluation featured the collection and analysis of both qualitative and quantitative data. Qualitative data collection occurred primarily through two sets of site visits to grantees. The six grantees that implemented their programs within the first 10 months of the grant were visited in the fall of 2009 these grantees are referred to as Cohort I . The 14 remaining grantees referred to as Cohort II were visited during the spring and summer of 2010. Quantitative data collection consisted of requesting copies of the quarterly performance reports submitted by grantees to ETA. EMPLOYMENT AND TRAINING ADMINISTRATION U.S. DEPARTMENT OF LABOR WASHINGTON D.C. 20210 3. Report Highlights . This final report presents findings based on all data collected during the evaluation. The report begins with an overview of the training programs and then focuses individual chapters on program design and development instructional methods and course structures and partnerships. The report concludes with a brief discussion of preliminary outcomes and lessons learned that have emerged from the implementation of the programs. Highlights of the report are provided below. Overview of Training Programs The characteristics of TBL programs varied considerably. Depending on the program the duration of training could be anywhere from 30 minutes to two years. TBL programs also provided training that led to a variety of occupational skill certifications professional licenses and college degrees including associate bachelor and master degrees . To provide access to course materials nearly all of the programs used a Learning Management System LMS most commonly Blackboard Learn or Angel. The LMS allowed participants to access training materials receive and submit assignments complete tests and communicate with peers and instructors. Most program operators made some accommodations to ensure that disabled participants could access materials on their LMSs. Because these systems were complex and had occasional problems all program operators provided some sort of technical support to participants. TBL programs provided participants with a number of non-training services. These services primarily included employment services such as career advising job placement job readiness soft skills training and internships or other work experience. Three programs also provided case management and supportive services most TBL participants who received case management however received it from partner programs such as community-based organizations vocational rehabilitation and local Workforce Investment Act WIA program providers and One-Stop Career Center operators. Design and Development of TBL Programs In designing and developing their TBL programs grantees and program operators typically engaged in a two-phase process with the first phase focused on making overall design decisions and the second phase aimed at designing implementing and testing new or upgraded curricula. The first phase involved designing the training program and usually began before the TBL grants were awarded. About half of the grantees determined that the best use of their grant was to convert existing curricula to an online or blended format while eight grantees determined that their program operator would need to acquire or develop new curricula. For most program operators the second phase of the design and development process the implementation of the plans developed during the first phase typically began immediately after they were awarded a TBL grant. Grantees and program operators generally adopted one of four approaches to this implementation process. The most common approach used by at least 15 TBL programs was to pair training program instructors with instructional designers or other program operator staff members who were knowledgeable about TBL methods. Once curricular changes or additions were implemented most program operators piloted their TBL curricula either formally or informally. All of these program operators indicated that this piloting process provided them with an important opportunity to identify and solve problems before the courses were made available to a wider audience. To guide the implementation of these curricular changes nine program operators regularly consulted with employers and other partners often through TBL advisory boards. At least two program operators also received input and feedback from consultants. The average duration of TBL programs design and development processes was about two years. In general programs that developed or made changes to a large amount of curricula created new curricula used complex technologies or had more experience with TBL spent more time on design and development. In a few cases major unexpected events occurring well after the TBL grants were awarded caused changes that also extended the length of design and development. Training Methods Sixteen programs 1 used a blended approach to instruction while four used a completely online approach and only one used a completely in-person approach. 2 Ten programs used a combination of synchronous learning that takes place at a specific time and place whether in classrooms or via an online or other mechanism and asynchronous learning that is not constricted by time or place but rather takes place independently activities nine incorporated only asynchronous activities into their coursework and two employed only synchronous activities. Creating opportunities for effective communication and interaction among participants and between participants and their instructors was considered an important pedagogical practice by instructors across multiple TBL programs. TBL programs used a wide range of tools and practices to encourage this communication. One practice was to require in-person sessions wherein communication and interaction would naturally occur another was to encourage or require the use of various online communications tools most commonly e-mail. Intriguing uses of technology to foster communication and interaction included one program s use of a social networking platform and another program s use of virtual reality software both of which provided participants with opportunities to communicate and interact in engaging ways. While many participants felt satisfied with the levels and means of communication afforded to them some participants and instructors expressed a desire for more in-person interaction. TBL programs measured participant achievement in a number of ways they assessed individual projects tested practical skills and conducted quizzes tests mid-terms final exams and certification exams. Frequency and mode of testing varied according to learning goals and course structure. Instructors and course designers endeavored to maintain academic integrity by providing multiple layers of assessment and or by structuring assessments in such a way that 1 While the 20 grantees offered a total of 26 TBL training programs only 21 of the programs were analyzed for the evaluation. The report describes the reasons for this in the footnotes of Exhibits I-2 and I-3 on pages I-5 through I-7. 2 This grantee used video conferencing equipment and software to broadcast synchronous lectures to remote locations. cheating would be difficult. Instructors in programs wherein industry-recognized certification was the final goal noted that it was in the participants best interests not to cheat if they wanted to gain the knowledge necessary to pass certification exams. Partnerships Developing or maintaining partnerships with employers was a major focus for nearly all TBL programs at the time of the site visits. These partnerships provided numerous benefits to TBL grantees and program operators. Specifically employers reviewed curricula provided information on future hiring and training needs and helped programs keep up with industry changes. Some employers also provided internships or clinical experiences space for training programs or funding while others allowed their employees to be recruited as participants or instructors for programs or hired program graduates. Employers in turn benefited from partnerships with TBL programs. Specifically they received skill upgrades for current employees avoided recruiting costs and had larger pools of skilled workers from which to hire employees. Nearly all TBL programs either developed or strengthened partnerships with agencies in the public workforce system particularly local WIBs local WIA program providers and One-Stop Career Center operators. These public workforce agency partners played a number of roles in the implementation of TBL grants they helped programs understand the needs of local employers identified gaps in the availability of training connected programs with possible employer partners and in one case contributed funding. A number of TBL programs formed strong relationships with one or more types of other organizations including educational institutions community-based organizations government agencies employer intermediaries and labor unions. TBL programs developed these partnerships for a variety of reasons including basic skills and soft-skills training wrap-around supportive and case management services and for enrollment in skilled training programs to prepare for employment opportunities. Preliminary Outcomes While TBL grantees and or program operators were still implementing their programs at the time data were collected for this report available preliminary data suggest that programs were mostly on track to achieve a number of expected outcomes such as improved access to training for individuals with geographic and time-related barriers increases in the number of individuals successfully completing training in high-growth industries achieving industry-recognized credentials and securing training-related employment and meeting the workforce needs of employers in targeted industries. Fully operational TBL programs appeared to have succeeded in improving access to training for individuals who would not have been able to participate otherwise. Through the use of technology these programs made training accessible for participants residing too far from training providers and for participants with family and work commitments that would have prevented enrollment in traditional courses. Due to limited data on whether participants successfully completed training programs or credentials assessments of these outcomes could not be made. However many program operators had in place various strategies designed to produce successful outcomes and reduce program attrition. These strategies included preparing participants for the time commitments required of TBL programs prior to enrollment actively monitoring participant progress so that 5 program staff members could intervene and provide support if someone was at risk of failing or dropping out and screening participants prior to enrollment to ensure that they possessed the self-motivation and other personal characteristics necessary for success in TBL courses. Most TBL grantees and or program operators also designed their course materials in ways that accommodated different learning styles and paces. Overall students reported being satisfied with their TBL training. They noted that training methods were convenient and that the use of asynchronous methods allowed them not only to integrate training into their already busy schedules but also to master program content. Generally students reported that the skills they learned in their programs were relevant to their current jobs or would be relevant to their future careers. While only a few TBL programs had any participants who had completed their training and entered employment at the time data were collected the few employers interviewed during the site visits expressed satisfaction with the training received by the program graduates they hired stating that their training had adequately prepared them for employment. Employers of incumbent workers appreciated that their employees had learned industry-relevant skills while being able to fulfill work obligations. 4. Availability . To view an abstract of this publication as well as to download the full report visit the ETA Occasional Paper Series Web site at http wdr.doleta.gov research keyword.cfm . 5. Evaluation Next Steps . Due to delays in the clearance of data collection methodologies originally planned for the evaluation these activities were postponed for completion at a later date 1 gathering detailed participant-level administrative data from grantees management information systems to supplement data collected from quarterly report submissions and 2 conducting an online survey of TBL participants. Approval was provided to ETA on January 31 2011 after conclusion of the SPR contract on December 26 2010. In July 2011 a new contract was awarded to Mathematica Policy Research Associates to complete these data collection tasks conduct analyses on the data and prepare a report and provide a briefing on the findings. This third report for the TBL Grants Evaluation pertaining to the collection of participant administrative and survey data is anticipated for release in late 2012. 6. Inquiries . Please contact Michelle Ennis in the Division of Research and Evaluation at ennis.michelle dol.gov .