TEN21-07acc.pdf

ETA Advisory File Text

NO. 21-07 EMPLOYMENT AND TRAINING ADMINISTRATION U.S. DEPARTMENT OF LABOR WASHINGTON D.C. 20210 TO STATE WORKFORCE AGENCIES STATE WORKFORCE LIAISONS ONE-STOP CAREER CENTER SYSTEM LEADS FROM EMILY STOVER DeROCCO s Assistant Secretary SUBJECT Effective Use of Assessment in the Public Workforce Investment System and Transmittal of Testing and Assessment A Guide to Good Practices for Workforce Investment Professionals 1. Purposes . This guidance notice is intended to encourage the effective use of assessment instruments within the workforce investment system to promote efficient investment in talent development that will drive regional econ omic competitiveness provide guidance to the workforce investment system and its partners on effective use of assessment instruments and release and transmit a new technical assistance resource Testing and Assessment A Guide to Good Practices for Workforce Investment Professionals the Guide . This is the third in a series of Testing and Assessment Guides see Section 2. References which was initiate d based on requests from workforce investment officials. This guidance notice and the attached new technical assistance guide are designed to support the workforce investment system in promoting talent development by ensuring that individuals served are able to make well-informed education and career choices. 2. References . Training and Employment Notice No. 10-99 announcing the guide Tests and Other Assessments Helping You Make Better Career Decisions which is targeted to individuals who may be taking assessments. The guide is available throu gh the O NET Resource Center at www.onetcenter.org dl files testAsse.pdf . Training and Employment Notice No. 33-98 releasing Testing and Assessment An Employer s Guide to Good Practices. The guide is available through the O NET Resource Center at www.onetcenter.org dl files empTestAsse.pdf . Both of the referenced guides also are available through the ETA Researc h Publication Database at wdr.doleta.gov research Hint Search using the keyword Testing. TRAINING AND EMPLOYMENT NOTICE DATE December 13 2007 - 2 - 3. Strategic Objectives for Effective Use of Assessment Tools . Human capital is the most valuable resource in today s globally competitive highly techni cal and increasingly diverse workplace. Investing in the skills of the workforce is critical for America s economic well-being and maintaining a high standard of living. Testing and assessment instruments can be critical tools in helping individuals bridge the skills gap that keeps them from obtaining high growth jobs. Thus assessment and testing are key services provided under the Workforce Investment Act to foster talent development. There are four main objectives for the use of assessments within the workforce investment system 1 exploring career options and person-career matches to ensure that individuals undertake skills preparation that is suited to their existing level of readiness as well as their aptitudes abilities and informed career go als 2 identifying skills gaps and education needs to ensure that individuals develop the skills and competencies that are in demand by high growth businesses and industries 3 referring qualified candidates to available job openings and 4 promoting attainment of industry-recognized credentials to ensure that candidates referred to business and industry partners are qualified for those posit ions. The strategic use of assessment instruments can help the public workforce investment system achieve these objectives help One-Stop Career Centers target customer services effectively and maximize the return on both the finan cial investment and the investment of time and effort that job seekers undertake as described below. A. Exploring Career Options . Individuals possess various interests aptitudes and behaviors that may make some in demand careers a better fit for them tha n others. Fortunately there is a wide range of in demand careers in the 21 st century economy in industries from health care to construction to energy technology. Assessments can help match individuals with likely career o ptions which helps assure the best possible return on investments of both time and money in education and skills training. B. Identifying Skills Gaps and Education Needs . Some individuals may need additional literacy or mathematics preparation before they are able to t ake full advantage of occupation-specific or industry-specific skills training. The use of assessments can help to identify educational readiness the ability to b enefit from specific workforce preparation or prerequisites that may need to be att ained prior to a course of study. Assessments can also help dislocated or inc umbent workers identify skills gaps that they need to address to increase their employment and advancement opportunities in the economy. C. Referring Qualified Job Candidates to Business . Effective use of assessments can improve the ability of the workforce investment system to refer qualifie d candidates to businesses seeking skilled workers. The results of severa l - 3 - assessment instruments along with educational attainment work experience and other credentials can assist One-Stop Career Center staff in identifying qualified job candidates who meet the skill needs specified by potential employers. D. Attaining Industry-Recognized Credentials . A number of assessments can lead to skills certifications recognized by business and industry. Individuals who demonstrate their competencies through successful performance on such assessments will receive certification or other credentials that can be an important part of their qualifications and assist them in finding employment. 4. Types of Assessment Instruments . To help achieve the objectives described above One-Stop Career Centers should begin by selecting a comprehensive toolki t of assessments from among the types described in this section. Workforce development assessment instruments differ in Purpose -- such as career counseling referral selection placement completion of training and development programs or competency attainment Format -- such as paper-and-pencil work sample or computer application Level of standardization objectivity and quantifiability-- such as subjective evaluations of informal interviews highly structured achievement tests or personality inventories with no specific right or wrong answer s. Assessment instruments and procedures vary greatly on these factors What they are designed to measure -- such as knowledge skills abilities personality traits work styles work values vocational interests mana gerial potential career success or job satisfaction A brief description and examples of two major types of assessments are p resented below. One major category of assessments includes those that measure personal characteristics that are important to job satisfaction. They can help individuals identify such characteristics as their work interests or work values. Interest Inventories - help identify individuals interests or activities that they like to do. Individuals can use information about their career-related interests to find education training or work that they would enjoy Work Values Instruments - help identify what a person values in a job independence recognition working conditions etc. A second major category of assessments includes those that help measure levels of - 4 - achievement or competencies of individuals that are important to successful training and job performance. They measure constructs such as knowledge skills abilities literacy and or basic work or career readiness. Frequently the quest ions presented in this type of assessment have right or wrong answers associated with t he commonly-understood concept of what constitutes a test. Individ uals can practice or study to improve their scores on these types of asse ssments. Work or Career Readiness Assessments - can be used to determine if an individual is work ready or may need remediation and can also be used a s a screening tool prior to more advanced achievement assessments referral to training or job placement. Work readiness is a term used to define whether individuals have the entry-level or foundational competencies needed to meet the demands of the workplace and to learn and advance on the job. Work or career readiness assessments can involve varying subject matter. They may assess soft skills such as the ability to cooperate wit h others resolve conflicts take responsibility learn and adapt to change. The y may also assess basic academic achievement such as the ability to communicat e orally in English to read and write locate information perform arithm etic calculations and solve problems. Various work or career readiness instruments assess different combinations of soft or interpersonal skill s and academic skills. A work readiness model defines which abilities and skills are considered to be essential and thus are included in and measured by the corresponding work readiness assessment instrument. Persons who reach certain levels of achievement or who have mastered particular competencies as indicated by their performance on the assessment test often receive skill proficiency or career work readiness certificates that convey their level of mastery in terms of a model of readiness competenc ies. Literacy Tests - measure how well individuals read write or use numbers in printed materials. Ability Instruments - measure skills related to how well individuals can learn or will perform in a training program. Typically these assessments hav e multiple parts and assess areas like arithmetic vocabulary and reading comprehension. Motor abilities may also be assessed. Achievement Tests - measure how well an individual has mastered a particular subject like computer programming or algebra. Achievement t ests may be part of a certification program. Skill and Competency Assessments - measure how well an individual performs particular skills. Skills measured can range from typing to w riting and active listening. - 5 - Chapter Four of the Technical Assistance Guide transmitted with this TEGL provides more detailed information on types of assessments. The guide Tests and Other Assessments Helping you Make Better Career Decisions also provides more information on different types of assessments see Section 2. Referenc es for information on how to access this publication . There are two other resources available to obtain information about asse ssments and their usage. One resource is www.Workforce3One.org which provides access to examples of promising practices and innovative uses of assessments to support talent development at the state and local levels. Another resource that can be used to identify competency-based assessment instruments is the searchable database in the Competency Model Clearinghouse within the ETA-sponsored CareerOneStop Web site at www.careeronestop.org CompetencyModel search.aspx . 5. Appropriate Usage of Assessment Tools . In order to serve both job seekers and business clients every local workforce area s One-Stop Career Center system should have the ability to provide their customers with access to an array of d ifferent assessment instruments relevant to the customer s needs. Also workforce investment system staff should understand which assessments will most effectively address different customer situations. This will often depend upon the background and characteristics of individual customers that is on factors such as their educational attainment life and work experience including volunteer o r extracurricular pursuits and whether or not their existing skills and competencies are still in demand in their regional economy. Typically individuals will benefit from more than one type of assessment for example they may need to engage in both career exploration assessment interests work values and abilities and work readiness assessment skills ach ievement or competency . Using multiple assessments to measure skills abilities and other job-r elevant characteristics contributes to whole person assessment which pr ovides a solid basis for individuals to make important career development decisions. U sing more information about an individual enables the job seeker and potential employers to get a more valid fair and complete picture of the individual s skil ls and strengths and therefore to make better career workforce investment and employme nt decisions. An assessment protocol that recommends the same battery of tests for eve ry individual who seeks assistance is not appropriate. To get the best results it is important for workforce investment professionals to select the appropriate instruments for use with specific customers based on client characteristics such as educational attainment work experience and career goals. These characteristics interact with each other in determining which assessments are appropriat e to administer. - 6 - For example interest inventories and work values assessments are partic ularly suited for individuals who may be unsure of what careers to pursue. They are also relevant for those who are considering a career change. This would include both in- school and out-of-school youth new labor market entrants regardless of age and dislocated workers who are unlikely to find employment within their regi onal labor market in their current field industry or occupation. The higher the level of education and the more recently it was completed the less likely it is that an individual will need basic literacy workforce readiness or achievement tests. For example a dislocated worker with a high school diploma obtained 20 years ago will probably benefit from assessments of literacy and basic academic skills. Other dislocated workers with more recent post-seconda ry education may benefit more from skill assessments to see if they possess the current skills in demand in their field rather than assessments of literacy or basic academic skills. The more work experience an individual has in a high growth high demand field the less likely abilities or work readiness tests need to be administere d. Achievement or skills competency tests are helpful to determine whether individuals possess skills in demand in the current and emerging labor market. The following table is intended to serve as a quick reference guide to t he types of clients who may benefit most from different types of assessment tools. The list is not exhaustive and does not represent all possible client groups for whom ce rtain assessments might be valuable. The table shows that the most important variable in determining whether to administer a career exploration instrument is whether the individual has existing career goals although this factor could correlate to some extent with w ork experience. Interviews also can help to determine whether individuals h ave formulated attainable career paths or may need some assistance. For most of the other types of assessment there is an interaction between the level of educational attainment and work experience as these are often substit utable for each other in employment decisions. Thus the educational attainment and work experience categories should be considered together in deciding what typ es of assessments to administer to an individual. If both are checked that is a strong indicator that the assessment may be necessary. If only one is checked that indicates that the assessment may be useful and could depend on information gained in interviewing the individual. If neither is checked it is unlikely that the individual with those characteristics would benefit from that type of assessment. - 7 - Type of Assessment Customer Attributes Career Goals Education Workforce Preparation Work Experience Unknown Established Job Career Changer Less than High-school High-school diploma GED Learners of English as a second language Some college Postsecondary certificate 2-year degree 4-year degree No work experience Limited work experience Dislocated Worker Incumbent Worker Job Changer Career Exploration Instruments -Interest Inventories -Work Values Work and Career Readiness Assessments Literacy Tests Ability Instruments Achievement Tests Skills Competency Tests - 8 - 6. Considerations for the Workforce System Using Assessment Tools with Workforce Information . The workforce investment system has invested significantly in providing improved and easily accessible information about the world of work and about spec ific industries and occupations. This information is available through a ran ge of national and state information resources and Internet sites including t he CareerOneStop electronic tools www.careeronestop.org Career Voyages www.careervoyages.gov O NET OnLine http online.onetcenter.org the Occupational Outlook Handbook http stats.bls.gov oco home.htm and a myriad of state-specific sites. It is important to leverage these investments in workforce and career information by integrating the information with results of assessment tools. Doing so will enhance the career decision-making process by providing individuals with a comprehensive picture of available employment and educational opportunities in the geographic area where they intend to work. One-Stop Career Center staff need to assist individuals in making the connection between their career aptitudes and the workforce needs of business to empower individuals to better manage thei r own careers and lifelong learning. 7. Considerations for the Workforce System Planning for Effective Use of Assessments . Strategies and policies for use of assessment tools are a critical aspect of state and local workforce system planning. Policies and plans to promote effectiv e utilization of assessment instruments can aid in achieving the goals of the workforc e investment system and in promoting talent development for regional economic competitiveness. State and local workforce agencies workforce investment boards and One-Stop operators need to recognize the critical role that assessment tools play in the delivery of workforce services to both individuals and businesses . Because state and local workforce systems and One-Stop delivery systems are organized differently throughout the country we recommend that this issue be considered at every level of the system. Examples of opportunities to d evelop strategies and policies for this purpose include Workforce Investment Act WIA and Wagner-Peyser State Plan development the WIA local planning process development of a memorandum of understanding among local One-Stop delivery partners and contractual documents with One-Stop operators. We encourage state and local workforce investment boards to develop uniform policy or guidance on the use of assessment tools within the One-Stop delivery system. For example some states have endorsed or adopted specific lite racy basic achievement or career work readiness assessments for use in the state. In - 9 - particular the policy should emphasize the appropriate types of assessm ents to be used with customers in One-Stop Career Centers for various purposes s uch as developing an individual employment plan or referrals to skill preparat ion or employment. There are several important areas to address when engaging in strategic planning or the development of policy guidance Obtain legal consultation and guidance when developing assessment policy and when selecting or procuring assessment instruments for use by staff within the One-Stop system to ensure that assessments used are valid reliable and non-discriminatory. Identify the suite of assessment tools that will be used to ensure that workforce investment professionals and their customers have access to a comprehensive set of assessment tools. Conduct staff development for personnel within the workforce investment system about the selection and effective use of assessment tools and instruments. The new technical assistance guide Testing and Assessment A Guide to Good Practices for Workforce Investment Professionals can serve as a valuable resource for structuring curriculum and materials for this purp ose. Endorse the good practices and assessment principles outlined in the guide including o the whole-person assessment concept and the importance of using more than one assessment instrument for an individual o familiarity with legal issues involved in using assessment instruments t o ensure proper use and administration of assessments o criteria to use when selecting appropriate assessment instruments for different purposes and o proper administration scoring and interpretation of assessment instruments and appropriate interactions with customers taking assessments. Explore ways to manage assessment costs . Development of a state-wide strategy can offer economies of scale and potentially reduce costs for s ome instruments. Some tools may be used in the education system or other programs so it may be valuable to develop strategic partnerships with o ther government agencies. When working with business and industry particularly around assessments for industry-driven competencies it is desirable to explore the extent to which the private sector may bear som e of the costs associated with procuring and administering assessment tools. - 10 - 8. Release of Technical Assistance Guide . The attached guide Testing and Assessment A Guide to Good Practices for Workforce Investment Professio nals serves to help career counselors and workforce development professionals use assessment practices in appropriate ways to support the overarching goal of workfor ce development. The guide conveys the essential concepts of testing and as sessment in easy-to-understand terms so that counselors trainers and other workforce development professionals can Evaluate and select assessment instruments procedures that support caree r exploration decision-making and guidance and aid in career developmen t and workforce preparation Administer and score assessment instruments that are the most efficient and effective for particular customer needs Interpret assessment results in an accurate manner and Understand the professional and legal standards to be followed when using tests and assessments in counseling workforce preparation and other ca reer development programs. Contents of the guide include 1. Assessment in Workforce Development Career Counseling Training and Development 2. Understanding the Professional and Legal Context of Assessment in Workforce Development 3. Understanding Test Quality Concepts of Reliability and Validity 4. Assessment Tools and Their Uses in Workforce Development 5. How to Select Tests Standards for Evaluating Tests a. Sources of information about tests b. Standards for evaluating a test information to consider in determining suitability of a test for your use c. Checklist for evaluating a test 6. Administering Assessment Instruments 7. Using Scoring and Interpreting Assessment Instruments 8. Issues and Concerns with Assessment 9. A Review Principles of Assessment Limitations of assessment instruments Do not expect any test or procedure to measure a personal trait or ability wit h complete accuracy for every single person. Do not expect any test or procedure to be completely accurate in predicting performance or job satisfaction. - 11 - The Guide is designed to provide important information regarding testing as part of workforce development programs such as career counseling training and development. It gives only general guidelines and must not be viewed as legal advice. 9. Summary of Principles of Assessment . This section lists each of the principles of assessment from the guide indicates the chapter of the guide where they are discussed in depth and provides a description of each principle an e xecutive summary of the assessment principles from the new guide. Use assessment instruments in a purposeful manner Chapter 1 Assessment instruments like other tools are helpful when used properly but can be useless harmful or illegal when used inappropriately. Often inapprop riate use results from not having a clear understanding of what you want to measur e and why you want to measure it. In selecting your assessment instruments always keep in mind the knowledge skills abilities characteristics and personal traits you want to measure and the population to be assessed. Once you are clear about your purpose you will be better able to select appropriate assessment instru ments and use them in an effective manner to achieve stated goals. Only use tests that are appropriate for your particular purpose. Use the whole person approach to assessment Chapter 1 An assessment instrument may provide you with important career- or train ing- relevant information about an individual. However no assessment tool i s 100 percent reliable or valid all are subject to errors both in measuring job-relevant characteristics and in predicting job performance. Moreover a single a ssessment instrument only provides you with a limited view of a person s qualif ications. Using a variety of tools to measure skills abilities and other job-rel evant characteristics provides you with a solid basis upon which to make important career and development decisions and minimizes adverse impact. Use only assessment instruments that are unbiased and fair to all groups Chapter 2 Using unbiased and fair tests will enable you to provide the most approp riate assessment services to clients. You should review the fairness evidence associated with assessment instruments by examining the test manual and independent test reviews. - 12 - Use only reliable assessment instruments and procedures Chapter 3 If a person takes the same test again will he or she get a similar scor e or a very different score A reliable instrument will provide accurate and consis tent scores. To meaningfully interpret test scores and make useful career or employment-related decisions use only reliable tools. Test manuals usually will provide a statistic known as the reliability coefficient giving you an indication of a test s reliability. The higher the reliability coefficient the more confidence you can have that the score is accurate. Use only assessment procedures and instruments that have been demonstrat ed to be valid for the specific purpose for which they are being used Chapte r 3 Validity is the most important issue in selecting assessment instruments. It refers to 1 the characteristic the assessment instrument measures and 2 how well the instrument measures the characteristic. Validity is not an inherent property of each assessment instrument itself it also depends on the context of its use that is an assessment instrument may be valid for certain uses but not for others. A test s validity is established in reference to a specific purpose it may not be valid for different purposes. For example a test that may be valid for assessing someone s job knowledge may not be valid for predicting his or her leadership skil ls. The test administrator must be sure that the instrument is valid for the purpose for which it is to be used. Selecting a commercially developed instrument does not r elieve the test administrator of this responsibility. The test manual usually provides a statistic the validity coefficient which will give an indication of the test s validity for a specific purpose under spe cific circumstances. It measures the degree of relationship between test performance and job performance i.e. job-relatedness of the test . Use assessment instruments that are appropriate for the target populatio n Chapter 3 An assessment tool usually is developed for use with a specific group i t may not be valid for other groups. For example a test designed to predict the wor k values of adults may not be valid for use with junior high school students. It is possible that many junior high school students are not vocationally mature enough for the assessment tool to be meaningful or perhaps the reading level of the test is not suitable for junior high school students. Tests should be appropriate for the individuals you want to test that is your target population. The manual should indicate the group or groups the test is designed to a ssess. Your target population should be similar to the group on which the test was d eveloped or - 13 - normed. In determining the appropriateness of an instrument for your target group also consider factors such as reading levels cultural backgrounds and language barriers. Use assessment instruments for which understandable and comprehensive documentation is available Chapter 5 Are the instructions for administration and interpretation understandable Is the information sufficiently comprehensive to evaluate the suitability of th e instrument for your needs Carefully evaluate the documentation provided by the te st publisher to be sure that the tools you select do the job you want them to do and furnish you with the information you need. If the documentation is not understandable or complete you run the risk of selecting inappropriate instruments. Test manuals should provide information about both the development and psychometric characteristics of tests. They should cover topics such as procedures for administration scoring and interpretation the recommended uses of an instrument the groups for whom the test is appropriate and test norms. They also should include a description of the validation procedures used and evidence of validity reliability and test fairness. Ensure that administration staff are properly trained Chapter 6 Assessment instruments must be administered properly to obtain valid results. Consult the test publisher and administration manual for guidelines on t he qualifications and training required for test administrators. These requirements will vary depending on the nature and complexity of the test. Only properly trained staff should be selected to administer these assessments. Administrators should be given ample time to learn their responsibilities and should practice by administering tests to other staff before administering tests to applicants. Some tes t publishers may run training sessions for test administration and interpretation. Administration staff also should be trained to handle special situations with sensitivity. An example would be responding to a request for accommodat ion based on a disability. Ensure that testing conditions are suitable for all test takers Chapte r 6 There are various extraneous influences that may affect the reliability and validity of an assessment procedure. For example noise in the testing room poor l ighting inaccurate timing and damaged test equipment may adversely affect test takers. Staff should ensure that the testing environment is suitable and that administration procedures are uniform for all test takers. - 14 - Provide reasonable accommodation in the assessment process for people wi th disabilities Chapter 6 To ensure that qualified individuals with disabilities have an equal chance to demonstrate their potential accommodations in the assessment process ma y be necessary. Under the ADA reasonable accommodation may involve ensuring physical accessibility to the test site modifying test equipment or the testing process or providing qualified assistance to the test taker. For example admin istering a Braille version of a test allowing extra time to complete the test or supplying a reader may be appropriate. It is important to become familiar with the types of accommodations that can be made without invalidating test results. If r easonable accommodation involving test administration cannot be made consider alt ernative assessment strategies. Maintain assessment instrument security Chapter 6 All materials used in the assessment process whether paper-and-pencil o r computer-based must be kept secure. Lack of security may result in som e test takers having access to test questions beforehand thus invalidating their scores. To prevent this test administrators should for example keep testing materials in locked rooms or cabinets and limit access to those materials to staff involved in the assessment process. Some instruments used in counseling and in self-assessments are not held to the same strict standards of security. The purpose of testing often will dictate the importance of security for test materials. Security is also the responsibility of test developers. The security of a test may become compromised over time. T o protect security test developers periodically introduce new forms of tests. Maintain confidentiality of assessment results Chapter 6 Assessment results are highly personal. Workforce development professionals must respect the test taker s right to confidentiality. Assessment results should be shared only with those who have a legitimate need to know. This would include staff involved in interpreting assessment results or making work-related decisions. Personal information should not be released to other organizations or in dividuals without the informed consent of the test taker. - 15 - Ensure that scores are interpreted properly Chapter 7 Tests are used to make inferences about people s knowledge skills abilities traits interests and values. The inferences should be reasonable well-founded and not based upon stereotypes. If test scores are not interpreted properly th e conclusions drawn from them are likely to be invalid thus leading to poor decision making. Ensure that there is solid evidence to justify your test score interpretations and the decisions you make based on those scores. The test manual should provid e instructions for properly interpreting test results. 10. Availability . The new guide is available as a downloadable file in PDF from the following Web sites The National O NET Consortium at www.onetcenter.org product index.html The ETA Advisories Web site at wdr.doleta.gov directives attach TEN ten2007 TEN21-07a1.pdf 11. Action Required . State administrators are requested to provide the above information to appropriate staff within the state workforce investment system and others who require an understanding of issues related to assessments and assessment practice. This may include administrators educators trainers personnel researchers interviewers employment counselors test administrators o ther workforce development professionals at local workforce investment boards and operators and staff of One-Stop Career Centers and elsewhere. 12. Inquiries . For additional information on assessment instruments and their use in the workforce investment system contact Pam Frugoli 202 693-3643 Jan Sten 202 693-2793 or the Division of Workforce System Support within the Office of Workforce Investment at 202 693-3045. For additional information on the guide and other related O NET products and services visit the Department of L abor s O NET Web site at www.doleta.gov programs onet or the National O NET Center Web site at www.onetcenter.org . Direct customer service inquiries to the National O NET Consortium s Customer Service Office. Direct e-mail t o onet ncmail.net . Fax 919 715-0778. 13. Attachment Testing and Assessment A Guide to Good Practices for Workforce Investment Professionals