Skip to page content
Secretary of Labor Thomas E. Perez
Bookmark and Share

DOL Annual Report, Fiscal Year 2004
Performance and Accountability Report

Significant FY 2004 Audits and Evaluations

The Department of Labor has worked hard to cultivate an "Evaluation Culture," a term used by the U.S. General Accounting Office (GAO) in a recent report.32 For the Department, this has meant welcoming objective reviews and audits by the Office of the Inspector General (OIG) and the GAO. Beyond that, we initiate self-examination by contracting with professional evaluators, inviting them to study our programs and assess our effectiveness.

During the course of the past year, the evaluation programs at the Department of Labor have supported a wide variety of studies, such as:

  • Process evaluations: reviewing our operating procedures and program implementation.
  • Outcome evaluations: moving us beyond a focus on our "outputs" and guiding us further in our commitment toward producing positive "outcomes."
  • Impact evaluations: distinguishing the precise contribution of our programs from that of other external factors.
  • Case studies: analyzing the delicate balance between the cost of operations and the yield in benefits.

32Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, GAO-03-454, May 2003.

While all of the audits and evaluations described below have their own unique focus and content, they each share in our common goal of trying to improve our efforts to develop a global economy for the 21st century.

Studies Conducted by the DOL Office of Inspector General (OIG)

1. Bureau of Labor Statistics: Government Information Security Reform Act (GISRA) Review of Current Population Survey (CPS), July 2002.

Performance Goal(s) Affected: 1.3A — Improve information available to decision-makers on labor market conditions, and price and productivity changes

Findings: There were four findings related to: (1) the CPS Continuity of Operations Plan (COOP); (2) the CPS System Security Plan (SSP); (3) security controls of the CPS; and (4) a Bureau-wide Uninterruptible Power Supply (UPS) system.
Recommendations: (1) Complete, provide training, and test the CPS Continuity of Operations Plan (COOP); (2) Complete risk assessment and update the CPS System Security Plan (SSP); (3) Add "Data Integrity and Validation" section to the CPS SSP; (4) Test and evaluate the security controls of the CPS; and (5) Procure new building-wide Uninterruptible Power Supply (UPS) system.
Actions Taken: BLS responded to the final report from the OIG on November 21, 2003. Each of the five recommendations is completed, except for Number 1. Number 1 includes COOP testing, which is a longer-term commitment.
Additional Information: A copy of the final evaluation may be obtained from the Bureau of Labor Statistics, Division of Management Systems, Room 4080, 2 Massachusetts Avenue, NE, Washington, DC 20212, or by calling 202-691-7628.

2. Employee Benefits Security Administration: Audit of EBSA Participant and Compliance Assistance Program (#09-04-001-12-121), March 2004.

Performance Goal(s) Affected: 2.2B — Enhance pension and health benefit security

Findings: The Office of Inspector General conducted an audit of the Employee Benefits Security Administration's Participant and Compliance Assistance program. The objective was to determine if EBSA's program: (1) provided information and assistance timely and accurately; and (2) whether the reported performance data, including benefit recoveries, was accurately reported for FY 2002. The OIG findings and recommendations are summarized as follows: (1) EBSA provided information and assistance timely and accurately; and (2) reported performance data was reasonably accurate.
Recommendations: None.
Actions Taken: None.
Additional Information:

3. Employee Benefits Security Administration: EBSA Needs Additional Authority to Improve Quality of Employee Benefit Plan Audits (#09-04-005-12-12), September 2004

Performance Goal Affected: 2.2B — Enhance pension and health benefit security.

Findings: The OIG reviewed the process used by EBSA to identify and correct substandard audits of employee benefit plans. The Employee Retirement Income Security Act (ERISA) requires employee benefit plans to be audited annually. The audit requirement is intended to help protect the interests of benefit plan participants and beneficiaries. The OIG found that although EBSA has made efforts to correct substandard audits, including rejecting annual report findings and referring auditors to oversight and or licensing organizations for possible investigation and disciplinary action, the process for identifying and correcting substandard audits has not been effective. For example, some referred audits had not been brought up to ERISA requirements. Moreover, OIG's review found that EBSA does not have the authority to take direct action against auditors who perform substandard audits on employee benefit plans.
Recommendations: Among the OIG's recommendations were that EBSA propose changes to ERISA to grant EBSA greater enforcement authority over such matter as registration, suspension, debarment, and civil penalties against employee benefit plan auditors. The IG also recommended that EBSA improve the accuracy of its case tracking system and develop targeting methods to identify substandard audits based on common attributes of plans with substandard audits.
Actions Taken: EBSA responded the OIG report, generally agreeing with its recommendations, identifying planned steps to address the recommendations, and citing ongoing efforts to improve employee benefit plan audit quality. With respect to its enforcement authority, EBSA responded that it recognizes deficiencies in the current law and is considering options for correcting those deficiencies. With regard to identifying substandard audits, EBSA is adopting an inspection approach, which focuses on CPA firms performing the most employee benefit plan audits.
Additional Information:

4. Employment Standards Administration: 2003 Financial Audit (#22-04-002-13-001), March 2004

Performance Goal(s) Affected: 2.2C — Minimize the human, social and financial impact of work-related injuries for workers and their families.

Findings: To determine continuing eligibility for some claimants' compensation, FECA is required to periodically review medical evidence. An Office of Inspector General audit concluded that the primary control ensuring that claimants submit medical evidence was not effective. Because OWCP did not follow its procedures, it could not terminate benefits if the medical evidence did not support continuing eligibility.
Recommendations: The Chief Financial Officer should ensure that ESA implements automated procedures to ensure FECA claims examiners review current medical evidence.
Actions Taken: The long-term solution rests with a new automated Periodic Entitlement Review application that checks files for the presence or absence of current medical evidence. In December 2004, FECA will begin using an automated tracking mechanism that will alert claims staff when medical evaluations are due. The FECA program believes that the new system and targeted training will eliminate this finding from future audits.
Additional Information:

5. Employment Standards Administration: Concerns Persist with the Integrity of Davis Bacon Prevailing Wage Determinations (#04-04-003-04-420), March 2004.

Performance Goal(s) Affected: 2.1A — Covered American workplaces legally, fairly, and safely employ and compensate their workers.

Findings: The Office of Inspector General audited the process of determining prevailing wage rates under the Davis-Bacon Act and found the Wage and Hour Division (WHD) had not sufficiently resolved past OIG and GAO concerns and recommendations. The OIG identified three issues that continue to affect the validity and usefulness of Davis-Bacon surveys: (1) errors in submitted wage data; (2) a methodology that may allow bias; and (3) time gaps between surveys and the resulting publication of the results.
Recommendations: OIG concluded that the current wage determination process should be replaced with a statistically valid methodology, and encouraged ESA to promote changes that would allow reliable and objective sources of data, such as those offered by the Bureau of Labor Statistics (BLS), to be used in prevailing wage determinations.
Actions Taken: WHD has contacted BLS about the possibility of using its data as a basis for Davis-Bacon prevailing wage determinations.
Additional Information:

6. Employment and Training Administration: New Hire Detection is a Better Method for Establishing UI Overpayments (#05-04-002-03-315), September 2004

Performance Goal(s) Affected: 2.2A — Make timely and accurate benefit payments to unemployed workers, facilitate the reemployment of Unemployment Insurance claimants, and set up Unemployment tax accounts promptly for new employers.

Findings: The UI program, a Federal-state partnership, is the DOL's largest income maintenance program. While the framework of the program is determined by Federal law, benefits for individuals are dependent on state law and administered by State Workforce Agencies. The UI system could attain significant savings by detecting overpayments through cross-matching UI claims against state and national new hire data. This would detect UI claimants who have returned to work but are still collecting UI benefits. Using new hire data to identify overpayments is more effective than the more common method of matching UI claims against employers' quarterly wage records because employers must report new hires within twenty days, whereas wage records are not available for months. The new hire method results in earlier detection of overpayments, reduces overpayment dollars, and increases the chance of overpayment recovery. In 2003, the OIG made recommendations for reducing overpayments by expanding states' use of new hire data. The full implementation of these recommendations, in OIG's opinion, would save the Unemployment Trust Fund (UTF) an estimated $428 million annually. In response, DOL implemented a pilot program in 6 states and improved its quality control program. DOL's Employment and Training Administration (ETA) also drafted policy changes and is collecting data for the pilot program. The pilot cost-benefit study results are due in January 2006. DOL will then need to make sure that lessons learned from the pilot are implemented. Despite the benefits of new hire detection, a recent OIG audit found 12 states had not used their own state new hire data to reduce overpayments.
Recommendations: OIG recommended that ETA 1) continue to provide technical assistance and resources to the state UI programs that are currently not using new hire detection in order to initiate and/or complete plans for implementation as soon as possible; 2) work with Department of Health and Human Services (DHHS) to communicate to Congress the need for amending the Personal Responsibility and Work Opportunities Reconciliation Act of 1996 (PRWORA), or introducing new legislation, to require employers to report a new hire's first day of earnings and provide a clear, consistent, nationwide definition for this date; 3) encourage state UI programs to access the National Directory of New Hire (NDNH) and coordinate efforts with DHHS and the state UI programs to accomplish this; 4) work with DHHS, the lead department, to encourage state agencies compiling the State Directory of New Hires (SDNH) to expand monitoring and outreach programs that will improve employer compliance and seek enforcement through penalties for employers who repeatedly fail to report new hires; and 5) assist the state UI programs in analyzing resources to determine the best detection methods, how to best allocate resources, and frequency of New Hire cross-matches.
Actions Taken: DOL was very pleased with the enactment of P.L. 108-295, which is based on draft legislation proposed DOL, and which gives state UI agencies access to the National Directory of New Hires. This will enhance states' ability to detect unreported work violations by UI claimants working in other states or for certain multi-state employers who may be all new hires to only one state. ETA is working with the Department of Health and Human Services on implementation details and will encourage states to use the directory when it becomes accessible. ETA continues to promote activities to prevent and detect overpayments. In FY 2004, $2.3 million in funds is being made available to states that submitted acceptable proposals to implement or enhance benefit payment control activities such as computer cross-matches to detect overpayments. An example of these activities would include the use of the states' Directories of New Hires as well as an electronic data exchange between state UI agencies and the Social Security Administration.
Additional Information:

7. OSHA's Future System Development Efforts Require Greater Use of Best Practices (#23-04-009-10-001), September 2004

Performance Goal(s) affected: 3.1C — Reduce occupational fatalities. 3.1D — Reduce occupational injuries and illnesses.

Findings: Based on observations made in FY 2002 and 2003, OIG identified three Project Management weaknesses in OSHA's IMIS Redesign project: (1) project management didn't cover the entire period; (2) lack of steady and assured funding increased project risk; (3) agency's Project Manager lacked system development knowledge/experience.
Recommendations: OIG made four recommendations: (1) Develop an SDLC (System Development Life Cycle) approach incorporating best practices to minimize risk in future system development; (2) Correct project management weaknesses identified in the report; (3) Ensure that every major system development effort is led by a senior program expert with experience and knowledge of the agency's mission, functions, and operations; and, (4) Obtain periodic status reports and information from the Project Manager.
Actions Taken: The agency accepted all of the OIG recommendations. The agency had previously suspended further work on the IMIS Redesign and had contracted for an independent assessment of its IMIS Redesign activities. OSHA is awaiting the contractor's final report. Receipt of that evaluation will assist the agency in determining how best to proceed in complying with the OIG recommendations.
Additional Information:

8. Office of the Assistant Secretary for Administration and Management: Departmentwide Reportable Conditions Resulting from OIG's Audit of General, Application, and Security Controls for Selected DOL IT systems that Support the Financial Statements (# 23-04-001-07-001), March 2004.

Performance Goal(s) Affected: IT2 — Continuously improve the performance of the Department's Cyber Security Program in accordance with the Federal Information Security Management Act (FISMA).

Findings: 1) The Department lacks strong logical security controls (LSC) to secure the Department's data and information. DOL has not developed and performed comprehensive tests of all continuity of operations/disaster recovery plans for critical systems and processes.
Recommendations: Recommend the Department continue to improve its security architecture and technical security standards, administrative and end-user guidelines and procedures, enforcement and monitoring processes, disaster recovery planning, and process for resolving known IT vulnerabilities and weaknesses.
Actions Taken: To correct the Logical Security Control (LSC) deficiencies, the Office of the Chief Information Officer (OCIO) has performed a complete review of Departmental operations manuals and proposed revisions to the documents; developed the DOL Technical Standards Manual (TSM) to provide hardening system standards and integrate actions for security configuration management; and revised the Computer Security Handbook. The Disaster Recovery/Contingency Planning Working Group has completed the following actions: (1) collected information from all DOL agencies related to the agencies processing priorities; (2) collected information from all DOL agencies related to the agencies' primary and backup data center status; (3) integrated collected information into the Department's consolidation efforts; and (4) further developed a bridging document that links Agency Contingency planning efforts to the DOL COOP, where applicable. This information helps clarify the role the Departmental Communications Network (DCN) will play in the failure and recovery process. OCIO has also developed a comprehensive test plan for the Emergency Management Center (EM-Net) and ECN/DCN networks to provide backup Internet and DOL agency connectivity; continued work to provide redundant agency inter-connectivity through a set of alternative DCN network interconnections; continued testing of core functions such as Active Directory, Common E-Mail and Internet connectivity; arranged for implementation of Backup Active Directory servers; prepared for Backup Common E-Mail services; and completed testing of the backup Internet connectivity. The OCIO Security team continues to monitor the agencies' implementation of their Plan of Action & Milestones; in response to a memo from the CIO to Agency Heads requesting that they resolve outstanding OIG recommendations by August, Agencies met with OIG and performed the necessary mitigation activities. As a result, the majority of the outstanding OIG recommendations have been closed and OIG is considering removing this recommendation from its reportable conditions.
Additional Information:

Studies Conducted by the U.S. Government Accountability Office (GAO)

9. Employment and Training Administration: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help (GAO-04-657), June 2004.

Performance Goal(s) Affected: 1.1A — Increase the employment, retention, and earnings of individuals register ed under the Workforce Investment Act Adult program. 1.1F — Increase the employment, retention, and earnings replacement of individuals registered under the WIA dislocated worker program.

Findings: The study found that Workforce Investment Act (WIA) performance data: (1) provides a national picture of workforce outcomes, but these data offer little information about current performance (because of time lags in Unemployment Insurance wage data from about 1 ½ years to 2 ½ years); and (2) represents a small portion of job seekers who received WIA services because of restrictions in the law and policies of Labor. The study also noted that states and local areas try to fill data gaps by using other locally defined indicators such as additional job-seeker measures, employer measures, and program partnership measures, but states and local areas would like more help from Labor in disseminating best practices on interim performance measures.
Recommendations: GAO recommends that Labor: (1) continue to allow supplemental data for reporting outcomes; (2) assist states in sharing best practices on interim indicators; (3) develop a systematic method to account for different populations and economic conditions when negotiating performance levels; and (4) expedite steps to implement an impact evaluation of WIA services. Lastly, Congress may also consider requiring that all WIA participants be tracked for reporting purposes.
Actions Taken: (1) DOL published guidance on the Common Measures for employment, retention and earnings in Training and Employment Guidance Letter 15-03. These measures require the use of Unemployment Insurance wage records for reporting on WIA performance outcomes, and continue to allow states to use supplemental wage data to assess program outcomes. Further guidance is being provided for using supplemental wage data, and states will be allowed to use and report supplemental wage data where appropriate. (2) For sharing best practices to states and local areas, ETA has created a system-based Web site where successful practices in this and other program areas can be posted by program operators. ETA Performance Enhancement Project is developing an on-line tutorial on WIA performance measures to assist states and local areas for improving performance outcomes. (3) ETA currently has a project with the State of Michigan to develop an adjustment model to account for economic conditions and different populations. Michigan has the lead in partnering with other states in the design and development of an adjustment model. (4) ETA plans to conduct an impact evaluation of the reauthorized WIA programs when the changes
Additional Information:

10. Employment and Training Administration: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services (GAO-04-308), February 2004

Performance Goal(s) Affected: 1.2A — Increase entrance and retention of youth registered under the WIA youth program in education or employment.

Findings: The study found that local areas primarily used the WIA program for dropout prevention and other efforts to improve academic achievement for in-school youth. Local areas emphasized learning-related summer employment for in-school youth. Local areas reported that serving out-of-school youth is difficult, expensive, and less effective, but when services are provided, they tend to focus on occupational skills training and supportive services. Despite DOL's guidance, areas face implementation challenges in identifying and retaining out-of-school youth, providing youth with mentoring and follow-up services, and using interim measures for on-going program assessment. Access to DOL's promising practice website and to technical assistance has been difficult for some local areas.
Recommendations: GAO is recommending that the Departments of Labor and Education coordinate efforts to clarify how schools can work with workforce officials to help connect school dropouts to local WIA youth programs.  GAO is also recommending that the Department of Labor provide states and local areas with technical assistance necessary to address the aforementioned ongoing implementation challenges and establish standard monitoring procedures to improve the quality of data reported by states. 
Actions Taken: DOL is setting up an interagency group with the Department of Education and will issue guidance on connecting dropouts to WIA youth programs. DOL will also issue guidance on a number of ongoing implementation challenges including serving out-of-school youth, providing mentoring services, providing follow-up services, and using interim measures to track program performance. The data validation initiative will ensure improvements in the quality of data reported by states.
Additional Information:

11. Employment and Training Administration: Trade Adjustment Assistance: Reforms Have Accelerated Training Enrollment, but Implementation Challenges Remain; GAO-04-1012, September 2004

Performance Goal(s) Affected: 1.1G - Increase the employment, retention, and earnings replacement of workers dislocated in important part because of trade and who receive trade adjustment assistance benefits.

Findings: The GAO report notes that at this early stage of implementation of the Trade Adjustment Assistance (TAA) Reform Act of 2002, several changes appear to be helping trade-affected workers. Overall, the study found that most workers are enrolling for service more quickly than in prior years because of a new 40-day time limit for DOL to process petitions which reduced the processing time from 107 days in 2002 to 38 days in 2003, and because of requirements for workers to register more quickly after the TAA certification or layoff event. One concern expressed in the GAO report was that this may not leave enough time to assess a worker's needs, particularly during a large layoff. Implementation of the Health Coverage Tax Credit has increased the workload in some areas. Demand for TAA training increased substantially in FY 2002, prior to implementation of the reforms, and States have struggled to meet the higher demand with available TAA training funds, even though TAA training funds available nationally doubled between FY 2002 and 2003. Most States have responded by using other federal employment and training resources. The study notes that information on TAA program results has been limited, but DOL is making improvements using wage records to track TAA outcomes and through a five year evaluation study initiated in FY 2003.
Recommendations: The study recommends that DOL monitor implementation of TAA for unintended consequences and propose legislative changes, if appropriate. In particular, DOL should track: (1) the ability of workers to meet the new training enrollment deadline and of states and local areas to provide appropriate assessments to impacted workers within deadlines; and (2) whether the eligibility criteria for the new wage insurance program are resulting in denial of services to some older workers who could benefit from the program.
Actions Taken: ETA shares GAO's concerns related to the possible negative impact on some workers of the retraining enrollment deadline and the new wage insurance provision having to do with "less than easily transferable skills" which may be resulting in denial of services to some workers who might benefit from them. ETA's Division of Trade Adjustment Assistance intends to work closely with states in upcoming months, when more experience with these issues is available, to better assess the possible need for legislative remedies. In addition, inclusion of the GAO concerns for further examination in the recently initiated TAA evaluation study will be explored.
Additional Information:
A copy of this report may be found at

12. Mine Safety and Health Administration: MSHA Devotes Substantial Effort to Ensuring the Safety and Health of Coal Miners, but Its Programs Could Be Strengthened, GAO-03-945, October 2003

Performance Goal(s) Affected: 3.1A — Reduce the mine industry fatal injury and all-injury incidence rates. 3.1B — Reduce the percentage of respirable coal dust samples and silica dust samples in mines exceeding applicable standards; and reduce noise exposures above the citation level in all mines.

Findings: To help ensure the safety and health of underground coal miners, MSHA staff review and approve mine plans, conduct inspections, and investigate serious accidents. In these three areas, GAO found that MSHA has extensive procedures and qualified staff. However, GAO found that MSHA can improve oversight, guidance, and human capital planning efforts.
Recommendations: (1) Monitor the timeliness of ventilation and roof control plan review to ensure that all inspections are completed by the district offices. (2) Monitor follow-up actions taken by district offices to ensure that mine operators are correcting hazards on a timely basis. (3) Update and consolidate guidance provided to district offices on plan approval and inspections to eliminate inconsistencies and outdated instructions, and clarify guidance on coordinating technical inspections with regular quarterly inspections. (4) Develop a plan for addressing anticipated shortages of qualified inspectors. (5) Amend the guidance provided to independent contractors engaged in high-hazard activities requiring them to report information on the number of hours worked at specific mines so that MSHA can use this information to compute the injury and fatality rates used to measure the effectiveness of its enforcement efforts. (6) Revise systems used to collect information on accidents and investigations to provide better data on accidents and better link injuries, accidents, and investigations.
Actions Taken: MSHA has: (1) improved headquarters monitoring of timely ventilation and roof control plan approvals; (2) improved headquarters monitoring of timely operator abatements; (3) updated and consolidated plan approval and inspection guidance, and procedures; (4) developed human capital strategies to address future inspector shortages; (5) initiated an independent contractor study of factors influencing contractor employment and potential methods to obtain contractor injury and employment data by individual mine; and (6) incorporated the existing Accident Investigation (AI) database into the MSHA Data Warehouse, allowing better linkage between accidents and investigations, and will develop additional information technology solutions to incorporate Part 50 data, including information currently residing in the AI database, into MSHA's new Standardized Information System (FY 2005 and 2006).
Additional Information: Copies of the report can be obtained at

13. Occupational Safety and Health Administration: OSHA's Voluntary Compliance Strategies Show Promising Results, but Should be Evaluated before They are Expanded, GAO-04-378, March 2004

Performance Goal(s) Affected: 3.1C — Reduce occupational fatalities. 3.1D — Reduce occupational injuries and illnesses.

Findings: OSHA's voluntary compliance programs appear to have yielded many positive outcomes, but OSHA does not yet have adequate data to assess their individual and relative effectiveness.
Recommendations: GAO recommended that OSHA: (1) identify cost-effective methods of assessing the effectiveness of OSHA's voluntary compliance programs; and (2) develop a strategic framework that articulates the priorities and resource allocations for the agency's voluntary compliance programs before further expanding the use of these strategies.
Actions Taken: At OSHA's request, DOL has funded an evaluation of the effectiveness of OSHA's Voluntary Protection Programs (VPP) in reducing injuries and illnesses from the inception of the employers' decision to participate in the VPP through to acceptance into the VPP. As part of this effort, the contractor conducting the evaluation is planning to develop a model to estimate VPP participants' changes in injury and rates over time that could be used for other OSHA voluntary compliance programs.
Additional Information:

14. Occupational Safety and Health Administration: OSHA's Complaint Response Policies: OSHA Credits Its Complaint System with Conserving Agency Resources, but the System Still Warrants Improvement, GAO-04-658, June 2004

Performance Goal(s) Affected: 3.1C — Reduce occupational fatalities. 3.1D — Reduce occupational injuries and illnesses.

Findings: The GAO stated that the extent of OSHA's area office supervisors' involvement in the decision-making process regarding which worker safety and health complaints would result in inspections and which would not varied across area offices. (OSHA's complaint policy sets forth certain responsibilities for area office supervisors in deciding which complaints should be investigated by inspection and which should be investigated by other means.) Although OSHA requires annual audits to identify the extent to which its area offices are correctly employing the OSHA complaint policies, some regions were not conducting these audits.
Recommendations: OSHA should ensure that area offices comply with complaint practices established by the agency and take additional actions to improve the quality of complaint information.
Actions Taken: OSHA is taking action to ensure compliance with its complaint policies.
Additional Information:

15. Occupational Safety and Health Administration: OSHA's Oversight of Its Civil Penalty Determination and Violation Abatement Processes Has Limitations, GAO-04-920, August 2004

Performance Goal(s) Affected: 3.1C — Reduce occupational fatalities. 3.1D — Reduce occupational injuries and illnesses.

Findings: GAO found that OSHA's oversight for ensuring that penalties are correctly determined and violations are properly abated had limitations. While the national office received copies of the regions' annual audits, it did not review or use them to monitor the extent to which penalties were calculated correctly and violations were properly abated. Even if OSHA had used the results of annual audits for oversight purposes, the information the audits provided was not always complete. In four of five regional offices that GAO reviewed, audits were not completed in accordance with OSHA procedures during fiscal years 2002 and 2003.
Recommendations: OSHA should ensure that regions complete audits in accordance with its required audit procedures and monitor audit results to oversee civil penalty determination and violation abatement processes. In addition, GAO recommended that the Secretary of Labor direct the Assistant Secretary for Occupational Safety and Health to evaluate the feasibility of using statistical modeling to help OSHA determine if penalties are being assessed correctly and identify if unanticipated factors are influencing penalty amounts.
Action Taken:
OSHA is taking action to ensure compliance with its civil penalty determination and violation abatement processes.
Additional Information:

Studies Conducted by Other Evaluators

16. Employee Benefits Security Administration: Assessing the Effectiveness of Methodologies Used to Evaluate EBSA's Enforcement Program Case Opening and Results Analysis and Baseline Compliance Studies, March 2004.

Performance Goal(s) Affected: 2.2B — Enhance Pension and Health Benefit Security

Conducted by: Mathematica Policy Research (MPR)

Findings: Compliance Baselines: (1) Compliance baselines are useful for allocating EBSA investigative resources but do not, by themselves, measure program impact; (2) Measuring baseline compliance would require an unreasonable diversion of investigative resources away from plans in which there is evidence of a violation, thus diminishing agency performance; (3) Estimating EBSA program impacts requires an experimental design because they cannot be measured by a baseline compliance audit alone; and (4) Neither a true experimental design nor the most reliable quasi-experimental design is feasible and the least reliable experimental design while feasible, would be expensive. When comparing National Project pension investigation cases: (1) Most effective at closing cases with results if the source of investigation is a participant complaint; (2) Most efficient cases are those that are regional office initiated; (3) When measured by highest average monetary result per investigation, those cases referred by state and local government are the most successful; (4) When measured by number of participants, EBSA was most effective and efficient with plans less than 100 participants but generated the highest average monetary result in cases with more than 2,500 participants. When comparing National Project health investigation cases: (1) Most effective and efficient at closing cases with results if the source of investigation is a national office referral; (2) When measured by highest monetary result per investigation, those cases referred by state and local government are again the most successful; (3) When measured by number of participants, EBSA is most effective with plans fewer less than 10,000 participants but generated the highest average monetary result in cases with more than 50,000 participants.
Recommendations: (1) Ensure that the Enforcement Management System (EMS) contains highly accurate data by executing edit checks and error detection reports at least annually; (2) Modify EMS fields so that they more accurately reflect plan data stored in the EDS; (3) Continue to use outcome measures like those contained in the FY 2003-2008 Strategic Plan; (4) consider computer modeling to address investigative balancing and coverage objectives to include cost effectiveness of various strategies.
Actions Taken: (1) EBSA has incorporated the suggested edit checks into EMS. The newest EMS version should be operational during the first quarter of FY 2005. (2) EBSA is reviewing the changes suggested to better reflect EDS data in EMS. (3) EBSA will evaluate the concept of computer modeling to address investigative balancing and coverage objectives.
Additional Information: A copy of the complete report can be obtained from the Employee Benefits Security Administration, 200 Constitution Avenue, N.W., Room N-5702 Washington, D.C., 20210.

17. Employee Benefits Security Administration: EBSA Customer Service Program, March 2004,

Performance Goal(s) Affected: 2.2B — Enhance Pension and Health Benefit Security

Conducted by: The Gallup Organization

Findings: The Gallup Organization conducted a comprehensive program evaluation comprising five distinct activities within the Office of Participant Assistance. The evaluation included a quantitative evaluation of the participant assistance program involving both a mystery shopping technique and customer satisfaction surveys of the participants served as well as selected program publications, the outreach program, and the EBSA website. Study findings and recommendations include: Participant Assistance Customer Satisfaction Surveys and Mystery Shopper: (1) EBSA met its customer satisfaction goal and received strong ratings overall but there was variance amongst offices and amongst staff. (2) Those offices and individuals scoring the highest were more likely to "delve into the customer's question in more detail", provide the "right level of detail" in their answers, and provide "as complete an answer as possible." Higher scores were also correlated to obtaining benefit recoveries. (3) EBSA needs to improve its score on being easy to find and reach. Publications Surveys: (1) Survey response rates were extremely low suggesting that other forms of evaluation are warranted. (2) Pension and Health Coverage for Dislocated Workers was viewed favorably by workers. (3) Health Benefits Under COBRA was viewed favorably but some readers suggested more detailed content, and (4) Savings Fitness was viewed favorably but some suggested that more graphics and illustrations would have been helpful. Outreach Surveys: (1) Attendees and sponsors of outreach seminars rate presenters favorably. (2) EBSA received its highest score for presenters being knowledgeable and well informed. (3) EBSA received its lowest scores for information being clear and easy to understand. Website Surveys: (1) A wide variety of users access the web site and for a variety of reasons. (2) Users spend a substantial amount of time and many are repeat visitors. (3) A majority of users give the web-site positive ratings. (4) The website has strengths (e.g. clear, concise, useful, and trustworthy) but also some weaknesses (e.g. search engine, layout, easy to find information).
Recommendations: Participant Assistance Customer Satisfaction Surveys and Mystery Shopping: (1) use qualitative evaluation results in conjunction with customer satisfaction results to direct next steps; (2) share individual results with advisors and managers to aid training and customer service improvement; (3) share best practices of top BAs and offices for meeting service attributes described above; (4) examine existing telephone systems and improve routing of calls, if necessary; (5) examine ways to make EBSA and its services easier to find and contact; and (6) examine the fit of advisers for the customer service role. Gallup also noted that EBSA's GPRA goal (75% of customers rating services a four or five on a five point scale) is too ambitious for the type of service that EBSA provides and recommended that a goal of reaching 67% over the next four years was more appropriate. Publications: (1) consider a qualitative evaluation instead of a survey; and (2) consider adding more details and graphics to the publications Health Benefits under Cobra and Savings Fitness. Outreach: (1) re-examine program content and format; and (2) continue to evaluate the presenters participating in the session. Website: (1) redesign layout; (2) improve links on Department of Labor's main site; and (3) improve the search engine.
Actions Taken: Participant Assistance Program: The Gallup Organization conducted training on the results of both participant assistance evaluations for each of the 11 offices that carry out this mission. Benefit Advisors in each office worked together to devise a plan for improving their office's scores. These plans are being implemented and will be reviewed throughout the fiscal year in conjunction with the new office-by-office satisfaction ratings to be developed by Gallup at the end of each quarter. The Department has accepted Gallup's recommendation to revise the GPRA goal from 75% to 67%. Publications: Suggestions will be taken into consideration when revising the publications. Outreach: EBSA will continue to evaluate this service and relate updated findings to the presenters and offices conducting outreach. Web site: EBSA has shared Web site survey results with the Department's Web site coordinator for his consideration regarding issues EBSA cannot resolve, including the search engine and the Web site's overall layout.
Additional Information: A copy of the complete report can be obtained from the Employee Benefits Security Administration, 200 Constitution Avenue, N.W., Room N-5702 Washington, D.C., 20210.

18. Employment Standards Administration: Federal Employees Compensation Act (FECA) Program Effectiveness Study, March 2004.

Performance Goal(s) Affected: 2.2C — Minimize the human, social and financial impact of work-related injuries for workers and their families.

Conducted by: ICF Consulting

Findings: The study, a broad review of program's effectiveness, focused on disability case management and wage-loss benefit payments. It included a statistical review of performance data, opinion surveys of FECA program staff and stakeholders, and review of promising practices in other governmental and private workers' compensation systems. ICF's assessment of FECA was generally positive, as indicated by the finding that "OWCP is currently employing practices that are consistent with those found in well administered, modern workers' compensation systems."
Recommendations: ICF recommended that: (1) OWCP reformulate staffing allocations and examine caseload levels in light of the heavy per examiner workload compared to other compensation programs; (2) increase collaboration among its district offices; (3) develop more in-depth data analysis capability; and (4) conduct regular and rigorous training for Federal employing agencies and work with those agencies to strengthen their disability management and injured worker reemployment programs.
Actions Taken: OWCP is currently reviewing these recommendations for adaptation to the FECA program. This review includes consideration of several industry disability and return-to-work practices and better informing Federal agencies of their own injured employees' FECA status as well as FECA's most current tracking and review methods.
Additional Information:

19. Employment Standards Administration: Evaluation of Office of Federal Contract Compliance Programs Westat Final Report, December 2003.

Performance Goal Affected: 3.2A — Federal contractors achieve equal opportunity workplaces.

Conducted by: Westat

Findings: The Westat research findings suggest that OFCCP's effect on the employment growth of non-favored groups is primarily through the review process, and not through the threat of review among all contractors. Because reviewed establishments were more likely to be discriminators (OFCCP targeted establishments based on expectations for finding discrimination), the results suggest that the OFCCP review process was highly effective.
Recommendations: Of the 11 report recommendations, the most substantive was that OFCCP should use a model to predict discrimination among reviewed contractors and estimate predicted rates of discrimination for contractor establishments to rank them for the review process based on their predicted rate of discrimination. In addition, Westat advised OFCCP to consider: (1) defining labor markets by taking into consideration the most commonly used industry and occupation hiring practices; (2) improving the process of targeting contractors for review by tracking the history of establishments' utilization of non-favored groups and findings of compliance reviews; (3) how the review process can be incorporated into the model to estimate the probability of discrimination because it is the review process that determines discrimination; (4) further research on the effect of equal employment opportunity and OFCCP review on changes in employment shares of non-favored groups; and (5) other measures to determine the effect of equal employment opportunity and OFCCP review.
Actions Taken: OFCCP has incorporated methodologies Westat recommended into its Federal contractor selection system to improve targeting of Federal contractors for compliance evaluations. OFCCP plans to undertake research to: (1) improve the review targeting system by using newly available Census data in its data collection and audit procedures; and (2) design a set of metrics to best measure OFCCP performance in future years.
Additional Information: Copies of the report may be obtained by contacting the Office of Federal Contract Compliance Programs, (202) 693-0101.

20. Employment and Training Administration: Creating Partnerships for Workforce Investment: How Services Are Provided Under WIA, September, 2003.

Performance Goal(s) Affected: 1.1A — Increase the employment, retention, and earnings of individuals register ed under the WIA Adult program. 1.1F — Increase the employment, retention, and earnings replacement of individuals registered under the WIA dislocated worker program.

Conducted by: Berkeley Policy Research Inc.

Findings: The Workforce Investment Act (WIA) of 1998 was the first major overhaul of the nation's workforce development system in more than 15 years. Among other things, the legislation expanded opportunity for public and private entities to deliver employment and training services through the public workforce investment system. This study explores local workforce investment system variation and how local boards use non-profit, for-profit, educational, and governmental agencies to deliver WIA services. Study findings are based on case studies of sixteen local boards across eight states conducted between December 2001 and September 2002.
Recommendations: None
Actions Taken: None
Additional Information:

21. Employment and Training Administration: The Workforce Investment Act in Eight States: State Case Studies from a Field Network Evaluation: Volumes 1-2, February 2004.

Performance Goal(s) Affected: 1.1A — Increase the employment, retention, and earnings of individuals register ed under the Workforce Investment Act Adult program. 1.1F — Increase the employment, retention, and earnings replacement of individuals registered under the WIA dislocated worker program.

Conducted by: Rockefeller Institute of Government

Findings: This two-volume set of state case studies is the result of eighteen months of work by the project's researchers to understand how states and localities interpreted and operationalized the provisions of the Workforce Investment Act (WIA) of 1998. The project took a close look at the orientation, governance, structure, and services of the workforce investment systems in eight states, and two local areas in each state, to provide information for the reauthorization of WIA. Field researchers in each state conducted a series of interviews with members of state and local boards and their staff, state and local elected officials and their staff, state agency officials responsible for workforce development and welfare programs, service providers, advocates, and other interested parties. Researchers also studied sample One-Stop Career Centers in each state. The case studies for Maryland, Michigan, Missouri, and Oregon are included in the first volume. The case studies for the states of Florida, Indiana, Texas and Utah are included in the second volume.
Recommendations: None
Actions Taken: None
Additional Information:

22. Employment and Training Administration: Internet Initial Claims Evaluation, October 2003

Performance Goal(s) Affected: 2.2A — Make timely and accurate benefit payments to unemployed workers.

Conducted by: HeiTech Services, Inc. and Mathematica Policy Research, Inc.

Findings: Over the past five years, 33 State Workforce Agencies have implemented systems which allow unemployed workers to file their initial unemployment insurance claim over the internet. In addition to providing unemployed workers with extended hours of access, these systems have reduced administrative costs and provided a more convenient service method for unemployed workers to file an unemployment insurance initial claim. Policymakers chose to evaluate state Internet UI claim filing systems, with a focus on service delivery, security, fraud and abuse controls, and cost effectiveness. The report concludes that Internet-filed claims are convenient for users, low-cost to states, and have no apparent adverse impacts on timeliness or accuracy.
Recommendations: Although the report made no formal recommendations, it gives an excellent overview of Internet claims filing, and supports ETA's decision to promote states' use of the Internet for UI administration. The report does caution, however, that states need to be vigilant to protect against fraudulent exploitation and system security compromises.
Actions Taken: ETA continues to support development of Internet-based operations. It funded claims-taking systems through implementation grants to 42 states. It is currently funding state systems to enable employers to register and submit tax and wage reports through the Internet.
Additional Information:

23. Occupational Safety and Health Administration: Evaluation of OSHA's Impact on Workplace Injuries and Illnesses in Manufacturing Using Establishment-Specific Targeting of Interventions, July 2004.

Performance Goal(s) Affected: 3.1D — Reduce occupational injuries and illnesses.

Conducted by: Eastern Research Group (ERG)

Findings: The impact of High Hazard Notification/Cooperative Compliance Program letters to employers without any subsequent OSHA inspections on the number of Lost Workday Injury and Illness (LWDII) cases over three years was estimated to be a 4.8-5.1 percent reduction. (This estimated reduction in the number of LWDII cases over three years would also reflect the positive effects of employers' safety and health initiatives in response to the greater likelihood of an OSHA inspection, including seeking OSHA-funded on-site consultation services.) When the letters were followed by Site Specific Targeting or Interim Targeting Program inspections, the reduction was between 12.0 and 13.8 percent. The analysis controlled for industry-wide declines in injury and illness rates, so that these reductions were in addition to the declines in injury and illness rates in manufacturing. Comparison to previous studies of OSHA's previous industry-level targeting shows that establishment-specific targeting is more effective.
Recommendations: ERG recommended that OSHA continue and refine its site-specific targeting system. Further research that OSHA could conduct to meet this goal would include determining: (1) which interventions (such as High Hazard Notification Letters, inspections, and onsite consultation visits) are most effective at reducing injuries and illnesses at establishments with given sets of characteristics; and (2) whether multiple interventions in combination with High Hazard Notification Letters have a cumulative positive effect. ERG also suggested that related analyses could be conducted to find out whether there is an optimal combination of types and numbers of interventions to maximize reduction in injuries and illnesses. This refinement might allow OSHA to estimate injury and illness reductions due to specific combinations of strategically timed OSHA interventions and would help improve the cost-effectiveness of its use of resources.
Actions Taken: OSHA is continuing its site-specific targeting system and is looking into the possibility of obtaining funding to find out whether there is an optimal combination of interventions to maximize reduction in injuries and illnesses.
Additional Information: Call (202) 693-2165 for a copy

24. Occupational Safety and Health Administration: Lookback Evaluation of OSHA's Standard for Presence Sensing Device Initiation (PSDI) of Mechanical Power Presses, May 2004

Performance Goal(s) Affected: 3.1D — Reduce occupational injuries and illnesses.

Conducted by: ICF

Findings: The technology for PSDI systems has not changed since the PSDI standard was adopted in 1988, but the technology for controlling mechanical power presses has changed considerably since OSHA promulgated that standard. A number of operating modes that are not addressed in OSHA's standard for mechanical power presses are now used, such as computer-controlled press operation. The PSDI Standard has never been implemented and so it has been ineffective and has not produced the benefits sought by OSHA, namely, allowing industry to use a system that would increase productivity and improve safety for employees.
Recommendations: If the benefits OSHA sought in the PSDI standard are to be gained (i.e., improved worker safety and employer productivity), the standard needs to be changed.
Actions Taken: OSHA has decided to update its Mechanical Power Presses Standard to the most current version of the industry consensus standard or something similar.
Additional Information: The complete report can be found on the Federal Register website at

25. Veterans' Employment and Training Service: Assessment of Unemployed Veterans' Needs for the Department of Labor's Veterans' Employment and Training Service, December 2003

Performance Goal(s) Affected: 1.1E Increase the employment and retention rate of veteran job seekers registering for public labor exchange services.

Conducted by: Battelle Memorial Institute — Centers for Public Health Research and Evaluation

Findings: This study collected data from a set of seven participating States on individual veteran registrants, including demographic characteristics, services provided, and subsequent earnings experiences. The primary analyses examined the extent to which different sets of services resulted in different earnings experiences for different subgroup of veterans. The overall findings are: (1) Strong evidence was obtained indicating that "the overwhelming majority" of unemployed veterans register with the Public Labor Exchange. (2) About nine-tenths of the registered veterans receive at least one service, while about two-thirds receive two or more services. (3) The relatively high rate of unemployment recorded for young veterans is very similar to the pattern observed for other new entrants to the civilian labor force, such as recent high school graduates. (4) The average quarterly earnings of veteran registrants during the year following registration are lower than their average earnings during the quarter prior to registration, which is typical for workers who experience a spell of unemployment. (5) Traditional labor exchange services, such as job search and referral, are associated with relatively positive subsequent earnings for veterans who are: (a) young (under 25); and (b) recently separated. (6) Case management is associated with relatively positive subsequent earnings for veterans who are: (a) older (over 44); and (b) recently separated. (7) For those registered veterans who are not recently separated, who are 25-44, who are typical UI (as opposed to UCX) claimants, there is no consistent association between any service or combination of services received and subsequent earnings.
Recommendations: Recommendations focused on two areas, with specific suggestions within each area: (1) Restructuring of service delivery philosophy: Public Labor Exchange services were found to typically involve brief interventions with job seekers who frequently return for services. To counteract that tendency, it was suggested that VETS consider a more holistic, as opposed to episodic, approach to service delivery. Such an approach might include provision of greater support following entry to employment to reduce the recurrence of unemployment. (2) Introduction of new services and linkages: Case Management might be more successful if linkages with other service providers were strengthened in an "integration of services" approach.
Actions Taken: VETS is planning to use data compiled by this study as a basis for developing preliminary baseline estimates of Earnings Gain, in anticipation of the application of that performance measure under the Common Measures. In addition, VETS is considering convening a work group to undertake a policy development initiative in light of the recommendations from this study.
Additional Information: To obtain copies of this report, contact the Veterans' Employment and Training Service at 200 Constitution Ave. NW, Room S-1325, Washington, DC 20210 or call (202) 693-4749.

26. Veterans' Employment and Training Service: Measuring Strategies to Achieve Performance Goals, February 2004

Performance Goal(s) Affected: 1.1E Increase the employment and retention rate of veteran job seekers registering for public labor exchange services.

Conducted by: The Bradson Corporation

Findings: With the advent of UI wage record based reporting, VETS is seeking performance measures that are linked to employment outcomes, but can be measured in something closer to real time. VETS needs reliable leading indicators that are clearly linked to performance and that can be used as a management tool to improve the return on the VETS budget in any given budget execution year. A proven management framework, the Balanced Scorecard (BSC) has been developed for this reason. BSC for public sector organizations generally includes four perspectives: (1) Customer Perspective; (2) Financial Perspective; (3) Internal Business Process Perspective; and (4) Organizational Capacity/Employee Perspective.
Recommendations: Study recommendations are based on the four perspectives identified above. (1) Customer Perspective: One proxy for EER that can be measured at the state level in something close to real time, based on BLS data is the Veterans' Relative Unemployment Rate, which is the ratio of the unemployment rate for veterans to the unemployment rate for non-veterans. State Directories of New Hires also are a potential source of leading indicators of EER. Some states already use this resource as a leading indicator, but additional exploration will be required before VETS can implement an indicator based on New Hire Reporting. (2) Financial Perspective: Until Efficiency Measures can be fully implemented, it is recommended that the VETS BSC include a measure of planned versus actual costs for DVOPs and LVERs. (3) Internal Business Process Perspective: The internal process perspective measures activities, efforts and workflow for those processes that are key to the organization's strategy. For the VETS BSC, the key categories identified are: (a) Defining the Input Stream; (b) Providing Services; (c) Providing Outreach; and (d) Implementing Priority of Service. (4) Organizational Capacity/Employee Perspective: The organizational capacity perspective addresses whether an organization has the right resources to implement its strategy. In the case of VETS, the key resources to be measured here are the DVOPs and LVERs. Four indicators are proposed that address capacity and occupancy for DVOP and LVER staff: (a) DVOP slots filled; (b) DVOP slots vacant; (c) LVER slots filled; and (d) LVER slots vacant.
Actions Taken: VETS is consulting with BLS regarding the Veterans' Relative Unemployment Rate measure because the use of this measure requires application of data with statistical error that is very high for small States. VETS seeks to determine the "cut point" in terms of State population, above which use of the measure would be appropriate.
Additional Information: To obtain copies of this report, contact the Veterans' Employment and Training Service at 200 Constitution Ave. NW, Room S-1325, Washington, DC 20210 or call (202) 693-4749.

27. Veterans' Employment and Training Service: Strategies for Implementing Priority of Service to Veterans in Department of Labor Programs, September 2004

Performance Goal(s) Affected: 1.1E Increase the employment and retention rate of veteran job seekers registering for public labor exchange services.

Conducted by: The Urban Institute

Findings: A range of program strategies are used to provide priority of service to veterans, as well as strategies used in serving non-veteran populations. Two categories of program strategies were identified: (1) Procedural Strategies: Workforce development offices currently use several strategies to serve veterans within their regular office operations. Five types of procedural strategies were identified: (a) outreach activities; (b) intake and registration procedures; (c) client flow adaptations; (d) role of DVOPs and LVERs; and (e) integration and collaboration with other programs. (2) Electronic web-based and other self-service strategies: Given the increased reliance on the Internet in recent years, there has been an explosion in web-based, self-service strategies in the publicly-funded workforce development system. These strategies fit within two categories: informational strategies and job-matching strategies. Four prototypical web-based models were identified for informational strategies. For job-matching strategies, the study found that many state job-matching websites provide veterans with the option of self-identifying, which allows for various priority options within the job-matching system.
Recommendations: The Jobs for Veterans Act gives states and localities latitude in implementing priority of service for veterans. There are three approaches that program administrators may consider: (1) Establish service goal priorities; performance measurement and goal-setting can help to clarify the intent of priority of service and serve as a motivating force for all program staff. (2) Consider local office procedural modifications that can be tailored to local operations to improve priority of service to veterans. (3) Improve self-service resources and electronic strategies. For example, state workforce development websites can be redesigned to improve access and information for veterans, using the following criteria: (a) visibility of information; (b) types of information; (c) active as well as passive features; and (d) evaluation component.
Actions Taken: Since the public labor exchange has historically observed priority of service, the impact of the new priority of service provision falls primarily upon other workforce development programs funded by ETA. In response to the new requirement, ETA has issued overall guidance and the individual ETA programs have developed guidance specific to their missions. General information about the overall ETA guidance and the related program-specific guidance can be accessed at:
Additional Information: To obtain copies of this report, contact the Veterans' Employment and Training Service at 200 Constitution Ave. NW, Room S-1325, Washington, DC 20210 or call (202) 693-4749.

Previous Section Next Section