UIPL22-05.pdf

ETA Advisory File Text

RECISSIONS None EXPIRATION DATE April 28 2006 CLASSIFICATION Data Validation CORRESPONDENCE SYMBOL OWS DPM EMPLOYMENT AND TRAINING ADMINISTRATION ADVISORY SYSTEM U. S. Department of Labor Washington D.C. 20210 DATE April 28 2005 ADVISORY UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 22-05 TO STATE WORKFORCE AGENCIES FROM CHERYL ATKINSON s Administrator Office of Workforce Security SUBJECT Unemployment Insurance Data Validation UI DV Program Software and Policy Guidance 1. Purpose . To provide policy guidance on how the data validation program will be administered and information about the characteristics of UI DV software currently under development. 2. References . Unemployment Insurance Program Letter UIPL No. 03-01 Unemployment Insurance Data Validation UI DV Program Update and Implementation Plans October 23 2000 UIPL 14-02 Unemployment Insurance Data Validation UI DV Program Implementation Plan February 22 2002 UIPL 10-03 Unemployment Insurance Data Validation UI DV Status and Guidance for Fiscal Year FY 2003 December 16 2002 UIPL 28-03 Call Memo for the Fiscal Year FY 2004 Unemployment Insurance UI State Quality Service Plan SQSP July 2 2003 UIPL 3-04 Unemployment Insurance Data Validation UI DV Program Status and Guidance November 21 2003 UIPL 26-04 Call Memo for the Fiscal Year FY 2005 Unemployment Insurance UI State Quality Service Plan SQSP June 24 2005 . 3. Background . The UI DV program is a cost-effective means of ensuring the accuracy of data on UI reports required by the Employment and Training Administration ETA . These reports are used for performance measurement as well as budget formulation and allocation. Beginning in 2002 validation activities were facilitated by PC-based software produced by Mathematica Policy Research Inc. MPR . ETA has developed validation software to run on the Sun computers resident in the states to replace the PC-based software developed by MPR. The Sun-based software has been extensively tested and will soon be released. 4. Software Development . The Sun-based software implements the same DV concepts and functions as the MPR-developed DV software. It uses the same logical structure to validate the same report cells as the MPR software uses the same investigation techniques for the samples and accepts the extract files developed for the MPR software. The new software makes several improvements over the MPR prototype many of them in response to comments from users of the MPR software including Reduced Data Entry Unlike the MPR software which requires states to hand-enter values from the ETA required reports being validated the new software will automatically retrieve reported values from the state s electronic copy of the required reports submitted to ETA. Easier Submissions Validation results will be uploaded to the national database once the state indicates that they are complete--the same way UI required reports are updated. This avoids the effort and technical issues currently associated with exporting results from state DV software and archiving them at ETA. Standardized Reports ETA is developing a suite of UI DV output reports summarizing validation results. States will be able to display their own validation results in ETA s report formats making technical assistance easier. Improved Structure To better reflect the uses made of UI benefits report data some report cells were grouped differently for pass fail purposes a random sample was added and other samples were redefined. The attachment Administration of Unemployment Insurance Data Validation explains the basic logic of UI DV and shows how the structure of validation will change coincident with the move from the MPR PC-based software to the Sun-based software. 5. Administering UI DV . Data validation is a systematic procedure for assessing the accuracy of UI required reports. All states must operate this system in order to determine the accuracy of required reports and if necessary to help improve reporting accuracy. UI DV subjects reported counts and their underlying data to a series of tests with attendant pass fail criteria. States will conduct UI DV every three years with the following three exceptions - 3 - Government Performance and Results Act GPRA data must be validated annually. These are o Timeliness of First Payments Benefits Population 4 New Intrastate and Interstate Liable First Payments group and random sample o Timeliness of New Status Determinations Tax Population 3 Total New Status Determinations and New 90 Days o Detection of Overpayments Benefits Population 12 group 227 Total Established and Total Established random sample and o Reemployment of UI Claimants when measure is in place --Benefits Population 4 New Intrastate and Interstate Liable First Payments group and random sample . Validation Failures . Pass fail groups and report items benefits random samples and tax sorts and File Integrity Validation FIV samples that fail to pass validation must be revalidated within one year. o For benefits validation results submitted prior to implementation of the Sun- based software pass fail determinations will be made on the basis of the groups and random samples defined under the MPR software. In the MPR structure groups and sample scores are independent. If a group fails the population for which it belongs must be validated again within a year. If a random sample fails the sample must be validated again within a year. It is expected that any causes for failure will be corrected before validating groups or samples again. o For benefits validation results derived from the Sun-based software pass fail determinations will be made using the revised structure of groups and random samples. In this revised structure 16 random samples relate directly to pass fail groups. For these groups if the random sample fails to pass validation not only must the sample be revalidated within a year but also the report validation must be repeated because the counts cannot be considered reliable. This generally implies the need to correct data quality problems in the extract file or the underlying database. o For all tax validation results obtained using both the MPR and Sun software pass fail determinations will be made on the basis of the report items sorts and FIV samples as defined under the MPR software. The tax structure has not changed. In tax if any validation counts dollars sort or FIV sample within a population fail validation the entire population fails and must be validated again within a year. UI Required Reports Produced by New Reporting Software. When states revise or replace software that could affect the data in required reports reports produced by that software should be revalidated within one year. - 4 - 6. Other Aspects of Data Validation. The preceding sections describe the bulk of data validation the most complex and automated portions. UI DV has two sets of other requirements described in Modules 4 and 5 of the validation handbooks which involve different and much less automated processes. Although previous implementation instructions have not emphasized these requirements they are an integral part of the validation design. Validation of the universes and sampling methods used to identify the samples used for the Benefits Timeliness and Quality BTQ nonmonetary determinations and Lower Authority Appeals quality reviews and the Tax Performance System TPS Acceptance samples for Status Determinations and Field Audits Module 4 . These areas will be validated every third year unless the universe from which the sample is drawn is not within 2 of the reference report count. In that case the universe match must be repeated the following year. o Sample Selection. In all cases if the validation finds that the sample selection is not random a new sample must be drawn and shown to be random before the review can proceed. Module 4 of the Data Validation Handbook outlines the procedures for determining whether a sample is random. o Timing. Because the validation of the sample must precede the BTQ or TPS review validation must occur as soon as the appropriate sample can be drawn. These reviews are to be done according to the following schedule For the TPS field audit acceptance sample when the data for the third quarter of the calendar year become available and For the TPS status determinations review as soon as the calendar year s data are available to draw the annual samples. Validation of the BTQ quality samples may be done in advance of any quarterly review during the validation year. Wage Item Validation Module 5 . Wage items reported on the ETA 581 report a workload item must be validated every third year. Procedures Under Exploration. In addition to validating the random selection of the BTQ and TPS samples the full validation design was intended to include supplemental reviews of data quality based on findings from those sample reviews. TPS acceptance samples were to be used to assess the accuracy of data on field audits and status determinations in the states tax databases. Similar information on nonmonetary determinations was to be obtained from BTQ quality samples e.g. were the correct date of detection and outcome entered into the state s database. This aspect of the validation design remains under development. We will explore whether and how these should be incorporated into the tax and benefits validation processes at a later date. - 5 - 7. Actions and Policies in FY 2005. Two major transitions for UI DV will occur during FY 2005 the replacement of the MPR software with Sun-based UI DV software and the synchronization of the validation schedule with the SQSP performance year. a. Installation of the Sun-based Software. A tutorial will be released about the same time as the new software. State data validation coordinators should carefully review the tutorial prior to using the new software. The Benefits and Tax UI DV handbooks will be revised to reflect the software changes and the content of this UIPL. Formal training in the new software will be offered through the Minnesota Training Center this summer. b. The Validation Year Concept. With implementation of the Web-based software DV will begin using a validation year that corresponds with the SQSP performance year. Validation activities will relate to reports submitted for the 12-month period from April 1 through March 31 and hence extracts must be constructed with data from this period. Although states may validate the validation-year data at any time reports of those validation results must be submitted by May 10 to enable integration into the upcoming SQSP process. The tax report ETA 581 data for the quarter ending March 31 are not due to the national office until May 20. Thus to meet the submission deadline tax validations must be performed for one of the quarters ending June September or December. c. Actions during FY 2005. March 31 2005 marked the end of validation year 2005. Although states are urged to begin familiarizing themselves with the new validation software as soon as it is released the MPR prototype software must be used for validations officially submitted for validation year 2005. These validations must be submitted by June 30 2005 no MPR results will be accepted after June 30 . Passing validation during validation year 2005 will permit a fresh start using the new software and structure for the next validation cycle. To enable all DV reports to be processed in time for the FY 2006 SQSP reports must be submitted by June 30 2005. All validation results submitted for report periods March 31 2005 and earlier will be considered done in validation year 2005. o Except for GPRA elements benefits groups and random samples and tax populations that pass both data element validation and report validation as of March 31 2005 will be due for revalidation during validation year 2008 i.e. results must be submitted by May 10 2008. If not already completed Module 4 and 5 validation reports are to be submitted by May 10 2006. d. Validation Year 2006. April 1 2005 began the new permanent validation cycle. - 6 - This validation cycle includes UI reports for the months or quarters for the period April 1 2005 through March 31 2006. These validations must be done using the new software. This would involve a new benefits structure as explained in the attachment to this UIPL and in the revised handbooks. The following validations are required to be done during validation year 2006 and submitted by May 10 2006 o GPRA elements o Any reports produced with new state reporting software during validation year 2005 and not previously validated as noted in 5 above o Benefits groups and random samples for report periods up to March 31 2005 that did not pass as of June 30 2005. In case there have been changes in benefits groups or random samples that failed to pass validation under the old software states conducting a revalidation using the new software should use the closest approximation under the new software alignment. Use Tables 1 and 2 of the Attachment as a guide. o If Benefits Population 14 must be resubmitted during validation year 2006 the new random sample added in the revised validation structure must be completed along with the summary validation. o Tax populations containing groups or report items that did not pass report validation or components of data element validation that did not pass by June 30 2005 and o TPS and BTQ quality sampling reviews not completed during validation year 2005. Although reports for validation year 2006 must be submitted by May 10 2006 states are encouraged to submit validation results as soon as they are completed. - 7 - Summary of Validation Requirements for Validation Years 2005 and 2006 Validation Year Program UI Reports for the Period Latest Report Due Validation Due Software to be Used Benefits Through 3 31 2005 May 1 2005 June 30 2005 MPR 2005 Tax Through 3 31 2005 May 20 2005 June 30 2005 MPR Benefits 4 1 2005 3 31 2006 May 1 2006 May 10 2006 Sun 2006 Tax 4 1 2005 3 31 2006 May 20 2006 May 10 2006 Sun In order to be submitted timely tax validations for FY 2006 and later years must be for either the June September or December quarters. 8. DV Contacts. Please refer questions to your regional office DV coordinator. The Web site maintained for the DV program can be accessed at www.ows.doleta.gov dv. This site is also the source for the MPR UI benefit and tax validation software and accompanying documentation and for all updates of materials related to UI data validation. Technical support for the software is available by calling the Office of Workforce Security Technical Support Hotline at 1-800-473-0188. 9. Actions . State Workforce Agency Administrators are requested to Distribute this advisory to appropriate staff Ensure complete implementation of UI data validation or establish a schedule for completion this includes the activities described in Modules 4 and 5 of the Handbooks By June 30 2005 send results of validations done using the MPR software to the national office dvrpts uis.doleta.gov with a copy to the appropriate regional office DV coordinator Identify technical issues administrative issues or other barriers that may impede implementation of the complete DV program and review this situation with appropriate regional office staff to devise workable solutions o Although it is strongly urged that states devote resources to implementing UI DV in cases where this is not feasible conduct the Workload Validation program in accordance with DOL policy and submit workload validation results to the regional office unless the national office grants a waiver based on the regional office s recommendation and - 8 - Ensure that validation staff makes preparations to transition to the Sun-system software including contacting Sun-system administrators to establish a Sun account and permissions. 10. Attachment . Administering Unemployment Insurance Data Validation.