DOL-ILAB SDC - ehsection3_relabelled
Pablo Diego-Rosell, PhD
2021-06-22
- Setup filenames
- Setup data, functions and create dictionary for dataset review
- Direct PII: variables to be removed
- Direct PII-team: Encode field team names
- Small locations: Encode locations with pop <100,000 using random large numbers
- Indirect PII - Ordinal: Global recode or Top/bottom coding for extreme values
- Indirect PII - Categorical: Recode, encode, or Top/bottom coding for extreme values
- Matching and crosstabulations: Run automated PII check
- Open-ends: review responses for any sensitive information, redact as necessary
- GPS data: Displace
- Save processed data in Stata and SPSS format
rm(list=ls(all=t))Setup filenames
filename <- "ehsection3_relabelled" # !!!Update filename
functions_vers <- "functions_1.7.R" # !!!Update helper functions fileSetup data, functions and create dictionary for dataset review
source (functions_vers)Visually inspect variables in "dictionary.csv" and flag for risk, using the following flags:
# Direct PII: Respondent Names, Addresses, Identification Numbers, Phone Numbers
# Direct PII-team: Interviewer Names, other field team names
# Indirect PII-ordinal: Date of birth, Age, income, education, household composition.
# Indirect PII-categorical: Gender, education, ethnicity, nationality,
# occupation, employer, head of household, marital status
# GPS: Longitude, Latitude
# Small Location: Location (<100,000)
# Large Location (>100,000)
# Weight: weightVar
# Household ID: hhId,
# Open-ends: Review responses for any sensitive information, redact as necessary Direct PII: variables to be removed
# !!! No Direct PIIDirect PII-team: Encode field team names
# !!! No Direct PII-teamSmall locations: Encode locations with pop <100,000 using random large numbers
!!!Include relevant variables, but check their population size first to confirm they are <100,000
dropvars <- c("dise")
mydata <- mydata[!names(mydata) %in% dropvars]
locvars <- c("a006_a_block_id", "a007_a_vill_id")
mydata <- encode_location (variables= locvars, missing=999999)## [1] "Frequency table before encoding"
## a006_a_block_id. 006 Block ID
## 1 2 3 4 5 6 7 8 9
## 210 169 196 406 99 197 156 426 531
## [1] "Frequency table after encoding"
## a006_a_block_id. 006 Block ID
## 279 280 281 282 283 284 285 286 287
## 210 426 169 197 196 531 156 99 406
## [1] "Frequency table before encoding"
## a007_a_vill_id. 007 Village ID
## 1 2 3 4 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22
## 17 16 17 15 20 30 28 14 16 15 17 24 24 15 18 21 16 17 18 30 22
## 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
## 20 19 32 27 26 18 15 15 24 26 22 16 30 19 17 21 27 16 16 18 16
## 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
## 29 20 23 22 19 17 17 16 18 26 24 27 18 16 23 13 24 20 16 19 18
## 65 66 67 68 69 70 71 72 73 74 75 76 77 78 80 81 82 83 84 85 87
## 29 16 19 21 24 13 18 19 18 23 23 17 22 29 30 16 22 19 17 13 16
## 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108
## 22 15 20 19 19 20 13 17 27 29 21 25 22 24 21 15 19 13 31 14 27
## 109 110 111 112 113 114 115 116 117 118 119 120 121 122
## 21 17 21 27 14 24 20 17 22 22 20 13 10 10
## [1] "Frequency table after encoding"
## a007_a_vill_id. 007 Village ID
## 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629
## 16 21 15 17 30 19 16 22 29 17 20 18 23 26 27 17 20 21 13 17 32
## 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650
## 17 10 24 14 31 29 15 13 27 16 27 18 16 16 30 20 25 16 27 22 16
## 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671
## 22 15 14 15 16 20 17 18 17 16 29 16 17 17 19 26 17 20 28 26 22
## 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692
## 19 22 16 22 29 22 22 19 18 24 18 19 30 24 19 27 10 19 20 18 24
## 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713
## 21 20 19 24 23 17 24 18 30 22 19 21 13 19 13 18 16 21 21 15 18
## 714 715 716 717 718 719 720 721 722 723 724 725 726 727
## 15 24 17 27 15 14 21 13 20 23 23 16 24 13Indirect PII - Ordinal: Global recode or Top/bottom coding for extreme values
# Focus on variables with a "Lowest Freq" in dictionary of 30 or less.
mydata2 <- top_recode (variable="a302_age_", break_point=18, missing=NA)## [1] "Frequency table before encoding"
## a302_age_. 302 How old is [Name]?
## 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
## 1 1 1 1 1 5 5 3 13 11 16 13 19 24 29 22 13
## 19 20 21 22 23 25 <NA>
## 2 1 1 1 1 2 2204
## [1] "Frequency table after encoding"
## a302_age_. 302 How old is [Name]?
## 2 3 4 5 6 7 8
## 1 1 1 1 1 5 5
## 9 10 11 12 13 14 15
## 3 13 11 16 13 19 24
## 16 17 18 or more <NA>
## 29 22 21 2204
Indirect PII - Categorical: Recode, encode, or Top/bottom coding for extreme values
# !!!Include relevant variables in list below (Indirect PII - Categorical, and Ordinal if not processed yet)
indirect_PII <- c("a310_hh_receive_money_",
"a313_location_work_pay_",
"a315_hh_recived_money_",
"a317_leave_currt_location_")
capture_tables (indirect_PII)Matching and crosstabulations: Run automated PII check
# Based on dictionary inspection, select variables for creating sdcMicro object
# See: https://sdcpractice.readthedocs.io/en/latest/anon_methods.html
# All variable names should correspond to the names in the data file
# selected categorical key variables: gender, occupation/education and age
selectedKeyVars = c('a302_age_', 'a303_gender_') ##!!! Replace with candidate categorical demo vars
# weight variable (add if available)
# selectedWeightVar = c('projwt') ##!!! Replace with weight var
# household id variable (cluster)
selectedHouseholdID = c('hh_id') ##!!! Replace with household id
# creating the sdcMicro object with the assigned variables
sdcInitial <- createSdcObj(dat = mydata,
keyVars = selectedKeyVars,
hhId = selectedHouseholdID)
sdcInitial## The input dataset consists of 2390 rows and 29 variables.
## --> Categorical key variables: a302_age_, a303_gender_
## --> Cluster/Household-Id variable: hh_id
## ----------------------------------------------------------------------## Information on categorical key variables:
##
## Reported is the number, mean size and size of the smallest category >0 for recoded variables.
## In parenthesis, the same statistics are shown for the unmodified data.
## Note: NA (missings) are counted as seperate categories!## Key Variable Number of categories Mean size Size of smallest (>0)
## a302_age_ 24 (24) 8.087 (8.087) 1
## a303_gender_ 3 (3) 93.000 (93.000) 87
##
## (1)
## (87)## ----------------------------------------------------------------------## Infos on 2/3-Anonymity:
##
## Number of observations violating
## - 2-anonymity: 12 (0.502%)
## - 3-anonymity: 20 (0.837%)
## - 5-anonymity: 33 (1.381%)
##
## ----------------------------------------------------------------------Show values of key variable of records that violate k-anonymity
notAnon <- sdcInitial@risk$individual[,2] < 2 # for 2-anonymity
mydata[notAnon,selectedKeyVars]## # A tibble: 12 x 2
## a302_age_ a303_gender_
## <dbl> <dbl>
## 1 4 0
## 2 3 0
## 3 2 1
## 4 23 1
## 5 5 0
## 6 6 0
## 7 9 1
## 8 20 1
## 9 19 1
## 10 21 0
## 11 19 0
## 12 22 1Show values of key variable of records that violate k-anonymity
#mydata <- labelDataset(mydata)
notAnon <- sdcInitial@risk$individual[,2] < 2 # for 2-anonymity
mydata[notAnon,selectedKeyVars]## # A tibble: 12 x 2
## a302_age_ a303_gender_
## <dbl> <dbl>
## 1 4 0
## 2 3 0
## 3 2 1
## 4 23 1
## 5 5 0
## 6 6 0
## 7 9 1
## 8 20 1
## 9 19 1
## 10 21 0
## 11 19 0
## 12 22 1sdcFinal <- localSuppression(sdcInitial)
# Recombining anonymized variables
extractManipData(sdcFinal)[notAnon,selectedKeyVars] # manipulated variables HH## a302_age_ a303_gender_
## 48 NA 0
## 49 NA 0
## 304 NA 1
## 388 NA 1
## 1286 NA 0
## 1346 NA 0
## 1582 NA 1
## 1583 NA 1
## 2022 NA 1
## 2152 NA 0
## 2153 NA 0
## 2186 NA 1mydata [notAnon,"a302_age_"][mydata[notAnon,"a302_age_"]>17] <- NA
mydata [notAnon,"a303_gender_"][mydata[notAnon,"a302_age_"]<18] <- NA
# 12 cases do not meet 2-anonimty by gender and age. 6 Gender suppresions for under 18, 6 age supressions for 18 and older
createSdcObj(dat = mydata,
keyVars = selectedKeyVars,
hhId = selectedHouseholdID)## The input dataset consists of 2390 rows and 29 variables.
## --> Categorical key variables: a302_age_, a303_gender_
## --> Cluster/Household-Id variable: hh_id
## ----------------------------------------------------------------------## Information on categorical key variables:
##
## Reported is the number, mean size and size of the smallest category >0 for recoded variables.
## In parenthesis, the same statistics are shown for the unmodified data.
## Note: NA (missings) are counted as seperate categories!## Key Variable Number of categories Mean size Size of smallest (>0)
## a302_age_ 19 (19) 10.000 (10.000) 1
## a303_gender_ 3 (3) 90.000 (90.000) 83
##
## (1)
## (83)## ----------------------------------------------------------------------## Infos on 2/3-Anonymity:
##
## Number of observations violating
## - 2-anonymity: 0 (0.000%)
## - 3-anonymity: 0 (0.000%)
## - 5-anonymity: 4 (0.167%)
##
## ----------------------------------------------------------------------Open-ends: review responses for any sensitive information, redact as necessary
open_ends <- c("a308_a_oth_", "a309_travel_oth_")
report_open (list_open_ends = open_ends)
mydata <- mydata[!names(mydata) %in% open_ends] # Drop as actually verbatim data in local languageGPS data: Displace
# !!! No GPS dataSave processed data in Stata and SPSS format
Adds "_PU" (Public Use) to the end of the name
haven::write_dta(mydata, paste0(filename, "_PU.dta"))
haven::write_sav(mydata, paste0(filename, "_PU.sav"))
# Add report title dynamically
title_var <- paste0("DOL-ILAB SDC - ", filename)