Maximizing the use of blocking in record linkage : theory and simulation

Abstract

Two data files can be compared only if they have common variables. Pairs are formed by the combination of two records, one from each file. The actual status of a pair of records is either a match or a nonmatch, but is usually unknown. Record linkage methodology is used to make a decision on the actual status of each observed pair. Decision-errors are committed whenever the actual status and the decision are different. The new decision-rule is based on minimizing the sum of these errors of misclassification. Discrete discriminant analysis has been used to discriminate between match and nonmatch pairs. It is assumed that there are chance factors associated with any pair of records belonging to the sets of match and nonmatch pairs. A high value of the chance factor implies that the reliability of the pair belonging to the corresponding set is low. This is the basic premise in this research. Two discriminant scores are computed from the conditional probabilities of the observed values among match and nonmatch pairs, and the true proportion of match and nonmatch pairs. A pair is decided as a link or a nonlink, depending on which score is larger. The discriminant scores are estimated by using the chance probabilities and applying Bayes' theorem on conditional probability. In order to reduce the number of unnecessary comparisons, the files are often blocked by one or more variables. A new method for choosing the best blocking scheme has been proposed in this study. Some of the actual match pairs may be missed because they fall in different blocks. The idea is to select the scheme for which the failure to pick up the match pairs is the least. In comparison to some of the commonly used linkage procedures, the new method is statistically sound, yet it only requires the frequency counts of the variables in the files. Simulated experiments using the new method have been very successful. without any data error the results were 100% correct. Even with 10% data errors 80% of the true matches were correctly decided.