Table Of ContentJournal of Official Statistics, Vol. 22, No. 4, 2006, pp. 681–710
Measuring the Disclosure Protection of Micro
Aggregated Business Microdata. An Analysis Taking
as An Example the German Structure of Costs Survey
1
Rainer Lenz
Dedicated to my teacher and friend Dietmar Schweigert, in memoriam
To assess the effectiveness of an anonymisation method with respect to data protection, the
disclosure risk associated with the protected data must be evaluated. We consider the scenario
where a possible data intruder matches an external database with the entire set of confidential
data. In order to improve his external database he tries to assign as many correct pairs of
records (that is, records referring to the same underlying statistical unit) as possible. The
problem of maximisation of the number of correctly assigned pairs is translated into a multi-
objective linear assignment problem (MOLP).
Regarding several variants of the micro aggregation anonymisation method applied to the
German structure of costs survey, we calculate approximative solutions to the MOLP obtained
by using two external databases as the data intruder’s additional knowledge. Finally, a
standard for so-called de facto anonymity is suggested.
Key words: Database cross-match; de facto anonymity; linear programming; micro
aggregation; scientific-use-file; statistical data confidentiality.
1. Introduction
The deep interest in secret data has a long tradition. Towards the end of December
1855 the deputy purveyor in chief in the Crimea, David Fitzgerald, submitted to the
chief medical officer a Confidential Report on the Nursing, since its introduction to the
Crimea on 23rd January. The contents of the confidential report soon became widely
known in medical and military circles. In it the purveyor criticised the nurses for lack
of discipline, for drunkennness and insubordination. In several letters Florence
Nightingale, running at this time a training establishment for nurses in Scutari,
expressed her displeasure concerning the report: “Having found that Mr. Fitzgerald’s
Confidential Report was Confidential only from myself, and has already ceased to be so
in the Crimea: : : ”
In the last ten years, the problem of confidentiality has become increasingly severe,
since the number of sources available to data intruders has risen, not least because of the
rapid expansion of the Internet. Disclosure occurs when an individual or an enterprise can
be re-identified in data that were collected with a promise to protect confidentiality.
1
Federal Statistical Office of Germany, Research Data Centre, Gustav-Stresemann-Ring 11, 65180 Wiesbaden,
Germany Email:
682 Journal of Official Statistics
Therefore distributors of data (such as statistical offices or private institutions) make sure
to handle confidential data with the utmost care. Their challenge is to pursue two
objectives, i.e., providing useful statistical information and ensuring confidentiality. That
is, the distributing institutions have to weigh the preservation of accuracy and analytical
validity of data against the anonymisation of data in order to minimise the risk of re-
identification of statistical units to which they relate.
In 1987, the Law on Statistics for Federal Purposes created the privilege of science,
allowing scientists and researchers access to so-called de facto anonymised
microdata. A data set was defined to be de facto anonymous if the costs of re-
identification exceeded the benefits of it (Sturm 2002). Before then, only completely
anonymised microdata files could be supplied to scientists. That is, the statistical offices
had to make sure that intruders had no chance to deanonymise data in order to gain
infomation about specific organisations. Moreover, the statistical offices have ethical
reasons for protecting respondents and confidentiality must be fully trustworthy to be able
to gather data from respondents.
In Rosemann et al. (2004) the analysis potential of several variants of micro aggregation
of the German structure of costs survey has been examined. In the present article the
associated re-identification risk is studied. Initial thoughts on this topic have been
presented in Lenz (2003) and Vorgrimler and Lenz (2003). In order to re-identify
statistical units, a data intruder needs additional knowledge about the units searched for
(e.g., in the form of an external database) containing variables the additional and the
confidential data have in common. Moreover, he needs knowledge about the participation
of the units in the target survey, the so-called response knowledge.
For an estimation of the re-identification risk (in business microdata), we consider three
scenarios of attack:
. A Assignment between original data and anonymised target data. (Calculation of
an upper bound for the re-identification risk)
. B1 Assignment between external data and formally anonymised target data (i.e.,
original datawithout direct identifiers). (Estimation of the natural protection in the data)
. B2 Assignment between external data and anonymised target data. (Realistic
scenario)
The results obtained from Scenario A do not represent the true picture. Nevertheless, it
would seem advisable to involve scenario A into the estimation of the re-identification risk
associated with the tested microdata file and hence into the decision about the de facto
anonymity of the file, since the available additional knowledge of a data intruder is
inaccessible. The results obtained from Scenarios B1 and B2 line out, how far by courtesy
of an assumption of the best possible additional knowledge in Scenario A the real re-
identification risk is overestimated.
2. Basic Definitions and Notations
Throughout the article we will use the following denotations: A (finite) graph G ¼
ðVðGÞ; EðGÞÞ is a relational structure, consisting of a (finite) set VðGÞ, the elements of which
2
are called vertices (or points), and a set EðGÞ # VðGÞ of unordered pairs of vertices,
Lenz: Measuring the Disclosure Protection of Micro Aggregated Business Microdata 683
called edges (or lines) of G. We denote these sets by V or E when there is no possibility of
confusion. We consider undirected graphs, fulfilling the implication
ða; bÞ [ E ) ðb; aÞ [ E. That is, E determines a symmetric binary relation. The edge
ðx; yÞ [ E is said to be incident with the vertices x and y, and x, y to be adjacent.
A graph S ¼ ðVðSÞ; EðSÞÞ is called a subgraph of G if VðSÞ # VðGÞ and EðSÞ #
EðGÞ hold.
G is called a bipartite graph with bipartition ðX; YÞ if VðGÞ is a disjoint union
V ¼ X < Y, so that every edge e is incident with both an x [ X and a y [ Y. Moreover,
if every x [ X is connected to every y [ Y, the graph G is said to be complete.
A matching M of G is a subgraph with the property that no two edges in M have a
0
common vertex. M is called maximum matching if there is no matching M with
0
M ,M . If v is a vertex of M, then M saturates v. Moreover, if every v [ V is
saturated, M is called perfect matching. A vector-weighted graph G is a
graph combined with a weight function
k
w : EðGÞ ! R ; e 7! ðw1ðeÞ; : : : ; wkðeÞÞ
which maps every edge e to a k-tuple of real numbers. In the case k ¼ 1 the graph G is a
weighted graph.
3. Types of Variables and Distances
In a database cross-match (see Elliott and Dale 1999), the data intruder matches an
external database B with the whole confidential database A. For this, he uses variables
which the external data have in common with the confidential data, the so-called key
variables Figure 1.
Fig. 1. Empirical key variables
Obviously, the reporting quality of these variables is crucial to the success of the
subsequent re-identification process. The set of key variables is partitioned into two
classes of variables, namely categorical and numerical variables, which are
described below.
3.1. Categorical and Numerical Variables
Numerical variables are defined to be discrete or continuous variables where the
difference between values has a meaning, e.g., “height”, “weight” of a person or “number
of employees”, “total turnover” of an enterprise. Regarding a numerical variable vi, its
2
values are distanced by diða; bÞ ¼ ðai 2 biÞ . In general, the treatment of numerical
variables admits less diversification than the treatment of categorical variables. When
analysing categorical data, it is always possible to work with counts or percentages of
684 Journal of Official Statistics
objects which fall within certain categories. We differentiate between nominal
variables (there is no ranking of the categories) and ordinal variables (the
categories have some kind of order, where differences between categories are
meaningless). Regarding a nominal variable vi, its values are compared for equality,
such that we define
(
0; if ai ¼ bi
diða; bÞ ¼ ð1Þ
1 otherwise
For a linear ordered variable vi, let c1 ,i c2 ,i : : : ,i cr be its ordered range (i.e.,
ð{c1; : : : ; cr};,iÞ is a well-ordered set). We then define
j{cjjminðai; biÞ #i cj ,i maxðai; biÞ}j
diða; bÞ ¼ ð2Þ
r
Since in practice there will often occur categorical variables with some non-linear partial
order (that is, there are at least two categories c1 and c2 with c1 æ c2 and c2 Ü c1Þ; formula
(2) does not in general fit. It is then possible to extend this non-linear ordering to a lattice-
order, where supremum sup{ci; cj} and infimum inf {ci; cj} exist for all pairs ðci; cjÞ. Using
th
an order preserving mapping f : {c1; : : : ; cr} ! R, the i component distance can be
defined by
diða; bÞ ¼ j f ðsup{ai; bi}Þ 2 f ðinf{ai; bi}Þj ð3Þ
For instance, consider the n-gram approach used for string variables (Efelky et al. 2002).
pffiPffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
The distance between two strings is defined as diða; bÞ ¼
;s j f aðsÞ 2 f bðsÞj, where
f aðsÞ and f bðsÞ are the number of occurrences of the substring s of length n in the two
strings a and b. Let us consider a small example, where n ¼ 3 and the strings HONEY and
pffiffiffi
MONEY are given. We obtain diðHONEY;MONEYÞ ¼ 2, since there are two non
common substrings of length 3: HON and MON. In the case of hierarchical variables, the
following distance function is suggested:
diða; bÞ ¼ min{f ðcjÞjcj , ai and cj , bi} ð4Þ
where for the order preserving mapping f holds f ðckÞ ¼ 0 if there is no cl with cl ,i ck.
Further techniques can be found in e.g., Porter and Winkler (1999). Hierarchical variables
may occur if some categorical variable was partially coarsened to different degrees in the
data set. For instance if in a business database the variable Branch of economic activity
(NACE code) is for some units specified on a 3-digit level and for other units specified on a
1- or 2-digit level (see Figure 2).
Here, the distance between hierarchical variables is preferable to the simple (0 2 1)-
distance in (1) being strongly separating, particularly if considerable deviations are
observed between the two data sets in respect of these variables.
For the planned adaption of different types of component distances, depending on the
types of variables vr, it is necessary to standardise the distances dr in order to avoid scaling
problems, e.g., by use of the max – min standardisation
drða; bÞ 2 minða;bÞ[A£Bdrða; bÞ
~
drða; bÞ :¼
maxða;bÞ[A£Bdrða; bÞ 2 minða;bÞ[A£Bdrða; bÞ
Lenz: Measuring the Disclosure Protection of Micro Aggregated Business Microdata 685
Fig. 2. Coarsening of the NACE code
For large data sets, it is recommended to partition the data into subsets, as described in
Subsection 3.2.
3.2. Blocking Variables
For a record pair to be a candidate for a possible assignment, it is necessary that the two
records coincide in their values of some specified variables. In the following these
variables are called blocking variables (see e.g., Jaro 1989), since they divide the
whole data into disjoint blocks.
The aim of blocking data is on the one hand to reduce the complexity of the subsequent
assignment procedure and the allocated main storage, on the other hand to reduce the
number of mismatches. Though the number of possible mismatches grows with the
number of wrongly classified records (that is, two records a and b which refer to the same
individual are possibly not members of the same block), mismatches have to be expected
especially in large blocks as there are many similar distances. Whether it will be possible
here to find a reasonable tradeoff depends on the quality of the variable used for blocking.
It is in general difficult to estimate the reporting error probability of some variable
intended for blocking. In the worst case, the corresponding blocks in both data sets are
disjoint, in the best case the chosen blocking variables are unique identifiers, such that
truly associated records belong to the same block. If possible, the data intruder will opt for
those variables which are known to have been left out of consideration within the
anonymisation process.
The treatment of specific variables like identifiers and blocking variables can also be
embedded into the calculation of distances, where identifier variables are handled like
nominal variables. Now let vi be a blocking variable. In the application in Section 7, the
appearing blocking variables are categorical ones, where the corresponding component
distances are defined by (1). Nevertheless, if a numerical variable v is intended for
blocking, it is strongly recommended to aggregate the range of v into intervals, such that
every value falls uniquely into some interval. That is, two values are distanced with zero if
and only if their intervals (categories) coincide.
From a theoretical point of view, the setting of blocking variables is a special case of
multidimensional clustering (see Schweigert 1999). Metrics often used in clustering
analysis like the general matrix metric
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
T
dðx; yÞ ¼ ðx 2 yÞ Cðx 2 yÞ
686 Journal of Official Statistics
where C is an arbitrary symmetric, positive definite matrix, are in most cases not of
practical relevance, since they involve an additional unacceptable computational amount
21
already for data files of medium size, particularly if C determines the inverse S of the
empirical covariance matrix.
Note that there are a number of alternative methods to preselect candidate pairs. A thorough
analysis of the methods can be found in Elfeki, Verykios, and Elmagarmid (2002).
A formalisation of the concept of allocating individual weights to all key variables is
given in the subsequent section.
4. Preference Functions and Matchings
Among others, the success of a record linkage algorithm depends on the choice of distance
measures and on the reliability of each variable. Therefore the decision maker rates the key
variables and prefers some of them to the others. This is done by use of so-called
preference functions.
þ k
Definition 1. Let L ¼ ðl1; : : : ; lkÞ [ ðR Þ be a k-tuple of positive real numbers. For a
ð1Þ ðsÞ
record r ¼ ðr ; : : : ; r Þ, where s is the number of all variables, let w.l.o.g. the entries
ð1Þ ðkÞ
r ; : : : ; r be the values of the key variables. We define a k-ary linear
k
preference function f L : R !R by
Xk
f Lðx1; : : : ; xkÞ ¼ lixi
i¼1
P P
k k21
Setting
i¼1li ¼ 1 and hereby lk ¼ 1 2 i¼1 li, we may reduce the set of parameters to
{l1; : : : ; lk21}. The permutation t, defined in such a way that
lt ð1Þ . lt ð2Þ . : : : . lt ðkÞ
can be understood as an individual ranking of variables by the decision maker. In the
theory of multicriteria optimisation, linear preference functions are used to turn
multiobjective optimisation problems into single-objective ones (Schweigert 1995).
Let M be a matching and M # A £ B its set of edges. We define componentwise
X
diðMÞ :¼ diða; bÞ; i ¼ 1; : : : ; k
ða;bÞ[M
and further
DðMÞ :¼ ðd1ðMÞ; : : : ; dkðMÞÞ
Definition 2. A maximum matchingM is called a preference matching if there is
0
a preference function f L such that f LðDðMÞÞ # f LðDðM ÞÞ holds for every maximum
0
matching M .
A single edge (a,b) can be regarded as a (non maximum) matching. Preference functions
involve for every ða; bÞ [ A £ B the distance
Xk
dða; bÞ :¼ f LðDða; bÞÞ ¼ lidiða; bÞ
i¼1
Lenz: Measuring the Disclosure Protection of Micro Aggregated Business Microdata 687
This expression can be regarded as a weighted sum of all component distances. Now we
are able to calculate the distances d(a,b) for a [ A and b [ B, split into component
distances associated with categorical or numerical variables.
k
X X X
~
dða; bÞ ¼ lidiða; bÞ ¼ t lidiða; bÞ þ lidiða; bÞ
i¼1 i[CV i[NV
where CV is the set of indices of the categorical variables and NV its complementary set of
~
indices of numerical variables. The parameter t ¼ li=li is an adaptive control parameter,
needed to balance the influence of categorical and numerical variables in order to achieve
a reasonable adaption. Key variables – besides blocking variables – which are involved in
distance calculations are tentatively called matching variables. Note that the
concept of blocking data can also be embedded into the calculation of distances as
mentioned in Subsection 3.2. The distances dða; bÞ can be split into their component
distances associated with blocking variables (BV) and matching variables (MV),
X X
dða; bÞ ¼ diða; bÞ þ lidiða; bÞ
i[BV i[MV
where identifier variables may be contained, for the sake of easy implementation, in the set
of matching variables, weighted with li ¼ 0. The weights of blocking variables are
allocated by li ¼ 1. If the records a and b coincide in their blocking variables, the first sum
P
is zero. Replacing li by li=
j[MV lj for each matching variable vi one obtains a convex
combination of the component distances, such that the second sum is a value less than or
equal to one. In other words, two records a and b are distanced by dða; bÞ # 1 if and only if
they are classified to the same block.
An alternative realisation of blocking data is to presort the whole data by blocking
variables and to read the data blockwise. Our experience has been, however, that
reading the data in and out block by block, a process usually not accounted for in
complexity analyses, is extremely time-consuming. Moreover, for large data sets the
summed-up distances dða; bÞ should be compared with some appropriate threshold value
c , 1 – determined a priori – to decide whether the overall distances are small enough to
classify the pairs ða; bÞ as true matches. (see also Fellegi and Sunter 1969)
5. Linear Assignment Problem
In a non-technical way, the concept of matching may be introduced as bringing together
pairwise information from two records, taken from different data sources, that are believed
to refer to the same individual. The records a and b are then said to be matched. Since in
general there is the possibility that the matching could be wrong, an attempt is made to
minimise the number of mismatches.
In the following let n ¼ jAj ¼ jBj ¼ m. Otherwise consider w.l.o.g. the case m , n.
Dually, the case n , m can be treated. We then define new objects bmþ1; : : : ; bn which
induce new pairs ðai; bjÞ for i ¼ 1; : : : ; n and j ¼ m þ 1; : : : ; n, at a distance
dðai; bjÞ :¼ ðmax d1ða; bÞ; max d2ða; bÞ; : : : ; max dkða; bÞÞ
ða;bÞ[A£B ða;bÞ[A£B ða;bÞ[A£B
688 Journal of Official Statistics
We obtain the multi-objective linear program described below:
X X
n n
d1ðai; bjÞxij
i¼1 j¼1
X X
n n
d2ðai; bjÞxij
i¼1 j¼1
Minimise Þ ðMOLPÞ
.
.
.
X X
n n
dkðai; bjÞxij
i¼1 j¼1
subject to xij [ {0; 1} for i; j ¼ 1; : : : ; n
Xn
xij ¼ 1 for i ¼ 1; : : : ; n and
j¼1
Xn
xij ¼ 1 for j ¼ 1; : : : ; n
i¼1
The constraints ensure that every ai is connected with exactly one bj and vice versa. That
is, xij ¼ 1 if and only if ai is matched with bj.
P
k
As described in Section 4, defining dða; bÞ :¼
i¼1lidiða; bÞ and abbreviated
dij :¼ dðai; bjÞ, the problem of finding an optimum matching is turned into a single
objective assignment problem (AP) using linear preference functions. The main idea is to
combine all objectives into one single value, as it is typically done in a linear program
formulation. Note that linear approaches in general lead to a considerable loss of useful
information. When the summed-up distances dij are calculated, the question of choosing
weights is often glossed over, but in fact it is extremely critical. In Schweigert (1995) it is
shown that under certain assumptions it suffices for the decision maker to define a range
for the weights. However, there arises the following single-objective assignment problem:
Xn Xn
Minimise dijxij ðAPÞ
i¼1 j¼1
subject to xij [ {0; 1} for i; j ¼ 1; : : : ; n
Xn
xij ¼ 1 for i ¼ 1; : : : ; n and
j¼1
Xn
xij ¼ 1 for j ¼ 1; : : : ; n
i¼1
This assignment problem can be formulated graph-theoretically as follows: Find a preference
matching on a vector-weighted bipartite graph. In other words, we have to look for
P
n
apermutationp of {1; : : : ; n} which minimises the sum
i¼1di;pði Þ. That is, in order to solve
(AP), we might produce all n! matchings of G and select one of minimum weight.
Unfortunately, this algorithm will certainly not be efficient and thus does not justify the
transition from problem (MOLP) to (AP). Though there are classical procedures like the
Lenz: Measuring the Disclosure Protection of Micro Aggregated Business Microdata 689
well-known simplex method (see e.g., Papadimitriou and Steiglitz 1998), which – despite
non-polynomialworst-caserun-time – turnedouttobeeffectiveinpractice,problemsappeared
already while studying data with moderate block sizes. Considering the coefficients connected
with the system of linear equations of the restrictions in (AP) the following matrix is generated
!
J1 J2 : : : Jn
A ¼
In In : : : In
th
where Ji defines a matrix of dimension n £ n, whose i row is the vector ð11: : :1Þ and whose
remaining entries are zero. In ¼ diagð1; 1; : : : ; 1Þ defines the identity matrix of dimension
2
n £ n. The resulting coefficient matrix A, possessing 2n rows and n columns, was in several
instances responsible for exceeding the working memory.
A way out is to use the Hungarian Method. We give a short description of this
method, originally proposed for maximum weight perfect matchings (see Kuhn 1955 and
Munkres 1957) and slightly modified for our purposes, in order to find a minimum weight
perfect matching or a preference matching, respectively. Let us consider a complete,
weighted bipartite graph G ¼ ðV; EÞ. A feasible vertex labeling l is a mapping
from the set V into the real numbers, where
lðaÞ þ lðbÞ # dða; bÞ
The number lðvÞ is then called label of v. The equality subgraph Gl is a subgraph
of G which includes all vertices of G but only those edges ða; bÞ fulfilling
lðaÞ þ lðbÞ ¼ dða; bÞ
A connection between equality subgraphs and matchings of minimum weight is provided
by the following theorem.
Theorem. Let l be a feasible vertex labeling of G. If the equality subgraph Gl possesses a
perfect matching M, then M is a minimum weight perfect matching of G.
0
Proof: Let M be a perfect matching of Gl andM be any perfect matching of G. Then it
holds that
X X
0 0
dðM Þ : ¼ dða; bÞ $ lðvÞ ðsinceM saturates all verticesÞ
ða;bÞ[M0 v[VðGÞ
X
¼ dða; bÞ ðby definition ofMÞ
ða;bÞ[M
¼: dðMÞ
Hence, M is a minimum weight perfect matching of GA.
When applying the algorithm, we use two vectors of labels, ðlða1Þ; : : : ; lðanÞÞ and
ðlðb1Þ; : : : ; lðbnÞÞ, to select admissible edges. Initially, we set
lðaiÞ ¼ 0 for i ¼ 1; : : :; n
and lðbjÞ ¼ min dðai; bjÞ for j ¼ 1; : : :; n
1#i#n
690 Journal of Official Statistics
Using the concept of so-called augmenting paths, we find a matching M of Gl which
saturates as many vertices as possible. If M is perfect, according to the above theoremM
is a minimum weight matching of G and the algorithm stops. M is then uniquely
determined up to equivalence. Else, if M does not determine a perfect matching, we relax
the values for some l(a) and l(b) so that new edges will be admissible.
A competing algorithm is the auction algorithm, introduced for the assignment problem
in Bertsekas (1979) and later extended to general transportation problems in Bertsekas and
Castanon (1989).
6. Matching Algorithm
In this section we suggest heuristic approaches to the single-objective linear assignment
problem. Though the greedy heuristics introduced below do not guarantee optimality, the
approaches are also discussed, since their undoubted advantage is that they work in
reasonable time, more precisely in square time complexity according to the number of
local units. On the other hand, even when the global solution is not reached, the reached
suboptimal solution is in our case a very good solution.
6.1. Greedy Heuristics
Often greedy algorithms are preferred on account of easy implementation and quick run-
time (e.g., see T. H. Cormen et al. 1990). In fact, the complexity of the procedures below is
of order OðnmÞ, where n and m are the numbers of records in A and B, respectively,
whereas the Hungarian method sketched in Section 5 has complexity of order
3
Oðmax{n; m} Þ and is therefore not practicable for data sets of large size. At this stage the
distances belonging to ða; bÞ [ A £ B can be taken for granted.
Procedure I: begin {PROC I}
M :¼ Y
i :¼ 1
While ði # n and B – YÞ do
0
b :¼ argminb[Bdðai; bÞ
0
M :¼ M< {ðai; b Þ}
0
B :¼ Bn {b }
i :¼ i þ 1
end {PROC I}
The procedure’s output is an assignmentM of A to B. Obviously, the output depends on the
enumeration of a1; : : : ; an and might be far from optimum. Let w.l.o.g. a1; : : : ; ar be
assigned to bpð1Þ; : : : ; bpðrÞ. In step r þ 1 the target object arþ1 is associated with a record b
of minimum distance to arþ1. Record b is one of the remaining m2r records in B, which
