De-identification: software and test data

The deid software package includes
code and dictionaries for automated location and removal of protected
health information (PHI) in free text from medical records.
The gold-standard corpus is a collection of
deidentified nursing notes that have been reidentified with realistic
surrogate data, for use in developing and evaluating software such
as deid.

For additional discussion of the methods used by the software
described here, see

Background

In the USA, the HIPAA Privacy Rule restricts exchange of medical data
containing protected health information (PHI), defined as any
information that might be used to identify the individual(s) from whom
the data were collected. Data known to contain PHI can be shared for
research purposes only under tightly controlled circumstances, typically
involving data use agreements under which the researchers involved must
obtain IRB or equivalent approvals for use of the data.

By contrast, medical data that do not contain PHI are exempt from the
restrictions of the HIPAA Privacy Rule and may be shared freely. The
data available on PhysioNet fall into this category.

Many of the research data sets currently being collected by PhysioNet are
accompanied by PHI, and the process of removing this PHI ("de-identification"
in the language of HIPAA, or "anonymization") is tedious and error-prone.
For many research projects, the cost of de-identification is a significant
barrier to data sharing.

The MIMIC II project has
invested several years' effort to develop and evaluate software that
is capable of removing PHI from text and, where appropriate, replacing
it by realistic surrogate PHI. (For example, names are replaced by
fictitious names, medical record numbers by fictitious medical record
numbers, dates by fictitious dates, geographic locations by other
geographic locations, etc.) The MIMIC II project has contributed to
PhysioNet both its deidentification software, which may be useful to
other researchers needing to remove PHI from their own data, and a
large corpus of text drawn from ICU nursing notes that were gathered
simultaneously with the signals, trends, laboratory reports, discharge
summaries and other data in the MIMIC
II clinical
and waveform databases.

Software

As is the case with all software available on PhysioNet, the
de-identification software (deid) is provided in source form so that
its workings can be studied, customized, and improved. This software
is free software. You can redistribute it and/or modify it under the
terms of the GNU General Public License as
published by the Free Software Foundation, either version 2 of the
License, or (at your option) any later version.

The authors of the deid software package are Margaret M Douglass,
Li-wei H Lehman, William J Long, and Ishna Neamatullah. An experimental
prototype by Jason M Levine provided useful guidance in the design of
deid.

Prerequisite

The deid package requires Perl (version 5 or later), which is
freely available for all popular platforms
here if you don't have it
already.

The current version of the deid software package was
developed and tested using perl 5.8.8 and perl 5.10 on GNU/Linux
(Fedora Core 10). It has also been tested on Fedora 6, 7 and 8,
Ubuntu 8.04, MacOS X 10.4.5, and on MS-Windows XP.

Downloading and installing deid

The deid software package can be downloaded as a single
file, deid-1.1.tar.gz.
(Instructions on unpacking tar.gz files
are here.) The contents
of deid-1.1.tar.gz unpack into a directory named
deid-1.1, which contains the files
listed below.

The deid software package is command-line
(text mode) software that must be run
in a terminal emulator (console) window.

Previous versions of the deid package are available for
reference purposes here.

On a typical current PC, the process will run to completion in
approximately 10 to 20 minutes. The three output files will be
written into the current directory; they should be identical to
the files of the same names that are included in the GSoutput
directory.

The program runStat.pl can be used to derive performance
statistics by comparing the id.phi file generated
by deid.pl with the reference id.deid file
provided with the deid software package. To do this, type
the command

To use the gold standard corpus to test the performance of another
deidentifier, arrange for that deidentifier to produce a file in the
format of id.phi, then use runStat.pl as above.

Customizing deid

Examine the configuration file, deid.config, to see how
to modify the behavior of deid.pl. For additional information,
see the user manual.

In order to customize this software to de-identify free text in other
medical records, you may replace our filter modules with your
data-specific filters. Additionally, at a minimum, you will have to
replace the six files that contain a priori information
(see below).

Depending on the text you wish to de-identify, you may wish to re-classify names
as ambiguous or not. For example, "Mae" is an unambiguous name in most
contexts, but in nursing and discharge notes, it may be an acronym (meaning
“moving all extremities”) and it is therefore ambiguous in those contexts.

Contents of the deid package

See above for information about downloading
the entire deid package in a single archive file. The
individual files in the package can also be viewed or downloaded by
following the links below.

[The six files in this group contain the surrogate PHI from
the gold standard corpus, not the original PHI from the files used to
create it! You will need to create your own versions of these six
files, using these as models, if you wish to use deid on your
own data.]

Except for functions related to calculating performance statistics (in
stats.pm), the deidentification code is contained in a perl
script (deid.pl). Its configuration file, deid.config,
can be used to set parameters of the deidentification process at run time.
Associated word and phrase lists are in lists/ (containing known
and possible PHI) and dict/ (containing probable non-PHI).

If the date shift filter is on, deid replaces all dates in
its input with surrogate dates. The shift.txt file contains
a randomly assigned date shift (between 1000 and 3000 days) for each
patient in the gold standard corpus; surrogate dates are generated by
adding the specified number of days to the dates in the input file.
The method used to generate the date shifts in shift.txt
differs from that used in PhysioBank databases.

The file id-phi.phrase is provided as a convenient index
to the PHI in the gold standard corpus; it is not used by the deid
software. Each line of this file contains 6 fields: PID,
Record_Number, PHI_Start_Location, PHI_End_Location, PHI_Type, and
PHI_Text.

The file id.deid also contains PHI locations in the gold standard
corpus, and it is used as a reference for calculating performance statistics
by the functions in stats.pm. It contains two types of lines. The
first type is of the form

Evaluating software for de-identification turns out to be quite difficult.
As in many of the projects contributing to PhysioNet, a reference database
is highly useful. Ideally the developer of de-identification software needs
an appropriate corpus of text in which all of the PHI has been labelled, so
that the software's PHI detection performance can be assessed quickly and
quantitatively. To compare different approaches to PHI removal objectively,
we need a standard corpus. And here's the dilemma: we can't share such a
corpus if it has any PHI in it!

The deid software package was developed and tested using a
gold standard corpus of 2,434 nursing notes that have been thoroughly
deidentified by a multi-pass process that included meticulous reviews
by three or more experts working independently, as well as by a
variety of automated methods. All detected instances of PHI in these
nursing notes have been replaced by realistic surrogate data in the
gold standard corpus. Although the deid software, as noted
above, may be redistributed under the terms of the GPL, the gold
standard corpus, because of the very small possibility that it may
contain one or more instances of undetected PHI, is currently
available only under terms that do not permit it to be redistributed.
This corpus is available only under the following conditions:

You agree not to attempt to identify any of the individual
subjects of the file.

If you find information within the file that you believe might
permit identification of any of the individual subjects of the file,
you agree to report this promptly by email
to webmaster@physionet.org,
citing the specific information in question so that it can be
investigated and removed if necessary.

You agree to exercise all reasonable and prudent care to avoid
disclosure of the identity of any of the individual subjects of the
file in any publication or other communication of your analysis of the
file.

If you agree to all of these terms and conditions, access to this
file will be granted to you as an individual. You may not share it
with anyone else. Your colleagues can obtain access to this file as
individuals via the same procedure you are following.

To download the corpus, you must be logged into a PhysioNetWorks account (you can create a free account in a few minutes here if you don't have one already). Go to the Deidentified Medical Text project and apply for access.