It seems that Hollywood is starting to catch on to what many viewers already know: Some of the best shows on TV are not on a network; they're streaming over your internet connection. Shows such as House of Cards and Orange Is the New Black from Netflix have been capturing people's imagination for years, although major awards have often eluded them. But in the first months of 2015, one announcement after another pointed toward a sort of streaming content renaissance.

Advances in networking and database technology have brought vast amounts of data together, and as search and querying technology improves, these vast stores of data become increasingly meaningful to even the casual user. In the right hands, such networked data and content can be invaluable—the doctor who needs vital patient records, the security analyst who wants to glean some intelligence from financial records. But for all its potential good use, the same data has great potential for misuse—either inadvertent or intentional. Mishaps have already happened and, while policies are in places (and new ones are soon to be implemented), the risk remains. Perhaps the real question then is to ask whether technology to preserve privacy can advance as quickly as the technology that seems to be putting privacy at risk.

In late December 2002, the U.S. Department of Defense reported that its efforts to computerize the medical records of military personnel were set back when hard drives containing the records of a half-million personnel were stolen. The records included names, social security numbers, and medical claims histories. According to the Associated Press, the Defense Department had seen the new computerized system "as a potential ‘data gold mine' for military physicians and other healthcare professionals that will provide quick and easy access to military patient records worldwide."

While this is perhaps the most spectacular recent privacy breach, it is not the only one. According to news accounts, patient record information has been compromised at a major pharmaceutical chain, a health insurance company, and an online retailer of healthcare products, to name a few places. In each of these cases, the compromise has been inadvertent: in one case, information was emailed to the wrong parties and in another case-sensitive information was accidentally posted to a public Web site. But when these accidental disclosures are considered in light of the Defense Department theft and some well-publicized security breaches at ecommerce companies, the concern begins to grow.

Indeed, many would argue that, when it comes to medical records, any compromise is unacceptable and that every reasonable effort should be made to safeguard such data. To that end, the federal government is mandating the enforcement of new patient privacy rules under the Health Insurance Portability and Accountability Act of 1996 (HIPAA). HIPAA is a broad law that called upon Congress to delineate what rights patients have to control their own medical information, and what procedures and mechanisms would be followed for appropriate sharing of that information. The result is a broad set of regulations to be followed by healthcare providers, insurers, and related organizations such as medical researchers—anyone who handles patient information.

Privacy is FundamentalThe assumption behind the protection of medical record information is that privacy is a fundamental right. In announcing the HIPAA regulations, the U.S. Department of Health and Human Services recognized that the new regulations would come at significant cost to the healthcare industry, but pointed out, "it is important not to lose sight of the inherent meaning of privacy: it speaks to our individual and collective freedom." While this may seem like lofty language, they cite the same basis many privacy organizations and advocates do—the Fourth Amendment guarantee that "the right of the people to be secure in their persons, houses, papers and effects, against unreasonable searches and seizures, shall not be violated."

To this end, HIPAA and regulations seek to control how patient information is collected, safeguarded, and used over time. The overarching requirement is in some ways obvious; only clinicians with a need to know—and to whom you have granted access—should have access to your medical information. But the actual implementation is complex, as more information is digitized, as more systems are interconnected, and as increasingly powerful tools for querying become available.

But the real tension between privacy and usefulness stems from the basic requirement for automating patient information in the first place—to give clinicians ready access to the information they need to make on-the-spot, critical decisions. "It's a balance between confidentiality and ease of use," notes Dr. John Halamka, who as both a practicing physician and CIO for a Boston-area hospital group has a comprehensive view of the problem. In describing the tools they have developed at CareGroup Health System, Halamka talked about "including knowledge in the workflow" for an application such as order entry. Halamka offered the example of a doctor who is prescribing a hypertension drug for a diabetic, where the doctor would ideally have the patient's latest lab results as well as recent and relevant research about the medication "in the context of taking the action."

Again, while the requirement is in some ways obvious, the implementation is likely complex. To begin with, doctors operate in an information-saturated world. Primary medical research alone is a deluge of information. Halamka points out that if doctors took time out "to read eight research articles a night, they would be 800 years behind after one year." To solve that problem, Halamka's technology team at CareGroup gives clinicians access to databases such as Uptodate.com, where experts in the field read, abstract, and summarize the world's literature.

Moreover, even an individual patient's record may be lengthy and complex, and, depending on the action being taken at a given time, the clinician likely needs selected information rather than every detail about that patient. Halamka notes that the same doctor prescribing a hypertension drug would indeed want recent lab results, but would likely not need to read a summary from a recent psychological visit.