Measuring Progress of Community Health Information Infrastructures

As dozens of communities move forward with their efforts to implement health information infrastructure (HII), there is a widespread and urgent need to be able to determine how these initiatives are progressing. This is necessary so that community stakeholders, outside observers, and policymakers can monitor and understand both what has been accomplished and those tasks that remain. Simply stating that a community is exchanging health information does not differentiate between a few pilot transactions and the availability of complete electronic information for all citizens.

What are the proposed HII measures?

To address this issue, my colleague Steven Labkoff, MD (of Pfizer Global Pharmaceuticals) worked with me to develop and test a proposed set of measures that can help to objectively assess how far a community has progressed on the path to a complete HII. This article summarizes the basic concepts we developed. A complete report of this effort has been accepted for publication and will appear in an upcoming issue of the Journal of Biomedical Informatics (proofs are now available with subscription on Science Direct).

The central idea is to examine the attributes of a fully operational, “complete” HII, and then define simple measures that would give an indication of how far along the path to completion a given community has progressed for each. We defined four such attributes:

1. INFORMATION: All the medical information about every person should be available when and where needed
2. USERS: All the physicians and all the consumers should be using the system
3. USES: The information should be used for the full spectrum of appropriate purposes: 1) providing health care; 2) public health; 3) medical research; 4) quality improvement; and 5) health care operations
4. FINANCING: The operation of the system should be financially sustainable, funded exclusively with ongoing operational income

We subdivided the first attribute (what information is available) into eight categories:

1) inpatient (hospital)

2) outpatient (ambulatory)

3) long-term care

4) home health/personal health record (PHR)

5) laboratory results

6) outpatient medications

7) imaging

8) insurance claims

Each attribute or sub-attribute was then scored on a 0-5 scale, where 0 represented no activity, 1 was 1-20% complete, 2 was 21-40%, etc. For the first attribute, separate scores were generated for all eight types of information. For the second attribute (who is using the system), separate scores were generated for physician and consumer use. The third attribute (uses of the information) was scored on a binary basis: if any of the information was being used for a specific purpose, a 1 was given — 0 indicated no activity. The last attribute (financial sustainability) was again scored on a 0-5 scale based on the percentage of funding derived from ongoing operational sources.

The scores for each attribute were then normalized so each counted for 25% of the total, and the overall score was expressed as a percentage of a perfect score, which would be 100%.

What are the results of using the measures?

When we applied these measures to four of the most advanced HII communities in the nation, the scores were in the 60-80% range indicating that additional work was needed to achieve a “complete” HII. The two specific items typically remaining incomplete are not surprising:

1) outpatient information was usually absent since most physicians do not have electronic health record systems (EHRs). Therefore, this information is not electronic and cannot easily be exchanged2) few consumers are using these systems — most communities do not yet provide consumers with access to their own information.

Of course, this rating system for assessing progress is very crude, and undoubtedly represents only the first step in the evolution of metrics for community HIIs. Implicit in this measurement scheme is agreement about the four key attributes: information, users, uses, and financial sustainability. If other attributes are deemed important, they must also be included.

In addition, a number of critically important factors are ignored. Privacy and confidentiality protection was not included because it was assumed that all HIIs would be required to deal with these issues before beginning operations. User satisfaction with the system, admittedly a critical characteristic to evaluate, was also omitted — primarily because it is somewhat time-consuming to measure.

Another item that is not represented in the measures is the degree to which the data being exchanged has been encoded. While it is clear that fully encoded, standardized data is much better (and essential for decision support), it was not clear how to appropriately assess this. Also, it was felt that at this early stage in the development of HII systems, it may not be necessary to do so. However, as HIIs mature, it seems likely that it will be helpful to differentiate communities that score 100% on this proposed assessment scheme by the extent to which the information is encoded.

As more and more communities begin to exchange health information, these measures should be very helpful in assessing their progress. We certainly hope this simple assessment scheme can contribute to a better understanding about how far we’ve progressed on the road to an HII both in individual communities and the nation as a whole.

As always, your comments and suggestions are welcome. What do you think?

Next time: Governance of community health information infrastructure systems

This entry was posted
on Wednesday, March 8th, 2006 at 2:24 pm and is filed under .
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.

2 Responses to “Measuring Progress of Community Health Information Infrastructures”

This is good data and perhaps can be worked into a scorecard for the NHII! I’m not surprised by the two results, either. But hold on, we are getting ready to see some changes in #1 and I don’t believe #2 is too far off!