eHealth Observatory

Metrics

Introduction

The following are some of the design components and metrics to evaluate HIS adoption/use.

Exhibits 1- 7

Exhibit 1

Exhibit 1.

Four study cohorts can be defined according to the adopter/user and HIS type.

Legend: *means only EMR or eDrug is adopted/used but not both; +/- means either EMR or eDrug is present for basic-adopters/users

Cohort

EMR

eDrug

EHR

A: Non-adopter/user

B: Basic-adopter/user*

+/-

+/-

C: Intermediate-adopter/user

√

√

D: Advanced-adopter/user

√

√

√

Exhibit 2

Exhibit 2.

Each cohort in 3a can be further divided into three care settings each with different HIS types

Setting

EMR

eDrug

EHR

Physician Office

√

√

√

Community Pharmacy

√

√

Acute/Tertiary/continuing care

√

√

Exhibit 3

Exhibit 3.

Each cohort in 3a can be further divided into three provider types each involved with different HIS types

Provider

EMR

eDrug

EHR

Physician

√

√

√

Pharmasist

√

√

Authorized health professional

√

√

Exhibit 4

Exhibit 4.

Different study design options available based on composition of the cohorts and time periods being examined. Legend: N is sample size; T is time period; subscripts 0 to 5 denote the sample sizes and time periods for each cohort; ∑ refers to the column, row and grand total sample sizes. For example, Cohort-A at Time-0 can be defined to include all three types of providers across all three settings who have not adopted/used EMR, eDrug and EHR at the baseline period.

Examples

Time-0

Time-1

Time-2

Time-3

Time-4

Total

Cohort-A

A(N1T0)

A(N2T1)

A(N3T2)

A(N4T3)

A(N5T4)

A(N1-5)

Cohort-B

B(N1T0)

B(N2T1)

B(N3T2)

B(N4T3)

B(N5T4)

B(N1-5)

Cohort-C

C(N1T0)

C(N2T1)

C(N3T2)

C(N4T3)

C(N5T4)

C(N1-5)

Cohort-D

D(N1T0)

D(N2T1)

D(N3T2)

D(N4T3)

D(N5T4)

D(N1-5)

Total

∑(N1T0)

∑(N2T1)

∑(N3T2)

∑(N4T3)

∑(N3T4)

∑(N1-5)

Exhibit 5

Exhibit 5.

Five kinds of studies to examine the information/system/service quality and use/satisfaction dimensions in the BE Framework. Legend: Cohorts-A, B, C, D are non-adopters/users with no HIS, basic-adopters/users with eDrug or EMR, intermediate-adopters/users with eDrug and EMR, and advanced-adopters/users with eDrug, EMR and EHR, respectively. Time-0 is the baseline period, Time-X is the next period, Time-Y the period after Time-X, and Time-Z after Time-Y, etc. Cohort-A with no HIS is excluded since these studies all require specific types of HIS being adopted/used.

Studies on BE Quality/use Dimensions

Cohort-A

Cohort-B

Cohort-C

Cohort-D

(1) Comparison: documented features vs. best practices

Time-0 to Time-X/Z

Time-X to Time-Y/Z

Time-X to Time-Y/Z

(2) Usability: ease of use, functionality and usefulness

Time-0 to Time-X/Z

Time-X to Time-Y/Z

Time-X to Time-Y/Z

(3) Qualitative case: survey, interview and observation on on percieved/actual use

Exhibit 6

Exhibit 6.

Five kinds of studies to evaluate the care quality and productivity dimensions in the BE Framework. Legend: Cohorts-A, B, C, D refer to non-adopters/users with no HIS, basic-adopters/users with eDrug or EMR, intermediate-adopters/users with eDrug and EMR, and advanced-adopters/users with eDrug, EMR and EHR, respectively. Time-0 is the baseline period, Time-X is the next period, Time-Y the period after Time-X, and Time-Z after Time-Y, etc.

Exhibit 7

Exhibit 7.

Two examples of time series delayed controlled matching cohorts. First is the two matching Pharmacy cohorts where baseline measures are taken at Time-0 with the original PharmaNet. Then Pharmacy1 deploys the new eDrug in Time-1 while Pharmacy2 continues with PharmaNet. Then Pharmacy2 deploys new eDrug in Time-2 while Pharmacy1 continues to use eDrug. Then Pharmacy1 deploys a new EMR while Pharmacy2 continues to use eDrug. Another example is the matching Physician cohorts for deployment of EMR, eDrug and EHR.

Examples

Time-0

Time-1

Time-2

Time-3

Time-4

Pharmacy 1

Original PharmaNet

New eDrug

eDrug

eDrug + New EHR

eDrug + EHR

Pharmacy 2

Original PharmaNet

Original PharmaNet

New eDrug

eDrug

eDrug + New EHR

Physician 1

Exsisting EMR

EMR + New eDrug

EMR + eDrug + New EHR

EMR + eDrug + EHR

EMR + eDrug + EHR

Physician 2

No HIS

New EMR

EMR + New eDrug

EMR + eDrug + New EHR

EMR + eDrug + EHR

Reactive Analysis

Reactive Analysis of an Information System, from the IT Interaction Model by Silver et al. [xx]

Was the information system's design objective to IMPROVE the organization incrementally or to TRANSFORM it?

What are the information system's FEATURES? What does it do?

How does the information system FIT the firm's EXTERNAL ENVIRONMENT?

How does the system FIT the firm's STRATEGY?

How does the system FIT the firm's BUSINESS PROCESSES?

How does the system FIT the organizational STRUCTURE and CULTURE?

Can the organization's existing IT INFRASTRUCTURE support the system? Does the information system leverage the infrastructure? Does it extend it?

How and how effectively was the system IMPLEMENTED?

Who USES the system and how do they USE it? As intended?

What are the CONSEQUENCES of the system for PERFORMANCE, PEOPLE, and FUTURE FLEXIBILITY? Did the system accomplish its objectives?

Guiding Questions

Guiding Questions from Kotter's Organizational Change Model [xx]

Has the sense of urgency for change been established within the organization?

Has a guiding coalition been created within the organization?

Has the vision and strategy been developed within the organization?

Has the vision and strategy been communicated throughout the organization?

What is the mechanism to empower broad-based action for change throughout the organization?

What are the short-term wins for the organization that can be generated from this change?

What are the efforts to consolidate gains and produce more change within the organization?

How are the new approaches being anchored within the organizational culture?