Transcription

1 Theme: Building health information systems Transforming data into information and knowledge: Examining the quality of Annual Reports in Pacific Island Countries and Territories Nicola Hodge Theme 1: Building health information systems health system information for decision-makers Working Paper Series Number 38 October 2013 DISSERTATION Strengthening health systems in Asia and the Pacific through better evidence and practice For the PDF version of this paper and other related documents, visit

2 About this series The Health Information Systems Knowledge Hub s Working Paper Series is the principal means to disseminate the knowledge products developed by the hub as easily accessible resources that collectively form a lasting repository of the research findings and knowledge generated by the hub s activities. Working papers are intended to stimulate debate and promote the adoption of best practice for health information systems in the region. The series focuses on a range of knowledge gaps, including new tools, methods and approaches, and raises and debates fundamental issues around the orientation, purpose and functioning of health information systems. Generally, working papers contain more detailed information than a journal article, are written in lessacademic language, and are intended to inform health system dialogue in and between countries and a range of development partners. Acknowledgments Many individuals with expert knowledge of health information systems in the Pacific have generously assisted in the preparation of this document. The Health Information Systems Knowledge Hub (School of Population Health, The University of Queensland) is grateful to Philip Davies, Audrey Aumua, Maxine Whittaker and Karen Carter for their valued contributions. Author details Nicola Hodge, Health Information Systems Knowledge Hub, School of Population Health, The University of Queensland Many working papers have accompanying products such as summaries, key points and action guides. The full range of documents, as well as other resources and tools, is available on the Health Information Systems Knowledge Hub website at tools. The opinions or conclusions expressed in the Working Paper Series are those of the authors and do not necessarily reflect the views of institutions or governments. The University of Queensland 2013 Published by the Health Information Systems Knowledge Hub, School of Population Health The University of Queensland Public Health Building, Herston Rd, Herston Qld 4006, Australia Please contact us for additional copies of this publication, or send us feedback: Tel: Fax: Design by Biotext, Canberra, Australia

3 Transforming data into information and knowledge: Examining the quality of Annual Reports in Pacific Island Countries and Territories Nicola Hodge This report is submitted in partial requirements for the award of Master of International Public Health at the University of Queensland December 2010

4 Abstract This dissertation is, in essence, a story of data: why we produce it; how we manage it; what we deem as worthy and appropriate ; who we share it with and those we exclude; what we do with it; and how it affects our lives Annual Reports, the focus of this research, provide a wealth of raw data. However they are often comprised of pages of complex tables, with little interpretation or descriptive analysis provided, thus limiting their usefulness in monitoring and evaluating health patterns. Despite growing recognition of the vital role health information systems (HIS) play in informing health care decisions, the area remains severely under-researched and under-resourced, with few systematic attempts at improving the quality of HIS as a whole. While sound statistics are becoming increasingly important for monitoring trends in population health, evaluating the success of interventions and managing health care services effectively; many systems still struggle with the task of transforming data into information and knowledge that is useful for action. The primary research question is concerned with what Annual Reports from Pacific Island Countries and Territories (PICTs) can tell us about the quality of data being produced from their HIS. Five dimensions of quality were selected for assessment (comparability, disaggregation, interpretability, presentation and timeliness), and methods of measurement developed accordingly. Secondary research questions were concerned with what the reports looked like, and what the data contained in them could tell us. Findings from this research are two-fold. Firstly, while many differences between reports were found, there are also striking similarities, most notably; Annual Reports from the Pacific present us with countries whose Page 2

5 health information systems are data rich but information poor. Reports are often excessively long, with a wide range of information on the entire health system and include pages of dense tables with little critical analysis on implications of what the data is showing. Due to the important formative role played by legislated reporting requirements, Annual Reports have become large compendiums of individual facts ; only presenting a snapshot of activities from the previous year with little attempt at linking the flow of information from objectives to outcomes, or determinants of health to health status, for example. The second main finding is related to the concept of quality. Overall, this research has found that the quality of data produced from HIS, as presented in Annual Reports, is poor. It is poor due to the limited success in each of the five quality dimensions assessed, especially in relation to comparability, which scored some of the worst scores and yet plays a crucial role in the potential use and usefulness of reports. The quality is also poor due to issues within HIS themselves though, and issues related to the production of Annual Reports, namely, the clarification of report purpose. In light of these findings, four recommendations are proposed: (1) the development of sub-reports; (2) the development of data quality assessment tools; (3) developing a regional reporting template; and (4) the development of a minimum data set. The wealth of data contained within Annual Reports deserve wider dissemination and use, and with a few improvements, could play a crucial role in evidence-based decision making and the monitoring and evaluation of health systems and health outcomes. Key words: Annual Report; Data; Health Information System(s); HIS; Information; Pacific Island Countries and Territories ( the Pacific ); Quality Page 3

6 Statement of originality This dissertation is composed of my original work, and contains no material previously published or written by another person except where due reference has been made in the text. I have clearly stated the contribution by others to jointly-authored works that I have included in my dissertation. I have clearly stated the contribution of others to my dissertation as a whole, including statistical assistance, survey design, data analysis, significant technical procedures, professional editorial advice, and any other original research work used or reported in my dissertation. The content of my dissertation is the result of work I have carried out since the commencement of my degree and does not include a substantial part of work that has been submitted to qualify for the award of any other degree or diploma in any university or other tertiary institution. I have clearly stated which parts of my dissertation, if any, have been submitted to qualify for another award. I acknowledge that an electronic copy of my dissertation must be lodged with the University Library and, subject to the General Award Rules of The University of Queensland, immediately made available for research and study in accordance with the Copyright Act I acknowledge that copyright of all material contained in my dissertation resides with the copyright holder(s) of that material. Nicola Hodge Page 4

7 Acknowledgements First and foremost, I want to thank the Health Information Systems Knowledge Hub for providing me with the opportunity to undertake this dissertation and also for their encouragement to both begin and finish this; the culmination of almost 12 months of work. I would also like to thank my supervisor, Philip Davies, for his continued support throughout this process; from providing me with creative ideas and alternative approaches in addressing my research questions; to assisting me with the writing and structure of the finished product. Your ideas challenged and inspired me, and I thank you for the time you have taken in guiding me through this process. I would also like to thank Audrey Aumua, who became my unofficialofficial other supervisor your comments and guidance made a significant impact on my approach to telling the story of Annual Reports in the Pacific. Also thank you to everyone at the University of Queensland who provided me with answers when I asked them for help and guidance when I was feeling a bit lost. Page 5

17 1 Introduction The following chapter provides an overview and introduction to this dissertation, including a description of where the research is located (Pacific Island Countries and Territories) and what the research is broadly concerned with: health systems, health information systems (HIS) and data. It is hoped that in reading this chapter, the reader will gain an appreciation of the current state of knowledge on HIS in the Pacific Region, and some of the common issues and challenges faced in terms of data quality. 1.1 Background Health information systems (HIS) are essential to the effective functioning of health systems worldwide. Within HIS, various types of data are obtained at different levels to be used by numerous people for many reasons. Providers use data for patient care; managers for enhancing efficiency and effectiveness; planners for decision-making; and finally policymakers for prioritisation and resource allocation. Put simply, sound statistics are a key component of quality evidence (AbouZhar & Commar 2008). They play a crucial role in evidence-based decision-making: the process of gathering, critically appraising and using high quality research evidence to inform policymaking and professional practice (AbouZahr et al 2007). However, rather than being provided with meaningful and useful information for decision-making, policymakers are often presented with competing, confusing, misleading or missing data (ibid). Statistical systems needed to deliver sound data are often weak, especially in developing countries where global health investments are commonly directed (Duran-Arenas et al 1998; AbouZahr et al 2007). Much of the data collected in HIS lack effectiveness due to the poor Page 15

18 quality and quantity of statistics used for monitoring, and the lack of incentives and capacity to collect, share, analyse and interpret health statistics (Shibuya 2008). While reliable and comprehensive data on disease levels, patterns, and trends in populations are required to monitor global and local epidemics and to assess the effectiveness of public health interventions, little is reliably known on the causes of morbidity and mortality in many developing countries (Boerma et al 2001; AbouZahr et al 2007). Despite this, many health agencies are actually producing increasing amounts of data: numerous HIS are not lacking information, but are overwhelmed by data. These data-rich but information-poor countries are not necessarily providing comprehensive information to users though, as data needs to be integrated with other information; presented and assessed in terms of past trends, current policy and fiscal considerations; and evaluated in terms of issues confronting the health system for it to be useful. Overall, it must be remembered that statistics alone have no value: in order for them to mean anything, they must be analysed and transformed into meaningful information that is then used. This gap in the availability and use of data is a cause for concern as the absence of health statistics has been described as a symptom and cause of underdevelopment, and the absence of good data also leaves room for policy-making that is driven by prejudice, speculation and uninformed ideology (AbouZahr & Commar 2008; AbouZhar et al 2007; Shibuya 2008; Bailey & Pang 2004). Better data can provide insight into public health problems and guide the development of policies: both resulting in improved health. The production of accurate, relevant and timely data on the health status of communities is an essential foundation of public health, as it assists in identifying risk factors and the characteristics of those people who use and need health services (WPRO 2003; HMN 2008). As more attention is placed on health systems and Page 16

19 HIS, sound statistics will also be increasingly required to track performance, monitor progress and evaluate effectiveness, efficiency and impact (Boerma et al 2001; Bailey & Pang 2004; Shibuya 2008). Despite the importance of HIS, little attention has been given to research needs in this area. While there has been a focus on research to generate better data for specific purposes or studies, there is a lack of research on how to improve health data in general. 1.2 Pacific Island Countries and Territories The Pacific Region is a diverse area and includes 22 island countries and territories with an estimated population of nine million people (SPC 2009). The islands are separated into three sub-regions of Melanesia (west), Polynesia (southeast) and Micronesia (north), based on their ethnic, linguistic and cultural differences (SPC 2009; Taylor 1990). Health outcomes in the Pacific are varied in each of these sub-regions, with traditional causes of mortality and morbidity continuing to be a major concern in some countries, while changes in life-style, urbanisation and diet have changed the epidemiology of other countries and caused a rapidly increased burden of non-communicable diseases (Lum On et al 2009). The degree of organisation and sophistication varies widely both between and within countries: from welfare state systems in Micronesia, to developing country systems in Melanesia, and transitional systems in the larger Polynesian Islands (Finau 1994). While the tendency to talk about the Pacific as a homogenous entity continues; due to variations in population, size, geography, political status, history, ethnic groups, language, natural resources and economic development, generalisations are unsurprisingly difficult (Taylor 1990; Haberkorn 1997). However, what can be said about Pacific Island Countries and Page 17

20 Territories is that they are characterised by remoteness, dispersed and small total populations, vast ocean distances and limited human resource and institutional capacity (PIFS 2002a; PIFS 2002b; Network Strategies 2010) making issues of access and communication vital for the effective functioning of their health systems (Hong et al 2002). With the diversity in history, economics, loyalty (to different colonial power structures) and culture in populations of the Pacific, it is unsurprising that their healthcare systems, which are inherently socially-based, are also diverse (Finau 1994). While some of the more wealthy nations have relatively high performing health systems, most nations of the Pacific suffer from under-development and poor infrastructure, and rely on external assistance for the provision of basic health services (Hong et al 2002). The World Health Organization (WHO) has, for example, provided much technical support to many nations in the Pacific over the years, including Tonga and Fiji (ibid). Overall, health systems are primarily funded through governments, however complex funding arrangements exist due to the multiplicity of bilateral and multilateral aid programs in place, which continue to provide vital services including water and sanitation, food and education (Taylor 1990). The quality of available data on health systems is variable and contradictory; up-to-date demographic information is unavailable in most countries or varies substantially from year-toyear; and there remains a paucity of reliable and current data on health outcomes (Taylor 1990; Haberkorn 1997). Death registration systems are generally regarded as deficient due to issues in identifying underlying cause-of-death, and morbidity is often more difficult to measure, mainly relying on individual contact with health care systems, special disease recording systems and ad-hoc surveys (Taylor 1990). Page 18

21 1.3 Health information systems Health information systems (HIS), defined by the World Health Organization (cited AbouZhar & Commar 2008: 1) as integrated efforts to, collect, process, report and use health information and knowledge to influence policy making, programme action and research are essential to the effective functioning of health systems worldwide. Routine HIS, such as those operated through health information departments or national statistics offices, provide information on risk factors associated with disease, mortality and morbidity, health service coverage, and health system resources (Lewin et al 2010). Furthermore, governments rely on the information provided to them from HIS for the production of high-quality, user-friendly statistical information on the health status of the community; the use and need of health services; formulating, monitoring and evaluating health policies; and measuring progress made in the provision of health services (WPRO 2003). HIS can also identify health problems; form effective health policies; respond to public health emergencies; select, implement and evaluate interventions; and allocate resources (Pappaioanou et al 2003). The Health Metrics Network (HMN) provides us with a conceptual representation of the components and standards of a health information system in Figure 1. HIS are part of the health and wider statistical system, which covers non-health sectors such as education and employment (AbouZahr & Boerma 2005). They include the legal and regulatory frameworks providing mechanisms for data availability, exchange, quality and sharing; core indicators to monitor local and national priorities; principles related to data sources, including standard principles for data collection and routine procedures to correct for bias and confounding; appropriate methods for data processing, compilation and management; data quality checks including assessments of reliability, transparency and completeness; and policies around dissemination and use (HMN 2008). Most traditional HIS collect data at a granular level by Page 19

22 various means such as surveys, clinical observation, diagnostic testing or management and financial information systems. They focus on individuals (citizens, patients, health care providers), characteristics of the services they need, use or deliver, the resources required to deliver those services and the impacts they achieve. Those data are then consolidated, analysed and reported in various ways to create summary information for use by service providers, managers, planners, researchers, commentators and others with an interest in the health sector (Davies et al 2010). Figure 1 Representation of the components and standards of a Health Information System (HMN 2008a: 4) Overall, HIS are a core building block of health systems (AbouZahr & Commar 2008). Health information underpins the entire health system: it strengthens stewardship (Shibuya 2008), can be used in strategic planning and priority-setting, as well as within clinical diagnosis and management, quality assurance and improvements, and global epidemics (Stansfield et al Page 20

23 2006). Healthcare information promotes excellence in care; describes the types of people using a service and the types of services received; helps coordinate services; provides meaningful information on the health status of the community; and ensures accountability (WPRO 2003). The core premise of the Health Metrics Network (2008) is that better health information will lead to better decision making, and as such, better health. While the link from information to decision making is not quite as clear as proposed, 1 the production of quality data that is transformed into useful information does have an inherently positive value (AbouZahr et al 2007). Decision makers, for example, cannot identify problems and needs, track progress, evaluate the impact of interventions or make evidence-based decisions when they lack information (HMN 2008). An underlying concern of all HIS is that of data quality, including ensuring the correct resources are dedicated to data generation and processing; indicators are used; sources are available; management structures are in place; information products are produced; and finally, dissemination strategies. It is vital then, that any research into HIS beings with an understanding of data Data Data are, in essence, the raw materials required for action (AHIMA 2008). They are the product of measurement or observation; a representation of facts, concepts or instructions presented in a formalised manner for communication, interpreting and processing (Lewin et al 2010; WPRO 2003). Data in health, for example, can relate to facts that describe the characteristics of individual patients (age, gender, cause-of-death), or aggregate measures of 1 See for example, King 2010, Goddard et al 2006 and Robinson 1999, for a brief introduction on the models of political-economy that highlight the broader social, political, institutional and environmental constraints health care decisions are made under Page 21

24 population health (life expectancy, maternal mortality) (WPRO 2003). Data is required for evaluating existing services, planning for the future, and introducing measures in anticipation; be they on sanitation and water, immunisation, infectious diseases, heart disease or road traffic accidents (ibid). Data also provide insight into public health problems, assist in developing rational and equitable priorities, and as discussed previously, contribute to improved health (Boerma et al 2001). As more attention is placed on HIS, sound data are becoming increasingly required to track performance, monitor progress and evaluate the effectiveness, efficiency and impact of health interventions (AbouZahr & Commar 2008; Shibuya 2008). Data are also driving more and more healthcare decisions, and many initiatives have been established to use data in monitoring performance improvement efforts, improving outcomes, and comparatively as benchmarks (AHIMA 2008). The elevated importance of data in health is reflected in the growing number of organisations and publications dedicated to the topic. In 1991 the US Centre for Disease Control and Prevention and the US Agency for International Development funded the Data for Decision Making Project (DDM) (Pappaioanou et al 2003; AbouZahr et al 2007). The goal of the DDM was three-fold; (1) to strengthen the capacity of decision makers to identify data needs and interpret and use data appropriately; (2) to enhance the capacity of technical advisors to provide valid, essential and timely data; and (3) to strengthen HIS collection, analysis, reporting, presentation and use at all levels (Pappaioanou et al 2003). Several other important events have also recognised the role of quality data and information in healthcare, including the release of the Fundamental Principles of Official Statistics (UNSTATS 1994); publications from the World Health Organization on improving data quality (WPRO 2003); and more recently, the establishment of the Health Metrics Network in 2005, Page 22

25 with its focus on improving global health and strengthening the systems that generate health information; followed up by the publication of the data quality framework by the Canadian Institute of Health Information (2009). 1.4 Issues and challenges The value of data... depends almost entirely on its uses, which may not even be fully known Tayi & Ballou 1998: 56 Despite unprecedented global interest and investment in health, and the statistics maelstrom this has produced, little is reliably known on the mortality or incidence and duration of disease in many developing countries (Boerma et al 2001; AbouZahr et al 2007). We still struggle, for example, to answer simple questions such as who dies from what for most of the world s population. While a basic building block of all HIS is counting births and deaths (HMN 2008): the stark reality remains that, most people are born and die uncounted, the reasons behind their deaths unknown (AbouZahr & Boerma 2005: 579). Due to historical, social and economic forces, most HIS are complex, fragmented and unresponsive to users needs. Furthermore, chronic under-investment in systems for data collection, analysis, dissemination and use, means that few developing countries have strong and effective HIS to monitor the health status of their populations or progress towards internationally agreed outcomes such as the Millennium Development Goals (HMN 2008; AbouZahr & Boerma 2005). As resources for interventions have surged; global health partnerships increased; and the amount of donor aid to developing countries risen dramatically, demand for more and better information has also increased (AbouZahr & Commar 2008; Shibuya 2008). However, while a Page 23

26 role of HIS is to generate, analyse and disseminate useful information, they rarely function systematically (AbouZahr & Boerma 2005). Many HIS have technical inefficiency: they lack a common database, standardised metrics and quality assurance procedures (Shibuya 2008). Human resource capacity in terms of statistical data skills are often overlooked, especially in developing countries, with staff poorly paid and undervalued (Stansfield et al 2006). Ministries of Health often do not manage large components of their HIS and much data collection is simply out of their hands. Furthermore, information systems in countries where global health investments are directed are usually weak and fragmented by disease-focused data requirements, leaving them overwhelmed by multiple, parallel information demands and overburdened by excessive reporting requirements (AbouZahr & Commar 2008; HMN 2008; AbouZahr & Boerma 2005; Mills et al 2006; AbouZahr et al 2007). As discussed in Chalkidou and colleagues (2010), decisions in many developing countries are driven by historical norms, donor interests and lobbying pressures, with little incentives or capacity to collect, share, analyse and interpret local data (Shibuya 2008). They provide the example of Rwanda and the grossly disproportionate funding allocated to HIV/AIDS in comparison to its relatively low infection rate, to highlight how local values, data and evidence are often ignored in favour of political pressure and standard-setting agencies such as the WHO (Chalkidou et al 2010: 284). There is also a noticeable paucity of evidence regarding HIS due to the limited role they play in research priorities, with the current body of knowledge on the topic referred to by Mills et al (2006) as ad-hoc, disjointed, and an unsystematic collection of facts, figures and points-ofview. Health information systems are a historically neglected field, and underinvestment continues to be the root cause of weaknesses (Stansfield et al 2006). There remains a large Page 24

27 disconnect between the need for information and a country s ability to respond. This tension between country needs and global demands (AbouZahr et al 2007) raises many questions around what essential information is, and who it is essential for (Bailey & Pang 2004). It also questions how information can be created and used locally, to respond to relevant local needs and demands (Duran-Arenas et al 1998, Bailey & Pang 2004). Large international surveys, for example, are not seen as being within HIS and are generally done to compensate for a lack of routine systems (Stansfield et al 2006). However continued investments in surveys, and other externally-driven information-gathering mechanisms, have allowed the ongoing neglect to develop comprehensive and sustainable HIS in many countries (AbouZahr et al 2007; Stansfield et al 2006). While there is a broad consensus that improved health outcomes need strengthened health systems (of which HIS are a core component), much of the data and information produced from HIS, remain unprocessed, or, if processed, unanalysed, or, if analysed, not read, or, if read, not used or acted upon (HMN 2008: 15). That is, as well as having their own inherent issues, HIS are also affected by issues related to their core building block: data. Raw data alone are rarely useful; they must be converted into credible and compelling evidence; complied, managed and analysed to produce information; integrated; and evaluated in terms of issues confronting the health system (AbouZahr & Commar 2008) (Figure 2). Data require an organised set of processes and procedures for this flow of collecting, collating, analysing and communicating (Stansfield et al 2006): they need a fully-functioning HIS. It should not come as a surprise then, that many developing countries struggle with this complex task and have become what AbouZahr and Commar (2008) and the HMN (2008) refer to as data-rich but information-poor. The issue of too much data and not enough information is Page 25

28 not restricted to the health sector however. In their research on rational data choice in politics, Mudde and Schedler (2010) remark that while there is an unprecedented abundance of cross-national political data, with datasets expanding every year, political actors are illequipped to deal with the luxury (and necessity) of choice. Due to issues of both information supply and quality, they conclude how, swimming in data wealth, we run the risk of drowning in numbers (Mudde & Schedler 2010: 411). Figure 2 Cyclic representation of transforming data into evidence (HMN 2008: 44) Furthermore, despite the fundamental importance data plays in healthcare management, planning, monitoring, and evaluation, there remains little awareness on the impact greater information use has in advancing health, and even less attention on the systems needed to provide accurate, timely and relevant information (AbouZahr & Commar 2008; Stansfield et al 2006). Assessment of health status in developing countries is often based on limited data from epidemiological research, health facility reports and national population-based surveys. While Page 26

29 data from health facilities is broad in coverage, it is often incomplete as the number of services delivered does not equal population need (AbouZahr et al 2007). Common barriers to the use of data include poor quality evidence; failure to frame issues in a policy context relevant for decision making; failure to package and present data in an understandable and compelling format; and a lack of trust in the overall quality of the HIS (Pappaioanou et al 2003). Factors compromising the quality of data include inadequate training for data collectors and processors; limited feedback from end-users; and a lack of understanding from most parties involved of the importance on data in health (Lewin et al 2010). As argued by Lewin and colleagues (2010) much local data is descriptive, providing simple summaries rather than comparisons, and this has implications when assessing its quality. There is a false assumption among many collectors and producers that data can be used directly by decision-makers, however it also must be presented, communicated and disseminated appropriately so that people understand the data and can link it to health issues, needs and actions (Pappaioanou et al 2003; HMN 2008). Overall, due to the multiplicity of potential problems with data, in that there are so many ways for it to be wrong (Tayi & Ballou 1998); decisions continue to be made despite the absence of reliable information: decisions that are highly sensitive to political opportunism, expediency and donor demands (AbouZahr & Boerma 2005). Page 27

30 2 Literature Review This chapter provides results from the literature review undertaken and has three main components. Part one is a comprehensive overview of the current state of knowledge on Annual Reports, including common features and types of reports globally, and a description of the purpose of Annual Reports in the Pacific Region; while the final section provides a summary of common limitations and weaknesses. The second part is concerned with exploring the concept of quality: its role in data collection and analysis; its importance for the validity and use of Annual Reports; and how it is defined and assessed. The third part provides a summary of previous research carried out on Annual Reports, with particular emphasis on research focused on assessing data quality, and assessments of reports from the Pacific Region. 2.1 Annual Reports Annual health statistics yearbooks or reports are one of the routes through which health information is transferred from data producers to end-users or decision makers WHO 2007: 2 As discussed previously, HIS are comprised of a number of components, ranging from resources and indicators, to standards for data management and dissemination. Annual Reports, one aspect of HIS, represent information products : the formatting and packaging of information into readily available formats such as dashboards, reports, queries and alerts (HMN 2008). Reports are generally comprised of numerical data on the characteristics of people using healthcare facilities and the services provided, and often contain a wealth of raw Page 28

31 data (AbouZahr et al 2007; WPRO 2003). Information on the types of diseases, number and sex of newborns, characteristics of deaths, and service utilisation are all commonly contained in Annual Reports. Such information can be used to inform comparisons of past and present performance and health status; planning; assessing the work performed by providers; and funding requirements (WPRO 2003). Annual Reports also play an important role in monitoring and evaluation (Boerma et al 2001; WHO 2007). In their workshop on country best practices, the WHO (2007) formulated three broad types of Annual Reports: Type 1: Raw Data. These reports include detailed tabulations of data on health facilities and health service performance, including monitoring progress towards health goals and health service use. These reports are of limited use other than to researchers. Type 2: Statistical Reports. Basic summary statistics with an analysis of the data in terms of comparisons between groups and areas, and overall trends. Also included are activities conducted within the health sector, and operational descriptions of health care facilities. Type 3: Summary Report with Interpretation and Analysis. This type of report includes characteristics from the previous two, as well as information on the program and policy implications of data and is suitable to a non-technical audience. It reflects, in essence, the best practice approach to reporting. While content differs between countries, Annual Reports are generally comprised of three elements: health status, service provision, and health management/health system Page 29

32 information. Also included are program specific data, for example progress reports on HIV or malaria campaigns. Reports also differ in their stated purpose. The purpose of Annual Reports on Public Health in England, for example, are to provide regular epidemiological assessments of population health; act as a basis for policy development; provide information for joint planning with other agencies; and set targets against which improvements in public health can be measured (Chambers & Bevan 1990). Similarly in Australia, the Department of Health and Ageing Annual Reports have specific reporting obligations, as set out in the Public Service Act 1999, Requirements for Annual Reports. The primary purpose of each report is to provide a useful and informative picture of performance over the past 12 months, in line with their core value and commitment to accountability (DOHA 2009). The National Health Care Quality Report from the Department of Health and Human Services is published to report on progress and opportunities for improving the health of the American population (AHRQ 2009). The report is part of a growing knowledge base on three key areas: the status of healthcare quality in America; where healthcare quality improvement is most needed; and how the quality of healthcare delivered to American s is changing over time. The report is comprised of over 200 measures across the four dimensions of effectiveness, patient safety, timeliness and patient centeredness (ibid) Annual Reports in the Pacific Annual Reports in the Pacific serve a broad range of purposes including compliance with legislative requirements; donor accountability; the provision of information to the public; planning; and international reporting agreements. In the three countries reviewed, all addressed the reporting requirements as set out by their respective Public Health Acts, in relation to the notification of infectious and dangerous conditions (which ranged from Page 30

33 dengue fever and HIV/AIDS, to conjunctivitis and polio). Both Fiji and Tonga also have specific legislation regarding the development and dissemination of Annual Reports; requiring them to provide a summary of action taken during the year and an update on the health status and health services of the country. The intended purpose, use and audience of reports are varied: the Cook Islands is the only country to clearly state that Annual Reports are to provide key health statistical information for the Cook Islands, which can be used by the Ministry of Health or any other interested party (Cook Islands Annual Report 2005: ii). Fiji and Tonga both provide a range of different purposes, ranging from providing a summary of occurrence of vital events; performance against health outcomes and MDG indicators; showcasing the roles and functions of particular health units; to providing indicators by which the Government s progress in policy/strategy implementation can be monitored and measured (Fiji Annual Report ; Tonga Annual Report 2007). Overall, while reports in the Pacific all generally contain information on the health status of their populations, and detailed operational information on the performance of their health system; the intent, and as such content, of each report are as varied as the countries making up the Pacific Region are themselves Common limitations and weaknesses While data published annually by most countries are assumed to be meaningful, this is not always the case (Shibuya 2008). A vast amount of data is collected within HIS, yet only a small amount is synthesised, analysed and used. In the case of Annual Reports, data is often collected and presented in crude formats, with limited attempts at analysing the data for use in day-to-day management and planning (HMN 2008). While, there is little point in engaging in the time- and resource-consuming process of data collection if there is no commitment to Page 31

34 analysing the data, disseminating the information and using it to improve health system functioning (HMN 2008: 14), many Annual Reports seem to do just that. As remarked by WPRO (2003) in their publication on improving the quality of reports, many Reports only present work done during the reporting period, which is not particularly useful for problem identification or decision-making. They argue that reports comparing a select number of indicators over time are far more useful for such purposes. There are many reasons for the poor quality of Annual Reports, ranging from issues of incomplete, inaccurate or insufficient source data; to poor transfer of data from one document to another; inaccurate coding; and the lack of standard terms (WPRO 2003). Further, the use of different sources, definitions and methods reduces data comparability between countries and within reports over time (AbouZahr & Boerma 2005). As such, assessing trends becomes difficult, and opinion, extrapolation and estimates are favoured above the reported data itself: a pattern clearly demonstrated in the limited international (and national) use of Pacificgenerated data. Annual Reports are primarily comprised of administrative data: data that is the by-product of delivering services to people. However, as argued by Iezzoni (1997), such data was never intended to assess outcomes; it is only due to its readily available and inexpensive nature that the use of administrative data has taken on a wider role in reporting. The HMN (2008) also comment on the limitations of administrative data due to its inherent bias in only reporting on the population using health services. Annual Reports often serve multiple purposes, including the development of statistical databases and acting as basic public health reports. More importantly, they act as the sole or main outlet for the dissemination of facility-based data: yet there is no standard reporting system guiding the content of such reports, or the processes around data analysis or Page 32

35 presentation (WHO 2007). More often than not, lower level managers are required to report on a vast quantity of data to higher levels: data for which they receive no feedback and data that is rarely used at higher levels, due to what AbouZahr and Boerma (2005) refer to as information overload. As such, processes for improving quality revolve around preparing reports in a logical, useful and meaningful manner; checking data for face validity and consistency; proof-reading; and explicitly defining the purpose, objectives and scope, through asking questions such as what information does the user want, what information is available, and what is routinely collected or will require additional work (WPRO 2003: 56). 2.2 Quality The ultimate goal is to manage quality. But you cannot manage it until you have a way to measure it, and you cannot measure it until you can monitor it Arah et al 2003: 377 Attention to quality in healthcare has become a central issue in recent years, with increasing awareness on the role quality plays in informing public policy, supporting health care management and building public awareness about the factors affecting health (CIHI 2009). As argued by WPRO (2003), hospitals, community health centres, clinics, aid posts and high-level health ministries and departments should all be concerned with the impact poor quality data has on the quality of health care provided to users. Overall, data quality is important for determining the current and future needs of patients; medico-legal responsibilities; ensuring diseases are being treated and procedures performed; measuring outcomes of health care interventions; obtaining information on the users of services; teaching health care professionals; and planning (WPRO 2003; HMN 2008). Page 33

36 In everyday language, quality represents where, on a scale of bad-good-excellent, a user may place a certain product with regard to its intended use, and also in light of comparisons with other available products (Elvers & Rosn 1997). Quality assessments generally take one of two paths: procedural or substantive. Procedural assessments are concerned with issues related to transparency, reliability and replicability how the data was processed and analysed (Mudde & Schedler 2010; Lewin et al 2010). Substantive assessments, on the other hand, deal with data outcomes and have criteria based on validity, accuracy and precision (ibid; ibid). However, ensuring the quality of data is much more difficult than ensuring the quality of other raw materials, and as Tayi and Ballou (1998) argue, this difficulty is further compounded by the low priority assigned to data quality assessments. Confidence in the quality of information produced by an agency 2 is vital for its survival: as soon as any information is regarded as suspect the credibility of an agency is called into question and their perception as being a trustworthy source is undermined (Brackstone 1999). When information in public health reports is not accurate or available when needed, potentially disruptive consequences can result, including debates becoming focused on who has the right numbers instead of the pros and cons of public health policy (Brackstone 1999; WHO 2003). In their work on quality, Lewin and colleagues (2010) are very clear that while local data 3 is important in contextualising and making global data relevant; we should remain cautious about using local data alone, as it is less reliable and can be misleading. They further argue that global data is often the best starting point for making judgements on the effects, 2 Here, an agency could refer to a Ministry of Health, Statistics Department, District hospital or individual healthcare clinic, for example 3 Defined as evidence available from the location the decision or action will take place in Page 34

37 modifying factors and ways to approach and address health problems, as much local data is difficult to locate and of poor quality. Caution over the use of local data is reflected in the long-held consensus that information and data from the Pacific is, incomplete, unreliable, obsolete and of poor quality (Finau 1994: 163). This consensus is clearly demonstrated at an international level: while Fiji s Annual Reports, for example, show a maternal mortality ratio (MMR) ranging between 31 and 51 during 2004 to 2008; the World Bank (2010), World Health Organization (2010), UNICEF (2010) and UNSTATS (2010) all officially report the MMR for Fiji in 2005 as 210. In the majority of these external sources no reference is made to the reported MMR provided from Fiji, and while calculations are provided for how the adjusted or modelled MMR was established, no justification is provided for why Fiji s MMR (which is approximately four-times lower than the modelled data) is not included in the official statistics. Overall, it would seem that due to the perception that the quality of data being produced in the Pacific is of dubious quality, much of the information is ignored and underutilised. The example of the limited and lessening use of data produced from within the Pacific Region is what Wang et al (1997) refer to in their seminal work on data quality as an intrinsic data problem (Figure 3). The logic behind an intrinsic data problem is as follows: 1. Mismatches in data provided from different sources initially causes a believability problem as users do not know which source is incorrect, only that the data conflicts 2. As information on the causes of the mismatches accumulate, evaluations on the accuracy of the data are generated 3. This leads to certain data gaining a poor reputation Page 35

38 4. As this reputation builds, the data are seen as having little value-add and so are used less. Intrinsic data problems can also stem from judgements on the data production process; for example, placing a higher value on raw data as opposed to aggregated (this is demonstrated in path (2) in Figure 3). The authors state that, a reputation for poor quality can also develop with little factual basis (Wang et al 1997: 105). This is of heightened importance for Pacific Island Countries and Territories, as they not only have to improve the quality of their data, but also improve the reputation of their entire HIS: an undoubtedly challenging and complex task. Figure 3 Intrinsic data quality problem (Wang et al 1997: 105) Defining quality While literature provides a wide range of techniques to assess and improve data quality, as information systems increase their size and scope, issues of quality are becoming more Page 36

39 complex and controversial (Batini et al 2009). Due to the contextual nature of quality, there remain discrepancies in definitions of its dimensions, and no agreement on which set of dimensions defines quality (ibid). Furthermore, assessments of data quality among qualitative research, or unstructured data, are virtually absent (Akkerman et al 2008; Batini et al 2009; Tayi & Ballou 1998). As discussed by Brackstone (1999), the traditional statistical concept of quality, related to measures of standard error and bias, does not adequately address the broader meaning quality has taken on in the management of organisations and systems. Here, he argues, quality refers to the fitness of final products and services in meeting the needs of users. However, if users needs are taken as the primary factor in assessing the success of products and services, and quality is taken to reflect the aspects of statistical outputs that reflect their fitness for use the varied number and needs of users mean that we are still left without an operational definition (Brackstone 1999). Furthermore, in defining quality in terms of its fitness for use this implies that the concept of quality is relative and that data with quality for one use might not have quality in another (Tayi & Ballou 1998); again providing no clear definition. Other authors and agencies have attempted to define the concept of quality: Elvers and Rosn (1997), the Canadian Institute of Health Information (2009) and Wang et al (1997) similarly regard quality as a measure of how well statistics meet users needs and expectations. In their work on quality, Arah et al (2003) argue that performance indicators (measures to capture health and health system trends and factors) provide an operational definition of quality, as performance indicators are essentially a quantitative measure of quality. The World Health Organization (WPRO 2003) regards quality as the production and dissemination of Page 37

40 understandable information for government policy-makers, community leaders, health planners and healthcare providers. As argued by Elvers and Rosn (1997), quality has taken on a descriptive meaning, and quality assessments need to consider both the product in question, and also its purpose. The Health Metrics Network (2008) echo this sentiment over a decade later, when they describe the process of assessing existing HIS in order to understand users current and perceived future requirements for statistical information. They propose that such assessments must be carried out, if we are to increase the availability, quality and use of health information vital for decision-making at country and global levels (HMN 2008a: 2). Overall, while there remains no single definitive definition of quality, most authors agree that it lies beyond the traditional statistical concept concerned with accuracy, and that it is madeup of a number of important components or dimensions (Brackstone 1999; Elvers & Rosn 1997). Again, while there is no universal consensus on which dimensions are required to produce quality, a number of dimensions are interrelated and there is significant overlap between different authors and agencies. In their review of the literature, Batini and colleagues (2009) provide a list of what they consider the four most basic quality dimensions as used by the majority of authors on the topic: accuracy, completeness, consistency and time-related dimensions. Table 1 provides a summary of the different quality dimensions defined by various authors and agencies. In general, the different dimensions of quality assess two main features: if information on the right topics is being produced, and if the appropriate concepts of measurement are being utilised (Brackstone 1999; IHP+ 2009). Page 38

42 component of relevance when discussing quality. All raise the question of whether the data is relevant to topical policy issues and adequately meeting the needs of users, or, as Brackstone (1999: 3) asks, if agencies are still counting buggy whips. The question of relevance seems, on face value, to be an important one to ask. However, as explicitly discussed by Elvers and Rosn (1997) and implicitly inferred by other authors and agencies in their exclusion of the dimension; relevance is not an intrinsic property of statistics. While some data may be highly relevant to certain users, for others it may have no value at all, due to their differing interests. Only users can decide the relevance of information (Elvers & Rosn 1997), and as such, it offers little practical guidance on quality assessment. Overall, despite the plethora of research on the role of quality and the need for thorough quality assessments; it remains difficult to define the concept of quality and best describe how to assess data quality. 2.3 Previous research Despite almost every country in the globe preparing health reports, representing a significant investment of efforts and resources (WHO 2007), very little systematic research into the content, quality, utilisation and potential areas for improvement of Annual Reports has been conducted. An exception to this is during the early 1990s in England, when Annual Reports on Public Health became a central topic of discussion and debate. In 1972, after the regular publication of Annual Reports for 100 years, public health authorities were released from their reporting requirements due to rising uncertainty over their value in improving health, and the considerable resources needed to produce them (Black 1989; Watt et al 1993). However by 1988 the reports were reintroduced with a new sense of optimism about their potential value (Fulop & McKee 1996). Annual Reports were argued as having a key role in improving and maintaining quality public health practice, clarifying responsibilities of health Page 40

43 authorities, contributing to accountability and focusing attention on important health problems (Black 1989). Following on from this emphasis on the importance of Annual Reports in public health, a number of researchers turned their attention to the issue of the quality of their contents. In their analysis of 28 different Annual Reports, Chambers and Bevan (1990) found the content and style of reporting varied significantly and that few reports met the legislative requirements for reporting. Overall, they concluded that while most reports were simple and easy to read, with well displayed compendiums on vital statistics, most reports were of a descriptive rather than analytical nature; only two reports made specific recommendations that could be used in planning; and the majority did not provide any basis for the joint planning of programs related to improving public health. Furthermore, information held on computerised National Health System databases were seldom used; there was little similarity in indicators presented; few reports attempted to link data (such as childhood immunisation with disease notification data); and most reports did not include data on the most vulnerable sections of the population, including the elderly, disabled or mentally ill (Chambers & Bevan 1990). Fulop and McKee (1996) went on to argue the central problem with such reports is their fundamental disagreement over who their audience is funders or the public. They also stated that while reports have the potential for great impact, political and financial pressures are major limiting factors. In their study into the use of Annual Reports for identifying health promotion activities, Anderson and colleagues (2003) concluded that local government reports are a useful source of information that can provide knowledge on the priorities and capacities of local authorities. They also remarked that reports are one of the only pubic sources of information stating political intentions. Page 41

44 2.3.1 Increasing global attention In July 2010, representatives from ten countries 4 and a number of international agencies (including GAVI, WHO, World Bank, CDC and Global Fund) convened a technical meeting to assess current country reporting practices. The review centred on the following themes: Well chosen and balanced indicator selection Appropriate data sources Quality assessment and processes Sufficient capacity for analysis and synthesis, and Effective communication of results to key audiences (WHO 2007). Main findings from this review were that while most countries had a list of core indicators, in some cases this included more than 100 indictors and they were often skewed towards particular elements in the results chain (ibid). The challenge here, reviewers argued, is to ensure an appropriate balance across the range of input, output, outcome and impact indicators. In terms of data sources, most countries were found to include references to the origin of the data, which ranged from administrative sources to household surveys. While the data contained within each report varied between countries, a common characteristic was the lack of systematic quality assessments, resulting in biased, incomplete, and late data. Capacity for analysis and synthesis was also limited, with most countries relying on external consultants. Other issues related to the production of Annual Reports reviewed included numerous reporting requirements, challenges between the demand from donors and available supply of data from countries, continued data gaps, and limited capacity at every point in the system 4 Benin, Ethiopia, Ghana, Kenya, Nepal, Mali, Mozambique, Rwanda, Thailand, and Uganda Page 42

45 (ibid). Furthermore, challenges affecting the use of reports in decision making ranged from issues with completeness and coverage; comprehensiveness; data quality and triangulation; data standards; timeliness; capacity to respond to different demands; and ability to cater to diverse audiences (ibid). Finally, in terms of communication and use, it was found that Annual Statistical Yearbooks, Abstracts or Reports were the most common mode for transferring information from data producers to end users. However, despite the considerable effort and resources invested in producing reports, they remain underutilised in the health and development community due to poor presentation (such as long and complex tables), limited accessibility (unavailable or un-downloadable from websites) and poor timeliness. A similar workshop facilitated by WHO was held in South Africa in October 2010, with the intent of enhancing the analytical capacity of countries to conduct comprehensive health progress and performance reviews in the context of national health plans and related global health goals. Furthermore, a third workshop is due to be held in Bangkok in March 2011 and it is planned to have representation from both Asia and, for the first time in this arena, the Pacific. Overall though, despite the growing international attention Annual Reports have received in recent years, little follow-up action has occurred (such as the production of country guidelines or training on data analysis that were due for publication by WHO in 2008) and no work has been carried out in the Pacific as yet Pacific examples Despite the lack of academic research on Annual Reports in the Pacific, one country provides us with an example of their own internal review and critique of reporting methods. As part of the World Bank funded Tonga Health Sector Support Project, work was carried out during 2005 to improve the HIS of Tonga, including revising the main information product of the Ministry Page 43

46 of Health : Annual Reports (Tonga Annual Report 2006: 17). As well as focussing significant efforts on improving data quality, information management processes and reporting procedures, a main goal of the project has been to accelerate an information culture within the Ministry of Health and Vaiola Hospital. Specific recommendations to update Tonga s reports include: Removing duplication Reporting against planning objectives Simplifying the format Standardising statistical presentation and accompanying narrative Establishing a clear link with the National Strategic Development Plan. Apart from this one example, no other research on Annual Reports from the Pacific was located: highlighting once more the paucity of information on HIS in the Region. Furthermore, despite the vital importance of quality assessments, due to the limited academic rewards and substantial efforts required, evaluations of data quality are not regularly featured in academic journals (Mudde & Schedler 2010), and no evaluations of the quality of Annual Reports from the Pacific, or globally, were located. This is endorsed by Brouwer and colleagues (2006) in their research on data quality improvement in general practice clinics. They remark on the explosion of data being collected by general practitioners over the past decade, and the general acknowledgement among collectors and end-users that such data still has ample space for improvement. However, despite the recent attention on quality and consensus over the need for improvement, they also remark on the surprisingly few studies dedicated specifically to the topic of quality improvement (Brouwer et al 2006). Furthermore, while striving for Page 44

47 completeness is regarded as the first broad step in improving the quality of general practice records, there remains no criteria on the acceptable level of quality, and no ideas on what is good or high quality, nor good-enough or high-enough (ibid). Overall, the area of Annual Reports, and specifically the quality of data within Annual Reports, is an under-researched area and this lack of research is even more pronounced when assessing reports from Pacific Island Countries and Territories. Page 45

48 3 Aims and Objectives From the limited previous research on Annual Reports available, it is apparent that most reports are underutilised; cumbersome and poorly presented; have too many tables with insufficient analysis and visual presentation; and have enormous variation in content and format (WHO 2007). However, as identified in the 1980s in England, and more recently through the work of the WHO, the data contained within Annual Reports have great potential value and deserve wider dissemination due to the influence such data can have on strategic and short-term planning, as well as in monitoring and evaluating the health status of populations (WHO 2007; Black 1989). Despite the importance of HIS, little attention has been given to research needs in this area. While there has been a focus on research to generate better data for specific purposes or studies, there is a lack of research on how to improve health data in general (HMN 2008). Furthermore, in their seminal paper on neglected health systems research, AbouZahr and Commar (2008) reiterate the general under-investment in HIS and propose three key actions to move the agenda of research on health information forward: (1) developing consensus around a prioritised research agenda; (2) enhancing communication between researchers and health information experts; and (3) mobilising resources for research in this field, especially implementation research. In light of the paucity of research in this field, and the absolute dearth of information related to Annual Reports in the Pacific, this dissertation addresses the overall question of, What can Annual Reports tell us about the quality of data produced from health information systems in three different Pacific Island Countries? Specific objectives are to: (1) detail what is presented Page 46

49 in Annual Reports and assess how this varies both between countries and over time; (2) explore the structure of reports; (3) describe what the data in Annual Reports tells us ; and (4) assess the quality of the data between countries and over time. After consulting the available literature on Pacific Island Countries and Territories, health systems, health information systems, and Annual Reports, initial hypotheses are: HIS in Pacific Island Countries are not suffering from a lack of health data, but rather from a lack of useful information produced by the analysis and contextualisation of such data The practical use and application of Annual Reports are limited, due to the complexity and length of reports Very little information on the meta-data 5 of each report is provided. 3.1 Potential public health benefits Through this research, a detailed analysis of one aspect of the health information system in the Cook Islands, Fiji and Tonga will be provided. In recognition of the importance of looking at HIS as a whole, rather than in relation to specific diseases, this research takes a holistic approach, focussing on building upon what already exists in the Pacific Region. As well as providing a detailed report on Annual Reports to contribute to the body of knowledge on this topic, one of the main public health benefits from this research is the potential of producing recommendations on developing a standardised template to be used when composing reports. Such a template could: 5 Meta-data is data about data. It covers definitions of data elements/variables, their use in indicators, data-collection methods, time period of data collection, analysis techniques used, estimation methods and possible data biases (HMN 2008) Page 47

50 Help guide the development of each report Specify what data is needed and how to analyse and present information in useful ways Provide recommendations for the format of each report (such as the use of appropriate graphs and tables) to increase the use of reports Standardise reporting across Pacific Island Countries, to enable cross-country comparisons Promote the use of Pacific-generated data, through the use of internationally-agreed standards and indicators. In doing so, two wider health systems benefits are possible: (1) improving the efficiency and effectiveness of health systems; and (2) supporting better targeting of resources by governments and/or donor agencies. Page 48

51 4 Methodology This chapter describes the overall methodological approach adopted in this research, including an outline of the overarching theoretical framework guiding the dissertation (Section 4.1). Section 4.2 describes the research design, including country criteria applied in selecting the Annual Reports for review and the research design adopted, which involved two main aspects: a descriptive and analytical review. Details of the descriptive review are provided in Section 4.3, and this includes aspects such as report structure, data characteristics and indicators presented. Finally, Section 4.4 describes the analytical review and its five quality assessment dimensions: comparability, disaggregation, interpretability, presentation and timeliness. 4.1 Theoretical framework While researchers have debated which method (quantitative or qualitative) is superior over the other, it is becoming clear that the focus should be on determining which method can provide the most convincing answers (Thomas 2003). In recognition of the fact that certain research problems are better suited to certain research methods; a mixed-method approach was adopted in this research (Creswell 2003). Quantitative research methods are concerned with counting and measuring; producing both descriptive and inferential statistics; evidence on cause-and-effect; and explanations of events that are easily replicable by other researchers (Gillham 2005; Merriam 2009; Thomas 2003). A qualitative approach, in contrast, aims to understand the meaning of what is going on, to illuminate issues, provide possible explanations and speak for the facts : it is, in effect, a search for deeper meaning (Gillham 2005; Polkinghorne 2010). As most research does not fit neatly into one method or the other, but rather features a combination of both, a mixed-methodology perspective offers greater Page 49

52 potential for understanding the evidence and theory behind the event being studied (Tashakkori & Teddlie 1998). Mixed-methods contain both numeric and textual information; they are pragmatic, consequence-oriented, problem-centred and pluralistic (Creswell 2003). Within this mixed-methodology perspective, a content analysis approach was adopted, to both generate evidence on Annual Reports and help make sense of it (Gillham 2005; Holdford 2008). In essence, content analysis is an empirically grounded method that seeks to explore the messages communicated from one party to another, through data, printed matter, images and sounds (Krippendorff 2010). Content analysis can tell us about three main categories: (1) messages communicated (what they are, who are they for, how are they communicated); (2) impacts; and (3) the affect messages may have on dependent variables (Holdford 2008). Put simply, such analyses can tell us what messages mean to people, and what impact the information conveyed in them has. Due to the explanatory nature of content analysis, it is regarded as a qualitative research method. However, it can take both a qualitative and quantitative approach, as has been applied in this research. The primary research question, which is focused on assessing the quality of data contained within Annual Reports, followed a quantitative content analysis approach: focusing on quantification, objectivity and outcomes (ibid). In this sense, data within the reports were collected using predetermined instruments to yield statistical data (Creswell 2003) that could be compared and applied to other settings (i.e. different countries). Secondary research questions, concerned with reports structure and content, were explored using a qualitative content analysis approach. Here, interpretation, subjectivity, flexibility and the influence of context were emphasised (Holdford 2008), in order to provide depth and fullness to the topic and developing themes emerging from the data (Creswell 2003; Polkinghorne 2010). Again, Page 50

53 the value in having a mixed-methods approach in this research lies in its ability to explore questions unanswerable through only applying a quantitative or qualitative method. 4.2 Research design The overall research design is an intensive desk-top review of Annual Reports to the Minister of Health (also called Annual Statistical Bulletins for the Minister of Health) produced by Pacific Island Countries and Territories. A desk-top review of publically-available documents was chosen as it offers an unobtrusive means of collecting information: reports are not affected by the researcher or the study, thus minimising potential ethical implications. Furthermore, informal discussions were held with a number of people involved in the production of Annual Reports at the Health Information Systems Knowledge Hub Short Course held in Brisbane in September 2010, and also via communication through the Pacific Health Information Network (PHIN). The purpose of such informal communications was to share knowledge, provide an opportunity for feedback, and create channels of communication between the researcher and those working in health systems in the Pacific. Three criteria were applied when selecting countries for review, namely: Annual Reports were readily available in English (either online or in hardcopy); the cross-section of reports selected were all from a relatively stable system; and the reports were from a variety of Pacific Island countries, representing different population sizes, level of development and cultural influences. After reviewing the Annual Reports available, and in light of the country inclusion criteria, the following three countries were chosen for review: 1. Cook Islands Page 51

54 2. Fiji 3. Tonga. While a number of countries had reports dating back a number of years, it was decided to use the most recent reports available, for a maximum of five consecutive years. Only three reports were sourced for the Cook Islands (2001, 2005 and 2007). Both Fiji and Tonga produced five reports during the period , and these were included in the review. The review of each Annual Report was divided into two main sections: descriptive and analytical. In terms of the descriptive review, two main approaches were taken to explore the contents of each report, namely; report structure and data characteristics. Further, as the primary research question is concerned with assessing the quality of data contained within each Annual Report, five different quality assessment dimensions were developed. These dimensions form the basis of the analytical section of the review. Figure 4 provides a conceptual overview of the methodology, including key research questions. Page 52

55 What can Annual Reports tell us about the quality of data produced from health information systems in three different Pacific Island Countries? Country inclusion criteria 4.3 Descriptive review 4.4 Analytical review Report structure Data characteristics What is the quality of the data? Comparability Disaggregation Interpretability Presentation Timeliness Figure 4 Conceptual representation of methodology Page 53

56 4.3 Descriptive Review This part of the review aimed to answer the question, What is presented? in an Annual Report, and also assess how this varied both between and within countries over time. The objective here was to provide a description of the kinds of data presented in an Annual Report, and note any comparability or major discrepancies in the types of information presented Report structure In describing how each report was put together (its structure) the question of, What does it look like? was addressed. Five aspects were considered when assessing report structure: 1. Audience, purpose and use the structure of each report is heavily influenced by its intended audience, purpose and use. For example, if the purpose of a report is to fulfill stated legislative requirements (such as reporting on specific diseases and health outcomes), it follows the report will be structured in a way to address this. 2. Type of report each report was classified as belonging to one of three types: raw data (type 1); statistical reports (type 2); and summary reports with interpretation and further analysis (type 3). 3. Number of pages as number of pages was used as a proxy measure for report length; factors such as page size, font size and layout were taken into consideration. Fortunately, all of the reports reviewed used standard A4-sized pages, similar font style and size (Times New Roman or Arial, font size 11 or 12), and similar page layouts, allowing for comparability. 4. Number of sections all reports included a contents page outlining main reporting divisions. The total number of sections, and section topics (i.e. mortality, dental Page 54

57 health, health facilities) were recorded to compare changes in number of sections and topics over time and between countries. 5. Number of tables and figures this refers to the total number of tables and figures in each report, not just the number referred to in the contents page (as a number of tables and figures were not included in contents pages) Data characteristics After describing the structure of each report, the next step was to address the question of, What does it tell us? This was done via assessing the characteristics of data contained within each report. As some of the reports were over 100 pages long, and the bulk of the data was contained within tables (and to a lesser extent, figures), it was decided to only assess data contained within tables and figures for this component of the analysis. Furthermore, as the text within each report was primarily a summary or description of the data contained within tables and figures (i.e. illuminating important trends or changes), excluding the text from analysis helped reduce the chance of analysing the same data twice. Data domain Due to the large amount of information contained in each report, it was decided to use a framework, defined by Klassen et al (2009) as a conceptual device to provide structure to a set of ideas, values, or conceptual entities, to provide the necessary tool for analysing data characteristics. There are a number of conceptual frameworks for monitoring, measuring, and managing the performance of health systems (see Arah et al 2003, for a detailed exploration of the conceptual bases and effectiveness of frameworks in health). After reviewing the conceptual frameworks of the United States of America, Canada, Australia and the numerous Page 55

58 frameworks produced by the World Health Organization; the Health Metrics Networks (2008), Domains of measurement for health information systems (Figure 5) was selected as the most appropriate tool to use. Figure 5 Domains of Measurement for health information systems and related health indicators (HMN 2008: 20) Choice of this framework was two-fold. Firstly, as a number of Annual Reports were roughly divided into thematic sections on population demographics and risk factors; general health system information, including workforce, infrastructure and finance; and health status indicators, the conceptual divisions within the Domains of Measurement framework transposed relatively cleanly onto the reports. Secondly, conceptual frameworks developed by other nations and organisations are simply measuring different things. The US Department of Health and Human Services National Healthcare Quality Report (2010), for example, is based on assessing and reporting against: (1) effectiveness; (2) patient safety, (3) timeliness; and (4) patient centeredness. Frameworks proposed by the Canadian Institute for Health Information Page 56

59 (2009) and Australian Institute for Health and Welfare (2010) are very similar, both focusing on health status and outcomes, determinants of health, health system performance and community and health system characteristics. However due to the large amount of data related to inputs and outputs in Annual Reports from the Pacific, and limited amount of data related to outcomes, determinants of health and health system performance, these frameworks did not adequately capture the content of each report. Overall, the Domains of Measurement framework offered a model that was able to differentiate between the broad categories of data within each report (i.e. health status as opposed to health system), while also drill down to finer levels of detail (such as input versus output). The Domains of Measurement framework breaks health information systems, and their core indicators, into three major components: 1. Determinants of health including socioeconomic, environmental, behavioural, demographic and genetic determinants of health or risk factors. These indicators were selected as they characterise the contextual environment in which the health system operates. 2. Health system indicators include inputs into a health system and related processes such as policy, organisation, human resources, health infrastructure, equipment and supplies. This domain includes output indicators such as health service availability and quality, and also immediate health system outcome indicators such as service coverage and utilization. 3. Health status this relates to levels of mortality, morbidity, disability and well-being. Health status variables depend on the efficacy and coverage of health services, and Page 57

60 determinants of health that may influence health outcomes independently of service coverage. Decision-analysis flow chart In classifying what domain the data contained within each table or figure belonged to, a decision-analysis flow chart was developed (Figure 6). Each table and figure was analysed and categorised via eight different questions and three rules. For example, as outcomes can refer to the ultimate (mortality and morbidity) and intermediate (service utilisation) results of system processes, it was vital that clear definitions for outcomes; health status (mortality, morbidity and well-being); and outputs were established. The rules used for each decision are provided in Table 2. Table 2 Rules used in decision-analysis flowchart Rule Between and within rule Health status rule Non-exclusive rule Definition A single indicator can be classified as more than one category between the domains (i.e. health system and health status) but not more than one category within each domain (i.e. mortality and morbidity) Data related to mortality and morbidity are classified as health status only (not as health system: outcomes) As tables and figures can contain multiple indicators, data contained within the same table or figure can belong to more than one domain Page 58

61 START Table or figure ACTION Classify as: Does the data tell us something about the distal (broad/highlevel) factors that might impact on the health of the population? (i.e. birth rate, population size) NO Does the data tell us something about the proximal factors that might impact on the health of the population? (i.e. number of people who smoke, number of women in childbearing age using contraception) NO Does the data tell us something about what is required or in place to run the service? (i.e. number of health facilities, staff appointments and vacancies) NO Does the data tell us something about service availability or the number of services performed? (i.e. number of pharmaceutical items dispensed, number of inpatient contacts) NO Does the data tell us something about how many people are covered or benefitting from the service? (i.e. number of people immunised, number of inpatients) NO Does the data tell us something about what the population is dying from? (i.e. number of cancer deaths, infant mortality rate) YES YES YES YES YES YES Determinants of Health: Socioeconomic and demographic factors Determinants of Health: Environmental and behavioural risk factors Health System: Inputs Health System: Outputs Health System: Outcomes Health Status: Mortality NO Does the data tell us something about what is making the population unwell? (i.e. number of people with food poisoning, prevalence of diabetes) NO Does the table or figure contain any other data that has not already been classified? (i.e. a summary table with mortality, morbidity and determinants of health data all together) NO YES YES Health Status: Morbidity Return to START and begin process again END Figure 6 Domains of measurement, decision-analysis flowchart Page 59

62 Indicators presented Further analysis was also conducted on the indicators presented in each report. All indicators were classified into their appropriate domain of measurement (i.e. health system) and subdomain group (i.e. inputs). Mortality and morbidity indicators were also classified via two other methods: 1. Indicator group mortality and morbidity indicators were grouped into one of the three main headings of infant and child; maternal; or general 2. ICD-10 coding under each of the three main headings, mortality and morbidity indicators were further allocated into their corresponding ICD-10 category (where possible). For indicators that were not clearly labelled with an ICD-10 code already, coding was performed via online searches of the World Health Organization s database, available at In categorising indicators, only those that were specific to a group were selected for example, while an infant may die from an acute respiratory infection, so may an adult. Therefore, a table with data on the mortality rate of acute respiratory infections (with no reference to age groups) was classified as being an indicator for general mortality ; whereas the under-five mortality rate can only relate to infants and children, and would be classified as an indicator for infant and child mortality. Furthermore, changes in the definition or level of disaggregation of an indicator were recorded as two separate indicators. For example if, in 2004 the indicator diabetes mellitus, absolute number was reported, however in 2006 the indicator was changed to diabetes mellitus, by age and sex, absolute number this would be counted as two different indicators. Indicators Page 60

63 were allocated this way as it is difficult to compare data over time when they are presented according to different definitions. 4.4 Analytical review This, the second part of the overall review aimed to address the question, What is the quality of the data? Five dimensions were used in assessing data quality: comparability, disaggregation, interpretability, presentation and timeliness. In selecting the five quality dimensions for assessment, three main considerations were taken into account: (1) results from the literature review; (2) practical limitations of what could be measured; and (3) relevance to the contents of Annual Reports in the Pacific. While the literature review provided no overall definition of quality; it did provide 31 various dimensions, of which accuracy, timeliness and consistency were the most referenced. Due to the practical limitations of this research (including time and the restricted focus on only using publically-available documents) and in light of the equally compelling literature arguing against the continued reliance on accuracy (see, for example, Tayi & Ballou 1998), accuracy was not a dimension in the review. Timeliness and comparability were selected due to their relative importance in the literature review, ability to be measured and relevance to report contents. The next three dimensions selected were disaggregation, interpretability and presentation: all scored a reasonable number of references within the literature review, were able to be measured and were relevant to the contents. As put forward by Brackstone (1999), most of the dimensions of quality are not directly observable by the end-user, especially in the absence of meta-data (a common issue of Annual Page 61

64 Reports). Furthermore, while a number of authors state that usefulness and usability are important aspects of quality and that quality cannot be assessed independently of the consumers who choose and use the products; very few quality assessments or dimensions address these issues (Wang et al 1997). As such, many dimensions like completeness, representativeness, security, collection methods, and adjustments could not be assessed. Dimensions such as usability, believability, currency, objectivity, and serviceability were also of limited relevance to this research, as they could not be assessed through a desk-top review of reports Comparability Also referred to as coherence or consistency; comparability is a measure of how well data can be compared, either internally within the same data set, between data sets or over time (HMN 2008). Factors enabling comparability include the use of common frameworks, methods and systems during measurement, such as the use of international standard classification systems (Brackstone 1999; CIHI 2009). The more stable a definition is over time, the higher comparability it will have, thus increasing the quality of the data, as users can be assured the same thing is being measured in the same way (Elvers & Rosn 1997). In a recent workshop, AbouZahr (2010) emphasised that comparability is crucial if health care authorities are to assess whether they are improving over time and reaching benchmarks. Three concepts were developed to assess indicator comparability: (1) indicator-comparability score; (2) indicator selection; and (3) indicator stability. The number of year-on-year comparisons that can be made with a data-set is vital in assessing trends over time. As such, a method for capturing the level of comparability of individual indicators was developed, based on the mathematical principles of combinatorics (Mazur 2010). Combinatorics is concerned Page 62

65 with counting the number of k-combinations that can be chosen from an n-element set: for example, combinatorics can provide the solution to how many different combinations of two cards (k = 2) can be made from five different cards (n = 5). A common way of calculating combinatoric problems is to use the ncr function on a scientific calculator, where C represents combinations or choices. In the example above, typing 5 ncr 2 asks the calculator, how many ways of picking r ordered outcomes (r = 2) from n possibilities (n = 5) are there? We find that there are 10 different combinations that can be produced from ordering five cards into different two-card pairs. Indicator-comparability score Based on the principles of combinatorics, it is possible to calculate the number of year-on-year comparisons that can be made when an indicator is presented for one, two, three, four, five, or x years. Take, for example, an indicator that is presented for three years, out of a possible sample frame of five years. A diagrammatic representation of the number of year-on-year comparisons would look something like this (Figure 7): X X - X - Figure 7 Diagrammatic representation of the number of year-on-year comparisons possible for an indicator presented for three years Three distinct year-on-year comparisons are able to be made: ; ; and If this same indicator was then presented for one extra year, taking it to a total of Page 63

66 four years out of five, then the number of year-on-year comparisons increases to six. Overall, when using the ncr function, we find that: An indicator presented for one year (N1) provides zero year-on-year comparisons (N1 = 0) An indicator presented for two years (N2) provides one year-on-year comparison (N2 = 1) An indicator presented for three years (N3) provides three year-on-year comparisons (N3 = 3) An indicator presented for four years (N4) provides six year-on-year comparisons (N4 = 6) An indicator presented for five years (N5) provides ten year-on-year comparisons (N5 = 10). After establishing the number of year-on-year comparisons possible for each indicator (based on the number of years presented), it is then a matter of adding up all the indicators presented for one year and multiplying them by their maximum comparability number; followed by adding up all of the indicators presented for two years and multiplying them by their maximum comparability number; and so forth; and then finally dividing that number by the maximum comparability number if all indicators were presented in all reports sampled. A formula was developed to calculate the overall indicator-comparability score for all of the indicators within a given report. The equation for the indicator-comparability score (ICS) is as follows: ICS = (N5*10) + (N4*6) + (N3*3) + (N2*1) + (N1*0) N*10 Definitions: ICS = indicator-year score N4 = number of indicators reported for four years N1 = number of indicators reported for one year N5 = number of indicators reported for five years N2 = number of indicators reported for two years N = total number of indicators reported in all years N3 = number of indicators reported for three years sampled NOTE: as only three Annual Reports were analysed from the Cook Islands, the equation was modified as follows: ICS = (N3*3) + (N2*1) + (n1*0) N*3 Page 64

67 The following example highlights the application of the equation. Five Annual Reports were sampled from Country A and it was found that 10 indicators were reported for all five years; eight were reported for four years; five for three; another 10 for two years; and three indicators were only reported for one year. The calculation would look as follows: ICS = (10*10) + (8*6) + (5*3) + (10*1) + (3*0) 36*10 ICS = ICS = 0.48 The above indicator-comparability score of 0.48 informs us that, on average, less than half of the indicators reported over the full sample of five reports were comparable. An indicator does not have to be presented continuously to score for comparability: while it is preferable to have data reported continuously, it is still possible to monitor changes in an indicator that is presented in 2005, and not again until 2007, as an example. This is taken into consideration in calculating the indicator-comparability score. While only 10 indicators (28%) were reported for the full five years in the above example, another 23 indicators (64%) had some degree of comparability. However, simply stating that 92% of indicators were comparable ignores the fact that many of them were only comparable for a limited number of years. Thus the need for the creation of a measure of comparability that takes the number of years an indicator is comparable for into account: the indicator-comparability score. Indicator selection For indicators not presented over the full number of reports reviewed, two concepts were developed to assess issues of limited comparability. Indicator selection refers to indicators that are not reported for the full number of reports reviewed (i.e. an indicator that is only Page 65

68 presented for two out of five years has an issue with selection). Indicator selection relates to limited comparability due to the limited number of years it is presented for. An example of indicator selection is provided below (Figure 8): while the infant mortality rate is presented in 2004, 2005 and 2007; it is not presented in 2006 or 2008, leading to a fragmented and seemingly haphazard process of selecting indicators for presentation in reports Infant mortality Infant mortality - Infant mortality - rate rate rate Figure 8 Example of poor indicator selection Indicator selection issue Indicator stability Indicator stability refers to indicators that are presented differently year-to-year. Indicator stability results in limited comparability due to the reduced similarity of definitions used, and is more common among statistical agencies and countries that do not use standardised indicator definitions (such as ICD-10 codes). In the example provided below (Figure 9), four different ways of presenting tuberculosisrelated morbidity are used over the five years: 1. Case notification by age group 2. Number of cases 3. Number of cases by age group and region 4. Case notification by region. Page 66

69 Selection issue Number of tuberculosis cases Tuberculosis case notification by age group Number of tuberculosis cases by age group and region Tuberculosis case notification by region Indicator stability issue Figure 9 Example of poor indicator stability While case notification and number of cases may be the same thing, unless meta-data is provided, it is difficult to know if the same thing is being measured each year. Furthermore, the differing disaggregation variables provided make it difficult to compare trends over time. As a note, the same indicator could have issues with both selection and stability: in the example above there is no data for tuberculosis in 2005, as well as differing definitions of the indicator in the years it is presented Disaggregation The level of disaggregation of data is an important indicator for representativeness (another quality component), as by only presenting aggregated data for a population, it is impossible to determine if important subpopulations are being under- or over-represented. The HMN (2008) discusses five key variables for data disaggregation: age, sex, socio-economic status, region and ethnicity. Page 67

70 Mortality and morbidity indictors were classified as being disaggregated by four main variables: age group; ethnicity; gender; and region (socio-economic status was not used in any of the reports analysed). While a small number of indicators were disaggregated by other factors (such as month), these were not included in the final analysis due to their limited number. The disaggregation groups were non-exclusive, in that one indicator could be disaggregated by more than one variable Interpretability In order for information to be used appropriately, users need to have an understanding of the properties of the data it was derived from, such as the underlying concepts, variables, classifications, and methods of collection and processing (CIHI 2009). This refers to the concept of meta-data, or data about data (HMN 2008: 53). In providing information about what has been measured, how it was measured, and how well it was measured, users can have confidence in the quality of the results and it is less likely that information will be misunderstood or misused (Brackstone 1999). Interpretability was assessed via two concepts: if the data was sourced (referenced where the data came from, i.e. hospital data or civil registration) and if the data had any meta-data notes (explanation on how indicators were produced or measured, i.e. rates per 1,000 population). Again, only data in tables related to health status (mortality and morbidity) were assessed for interpretability. Page 68

71 4.4.4 Presentation The layout of data has a significant influence on its use, including the order data is presented, the location on a page or screen and the use of directions and labels (Elvers & Rosn 1997). Overall, presentation is important in ensuring quality: it is vital that data presented is clear and concise, and that users are not left with any doubt over what the figures represent (WPRO 2003). Youmans (cited WPRO 2003) further explains that tables are the simplest means of summarising a set of observations, while figures and graphs provide a pictorial representation of numerical data and are the best medium for a quick review of the relationship between variables. However it is important that data in tables and figures is appropriately presented, and this includes a number of factors such as having an appropriate title, column and stub headings and totals (for tables), and use of scales and axis titles for figures. In recognition of the importance of presentation in the overall quality of data, one-quarter of the combined number of tables and figures were randomly selected from each report to be reviewed (Table 3). Table 3 Sample size for presentation analysis, based on number of tables and figures for each country and Annual Report by year Country Annual Report Tables & figures (N) Sample size (25%) Cook Islands Fiji Tonga TOTAL sampled 253 Page 69

72 All of the tables and figures were listed in an Excel spreadsheet, in the order they appeared through each report, and numbered starting from one. Using the =RAND() function in Excel (which assigns a random six-digit number next to each cell) the tables and figures were then sorted according to their random number. Counting down from the first cell, A1, samples were then populated for each Annual Report. Once the sample of tables and figures was decided, they were assessed via the essential components of a table or graph, as described by Youmans (cited WHO 2003). Each table or figure was assigned either a pass, fail or not applicable to each of the following criteria: Title this must explain in simple words what is contained in the table or figure (a common method of assessment here was to ask; would this table/figure make sense if it was presented on its own) Source the source for any data should be identified in a footnote or other appropriate means Legible the data must be clear (i.e. font size, use of different line styles to differentiate between groups) Stub-headings (table only) the title or heading of the first column. Must be there and make sense (i.e. having a stub-heading of years, however then showing months would get a fail ) Column headings (table only) the headings or titles for the columns. Must be there and make sense Totals (table only) totals must be there and add up correctly Page 70

73 Legend or key (figure only) when several variables are included on the same graph, i.e. males and females, it is necessary to identify each by using a legend or key. Must be there and make sense Scale options (figure only) placed on both axes to identify the values clearly (also referred to as axis headings). Presentation score A score of 1 was given for each passed criteria; 0 for each failed; and those criteria that were not applicable (such as a figure only criterion when assessing a table) were not taken into consideration. Average scores were calculated by dividing the total number of tables or figures that passed each criterion, by the total number of tables or figures that passed plus the total number that failed. An average score was calculated for each of the eight different criteria. These were then averaged to produce the overall presentation score for each country Timeliness Available statistics should be up-to-date (Elvers & Rosn 1997): the production of accurate information on relevant topics is of no use if it arrives after decisions have been made. Timeliness then, refers to the length of time between the reference point information is related to (such as a specific day or year) and when it is made available (Brackstone 1999; HMN 2008; CIHI 2009). While timeliness varies with other considerations, such as the frequency of measurement and demand for completeness, the quality of data that are released several years after collection is severely limited, as they have no use in decisionmaking or resource-allocation (WHO 2007). Page 71

74 Two concepts were developed to assess the timeliness of data in the reports reviewed: 1. Production time refers to the delay (in months) from the stated year of the Annual Report, to the stated publication date. For example, a 2006 Annual Report that was published in October 2007 has a production time of 10 months 2. Reference time refers to the age of the data within the report. Data was classified as having a reference time of: (1) year of the report; (2) one year prior to the report; (3) two years prior; (4) three years prior; (5) four years prior; (6) five years prior; (7) six to 10 years prior; and (8) more than 11 years prior to the year of the report. Page 72

75 5 Results Chapter five provides an overview of the results full details are available as appendices at the end of this dissertation and provide results by country. Statistical tables of the descriptive and analytical reviews for each country are provided in the CD attached to the inside cover. Section 5.1 describes the main results of the descriptive review, including significant commonalities and differences between each country in terms of report structure and data characteristics. Similarly, results of the analytical review are provided in Section 5.2, with country comparisons for each of the five quality dimensions assessed (comparability, disaggregation, interpretability, presentation and timeliness). 5.1 Descriptive review As will be discussed below, results from the descriptive review highlight the diversity of reporting practices occurring throughout the Pacific Region. The number of pages, tables and figures fluctuates dramatically between countries and over time (Section 5.1.1); the stated potential audience, purpose and use of reports vary considerably; and this is most likely a result of the equally varied legislative reporting requirements. As will be discussed, none of the countries achieved the WHO s best practice level of producing a type 3 report, due to the large amount of unanalysed data presented. The number of indicators presented varied significantly between countries and over time, and most reports focused on health system and health status data, as discussed in Section Page 73

76 5.1.1 Report structure While the number of pages contained in Annual Reports produced by the Cook Islands has remained fairly stable over the three years analysed, both Fiji and Tonga have seen a steady increase in the total number of pages in their reports, with Tonga reaching a peak of 160 pages in After taking into consideration the missing pages from the 2001 report, the number of tables and figures presented in the Cook Islands has also remained fairly stable, at around 68 per report. The number of tables and figures rose and peaked in 2005 for Fiji (80) and in 2006 for Tonga (130), and has remained fairly stable from then on. Overall, while there are a number of different sections both between countries and over time within the same country, the structure of Annual Reports can be grouped into four broad themes: (1) population characteristics and the determinants of health; (2) health system performance; (3) morbidity; and (4) mortality. Audience, purpose and use The Cook Islands and Fiji both define their respective audience, purpose and use at the beginning of each report. Overall, the purpose of Annual Reports from the Cook Islands is to provide key health statistical information for the Cook Islands for both the Ministry of Health and other people interested in the health needs and priorities of Cook Islanders. Over the five reports analysed, Fiji defined a number of different purposes of their Annual Report, including: providing a summary of the Ministry s major activities and performances; highlighting performance in delivering services; illustrating the effort, commitment and achievements of staff; providing a summary of occurrence of vital events; and showcasing the roles and functions of Mataika House and the Paediatric Unit. The potential audience, purpose and use of Annual Reports from Tonga were not specifically mentioned in any of the reports Page 74

77 reviewed between 2004 and However, reports from 2007 and 2008 have been structured to address the major objectives of the Strategic Development Plan VII, thus providing a general reporting purpose (monitoring progress of the plan). Legislative requirements There is a mixture of legislative requirements governing the production and dissemination of Annual Reports in the Pacific. While there is no specific legislation dedicated to the development and dissemination of Annual Reports in the Cook Islands, both Fiji and Tonga are required to report on set topics at set intervals as prescribed in a number of different Acts and Regulations. Furthermore, all three countries were subject to mandatory reporting as set out in their Public Health Act (or equivalent), which sets a prescribed number of notifiable diseases required for reporting. The influence of these Acts is clear on the contents of each report, with each country consistently reporting on the majority of notifiable diseases. Report type Using standards as set out by the World Health Organization in their report on Country Best Practices for Reporting (2007), the Cook Islands Annual Reports can be classified as type 1 : reports characterized by a large amount of crude or raw data; simple tabulations; limited explanations of definitions of variables; and a mixture of rates, ratios and absolute numbers. Fiji and Tonga represent the next level up in complexity: including both crude or raw data and basic summary statistics and analyses. None of the reports were at the highest level ( type 3 ), which are characterised by presenting policy and program implications with the data, and are suitable for non-technical audiences. Page 75

78 5.1.2 Data characteristics Domain of measurement In general, very little data contained within Annual Reports from the Pacific is related to describing the determinants of health (Figure 10). In both Fiji and Tonga, over half of all data contained within tables and figures is related to measuring aspects of health system performance, such as inputs, outputs and outcomes. While reports from the Cook Islands have a large number of tables and figures related to health status, this only makes up around onethird of all data in Annual Reports from Fiji and Tonga. 70 Tables and figures by domain of measurement: Cook Islands, Fiji and Tonga Tables and figures (%) Cook Islands Fiji Tonga 0 Determinants of Health Health System Health Status Domain of measurement Figure 10 Comparative distribution of data (contained in tables and figures) by domain of measurement, Cook Islands, Fiji and Tonga, all reports Indicators presented In terms of the absolute number of indicators presented over all reports analysed, Tonga produced almost double that compared to the Cook Islands and Fiji (1,019 compared to 538 and 456 respectively). Page 76

79 Determinants of health As shown in Figure 11, on average 5% of all indicators presented across all reports and countries were related to describing the determinants of health. Very few indicators were related to environmental and behavioural risk factors, with the data tending to focus on aspects such as population (by age, gender and region); population growth; crude birth rate and characteristics of mothers giving birth (i.e. age, number of previous children or pregnancies). Health system Compared to the Cook Islands and Tonga; Fiji s Annual Reports contained the greatest number of indicators related to the health system, at just under 30% of the total. Overall, both the Cook Islands and Tonga primarily focused on output indicators, followed by outcomes and then inputs. In contrast, Fiji primarily focused on outcome indicators, followed by inputs and then outputs. Health status The vast majority of indicators presented over all 13 reports analysed for the Cook Islands, Fiji and Tonga were related to describing health status, such as mortality and morbidity. No less than 65% of the total indicators presented for all reports and countries are related to health status measures. Within this domain, the majority of indicators were mortality-related and a mixture of individual cause-of-death (such as haemorrhage due to childbirth ) and aggregatelevel (such as neoplasms ) measures. Page 77

80 Indicators by domain of measurement: Cook Islands, Fiji and Tonga Indicators (%) Determinants of Health Health System Health Status Domain of measurement Cook Islands Fiji Tonga Figure 11 Relative distribution of indicators by domain of measurement, Cook Islands, Fiji and Tonga, all reports 5.2 Analytical review This section outlines the comparative results for each of the countries in terms of the five specific quality criteria. Section outlines the results for comparability, an extremely important aspect of data quality, and it shows the limited comparability of data contained within Annual Reports from the Pacific. Results for disaggregation are provided in Section and they highlight the variation in disaggregation variables used across the three countries. As shown in Section 5.2.3, results for interpretability were generally poor, with few countries providing sources or meta-data for health system data. Section highlights the different presentation scores achieved for the Cook Islands, Fiji and Tonga, which were all relatively high. And finally, results on timeliness are discussed in Section Page 78

81 5.2.1 Comparability Overall, Fiji and Tonga scored very low indicator-comparability scores (less than 0.3 across all domains) (Figure 12). In effect, this has resulted in large variances in the type and number of indicators presented each year, producing Annual Reports that appear fragmented and haphazard. The main issue affecting data comparability across all three countries is that of indicator selection. That is, most indicators have low comparability as they are only presented for one or two years, and then are not presented again. Indicator stability is the other cause of low comparability and it arises due to changes in the presentation or definition of indicators between reports. Countries often switch, for example, between presenting data as absolute numbers or rates; high-level cause-of-death categories (i.e. infectious and parasitic diseases) to individual cause-of-death (i.e. tuberculosis); and between disaggregated or aggregated data. The clear outlier is that of the Cook Islands, and it is unique for two reasons. Firstly, due to the limited number of reports reviewed (three compared to five each for Fiji and Tonga) and the missing pages in the 2001 report, the very real possibility exists that the data are biased and have produced an overly-inflated indicator-comparison score. However, in the 2005 and 2007 reports (which were complete) the structure and contents were virtually identical: including the choice of indicators. This suggests that the relatively high indicator-comparability score of the Cook Islands (overall score: 0.6) might be reflecting the true situation in terms of data comparability. Page 79

82 Indicator-comparability score Indicator-comparability score by domain of measurement: Cook Islands, Fiji and Tonga Determinants of Health Health System Domain of measurement Health Status Cook Islands Fiji Tonga Figure 12 Indicator-comparability score by domain of measurement, Cook Islands, Fiji and Tonga, all reports Disaggregation Overall, only 29% of Fiji s health status data was disaggregated, well behind the Cook Islands (72%) and Tonga (73%) (Figure 13). In terms of individual disaggregation variables, there appears to be little overall similarities in the region. While ethnicity was the primary disaggregation variable of interest in Fiji; no data from the Cook Islands or Tonga were presented by ethnic group. Age group was a common variable for the Cook Islands and Tonga, but not for Fiji. A large number of health status indicators in Fiji and Tonga were disaggregated by gender, while only a limited number from the Cook Islands were. And finally, very few indicators were disaggregated by region for all three countries. Page 80

83 Indicators by disaggregation variable: Cook Islands, Fiji and Tonga Indicators (%) None Age Group Ethnicity Gender Region Disaggregation variable Cook Islands Fiji Tonga Figure 13 Distribution of indicators disaggregated by variable of interest, Cook Islands, Fiji and Tonga, all reports Interpretability While results for interpretability varied between country and the two aspects of interest (source or meta-data), generally speaking, only around 10% of all tables and figures with health status data from the three countries had an acceptable level of interpretability. There are two clear outliers to these generally poor results, as shown in Figure 14 and Figure 15. Firstly, Tonga scored exceptionally well for interpretability in relation to the number of tables and figures that were appropriately sourced (over 50% of all tables and figures were sourced). This is most likely a result of the review undertaken of their reporting practices in 2005, which recommended improving the current levels of sourcing provided. Secondly, the Cook Islands provided meta-data descriptions for 47% of their tables and figures containing mortality data much higher than the level provided for Fiji and Tonga. Page 81

86 5.2.5 Timeliness Production time The production time for Tonga s Annual Reports was not able to be calculated due to insufficient data. While production time appears to be decreasing for both the Cook Islands and Fiji (from ten months to seven; and ten months to one respectively) due to limited data for these countries, it is difficult to conclude on any trends. Reference time Overall, the reference time for health status indicators is very similar for Fiji and Tonga, with the majority of indicators representing data from the year of the report (Figure 17). The pattern is quite different for the Cook Islands, with an almost equal number of indicators in each reference time group, and a gradual decline from recent to older data. Indicators (%) Reference time of health status indicators: Cook Islands, Fiji and Tonga Year of Report 1 Year Prior 2 Years Prior 3 Years Prior 4 Years Prior Reference Time 5 Years Prior 6-10 Years Prior 11+ Years Prior Cook Islands Fiji Tonga Figure 17 Reference time (age of data) of health status indicators, Cook Islands, Fiji and Tonga, all reports Page 84

87 6 Discussion There are two stories to be told here. The first is the story of Annual Reports in the Pacific: what is presented in them; what they look like; and what they can tell us. Overall, while many differences between Annual Reports were found, there are also striking similarities. A key finding here is the vital role played by legislative reporting requirements, and the impact they have on the content (and quality) of Annual Reports. Secondly is the story of quality: while this research was interested in measuring five distinct dimensions, report purpose has emerged as an important overarching factor in determining the quality of reports. Comparability also emerged as the single-most important dimension of quality, due to the impact it has on the remaining quality dimensions and the potential use and usefulness of the report itself. 6.1 Annual Reports in the Pacific Annual Reports from the Pacific present us with countries whose health information systems are data rich, but information poor. Each report has anywhere between 25 and 80 tables, 50 to 160 pages, and in most cases, hundreds of indicators: effectively making it hard to understand or even see the information from the data. As remarked earlier, countries of the Pacific are truly running the risk of drowning in numbers as they swim through the vast sea of data they continue to produce (Mudde & Schedler 2010: 411). Very few of the tables or figures are referred to in the text, with countries opting to place a significant amount of tables as appendices, with no attempt at linking them into the main report itself. Furthermore, most of the text is descriptive in nature and lacks any critical reflection or analysis on what the data is showing. There are, for example, few occasions when reports compare data over time or between groups, or in light of government policy or objectives; highlighting the apparent lack Page 85

88 of appreciation among data producers that data alone means very little, and that it is only through context, comparison, and explanation that it can begin to tell its story. In both Fiji and Tonga there is a strong emphasis for reporting on measures of health system performance: over half of all tables and figures were based on data related to system inputs, outputs and outcomes. The pattern of reporting for the Cook Islands is different, with over half of the data in their reports relating to measures of health status. Overall however, a common theme linking these countries is the presentation of data as individual facts : there is little or no attempt at linking objectives to inputs, outputs, outcomes and health status. Rather than being presented with information on the flow of resources through the health system, users are presented with segregated data on, for example, number of bed days available, number of surgical procedures performed, and number of surgical-related infections, with no appreciation of the overall picture of what is happening. While the data in Annual Reports may be able to provide a number of key indicators on the health system or health status, as it stands, they can only provide a snapshot of performance for the year in question. Furthermore, this snapshot only skims the surface of the bigger picture. The majority of health status indicators that are presented consistently are high-level aggregate measures of mortality or morbidity (such as top-ten causes ), thus providing no information on differences between age groups, gender or regions. Due to the heavy reliance on civil registration and hospital administrative systems, most of the data within Annual Reports can only inform us about the people in contact with health services, and very little attempts have been made at assessing the burden of disease among those not engaged with the system. This reliance on systems-based data is also apparent in the paucity of information related to the socioeconomic Page 86

89 or demographic factors impacting on health, including environmental and behavioural risk factors. Information such as this is crucial in the management, planning and implementation of health services, yet very little data within Annual Reports is dedicated to this topic Legislative requirements The legislative requirements and stated potential audience, purpose and use has a direct impact on report content, and this is clearly demonstrated in Tonga. Overall, it would seem that Tonga is suffering from its own success. While there are three legislative acts regarding the development and dissemination of Annual Reports; the guidelines concerning report content are overly broad and ambitious. In the Health Services Act of 1999, for example, under the heading Annual Reports, it states that,... and if the Legislative Assembly shall wish to know anything concerning the department of any minister he shall answer all questions put to him... and report everything in connection with his department (emphasis added). This broad requirement to answer all questions on anything the Legislative Assembly wishes to know, and report on everything in connection with their department, may very well explain why Tonga produces the largest reports (up to 160 pages) with an enormous amount of indicators (over 1,000 in the five years analysed). It may also explain why indicators range from seemingly unimportant measures of health system performance, including boiler fuel consumption and the number of transport drivers; to everything in-between, including the number of wound dressings applied and number of pharmaceutical items dispensed; to what could be regarded as exceptionally relevant measures of population health and system outcomes, such as infant mortality, immunisation coverage and service utilisation. Page 87

90 Furthermore, all three countries had Public Health Acts that defined notifiable and dangerous diseases for mandatory reporting. In all three countries, these notifiable diseases represented the majority, if not all, of indicators that were consistently reported year-to-year. It is unsurprising that legislative requirements play such a formative role in defining what is presented in an Annual Report; what an Annual Report looks like; and what it tells us. What is surprising is the lack of attention to updating legislation, especially in relation to health, which has undergone dramatic changes recently as Pacific Island Countries and Territories have entered into the demographic transition. Overall, it would appear that Annual Reports are regarded as a means of satisfying legislated reporting requirements, and their potentially broader role in guiding evidence-based decision making, or use in monitoring and evaluating national and international strategies, is not being fully realised. 6.2 Quality Overall, this research has found that the quality of data produced from HIS in the Pacific, as presented in Annual Reports, is poor. It is poor due to the limited success in each of the five quality dimensions assessed. However it is also poor due to issues within systems themselves and issues related to the production of Annual Reports, namely, clarification of report purpose. Results from the WHO (2007) workshop, one of the only major initiatives aimed at systematically reviewing Annual Reports identified in this research, provide similar findings. From their analysis of 13 country reports, they generated four common challenges: (1) quality; (2) comprehensiveness; (3) the use of standards; and (4) the ability to cater to a wide range of audiences. Page 88

91 Of the five quality dimensions assessed, some of the worst results were related to comparability. Much of the data presented in Annual Reports has limited meaning, as it cannot be compared over time or space. The inconsistent choice of indicators provide a fragmented picture of health system performance, and (as demonstrated in Appendices four to 15) a patchwork of different indicators presented for a varied number of years. As well as varied indicator selection, there are also issues of indicator stability a concept closely related to the quality dimension of interpretability. Most of the data is presented without reference to its source or with any meta-data explanations, making comparisons over time and between countries difficult. While the level of disaggregation of indicators was generally acceptable, this also changed over time and space, making comparisons even more challenging. Presentation is the one quality dimension that scored relatively well, however there remains ample room for improvement, including reducing the use of large, cumbersome tables and providing more user-friendly methods of presenting data, such as simple tables and figures. The final quality dimension assessed was timeliness, and while all countries showed signs of improvement, the delay between reporting period and report publication is still a major limiting factor in the usefulness of Annual Reports. Reference-times calculated for each country showed the majority of data was from the reporting period, with a gradual decline in the presentation of data as it aged. Results from this aspect of the review also support the findings in relation to comparability. Due to large fluctuations in the number and type of indicators presented each year, it is not surprising that most data is related to the reporting period: if data were being presented consistently, then the reference-time pattern would change, with a relatively equal number of data for each time-frame. Page 89

92 All of this is, however, not purely a problem of the reports, but rather a problem of health information systems themselves. As has been discussed previously, information systems in the Pacific, and the data they produce, have a number of inherent problems including fragmentation and inadequate resourcing. It should not come as a surprise then, that Annual Reports are also affected by those same issues, as they are a product of HIS. In his review of six independent health care systems in the Pacific in 1990, Taylor, for example, found that death registration systems were often inaccurate and incomplete; not disaggregated by age, sex or ethnicity; difficult to compare due to differences in coding; and most presented mortality data on the top-ten causes of death, if at all. He further commented on how health care systems were usually defined in terms of personnel, facilities and equipment, and the number of patients processed and resources consumed. Such results are clearly replicated here: highlighting that little has changed for HIS or Annual Reports in the 20 years since his research Comparability Of the five dimensions assessed, issues related to comparability deserve special attention due to their impact on the remaining quality dimensions. Both Fiji and Tonga scored alarmingly low indicator-comparability scores, due to large fluctuations in the number and type of indicators presented each year. As well as severely limiting comparability over time and space, such fluctuations present us with reports that contain information that is both seemingly haphazard and fragmented. Issues relating to indicator selection are the primary cause of this limited comparability, and arise due to a number of factors, including the lack of a minimum data-set for reporting. Core indicators form the backbone of health information systems and they need to reflect changes over time, while being valid, reliable, specific, sensitive and feasible to measure (HMN 2008). They also need to be relevant and useful for decision- Page 90

93 making, and as such, reviewed: an area in desperate need of improvement for Annual Reports in the Pacific. A large number of indicators within Annual Reports are linked to early Public Health and Notifiable Disease Acts, which list specific diseases for monitoring and reporting. While a number of recent diseases, such as HIV/AIDS, have been added to the lists, a large number of indicators with dubious present-day importance remain (for example, conjunctivitis). The continued inclusion of such indicators needs to be assessed, despite our intuitive fears over removing anything of potential importance, before reporting requirements expand further beyond the capacity of current HIS. A natural question that arises here is, if there are processes in place for validating the inclusion of indicators into national reporting requirements, are there processes for their exclusion? A second factor linked to the issue of limited comparability is the role of the international community, and the need for countries to report on global health agreements such as the Millennium Development Goals (MDGs). There has been a clear shift in Fiji, for example, to include MDGs as a core component of their Annual Reports. Arguments over the utility of MDGs aside, it is vital that by incorporating global reporting requirements into national reports, countries do not lose sight of important local health issues. While all three countries have devoted a significant amount of their reports to information on non-communicable diseases, infectious and parasitic diseases remain the top cause of morbidity. This highlights that while an appreciation of the global shifts in patterns of mortality and morbidity are important, countries must still be aware of the realities of their local conditions. Influences of the international community are also seen in other aspects of Annual Reports, such as in the Page 91

94 massive increase in reporting on neoplasms during 2004 and 2005, due to an international survey, followed by a relative dearth of cancer-related indicators Report purpose An overarching theme that has emerged from this research (one that affects both the content and quality of Annual Reports) is the need for clarification on report purpose. The cause of many issues with the reports is that they have no clear idea of their potential audience, use or purpose, and this is translated into long and complex reports providing poor quality information on an overly wide range of topics. In 2008 the HMN proposed a framework linking information needs and tools at various levels of collection within the health system (Figure 18). The reasoning behind this was the need to identify the different types of data needed for management, disease control and response, strategic decision making and policy development, and produce information accordingly. Figure 18 Information needs and tools at different levels of data collection (HMN 2008) Page 92

95 While this framework is overly simplistic in assuming that information required for health system policy development is simply a summation of data from lower-levels of the system, it does highlight a salient point for countries producing Annual Reports: they must be clear about what data needs reporting on, and for what purpose. Annual Reports from the Pacific are exceptionally broad in their attempt to provide a summary of everything of potential value related to the health system. This over-ambitious approach has resulted in poor quality reports containing high-level population health indicators such as infant mortality rate; down to individual facility measures of outputs, including the number of telephone calls received and loads of washing performed. 6.3 Limitations There are a number of limitations to this research. Data from the Cook Islands was limited due to two important factors: the missing pages from the 2001 report, and the overall limited availability of reports (only reports from 2001, 2005 and 2007 were sourced). It is disappointing that the missing pages from 2001 were not discovered until late in the research process, otherwise a different country would have been used in substitution. Overall, it is possible that the missing pages and limited number of reports have biased the results for the Cook Islands, making comparisons over time difficult. The quality dimension of presentation and its specific measurement of presentation score offered little differentiation between the countries, highlighting the limited use of the measure. Furthermore, in breaking presentation down into eight different criteria, the overall essence of the presentation of a table or figure was missed. A table with an appropriate title, stub- and column-headings, for example, may also be confusing, spread out over multiple Page 93

96 pages and generally hard to follow. While such a table would score highly in its specific presentation criteria, this does not mean the table was presented in the best way, was appropriate, or easy to understand. A more subjective method of assessing presentation may offer more illuminating results, and this could include asking users to rank tables and figures in regards to their ease of use, understandability, or overall presentation. Due to time limitations, a number of quality dimensions were only assessed for health status (mortality and morbidity) data, and all quality dimensions were only assessed in terms of the data contained within tables and figures. While health status information often formed the bulk of Annual Reports, and the majority of data was presented in tables, in effect this has limited the results of the study as large parts of reports were not analysed. This research project was initially planned to assess Annual Reports from five different Pacific Island Countries, however it became clear soon after beginning work on the project, that this was not a feasible goal within the time limitations. While the three countries used have provided interesting and useful comparisons, it would have been preferential to expand the number of countries reviewed to further strengthen the research findings. And finally, due to the broad scope of the research questions, there was limited time for refining the methodological approach in assessing the quality dimensions. A better approach for future research projects may be to limit the scope to one or two quality dimensions for assessment, and dedicate more time to experimenting and refining approaches to the assessment of quality. In taking such an approach, issues such as those encountered with the dimension of presentation would be minimized, and the overall research would have a higher level of academic rigor and assurance surrounding its methods. Page 94

97 7 Recommendations Overall, this research has generated four main recommendations for further action and research. The first is concerned with an immediate approach to improve the quality of Annual Reports through the relatively simple method of splitting the report into three smaller subreports. The next three require a more systematic and invested approach to the review and improvement of Annual Reports, and represent long-term goals for the Pacific. 7.1 Development of sub-reports An initial step for improving the quality of Annual Reports could be to separate them into a number of different sub-reports. While it is undoubtedly useful for a senior decision maker to know how many infants are dying in their country, and if this pattern is changing over time or between regions, it is unlikely the operations manager would need to know this information, for example. They would be more interested in operational outputs such as washing demands or the number of admissions in order to efficiently run their facility. While each piece of information may be relevant to one group, that same information may be highly irrelevant to the other, producing a report that is not only excessively long, but also ill-suited for meeting the needs of its different audiences. Annual Reports in the Pacific would benefit immensely from having a clearer idea of who their audience is and what their information needs are. This could lead to the division and separation of reports into smaller sub-reports, such as those dedicated to the running of facilities; those with high-level information for decision making and planning; and those with general health status information for the public. Dividing the reports into three distinct sections following the lines of the Domains of Measurement could also offer a conceptually sound approach to this. Page 95

98 This sentiment reiterates those expressed during the reform process of health system reporting in the 1990s in England, as highlighted in Watt et al (1993) and their discussion on the need for evaluating Annual Reports against set national criteria. While they are quick to point out that this does not equate to rigid dictations of the content and format of reports, they argue that providing set content criteria is a vital step forward in developing the shared agreement of the purpose and objectives of Annual Reports. Fulop and McKee (1996) also discuss the clear problems in producing a document with such a wide audience, and the need for better targeting of information in reports. While it is true that most reports are disadvantaged due to their need to cater for a number of varied audiences, questions such as, are the reports producing the type of information required by healthcare facilities and Governments, are the reports being used and how much of the reports is not used (WHO 2003), provide useful starting points for the development of sub-reports. Providing smaller, more frequent reports could improve also timeliness and relevance, and potentially, have a more impact on strategic decision making and planning. 7.2 Development of data quality assessment tools One of the main utilities of this research is in the development of specific methods for assessing the quality of data presented in Annual Reports. While a number of tools have been developed for assessing national health information (HMN 2008a) and vital statistics systems (Mikkelsen & Lopez 2009), few have been developed for assessing the information products of HIS. This apparent niche is further complicated by the wealth of quality dimensions presented in the literature, often with limited practical advice on how to measure comparability, usefulness or comprehensiveness, for example. As such it is strongly recommended that further research is dedicated to developing a tool for assessing the quality of data within Page 96

99 Annual Reports. Part of this tool may involve prioritising the most important aspects of quality based on recommendations from the literature, or from areas in need of improvement as identified by previous research in the Pacific. Creating a tool that provides operational definitions of quality and how to measure the concept without the need for external consultants or expertise would give countries in the Pacific Region the means to apply the tool themselves, thus building local capacity. Research on the methods for measurement is also required. As demonstrated here, the utility of certain dimensions (such as presentation) offered limited practical guidance on the assessment of data quality, while comparability offered a wealth of information, as well as offering potential areas for improvement. More research needs to be done on refining the methods for quality assessments, and building further on the work here. 7.3 Development of regional reporting templates Research on the development of a regional Annual Report template, including a minimum data-set and the use of standard data definitions, would be of exceptional benefit to HISstrengthening in the Pacific. In providing a standardised template, the production time of reports should be reduced, while aspects of quality such as presentation and interpretability should increase dramatically. A standardised report would enhance comparability over time and between countries, and present a stronger, untied Pacific voice in terms of emerging trends for the region. Furthermore, in providing a minimum data-set with standard definitions and reporting requirements, countries still have the option of reporting more than what is required; however a minimum amount of data for comparison will be guaranteed. There is also the option of presenting specific annual themes, such as on non-communicable diseases or sexually transmitted infections, to highlight topics of interest in greater detail. Page 97

100 As part of such a template, Excel spreadsheets with formulae already inserted and clear instructions on how to input data could be developed. Such spreadsheets could calculate simple three- or five-year moving averages of indicators that are sensitive to small populations (such as the infant mortality rate and maternal mortality ratio) and also produce basic graphs for presentation. This would again increase the comparability of data, and also strengthen the reputation of data generated from the Pacific, which is often unfairly regarded as unreliable or obsolete, when large fluctuations in reported figures for certain indicators are a product of small population sizes and not a product of poor quality data. In providing templates for both the production and presentation of data, the timeliness of reports could also be improved, as data can be added monthly or quarterly, rather than at the end of each year. 7.4 Development of a minimum data-set A final recommendation is the development of a minimum data-set. As the primary purpose of recording data is for communication (WPRO 2003); the process of standardisation, including the development of minimum data-sets and dictionaries, is vital as it ensures communication across time and space. While indicators may form the backbone of any HIS, a nationally defined minimum set of indicators for national program planning, monitoring and evaluation is the ultimate goal (HMN 2008). This should include core indicators that reflect changing needs over time, based on the epidemiological profile and development needs of countries, as well as being able to monitor local and national priorities, while meeting international technology standards and linked to key international initiatives such as the MDGs, GFATM and GAVI. However the challenge is to keep the minimum set small and based on a specific framework for selection. While a number of countries still struggle with this, PAHO (2007) have produced a compendium of over 100 indicators, along with their definitions and methods of Page 98

101 measurement, that may provide a solid reference point for any future reviews of Annual Reports in the Pacific. In his review of HIS reform in South Africa, Shaw (2005) also remarks on the importance of an essential data-set of clearly defined indicators for monitoring and evaluating services, which is integrated into an overall coherent data system. He also states the need for such reporting requirements to be able to change over time in response to the changing needs of data users and the importance of asking questions such as why do we want to collect this information and how will we use it (Shaw 2005: 638). Overall, a key lesson for Pacific Island Countries and Territories is to limit the number of reporting requirements and integrate them. Page 99

102 8 Conclusions Results from this research will generate discussion and debate on the role of Annual Reports, and reporting in general, among producers and users of data in the Pacific. The practice of collecting information for the sake of collection, with little reflection on why the information is important and how it could be used, must be changed. Collection is a means to an end: it should not be an end to itself. As such, reflection on what Health Ministries or local Governments want from a HIS product, such as an Annual Report, is needed. Clarification on the purpose of reports is vital: is their purpose to highlight what is making the population sick; or who has been admitted to hospital; or what interventions are being done to help; or what the state of health services are? At the moment, reports provide an excessively broad range of individual facts on activities from all levels and facets of the health system, with little attempt at transforming the data into useful information and knowledge for action As discussed previously, data contained within Annual Reports could play a vital role in health; from providing an evidence-base for use in strategic decision making, to monitoring the trends in population health, and evaluating the impact of interventions. However, due to longstanding issues of quality, HIS in the Pacific and the data they produce are often regarded with suspicion and simply not used. While this research has highlighted the limited quality of data within Annual Reports as they currently stand, many Pacific Island Countries and Territories have taken steps to improve the quality of their reports and it is now up to the international community to provide them with the necessary tools and capacity to strengthen their HIS, rather than continuing to rely on externally produced estimates and models. Page 100

104 Creswell J Research design: Qualitative, quantitative and mixed-method approaches. Sage Publications: California. Davies P, A Aumua, N Hodge, A Malik, Y Lee and L Skiller Determining the health information needs of senior decision makers. HIS Knowledge Hub [unpublished working paper]. Department of Health and Ageing (DOHA) Department of Health and Ageing Annual Report. Department of Health and Ageing, Australian Government. Available at [Accessed 12 November 2010]. Duran-Arenas L, C Rivero, S Canton, R Rodriguez, F Franco, R Luna, and J Catino The development of a quality information system: a case study of Mexico. Health Policy and Planning 13(4): Elvers E, and B Rosn Quality concept for official statistics (pp ). Encyclopaedia of Statistical Sciences, Update Volume 3. Wiley-Interscience: New York, United States of America. Finau S National health information systems in the Pacific Islands: in search of a future. Health Policy and Planning 9(2): Fullop N and M McKee What impact to annual public reports have? Public Health 110: General Data Dissemination System (GDDS) The General Data Dissemination System: The GDDS Document. Available at [Accessed 9 February 2010]. Gillham B Real world research: Case study research methods. MPG Books Ltd: Cornwall. Goddard, M., A. Preker, and P. Smith Priority setting in health a political economy perspective. Health Economics, Policy and Law 1: Government of Fiji Public Service Act. Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of the Cook Islands Births and Deaths Registration Act. Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of the Cook Islands Public Service Act. Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of the Cook Islands Public Health Act (amended). Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of Tonga The Constitution of Tonga. Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of Tonga Health Services Act. Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Government of Tonga Public Health Act 1992 (amended). Pacific Islands Legal Information Institute (PacLII). Available at [Accessed 17 June 2010]. Haberkorn G A sea of islands a myriad of indicators: on the interface between demography and planning in the Pacific Islands. Asia Pacific Viewpoint 38(3): Health Metrics Network (HMN) Framework and Standards for Country Health Information Systems, 2 nd Edition. World Health Organization: Geneva. Health Metrics Network (HMN). 2008a. Assessing the National Health Information System: An Assessment Tool, Version World Health Organization: Geneva. Holdford D Content analysis methods for conducting research in social and administrative pharmacy. Research in Social and Administrative Pharmacy 4(2): Hong K, A Cum Laude and A Huibin Towards a comprehensive analysis of health system reform in the Asia-Pacific region. Asia-Pacific Journal of Public Health 14(1): Page 102

109 Appendix 1 Cook Island Annual Reports 2001, 2005 & 2007 This appendix contains results of the descriptive and analytical review of three Annual Reports produced from the Cook Islands dated 2001, 2005 and Section A1.1 and A1.2 provide the descriptive review results, including report structure, data characteristics and indicators presented. Section A1.3 covers the results from the analytical review, including the five quality dimensions data was measured against. The remaining appendices are located either on the CD attached to the inside cover (printed version); or online in the separate document, Hodge_Transforming data into information and knowledge_appendices. Appendix 4 is a table of determinants of health indicators by domain sub-group, indicator title, unit, disaggregation and year. The table for health system data is found in Appendix 5. Appendices 6 and 7 provide health status indicators by domain subgroup (mortality or morbidity) and indicator group (infant and child, maternal or general). A1 Descriptive review A1.1 Report structure As demonstrated in Table 4, the structure of Annual Reports has remained constant over the three reports analysed, with only minor fluctuations in the total number of pages, sections, tables and figures. While there is a large number of different sections (approximately 20 per report), they can be grouped into the following four broad themes (topics after each theme are headings from the reports): Population characteristics population; vital statistics; births; family planning; immunisation Health system performance health facility; hospital indices; outpatient; laboratory services; x-ray services; surgical operation; physiotherapy services Morbidity general morbidity; notifiable diseases; malignant neoplasm; noncommunicable diseases; fish poisoning; measles; motor vehicle traffic accidents; and Mortality general mortality. Page 107

110 Table 4 Summary statistics, Cook Island Annual Reports 2001, 2005 & 2007 Annual Report Total number of Pages Sections Tables Figures Tables & Figures 2001* 35 (53) Average** * There are 18 pages missing from the copy of the 2001 report analysed ** Due to the missing pages from 2001, averages have only been calculated for 2005 and 2007 A1.1.1 Audience, purpose and use The potential audience, purpose and use of the Cook Islands Annual Statistical Bulletins (referred to hereafter as Annual Reports) are clearly defined in the Preface of each report (see Table 5). Two audiences are specified as main users of the reports: namely the Ministry of Health and others interested in the health needs and priorities of Cook Islanders. The purpose of the report is stated as to provide key health statistical information for the Cook Islands. Furthermore, potential uses of the report include: Assisting in identifying public health needs Formulating future plans Advising the Minister on how needs should be addressed Assessing progress towards health outcome targets Providing information needed to debate health needs and priorities Providing key health indicators for creating frameworks for planners in the public and private sector. Table 5 Preface, Cook Islands Annual Report 2001, 2005 & 2007 [The] Medical Records Unit Annual Statistical Bulletin... provides key health statistical information for the Cook Islands on an annual basis. The information contained in this report will be used by the Ministry of Health (MOH) to assist in identifying public health needs, in formulating future plans and advise (sic) the Minister of Health on how needs should be addressed, and in assessing progress towards the health outcome targets. We believe that this report will also be of interest to others, because it provides information that is needed to debate about health needs and priorities that will benefit the people of the Cook Islands. This bulletin provides key health indicators for creating framework (sic) adopted for our planners both in the public and the private sector. Page 108

111 A1.1.2 Legislative requirements While no specific legislation dedicated to the development and dissemination of Annual Reports for the Cook Islands was located, Part II of the Public Health Act 2004 (notifiable and dangerous conditions) lists a number of diseases that medical practitioners are required to report on to the Director of Health (Table 6). Furthermore, the Public Service Act 1975 and Births and Deaths Registration Act 1973 also include a number of reporting requirements that are likely to have influenced the development of Annual Reports. Table 6 Notifiable conditions, Public Health Act 2004, Cook Islands Acute anterior poliomyelitis AIDs (Acquired Immune Deficiency Syndrome) Cerebro-spinal meningitis (meningococcal) HIV (Human Immune Deficiency Virus) MRSA (methicillin resistant staphylococcus aureus) Tuberculosis: pulmonary and other sites Anthrax Cholera Chickenpox (Varicella) Conjunctivitis Dysentery: Amoebic, Bacillary (Shigellosis), and other types Enteric fevers (Typhoid fever, Paratyphoid fever) Infantile diarrhoea Infectious hepatitis Influenza Asthma Bronchitis Cancer (all varieties) Diabetes mellitus Fish poisoning (ciguatera) Food poisoning Hypertension Otitis media Pneumonia Rheumatic fever Diphtheria Dengue Hepatitis B Measles Mumps Poliomyelitis Tetanus Whooping cough (Pertussis) Gonorrhoea Syphilis Venereal warts Trichomonas Candidiasis Leprosy Measles (Rubella or Morbilli) Ringworm (Tinia Imbricata) Scabies Yellow fever A1.1.3 Report type The Cook Islands Annual Reports can be classified as Type 1. Type 1 reports are characterised by a large amount of crude or raw data; simple tabulations; explanations of definitions of variables; basic description of sources; and the use of rates, ratios and absolute numbers. Overall, the majority of each report is comprised of tables and figures (on average, each page has between two to three tables or figures), with a small amount of explanatory text at the start of each section. The explanatory text provides a brief summary of the main events for that year (i.e. number of births or deaths, number of admissions for hypertension); very little of the text is dedicated to exploring any trends or patterns over time, between groups or regions. Page 109

112 A1.2 Data characteristics A1.2.1 Domain of measurement As shown in Table 7, the majority of data contained within the Cook Island s Annual Reports relates to the health status of the population (mortality and morbidity), followed equally by determinants of health and health system indicators. This pattern of reporting has remained constant over the two comparable reports (2005 and 2007). The vast majority of determinants of health data are related to socioeconomic and demographic factors. Very little data is specific to health system inputs, with the majority of health system data describing system outputs and outcomes. Furthermore, most health status data is related to morbidity. Table 7 Allocation of tables and figures by domain of measurement, Cook Islands Annual Reports 2001, 2005 & 2007 Domain of Measurement Number of tables and figures in each Annual Report Contribution of domain (%) TOTAL Within Between Determinants of health Total % Socioeconomic and demographic factors % - Environmental and behavioural risk factors % - Health system Total % Inputs % - Outputs % - Outcomes % - Health status Total % Mortality % - Morbidity % - All domains TOTAL % In looking at the patterns of reporting over time, there appears to be little difference over the three reports analysed (Figure 19). While there was limited health system data in 2001, the report was incomplete, with a large number of pages missing thus significantly biasing the results. On average, each report is focused primarily health status data (approximately 50% of all data), followed by data on the determinants of health, and then health system. Page 110

113 Contribution of each domain 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Cook Islands: Structure of report by domain of measurement Annual Report Health status Health system Determinants of health Figure 19 Comparative contribution of domains of measurement over time, Cook Islands Annual Reports 2001, 2005 & 2007 A1.2.2 Indicators Presented Determinants of health A limited number of indicators related to the determinants of health were presented over the three reports analysed. The vast majority (over 80%) described socioeconomic and demographic factors impacting on health, and covered indicators such as: adult literacy; total population; annual growth rate; number of births; total fertility rate; rural/urban split; and characteristics of mothers giving birth (including age). Many of the indicators were disaggregated by region and age group. Indicators related to environmental and behavioural risk factors included the proportion of the population with access to excreta disposal, local health services, and safe water; and the number of low birth-weight babies. The pattern of reporting over time has remained constant over the three reports, with no changes in indicators presented on environmental and behavioural risk factors, and minor fluctuations in socioeconomic and demographic factors. Page 111

114 Health system A total of 23 different indicators related to health system data were presented in the three reports analysed, as shown in Table 8. Just under half (48%) of health system indicators were related to measuring outputs, including hospital bed availability, and number of operations, consultations, radiology examinations and laboratory tests performed. A limited number of output indicators were disaggregated by gender and region. Input indicators included health expenditure and number of health facilities by region. Outcome indicators focused primarily on contraceptive use and immunisation coverage, as well as service utilization. Due to the missing pages from the 2001 Annual Report, comparisons are difficult. However a few interesting observations were made. Indicators on health expenditure (including absolute expenditure, percent of GDP and percent of total budget) were only reported in the 2001 report there is no such data in either 2005 or The remaining indicators for outputs and outcomes were almost identical in 2005 and 2007, suggesting that the selection and presentation of health system indictors are relatively stable. Table 8 Number of indicators presented by domain of measurement, domain sub-group and indicator group, Cook Islands Annual Reports 2001, 2005 & 2007 Domain of measurement Domain Sub-group Indicator Group TOTAL* Determinants of health Socioeconomic and demographic factors Environmental and behavioural risk factors Health system Inputs Outputs Outcomes Health status Mortality Infant and child Maternal General Morbidity Infant and child Maternal General TOTAL indicators * NOTE: total is the number of different indicators presented over the five years, and is not the sum of indicators presented from (as this would produce double-counting) Page 112

115 Health status: mortality data In the three Annual Reports analysed, 199 different mortality indicators were presented. Unsurprisingly, the vast majority of indicators were related to general mortality, followed by infant and child and then maternal mortality. The primary mode of presenting mortality data was in large tables arranged by ICD-10 code, age group and gender for each individual occurrence of death. There were also a number of tables with aggregate measures of mortality, including absolute number of deaths and crude death rate by year and region. Infant and child mortality Reporting on infant and child mortality has remained fairly stable. Indicators reported consistently over the three reports include high-level aggregate measures of mortality, such as number of foetal and infant deaths, and the infant and under-five mortality rates. Maternal morality There has been no change in the maternal mortality indicators presented over the three reports analysed. All reports include data on maternal deaths (absolute number and rate) and also the maternal mortality ratio. Data on maternal mortality is presented at an aggregate level by year, and as such, no other information on age group, ethnicity or region is provided. General mortality In looking at the contribution of each ICD-10 group to the overall number of general mortality indicators presented (Figure 20), over 50% of the data is related to neoplasms, diseases of the circulatory and respiratory systems and certain infectious and parasitic diseases. The remaining 10 groups make up between nine and one percent of all other general mortality data. In terms of the absolute number of indicators presented, there has been an increase in the number of mental and behavioural indicators; no change in all other causes ; and a decrease in reporting on all other ICD-10 groups. All reports include data on crude death rate and absolute number of deaths. Data on general mortality is presented in a number of tables including aggregate indicators (e.g. crude death rate ) by region and year; and disaggregated indicators (e.g. septicaemia, hypertension ) by age group, gender and region. Page 113

116 35 Health Status: Indicators by domain sub-group and ICD- 10 code, Cook Islands Indicators (%) I II III IV V VI VII VIII IX X XI XII XIII XIV XVIII XIX XX XXI NA ICD-10 Group Mortality Morbidity Figure 20 Comparative contribution (in percent) of mortality and morbidity indicators by ICD-10 group, Cook Islands Annual Reports 2001, 2005 & 2007 Key Definition Key Definition I Certain infectious and parasitic diseases XI Diseases of the digestive system II Neoplasms XII Diseases of the skin and subcutaneous tissue III IV Diseases of the blood and blood-forming organs and certain disorders involving the immune mechansim Endocrine, nutritional and metabolic diseases XIII XIV Diseases of the musculoskeletal system and connective tissue Diseases of the genitourinary system V Mental and behavioural disorders XVIII Symptoms, sings and abnormal clinical and laboratory findings not elsewhere defined VI Diseases of the nervous system XIX Injury, poisoning and certain other consequences of external causes VII Diseases of the eye and adnexa XX External causes of morbidity and mortality VIII Diseases of the ear and mastoid process XXI Factors influencing health status and contact with health services IX Diseases of the circulatory system NA All other causes (non-icd-10) X Diseases of the respiratory system Health status: morbidity data A total of 290 different morbidity indicators were presented over the three Annual Reports analysed, representing 59% of all health status data (Table 8). The majority of data are related to general morbidity; followed by maternal; and infant and child. Due to the missing pages from the 2001 report, morbidity data from that year cannot be compared. However in looking at data from 2005 and 2007, there appears to be very little difference in the number of Page 114

117 indicators reported. The primary mode of presentation of data is in large inpatient morbidity tables, with information on ICD-10 code, age group and gender. There are also a number of tables and figures related to individual diseases (such as sexually transmissible infections, noncommunicable diseases, and cancer). Infant and child morbidity Reporting on infant and child morbidity has remained constant over the two reports compared (2005 and 2007). Indicators presented consistently that are specific to infant and child morbidity are certain conditions originating in the peri-natal period and congenital malformations, deformations and chromosomal abnormalities and these are presented by age group and gender. Maternal morbidity There has been no change in the maternal morbidity indicators presented over the two reports. All reports include indicators on pregnancy, childbirth and the puerperium and also individual causes of morbidity including complications of labour and delivery and oedema, proteinuria and hypertensive disorders. Data on maternal morbidity is presented by age group. General morbidity In looking at the percent contribution of each ICD-10 group to the overall number of general morbidity indicators presented (Figure 20), approximately half of all data is related to infectious and parasitic diseases, external causes, diseases of the respiratory system and diseases of the circulatory system. The remaining 13 groups comprise between five and one percent of morbidity data. Of the 44 different notifiable and dangerous conditions listed in the Public Health Act 2004, 30 were included in all or some of the Annual Reports reviewed between 2001 and The primary mode of presentation of data is in large inpatient morbidity tables, with information on ICD-10 code, age group and gender. Page 115

119 Indicator stability and selection As demonstrated in Table 10, data in the Cook Islands Annual Reports are almost equally affected by issues of indicator stability (55%) and selection (45%). While indicator title (i.e. foetal deaths or tuberculosis ) and unit (absolute number, rate or ratio) have remained constant over time, changes in the level of disaggregation have affected the stability of mortality and morbidity indicators, and as such, limited their comparability. In 2001, for example, most general mortality indicators were disaggregated by region, however in 2005 and 2007 indicators were disaggregated by age group and gender, thus making it difficult to compare categories across time. Table 10 Summary of indicator comparability by domain of measurement; domain sub-group; and indicator group, Cook Islands Annual Reports, 2001, 2005 & 2007 Domain of measurement Domain subgroup Indicator group Indicators (N) Comparable over all reports (N) Stability issue (N) Selection issue (N) Determinants of Health Socioeconomic and demographic factors Environmental and behavioural risk factors Health System Inputs Outputs Outcomes Health Status Mortality Infant & child Maternal General Morbidity Infant & child Maternal General ALL indicators (64%) 110 (20%) 88 (16%) While indicator stability was an issue for all health status data, this result would have been biased from the missing pages in the 2001 report, which made it impossible to compare indicators over the full three reports. However, issues of stability are apparent in the other two domains, with a number of indicators reported for one or two years, and then not presented again. Page 117

120 A1.3.2 Disaggregation Table 11 summarises the level of disaggregation in data from the Cook Islands. The level of disaggregation is very high for morbidity data (86%). Overall, 72% of health status indicators are disaggregated. Furthermore, age group is the primary variable of interest (75%); followed by gender (37%); and region (20%). No indicators were disaggregated by ethnicity. Table 11 Health status indicators by indicator group, level and type of disaggregation, Cook Islands Annual Reports 2001, 2005 & 2007 Indicator group Indicators Disaggregated Disaggregation variables (N) (N) Age Ethnicity Gender Region Mortality total Infant and child mortality Maternal mortality General mortality Morbidity total Infant and child morbidity Maternal morbidity General morbidity Health Status TOTAL Mortality Just over half (52%) of the 199 different mortality indicators are disaggregated by at least one variable in the three Annual Reports analysed. Approximately 44% of infant and child and 53% of general mortality indicators were disaggregated: no level of disaggregation was provided for any data related to maternal mortality for the Cook Islands population. The primary variable of interest for mortality data was gender; followed by age group and then region. None of the data was disaggregated by ethnicity. Morbidity The situation is markedly better for morbidity data: of the 290 different indicators presented, 250 (86%) are disaggregated by at least one variable. All of the indicators related to infant and child and maternal morbidity are disaggregated, while 85% of general morbidity-related indicators are. Age group is the main variable morbidity data is disaggregated by; followed by gender; and then region. Approximately 13% of indicators were disaggregated by month (highlighting the seasonal trends in many diseases). Again, none of the data was disaggregated by ethnicity. Page 118

121 A1.3.3 Interpretability Source The amount of data contained in tables and figures that has reference to a source is very low: out of the 94 tables and figures with health status data, only three (3%) are sourced (Table 12). It is more common for mortality data to be sourced than morbidity. Meta-data Overall, 19% of health status data in the Cook Islands Annual Reports has some degree of meta-data notes, including indicator definitions ( total birth does not include stillbirth ) and calculation explanations ( rates are per 1,000 population ). Again, it is much more likely for mortality data to have meta-data. Table 12 Interpretability (by source and meta-data) of health status data, Cook Islands Annual Reports, 2001, 2005 & 2007 Health Tables/Figures Source Meta-data Status (N) Number Percent Number Percent Mortality % % Morbidity % 3 4.8% TOTAL % % A1.3.4 Presentation Presentation score From the sample of 40 tables and figures selected over the three reports, the Cook Islands scored an overall presentation score of 0.6 (Table 13), indicating that the majority of data contained in tables and figures is being adequately presented. There was no deviation in the overall score over the three reports sampled. In terms of specific criteria, the Cook Islands scored highly for legible, stub-headings, and legend or key (all receiving a perfect score of 1.0). Most tables and figures also had an appropriate title that clearly explained what was contained. Very few tables had columns or rows containing data totals (score = 0.3), and even less tables and figures had reference to their source (score = 0.1). Out of the 14 figures sampled, all had an appropriate legend; however none had scale options (axis labels). Page 119

123 Table 14 Production time, Cook Islands Annual Reports, 2001, 2005 & 2007 Annual Report Date of Publication Production Time 2001 October months 2005 August months 2007 July months Reference time The reference time of data contained within the Cook Island s Annual Reports is relatively even between each time-frame category (Figure 21). Approximately 50% of all data contained in the reports is either from the year of the report, or one or two years prior. The remaining five time-frame categories (three, four, five, six to 10 and 11+ years prior) all comprise approximately 10% each of the data. The oldest data contained within the Cook Island s reports is from Reference time: Health status indicators by age, Cook Islands Indicators 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Annual Report 11+ years prior 6 to 10 years prior 5 years prior 4 years prior 3 years prior 2 years prior 1 year prior Year of report Figure 21 Number of health status indicators by reference time, Cook Islands Annual Reports, 2001, 2005 & 2007 Page 121

124 Appendix 2 Fiji Annual Reports This appendix contains results of the descriptive and analytical review of the five Annual Reports produced from Fiji during 2004 to Results from the descriptive review, including report structure, data characteristics and number and type of indicators presented are in Section A2.1 and A2.2. Section A2.3 provides the results from the analytical review, including the five quality dimensions data was measured against. The remaining appendices are located either on the CD attached to the inside cover (printed version); or online in the separate document, Hodge_Transforming data into information and knowledge_appendices. Appendix 8 is a table of determinants of health indicators by domain sub-group, indicator title, unit, disaggregation and year. The table for health system data is found in Appendix 9. Appendices 10 and 11 provide health status indicators by domain subgroup (mortality or morbidity) and indicator group (infant and child, maternal or general). A2 Descriptive review A2.1 Report structure A2.1.1 Audience, purpose and use The purpose of each Annual Report has varied each year for Fiji, and no specific uses or target audiences were identified over the five reports analysed. Information on the purpose of the report is generally found in the introductory remarks or letter to the Minister (attached to the front of the report) and includes: To provide a summary of the Ministry s major activities and performance in programs under their responsibility (2005, 2006, 2007, 2008) To highlight the Ministry s performance in delivering services and contributing to outcomes targeted by the Government (2006) To illustrate the effort, commitment and achievements of staff and partners (2006, 2007, 2008) To feature major activities during the year that deserve special attention (2007) Page 122

125 To provide a summary of occurrence of vital events, performance against health outcomes and MDG indicators, and achievements of performance indicators (2007) To showcase the roles and functions of Mataika House and the Paediatric Unit (2007). A2.1.2 Legislative requirements There are two components within the Public Service Act 1978, Chapter 111 Public Health, which are likely to have influenced the development and structure of Fiji s Annual Reports. Under the Subsidiary Legislation Section S-7, clause (m), it states that, he shall also make an annual report to the Board up to the end of December in each year, comprising a summary of the action taken, or which he has advised the Board or local authority to take during the year for preventing the spread of disease and an account of the sanitary state of his sanitary district at the end of the year. The legislation also goes on to state that, the report shall also contain tabular statements... of the sickness and mortality within the sanitary district, classified according to diseases, ages and localities. The First Schedule of the Public Service Act, Public Health, also lists 35 different infectious diseases that are required for either immediate or weekly notification. Table 15 provides a summary of those diseases. Table 15 Infectious diseases, Public Service Act 1978, Chapter 111 Public Health, Fiji Cholera Plague Food poisoning Smallpox Typhus Yellow fever Acute poliomyelitis (paralytic and non-paralytic) Diptheria Enteric fevers (typhoid or para-typhoid) Antrax Brucellosis Encephalitis Dystentry (amoebic or bacillary) Infective diarrhoea or enteritis under 2 years (severe) Relapsing fever Infective hepatitis Leprosy Leptospirosis Malaria Meningitis Puerperal pyrexia Rheumatism (actue) Tetanus neonatorium Tetanus Tuberculosis (pulmonary or other) Venereal diseases (gonorrhoea, granuloma venereum, gonorrhoeal opthalmia, lymphogranuloma inguinale, soft chancre, syphilis, venereal warts) Yaws Dengue fever Chickenpox German measles Infective diarrhoea or enteritis under 2 years (mild) Influenza Measles Trachoma Whooping cough A2.1.3 Report type According to the Country Best Practices Report (WHO 2007), Fiji s Annual Reports can be classified as Type 2 (Statistical Reports). Reports such as these are comprised of both Page 123

126 statistical and descriptive components, including data analysis, comparisons between groups or geographical area, and trends over time. On average, there is less than one table or figure per page, with a large amount of text dedicated to describing the objectives of different sections within the Ministry of Health. Most of the text is a simple description of data contained within tables or figures, with very little reflection on major trends in the data, or explanations of variances or patterns. A2.1.4 Report structure The length of each Annual Report has increased dramatically: from 64 pages in 2004, to 109 in 2008, representing a 70% increase. The number of tables presented in each report has decreased, however this has been supplemented by an increase in the number of figures included each year. Table 16 summarises the main aspects of Fiji s report structure over the five years analysed. Table 16 Summary statistics, Fiji Annual Reports Annual Report Total number of: Pages Sections Tables Figures Tables & figures Average Overall, the structure of Fiji s Annual Report remained fairly constant throughout the five-year period in question. While the total number of sections has decreased from five to three; this reflects the collapsing of information into a smaller number of categories, rather than the subtraction of any components to the report. Reports are comprised of three main sections, each with their own specific purpose as set by the report authors (text in italics): 1. Vital statistics such as births and deaths, which are the key population components relevant to the work of the Ministry of Health. Rates of occurrence of vital events are also recorded. Infant and child mortality, maternal mortality, other deaths and birth rates are also included, as they are broad indicators of social and economic development Page 124

127 2. Health service utilisation measure of workload 3. Disease burden morbidity and mortality. A2.2 Data characteristics A2.2.1 Domains of measurement As shown in Table 17; the majority of data contained within Fiji s Annual Reports relates to health system inputs, outputs and outcomes, followed by health status and determinants of health data. The majority of determinants of health data are related to environmental and behavioural risk factors. Very little data is specific to health system outputs, with the majority of data describing inputs or outcomes. Furthermore, most health status data is related to morbidity. Table 17 Allocation of tables and figures by domains of measurement, Fiji Annual Reports Domains of measurement Contribution of Number of tables and figures in each Annual Report domain (%) TOTAL Within Between Determinants of health Total % Socioeconomic and demographic factors % - Environmental and behavioural risk factors % - Health system Total % Inputs % - Outputs % - Outcomes % - Health status Total % Mortality % - Morbidity % - All domains TOTAL % In looking at the patterns of reporting over time, there has been an apparent decrease in reporting on determinants of health (from 16% of data in the 2004 report; to 5% in 2008). A similar decrease has also occurred for health status, although the change is not as stable (from 35% in 2004, to 23% in 2008). Data relating to the health system domain has increased steadily over the five reports analysed: from 49% of all data in the 2004 report; to 62% in 2005; Page 125

128 66% in 2006; and its current level of 73% in 2007 and Figure 22 highlights the patterns of reporting over time. 100% 90% Fiji: Structure of report by domain of measurement Contribution of each domain 80% 70% 60% 50% 40% 30% 20% 10% Health status Health system Determinants of health 0% Annual Report Figure 22 Comparative contribution of each domain of measurement by report, Fiji Annual Reports A2.2.2 Indicators presented Determinants of health A very small number of indicators (6%) related to the determinants of health were presented over the five reports analysed (Table 18). The majority of such indicators described socioeconomic and demographic factors impacting on health, and included: population, crude birth rate, fertility rate and proportion of population living below the poverty line. A small number of indicators were disaggregated by ethnicity and region. Indicators related to environmental and behavioural risk factors include percentage of the population with access to safe water and sanitation, and housing conditions. The pattern of reporting over time has remained fairly constant over the five reports analysed for socioeconomic and demographic Page 126

129 factors, with more variance in the indicators presented year-on-year for environmental and behavioural risk factors. Health system Approximately one-third of indicators presented over the five reports analysed were related to health system data. Within this domain, the majority of indicators described system outcomes (such as contraception coverage, immunisation coverage and TB cure rate); followed by inputs (health facilities; staff numbers; revenue collected); and then outputs (drug stock, cases treated; services provided). There has been an apparent increase in the number of input indicators presented over the five years, with a decrease in output and outcome indicators. However, apart from the overall decrease in reporting in 2006 (across all domains), the total number of health system indicators presented has remained stable. Table 18 Number of indicators presented by domain of measurement, domain sub-group, indicator group and year, Fiji Annual Reports Domain of Domain subgroup group Indicator measurement TOTAL* Determinants of health Socioeconomic and demographic factors Environmental and behavioural risk factors Health system Inputs Outputs Outcomes Health status Mortality Infant and child Maternal General Morbidity Infant and child Maternal General TOTAL indicators * NOTE: total is the number of different indicators presented over the five years, and is not the sum of indicators presented from (as this would produce double-counting) Page 127

130 Health status: mortality data As demonstrated in Table 18, 76 different mortality indicators were reported over the five reports analysed. The majority of mortality indicators relate to general mortality, followed by maternal, and infant and child. Apart from a spike in the number of indicators presented in 2005, the overall pattern for reporting mortality indicators has remained stable between 2004 and Infant and Child Mortality Reporting on infant and child mortality has remained fairly stable. Indicators reported consistently over the five reports analysed include population-based measures of mortality, such as the peri-natal and post-natal mortality rates, and infant and under-five mortality rates. Maternal Mortality In looking at the pattern of reporting over time, there was a large decrease in the number of maternal mortality indicators reported after 2005: this was due to the change from presenting data by individual cause-of-death indicators, to grouped indicators by ICD-10 code. The only indicator that has been reported consistently over the five reports analysed is the maternal mortality ratio. General Mortality In looking at the contribution of each ICD-10 group to the overall number of general mortality indicators presented, over 50% of the data is related to infectious and parasitic diseases and diseases of the circulatory system (Table 22). The remaining 11 different ICD-10 groups each contribute less than 10% of all other general mortality data. In terms of the absolute number of indicators presented, there has been very little change in the number of indicators each year, apart from a small spike in the number of indicators related to diseases of the circulatory system in 2006 and All reports contained the crude death rate. Data on general mortality is presented in a number of different tables and figures, including aggregate vital statistics tables; summary measures of the top ten causes of mortality by year; and disaggregated indicators (i.e. individual cause-of-death) by age group, gender or ethnicity. Page 128

131 Health Status: Morbidity Data Of the 165 morbidity indicators presented over the five years analysed, the vast majority relate to general morbidity. The number of morbidity indicators has decreased during the reporting period analysed; from 100 in 2004, to 66 in Morbidity data is presented in a number of different tables and figures, including large inpatient tables; disease-specific reporting (i.e. sexually transmissible infections or non-communicable diseases); and summary tables of the top ten causes of morbidity. Infant and Child Very few indicators specific to infant and child morbidity were presented over the five reports analysed, and no indicators were presented for the full five years. Maternal Again, very few indicators specific to maternal morbidity were presented over the five reports analysed, and no indicators were presented over the full five years. One indicator was reported consistently ( puerperal pyrexia ) during 2004 to 2007; however it was not included in the 2008 report. Health status: Indicators by domain sub-group and ICD- 10 code, Fiji Indicators (%) I II III IV V VI VII VIII IX X XI XII XIII XIV XVIII XIX XX XXI NA ICD-10 group (coded) Mortality Morbidity Figure 23 Comparative contribution (in percent) of mortality and morbidity indicators by ICD-10 group, Fiji Annual Reports Page 129

132 Key Definition Key Definition I Certain infectious and parasitic diseases XI Diseases of the digestive system II Neoplasms XII Diseases of the skin and subcutaneous tissue III IV Diseases of the blood and blood-forming organs and certain disorders involving the immune mechansim Endocrine, nutritional and metabolic diseases XIII XIV Diseases of the musculoskeletal system and connective tissue Diseases of the genitourinary system V Mental and behavioural disorders XVIII Symptoms, sings and abnormal clinical and laboratory findings not elsewhere defined VI Diseases of the nervous system XIX Injury, poisoning and certain other consequences of external causes VII Diseases of the eye and adnexa XX External causes of morbidity and mortality VIII Diseases of the ear and mastoid process XXI Factors influencing health status and contact with health services IX Diseases of the circulatory system NA All other causes (non-icd-10) X Diseases of the respiratory system General Overall, just over half of all morbidity data is related to infectious and parasitic diseases (Figure 23). Neoplasms contribute a further 34% of data, followed by diseases of the circulatory system (8%). The remaining six ICD-10 groups comprise between one and two percent of morbidity data. There has been a slight decrease in the total number of morbidity indicators presented over the five reports analysed, from 99 in 2004 to 66 in Of the 35 different infectious diseases as stated in the Public Service Act of 1978, 32 of them have been continuously reported for the past five years. Furthermore, another four indicators (including a number of sexually transmissible infections) have been included over the past three to four years. There was a dramatic spike in the number of indicators relating to neoplasms in 2004, and to a lesser extent, Out of the 54 individual indicators presented over these two years, none of them have been included after Rather, they have been replaced with a single summary measure ( neoplasm prevalence ). A similar pattern occurred with circulatory system indicators, with 12 new indicators introduced in Out of these, three were reported on over the next two years, however the rest have not been included again. Page 130

133 A2.3 Analytical review A2.3.1 Comparability Indicator-comparability score As demonstrated in Table 19; there is little consistency in indicators reported during the five years analysed, resulting in an overall low indicator-comparability score for Fiji of In looking at the domains of measurement, indicators related to determinants of health scored the highest, followed by health status and then health system. The indicator group of infant and child mortality scored the highest overall indicator-comparability score out of all other possible groups, representing a high number of comparable indicators. Interestingly, infant and child morbidity scored the lowest (0.03), showing very little comparability in indicators presented over the five reports analysed. Table 19 Indicator-year score by domain of measurement, domain sub-group and indicator group, Fiji Annual Reports Domain of measurement Domain sub-group Indicator group Indicator-year score Determinants of health 0.26 Socioeconomic and demographic factors Environmental and behavioural risk factors Health system 0.14 Inputs Outputs Outcomes Health status 0.22 Mortality 0.22 Infant and child 0.53 Maternal 0.18 General 0.17 Morbidity 0.23 Infant and child 0.03 Maternal 0.30 General 0.23 ALL indicators 0.20 Page 131

134 Indicator stability and selection As displayed in Table 20, data from Fiji are overwhelmingly affected by issues of indicator selection; that is, a high number of indicators are presented for one or two years, and then not reported again. There does seem to be progress being made in the area of indicator selection for maternal mortality: one indicator has not been reported on since 2004, and another 22 have not been reported on since 2005 highlighting the move away from presenting individual cause-ofdeath data to standardised indicators (following ICD conventions). Overall however, the mortality indicators selected in Fiji s reports appear in a haphazard and fragmented manner, with many indicators only reported for one or two years. There are also a high number of indicators reported for one year, then absent in the following two, only to appear again the next two years. Table 20 Summary of indicator comparability by stability or selection issues, Fiji Annual Reports Domain of measurement Domain subgroup Indicator group Indicators (N) Comparable over all five reports (N) Stability Issue (N) Selection Issue (N) Determinants of health Socioeconomic and demographic factors Environmental and behavioural risk factors Health system Inputs Outputs Outcomes Health status Mortality Infant and child Maternal General Morbidity Infant and child Maternal General ALL indicators (11%) 89 (19%) 328 (70%) Page 132

135 In 2004 and 2005, 57 new individual indicators on neoplasm s were introduced into the reports (this is most likely related to a cancer study which took place in the Pacific at that time). From 2006, none of the indicators have been reported on again; being replaced with the aggregate indicator neoplasm prevalence instead. A similar pattern occurred with data on mental and behavioural disorders: of the 28 indicators reported in 2004 and 2005, none have subsequently occurred (however there has been no attempt at producing any aggregate indicators for mental health). The problem of indicator stability is highlighted in health system (outcome) data: attendance at oral health clinics was disaggregated by ethnicity in 2005; region in 2006; age group and ethnicity in 2007; and age group in Similarly for health system (input) data: there are eight different indicators used for budget data, all slightly different (budget by expenditure group, budget by allocation, final budget, etcetera). Changes in the stability of indicator definitions such as this make it difficult to compare over time. A2.3.2 Disaggregation Mortality The level of disaggregation among data presented in Fiji s Annual Reports is low: approximately 39% of the 302 different health status indicators are disaggregated by at least one variable (Table 21). Out of the 76 different mortality indicators reported over the five years in question, only three (4%) were presented in a disaggregated format. The data were all related to infant and child mortality: no level of disaggregation was provided for any data related to maternal or general mortality for the Fijian population. The primary variable of interest for mortality data was ethnicity (100% of disaggregated data included ethnicity); followed by age group (33%). None of the data was disaggregated by gender or region. Morbidity The situation is markedly better for morbidity data: of the 226 different indicators presented, 115 (51%) are disaggregated by at least one variable. Approximately half of the indicators related to infant and child and general mortality were disaggregated, while none of the maternal morbidity-related indicators were. Ethnicity and gender are the two main variables Page 133

136 morbidity data was disaggregated by; while 15% of disaggregated data was presented by age group, and approximately 1% by region. Table 21 Health status indicators by indicator group, number disaggregated and disaggregation variable, Fiji Annual Reports Indicator group Indicators Disaggregated Disaggregation variables (N) (N) Age Ethnicity Gender Region Mortality Infant and child mortality Maternal mortality General mortality Morbidity Infant and child morbidity Maternal morbidity General morbidity Health Status TOTAL (39%) 18 (15%) 112 (95%) 110 (93%) 1 (1%) A2.3.3 Interpretability Source Data contained in tables and figures with reference to a source was very low: out of the 87 tables and figures with health status data, only nine were sourced (Table 22). While it is positive to see that there are a handful of tables and figures that are sourced in later Annual Reports (none of the data from 2004 or 2005 had their source listed), there is a considerable amount of room for improvement in terms of interpretability for Fiji. Meta-data Only 3% of tables and figures with health status data had meta-data notes, which included explanations of calculations and indicator definitions (i.e. rates are per 1,000 population ). It was more likely for mortality data to have meta-data notes than for morbidity. Table 22 Interpretability of health status data, Fiji Annual Reports Health Status Tables/figures Source Meta-data (N) Number Percent Number Percent Mortality Morbidity TOTAL % 3 3.4% Page 134

137 A2.3.4 Presentation Presentation score From the sample of 78 tables and figures selected over the five years, Fiji scored an overall presentation score of 0.6 (Table 24), indicating the majority of data contained in tables and figures is being adequately presented. There was no major trend over the five years sampled (range: 0.6 to 0.7). In terms of specific criteria, Fiji scored highly for title and stub-headings (both receiving a score of 0.9). However only one table sampled had reference to its source, and on average, around half of the tables had rows or columns with data totals presented. Furthermore, out of the six figures sampled, very few of them displayed scale options (axes labels). A2.3.5 Timeliness Production time Overall, the production time of Fiji s Annual Reports appears to be improving. In the introductory remarks of the 2005 report, it was noted that the report was most likely the first ever to be completed and presented by the Ministry of Health within a year after the year ending. While the production time cannot be calculated for 2004 and 2005, as Table 23 demonstrates, the production time of earlier reports was up to ten months after the year ended, however by 2008, the report was available one month after year end. Considering the large amount of data contained within each Annual Report, a production time of between one and three months is a considerable achievement. Table 23 Production time, Fiji Annual Reports Annual Report Date of publication Production time (Parliamentary Paper 123) (Parliamentary Paper 60) October months 2007 March months 2008 January month Page 135

139 Reference time The reference time of data contained within Fiji s Annual Reports predominately relates to data that is from the year of the report in question. Overall, approximately 40% of data in each report is from the current year; data that is from one year prior makes up 10%, as do data that are from two, three and four years prior; data from five years prior contribute 5%; and the remaining 15% of report contents are from data that range from six years prior, to over 11 years. The oldest data contained within Fiji s reports is from Figure 24 demonstrates the reference time of Fiji s Annual Reports over the five years analysed. 100% 90% 80% Reference time: Health status indicators by age of data, Fiji Indicators (%) 70% 60% 50% 40% 30% 20% 10% 11+ years prior 6 to 10 years prior 5 years prior 4 years prior 3 years prior 2 years prior 1 year prior Year of report 0% Annual Report Figure 24 Number of health status indicators (in percent) by reference time (age of data), Fiji Annual Reports Page 137

140 Appendix 3 Tonga Annual Reports This appendix contains results of the descriptive and analytical review of the five Annual Reports produced from Tonga during 2004 to Section A3.1 and A3.2 provide the descriptive review results, including report structure, data characteristics and indicators presented. Section A3.3 covers the results from the analytical review, including the five quality dimensions data was measured against. The remaining appendices are located either on the CD attached to the inside cover (printed version); or online in the separate document, Hodge_Transforming data into information and knowledge_appendices. Appendix 12 is a table of determinants of health indicators by domain sub-group, indicator title, unit, disaggregation and year. The table for health system data is found in Appendix 13. Appendices 14 and 15 provide health status indicators by domain sub-group (mortality or morbidity) and indicator group (infant and child, maternal or general). A3 Descriptive review A3.1 Report structure A3.1.1 Audience, purpose and use The potential audience, purpose and use of Annual Reports from Tonga were not specifically mentioned in any of the reports reviewed. However, the Strategic Development Plan VII, which came into effect in 2007, and informs activities of the Ministry of Health, lists four priority objectives: 1. Guide the formation of the public sector s corporate and management plans and the annual budgets through which resources are allocated 2. Inform the private sector and civil society of Government s policy intentions 3. Provide the foundation on which Government can develop its external economic relations and donors can construct their country strategies and assistance programs Page 138

141 4. Provide indicators by which Government s progress in policy/strategy implementation can be monitored and measured. Reports from 2007 and 2008 have been structured to address the objectives as listed above, thus providing a general purpose of the reports (to monitor progress of the Strategic Development Plan). A3.1.2 Legislative requirements There are three legislative requirements related to the generation and dissemination of Annual Reports in Tonga. The Health Services Act of 1999, Part II General Admission, clause seven, states that, It shall be the duty of the Minister to present annually to His Majesty, a report on the state of the public health and the working of the health services for the year just ended. Furthermore, The Constitution of Tonga 1988, Part II Form of Government, clause 51, also states that, The Cabinet ministers... shall draw up a report once every year acquainting the King with the affairs of his department and such report shall be forwarded by the King to the Legislative Assembly at its next meeting and if the Legislative Assembly shall wish to know anything concerning the department of any minister he shall answer all questions put to him by the Legislative Assembly and report everything in connection with his department. There is also the Public Health Act of 1992, Section 138, Schedule four, which lists 42 notifiable diseases that must be reported on (Table 25). Table 25 Notifiable conditions, Public Health Act 1992, Tonga Acquired Immunodeficiency Syndrome (AIDS) AIDS Related Complex Anthrax Brucellosis Cholera Dengue fever Diphtheria Dysentery, all forms Encephalitis, acute Filariasis Food poisoning, suspected Gastroenteritis Hepatitis A Hepatitis B Human Immunodeficiency Virus 1 (HIV-1) Human Immunodeficiency Virus 2 (HIV-2) Infectious conjunctivitis Influenza Leprosy Leptospirosis Malaria Measles (morbilli) Meningitis, all forms Mumps Opthalmia neonatorum Paratyphoid fever Pertussis (whooping cough) Plauge Pneumonia, all forms Poliomyelitis Psittacosis Puerperal fever Rabies Rheumatic fever Rubella Tetanus Trachoma Tuberculosis, all forms Typhoid fever Viral haemorrhagic fever Yaws Yellow fever Page 139

142 A3.1.3 Report Type According to the Country Best Practices Report (WHO 2007), Tonga s Annual Reports can be classified as Type 2 (Statistical Reports). Reports such as these are comprised of both statistical and descriptive components, including analysis of data, comparisons between groups or geographical area, and trends over time. A3.1.4 Report Structure Overall, the structure of the Annual Report has remained fairly constant throughout the fiveyear period in question, with a noticeable increase in the total number of pages, tables and figures until 2006, before beginning to decline again (Table 26). Reports are structured so that all of the text is at the beginning of the document, with a number of appendices (statistical tables) at the end. While there are a large number of different sections (around 11 per report), they can be grouped into the following broad themes: Health system organisational objectives and functions; health administration and management; health resources; international collaboration; health districts; administration; health planning and information; leadership, policy advice and program administration Service provision public health services; medical services; nursing services; dental services; preventative services; curative services Health status overview of health indicators. Table 26 Summary statistics, Tonga, Annual Reports Annual Report Total Number of Pages Sections Tables Figures Tables & figures Average Page 140

143 A3.2 Data characteristics A3.2.1 Domains of measurement As shown in Table 27, the majority of data contained within Tonga s Annual Reports relates to health system inputs, outputs and outcomes. Within this category, the majority of data (45%) is related to outcomes (such as service utilisation). Approximately one-third of data is related to health status, and very little data is dedicated to describing the determinants of health. Table 27 Allocation of tables and figures by domain of measurement, Tonga, Annual Reports Domains of measurement Number of tables and figures in each Annual Report Contribution of domain (%) TOTAL Within Between Determinants of health Total % Socioeconomic and - demographic factors % Environmental and behavioural - risk factors % Health system Total % Inputs % - Outputs % - Outcomes % - Health status Total % Mortality % - Morbidity % - All domains TOTAL % In looking at the patterns of reporting over time, reporting has remained fairly constant over the five reports analysed (Figure 25). Overall, the majority of each report is comprised of data related to the domain of health system (such as inputs and outputs). Very little data is related to the determinants of health, and this has remained fairly stable at around 10% of each report. The number of data related to health status has slightly increased over time, from around 22% to over 30% in Page 141

144 Tonga: Structure of report by domain of measurement Contribution of each domain 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Health status Health system Determinants of health 0% Annual Report Figure 25 Comparative contribution of domains of measurement over time, Tonga, Annual Reports A3.2.2 Indicators Presented Determinants of health A limited number of indicators related to the determinants of health were presented over the five reports analysed (approximately 5% of the total), as demonstrated in Table 28. The majority of indicators covered socioeconomic and demographic factors, such as population growth and number (by age, region and gender), and absolute number of births, crude birth rate and total fertility rate. Indicators related to environmental and behavioural risk factors included infant nutritional mode and proportion of population with access to safe water or sanitation. Health system Approximately one-fifth of indicators presented over the five reports analysed were related to measuring health system performance. Input indicators were mostly related to finance and human resources. Output indicators included number of activities and services provided, Page 142

145 while outcome indicators measured aspects such as number of admissions, immunisation coverage, contraceptive coverage and cure rates for tuberculosis. Table 28 Number of indicators presented by domain of measurement, domain sub-group and indicator group, Tonga, Annual Reports Domain of Domain Indicator measurement sub-group group TOTAL* Determinants of health Socioeconomic and demographic factors Environmental and behavioural risk factors Health system Inputs Outputs Outcomes Health status Mortality Infant and child Maternal General Morbidity Infant and child Maternal General TOTAL indicators * NOTE: total is the number of different indicators presented over the five years, and is not the sum of indicators presented from (as this would produce double-counting) Health status: mortality data The vast majority of data (76%) over the five reports were related to health status indicators. Within this domain, over half of the indicators measured mortality. Unsurprisingly, the majority of mortality indicators were related to general mortality; however there was also a large amount (almost 20%) of indicators measuring infant and child mortality. The primary mode of presenting mortality data was in large tables arranged by ICD-10 codes, age group and gender for each individual occurrence of death. There were also a number of tables with aggregate measures of mortality, such as crude death rate and infant mortality rate. Infant and child mortality Reporting on infant and child mortality has fluctuated over the five years analysed, with over 100 different indicators presented. Indicators presented consistently include aggregate levels Page 143

146 of mortality, including infant mortality rate, number of infant deaths, and peri-natal mortality rate. Data on infant and child mortality are commonly presented in large tables by gender and age group. Maternal mortality Very few indicators relating to maternal mortality were presented over the five years analysed, and the only indicator to be presented consistently was the maternal mortality ratio. General mortality In looking at the contribution of each ICD-10 group to the overall number of general mortality indicators presented (Figure 26), over 50% of data was related to neoplasms, diseases of the circulatory system, injury and poisoning, and diseases of the respiratory system. The remaining 15 groups made up between one and seven percent of other general mortality data. All reports included data on the crude death rate and absolute number of deaths. Health status: morbidity data The majority of morbidity data is related to general morbidity, followed by infant and child and then maternal. The primary mode of presentation was in a number of disease-specific tables, which presented data by gender, region or age group. Infant and child Reporting on infant and child morbidity has fluctuated over the five years analysed, and this is due to changes in reporting on the causes of admissions to the paediatric ward and special care nursery. Acute respiratory infection and acute gastroenteritis are the only two indicators reported on consistently over the five years, and very few indicators are disaggregated. Maternal Indicators presented for maternal morbidity have remained fairly stable, with a number of individual causes of morbidity not being included after Four indicators were reported consistently over the five years analysed, and these are related to common causes of complaint during pregnancy or reason for admission. Approximately half of maternal morbidity indicators were disaggregated by region. General In looking at the percent contribution of each ICD-10 group to the overall number of general morbidity indicators presented (Figure 26), approximately half of all data is related to Page 144

147 infectious and parasitic diseases and endocrine, nutritional and metabolic disorders. The remaining 17 groups comprise between one and 14% of morbidity data. 38 different indicators related to mental and behavioural disorders were presented consistently over the five years. Apart from those 38 indicators, very few general morbidity indicators were presented consistently. The primary mode of presentation of data is in a number of different disease-specific tables. Indicators (%) 35% 30% 25% 20% 15% 10% 5% Health status: Indicators by domain sub-group and ICD- 10 code, Tonga 0% I II III IV V VI VII VIII IX X XI XII XIII XIV XVIII XIX XX XXI NA ICD-10 group Mortality Morbidity Figure 26 Comparative contribution (in percent) of mortality and morbidity indicators by ICD-10 group, Tonga Annual Reports Key Definition Key Definition I Certain infectious and parasitic diseases XI Diseases of the digestive system II Neoplasms XII Diseases of the skin and subcutaneous tissue III IV Diseases of the blood and blood-forming organs and certain disorders involving the immune mechansim Endocrine, nutritional and metabolic diseases XIII XIV Diseases of the musculoskeletal system and connective tissue Diseases of the genitourinary system V Mental and behavioural disorders XVIII Symptoms, sings and abnormal clinical and laboratory findings not elsewhere defined VI Diseases of the nervous system XIX Injury, poisoning and certain other consequences of external causes VII Diseases of the eye and adnexa XX External causes of morbidity and mortality VIII Diseases of the ear and mastoid process XXI Factors influencing health status and contact with health services IX Diseases of the circulatory system NA All other causes (non-icd-10) X Diseases of the respiratory system Page 145

148 A3.3 Analytical review A3.3.1 Comparability Indicator-comparison score Overall, of the 1,019 health status indicators presented between 2004 and 2008, only 68 (9%) of them were presented over the full five years, providing complete comparability of data. The low comparability in health status indicators has resulted in an overall low indicator-year score for Tonga (0.18) (Table 29). Overall, the main issue with data comparability stems from inconsistent data selection: many indicators are reported for one or two years, and then not included again, in a seemingly haphazard and unplanned manner. Table 29 Indicator-comparison score by domain of measurement, domain sub-group and indicator group, Tonga, Annual Reports Domain of measurement Domain sub-group Indicator group Indicator-year score Determinants of health 0.28 Socioeconomic and demographic factors Environmental and behavioural risk factors Health system 0.22 Inputs Outputs Outcomes Health status 0.16 Mortality 0.09 Infant and child 0.11 Maternal 0.28 General 0.08 Morbidity 0.33 Infant and child 0.35 Maternal 0.35 General 0.33 ALL indicators 0.18 Of the 562 different mortality indicators included (Table 30), only 24 (4%) were presented consistently over the five years analysed, allowing for a full comparison of the data. The indicator groups of infant and child and general had very similar indicator-year scores (0.11 and 0.08 respectively), reflecting the overall poor comparability of such data. However, maternal had a much higher score of 0.28, reflecting the relatively higher number of Page 146

149 indicators reported for more than one year, allowing for improved comparability. Indicators presented consistently were primarily aggregate-level measures of mortality, such as: Peri-natal and infant mortality rate Maternal mortality ratio Crude death rate Absolute number of deaths for the population, and Absolute number of deaths for the population by common disease categories (infectious and parasitic; neoplasms; diseases of the circulatory system, etcetera). The situation is slightly better for morbidity: of the 213 indicators included, 44 (21%) were presented consistently over the five years analysed. All of the indicator groups had a score close to 0.30, reflecting a reasonable number of indicators presented for more than two or three years. Infant and child morbidity scored the highest indicator-year score due to the large number of indicators reported for four and five years, providing a high level of comparability between the data. The number of maternal and general indicators reported for one or two years resulted in their lower indicator-year scores. A major issue with these indicator groups is the use of individual cause-of-death data, which naturally fluctuate between years, thus reducing comparability. Indicator stability and selection Overall, 64% of data in Tonga s Annual Reports were affected by issues of indicator selection (Table 30). This means that a high number of indicators are presented for one or two years, and then not reported on again. Changing the definition of indicators (i.e. from condom use to contraceptive prevalence rate or from population by age group to population by gender and region ) makes it difficult to compare data over time, thus decreasing indicator stability. Page 147

150 Table 30 Summary of indicator comparability by stability or selection issues, Tonga Annual Reports Domain of measurement Domain sub-group Indicator group Indicators (N) Comparable over all reports (N) Stability issue (N) Selection issue (N) Determinants of health Socioeconomic and demographic factors Environmental and behavioural risk factors Health system Inputs Outputs Outcomes Health status Mortality Infant and child Maternal General Morbidity Infant and child Maternal General TOTAL indicators (9%) 363 (33%) 643 (58%) A3.3.2 Disaggregation Mortality Over 80% of the 562 different mortality indicators were disaggregated by at least one variable. The primary variable of interest is gender and age group. A limited number of indicators were disaggregated by region, and none by ethnicity (Table 31). Morbidity The situation is not as good for morbidity indicators, with less than half disaggregated. While a reasonable amount of general morbidity indicators were disaggregated, very few infant and child or maternal morbidity indicators were. The primary variable of interest is region, followed by gender and age group, with no indicators disaggregated by ethnicity. Page 148

151 Table 31 Health status indicators by indicator group, number disaggregated and disaggregation variable, Tonga Annual Reports Indicator group Indicators Disaggregated Disaggregation variables (N) (N) Age Ethnicity Gender Region Mortality total Infant and child Maternal General Morbidity total Infant and child Maternal General Health Status TOTAL (71%) 467 (84%) 0 (0%) 494 (89%) 49 (9%) A3.3.3 Interpretability Source Overall, just over half of the 186 tables and figures presented in the five reports analysed had reference to their source (Table 32). There has been a dramatic improvement in presenting tables and figures with reference to their source in Tonga s Annual Reports, from no sourcing in 2004, to complete sourcing of all data contained within tables and figures in 2007 and There was no difference in levels of sourcing between mortality and morbidity data. Table 32 Interpretability of health status data, Tonga, Annual Reports Health Tables and Source Meta-data Status figures (N) Number Percent Number Percent Mortality % 7 8% Morbidity % 18 18% TOTAL % 25 13% Meta-data Very few tables and figures containing health status data had reference to meta-data. Of those that had meta-data notes, they were generally explanations of calculations ( calculated based on the assumption fertility rates will decrease ), indicator definitions ( rates are per 100,000 live births ), or general statistical information ( amended from statistics published in 2001 ). No trend is apparent in the use of meta-data in Tonga s reports, with percentage levels fluctuating between five and 33%. Slightly more tables and figures with mortality data had reference to meta-data than for morbidity. Page 149

152 A3.3.4 Presentation Presentation score From the sample of 102 tables and 33 figures selected over the five reports analysed, Tonga scored an overall presentation score of 0.7 (Table 33). This indicates that the majority of health status data presented in tables and figures is being presented adequately. There was no major trend in overall presentation score for the five reports sampled (range: 0.6 to 0.8). In terms of specific criteria, Tonga scored very high scores (0.9) for readable, column headings and legend, showing the general good quality of tables and figures. Another six criteria scored above 0.7. Tonga scored poorly for scale (0.3) and source (0.5). The most common presentation error was not placing axis labels on figures. Other errors included: Missing or ambiguous titles No reference to the data source The use of abbreviations, small font sizes and small figures, making readability difficult Tables not being presented with their totals. A3.3.5 Timeliness Production time The production time for Tonga s Annual Reports was unable to be calculated due to a lack of data. Reference time Apart from 2007, data contained within Tonga s reports is primarily from the year of the report, or one year prior (Figure 27). In the 2004 Annual Report, data from that year comprised over half of the entire report, however this number steadily declined until 2007 (where just under 30% of data was from the year of the report), before increasing again in Overall, data that is five years prior to the year of the report, and six to ten years prior, has been increasing over the years analysed (from approximately 5% to over 10%). No data was older than 11 years, with the oldest data relating back to Page 150

The Healthy Asia Pacific 2020 Roadmap INTRODUCTION: THE HEALTHY ASIA PACIFIC 2020 INITIATIVE In the 2014 APEC Leader s Declaration and Joint Ministerial Statement, it is recognized that the prospect of

The EU role in global health QUESTIONNAIRE: Question 1: In your opinion, does the proposed concept global health cover the most relevant dimensions? If not, which other essential factors would you suggest?

Occasional Paper: Strengthening Clinical Governance for Health Service Boards This work draws on research undertaken as collaboration between VMIA, The Centre for Health Policy, University of Melbourne,

Public sector November 2007 Improving information to support decision making: standards for better quality data A framework to support improvement in data quality in the public sector Improving information

Issues paper: The EU role in global health Comments from the Institute for Global Health, 9 December 2009 IGH is submitting answers to 14 of the questions posed in the consultation document. Question 3:

A national framework for the development of decision-making tools for nursing and midwifery practice Introduction The Nursing and Midwifery Board of Australia The Nursing and Midwifery Board of Australia

The Future Use of Electronic Health Records for reshaped health statistics: opportunities and challenges in the Australian context The health benefits of e-health E-health, broadly defined by the World

Basel Committee on Banking Supervision Working Paper No. 17 Vendor models for credit risk measurement and management Observations from a review of selected models February 2010 The Working Papers of the

Water, sanitation and hygiene in health care facilities in Asia and the Pacific A necessary step to achieving universal health coverage and improving health outcomes This note sets out the crucial role

Monitoring and Evaluation Framework and Strategy GAVI Alliance 2011-2015 NOTE TO READERS The 2011-2015 Monitoring and Evaluation Framework and Strategy will continue to be used through the end of 2016.

Review of the Measurement of Ethnicity Policy Perspectives Paper Main Paper July 2001 Kate Lang Social Policy Division Statistics New Zealand Statistics New Zealand Review of the Measurement of Ethnicity

Glossary Monitoring and Evaluation Terms This glossary includes terms typically used in the area of monitoring and evaluation (M&E) and provides the basis for facilitating a common understanding of M&E.

Page 1 of 8 Faris Dababneh et al. Improving mortality data in Jordan This online first version has been peer-reviewed, accepted and edited, but not formatted and finalized with corrections from authors

Message from the Chief Executive of the RCM The Midwifery Leadership Competency Framework has been derived from both the NHS Leadership Qualities Framework and the Clinical Leadership Competency Framework.

Commonwealth of Australia 2014 This work is copyright. In addition to any use permitted under the Copyright Act 1968, all material contained within this work is provided under a Creative Commons Attribution

THE MINISTRY OF HEALTH AND SOCIAL WELFARE POLICY FOR THE NATIONAL HEALTH MANAGEMENT INFORMATION SYSTEM OCTOBER 2008 Support for this publication was provided by USAID/Liberia and the USAID/BASICS Project

Overview of Water Utility Benchmarking Methodologies: From Indicators to Incentives Sanford Berg and Julie C Padowski sanford.berg@cba.ufl.edu Public Utility Research Center (www.purc.ufl.edu) University

A Review of the NHSLA Incident Reporting and Management and Learning from Experience Standards Assessment Outcomes April 2003 - March 2004 September 2004 1 Background The NHS Litigation Authority (NHSLA)

Registered nurse professional practice in Queensland Guidance for practitioners, employers and consumers December 2013 Registered nurse professional practice in Queensland Published by the State of Queensland

United Nations Statistical Institute for Asia and the Pacific Seventh Management Seminar for the Heads of National Statistical Offices in Asia and the Pacific, 13-15 October, Shanghai, China New Zealand

Global Conference on a Transformative Agenda for Official Statistics: Towards a Strategic Framework for Statistics in Support of the Post-2015 Development Agenda Concept Note Purpose and background The

Medical Management Plan Kenya OVERVIEW Company Mission Our mission is to encourage young people to volunteer for worthwhile work in developing countries. We expect that doing this kind of voluntary work

Guide to the National Safety and Quality Health Service Standards for health service organisation boards April 2015 ISBN Print: 978-1-925224-10-8 Electronic: 978-1-925224-11-5 Suggested citation: Australian

Translating System Thinking for Health Systems Strengthening and Development 1. The journey from real world practice to systems thinking Marcel Tanner & Don de Savigny Swiss Tropical & Public Health Institute

Technical guidance note for Global Fund HIV proposals in Round 11 UNAIDS I World Health Organization I August 2011 Rationale for including the development of a system in the proposal With the global momentum

Assessment of compliance with the Code of Practice for Official Statistics Statistics on Drug and Alcohol Treatment for Adults and Young People (produced by Public Health England) Assessment Report 264

Master of Public Health General track: Master of Public Health Specialisation tracks: Epidemiology International Public Health The general track in Master of Public Health provides a thorough grounding

AQTF Audit Handbook The following publication was endorsed by the former Standing Council for Tertiary Education Skills and Employment (SCOTESE). On 13 December 2013, COAG agreed that its council system

THE STRATEGIC PLAN FOR THE DEVELOPMENT OF THE NATIONAL HEALTH INFORMATION SYSTEM FOR 2008-2018 REPUBLIC OF ARMENIA YEREVAN 2008 TABLE OF CONTENTS LIST OF ABBREVIATIONS... iii PREFACE...v 1. INTRODUCTION...1

3.7.2009 Official Journal of the European Union C 151/1 I (Resolutions, recommendations and opinions) RECOMMENDATIONS COUNCIL COUNCIL RECOMMENDATION of 9 June 2009 on patient safety, including the prevention

At a glance 63 October 2013 Partnership working in child protection: improving liaison between acute paediatric and child protection services Key messages Both in-hospital and communitybased social work

The National Health Plan for Young Australians An action plan to protect and promote the health of children and young people Copyright 1997 ISBN 0 642 27200 X This work is copyright. It may be reproduced

DRIVING FORWARD PROFESSIONAL STANDARDS FOR TEACHERS The Standards for Leadership and Management: supporting leadership and management development December 2012 Contents Page The Standards for Leadership

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this

UMEÅ INTERNATIONAL SCHOOL OF PUBLIC HEALTH Master Programme in Public Health - Programme and Courses Academic year 2015-2016 Public Health and Clinical Medicine Umeå International School of Public Health

Guidelines for Gender Sensitive Programming Prepared by Brigitte Leduc and Farid Ahmad November 2009 What is a Gender-Sensitive Programming Process? Gender is a critical factor that determines an individual

INTRODUCTORY NOTE TO THE G20 ANTI-CORRUPTION OPEN DATA PRINCIPLES Open Data in the G20 In 2014, the G20 s Anti-corruption Working Group (ACWG) established open data as one of the issues that merit particular

AUSTRALIAN PROFESSIONAL STANDARDS FOR TEACHERS QUALITY TEACHING I L C O U N C Contents Introduction 2 Organisation of the Australian Professional Standards for Teachers 4 Professional Knowledge 8 Professional

Staff Guide to Qualification and Level Descriptors for Taught Programmes Issued by the Quality Assurance and Enhancement Unit August 2009 1. Overview This document is designed to assist staff in programme

Board of Member States ERN implementation strategies January 2016 As a result of discussions at the Board of Member States (BoMS) meeting in Lisbon on 7 October 2015, the BoMS set up a Strategy Working

Principles and protocols for reporting on schooling in Australia June 2009 1 Principles and protocols for reporting on schooling in Australia Background...3 Principles for reporting on schooling in Australia...4

RCN Policy Unit Policy Statement 16/2006 Acute and Multidisciplinary Working The Royal College of Nursing of the United Kingdom and the Royal College of Physicians (London) September 2006 Royal College

Investing in Health 2007: An update to the recommendations of Investing in Health: A Framework for Activating Primary Health Care Nursing (2003, Ministry of Health) September 2007 Investing in Health 2007:

1/5 Draft political declaration of the high-level meeting of the General Assembly on antimicrobial resistance We, Heads of State and Government and representatives of States and Governments, meeting at

...? series New title The NHS and HTA Supported by sanofi-aventis What is world class commissioning? Michael Sobanja Dip HSM Dip IoD Cert HE FRSM Chief Executive, NHS Alliance For further titles in the

THE BRITISH LIBRARY Unlocking The Value The British Library s Collection Metadata Strategy 2015-2018 Page 1 of 8 Summary Our vision is that by 2020 the Library s collection metadata assets will be comprehensive,

PROMOTING ACCESS AND MEDICAL INNOVATION: INTERSECTIONS BETWEEN PUBLIC HEALTH, INTELLECTUAL PROPERTY AND TRADE Joint study by the WHO, WIPO and WTO Secretariats Outline and overview The WHO, WIPO and the

Appendix 10 Blaby District Council Housing Stock Transfer CHANGE MANAGEMENT PLAN 1 Change Management Plan Introduction As part of the decision making process to pursue transfer, the Blaby District Council

Exposure draft of a report from the UK Statistics Authority: Quality Assurance and Audit Arrangements for Administrative Data July 2014 2 Contents Foreword from the Head of Assessment... 4 Summary... 6

A FRAMEWORK FOR NATIONAL HEALTH POLICIES, STRATEGIES AND PLANS June 2010 A FRAMEWORK FOR NATIONAL HEALTH POLICIES, STRATEGIES AND PLANS June 2010 This paper reviews current practice in and the potential

National Clinical Effectiveness Committee Prioritisation and Quality Assurance Processes for National Clinical Audit June 2015 0 P age Table of Contents Glossary of Terms... 2 Purpose of this prioritisation

International Open Data Charter September 2015 INTERNATIONAL OPEN DATA CHARTER Open data is digital data that is made available with the technical and legal characteristics necessary for it to be freely

Module 6: Data quality checks, access and use 6.1 Introduction The previous modules in this resource kit have focused on the systems and processes that need to be in place in order to collect information

Medical Management Plan Philippines OVERVIEW Company Mission Our mission is to encourage young people to volunteer for worthwhile work in developing countries. We expect that doing this kind of voluntary

Aboriginal and Torres Strait Islander Health Workers / Practitioners in focus i Contents Introduction... 1 What is an Aboriginal and Torres Strait Islander Health Worker?... 2 How are Aboriginal and Torres

A private sector assessment combines quantitative and qualitative methods to increase knowledge about the private health sector. In the analytic phase, the team organizes and examines information amassed