The proliferation of research witnessed in the past couple of decades has increased the need for distinguishing the impact of publications. As it could take a long time for a scientific finding to be translated into action, surrogate measures are sought after. Citation of a published work is often considered the ideal surrogate measure. The growth of social media and the use of published work in policy documents and news media have called for other alternate measures. The measures of scientific impact are formulated into impact indicators. Journal impact factor is a widely known indicator of scientific impact. Initially intended for the purpose of rating journals, it is often used for rating the article and the author. Impact factor is an average and is influenced by the outliers. The field of research, citation index, and the problem of deliberate self-citation influence it. The h-index, an author-level metric, should be interpreted with other complementary indices such as the e-index and q-index. Altmetric Attention Score, a novel indicator which uses alternative measures of impact such as used in social media, discussions, and policy documents, is a promising indicator.

Keywords: Altmetrics, h-index, impact factor, scientometrics

How to cite this article:Akkilagunta S, Deshmukh P, Bhatia V. Indicators of scientific impact: The need for a tectonic shift. Indian J Community Fam Med 2019;5:97-9

Scientific journals and scholarly articles have seen a widespread proliferation in the present age. An estimate says that there is an increase in scholarly articles by 3% every year.[1] The growth is contributed in part due to an increase in poor-quality predatory journals and poor publishing ethics. Owing to this, the need to distinguish oneself in the scholarly world has become crucial for getting research grants and promotions. The field of scientometrics, involved in evaluating the impact of scientific research, has been growing proportionately to address this need. At this juncture, it is pertinent to address two questions – What are all the measures of scientific impact? Moreover, which indicator/composite indicator gives a satisfying answer to address the need of identifying scholarly research?

Measures of Scientific Impact

Quantifying the impact of research has been a pursuit spanning decades since the inception of the impact factor by Garfield.[2] The scientific impact of research is the best measured by its application to benefit humanity. The benefit may vary in terms of its intensity and reach. However, not all published research show a direct tangible benefit. It may take decades for scientific research to be applied in action. Due to this, surrogate indicators of research outcomes were sought after. Of these, the citation is the widely used measure, included in most of the scientometric indicators. Citation depends on the access to the article, the type of publication, the field of research, age of the article, and the topic itself. For example, exploratory research might garner fewer citations compared to a clinical trial, and similarly, new and novel topics against the old topic. Review articles gather far more citations compared to the original research. The problem of deliberate self-citation could be used to manipulate. Due to these reasons, an effort has been made to include research outcomes apart from citation.

Few databases have used alternative measures of scientific impact. For example, page views, downloads, and social media sharing have been used in public library of science metrics.[3] These measures are still plagued by the limitations of access, and hence, more relevant to open-access journals. Even a question arises whether they reflect impact or mere feedback. Other alternative measures include media coverage, blogging, and use in Q and A threads. However, important of them all would be the use of research as a reference in policy documents. The use of research in policy would be the closest to scientific impact in terms of its benefit to humanity. Mere citation in policy documents may not implicate its direct contribution to policy. However, it is a beginning in the right direction.

Journal Impact Indicators

The second question relates to finding the best available composite indicator of scientific impact at all three levels – Journal, Article, and Author. The impact factor is the widely known journal-level metric of journals in the Web of Science database and is published every year as Journal Citation Reports. The impact factor measures the average number of citations received in a year by all the articles published in the previous 2 years. Since the measure is an average, it is affected by the skewness of the distribution. A study on biochemical journals showed that the top-cited 15% of the articles contribute to 50% of the citations.[4]

The original purpose of the impact factor had been to help librarians choose journals for a subscription. However, it has become the basis for the author to choose a journal for submission, a judge of the researcher's capability, and the basis for funding research projects. It has been used to distinguish authors and articles. An article or author may be published in a journal with a high impact factor, but itself may not be necessarily cited. Other journal-level metrics include SCIMAGO Journal rank, impact per publication, and Source Normalized Impact per Paper using the SCOPUS database. Article influence score and Eigenfactor score use the Web of Science database. These indicators vary from each other in the time window of citation, subject field limit, the range of citable items, and the adjustment for self-citation.[5]

Author-Level Indicators

Author-level metrics are used to distinguish between researchers based on the impact of their publications. H-index introduced by a physicist, Hirsh, has been widely reported. To calculate the h-index, the papers of an author are arranged in decreasing order of the number of citations received. The h-index is then the largest rank “h” such that the paper on this rank has h or more citations.[6] For example, an author with h-index of 23 has 23 papers with 23 or more citations. Since its conception, improvements have been made to refine it further. Individual h-index has been developed to reduce the contributing effect of coauthors.[7] Due to its nature, the h-index ignores papers with fewer citations and also those with a very high number of citations. As a result, two researchers with a huge difference in the citation count may have an identical h-index. A complementary metric termed e-index has been developed to account for those citations not included in the estimation of h-index.[8] Another index termed q-index evaluates deliberate self-citation by researchers.[9] Interpreting h-index requires the use of other complementary indices. Another commonly used indicator is i-10 index, i.e., number of articles with at least 10 citations.

Experts proposed g-index to overcome the poor weightage given by h-index to total citations and papers with a large number of citations. The papers by an author are similarly ranked in descending order of their citations. The g-index is the highest rank that the top g papers cumulatively have at least g2 citations. A researcher with a g-index of 10 has 10 papers that have together received at least 100 (102 citations).[10] However, both the h-index and g-index do not adjust for the age of the article. A contemporary h-index tries to address this issue by giving weightage to recent well-cited articles.[11]

Other author-level metrics are the SIGAPS score and ResearchGate (RG) score. RG score is a composite score that includes citation indirectly in the form of journal impact factor.[12] Other components of the RG score are discussion (question and answers) and followers. However, it has been criticized due to its lack of transparency and the inclusion of journal impact factors in its assessment.[13]

Article-Level Indicators

The commonly used article-level metrics are – mean normalized citation score, Relative Citation Ratio, and Altmetric Attention Score. While the first two use citations, the Altmetric Attention Score is comprehensive including traditional measures such as citation and contemporary indicators. The contemporary indicators include media attention, discussion, blogging, social media, and citation in policy documents.[14] Altmetrics, however, faces drawbacks such as heterogeneity, data quality, and lack of reliability. The accuracy of data on social media attention, blogging, etc., is questionable. The error in estimation is multiplied due to the inclusion of a wide variety of indicators. The methodology is revised often which makes it unreliable.[15] Although Altmetrics is appealing due to its contemporary nature, it is still in its nascent phase. Further research is required to overcome these limitations and enhance its use in future.

Conclusion

Citation still forms a base for most of the research impact indicators. In this era of digital publication, open-access and social media influence demands the inclusion of other research outcome indicators. Altmetrics and other similar indices while promising need further research.

To conclude, the growth of scientometrics has enabled a long list of impact indicators. There is a need for a unified, comprehensive indicator at each level to allow for simplicity and transparency in the assessment of scientific impact. Despite its drawbacks, the journal impact factor is still used by funding agencies for decision-making. In India, impact factor and h-index have been reported for funding agencies such as Indian Council of Medical Research as a measure of research output.[16] As discussed above, no single indicator allows for a comprehensive, unbiased assessment. Ensuring awareness and access to the key scientometric indices is necessary for informed decision-making.