1. INTRODUCTION

Many evaluations of research institutions apply purely quantitative scientometric indicators, which are based upon publication and citation counts (Yang & Lee, 2013). The challenge of scientometrics is to quantify quality (Yang & Lee, 2018) as well as possible. Often, research evaluations use only a few indicators or even just one. In practice, problems with the ranking of research institutions often become apparent in the course of university rankings, since these are carried out frequently. In particular, worldwide university rankings are very popular (Kochetkov, 2024) and sometimes even lead to reports in newspapers. Some examples of such rankings with different approaches include:

the Academic Ranking of World Universities (ARWU) (established in 2003), initially published by the Shanghai Jiao Tong University, and since 2009 distributed by the Shanghai Ranking Consultancy, located in China,

the Quacquarelli Symonds World University Rankings (QS) (founded in 2004 together with THE), which has its headquarter in London (UK) and has numerous offices in many countries of the world,

the Times Higher Education World University Ranking (THE) (established in 2004, independent from QS since 2010), published by TSL Education and located in London, UK,

the Webometric University Ranking (since 2004) by the Cybermetrics Lab of the Spanish National Research Council in Madrid,

the Leiden Ranking, existing since 2006, compiled by the Centrum voor Wetenschap en Technologische Studies at Leiden University in the Netherlands, and, finally,

the U.S. News Best Global Universities, produced by U.S. News & World Report with headquarters in Washington, DC, and New York, NY (established in 2014 for the international ranking, while the national American ranking started in 1983).

ARWU, also known as Shanghai Ranking, considers research awards (e.g., Nobel prizes), field-specific and highly cited researchers (from Clarivate), articles published in top journals (such as Nature or Science) and indexed by multidisciplinary bibliographic information services (here Web of Science [WoS]), and performance per capita (Liu & Cheng, 2005). Very important aspects of ARWU are overall research output and top-notch researchers (Dehon et al., 2010). In contrast, THE Ranking applies data from opinion surveys on university reputation, the faculty per student ratio, the number of citations per faculty, the extent of international faculty, and the number of international students (Baty, 2013); furthermore, THE changes their selection of concrete indicators from time to time. It covers mainly the university’s perceived reputation, its research impact (based upon data from WoS), its resources for teaching, and its internationality. QS Ranking is similar to THE (both institutions originally cooperated). QS works with surveys on research activity, employers’ surveys, citations per faculty, the faculty-student ratio, and the numbers of international faculty and students (Huang, 2012). In THE Ranking, survey data is weighted at 30%, while QS is more dependent on surveys (50%). Leiden Ranking is based on publication output, citation impact, and scientific collaboration; all data are retrieved from Clarivate’s Science Citation Index and Social Sciences Citation Index (Waltman et al., 2012). The U.S. News Best Global Universities Ranking uses survey data on research reputation by Clarivate (weighted at 25%) and the number of publications, as well as the number and percentage of the most cited publications (data from Clarivate’s InCites), and international collaboration (Morse & Wellington, 2024). The Webometric University Ranking analyses the universities’ web presence performance, such as the number of web pages, “rich” files (e.g., PDFs), and external links to the university’s websites (Aguillo et al., 2008). This may be an indicator for the size of online published information and of the access to this information, but is by no means a research or teaching indicator.

Each approach brings its own ranking into being frequently inconsistent with the other rankings. For example, THE ranks the Universities of Oxford (UK) and Cambridge (UK) in the first two places, while QS ranks Massachusetts Institute of Technology (US) and Stanford University (US) at the top. The California Institute of Technology is no. 3 in THE and no. 4 in QS, but no. 193 in the Leiden Ranking (all data as of 2018; Fauzi et al., 2020, p. 89). However, Aguillo et al. (2010) found also some similarities between certain pairs of rankings (e.g., between ARWU and a ranking from Taiwan). But for most rankings, “correlations highlight the significant differences among some indexes even in measuring the same criterion such as teaching or research” (Olcay & Bulu, 2017, p. 153).

All this has led the majority of researchers to view worldwide university rankings extremely critically (Bellantuono et al., 2022; Billaut et al., 2010; Chirikov, 2023; Gadd, 2021; Kochetkov, 2024; Lim, 2018; Moed, 2017; Selten et al., 2020). For example, Altbach (2006) concludes, “they (university rankings) often measure the wrong things, and they use flawed metrics to do the measurements.” If rankings try to measure “reputation” using surveys—and such reputation indicators play an important role in the rankings of QS and THE (Hou & Jacob, 2017)—problems arise with the “reputational bias derived from the anchoring, echo, halo and Matthew effects” (Safón, 2019, p. 910). University rankings clearly exhibit shortcomings such as “adjustment for institutional size, differences between average and extreme, defining the institutions, measurement of time frame, credit allocation, excellency factors as well as adjustment for scientific fields” (Anowar et al., 2015, p. 559). In a review of articles on university rankings, Vernon et al. (2018, p. 13) conclude that “current indicators are inadequate to accurately evaluate research outcomes and should be supplemented and expanded to meet standardized criteria.” For van Raan (2005, p. 140) some university rankings can be considered as “quasi-evaluations,” which is “absolutely unacceptable.”

In view of these criticisms, the question arises as to how a meaningful ranking could be carried out at the institutional level. It is obvious that there is no universally valid research ranking. Rather, it seems more sensible to take various aspects into account in order to avoid an overly narrow view and possible misjudgments. And this is precisely where our study comes in, which on the one hand attempts to identify the dimensions (and their attributes) that affect the results of research evaluations at the institutional level, such as at the level of departments, faculties, or universities. In addition, a case study will be carried out to examine whether identical data but modified indicators and attributes lead to different rankings of two specific research institutions. If this is the case, a combined consideration of various aspects could provide a more complete picture of the research performance of the two exemplary institutions.

For resource-economic reasons, our study focuses on the research performance of institutions in the form of publications (publication output and publication impact), as we believe these are of central importance for assessing the overall performance of institutions. Therefore, we focus on the inconsistencies of rankings that make use only of quantitative scientometric data and indicators. From the literature (e.g., Reichmann & Schlögl, 2021; 2022; Reichmann et al., 2022), we identified several dimensions of research output and of research impact, which may influence the results of a quantitative research evaluation more or less strongly. “Output” (“production”) means an institution’s publications and “impact” the citations these publications attracted (Abramo, 2018; Stock, 2001; 2010).

For our case study, we focused on two university departments that we know very well (those where we are currently working or have previously worked), as this was the only way to fully capture the required data. Departments, like universities, are also located at the meso-level, and so the dimensions and attributes we identified in the first step are fully relevant.

Overall, in our study we want to answer the following two research questions (RQs):

RQ1. Based on a purely theoretical consideration: Which dimensions and which attributes of the dimensions are useful for the description and subsequent ranking of research institutions? (Section 2)

RQ2. Based on a case study of two research institutions: Is it useful to produce different rankings concerning their research output and research impact? (Sections 3 and 4)

2. DIMENSIONS OF SCIENTOMETRIC RESEARCH EVALUATION

In the following section (2.1.) we present the dimensions having an impact on the results of research evaluations in more detail (RQ1). As is e xhibited in Table 1, they differ partly depending on whether they refer to research output via publications (P) or research impact via citations (C). Concerning research output we found nine, and for research impact we identified 14 relevant dimensions. In Table 1 we list each dimension and its possible attributes to provide guidance for a “fair ranking” of institutions (Vavryčuk, 2018), if something like that is ever possible.

We did not consider compound indicators, which are derived from two or more raw indicators. Examples are normalization approaches for research fields, such as the Leiden crown indicator (Abramo & D’Angelo, 2016; Opthof & Leydesdorff, 2010; van Raan et al., 2010), and the H-Index combining publication and citation counts (Hirsch, 2005; 2023; Kamrani et al., 2021). We also did not consider indicators from altmetrics, e.g., mentions of an institution’s research activities on social media services (Stock et al., 2023b, pp. 46-48).

3. CASE STUDY METHODOLOGY

Based on our second RQ, we want to investigate what impact follows on research performance if we vary some attributes of the scientometric dimensions discussed before. In order to do this, we are following a case study approach and compare the research performance of two similar information science departments: the Department of Information Science at the University of Düsseldorf, which was part of the Institute of Language and Information, and the former Department of Information Science and Information Systems at the University of Graz. The main reason for the selection of these two research institutions was that they are comparable to some degree. Partly they do research on similar topics, in particular scientometrics and information literacy, and partly they work in different research areas: Düsseldorf on smart cities and social media services, and Graz on topics related to information systems. Another reason was that the authors were staff members of the two institutions, which facilitated both data collection and interpretation of the results.

One major difference between the two organizational units relates to the staff structure. While the department in Düsseldorf consisted of one professor and more than ten predocs and postdocs, the department in Graz had one full and three associate professors as well as one predoc researcher. As a consequence, the department in Düsseldorf had slightly more than two times the research staff compared to that in Graz. The department in Düsseldorf is shut since the beginning of 2024, while that in Graz was merged with two other departments in 2020, forming the Department of Operations and Information Systems. Due to data availability, we chose an evaluation period from 2009 to 2018 (i.e., 10 years). Publication data was collected for all research staff employed (at some point of time) between 2009 and 2018. It should be noted that obtaining the appropriately detailed employment data took some effort. This shows how difficult it is, even for relatively small research units, to determine the research output per FTE, in particular for longer evaluation periods.

For Düsseldorf, the data of all personal publication lists was cumulated. For Graz, there exists an institutional research database in which researchers have to enter their research activities, including their publications. All publication data was entered into a MS Access database (Microsoft, Redmond, WA, USA) developed for this study. Besides the most important bibliographic data, the number of co-authors, the publication type (books, journal articles, articles in proceedings, articles in other edited books, and other publications), the publication language, and the length (in pages) were also recorded for each publication. Finally, we recorded whether a publication was included in WoS Core Collection (without consideration of the Book and the Proceedings Citation Index), Scopus, or Google Scholar, and how often it was cited in these databases. Data collection took place in December 2020 for WoS and in March 2021 for Scopus and Google Scholar. The Access database comprises a total of 345 publications by authors from Düsseldorf and 228 publications by authors from Graz. Fig. 1 shows the publication output during the observation period by document type for Düsseldorf and Graz.

Fig. 1

Publication output by document type for Düsseldorf and Graz (2009-2018).

As can be seen, journal articles constitute only slightly more than one quarter of the total publication output at both institutions. Therefore, we decided not to base our publication analyses solely on journal articles (dimension P6). Variant 1 covers all document types, which were arbitrarily weighted as follows: books – 10 points, journal articles – 5 points, articles in proceedings – 3 points, articles in other edited books – 2 points, other publications – 1 point. Variant 2 only considers journal articles. This means that in variant 2 there is no weighting of publication types. For each of these two variants, we used the following attributes for the various dimensions to calculate the so-called base of operations:

Starting from the base of operations, we then calculated alternative research rankings (in our case study, they consisted only of a comparison of the values for the two compared institutions), in which only one attribute of one dimension was changed compared to the base of operations (e.g., dimension P4 – data source: WoS instead of personal publication lists/institutional research database). In this way, we calculated 12 (9) alternatives with regard to variant 1 (2), the results of which are displayed in the following two tables, both size-dependent and size-independent. Each of the tables also shows for each ranking by what percentage Düsseldorf is better or worse than Graz (Graz=100% in each case).

We chose the same procedure for the citation analysis as for the publication analysis, i.e., there are variants 1 and 2 with regard to the cited documents (dimension C10: all document types without weighting vs. journal articles), as well as a size-dependent and a size-independent presentation of the results (dimension C3). For both variants, we again calculated a base of operations, on the basis of which we then calculated seven alternatives according to the procedure described regarding the publication analysis. We used the following attributes for the various dimensions to calculate the base of operations for the citation analysis:

4. CASE STUDY RESULTS

5. DISCUSSION AND CONCLUSION

Our first RQ tried to identify the dimensions which have an impact on the results of research evaluations at the institutional level. Since each of the identified dimensions consists of several attributes, there are hundreds of thousands (regarding publications) or even millions (regarding citations) of possible combinations to rank institutions. Definitely, many of these combinations do not make sense. However, a greater number of them can be relevant in research evaluations.

In practice, when evaluating research institutions, for example in the context of university rankings, only a fraction of these variants can of course be used. However, some variants are used that we would never recommend for reasons of fairness (Anowar et al., 2015; Fauzi et al., 2020; Moed, 2017). For example, many international university rankings are entirely size-dependent or use only individual size-independent indicators. This is probably primarily because in many cases it could be difficult to obtain reliable data on size. If, as in our case, a study is restricted to the area of research, the average number of researchers at a university during the observation period would be needed; ideally in the form of FTE. As our study has shown, obtaining this data is already quite complex for two departments. But international university rankings usually refer to an observation period of just one year (Selten et al., 2020), which in turn makes it much easier to collect size-relevant data, as a single data collection point seems sufficient (all relevant data, such as the number of employees, are only shown on an annual basis anyway), which is why we strongly recommend collecting and taking such data into account as part of research evaluation at the institutional level.

In our study, we chose a ten-year observation period due to possible annual fluctuations in the publication output of individual researchers, as we only considered a relatively small number of researchers due to the restriction to two departments. Due to the much larger number of researchers at entire universities, annual fluctuations in performance by individual researchers should not play a role in university rankings even when looking at the data on an annual basis.

Another dimension that, in our view, is often not given enough attention in international university rankings is the handling of multiple authorships. Full counting is often used with regard to publication output, and this is even the standard in terms of research impact. This seems rather unfair to us, as it puts single authorships at a great disadvantage. However, it should be stressed that the use of fractional counting would mean a considerable additional effort. In the case of fractional counting, we would recommend using the formula 1/n, which is also the most important in practice, as our literature study has shown (see Section 2).

Based on the second RQ, we wanted to explore whether modified indicators and attributes lead to different rankings and how large the differences in the rankings can be. Hence, we have selected several combinations from the options elaborated theoretically (see Section 2) that we consider useful for research evaluation at the meso-level. As the results show, the rankings can vary greatly.

In addition to the two dimensions already mentioned above (size of the institution and handling of co-authorship), the following dimensions also had a major influence on the results of our case study: a) the representatives, whereby we recommend that the entire research staff should be taken into account in a research evaluation. This is the only way to capture the entire research performance of an institution; and b) the data source, whereby the use of personal publication lists or institutional research databases would be optimal for reasons of completeness. However, such data sources sometimes do not even exist and their use in the context of international university rankings would be extremely complex. In practice, such rankings usually rely on a single one of the common information services (Billaut et al., 2010), which can lead to considerable distortions, as our case study has shown. It would therefore be desirable if these information services were used in combination. One consequence of using these information services as a data source is that the attributes with regard to the dimensions “language” and “document types” are practically fixed, since these information services mainly contain papers in English.

Table 6 summarizes the largest differences depending on the chosen procedure (attributes). As can be seen from our case study, Düsseldorf is clearly ahead in all citation rankings, while the results of the publication rankings are more diverse. Depending on the particular underlying “research reality,” the selection of a certain attribute of a research dimension is in favor of either the one or the other department.

The large differences between the results of the publication analysis and those of the citation analysis revealed in our case study illustrate the great importance of using both analyses together. In our view, too much emphasis is often placed on citation analyses in international university rankings (Baty, 2013; Huang, 2012). This is critical because, due to the limited data sources for citation analyses, there is little scope for design with regard to other dimensions, such as language.

However, the results of our case study also show that an institution that is significantly better in one area, such as Düsseldorf in terms of citations, performs better in all related rankings, regardless of which (somewhat meaningful) attributes are used. This seems very reassuring to us with regard to the selection of these attributes.

One major limitation of our case study is that it lacks generalization. Therefore, it cannot be taken for granted that other studies also reveal similarly significant differences in their rankings – or even greater differences. Overall, we assume that the variability of the rankings does not only depend on the particular circumstances of the institutions under consideration (for instance, more project staff vs. more long-term employed staff) but also on the discipline. In some natural sciences disciplines it might, for instance, be sufficient to concentrate on journal articles and citations to them. In contrast, in many social sciences and humanities disciplines this would be a much too narrow view, as is also confirmed in our case study. Here, it definitely makes sense to follow a broader approach. However, it must be pointed out that if an institution designated for a research evaluation is invited to participate in the determination of the indicators, it could be “attempted” to suggest those dimensions and attributes which are in favor of it.

Scientometric indicators are not “good” or “bad” per se. However, as our case study shows, a wise use of different indicators can reflect research reality much more adequately. Therefore, rankings of institutions such as the rankings of the world’s universities with only one composite indicator are in no way purposeful. As there are numerous possible combinations of quantitative dimensions and their attributes, there is never a “true” measurement and all indicator selection is more or less arbitrary.

Therefore, all mentioned rankings of universities are suboptimal. The research literature on global rankings of universities is very critical. But the producers of the rankings have failed to adjust their methods. After all, if the publishing institutions insist on producing their rankings in the future, following the literature and the results of our study, we have some serious recommendations:

If evaluators fail to follow these hints because the necessary activities are too complex or too costly to realize, the Austrian philosopher Wittgenstein (n.d., ch. 7) should be remembered: “Whereof one cannot speak thereof one must be silent.”