Blog

Research assessment reform is a hot topic of conversation in academia and scholarly publishing, with institutions, governments, and industry organisations coming together worldwide to tackle an important question: how should research and academic outputs be assessed?

We explore the context of research assessment, ongoing work globally to address some of the current issues, and how open research initiatives like HRB Open Research can help contribute to research assessment reform.

What is research assessment and how is it assessed currently?

Research or academic assessment evaluates researchers’ work for decisions around recruitment, funding, and promotion. With finite resources, it helps decision-makers allocate opportunities to support scientific progress across critical priorities and to evaluate return on investment.

Historically, this assessment has relied on quantitative indicators, such as how many papers a researcher has published and the Journal Impact Factor of a researcher’s chosen publishing venue.

What are the issues with this current approach?

It’s now widely acknowledged that these indicators are inadequate to evaluate research output effectively.

Issues include:

• Prioritising how many papers a researcher has published, which can lead to quantity over quality and poorer science due to pressure to publish as often as possible.
• Judging the impact of work through journal-level metrics like Journal Impact Factor, which are often maintained through a high rejection rate of articles, regardless of quality, and a focus on author prestige.
• Limited access to higher impact journals for early career researchers due to this focus on author prestige, which can have knock-on effects on funding and career progression.
• Deprioritising sound but not novel research, which is often published in lower impact journals, despite the research still providing a quality contribution to the scientific record.
• Considerable variance of journal-level metrics between fields and document types, which makes it difficult to establish inter-field comparisons.
• Skewed metrics within a journal itself, with a small number of articles having lots of citations while most other articles have far fewer, and thus failing to be indicative of individual article quality.

Global research assessment reform

As a result, initiatives to reform research assessment are now centre stage worldwide, including the San Francisco Declaration on Research Assessment (DORA) and the Coalition for Advancing Research Assessment (CoARA).

These initiatives seek to ensure that assessment considers a wide range of research activities, considering both qualitative and quantitative measures.

Qualitative metrics for research assessment could include:

• Participation in peer review
• Sitting on editorial boards
• Working with stakeholders and policymakers
• Participating in academic citizenship
• Providing teaching and educational resources
• Providing mentoring
• Developing open access materials, including open source software and code, and open science guidance

These initiatives also support a move away from journal-level metrics like Impact Factor. Instead, the focus is on the responsible use of article-level quantitative metrics, such as:

• Article citations
• Article downloads
• Article shares
• Article views
• Altmetric scores, which can demonstrate impact on policies and other non-academic sources

However, both CoARA and DORA emphasise that decision-makers should only use quantitative measures to support the types of qualitative indicators already identified.

Finally, there is a call for research assessment reform to broaden our perception of what constitutes a quality paper. Rather than focusing solely on novelty, we should consider the broad range of outputs and results from projects.

For example:

• Publishing outputs such as datasets, software, and methods, which can show innovation outside of the traditional Research Article.
• Publishing negative or null results, which can still add to the scientific record and reduce unnecessary repetition.
• Publishing confirmatory results, which can bolster existing evidence of crucial research questions.

Overall, these initiatives seek to ensure that research assessment places value on the diversity of outputs that researchers produce and the variety of activities and practices that researchers undertake, such as mentoring and peer review.

HRB Open Research and research assessment reform

Many open research practices help support research assessment reform, including all practices underpinning HRB Open Research. These include:

• Not having a Journal Impact Factor, instead promoting the responsible use of individual article indicators.
• An open data policy that facilitates the reproducibility of research and helps demonstrate researchers’ involvement in open science practices.
• Supporting 11 article types, including dedicated Data and Software article types, and brief article types that can aid in working with stakeholders and policymakers, such as Research Notes, Case Reports, and Correspondence.
• A broad scope that supports all results, including publication of confirmatory, null, and negative results.
• Opportunities for increased credit for additional activities, such as peer reviewing (each peer review report can be cited independently from the article).

Research assessment reform is critical to the move toward open science in academia and scholarly publishing. The Health Research Board is committed to supporting this reform as both a signatory of CoARA and DORA and in providing HRB Open Research.

Find out more about research assessment with the HRB.

Publish your work with HRB Open Research

If you’re an HRB-funded researcher, you can publish your work with HRB Open Research at no cost and benefit from these alternative measures in research assessment, including rapid publication, open access, open peer review, and diverse article types.

Find out more about submitting your research today.

• Follow us on Twitter
• Sign up to our newsletter