Reading Time: 3 minutes
Pressure gauge for a 19th-century steam engine

Universities spend considerable time and money trying to measure research influence, but do those metrics truly capture scholarly value? I talked with Rachel Miles, research impact coordinator at Virginia Tech Libraries, to find out more about why metrics have become central (and sometimes contentious), and how emerging tools—especially AI—are reshaping how we interpret scientific influence.

The Evolution of Research Impact in Libraries

Rachel Miles’s career path highlights how research impact has become a pivotal part of library services. Originally, libraries focused on collections and reference services; now, they actively engage in understanding how research is communicated, its dissemination, and its influence on society. With more research moving online and becoming accessible, libraries started tracking how publications, data, and scholarly activities contribute to society and academia. This shift was driven by digital tools and the internet, which provided unprecedented ways to quantify and analyse research influence.

Why have metrics gained such popularity?

The explosion of the popularity of metrics is driven largely by university rankings. Metrics influence institutional reputation, faculty evaluations, and funding, creating a feedback loop that elevates their importance. But this focus can distort priorities. Many metrics—such as journal impact factors—don’t necessarily align with an institution’s mission or the broader societal impact of research. Instead, they often serve institutional competitiveness, encouraging practices that may skew genuine scholarly contributions.

Specifically, Rachel Miles points out several challenges in the use of metrics:

  • Field differences: Impact measures favor STEM fields with citation practices that differ from humanities or social sciences.
  • Geographical bias: Research from the Global South or non-English sources often goes under-represented.
  • Goodhart’s Law: When a measure becomes a target, its usefulness diminishes. If researchers start optimizing for citations rather than meaningful impact, the metric loses integrity. She cites recent studies showing that overall research productivity hasn’t increased over a century; rather, papers have shortened, and co-authorship has expanded, skewing perceptions of output.

Hence, relying solely on citations or impact factors can be misleading. A nuanced, multi-layered approach that considers different metrics and disciplinary contexts provides a better assessment.

Towards responsible research evaluation

Rachel advocates for a cultural shift—highlighted by initiatives like DORA (the Declaration on Research Assessment)—that promotes responsible use of metrics. These emphasize qualitative assessments, peer review, and context-sensitive evaluation, rather than fixating on numerical indicators alone. She underscores the importance of a balanced metric ecosystem: combining traditional citation-based indicators with new tools like altmetrics and policy citations, to capture societal impact more comprehensively.

The role of bibliometrics and AI

Bibliometricians study citation networks, understanding how science evolves over decades. Interestingly, a 2023 article highlighted the most-cited papers in bibliometrics—raising questions about whether citation counts equate to quality: the core debate of the field.AI and machine learning are increasingly implicated, with tools like Semantic Scholar attempting to categorize citation contexts—distinguishing mere references from impactful citations. AI currently struggles to interpret nuance, such as positive versus negative citations, but potentially could revolutionize impact assessment. She warns against over-reliance on AI, especially given the time lag in recognizing interdisciplinary or impactful work, which tends to accrue citations slowly.

Open Science and impact beyond citations

Virginia Tech has carried out several initiatives—like institutional repositories, open education grants, and community-engaged research—that aim to broaden access and societal relevance. She discusses altmetrics—indicators like social media mentions—and policy citations as emerging impact metrics. Rachel views altmetrics as supplementary; they shine a light on broader engagement but have limitations, like arbitrary scoring and difficulty capturing context. Research should connect with policymakers and communities: impactful research often reaches beyond academia into public policy or societal change, sometimes without researchers being directly involved.

The future of Open Science and scholarly communication

Virginia Tech’s open science initiatives include platforms facilitating open peer review and preprints. However, while such innovations promote transparency, accelerate dissemination, and democratize access, there are measurement challenges, such as multiple standards for books, or tracking usage across platforms. She advocates for community-led infrastructure, better repositories, and transparent metrics that adapt to new publishing models. Her concern is how to balance open access with the rising costs of publication fees, which can favour researchers in well-funded institutions and countries.

Conclusion: Navigating the world of research metrics

Rachel Miles’s recommendations include:

  • Don’t rely on a single metric; research impact is multifaceted;
  • If you’re involved in research evaluation or scholarly communication, consider diversifying your assessment approaches;
  • Support open science;  
  • Stay informed on technological and ethical developments in impact measurement.

Views: 126