EARMA Conference Oslo

Measuring societal impact; the quantitative vs qualitative approach

A discussion towards a common societal impact analysis framework

Format: Fifteen-Minute Discussion Tables

Category: Discussion Starter

Topic: Impact

The societal impact of academic research is increasingly getting attention. The impact planned for in the first Horizon Europe grant proposals where this was a requirement, should by now have materialized. But the EU lacks a clear framework for societal impact analysis. Therefore, a debate about how to measure societal impact is long overdue.

Working in research support in a university hospital in the Netherlands, the UMC Groningen, I notice a stark contrast between two ways of showcasing societal impact. My librarian colleagues focus strongly on measuring societal impact metrics of our institute as a whole. The way they are able to collect impact metrics is nothing short of impressive, but in my view this approach ignores the intricate nuances of a given impact project somewhat. On the other hand, my colleagues of the communication department have a keen eye for the story, but without quantitative support, the showcased impact lacks hard evidence and makes comparisons between institutions or projects impossible.

My experience in the United Kingdom taught me that both strategies could be reconciled. When I was working in an impact team at a Welsh university, I noticed there was a very strong focus on evidencing impact. Per impact project an impact case study was written, where often quantitative data was collected to show the reach of the claimed impact. This was then combined with qualitative data, to show the significance, or depth, of the impact. Both this quantitative and qualitative data are then brought together into one narrative and presented as an impact case study. For instance: “Our newly developed medical treatment has had the following positive effects on these ten interviewed patients. The treatment has been applied a thousand times, therefore, the health benefits of the treatment can be extrapolated, giving a sense of both reach and significance.”

Combining both quantitative and qualitative evidence in an impact case study, much like the British approach, allows for showcasing all facets of societal impact of research. However, benchmarking against other institutes is arbitrary (when compared to a purely quantitative approach). Where do you stand in this debate? Should your institution focus on qualitative evidence, allowing for the broadest range of impact possible but forsaking comparability? Or should we develop a finite list of impact indicators so we could compare our institution against others? Or thirdly, as proposed here, a combination of the two; best of both worlds or the worst?

Take home message: increasingly, research funders focus on the societal impact of the projects they fund. To assess whether higher education institutions have delivered on their promises requires a societal impact assessment framework. What should impact assessment look like, should we follow a qualitative or a quantitative approach, or a mix of both?

New demands in a new era: Cluster building for greater impact

Taking advantage of the experiences of the EURION cluster

Format: Fifteen-Minute Discussion Tables

Category: 3. Good Practice

Topic: Impact

With the increasing emphasis on impact of
research and synergies across research activities and projects, new requirements
are emerging for research managers and administrators to facilitate this strive
for greater impact. This session will scan the horizons for new requirements, opportunities,
skills and potential obstacles for joint collaboration across EU-funded
research projects. The participants will discuss and share their ideas,
experiences and lessons learned on joint efforts from research management as
well as communication and dissemination perspectives. 

Assessment of the impact of research and innovation projects sponsored by the European Commission: insights from a case studies approach

Format: Oral 30 Minutes

Category: 1. Case Study

Topic: Impact

Research impact assessment is a “wicked” issue as it involves necessarily tangible and intangible aspects. This may include e.g. scientific outputs and economic impacts, but also tacit knowledge, social and environmental impacts, by nature often more difficult to appraise. The topic of circular bioeconomy, at the intersection of the circular economy and bioeconomy concepts, will be used as a case study to demonstrate how the impact of research projects on the society at large can be assessed. Those attending will gain a practical perspective about how to assess research impact and about short, medium and long-term effects of research topics driven by public policies.

CHALLENGING ‘THE NORMS’ IN ACADEMIA

IS THE SECTOR LOSING ITS APPETITE FOR CHANGE?

Format: Oral 30 Minutes

Category: 1. Case Study

Topic: Impact

Over the last few years, many academics have become
increasingly disillusioned by ‘the norms’ within research culture that are
holding back progress. The academy’s tie to incentives and its influence over
career progression, a trend towards insecure contracts, and a lack of
diversity, are among the issues that have come to the fore. COVID has
compounded these issues, leading to greater insecurity and deepening
disparities. The need for real societal change to encourage a fairer, more
equitable environment for research to thrive is becoming increasingly more
urgent.

In this session we will share a whistle-stop tour of Emerald's '2021 Time
for Change' report. Now in its third year, it reflects the views of over
2,000 researchers world-wide.  The report looks at trends in attitudes to
research evaluation, academic culture, openness and transparency and the
evolving role of the publisher. It explores the challenges researchers face,
and the impact on personal lives and career aspirations.

Claire Jackson, Emerald’s Head of Community Engagement, will
present the main findings, explore regional differences, and give examples of where
positive change is happening plus, discuss the growing role of publishers as
facilitators of research impact. 

Connecting research support structures to stimulate an impact culture among researchers

Format: Oral 30 Minutes

Category: Discussion Starter

Topic: Impact

In many universities the turn to recognizing and rewarding impact beyond the academic realm has been taken. At Ghent university such desire to recognize and appreciate other forms of impact has been formalized into policies and support structures at the central and faculty level. However, challenges remain on how to implement and operationalize those policies. What kind of support (structure) do our researchers need? What initiatives lead to a conducive or stimulating environment or culture to focus on societal and economic impact of one’s research?

DESIGN OF ACADEMIC RESEARCH PROJECTS WITH SOCIAL IMPACT

Format: Oral 30 Minutes

Category: 2. Theoretical

Topic: Impact

Academic research cannot be oblivious to social problems and needs, so projects with the capacity for transformation and impact have to prevail, especially in a context of uncertainty and change. In order to design projects with social impact, we will introduce the Theory of Change, as a project design methodology used to explain how and why the activities of a project will lead to the desired changes, expressed as a medium and long-term benefit obtained by the target population.

Impacting Research Management through North-South capacity development partnerships (30 min presentation)

Format: Oral 30 Minutes

Category: 1. Case Study

Topic: Impact

There is rigorous
debate as to where "academic impact" (i.e. peer-reviewed
publications) ends and "research impact" (i.e. beyond the academe)
begins. The StoRM and the TReMOR Projects are two recent, international North-South
partnerships towards improving Research Management capacity across the
participating countries across SADC, the UK, and the EU. This presentation
showcases their capacity development initiatives, and how their online
resources, under the "new normal", have impacted RM capacity
development on the African continent and further afield.

Look inside – think beyond: A toolbox to support researchers in finding alternative ways of exploiting project results

Format: Oral 30 Minutes

Category: 1. Case Study

Topic: Impact

Impact is becoming increasingly important to funding institutions, but for researchers who are the experts for excellence in their research areas it is not an easy point to address in project proposals. Our exploitation toolbox aims to bridge this valley.

Its presentation will give an overview of how researchers can be coached their way across this bridge: to align their excellence in research with stakeholders’ expectations and to find alternative ways for exploiting their results.

Open innovation between Universities of Applied Sciences

A case study of national Centres of Expertise in NL

Format: Oral 30 Minutes

Category: 1. Case Study

Topic: Impact

Universities of Applied Science play a key role in innovation for tackling the societal missions of our time. This paper presents guidelines and best practices in the challenge to form nation wide cooperation between UAS research groups and societal actors.
It is presented in the form of a case study of the Dutch Centres of Expertise that have been set-up in 2018 as an instrument to create critical mass in practise based research.
The author is initiator of several centres and works across the institutional borders of universities, industry, public services and citizen stakeholder groups. With a multitude of centres focussed around six of the national innovation missions, a vibrant network of researchers is build that delivers impact. The presentation explains the reserach management principles that are applied to stimulate coorperation, interdisciplinary work and entrepreneurship.

The EU Policy Making Process (The Potential Impact of Research Results)

Format: Oral 30 Minutes

Category: Methodological Session

Topic: Impact

Pillar II of Horizon Europe is designed to address Global Challenges and to support European Industrial Leadership. In the proposal forms of Pillar II the researchers are asked:

“Describe possible feedback to policy measures generated by the project that will contribute to designing, monitoring, reviewing & rectifying (if necessary) existing policy & programmatic measures or shaping & supporting the implementation of new policy initiatives & decisions.”

The EU Policy Making Process is a formal process that can last between two to five years. During this process ‘evidence’ is needed by the different players to help in the policy decision making.
One of the biggest problems in EU policy making is the difference in the language between the Policy Makers and the Researchers.

The policy makers state that “Policy-makers need information which will inform their decision making process. The information must be accessible, politically useful & contribute to finding practical solutions to problems.”
The Policy makers also state that “research reports are often inaccessible and not sufficient to ensure that research findings are used to inform policy.”

This presentation provides and overview of the EU policy making process and how researchers should design their proposals to address these communication concerns.

Evolution of research impact (30 min presentation)

Format: Oral 60 Minutes

Category: Discussion Starter

Topic: Impact

This presentation will look at research impact from RMA’s point of view: how the concept has been understood previously and where we are now. We will discuss the state-of-the-art impact from the 1990s to 2022 and beyond, how the idea of research impact has evolved from counting publications to the current understanding of science-society relations, and how the creation of impact is nowadays viewed as an interactive process. We will also examine the demands of research funders, for example, how Horizon Europe uses the impact pathway model, and bring up examples of national funders’ requirements. We will look at some of the tools and guides available so far and used by RMAs to help researchers understand their impact and write more enticing funding proposals.

How to improve researcher competencies in policy advice by creating a trainer network across Europe?

Format: Oral 60 Minutes

Category: 1. Case Study

Topic: Impact

This session gives an insight into how both a research policy unit at a university and a training unit at JCR are jointly looking towards increasing researcher competencies in advising and shaping policy.

Towards a value-driven research culture

Impact expert – an essential role in a value-driven research culture. Where we want to be in 5 years and what might help us to get there.

Format: Oral 60 Minutes

Category: Interactive Session

Topic: Impact

How are research professionals developing in this evolving culture to meet today’s challenges, and those that lie over the horizon? How are we developing and honing our skills and expertise in order to bring impat to the fore in our day-to-day work? How will this shape our work profiles and responsibilities in the future and what skills will we need to acquire to meet the challenges ahead? Importantly, how will we exchange impact knowledge and expertise that often eminates from a number of sources within our respective institutions?

Together with the participants of this session, we would like to discuss these questions, share our knowledge and finally co-create some storylines that will paint a picture of how the ”impact expert” of the future may look. This is an interactive session, so please come and share, be creative and feel welcome to think outside the box.

Intellectual Assets Management (IAM): does it work?

Format: Pecha Kucha

Category: Methodological Session

Topic: Impact

The
learning outcome for the session participants will be a glim to the
methods they could use in capturing, managing, and preparing the strategy for
the intellectual assets’ management and impact. The concept of useful
use is no longer associated only with technological innovations commercialised,
but they also include social, public, inclusive, grassroots and
challenge-driven innovations created in collaboration between academia and
other societal actors. A one-sided focus on patents and intellectual property
can prevent the spread of broader societal benefits, as lock-in knowledge can
slow down the development and dissemination of project results and innovations.
It is essential to capture the Intellectual Assets that research generates and
decide how they will be managed. LTU participated in the project where large
number of universities participated to have a common view on how universities
should manage their intellectual assets https://imp-act.se.
One of the pilots was carried out at LTU.  We had an excellent opportunity
to support researchers to ensure that the intangible assets generated by
project is managed properly. We tested the IAM methods and developed our
research support services in the management of intellectual assets. In this
session the overview of the service package and methods will be presented. 

Promote Your Research

UCD’s new website to help researchers increase the visibility of their work

Format: Pecha Kucha

Category: Practical Initiatives

Topic: Impact

Millions of research outputs are published every year. Unfortunately, this means a lot of excellent research gets lost in a sea of publications, prototypes, creative works, and datasets. But there are steps that researchers can take to make their research stand out. UCD’s new “Promote Your Research” website gives advice on how to increase the visibility of outputs, making it more likely that people will discover, use, and cite them. As well as increasing citations, the website helps researchers to build their profile, find future collaborators, and connect with those who stand to benefit from their work (giving it a better chance of having a positive impact on society).It includes tips on how to prepare for publication, how to identify your audience, how to develop your message, how to promote research using social media platforms, how to create multimedia resources, how to reach wider audiences, and how to monitor where outputs are being picked up and used.  In this session, David Bennett, UCD’s Research Impact Officer, will introduce UCD’s new website. He will discuss why UCD felt the need to make it and how they pulled it together, and he will describe the various tools and resources it contains.

From Need Identification to Impactful Projects – Co-Creative Process to Support Project Idea and Proposal Preparation

Format: Poster

Category: 3. Good Practice

Topic: Impact

World’s multidimensional global challenges need to be addressed using all intertwined potential of science and research, innovation, business, public sector and civil society. Identification of the large-scale challenges and more specific needs at local level is a joint effort. Utilisation of foresight information is of utmost importance in finding the pathways towards sustainable and inclusive solutions and impact for societies.

To capitalise the value of impact, it’s essential to ensure that impact is considered as the starting point for project ideation process. The link between project portfolio and institutional strategy needs to be strengthened, as strategy defines the pathway towards desired impact On the other hand, it also constitutes the main tool to prioritise operative actions during times of scarce resources.

Tampere University of Applied Sciences (TAMK) implements the project idea process and platform (EARMA Conference 2021). New elements include incorporation of a pre-ideation phase and clarification of impact targets of project ideas. Yet, the main aim remains to be to secure the alignment of externally funded project portfolio with the institution strategy. The objective of the new elements is to increase the number of new, more mature project ideas, to improve the quality of early-stage idea expert support, and to guarantee an open, collaborative platform for staff members for cross-disciplinary project idea maturation.

In the new phase regarding raw project ideas, idea description can still be short and unorganised. The online platform is accessible for all staff members offering the opportunity for collaborative and cross-disciplinary reflection among peers and early support from RDI experts. There is a direct channel to the second phase where the project idea is elaborated in more detail. The main project objectives must be described, and a financial plan defined. Alignment with institution’s strategy is ensured, and targeted impacts have to be described. The process also requires reflecting the idea against the UN Sustainable Development Goals.

In the transparent decision-making process, the Vice President and Head of External Funding give either approval to proceed with the application preparation with necessary resources, or the idea is returned with constructive feedback.

The coverage of project idea management with one process and platform supported by RDI specialists effectively supports TAMK’s strategic, managerial, financial and quality processes and facilitates impact creation. Systematic supportive approach from project idea need identification to early-stage impact consideration has ensured that TAMK’s project portfolio has grown both in number and in quality. Process also strengthens joint institutional values such as co-creation, transparency, equality, cross-disciplinarity and openness. TAMK is more ready to identify and tackle multidimensional local and global challenges with the collaboration of relevant actors to achieve impactful solutions.

The strategic idea process has been implemented in TAMK since 2014. Besides the successes, there still are obvious places for improvement. These include the growing demand for continuous competence development of RDI support experts and research managers in areas such as impact thinking, early-stage idea facilitation, forecasting capabilities, and knowledge of various funding schemes and the policy frameworks behind them.

The Impact Workshop- Success Story on Collaboration of Research Funding and Innovation Advisors

Format: Poster

Category: 3. Good Practice

Topic: Impact

We began the development of The Impact workshop 2013. The starting point was that funding organisations expect research projects to have an impact that is not just innovation development, but a wider concept where the results of projects are exploited by society at large. To meet this need an “Impact Clinic” was developed as a collaborative effort of research funding and innovation advisors at the University of Jyväskylä. In the presentation, we explore the development phases of the workshop from an open event to a tailored workshop offered to support the design of individual projects.