Projects

Introduction

This publication finalises the work IDEA Consult experts have been doing for the Directorate General
on Research and Innovation on the evaluation of excellent science aspects the European framework programmes for R&I.

The study was performed in two phases to provide input for two evaluations:

• Phase 1 fed into the ex-post evaluation of Horizon 2020, the 8^th EU Framework Programme for the period 2014-2020 with a budget of nearly EUR 80 billion.
• Phase 2 supported the interim evaluation of Horizon Europe, the 9^th Framework Programme for the period 2021-2027 with a budget of EUR 95.5 billion.

Through this study, IDEA Consult experts, together with its partners, have analysed the relevance, effectiveness, efficiency, coherence, and European added value of the programmes associated to excellent science. The programmes analysed by IDEA Consult under Horizon 2020 were:

• Research Infrastructures
• Spreading Excellence and Widening Participation (SEWP)
• Science with and for Society (SwafS)

For the Horizon Europe, IDEA Consult analysed the European Research Council (ERC), Research Infrastructures, and Widening participation and Spreading Excellence (WIDERA).

Process & Results

The analysis performed by our experts showed that the EU Framework Programmes have been successful in terms of production of excellent science, measured in terms of the number highly-cited publications and initial evidence suggests advancements in the generation of high-quality knowledge in Horizon Europe. However, this excellence is still not as widespread as desired and there continues to be a need for dedicated support for less advanced R&I systems to build R&I capacity. Moreover, both Horizon 2020 and the evidence stemming from the first projects under Horizon Europe appear to have strengthened the human capital engaged in R&I, as shown by the positive effects they have had on the career prospects and reputation of researchers, as well as skills development, international and intersectoral mobility, and working conditions.

These results have been achieved in a context characterised by important external shocks such as the COVID pandemic, the energy crisis or the war in Ukraine – the analysis shows that the EU Framework Programmes were able to cope with these challenges and keep on delivering value meeting its efficiency goals. Finally, it shall be noted that the EU Framework Programmes demonstrate a very strong EU added value by financing research projects characterised by having a greater scale, scope and scientific complexity than those typically funded at national or regional level. Additional efforts would need to be invested, however, in ensuring a greater coherence of some of the actions at EU level to achieve an optimal impact.

Finally, the study indicates that is important to combine a challenge-oriented approach with sufficient room for ‘blue skies’ research, where “real-world” applications are not immediately apparent, in order to be ready for future challenges, foster collaborative top-down research while keeping actions oriented towards specific challenges or topics.

Methodology

The study designed and applied a tailored methodology consisting of quantitative (e.g., bibliometric/scientometric analysis, econometric modelling, social network analysis) and qualitative methods (including interviews, case studies and international benchmarking exercises).