Talking with the WP6 team members
The general objective of Work Package 6 (WP6) of the VAX-TRUST project was to design and implement the evaluation plan for the interventions developed in VAX-TRUST.
The VAX-TRUST project runs until February 2024 and is funded by the European Union’s HORIZON 2020 program. WP6 was conducted by a team from the Instituto de Higiene e Medicina Tropical from the NOVA University of Lisbon, IHMT-NOVA, led by Tiago Correia and which includes the researchers Cátia Sá Guerreiro, Rita Morais and André Beja.
Between 24th and 26th of October, WP6 team hosted a seminar to reflect on developing interventions for healthcare professionals to address vaccine hesitancy, an event included in the workplan of VAX-TRUST. We spoke to the WP6 team members, who detailed the challenges of conducting an evaluation process of health interventions.
How important is it to include evaluation in a project of this nature?
This project planned and implemented interventions aimed at increasing awareness on the complexity of vaccine hesitancy among healthcare professionals involved in discussing childhood vaccines with parents.
We firmly believe that evaluation is fundamental at every stage of an intervention, starting with its design and continuing throughout its implementation. Evaluation processes make it possible to adjust the objectives of interventions to the context in which they take place, define realistic expected results and quantify and analyse them at the end of implementation. This conviction is supported by literature that emphasises that the purposes of evaluation include promoting accountability and learning. Evaluation aims to understand why — and to what extent — intended and unintended results were achieved and to analyse the implications of the results. It should offer credible, useful evidence-based information that allows the timely incorporation of findings, recommendations, and lessons into the decision-making processes of organisations and stakeholders.
Can you describe the evaluation process implemented in this project?
The guidelines created by the WP5 team oriented the definition and subsequent implementation of interventions in the seven Target Regions. Evaluation begins in the first moment – when you are designing an intervention, you should ask “Will it be possible to evaluate the implementations of this intervention in the future?”. To help answer this question, WP6 team conducted an evaluability study, and made suggestions for the improvement and strengthening of the design of each intervention, seeking to ensure that they would be in the best condition possible to be evaluated.
These interventions’ implementation was subjected to an internal evaluation, conducted by the implementation team, and an external evaluation, conducted by the WP6 team with the support of local evaluators. Before the implementation phase, the local evaluators involved in the external evaluation, who were expected to monitor and assess the implementation in each target region, were trained in Lisbon (27 February and 3 March 2023).
In short, once implementation was complete, for each Target Region, WP6 had an internal evaluation report and an external evaluation report, preceded by the results of the evaluability study. It was then possible to carry out an implementation analysis with the objective of understanding how these interventions were implemented. For this purpose, the objectives of each intervention, the planned activities and their results were scrutinised, in constant interaction with the context in which the intervention took place.
The following framework systematises the evaluation process that we carried out, which was a huge challenge for all of us.
Why do you say it was a huge challenge?
Yes, it was a real challenge! Evaluation is still often seen from the perspective of non-constructive criticism, of saying what’s not going well. In this project it was a challenge to get across to the consortium the message of the usefulness and importance of evaluation processes.
On the operational side, regarding the conduct of the entire evaluation process, we would emphasise that specific tools were created for each stage, and they can be used in future processes. You can find details of these tools in the deliverables produced. We consider this as an innovative process, based on scientific knowledge and emphasising the importance of evaluation in processes of this nature.
What are the key messages from this analysis?
First and foremost, this work has strengthened our conviction of the importance of evaluating interventions, which should be integrated into the planning process and accompanied throughout implementation, with a conclusive analysis of the results at the end.
No less important is the conclusion that the planning process is fundamental to successful implementation. Considering information about the context in which implementation will take place, along with realistically adapting the plan to the time available for the implementation phase, are crucial aspects. In this sense, for any intervention to be successfully implemented, the context in which it takes place must be considered with detail. The context analysis associated to a detailed schedule, are conditions to the design of realistic interventions that can be successfully implemented.
The way we define objectives is also important: the clearer the definition of the interventions’ objectives, the more appropriate the definition and application of the evaluation indicators will be, making it possible to analyse the results achieved with greater skill and detail.
It’s also important not to forget that risks are a reality in these processes and must be considered, along with a clear mitigation plan. Even so, in the implementation phase, the flexibility of the implementers will be an asset in adapting the plan to reality, while keeping the focus on results.
How was your experience of working in evaluation on this project?
As said, it was an innovative process that required creativity and perseverance. Although we already had a background in evaluation, we have learnt a lot, both as a team and individually! It has been a challenge to get to know the different teams, to understand the contexts, to work with the local evaluators.
A take-home message to healthcare professionals who encounter vaccine hesitancy in their daily work?
Well, first of all, we would like to emphasise that the relevance of the issue of vaccine hesitancy for healthcare professionals and the usefulness of reinforcing their knowledge and training to deal with this type of situation were demonstrated. So, let’s go on working on it!
We would also like to leave a message of boldness in the way you think about these processes, a reinforcement of the importance of planning them well, basing everything on the specific context and on scientific bases. And why not use tools that have already been built and tested?
Above all, we can see the usefulness and relevance of these processes. We encourage to invest in them creatively, never forgetting evaluation as a fundamental part of any process in which we want to succeed.
