General evaluation framework
The evaluation framework has been contextualised to cover the needs of the individual pilot sites. Each pilot’s evaluation plan contains services’ metrics that incorporate stakeholders interests and views. The contextualized evaluation plan is based on what each stakeholder needs to evaluate In short, the evaluation framework aims to establish a “common ground” through which the results of the activities of the three pilot sites of the CO3 project are going to be evaluated.
Pilots evaluation plan: 3 steps
Evaluation Plan Steps
Following the realistic evaluation approach, the design of the pilot’s specific evaluation plans has been framed around three steps:
- Identification of the main hypothesis to be tested through the evaluation (together with pilots representatives, taking into account stakeholders expectations)
- Definition of the evaluation actions to be carried out in order to test the main hypothesis, defined in step 1
- Definition of the evaluation roadmap, in order to collect data for the evaluation actions defined in step 2
Evaluation actions
On a subsequent stage of the discussion with the pilots we went on to map the evaluation actions that will be carried out in every pilot.
The goal here was to map these evaluation actions in order to:
- Design actions able to collect data in order to test the hypotheses developed
- Attempt to link evaluation actions with engagement actions to achieve a more coherent pilot development course of action
- Use them as feedback to create the overarching framework and decide on the general action types and phases
The definition of the evaluation actions helps us to identify which are the data to be collected during the pilots. Specifically, two main types of data are going to be collected:
- Usage data: are the data that will be collected from the CO3 platform during the whole pilot execution, considering all the CO3 systems: Liquidfeedback, wallet, AR app, FirstLife, Gamification (Ontomap)
- Qualitative data: are the qualitative data about stakeholders expectations that will be collected in specific moments of the pilots lifetime. They could be collected through questionnaires, workshops, focus groups, interviews with experts, etc.
Evaluation roadmap
The roadmap aims to provide local pilots and partners in Task 4.2 a clear way to organize the contextualized evaluation actions and is going to be adapted to specific pilots needs and characteristics. Not all phases must be implemented in each pilot in the same way and with the same order. It depends on the local environment. The roadmap incorporated the following phases that are linked with evaluation methods, and particularly with data collection methods.
The main outputs for this step are the following:
– For usage data: a unique table (one for all the pilots) that makes reference to the main data sources to be used. The final list of usage data that will be collected through the CO3 platform in the three pilots that could slightly differ from the present one considering eventual refinements in the evaluation framework.
– For qualitative data: for each pilot the evaluation stages for collecting the requested data are defined following a common tabular template that considers the following variables:
- Evaluation stage: Reference to a specific evaluation stage, defined in the evaluation roadmap
- Main action – code: for each evaluation stage one or more actions are identified by a code.
- stage main action – title: Title of the main action
- stage main action – description: Full text description of the main action
- Evaluation action: Reference to the evaluation action(s) to which this main action refers to
- Engagement action: Reference to specific engagement actions (code) presented in the engagement plan
- Data collection: Description of the data collection method.
Possible options: report from workshop facilitators, feedback from stakeholders, surveys, etc.
- Data analysis: Description of data analysis method. It could be: thematic analysis, quantitative analysis, etc
- Supporting material: Link to supporting material and templates prepared for facilitating data collection (if available)
- Time table: Indicates when the main action is going to be carried out.
