Evaluation methods

The goal of impact assessment is to estimate the magnitude of effects on specific indicators when a CCAM system is introduced or in use (treatment), compared to the situation before its introduction or when it is not in use (baseline). An effect refers to the relative difference in an indicator value that results from the use of CCAM.

Once the research questions are established, indicators are defined to answer them. Indicators translate research questions into measurable metrics that can be evaluated using different methods. They bridge the gap between theoretical research questions and the practical, measurable dimensions of what those questions aim to capture. If hypotheses have been formulated, they may also influence the selection of indicators.

The next step involves selecting suitable methods and tools for estimating the indicator values. The evaluation method specifies how each research question will be addressed after data collection from CCAM experiments. While some research questions might be answered using indicators that can be directly measured or calculated, others may require stages with intermediary indicators. Each step in this process may involve different methods, tools, and datasets to derive or assess interim and final outcomes. 

Scenarios are useful for structuring evaluations, a scenario is defined as a structured depiction, specified by a set of pre-determined conditions and variables, representing actual or theoretical states, situations, or interactions. All evaluation and impact areas require scenario setting, though the scope and detail of scenarios may differ across areas.

EU-CEM defines seven distinct scenario categories for evaluation:

• Societal scenarios describe the broader societal context.
• Service scenarios describe a single CCAM service or a chain of transport services that CCAM is part of.
• Traffic scenarios represent traffic characteristics and road infrastructure on specific road segments or networks.
• Driving scenarios depict short, specific periods of driving defined by primary driving tasks (e.g. car following, lane change) or triggered by specific events (e.g. an obstacle in the lane).
• Test scenarios describe sequences of triggers, events, and actions among road users (e.g. ego vehicle, adjacent vehicles, pedestrians) to achieve a specific testing goal.
• User scenarios describe population segments and their state related to CCAM, such as mobility behaviour, attitudes, and socio-demographic characteristics.
• User behaviour scenarios describe the direct or indirect interaction between different users and the CCAM system.

Field experiments typically address lower-level scenarios, such as driving or user scenarios, on a particular road section or within a limited area or based on a limited group of participants. However, broader societal impacts of CCAM require assessing them within societal scenarios that cover entire networks and all trips made in them, including non-automated vehicles and non-motorised travel.

Depending on the impact areas and research questions, it may be necessary to scale up the impacts from lower-level scenarios to broader societal contexts reflected by the societal scenario. This requires a method for scaling up results to a societal level, reflecting a broader spatial and temporal frame than the lower-level scenarios.

In addition to direct impacts, it is important to consider cascading indirect and possibly unintended impacts of CCAM. These include increased car usage due to improved comfort and behavioural adaptations among non-users. These indirect impacts can be difficult to detect and quantify, as they may result from gradual processes. For example, behavioural adaptations among CCAM users and non-users can take months or years to fully manifest. Nevertheless, qualitative considerations of these impacts are recommended.