Experimental design

The EU-CEM provides guidelines for designing appropriate experiments to collect the data needed to answer each research question. Selecting methods for data collection—whether through field experiments, simulations or other approaches—requires careful planning to ensure the validity and reliability of data. Key considerations include defining treatment and baseline conditions, determining sample sizes, selecting participants, and structuring data collection phases.

Agreement on the experimental design is important for the feasibility of the evaluation plan execution in CCAM projects. Real-world testing opportunities are often limited to systems of low technology readiness level (TRL), making it impossible to accommodate every preference of the evaluation team. Some experimental design elements may have been established during the project preparation phase, particularly if they are also key project characteristics. This chapter focuses on selecting suitable design elements that can still be determined during the project.

The experimental design must support the data collection required for each research question. If not, iterative adjustments to the evaluation plan are necessary. For example, if a research question requires data from ordinary drivers but regulations prohibit such testing on public roads, alternative sources, such as test track experiments, should be considered.

A single CCAM project may involve multiple experiments, each potentially involving several teams, such as a technical team, a test site team, and an evaluation team. Early and consistent communication is key to ensure a feasible evaluation plan and efficient use of resources, and to avoid a mismatch between evaluation needs and the practicalities at test sites. Establishing standardised working procedures with clearly defined roles, responsibilities, goals and schedules early on is highly beneficial. Additionally, ensuring a common understanding through test site visits or direct discussions between evaluation and test site teams is recommended.

The experimental design must specify the baseline and treatment conditions. In some cases, the baseline may reflect future rather than current conditions. Factors such as demographic shifts (e.g. aging population or more non-motorists), increased adoption of automation, and vehicle fleet electrification can significantly alter the results if unaccounted for. Predicting these shifts when defining a future baseline helps ensure that the experimental design captures the emerging impacts caused by CCAM only.

The EU-CEM includes guidelines on the following topics:

• How to choose the experimental approach for a research question
• How to consider the elements of the experimental design in the evaluation plan
• How to plan the experimental design of field experiments
• How to plan the experimental design for virtual environments with participants
• How to select participants for your experiment
• How to provide participants with the most realistic user experience possible
• How to define the treatment and baseline conditions