Gaps to be addressed in the Common Evaluation Methodology

In the ARCADE 23 November 2020 workshop on Common Evaluation Methodology (CEM), a long list of methodology gaps were identified (see Proceedings, Chapter 4). Evaluating CCAM and following FESTA may be quite difficult and complex. Methodology helps but does not change complexity. 

The ARCADE team further analysed the gaps and identified where we can expect to get information to address these gaps. The gaps are organised along the FESTA-V

We could classify the information needed to (partially) address the gaps into four categories: 

  1. FESTA: the topic is addressed in the FESTA handbook version 8.  
  1. Process: processes such as harmonisation, standardisation or management, not further specified. These processes could, for example, be performed by standardisation bodies, but are at the moment outside the scope of an evaluation methodology.  
  1. Projects: the topic is addressed by CCAM projects running after 2020.  
  1. CCAM call: the topic will be addressed by one of the upcoming CCAM projects from the 2021 and 2022 calls 

The table was first created at the end of 2021 and should be seen as an inventory of information sources that could be used to address the gaps. You are invited to suggest further additions and new developments (please use Feedback form).


Gap (organised along the FESTA V): 
 
Progress expected from:
1. Implementation plan  
1.1. Framework for coordinating FOTs with multiple locations, OEMs, countries  
Harmonisation 
Project management 
HORIZON-CL5-2021-D6-01-06 (FAME)
1.2 Hard to compare FOTs or between multiple locations, countries or OEMs Harmonisation 
HORIZON-CL5-2021-D6-01-06 (FAME)
2. Function identification and description  
2.1. Common description method (ontology, terminology, format) for ODD, use cases and services (and secondly requirements, vehicles, functions, accidents)  Standardisation 
HORIZON-CL5-2021-D6-01-06 (FAME)
3. Use cases  
3.1. Common source for describing ODD in terms of driving behaviour, accidents, scenarios and edge cases  Standardisation 
HORIZON-CL5-2021-D6-01-06 (FAME)
3.2. Method to define, find and use edge cases  EU Hi-Drive
4. Research questions and hypotheses  
4.1. Method to define and prioritise research questions  FESTA 
4.2. Method to define future scenarios  FESTA  
HORIZON-CL5-2021-D6-01-06 (FAME)
5. Performance indicators  
5.1. Common set of safety indicators with known relation to safety impact  FESTA 
EU Hi-Drive 
5.2. Accepted set of indicators or model for communication and positioning  EU HEADSTART 
6. Study design  
6.1. Approaches for achieving a realistic and rich user experience with prototype vehicles  
Complex by nature
No approach identified
6.2. Method to compare human and automated driving  L3-Pilot 
HORIZON-CL5-2022-D6-01-03 
6.3. Reference to compare new services with. 
What is the baseline for a service in impact assessment? 
HORIZON-CL5-2022-D6-01-06 (V4Safety)
6.4. Method to define and measure a clear baseline for a FOT impact assessment. What is better?  FESTA 
HORIZON-CL5-2022-D6-01-06 (V4Safety)
6.5. Shared assumptions on human driving or shared human driving models  HORIZON-CL5-2022-D6-01-03 
6.6. Method to validate cybersecurity  HORIZON-CL5-2021-D6-01-04 
6.7. Method to balance scale of experiment versus generalisation acceptable in impact assessment  FESTA 
6.8. Methodology that can be scaled down for small projects or handles multiple scales of research questions  Micro-FESTA 
7. Ethical and legal issues  
7.1. Need for an innovation friendly framework for running pilots, FOTs and operation  
Project management 
FESTA 
HORIZON-CL5-2021-D6-01-06 (FAME)
8. Data acquisition
8.1. Data sharing: approaches to handling lack of data and lack of willingness to share  Data sharing Framework 
8.2. Guidelines for efficient and effective process to public road test permission  National type approval bodies
Harmonisation
8.3. Common solutions for data management (release, flow, models, formats)  Standardisation 
Data sharing Framework 
L3Pilot Common Data Format 
8.4. Agreed principles for data sharing among industry and research, respecting industrial sensitivity of the data  Data sharing Framework 
8.5. Practical solutions for GDPR-compatible handling of video  Data sharing Framework 
8.6. Overview of urban traffic environments   HORIZON-CL5-2021-D6-01-06 (FAME)
9. Data analysis  
9.1. Lack of accident data for socio-economic impact assessment  HORIZON-CL5-2022-D6-01-06 (V4Safety)
9.2. Approaches to simulate at multiple levels of detail (sensor, AD function, vehicle, traffic, city), including effect of safety strategies at vehicle level on traffic level  HORIZON-CL5-2022-D6-01-06 (V4Safety)
EU Hi-Drive 
9.3. Standardisation of modelling of scenarios and AD functions in simulations  Harmonisation 
HORIZON-CL5-2021-D6-01-06 (FAME)
9.4. Handling diversity in sensors, data sources, locations, formats – preferably in an automated process  Industry
10. Impact assessment  
10.1. Shared framework to come from KPIs to assessment (data evaluation architecture, combining various test results)  FESTA 
10.2. Shared assumptions to be used in impact assessment and generalisation   HORIZON-CL5-2021-D6-01-06 (FAME)
10.3. Shared assumptions on changing human behaviour with higher share of CCAM   EU Hi-Drive 
10.4. Shared assumptions to estimate impact on VRUs   HORIZON-CL5-2022-D6-01-06 (V4Safety)
10.5. Accepted method to come from test results with prototypes to impact assessment for mature CCAM and full scale  FESTA 
HORIZON-CL5-2021-D6-01-06 (FAME)
10.6. Shared future scenarios for generalisation of impact assessment  HORIZON-CL5-2021-D6-01-06 (FAME)
10.7. Methods for evaluating impacts on certain accident types  FESTA 
HORIZON-CL5-2021-D6-01-06 (FAME)
HORIZON-CL5-2022-D6-01-06 (V4Safety)
10.8. Methods to evaluate AI processes and decisions  HORIZON-CL5-2022-D6-01-05 
Gaps in Evaluation Methodology

Feedback form

Have feedback on this section? Let us know!

Send feedback

Feedback

Please add your feedback in the field below.

Your feedback has been sent!
Thank you for your input.

An error occured...
Please try again later.