Meta-analysis for ground truth evaluation and selection

The collection of high-quality data for machine learning purposes is a complex process, especially when the task is prone to human intervention or evaluation. In such tasks, ground truth is often gathered from groups of experts, whose outputs are generally heterogeneous. This procedure raises questions on how to compare the results obtained by automated algorithms to multiple ground truth items. Secondly, it raises questions on the meaning of the divergences between experts. This work analyzes how the use of meta-analysis can lead to a better understanding of the information. Also, how the application of dedicated meta-analysis measures can lead to the generation of reliable ground-truth selection and/or evaluation processes

Keywords:

Information fusion , Image Processing , Ground-truth management , Twofold Consensus Ground Truth (TCGT) , Meta-quality analysis