November 15, 2018

Combining relative assessments for melanoma classification

The project addresses melanoma classification in skin lesion images. Typically machine learning algorithms for this application would learn from images which have been labeled as melanoma or not. A less explored option is to learn from relative assessments of images, for example, whether images are similar to each other or not. Such assessments can be used to learn a good representation of the images, and the representation can then be further trained with a traditional dataset. An advantage of this method is that relative assessments may be more intuitive to provide than diagnostic labels, which could allow a large number of assessments to be collected via crowdsourcing.

In this project you will develop a deep learning algorithm which uses two types of input: melanoma labels provided by experts, and relative assessments provided by the crowd. The relative assessments were collected as part of the MelaGo project at TU Eindhoven, where participants could rate images via a (gamified) app. One of the goals is therefore also to investigate how gamification affected the quality of the assessments. Another goal is to investigate how to best combine the assessments, and whether filtering annotators by quality can improve the results.


Some experience with machine learning is required, experience with Python is preferred.

Supervisor: Dr. Veronika Cheplygina (v.cheplygina at


Khan, V.-J, Pim Meijer, Michele Paludetti, Reka Magyari, Dominique van Berkum, and Veronika Cheplygina. “MelaGo: Gamifying Medical Image Annotation”, 2018. PDF

Schultz, M., & Joachims, T. (2004). Learning a distance metric from relative comparisons. In Advances in neural information processing systems (pp. 41-48).

Ørting, S. N., Cheplygina, V., Petersen, J., Thomsen, L. H., Wille, M. M., & de Bruijne, M. (2017). Crowdsourced emphysema assessment. In Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis (pp. 126-135).