To be held in conjunction with the 25th Conference on User Modeling, Adaptation and Personalization, UMAP 2019, June 2019, Larnaca, Cyprus

Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behavior of the users involved in trialing the systems. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research.

We envision that EvalUMAP 2019 will be the starting point for an annual comparative evaluation challenge at future UMAP conferences. The two previous iterations of this workshop (EvalUMAP 2016 and 2017) have created a platform for this topic to be discussed and advanced. In this iteration of the workshop series, final steps towards shared challenge generation will be made. A white paper mapping a proposed shared challenge approach is now available, consisting of exemplar data, desirable metrics and a plan for what engaging in the challenge task would look like. Feedback in the form of reflection, critique and alternate approaches will be solicited from the community via short submissions to the EvalUMAP workshop.

The planned outcome of the workshop will be a roadmap to develop initial shared task(s) that will be published ahead of UMAP 2020, providing opportunity for participants to test and tune their systems and complete the task in order for comparative results and associated publications to be prepared for and presented at UMAP 2020. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviors, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of metrics and observations for participants to perform in order to facilitate comparison will be outlined.

Workshop topics are evaluation focused and include, but are not limited to:

Papers should be submitted in pdf format through the EasyChair system (https://easychair.org/conferences/?conf=evalumap2019) no later than midnight Pacific Daylight Time on March 20, 2019. Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the UMAP 2019 EvalUMAP workshop proceedings.