EvalUMAP 2019 Workshop

Towards comparative evaluation in user modeling,
adaptation and personalization

To be held in conjunction with the 25th Conference on User Modeling,
Adaptation and Personalization, UMAP 2019, June 2019, Larnaca, Cyprus

EvalUMAP 2019 Agenda

Sunday 9th of June

14:30 - 14:40

Workshop Introduction
(Welcome, purpose, goals, agenda, Introduction of participants)
Bilal Yousuf (10 Minutes)

14:40 - 16:00

Paper Presentations (Each Presentation 15 minutes plus 5 minutes Q&A)

Oznur Alkan, Elizabeth M. Daly and Adi Botea
An Evaluation Framework for Interactive Recommender Systems

Alexander Nussbaumer, Christina M. Steiner and Owen Conlan
Towards a Multi-Modal Methodology for User-Centred Evaluation of Adaptive Systems

John Dowell and Marios Constantinides
Extending the evaluation paradigm for personalisation: a categorisation use case

Kieran Fraser, Bilal Yousuf and Owen Conlan
Generation and Evaluation of Personalised Push-Notifications

16:00 - 16:30

Coffee/Tea Break

16:30 - 17:00

Discussion Session
White paper challenge proposal overview (Owen Conlan)
Program Committee memebers' perspectives

17:00 - 18:00

Discussion Session
(on next steps and challenge formation)

Call for critiques of proposed shared evaluation challege approach & Call for Papers

Max paper length: 2-pages

Submission deadline: March 20, 2019 March 25, 2019

Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behavior of the users involved in trialing the systems. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research.

We envision that EvalUMAP 2019 will be the starting point for an annual comparative evaluation challenge at future UMAP conferences. The two previous iterations of this workshop (EvalUMAP 2016 and 2017) have created a platform for this topic to be discussed and advanced. In this iteration of the workshop series, final steps towards shared challenge generation will be made. A white paper mapping a proposed shared challenge approach is now available, consisting of exemplar data, desirable metrics and a plan for what engaging in the challenge task would look like. Feedback in the form of reflection, critique and alternate approaches will be solicited from the community via short submissions to the EvalUMAP workshop.

The planned outcome of the workshop will be a roadmap to develop initial shared task(s) that will be published ahead of UMAP 2020, providing opportunity for participants to test and tune their systems and complete the task in order for comparative results and associated publications to be prepared for and presented at UMAP 2020. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviors, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of metrics and observations for participants to perform in order to facilitate comparison will be outlined.

Workshop topics are evaluation focused and include, but are not limited to:

  • Reflecting upon and critiquing the proposed shared challenge proposal white paper
  • Understanding UMAP evaluation
  • Defining tasks and scenarios for evaluation purposes
  • Identification of potential corpora (datasets) for shared tasks
  • Automated and semi automated processes on creating appropriate datasets and simulating user behaviours etc in order to accommodate a shared task
  • Interesting target tasks and explanations of their importance
  • Critiques or comparisons of existing evaluation metrics and methods
  • How we can combine existing evaluation metrics and methods?
  • Reusing or improving previously suggested metrics and methods
  • Reducing the cost of evaluation
  • Proposal of new evaluation metrics and methods
  • Technical challenges associated with design and implementation
  • Anonymization of datasets, Privacy, Ethics and Security issues on the use of datasets

Workshop format:

This will be an interactive workshop structured to encourage group discussion and active collaboration among attendees.

Paper Submissions

The workshop is now accepting paper submissions. Position papers (max 2 pages) that either (1) critique the white paper we have made available proposing an approach for a shared challenge task (http://evalumap.adaptcentre.ie/whitePaper.pdf), (2) propose an alternate challenge approach, or (3) reflect on experiences in the space are invited. Submissions should be in ACM SIGS format. LaTeX and Word templates are available at (https://www.acm.org/publications/proceedings-template).

Papers should be submitted in pdf format through the EasyChair system (https://easychair.org/conferences/?conf=evalumap2019) no later than midnight Pacific Daylight Time on March 20, 2019. Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the UMAP 2019 EvalUMAP workshop proceedings.

Important Dates

March 20, 2019 March 25, 2019: Deadline for paper submission (midnight Pacific Daylight Time)

March 27, 2019 March 29, 2019: Notification to authors

April 3, 2019: Camera-ready paper due

June, 2019: EvalUMAP Workshop at UMAP

Further Information

Further information is available by emailing the workshop organizers at evalumap@adaptcentre.ie or by emailing the workshop organizers.

Workshop Chairs

Liadh Kelly, Maynooth University, Ireland

Bilal Yousuf, Trinity College Dublin, Ireland

Workshop Co-organizers

Owen Conlan, Trinity College Dublin, Ireland

Kieran Fraser, Trinity College Dublin, Ireland

Kevin Koidl, Trinity College Dublin, Ireland

Séamus Lawless, Trinity College Dublin, Ireland

Program Committee

Paul De Bra, Eindhoven University of Technology, The Netherlands

Eelco Herder, Radboud University

David Chin, University of Hawaii

Stephan Weibelzahl, Private University of Applied Sciences Göttingen, Germany

Alexandros Paramythis, Contexity, Switzerland

Tsvi Kuflik, The University of Haifa

Past Editions of the Workshop

2017 Workshop
2016 Workshop