Call for critiques of proposed shared evaluation challege approach & Call for Papers
Max paper length: 2-pages
Submission deadline: March 20, 2019 March 25, 2019
Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behavior of the users involved in trialing the systems. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research.
We envision that EvalUMAP 2019 will be the starting point for an annual comparative evaluation challenge at future UMAP conferences. The two previous iterations of this workshop (EvalUMAP 2016 and 2017) have created a platform for this topic to be discussed and advanced. In this iteration of the workshop series, final steps towards shared challenge generation will be made. A white paper mapping a proposed shared challenge approach is now available, consisting of exemplar data, desirable metrics and a plan for what engaging in the challenge task would look like. Feedback in the form of reflection, critique and alternate approaches will be solicited from the community via short submissions to the EvalUMAP workshop.
The planned outcome of the workshop will be a roadmap to develop initial shared task(s) that will be published ahead of UMAP 2020, providing opportunity for participants to test and tune their systems and complete the task in order for comparative results and associated publications to be prepared for and presented at UMAP 2020. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviors, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of metrics and observations for participants to perform in order to facilitate comparison will be outlined.
Workshop topics are evaluation focused and include, but are not limited to:
- Reflecting upon and critiquing the proposed shared challenge proposal white paper
- Understanding UMAP evaluation
- Defining tasks and scenarios for evaluation purposes
- Identification of potential corpora (datasets) for shared tasks
- Automated and semi automated processes on creating appropriate datasets and simulating user behaviours etc in order to accommodate a shared task
- Interesting target tasks and explanations of their importance
- Critiques or comparisons of existing evaluation metrics and methods
- How we can combine existing evaluation metrics and methods?
- Reusing or improving previously suggested metrics and methods
- Reducing the cost of evaluation
- Proposal of new evaluation metrics and methods
- Technical challenges associated with design and implementation
- Anonymization of datasets, Privacy, Ethics and Security issues on the use of datasets
Workshop format:
This will be an interactive workshop structured to encourage group discussion and active collaboration among attendees.
Paper Submissions
The workshop is now accepting paper submissions. Position papers (max 2 pages) that either (1) critique the white paper we have made available proposing an approach for a shared challenge task (http://evalumap.adaptcentre.ie/whitePaper.pdf), (2) propose an alternate challenge approach, or (3) reflect on experiences in the space are invited. Submissions should be in ACM SIGS format. LaTeX and Word templates are available at (https://www.acm.org/publications/proceedings-template).
Papers should be submitted in pdf format through the EasyChair system (https://easychair.org/conferences/?conf=evalumap2019) no later than midnight Pacific Daylight Time on March 20, 2019. Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the UMAP 2019 EvalUMAP workshop proceedings.
Important Dates
March 20, 2019 March 25, 2019: Deadline for paper submission (midnight Pacific Daylight Time)
March 27, 2019 March 29, 2019: Notification to authors
April 3, 2019: Camera-ready paper due
June, 2019: EvalUMAP Workshop at UMAP
Further Information
Further information is available by emailing the workshop organizers at evalumap@adaptcentre.ie or by emailing the workshop organizers.
Workshop Chairs
Liadh Kelly, Maynooth University, Ireland
Bilal Yousuf, Trinity College Dublin, Ireland
Workshop Co-organizers
Owen Conlan, Trinity College Dublin, Ireland
Kieran Fraser, Trinity College Dublin, Ireland
Kevin Koidl, Trinity College Dublin, Ireland
Séamus Lawless, Trinity College Dublin, Ireland
Program Committee
Paul De Bra, Eindhoven University of Technology, The Netherlands
Eelco Herder, Radboud University
David Chin, University of Hawaii
Stephan Weibelzahl, Private University of Applied Sciences Göttingen, Germany
Alexandros Paramythis, Contexity, Switzerland
Tsvi Kuflik, The University of Haifa
Past Editions of the Workshop
2017 Workshop
2016 Workshop
