logo



EvalUMAP 2016 Workshop


Towards comparative evaluation in user modeling,
adaptation and personalization


To be held in conjunction with the 24th Conference on User Modeling,
Adaptation and Personalization, UMAP 2016, July 2016, Halifax, Canada

Workshop Proceedings

EvalUMAP workshop proceedings available at: http://ceur-ws.org/Vol-1618/#EVAL

EvalUMAP 2016 Agenda

Saturday 16th of July 2016

Room 150


9:00 - 9:20

Workshop Introduction
(Purpose, Goals, Agenda, Introduction of participants)
(20 Minutes)

Owen Conlan

9:20 - 10:00

Keynote
(30 Minutes + 10 Minutes Q&A)

David Chin
The Challenges of Shared Evaluation Tasks for UMAP

10:00 - 10:30

Paper Presentations I (Each Presentation 10 minutes plus 2 minutes Q&A)

Harshvardhan Pandit, Roghaiyeh Gachpaz Hamed, Shay Lawless and David Lewis
The Use of Open Data to Improve the Repeatability of Adaptivity and Personalisation Experiment

Kevin Koidl, Killian Levacher, Owen Conlan and Ben Steichen
ECP: Evaluation Community Portal A Portal for Evaluation And Collaboration in User Modelling and Personalisation Research

10:30 - 11:00

Break

11:00 - 12:30

Paper Presentations II (Each Presentation 10 minutes plus 2 minutes Q&A)

Paul De Bra
Evaluating Adaptive Systems and Applications is often Nonsense

Ifeoma Adaji and Julita Vassileva
Evaluating Persuasive Systems Using the PSD Framework

Ali Hosseinzadeh, Roghaiyeh Gachpaz and Kevin Koidl
A Review of User-centred Information Retrieval Tasks

Liadh Kelly
Living Labs for UMAP Evaluation

David Chin and William Wright
Evaluation Metrics for Inferring Personality from Text

Bilal Yousuf and Owen Conlan
Motivating Behavioral Change through Personalized Explorable Visual Narratives

12:30 - 13:30

Lunch

13:30 - 14:00

Paper Presentations III (Each Presentation 10 minutes plus 2 minutes Q&A)

Athanasios Staikopoulos and Owen Conlan
Towards comparative evaluations of user-adaptive software systems

Veronika Bogina and Tsvi Kuflik
Building a Bridge between User-Adaptive Systems Evaluation and Software Testing

14:00 - 15:00

Discussion Session

15:00 - 15:30

Break

15:30 - 16:30

Discussion Session

16:30 - 17:00

Wrap up and next steps

18:00

Social Evening Event (TBA)


Call for Papers

Submission deadline: May 16, 2016

Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behaviour of the users involved in trialling the systems. To-date this topic has received relatively little attention. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research, which to-date has been very limited.

Taking inspiration from communities such as Information Retrieval and Machine Translation, the first EvalUMAP Workshop seeks to propose and design one or more shared tasks to support the comparative evaluation of approaches to User Modelling, Adaptation and Personalization. The workshop will solicit presentations from key practitioners in the field on innovative approaches to evaluating such systems and will provide a forum to start scoping and designing tasks for the following year. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviours, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of metrics and observations will be outlined, that participants would be expected to perform in order to facilitate comparison.

The planned outcome of the EvalUMAP Workshop 2016 will be a roadmap to develop initial shared task(s) that will be published well in advance of UMAP 2017, giving an opportunity for participants to test and tune their systems and complete the task in order for comparative results and associated publications to be prepared for and presented at the EvalUMAP Workshop in 2017. We envision that EvalUMAP 2017 will be the starting point for an annual comparative evaluation challenge at future UMAP conferences.


Workshop topics are evaluation focused and include, but are not limited to:

  • Understanding UMAP evaluation
  • Defining tasks and scenarios for evaluation purposes
  • Identification of potential corpora for shared tasks
  • Interesting target tasks and explanations of their importance
  • Critiques or comparisons of existing evaluation metrics and methods
  • How we can combine existing evaluation metrics and methods
  • Improving on previously suggested metrics and methods
  • Reducing the cost of evaluation
  • Proposal of new evaluation metrics and methods
  • Technical challenges associated with design and implementation
  • Privacy, Ethics and security issues
  • Legal and ethical issues

Workshop format:

This will be an interactive workshop structured to encourage group discussion and active collaboration among attendees. The workshop will feature a keynote talk, lightning round presentation session for position papers, multiple (parallel) breakout sessions, and a final discussion session to wrap up the event.


Paper Submissions

The workshop is now accepting paper submissions. Position papers (max 2 pages) describing approaches or ideas / challenges on the topics of the workshop are invited. Submissions should be in ACM SIGS format. LaTeX and Word templates are available at www.acm.org/sigs/publications/proceedings-templates.

Papers should be submitted in pdf format through the EasyChair system (https://easychair.org/conferences/?conf=evalumap2016) no later than midnight Pacific Daylight Time on May 16, 2016. Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the UMAP 2016 EvalUMAP workshop proceedings, which will be indexed with CEUR. Authors of select papers may be invited to contribute to a journal publication which describes the outcomes of the workshop.


Important Dates

May 16, 2016: Deadline for paper submission (midnight Pacific Daylight Time)

June 1, 2016: Notification to authors

June 7, 2016: Camera-ready paper due

July, 2016: Full-day Workshop during UMAP

Further Information

Further information is available by emailing the workshop organizers at evalumap [at] adaptcentre.ie.


Workshop Organizers

Owen Conlan, Trinity College Dublin, Ireland

Liadh Kelly, Trinity College Dublin, Ireland

Kevin Koidl, Trinity College Dublin, Ireland

Séamus Lawless, Trinity College Dublin, Ireland

Killian Levacher, Trinity College Dublin, Ireland

Athanasios Staikopoulos, Trinity College Dublin, Ireland


Programme Committee

Paul De Bra, Eindhoven University of Technology, The Netherlands

Iván Cantador, Universidad Autónoma de Madrid, Spain

David Chin, University of Hawaii, USA

Eelco Herder, L3S Research Center, Hannover, Germany

Geert-Jan Houben, Delft University of Technology, The Netherlands

Judy Kay, University of Sydney, Australia

Tsvi Kuflik, The University of Haifa, Israel

Alexandros Paramythis, Contexity, Switzerland

Francesco Ricci, University of Rome, Italy

Alan Said, Recorded Future, Sweden

Vincent Wade, Trinity College Dublin, Ireland

Stephan Weibelzahl, Private University of Applied Sciences Göttingen, Germany






Confirmed Keynote Speaker: David Chin, University of Hawaii, USA


Talk Title:
The Challenges of Shared Evaluation Tasks for UMAP


Abstract:
There are many challenges for developing shared evaluation tasks for the UMAP community. First, there is the breadth of both types of user models and applications. A shared evaluation task for inferring user knowledge level would require very different data than one for inferring user product preferences. Even for the same type of user model, say user knowledge level, the data needed would be completely different for an NLP advisor system versus an ITS. Another difficulty is persuading researchers to release their hard earned data, which often required non-trivial expense (e.g., by paying subjects) and/or effort to acquire. Also, exclusive access to a large dataset confers advantages for research and publication. The final concern is privacy. Even when all personal identification information is stripped from a large dataset, it is often possible to reidentify user records by name even. The community may need to research techniques for foiling reidentification such as adding random noise to the entire dataset or small changes to the data of unique or rare population groups. Despite these challenges, shared evaluation tasks often spur advances in the state-of-the-art of algorithms, so are a worthwhile pursuit for the UMAP community.