User Evaluation report published

Kate Fernie

The report presenting the results of evaluation of the second PATHS prototype is now available.  The report forms part of a programme of user evaluation that began with the PATHS first prototype whose results have been central to the development of the PATHS system.

MDR PATHS Workshop

Figure 1: Evaluation workshop held in the UK by PATHS partners, MDR Partners

The evaluation took place during the summer of 2013, with groups of users being invited to take part in demonstration sessions in the UK, Span and Greece, and in lab-based trials which took place at the University of Sheffield.  The authors of the report collected both qualitative feedback and quantitative data which was analysed ant the results are presented in this report, which is available here .

The findings of the report show that the majority of participants had an overall positive response to PATHS, finding it mostly Interesting , Understandable , Easy , Useful  and Fast . The system was also seen as offering novel functionality that could be useful in a number of different user scenarios.

Exploration and support for tasks were developed and extended considerably for the second prototype in light of findings and results from participants of the first prototype and now show high levels of positive responses for tasks related to Serendipity and discovery, Finding items on a topic, Exploring a collection, Sharing content with others and Creating resources. The path following task, Using content created by others received no negative responses, and responses have improved substantially across all tasks, including those which are core to PATHS, over responses given for the first prototype.

Screenshot of the path creation workspace

The most popular tasks  amongst participants of the second prototype were Exploring the collection and Creating resources, placed first by both Laboratory and Demonstration participants. Serendipity and discovery, Sharing content with others and Finding items on a topic all ranked amongst the top three uses for PATHS. The redevelopment of exploration modes and support for users of PATHS appears to have had a positive impact on participant responses to the second prototype.

Finally, the results of the demonstration and laboratory activities reported here have enabled us to confirm that the second PATHS prototype system does meet user requirements as
identified in our 2011 User Requirements Analysis  and is viewed as usable, useful and supportive of exploration of cultural heritage collections.

Acknowledgements

We wish extend our gratitude to the following organisations for their support in hosting demonstration sessions:

  • Tekniker-IK4, Spain
  • Euskomedia, Spain
  • Elhuyar-Eleka, Spain
  • CIC TourGune, Spain
  • Ministry of Culture of Spain/Hispana
  • Byzantine Museum, Athens, Greece
  • Royal Institution, London, UK
  • Manchester Institute for Research and Innovation in Art and Design (MIRIAD) , UK
  • Department of Media, Manchester Metropolitan University, UK
  • Department of History, Manchester Metropolitan University, UK
  • North West Film Archive, UK
  • Samuel Johnson Birthplace Museum, UK

We would also like to thank all those individuals who took part in both the laboratory and demonstration activities, your input has been invaluable.

Share |