Posts about our text and social media analysis work and latest news on GATE (http://gate.ac.uk) - our open source text and social media analysis platform. Also posts about the PHEME project (http://pheme.eu) and our work on automatic detection of rumours in social media. Lately also general musings about fake news, misinformation, and online propaganda.

Thursday, 25 April 2013

EnviLOD: Summary of User Feedback on the Semantic Search Interface

We have now completed the analysis of the user feedback received during two evaluation
sessions, carried out towards the end of the 7 month long #EnviLOD project:

The
first smaller user evaluation session was held internally at HR Wallingford and
was used as an initial test of our semantic search user interface (UI). Following this, some minor adjustments were made to the UI.

Overall,
workshop participants found the EnviLOD semantic search UI (see a screen shot at the end of this post) easy to learn and
use. In some detail, 87.5% of users disagreed or strongly disagreed
with the statement that the UI is unnecessarily complex. Similarly, 81.25% agreed or strongly
agreed that the EnviLOD UI is easy to use. 93.75% of participants also felt they can use the system without needing to learn
more about it first. With respect to quality, we asked whether the results returned by the semantic search UI
made sense to the users, where 75% agreed or
strongly agreed with this statement.

The group discussions and the three written feedback questions on the
user feedback forms allowed us to elicit a number of small, easy to implement changes
to the UI, which we hope will lead to improved usability in the future. We are
planning to implement these during follow-up research in the next 1-2 months and then carry out a
second user-based evaluation. This time we will include users recruited online, who will
not be shown a demonstration of the semantic search UI in advance.

In
addition, we also elicited a number of more challenging ideas for future
improvements, which cannot be easily addressed within the scope of short, informal follow-up work. The most
substantial of these include the implementation of a natural language
interface, map-based visualisations, support for user feedback on search
results and search query refinement, for
example. These are all valuable possible extensions to this work, including
building a natural language interface (see (Damljanovic et al, 2013) for some
preliminary work which we have done on this already). However, to fully address
these, a much longer and larger, two or three year project would be required.

Lastly, a known limitation of this evaluation which could have impacted on our results came from the limited amount of content which we could index in the experimental system. This was due to copyright issues with the majority of environmental science content at the British Library. Due to this limited content, participants were not always able to judge how comprehensive or accurate the semantic search was, especially if compared against results offered by Google. Since the British Library is now planning to integrate the EnviLOD semantic enrichment tools within the advanced Envia Labs functionality, future work on this tool could potentially be able to evaluate on this more comprehensive data, through the Envia system.