The internet is an incredible resource for information and learning. By using search
engines like Google, information is usually just a click away. Unless you are a child,
in which case most of the information on the web is either (way) too difficult to read
and/or understand, or impossible to find. This research aims to successfully combine
the areas of readability assessment and gamification in order to provide a tech-
nical and theoretical foundation for the creation of an automatic large scale child
feedback readability assessment system. In which correctly assessing the readability
level of online (textual) content for children is the central focus. The importance
of having correct readability scores for online content, is that it provides children
with a guideline on the difficulty level of textual content on the web. It also allows
for external programs i.e. search engines, to potentially take readability scores into
account based on the known age/proficiency of the user. Having children actively
participate in the process of determining readability levels should improve any current
systems which usually rely on fully automated systems/algorithms or human
(adult) perception.
The first step in the creation of the aforementioned tool is to make sure the underlying
process is scientific valid. This research has adapted the Cloze-test as a
method of determining the readability of a text. The Cloze-test is an already established
and researched method of readability assessment, which works by omitting
certain words from a text and tasking the user with filling in the open spots with the
correct words. The resulting overall score determining the readability level.
For this research we want to digitize and automate this process. However, while
the validity of the Cloze-test and its results in an offline (paper) environment have
been proven, this is not the case for any digital adaptation. Therefore the first part of
this research focusses on this central issue. By combining the areas of readability
assessment (the Cloze-test), gamification (the creation of a digital online adaptation
of the Cloze-test) and child computer interaction (a user-test on the target audience
with the developed tool) this validity was examined and tested. In the user-test the
participants completed several different Cloze-test texts, half of them offline (on paper)
and the other half in a recreated online environment. This was done to measure
the correlation between the online scores and the offline scores, which we already
know are valid. Results of the user-test confirmed the validity of the online version
by showing significant correlations between the offline and online versions via both
a Pearson correlation coefficient and Spearman’s rank-order analysis.
With the knowledge that the online adaptation of the Cloze-test is valid for determining
readability scores, the next step was to automate the process of creating
Cloze-tests from texts. Given that the goal of the project was to provide the basis of
a scalable gamified approach, and scalable in this context means automated. Several
methods were developed to mimic the human process of creating a Cloze-test
(i.e. looking at the text and selecting which words to omit given a set of general
guidelines). Included in these methods were TF.IDF and NLP approaches in order to
find suitable extraction words for the purposes of a Cloze-test. These were
tested by comparing the classification performance of each method with a baseline
of manually classified/marked set of texts. The final versions of the aforementioned
methods were tested, and resulted performance scores of around 50%, i.e. how well
they emulated human performance in the creation of Cloze-tests. A combination of
automated methods resulted in an even bigger performance score of 63%. The best
performing individual method was put to the test in a small Turing-test style user-test
which showed promising results. Presented with 2 manually- and 1 automatically
created Cloze-test participants attained similar scores across all tests. Participants
also gave contradicting responses when asked which of the 3 Cloze-tests was automated.
This research concludes the following:

Results of offline- and online Cloze-tests are highly correlated.

Automated methods are able to correctly identify 63% of suitable Cloze-test
words as marked by humans.

Users gave conflicting reports when asked to identify the automated test in a
mix of both automated- and human-made Cloze-tests.

This entry was posted
by Djoerd Hiemstra
on Wednesday, November 30th, 2016 at 6:33 pm and is filed under IR for children.
You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.