Summary Impact Type

Research Subject Area(s)

Download original

Summary of the impact

Academic English language skills are essential if overseas students are
to perform effectively in
university/college contexts and to engage in study with relative
independence and adequate
comprehension. The Centre for Research in English Language Learning and
Assessment
(CRELLA) has developed a theoretically sound and practically efficient
methodology for the
construction and validation of tertiary-level Academic English tests used
in the selection of millions
of students by universities worldwide. Greater precision in language tests
enables academic
institutions to target their resources in ways that bring maximum benefit
to students, both pre- and
post-entry. CRELLA's research into how English is employed in academic
contexts has led to
significant improvements in test design and practice, and added to the
commercial impact of both
the existing tests to which they have contributed and the new tests they
have created. Over £1.6
million was generated by our development of the Password test
alone — see Section 4 below.

Underpinning research

Supported by British Council funding, staff in CRELLA have conducted
extensive research since
2005 into the validity of IELTS (International English Language Testing
System), the internationally
recognised and widely used Academic English proficiency test for
evaluating students wishing to
access higher education opportunities (over 2 million test takers in 2013)
(Refs3.1, 3.2, 3.5 and
3.6). Our research has established criteria that enable testers to
align their tests more precisely
with the contexts and the cognitive processes associated with using
English for academic
purposes. It has facilitated the inclusion of these key parameters in
practical test specifications and
their incorporation in test provisions, with major implications for the
validity of the test.

Contextual parameters in academic reading

Weir (Professor 2005-present), Green (Principal Lecturer
2006-2008, Reader 2008-2013,
Professor 2013-present), Bax (Reader 2008-2012, Professor
2012-present) and Hawkey (Senior
Research Fellow 2006-2009, Visiting Professor 2009-present) have been at
the forefront of this
research strand, winning funding for a series of projects. One
study investigated the nature of
Academic English reading in British universities and identified problems
international students had
in coping with these demands (Ref3.1: pp.37-119). IELTS was
evaluated with respect to these
data and improvements suggested. Ref3.2 employed automated text
analysis tools to measure key
textual features and explored the validity of the IELTS Academic English
Reading test by
comparing the features of texts used in the test with those of core
textbooks used by
undergraduates. The study identified disparities requiring intervention
which were subsequently
addressed by the test developers through changes to the specifications.

Cognitive parameters in academic reading

Ref3.1 (pp.212-269) also highlighted the cognitive processes
engaged by the IELTS Academic
Reading test compared with those demanded of students in higher education
(see Ref3.3 for our
theoretical framework for reading and Ref3.4 by Bax for
later elaboration). Bax made innovative
use of eye-tracking technology to investigate the cognitive processes of
readers taking online
Academic English reading tests. His research investigated whether test
items required higher level
cognitive processing from candidates (Ref3.5). Green and
Hawkey (Ref3.6) investigated the
extent to which appropriate contextual and cognitive parameters were
considered and
operationalised by item writers working on the IELTS Academic Reading
test.

Cognitive and contextual parameters in academic listening and speaking

Three other staff broadened this investigation of Academic English
literacy to cover other language
skills as operationalised through IELTS. In the assessment of Academic
Listening ability, Field
(Senior Lecturer 2011-present) developed a research methodology for
investigating the cognitive
validity of an Academic English Listening test, i.e., the extent to which
it taps into processes
employed in real-world academic listening events. This methodology was
employed (Ref3.1:
pp.391-453) to examine the effects of test method on listener behaviour. Field
edited a special
issue of the Journal of English for Academic Purposes (2011
Vol.10/2) exploring research and
thinking relating to listening in academic contexts. Taylor
(Senior Lecturer 2011-present)
contributed to the issue a major review article on the testing of academic
listening. This work is
complemented by innovative research by Nakatsuhara (Lecturer
2010-2012, Senior Lecturer
2012-present) into the relationship between test-takers' listening
proficiency and their interactional
performance on the IELTS Speaking test (Ref3.1: pp.519-573).

All of the above references are being submitted as Outputs in UoA 29
REF2014. Follow-on annual
funding from major EAP test providers for this underpinning research
evidences clear pathways to
impact. Since 2008, our research into IELTS funded by the British Council
has been awarded a
total of £99,100 across 5 separate projects. Our research for the Eiken
Foundation of Japan
produced a total income of £102,884, in 2010-2013. We have received
£216,621 from English
Language Testing (Password) for annual research and development
work since 2008.

Details of the impact

Tertiary students need appropriate English language skills to benefit
fully from academic study.
Furthermore, if under-qualified students are admitted to universities, the
economic costs to the
institutions and the negative social and personal impact on the students
can be enormous (Ref5.1
provides empirical data on the severity of these costs). Soundly conceived
Academic English tests
provide accurate and valid information to universities for making access
decisions and determining
the length and nature of any remedial language instruction required.

CRELLA's research has contributed to a deeper understanding of the nature
of Academic English
and provided a theoretically sound and practically efficient framework for
its measurement. Its
research has been utilised by all five English language test providers
officially credited as meeting
the UK Border Agency's requirements for student visas (Cambridge, City and
Guilds, IELTS,
Pearson and Trinity College London), as well as by other international
test providers.

Evidence of Commercial Impact

The Password academic placement test

CRELLA's research into language-related academic proficiency underpinned
the design and
development of the Password test to assess students' English
language levels before or after
acceptance on tertiary-level language programmes (Ref5.2 and Ref5.3).
Our original commercial
proposal became the basis for a joint venture business (English Language
Testing (ELT) Ltd) in
2008 with offices in London, Australia and the USA employing five
full-time staff members, 15 item
writers and seven outsourced suppliers. Now used by some 170 HE
institutions around the world
(see Ref5.3/Password clients), the test has had over
150,000 candidates since 2008 (generating
over £1.6 million in business for ELT) and proved itself as an efficient
means of assessing
students' language levels and identifying those needing language support.
One user, who has
carried out an evaluation study, considers that: `There is no test that
can compete with Password
for its combination of state-of-the-art question design, security,
reliability and simplicity of use,
delivered at such a low cost' and describes it as `an extremely reliable
indicator of academic
language ability and predicted final [language course] grade' (see Ref5.3/testimonials).
CRELLA
recently developed Password Pupil, Password Intro, Password
Arabia and the Password Reading
Test. Listening and speaking tests are currently under development
in CRELLA.

Evidence of Educational Impact

The International English Language Testing System (IELTS)

IELTS is taken by over 2 million people in 125 countries by those needing
to meet English
language entry requirements for university, professional and immigration
purposes. It facilitates
access to education worldwide and is used increasingly as an entry
criterion to professional bodies
(Ref.5.4). More than 7,000 education institutions, faculties,
government agencies and professional
organisations around the world recognise the test.

The IELTS partners are committed to `continual research-based
improvement'. According to
Davidson and Pollock, Chief Executives of the British Council and IELTS
Australia: `The steady
evolution of IELTS in response to research outcomes ensures its
contemporary relevance and
continued usefulness for test stakeholders' (Ref5.5). CRELLA's
research has significantly informed
the revision of the test's design, content and production. 86 versions of
the test are in use at any
one time so receiving institutions need to know that these are equivalent.
CRELLA's research has
given IELTS a systematic methodology for ensuring the consistency of content
from one version
of the reading test to the next. Specific IELTS research studies by CRELLA
staff have
demonstrated the necessity of creating reading and listening tasks which
are cognitively valid,
i.e., involve cognitive processing representative of the processes
occurring in real-life academic
activities. This has been incorporated into the IELTS test validation
process (Ref5.6).

CRELLA's research has also contributed to the test's strong international
reputation and
commercial success in recent years (Ref5.7 confirms). The annual
number of IELTS test takers
has increased from just over 1 million in 2008 to over 2 million in 2013
(at c£150 per candidate,
generating over £300 million p.a.).

Educational institutions need to be able to predict students' performance
on academic courses
accurately from test results. IELTS makes a useful contribution to the
prediction of overall
performance on academic courses. Weir et al found that in general, the
better the IELTS score the
more likely the student is to perform well academically (Ref5.8:
p28). Based on returns from 100
staff in 27 departments in 13 UK universities, Hyatt and Brooks report
that 88% of respondents felt
the IELTS Test to be a useful indicator of Academic English proficiency (Ref5.9:
p.16) enabling
them to reject students with inadequate English language proficiency and
target appropriate
remedial instruction for those who fall just below an acceptable
threshold. Correct use of individual
IELTS profile scores for acceptance purposes invariably contributes to a
fall in the number of
students underperforming in their degree courses because of language
deficiency (Ref5.1).

The Test of English for Academic Purposes (TEAP) in Japan

CRELLA's research and reputation in the Academic English field also led
in 2011 to an invitation to
develop the Test of English for Academic Purposes (TEAP), a new university
entrance examination
for Japan, in partnership with the Eiken Foundation of Japan (formerly
STEP), the leading English
language testing organisation in Japan, with over 2 million test takers
annually and Sophia
University, one of the leading private universities in Japan. TEAP has
Japanese government
support. The major aim behind the development of TEAP was to impact
positively on English
education in Japan at the high school level (3.4 million students). Ref5.10
states: "The long-term
impact of the TEAP project will be felt not only at the immediate level of
the universities which
choose to recognize the results, but will play an important part in
removing the impediment to the
improvement of EFL teaching and learning practices in Japan that the
university entrance exam
system has posed".

The Eiken Foundation is conducting iterative washback studies working
with Green of CRELLA to
evaluate the success of this intention. TEAP is intended to supersede the
disparate approaches to
English testing currently used in university admissions in Japan in which
speaking and writing skills
go untested. It serves as a model of the full range of Academic English
skills needed by Japanese
students to cope with the requirements of tertiary-level English-medium
study in Japan. Our
research into Academic English provided the theoretical and empirical
basis for the TEAP test
specifications, and CRELLA developed both the speaking and writing tasks
and the rating scales.
(Ref5.10). The new test was announced in 2012 and the Eiken
Foundation has made provision for
up to 200,000 candidates in the first full scale administration in 2014.

The Cambridge Academic Literacy Test

In 2013 CRELLA was commissioned to assist Cambridge English Language
Assessment in the
development of the new Cambridge Academic Literacy Test which will be
aimed at all prospective
UK undergraduate and postgraduate students. CRELLA provided the initial
position paper on the
nature of academic literacy and subsequently helped write the theoretical
rationale for the test. It is
currently helping develop the specification and test content.

Ref5.9 Hyatt, D. and Brooks, G. (2009) Investigating stakeholders'
perceptions of IELTS as an
entry requirement for higher education in the UK, IELTS Research
Reports, Volume 10. IDP,
IELTS Australia and British Council