ICWE / Event Online Educa BerlinICWE is an international events and media company, which specialises in organising large-scale events and communication of international media campaigns for clients, e.g. the European Commission. ICWE&apos;s international conferences and exhibitions are the leaders in their field, for example Online Educa Berlin and eLearning Africa.The partnership between ICWE and ELIG continues through Online Educa Berlin and other common engagements.

The HOTEL (Holistic Approach to Technology Enhanced Learning) project originates from the observation that most of the TEL research so far has concentrated mainly on the development of ad hoc technologies for learning, failing to capture both the potential adoption of emerging technologies not originally designed for learning in education and training environments and the innovative use that is made of technologies in non-formal and informal ways of learning and the extent to which this could be transferred / adapted to formal learning environments. This is believed by the Consortium to provide a misleading and fragmented picture of the extent to which new forms of using technologies (already mainstreamed and/or successfully piloted and/or emerging) support learning. Innovative practices in the use of technologies for learning (especially in non-formal and informal learning environments) are often not sufficiently considered by research whereas bottom-up innovation is playing an increasingly important role in the field of TEL, which might lead to new theories for learning. On the other hand, there is a need to verify the impact of existing learning theories on TEL practices to determine whether this has led / is leading to innovation. Furthermore, the lack of a holistic approach in TEL as described above puts at risk the effectiveness and mainstreaming of new ways of using ICT for learning purposes: too often the timespan between the identification of technologies that have a potential for learning, the theoretical analysis of pedagogical implications, the piloting of such technologies and their adoption (first at small scale and then mainstreamed) is so long that the technology itself becomes out-dated compared to the changing environment and learning needs.The assumption based on which a project was conceived is that the innovation cycle (idea -&gt; prototype -&gt; test -&gt; innovation -&gt; market) in the field of TEL is so long that it seldom leads to exploitable innovations and often traps potentially disruptive innovations into the chronological limits of EU-funded projects in the fieldHOTEL tries to address this challenge by designing and testing an innovation support model (specific for TEL) in three labs - the HOTEL Learning Exploratoria – that officially launched in September 2013. The Learning Exploratorium Labs will act in the field of adult learning covering respectively: the higher education setting, the corporate setting, the informal learning setting within professional networks.An Open call for innovators was launched in June 2013 to gather applications from TEL innovators to become members of the Labs and have their innovations discussed and /or tested within the Labs

Pearson is the world’s leading learning company. We have always been focused on enabling people to make progress in their lives through learning. Now we are further increasing our focus on this. Indeed, our aim is to become the ‘efficacy company’. By this we mean making a measurable impact on improving someone’s life through learning. We’re transforming ourselves from a company that creates inputs to education – such as online courses, books, assessments and other learning tools - to one that is focused on learning outcomes. How will we do this? Efficacy.

There is clearly big demand for high quality education, and in this context there are three important factors that have made us realise that now is our best chance to make a difference: The recognition that education can drive personal, economic and societal growthThe subsequent increase in global investment in educationThe advancement of technology that gives us access to real-time data on how well a product or service is helping a learner reach their goals Together, these factors create a unique opportunity to work with others to transform education, and in doing so, the lives of learners across the world.

Efficacy has always been important to Pearson, and we have many examples of products where we can demonstrate their precise impact on learners, but we have not had a consistent methodology for measuring efficacy across every single one of our products and services...until now. Eighteen months ago, we started work on a new Efficacy Framework: a comprehensive measurement tool that would help us to identify what impact our products and services are having on learners, and what we need to do to increase this impact. We are using this framework to review all of our products and services. We know that we have a long way to go as a company to achieve this vision in full, and, that for some of our products it will take time to demonstrate efficacy. However, we will ensure these products are on a path to efficacy by supporting them through regular check-ins to make sure that progress is on track.

We are applying efficacy to our products and services, but we are also applying it to all of our people and our whole business strategy to make sure that right across the company educational impact is just as important as financial objectives, and all of our decisions take into account what is best for the learner.Ultimately, we would like to demonstrate that social and educational value and financial value can be delivered by the same strategy, and encourage others to take the same approach.

That said, we know that we don’t have all of the answers. The global education challenges are too big and too pressing for any one organisation to resolve alone, and we are just one small part of the education system. This is why we plan to partner with other education organisations, to debate and discuss the challenges, and agree and invest in solutions.We have also made our efficacy framework available to anyone who would like to use it, and are inviting feedback on how it works in practice and how it can be improved.

We have made a commitment to review all of our products and services with an annual revenue of over £1m by 2015.All products will have clear action plans for improvement by 2018, and if they don’t meet the targets set out in those action plans they will be discontinued. We have begun to review all of our acquisitions and investments for efficacy. Our ambition to start a global conversation about efficacy is also underway. We are creating a new section of our website called Open Ideas, which will give everyone unlimited access to a growing pool of research into best practice in education.

The Efficacy Framework is divided into four sections: Outcomes: to define what we are trying to achieveEvidence: to make sure that our assessment of and ambitions for the product/service are grounded in real experience and resultsPlanning and implementation: to outline how we intend to achieve our goalsCapacity to deliver: to check that we have the people, knowledge and skills we need to reach those goals

In line with the recent public commitment to efficacy (http://efficacy.pearson.com/) the learnshop is seen to be an appropriate showcase for the application of the efficacy framework to a wide variety of interested parties. Inside and outside Pearson “efficacy” has different meanings. At Pearson we have agreed on a definition of efficacy. Efficacy is defined as: “A measurable impact on improving someone’s life through learning.”We need to be able to identify the specific impact for a learner. Efficacy has direct and obvious applications for those who are designing and delivering products, services and solutions to learners. The Efficacy Framework was developed by Sir Michael Barber (Chief Education Advisor) and his team. It draws on best practices about delivery from Pearson, and the public and private sectors. The Efficacy Framework has two purposes: to understand whether we are delivering efficacy, and to identify a path to improve efficacy. This is outlined below, with the four key questions asked as part of the framework and a set of ratings for identification.

I hope you have found this useful and interesting, and that you feel like our efficacy focus will help you and your students/pupils to achieve your goals.For more information, please visit efficacy.pearson.com, where you will find lots more detail about our approach, and be able to hear from some of our leadership team and external education experts about what it could help to achieve.

4.
1. An Introduction to the Learnshop
This interactive Learnshop aims critically to reflect how to innovate in a
profoundly changing education ecosystem.
Pearson‟s Efficacy Framework will be tested as a [e.g. stand-alone] means to
support the different stakeholders to innovate in TEL / education
Key objectives
For attendees the learnshop would provide the opportunity:
1.
2.
For a case owner: to examine your own „case‟ through the lens of a rigorous
and structured framework, providing key takeaways at both the transversal
and individual case level
3.
4
For all attendees: to acquaint yourself with the efficacy framework as a tool
to engender learning-focused discussions when assessing and evaluating
prospective innovations
For a participant: to be involved at the input phase of the project, allowing
for feedback provided on evaluating the efficacy framework as a support
model to be incorporated into future iterative decisions of the holistic model
Online Educa Berlin l December 2013

6.
2.2 The HoTEL Project
•
The assumption based on which a project was conceived is
that the innovation cycle in the field of TEL is so long that:
1.
2.
•
6
It seldom leads to exploitable innovations
Often traps potentially disruptive innovations into the
chronological limits of EU-funded projects in the field
HOTEL tries to address this challenge by designing and
testing an innovation support model (specific for TEL) in
three labs (the HOTEL Learning Exploratoria)
Online Educa Berlin l December 2013

7.
2.3 Pearson aims to use its position in
education to improve the lives of learners
•
•
We want to be able to prove that our products and services
have a measurable impact
•
7
Our vision is to help millions of people make progress in
their lives through learning
Efficacy is how we’ll do this.
Online Educa Berlin l December 2013

8.
3.1 The moment is now to make a lasting
impact on learner lives
•
•
Governments, individuals, employers and
institutions recognise the need to deliver
high quality learning
•
8
More than ever before, there is a shared
understanding that high quality education
drives personal, economic and societal
growth
New technology makes it increasingly
possible to see what works and what doesn‟t
in education to drive our ability to help
learners achieve their goals
Online Educa Berlin l December 2013

9.
3.2 Measuring and improving
•
•
Online Educa Berlin l December 2013
We introduced a framework that
is systematically being applied
across the company to help us do
this
•
9
We are reviewing all of our
products and services to
ensure they deliver on learner
outcomes
For some products it will take
time to demonstrate efficacy but
we will work hard to ensure that
they are set on a path to
efficacy

10.
3.3 Embedding and leading
•
We adapted core business
processes driving the
company to focus on delivering
learner outcomes
•
We will measure our success
not only by financial returns but
also by learner outcomes
•
We hope to demonstrate the
knowledge gained from this
approach and encourage others
to adopt it
10 Online Educa Berlin l December 2013

11.
3.4 Collaborating and exchanging
•
We cannot do this alone - we are just one part of the
education system
•
We want to work with partners, leading educators and
organisations to co-create solutions
•
We will share our approach with everyone, invite
feedback on it and work together to tackle the great
unmet education needs worldwide
11 Online Educa Berlin l December 2013

12.
3.5 Our commitment
•
We will review all our acquisitions, investments and
existing products and services to ensure they improve
learner outcomes
•
We will build and share a global bank of evidence on best
practices in education
•
We will publicly report our impact on learning outcomes,
and set targets for improvement, across our business by
2018
12 Online Educa Berlin l December 2013

14.
4. Our efficacy framework aims to measure
likelihood of impact
•
What outcomes are we trying to achieve?
•
What evidence do we have to believe it is possible
to achieve these outcomes?
•
What plans are in place to implement these
outcomes?
•
What capacity is in place to deliver on outcomes?
14 Online Educa Berlin l December 2013

16.
4.1 The Efficacy Framework in Practice
Purpose:
•
To better understand the use of the Efficacy
Framework as a tool to support TEL
innovation
Instructions:
•
1 case owner per group – run through the
efficacy framework
•
Participants to join a case group and ask
questions to the case owner aligned to the
framework
•
Based on additional knowledge and context,
group to rate the case using the efficacy
framework
•
Pearson & ELIG team to facilitate the
discussion and note down next steps
Timing:
•
~ 45 minutes total
16 Online Educa Berlin l December 2013

17.
Efficacy Framework: Outcomes
Intended outcomes
• Have you identified specific outcomes for your target
group?
• Do you have a way to measure the intended
outcomes?
• Do you have ambitious and measurable targets in
place, and deadlines for achieving them?
• Are your intended outcomes clearly documented and
understood by the relevant people within and
outside your organisation?
Overall design
• Is the product designed in a way that will most
effectively help your target group reach their goals?
• Does the design allow you to automatically collect
evidence of your progress?
• Have you adapted the design based on feedback
from users?
• Could the design by used by others?
Value for money
• Do you understand the benefits of your product or
service to your target group, relative to other
options?
• Is the cost of the product/service competitive,
considering the benefits it would deliver?
Example of green rating
Example of red rating
• All outcomes are specific and
clearly documented.
• People within and outside my
organisation understand the
intended outcomes and are
able to communicate them
clearly.
• Future targets are ambitious
and achievable.
• Outcomes can be regularly
measured against set targets.
• Outcomes are not documented
or specific.
• People within and outside my
organisation do not
understand the intended
outcomes or communicate
them in the same way.
• Targets do not exist to
measure outcomes against.
• Outcomes are only defined at
a high level.
• Design is superior to other
options/competitors with
features focused on
delivering outcomes.
• Real-time evidence is
generated.
• The design can be adapted
and developed.
• Others could use this design,
and it has been shared with
them.
• The design does not meet
target group expectations
and is difficult to use.
• The design does not reflect
intended outcomes.
• The design does not allow
for the collection of
feedback.
• The design is specific to a
local situation and cannot be
replicated.
• Feedback/research has
allowed me to identify what
benefits the product/service
needs to deliver to users.
• Feedback and return-oninvestment research shows
that the cost of the
product/service reflects the
benefits delivered.
• No feedback from users
exists (either formal or
informal), and the benefits of
using this product/service
are unclear to our team and
our users.
• Perceptions of value for
money and user experience
are poor.

18.
Efficacy Framework: Evidence
Comprehensiveness of evidence
• Do you collect evidence using a range of methods
(quantitative, qualitative, internal and external for
example)?
• Do you collect evidence for all stages of your
product/service (from early conception to design and
then to implementation)?
• Do you have evidence from all users of your
product/service?
Quality of evidence
• Does the evidence you have collected link directly to
what you are trying to achieve?
•
Is the evidence you have collected unbiased;
applicable to your product/service; recent; and does
it measure success over a period of time?
• Is the evidence you have collected relevant,
representative and where possible at an individual
level?
Application of evidence
• Is the evidence stored and accessible to relevant
people? Is it available in an electronic and searchable
format?
• Has the evidence you have collected been analysed
to help inform the design of your product/service?
• Has the evidence you have collected been analysed
to help inform other decisions about your
product/service?
Example of green rating
• A wide range of evidence has
been collected via
internal/external, and
quantitative/qualitative
methods.
• Evidence relates to all stages
of my product/service.
• Evidence exists from all users.
Example of red rating
• Evidence is collected via a
limited range of methods and
does not balance qualitative
and quantitative sources.
• Evidence is mainly anecdotal
and patchy, and does not take
into account the
product/service‟s lifecycle,
features, or users.
• Evidence collected effectively
proves how well we are
meeting our objectives.
• Rigorous research methods
have been used.
• Evidence relates to the
specific and relevant use of
the product/service.
• Evidence was gathered over
a period of time.
• Of the evidence that does
exist it is not linked directly to
what I am trying to achieve.
• The evidence that exists is:
biased; not from a relevant
use of the product/service;
out of date.
• The evidence is not
representative of how a
learner would use this
product/service.
• All evidence is readily
accessible and searchable.
• The evidence is used
regularly to inform the design
of my product/service.
• Collected evidence is also
used to inform non-design
decisions.
• The evidence that exists cannot
be accessed quickly via
electronic means.
• The design of my product/service
has not been changed as the
result of evidence.
• Major decisions about my
product/service are not
underpinned by evidence.

19.
Efficacy Framework: Planning & Implementation
Action plan
• Do you have a plan in place to achieve your
outcomes, including milestones, actions,
responsibilities and timelines?
• Does your plan include short- and long-term
priorities.
• Have you identified any potential risks and included
actions to mitigate these in your plan?
• Do you regularly update your plan and communicate
changes to relevant people/institutions?
Governance
• Do people within and outside your organisation
understand who is responsible for decision-making
regarding your product/service?
• Have you documented who is responsible for the
work, and who should be consulted and informed?
Do the relevant people understand this?
• Have you identified the key processes required to
implement your product/service and are these
clearly documented?
Monitoring and reporting
• Do you update your plan based on progress, adapt it
where necessary and communicate this with your
stakeholders?
• Do you get/have access to real-time feedback from
your users?
• Do you identify issues early, discuss these honestly
and find solutions?
• Do you have tools and routines in place to monitor
progress (such as emails, calls, document-sharing)?
Example of green rating
Example of red rating
• Electronic plan exists with
clearly identified steps,
responsibilities and
deadlines.
• The plan includes short and
long-term priorities.
• The plan is regularly updated
and all relevant parties are
aware of the changes.
• No electronic plan exists.
• Plan is informal with actions,
responsibilities and timelines
unclear.
• Milestones lack clarity and
are either too ambitious or
not stretching enough.
• Potential risks have not been
formally identified or planned
for.
• Team-members know who
makes decisions, and each
member of the team (within
and outside my organisation)
is clear about their role.
• The processes we have in
place are documented and
well understood, and new
members are fully briefed.
• Team-members do not know
who makes key decisions.
• Roles for people outside the
core team are poorly defined.
• New team members are
unclear of key processes and
do not have documentation to
refer to.
• Data is collected in real-time
and analysed to provide
feedback.
• Monitoring of the
product/service alerts me to
issues in real time.
• Tools and routines are in
place to identify and solve
problems.
• Our action plan has not been
updated and adapted.
• Where feedback exists, it is
delayed.
• Our team is are unaware of
issues or fails to act on them.
• Team routines are informal
and not focused on
monitoring progress.

20.
Efficacy Framework: Capacity to deliver
Internal capacity and culture
• Does your organisation have the right number of
people, and people with the right skillsets to enable
you to deliver your desired outcomes?
• Does your organisation have a culture focused on
delivering outcomes, and is it collaborative and
innovative?
• Do leaders within your organisation support your
work and are there opportunities to work with
others across the organisation?
User capacity and culture
• Do the target group understand the objectives and
their roles in achieving them?
• Does the product/service reflect the user‟s skillset
and available resources?
• Do users have the people, skills, time, or resources
to achieve their goals?
• Have you put measures in place to build users‟
skills?
Stakeholder relationships
• Have you identified who your key stakeholders are
and do you understand their needs and concerns?
• Do you regularly communicate with your
stakeholders?
• Is there a culture of partnership and collaboration
between your organisation and your stakeholders?
Example of green rating
Example of red rating
• Team has right number of
people with appropriate
skillset and experience.
• Culture is focused on
delivering outcomes and is
collaborative and innovative.
• Team has appropriate
budget.
• Our team lacks the
appropriate skills and
resources to deliver the
desired outcomes.
• Our culture feels negative,
traditional and not focused on
outcomes.
• The target group understand
the objectives and their
roles.
• The product/service takes
the user‟s skillset into
account and there are
mechanisms in place to build
skills.
• Users have the appropriate
resources to achieve their
goals.
• The target group and existing
users are not aware of what
the product/service should
help them to achieve and
what they need to do to get
there.
• The product/service is illsuited to the user and
attempts to build users‟ skills
are ineffective.
• Users do not have the
resources and skills to meet
their goals.
• We meet with stakeholders
frequently, and have formal
and informal conversations.
• Conversations with
stakeholders have led to a
culture of trust and
partnership over a sustained
period of time.
• The team and stakeholders
have uncertain relationships.
• Miscommunication occurs
frequently and solving
problems in a joint fashion is
difficult.

21.
Reference material: Efficacy website
On November 15th, Pearson
launched a dedicated
website:
http://efficacy.pearson.com
outlining the company‟s focus
on efficacy and commitment
to put the learner at the
heart of the global strategy.
An interactive version of the
efficacy framework also
features on the website.
21 Online Educa Berlin l December 2013

22.
Reference material: Efficacy publications
Identifying dialogue and collaboration with the wider education community as crucial to accelerate
progress, Pearson has also published two reports:
• The first, Asking More: The Path to Efficacy, sets out the imperative for measuring and improving
learning outcomes worldwide.
• The second, The Incomplete Guide to Delivering Learning Outcomes, shares in detail the
company‟s new approach to contributing to that goal and the progress it has made so far
22 Online Educa Berlin l December 2013

23.
Reference material: Alive in the Swamp
“The future will belong not
to those who focus on the
technology alone but to
those who place it in this
wider context and see it as
one element of a wider
system transformation.”
Quote is from Michael Barber, Chief Academic Advisor, Pearson
23 Online Educa Berlin l December 2013

24.
How can I find out more?
You can visit http://hotel-project.eu/, for:
•
•
More on the Learning Exploratorium Labs
•
Details of future events
•
Find us on LinkedIn:
(TEL Innovation laboratory)
•
24 Online Educa Berlin l December 2013
Project overview: Work programme and
Innovation model
Contact: Andreas Mesizner
(andreas.meiszner@elig.org) if you are
interested in further opportunities for
collaboration

25.
How can I find out more?
You can visit efficacy.pearson.com, for:
•
•
To use our online interactive efficacy tool
•
Keep up to date on our events
•
Read up on the role of efficacy in education
in two publications: Asking More, and The
Incomplete Guide
•
Find us on LinkedIn (Open for Learning)
and Twitter (@PearsonPLC)
•
25 Online Educa Berlin l December 2013
More information about our approach
Contact: efficacy.global@pearson.com if
you are interested in further opportunities
for collaboration