A qualitative study of learning from CAL Programs in two tertiary education courses

Within the range of current investigations into the effectiveness of education and student learning, a wide range of measures have been developed to assess the effectiveness of student learning and other educational outcomes. These measures include data from case studies, student interviews, large scale surveys of student approaches to tertiary education, and cognitive studies of learning style and memory which, together with attainment tests, provide a comprehensive picture of the motivational, attitudinal and cognitive approaches taken by students, and the subsequent achievement in terms of performance.

When the use of CAL and multimedia is considered, the focus of studies of student learning seems to be much narrower. Many of the studies of learning from CAL use a test of attainment at the end of the program as being the only measure of student learning. This raises questions about the nature of such testing, or of the actual test itself: is it primarily a test of recall or does it test the ability to reason; is there a variety of question styles; do the students have to apply their learning to previously unencountered situations; and what is the significance of the learning task to the learner? These questions relate to the effectiveness of the learning task in terms of its value to the student and, while they have wider implications than can be addressed by individual studies, they represent qualitative aspects that have educational significance, but are generally ignored in studies of CAL.

The purpose of this paper is to consider a range of qualitative aspects and measures of student learning in current use, and to consider the insights into student learning that these provide. This will be considered in relation to the measures used to gauge student learning from CAL packages. An approach to CAL that uses qualitative measures appears to be lacking from the research literature. One such approach, carried out by the author, is described. In this study a measure of depth of processing, the Structure of Observed Learning Outcomes (SOLO) taxonomy developed by Biggs & Collis (1982) was applied.

Qualitative measures of learning

Qualitative measures of learning are aimed at informing us about aspects of learning that do not emerge from conventional test scores. Aspects, such as the learning style employed by the student, the students' perception of the nature and importance of the task and the approach taken, or strategy used to achieve learning, and the learning outcome in terms of its meaningfulness to the student or depth of learning are all important issues in relation to the overall educational aspect of the learning task. The level of student attainment is clearly related to the way in which students approach their studies. This approach can be considered as a style or a strategy tor learning, with more effective strategies clearly leading to a greater level of success. An understanding of the learning strategies used by students can provide some guidance as to the way that learning tasks are best structured to bring about the desired outcome.

To identify approaches to learning, Biggs (1987), Schmeck (1983), and Entwistle & Waterson (1988) have all developed questionnaires aimed at identifying approaches to learning. An example is the Study Process Questionnaire developed by John Biggs (1987). In this questionnaire, students are asked to report on their normal approach to a wide range of aspects of the learning task. From these responses, it is possible to determine whether the student is intrinsically motivated to learn in depth in order to achieve a level of understanding, or is likely to take an approach of rote learning specific facts. The questionnaire also measures the extent to which achievement, in the sense of high grades, motivates the student. The questionnaire measures approach and strategy on each of the Deep, Surface and Achieving dimensions, which can be simplified to a dichotomy between Deep Achieving and Surface Achieving approaches. By using a questionnaire of this kind, insight can be gained into how particular groups of students approach their study, and also into the range of styles that are used by students in accomplishing a learning task.

Questionnaires measure a basic approach, however other measures have identified that this is not necessarily consistent for each student across every task. Laurillard (1984) found that the depth of learning aimed tor and achieved by students, as identified by a procedure of asking the student to teach back the material that had been learned plus interviewing the students about the strategy used, would vary according to the student's perception of the importance of the task. This means that the quality of learning is influenced to a high degree by whether the student sees the task as worthy of deep study. Some students reported that if they thought the assessment would be a test of memory or recall, then an attempt to understand the material in depth was not necessary, and a more superficial approach would suffice. While the SPQ measures student approach, Laurillard used a qualitative measure of both approach and outcome.

The identification of depth of learning as an outcome has been identified by Biggs and Collis (1982), who developed the SOLO taxonomy to measure this. This taxonomy is used to classify responses to an open ended question, in terms of the relationships the learner draws between the concepts learned and whether the learner is able to structure these according to an appropriate conceptual framework. By using a measure such as this, student learning can be classified according to the extent to which the student has created meaningful associations with the newly acquired knowledge, as opposed to the extent the student has memorised or reproduced the information.

Many of these aspects are the subject of ongoing study by academics involved with research into the quality of teaching and learning, as they are of value tor the insights they provide into what it is about learning tasks that motivate students to learn effectively and perform well.

Depth of processing and CAL design

Depth of learning is associated in cognitive terms with the concept of depth of processing first identified by Craik & Lockhart (1972) as being a major attribute of effective learning. This concept refers to the extent of the meaningful network of associations that the learner makes when learning new material. Craik & Lockhart identified a hierarchy of levels of cognitive processing, with greater depth of processing associated with the extraction of meaning from the material being learned. This they associated with more effective and efficient learning, as the creation and development of meaningful relationships make the efficient processing of large amounts of information possible. They characterise processing as the manipulation of conceptual schema, with efficient processing of large amounts of information occurring through schemes that contain a rich association of relevant information.
The relationship of these aspects of the learning process to CAL development have been identified by Hannafin & Rieber (1989) who argue that the methods frequently used in the design of CAL programs, such as

small learning units

controllable sequences

discrete discernible steps

behaviourally defined objectives and performance criteria

do not encourage deep mental processes. This implication is developed further by Jonassen (1988), who identifies four levels of information processing strategies; recall, integration with existing schema, organisation of existing and new schema, and elaboration, involving using and making judgements on the material. For Jonassen (1988, p154), CAL programs require what he describes as "generative learning strategies", such as "generating mnemonic memory aids, note taking, underlining, paraphrasing, summarising, generating questions, creating images, outlining, and cognitive mapping" to induce the learner to engage in an active process of constructing meaning during learning.

The argument presented by Hannafin & Rieber, and Jonassen, is that CAL programs are potentially superficial in the mental processes that the learner is required to engage in, and that the resultant learning is likely to be superficial. While this claim has intuitive merit in the sense that it is only by the student making an active response that learning is likely to occur or be effective, it may be oversimplified in that it does not take into account the mental processing that the student may engage in that is not required by the program, which may reflect the student's own learning style or motivation. This issue is investigated by examining whether depth of processing is an outcome even if programs do not, as most don't, use some form of generative learning.

Testing for depth of processing

The first issue is whether students do, in fact, learn in depth when using CAL programs. To do this an evaluation study was carried out on two tertiary level groups of students who, as part of their planned course of studies, used a CAL program to learn a particular aspect of their course.

The testing procedure tor depth of learning was carried out after the students had used the CAL program, and independently of it. The students were asked a question about the material they had studied which required an open ended and structured response. This question was assessed using the classification in the SOLO taxonomy, which assessed depth of learning in terms of the relationships the student constructed, and the structured nature of the student's response.

Each study provided a different pattern of responses, with one study showing a high level of SOLO responses, indicating an in depth response by a majority of the students, and another study showing a very low level of student responses. In this second study, most learners reproduced elements of the information provided, without attempting to relate these elements in any way. either to other newly acquired concepts in terms of a structure, or to their own experience in a way that would indicate a process of assimilation with existing knowledge and cognitive structures had taken place. Some answers, showing a deeper, more related, response, were observed. however these were in the minority. These evaluations indicate a contradictory response to the issue, except to say that some students did respond in depth, however the studies were not directly comparable due to contextual and motivational factors that are considered below.

Depth of processing in relation to learning style

Research using the Study Process Questionnaire (SPQ) has indicated that many learners do not use an approach that would lead to a deep level response. The type of student approach that leads to memorisation of facts and procedures, rather than trying to understand the meaning and implication of these in relation to the learner's own cognitive process, will most likely result in a non-relational response when evaluated using the SOLO taxonomy. By examining the relationship between the student's tendency to learn in a deep or surface way, compared to the observed outcome of learning, more insight into the effectiveness of the learning program can be gained.

All students were asked to complete the SPQ questionnaire at the beginning of each study in order to provide additional information on the effect of the CAL program on learning. As the SPQ provides a measure of deep learning as an approach to a learning task, and the SOLO taxonomy provides a measure of depth of learning as an outcome of the learning task, a degree of correlation between the two scores can be expected. Studies by Biggs (1979) and Watkins & Hattie (1981) have established a correlation between the two measures, although the correlation is not necessarily strong.

The comparison between the SPQ and SOLO scores highlights the difference between the two groups of students evaluated. In Group A, in which a high level of SOLO responses was recorded, the comparison between the SPQ and SOLO scores shows a level of in depth responses that is higher than the extent of the deep learning approach as indicated by the SPQ scores. In Group B. the level is lower, indicating that many students who are inclined to take a deep approach to learning did not respond in depth.

These differences can only have come about through the circumstances under which the learning took place, including the CAL programs used. Neither program used generative learning strategies, however there were other differences between the programs that partly explain the differences in the results. There were, however, differences in the context in which the programs were used that also partly explain the results. In order to identify all factors that may have affected student performance, additional data was obtained from the students by interview and questionnaire.

Interpretative data

In Group A, a selection of students were interviewed about their use of the program. From this data, a number of trends are apparent which illuminate the data from the SPQ and SOLO scores. The trends in the student responses indicate:

The students found the program easy to use.

The program fitted in well with the course, by providing information and learning activities that were important within the context of the overall learning task.

While the students also used other sources of information, including a lecture and a text book, they mostly reported that the CAL program was the source that required them to think most deeply about the topic.

The CAL program helped the students to form an in depth understanding of the topic. Many students commented on the inclusion of a case study as being particularly helpful.

It must be noted that the students in group A were preparing for an examination question on the topic, and used other sources of information apart from the CAL program. This clearly influenced the motivation to study the topic, to the extent that some students reported taking notes from the program to use later. The question that was subjected to analysis using the SOLO taxonomy was the exam question on the topic.

Interpretative data was gathered from Group B also, this time by questionnaire. While the responses were not uniform, trends are apparent that indicate the following:

The students did not find the program easy to use.

The program content was seen to be related to the course content, however the reason for using the program was not clear to the students.

The program did not encourage the students to think deeply about the topic.

The program did not enable the students to form a deep understanding of the topic.

The students felt that the program did not provide enough information to effectively learn about the topic.

From this information it is clear that the programs differed from each other in terms of their effectiveness in meeting the learning needs of the students, in part because of the context in which they were used, and in part due to the effectiveness of the information provided and the learning activities required. Contextual factors are clearly important. The students in Group A saw a clear need for the knowledge contained within the program, and were happy to use it. Group B students, by contrast, did not understand why they were asked to use the program, and did not want to use it, despite the fact that the content was clearly related to their course of study. This aspect of lack of preparation is likely to have influenced performance.

It is difficult to assess by how much, however. The group A students were generally satisfied with the information provided by the program, and found it helpful in reaching an in depth understanding, a self report that is supported by the SOLO data. The group B students found it difficult to obtain the information that was needed, and reported that this did not provide a comprehensive explanation of the topic. These factors were clearly an important influence on the lack of depth that was generally evident in the results from this group.

Conclusion

The data from this study so far is not conclusive. Although neither program used generative learning strategies which, according to Jonassen (1988), means that the resulting learning will inevitably be superficial, many students in one group did show evidence of deep learning. As this group sat tor an examination, it is entirely possible that the students used their own generative learning strategies, such as writing summaries, and that these were instrumental in attaining the in depth learning as assessed by the SOLO taxonomy, even though they were not built in to the program. It is clear, however, that the programs differed markedly in their effectiveness as aids to learning. Further investigation is needed to establish the factors about program design that influence depth of learning.