Wendy Richards
New South Wales Department of Technical and Further Education

The purpose of this paper is to report on some aspects of the research
design and methodology developed to evaluate a number of women's access
courses provided by TAFE in NSW. The evaluation of the New Opportunities
for Women (NOW) program was designed as a two-part project, involving a
state wide survey of all participants since 1985 (Part 1) and a case study
of one course using a number of qualitative data gathering techniques (Part
2).1 The data gathering stages of both parts of the project have been completed,
and analysis of the results has confirmed that the research methodology
developed for the evaluation was appropriate and productive (the findings
from Part 2 are presented in Richards 1987).

To demonstrate the effectiveness of this two-part approach, and the
different techniques used within it, this chapter considers two aspects
of the evaluation methodology. Firstly, the discussion will show how the
methodological choices made during the process of project design and development
were influenced by the kinds of questions being asked of the research and
by the circumstances within which some answers could be found, These methodological
decisions were made prior to the conduct of the fieldwork phases of both
parts of the research. The second aspect of the evaluation methodology
considered here concerns the relationships that can develop between a researcher
and those who are the focus of the research, in a project where qualitative
methods are used over an extended period of time. In practice, these relationships
have important implications for the kinds of information that research
subjects are prepared to disclose. At the level of theory, the effect of
these relationships upon the process of data gathering are part of the
issue of scientific objectivity and the related questions of bias and the
distortion of data. Although this issue was taken into account during the
design stage of the project, its significance in practice was appreciated
more fully once the research had begun.

Women's access courses in New South Wales TAFE

Since the mid 1970s, both Federal and State governments have been concerned
with developing equal opportunity policies and initiatives, in particular
in employment and education. In response to these moves, TAFE Authorities
throughout Australia have formulated strategies to make their provision
more equitably accessible to all groups in the community.

As part of this recent commitment to equality of opportunity, several
of the States have developed programs designed to increase women's access
to TAFE and through increased education and qualifications, to those sections
of the workforce for which TAFE provides vocational training. The women's
access program offered within the NSW TAFE system is the New Opportunities
for Women (NOW) program. The NOW course is directed to women over the age
of 25 who have limited education or outdated vocational qualifications
who would like to undertake further education and training, and/or re-enter
employment, and who are uncertain how to go about this" (NOW Syllabus,
p.l). The course aims "to broaden and give direction to the personal
and educational options of mature age women beyond traditional notions
of what is appropriate for women" (NOW Syllabus, p.l), into those
areas of technical employment with prospects of secure and satisfying work.
The attention given in the course to technical occupations and qualifications
follows from the hope that this focus "will contribute to an overall
change in labour market opportunities for women at a time of rising under
and unemployment, when traditional women's skills are becoming redundant
in many industries and occupations" (Fell, 1983, p.l).

Many of the women in the NOW target group have withdrawn from the workforce
for long periods to raise children. It has been argued by course developers
that these women in particular lack the qualifications, skills and confidence
which are necessary for successful competition for jobs in today's tight
labour market. To this end the course also provides vocational guidance,
support and assistance which will maximise opportunities for re-entry into
further training or employment.

NOW courses are offered by TAFE in South Australia, Victoria, ACT, Western
Australia and Tasmania. In NSW, the NOW course began in 1982 at three TAFE
colleges. By 1986 over 80 courses were offered at both metropolitan and
country colleges. The course has been developed in four formats to meet
the needs of particular groups of women: NOW Base, NOW for Aboriginal women,
women of non-English speaking background and deaf women.

Designing a course evaluation: Some methodological
considerations

Several evaluations of NOW courses in NSW and other states had been
conducted prior to the development of the project discussed in this chapter.
In 1983, the first NOW evaluation in NSW, based on self evaluation and
questionnaires administered by course coordinators, found that students
were being drawn from the expected target group, and that their responses
to the course were generally positive (Fell, 1983). Unfortunately however
the evaluation was hampered severely by methodological difficulties such
as low response rates, non-comparability across research instruments and
too small a final sample to make statistical analysis possible. While many
of these difficulties were a consequence of the newness of the course at
the time of the research, the result was a very limited evaluation of the
course. No interviews were conducted with either students or staff to gain
greater understanding of issues such as the effectiveness of the teaching
strategies used in the course, the nature of group formation amongst students
and its role in the outcomes of the course, the possible sources of the
reported increase in students' self confidence and issues related to the
processes and dynamics of the course over time. Evaluations had also been
completed of the Western Australian, Victorian and South Australian TAFE
women's access programs (Pine, 1985; Jenkins, 1984; Jones, 1983 respectively).
However, in terms of their relevance to the design of the evaluation discussed
here, these previous evaluations were conducted within differing TAFE systems,
involved different combinations of access programs, and varied considerably
in their research objectives and choice of respondents.

Despite these variations in methodology, focus and research subjects
however, all previous evaluations produced findings and conclusions which
were relevant to the later NSW project. Each nominated childcare, timetabling,
accessibility and income support as the most influential factors in the
successful development and maintenance of women's access programs. The
draft report by Pocock (1985) supported these findings. This important
review of strategies for improving access and equity in education for women
within the TAFE system at both the national and State levels provided an
understanding of the larger institutional setting within which the various
NOW courses operated. However, given its purpose and methodology, this
research could not explore how the NOW courses work internally (ie, from
the students' point of view). Thus, although all previous evaluations arrived
at similar findings in terms of the factors which influence the success
or otherwise of women's access programs, none of them reported on how these
factors worked from the students' experience of them and none was in a
position to draw a broader picture of an established women's access course
offered across extensive areas of a State system. Thus the NSW course providers
had very little information, either from NSW or from experiences in other
States, about those issues considered central to the NOW program, namely,

the effectiveness of the course in facilitating students' re-entry
into work or further study

the relevance of the curriculum to the needs of the students

the effectiveness of the teaching and learning techniques employed
in the program

the composition of the target group.

During the design stages of the project discussed here, it was felt
that an evaluation which provided a State wide examination of the target
group and course outcomes, and at the same time a detailed, intensive investigation
into the day-to-day processes of the course and the women's experiences
of these, would provide a greater understanding of the course than had
been possible in previous evaluations.

It was decided that the most effective technique for obtaining a state
wide picture of the target group and the measureable outcomes of the course
would be a series of standardised questionnaires. However a detailed examination
of students' responses to the course over its 18 weeks would require a
methodology which allowed access to the day-to-day environment and activities
of the course. To accommodate these differences in research objectives,
the evaluation was designed as a two part project. Part 1 was a survey
of all course participants in NSW over the two year period from Semester
2, 1984 to Semester 1, 1986. During that period, questionnaires answered
by students at the beginning, the end and six months after completion of
the course, provided data needed for

construction of a socioeconomic, educational and vocational profile
of both applicants and enrollees

examination of the extent of unmet demand for the course and whether
the course caters for those who could benefit most from it

comparison of students' aspirations at the beginning, the end and six
months after completion of the course in order to measure the effect of
the course on students' future plans

examination of students' responses to the course overall, and to its
various components · comparison of the four NOW formats according
to the above.

While it was felt that questionnaires administered at particular points
during and after the course would yield standardised and statistically
meaningful data about all NOW participants in the State, a technique such
as this could not uncover the dynamics, interactions and processes which
develop throughout the course and which, according to teachers' and students'
anecdotal reports, contribute significantly to its outcomes. Course providers
wished to learn more about the processes of change which occur amongst
students during the course, the interactions amongst students and between
students and teachers, the relationships between the women's lives outside
the course (both past and present) and their experiences in the course,
and the ways in which mature age women learn and define what is relevant
to them in learning. These questions required a research methodology which
would enable the researcher to become a part of the research setting and
to be responsive to those participating in the course and to events as
they unfolded. Such a method also needed to be flexible to changes in the
research design if they were called for, be capable of being sustained
over a long period and be open to a variety of sources of data.

All of these methodological requirements needed to be met if a fuller
understanding of the course was to be gained from this evaluation. Since
it is possible to satisfy many of these requirements via the techniques
of observation, participant observation, open ended interviews and informal
conversations, the second stage of the evaluation was designed along these
lines. Thus Part 2 was to be an ethnographic study of the NOW course at
Gilmour College of TAFE in Sydney, commencing in Semester 1, 1986. This
part of the study sought to understand the effectiveness of the course
from the perspective of students.2

Unfortunately, over the four and a half months of the course, access
to all areas of the lives of the group at Gilmour was not possible. This
considerably restricted the scope of interaction with the students and
thus the range of data gathering opportunities. In light of these limitations,
the evaluation methodology became that of a case study rather than an ethnography,
based on a combination of participant observation during part of the course
and in-depth interviews with students both during and after the course.

As we can see, the decisions about methodology made during the process
of research design were determined partly by logistics. It is almost impossible
to interview meaningfully every member of a population of almost two thousand
students, while it is possible to ask questions of them through the mechanism
of a survey. These decisions were made also in consideration of the kinds
of research questions which could he asked and the nature of the data that
would answer them best. While it is possible with a large group of very
scattered respondents to gather up the widest sweep of information through
survey techniques, such techniques can only bring together certain kinds
of information. Information about dynamic and interactive processes which
require observation on the part of the researcher, as well as verbal description
and explication from those involved in these processes, would inevitably
fall through the far flung survey net.

Theorising method

The NOW course was developed originally within the framework of a feminist
analysis of the effects of the sexual division of labour on women's position
in the workforce, the lack of access for mature age women to vocational
training within TAFE, and the kind of specialised skills training and guidance
needed by mature age women who wished to re-enter employment or education
after long absences. Over the last two decades, feminist researchers and
theorists have documented the personal and collective experiences of women
in almost all aspects of their lives, and used these to elaborate theories
about the nature and origins of women's consistently subordinate positions
in larger social structures and their experiences of power and domination
in personal relationships. At the same time, feminist researchers have
developed extensive critiques of some of the dominant paradigms in social
science philosophy and method (Bowles & Duelli-Klein, 1983; Stanley
& Wise, 1983; Roberts, 1981; Westkott, 1979). Some of these criticisms,
which are particularly relevant here, point to the omission of women's
experiences, as they speak and feel about them, from theoretical and methodological
concerns in mainstream social science, the imposition of abstract, a priori
theory upon data which often is predetermined by the method used to obtain
it, the fragmentation of reality into discrete, methodologically and theoretically
"manageable" units of analysis, and the distinction made between
the researcher and the "object" of research in the interest of
scientific objectivity. Du Bois (1983) amongst others (Reinharz, 1983;
Duelli-Klein, 1983) argues for an alternative methodology which can make
good some of the sins and omissions of traditional enquiry. In her view,
research about and for (not on) women needs to generate concepts and frameworks
that are:

...firmly and richly grounded in the central experiencing of women.
And this demands methods of inquiry that open up our seeing and
thinking, our conceptual frameworks, to new perceptions that actually derive
from women's experience ... Our scientific methods ... require seeing things
as they are: whole, entire, complex ... (and) in context ... It is only
within its (context) that experience, reality can be known. And this (context)
includes the knower (1983, pp.110-11, emphases in original).

This last point, the relationship within the research setting between
the knower and the known, has had particular significance for the NOW evaluation,
as illustrated in the final section of this chapter. Positivist (and, as
Hammersley & Atkinson, 1983, argue, also naturalist) scientific paradigms
distinguish between the knowledge and actions of the researcher and those
of the individuals who are the focus of the research. This leads to their
"joint obsession with eliminating the effects of the researcher on
the data" (Hammersley & Atkinson, 1983, p.14). This determination
to remove researcher bias and distortion of data also means removing any
recognition of the possibility of a relationship between the researcher
and the researched. Even at the level of common sense and everyday experience,
however, this search for "scientific objectivity through detachment
from the research process" is doomed to fail, as we necessarily and
inevitably are part of the world we research. It is almost unavoidable
that some sort of relationship is established between the researcher and
the research subjects and that some effect flows from the researcher's
presence in the research setting or from intrusion into it through some
medium (eg, by telephone or mail-out questionnaire).

Rather than a source of bias and distortion, the outcomes of relationships
between researcher and researched may form part of the data, as Hammersley
and Atkinson point out (1983, p.15). For this to happen, "historical
truths (should be) grasped not by attempting to eliminate subjectivity
but through the intersubjectivity of meaning of subject and object"
(Westkott, 1979, p.426). This intersubjectivity does not mean, however,
that the researcher and the researched may identify with each other, but
that a "dialectical relationship" (Westkott, 1979, p.426) of
question and answer (and perhaps confrontation, evasion, counter-question
and silence) is established between them. It could be argued further that
it is by virtue of this relationship that in many instances we are able
to do research at all, and that often we increase our knowledge of social
reality because of the relationship between researcher and researched and
not, as is commonly held, in spite of it.

... (T)he mythology of "hygienic" research with its accompanying
mystification of the researcher and the researched as objective instruments
of data production [should] be replaced by the recognition that personal
involvement is more than dangerous bias - it is the condition under
which people come to know each other and to admit others into their lives
(Oakley, 1981, p.58, emphasis added).

During the two and a half months which I spent in the NOW course at
Gilmour College, the relationships that I established with the 15 women
in the group (who were aware of my research role) enabled me to gain an
understanding of what the course meant to its students beyond any that
had been possible in previous evaluations. During this time I was aware
that, in developing these relationships, what I saw, heard and noted down
as data was not only material absorbed by me, but was also in part created
by my presence there and by the history and nature of these relationships.
Within the group, as a researcher and a fellow student, I was both insider
and outsider. Thus at times I knew I was not being told everything and
yet, at other times and because of these relationships, I was often "privileged
with confession, the otherwise unsaid, the heartfelt and the bitter"
(Ball, 1984, p.17).

It has been argued that "idiosyncrasies of person and circumstance
are at the heart, not the periphery, of the scientific enterprise"
(Johnson in Bell & Newby, 1977, p.9). The centrality of the personal
to scientific inquiries into the social is demonstrated in the history
of one woman in the NOW course at Gilmour College. The significance of
the relationship between the researcher and the researched to that inquiry
is also demonstrated in her history, and our joint history in the course.
As the following discussion shows it was through coming to know each other
through our experiences during the course that I was privileged to gain
information that would have remained, under different circumstances of
research, the "otherwise unsaid".

Privileged information:
"What do you say when you talk to someone about it?"

Julie is a Maori woman in her mid-thirties who arrived in Australia
10 years ago with her two small children. She finished school in New Zealand
at 15 and, in the years before her marriage, worked mainly in textile factories
as a machinist. Since her divorce five years ago, she has been living on
Supporting Parents Benefit, with occasional part-time factory work.

This was the picture of Julie's life given by the information asked
of her on the NOW course application form and it was all that I or the
teachers knew of her when the course began. Throughout the next few months,
she often remarked that she had enrolled in the NOW course to "brush
up on my maths and English". However, as I eventually discovered,
her unstated agenda in coming to Gilmour was not to enrol in a short refresher
course in mathematics and English, but to find some solution to her problems
with literacy and numeracy. Julie was almost illiterate and innumerate,
yet although she hoped to find ways of changing this in the course, her
anxiety about not being able to read and write easily, her well developed
skills in concealing this inability and her great unwillingness to broach
the subject with anyone, including her close friends, made it almost impossible
for her to ask for the help she needed. Throughout her adult life, in situations
which looked like exposing her difficulties with words and numbers, she
would either bluff her way through them or find legitimate ways of withdrawing
from them altogether. With practice, she had become very successful at
determining the limits to what anyone would know about her ability to read
and write, her formal qualifications and her past educational experiences.
When these well rehearsed strategies were brought to a situation like the
NOW course, which works within a cooperative, sympathetic and non-threatening
teaching and learning environment, it was very easy for Julie to avoid
those moments in which the true extent of her illiteracy and innumeracy
could become known.

I fooled my way through, like I do a lot of things. When we had to write
things in class I'd sit there, you know, and watch the others. They were
so good at it. Then (the teacher) would say, "O.K. Have you finished?"
and I'd say, "Yeah, I've finished", even though, you know, I
didn't write a thing. I had it all in my head, right, like I'd get everything
ready in my head so if she's asked me I could say it. Then I'd hope like
mad she wouldn't come and look at it.

This determination to avoid exposure as an illiterate adult coexisted
uneasily with the often overwhelming need to find out just how bad the
situation was and what she could do about it. Her strong desire to "do
things with my life", and her confidence in her ability to take on
whatever she put her mind to if only she were "educated", were
frustrated by this ever present problem.

To me it's a very big problem. Because I don't want this you know. I
want to do so much, with my sons and with myself, and I can't ... and when
I think about that, that's depressing. It's a problem finding out what
level I'm at. I don't know where I am in my learning. If I knew that I'd
feel better about myself ... See, I know I've got the confidence in myself
to go out and do these things if I had the education. I'd be off, there'd
be no stopping me, you know, I'd do a lot, I know I would, but I can't
... And it's really hard to know what to do. What do you say when you talk
to someone about it? If I ring on the phone, do I say, "Hello my name's
Julie and I can't read"?

Although she gained in many other ways from her experiences in the NOW
course, Julie felt she had made no real advance in overcoming a central
problem in her life.

The first clue that Julie might be having trouble with words and numbers
came very early in the course when we worked together on a project that
involved writing. During this exercise I noticed that her handwriting was
the large, slow printing of early primary school years and that she was
very reluctant to show anyone what she had written. I thought that perhaps
she lacked basic literacy and numeracy skills and that, if this were the
case, then her educational needs and past experiences would be quite different
from those of the other women in the group. Without this original non-verbal
clue that my combined role of student and participant observer gave me,
I might not have been able eventually to understand the real nature of
Julie's experience of the course and to assess the extent to which the
course was of benefit to her particular needs. Because of the research
techniques used during the case study at Gilmour College, I gained an insight
into one person's experience of the course that other methodologies could
not have provided. From observation and participation, I discovered firstly
that the problem existed and secondly that, given her efforts to hide it,
it was likely that only under certain circumstances would she discuss it
and its effect on her life.

Qualitative research is often carried out in circumstances within which
it is possible to establish a personal relationship with those involved
in the research, through shared experiences over a period of time. The
importance of the relationships that are possible through this methodology
to the kinds of research questions that are being asked is illustrated
by the consequences of the relationship Julie and I established during
the course. Through observing the energy and skill she invested in camouflaging
her literacy and numeracy difficulties, and guessing at the anxiety behind
the spirited manner with which she maintained this cover, I sensed that
only if and when she felt comfortable with me could I talk about it with
her, and piece together the events which had placed her in this position.
Without this understanding of the events in her life, and without hearing
from her how she perceived herself and her needs, I could not assess the
strengths and weaknesses of the course from her point of view. And without
a relationship between us based at least on her confidence that she would
be understood and accepted, we could not arrive at the point of talking
directly about her needs of the course. It was through the experiences
we shared during the months of the course that such a relationship grew,
until eventually eight months after the course began we were able to discuss
at length how it came about that she could barely read and write.

Julie's family had moved constantly around New Zealand while her father
looked for seasonal work and battled with the financial difficulties of
raising and educating a family of 14 children. For Julie the result was
a badly disrupted primary school career, after which she left high school
at 15 with no idea of her real educational level, or of her abilities.

Like, we'd move to one school and that'd be great, and then we'd move
on, and then by the time high school came the teachers more or less had
to send me on because I was too old. And when I got (there),just honestly,
I didn't know where I was going so I just didn't bother. I'd try but it
didn't make any difference ... I was lost, you know, I really was, I was
absolutely lost. I really didn't know what I was learning, it was just
too fast for me. I didn't know what level I was at and I really thought
there was no hope.

The strategies that she used in the NOW course were echoes of those
which she began using at school and of the "bluff' she often described
using to get jobs later on. Her unwillingness to talk about her school
years seemed to stem from a sense of shame at being the only one she knew
of like this and from a great fear of ridicule.

At school it got to the point where I was too embarrassed to say that
I didn't know and put my hand up in front of the class and say it ... I
was so far behind anyway that I didn't speak out and say anything because
I didn't want to be put aside and treated as someone different or a bit
slow... [And when the teacher said], "Does everyone know it?"...
I'd just agree and say, "Yeah, I know it". What could I do? ...
I haven't told anyone. Actually I've told you and another friend of mine
... it's funny, she's the only one of my friends I've ever told. I just
felt good talking about it with her because I knew in my heart she wouldn't
laugh at me, that she wouldn't talk about it or judge me. I know a lot
of people would. They'd find it quite funny actually, you know, make fun
of me. That's why I don't tell them.

It is highly unlikely that Julie would have written these words on a
questionnaire, or spoken them to an unknown interviewer asking for her
opinions of the course. From her account, she only became willing to talk
to me because of the degree of trust and understanding that had developed
between us. ("I feel like I can talk to you about it, you know. You
understand what I'm trying to say, you don't make me feel bad.") The
opportunity and circumstances for that trust and understanding to grow,
to form the basis of our relationship, were provided by the place in the
course that the research role of participant observer gave me. Without
that relationship, and the questions and answers that moved to and fro
within it, important insight into the workings of the NOW course would
have been lost.

Conclusion

This paper began by arguing that the choice of research methods used
in the NOW evaluation was governed by the nature of the research questions
being asked. The NOW course was developed on the basis of a feminist analysis
of the educational needs of socio-economically and educationally disadvantaged
women, particularly in relation to curriculum content and teaching and
learning processes. The program deliverers, who instigated the evaluation,
required different kinds of data to measure the accuracy of that analysis,
and to explore the effectiveness of the program which was developed from
it. These were, firstly, State-wide information about the composition of
the target group, postcourse activities of students and variations between
the formats of the course and regions within the State and, secondly, a
more detailed and holistic understanding of mature age women students'
responses to a re-entry course of this kind.

In the design stage of the evaluation, it was decided that only qualitative
research methods could yield the type of data necessary to assess the effectiveness
of the course from the students' experience of it. This brief account of
the relationship that developed during the course between Julie and me
illustrates the usefulness of these techniques, and just how misleading
data collected via quantitative methodologies alone might have been in
understanding the complex elements of this experience.

Endnotes

Part 1 of the NSW evaluation was funded from the TAFEC Designated Grants
Program (Curriculum and Education Resource Development Funds). Part 2 was
funded by the TAFE National Centre for Research and Development.

The names of the college and all those who participated in the course
which was the subject of the research are pseudonyms.