Methodology

Systematic reviews are a method for identifying and synthesising research evidence in the literature to answer a specific question. They are common in the health field and they are increasingly being used in a variety of social policy fields. Systematic reviews use a thorough, systematic and transparent search process to identify relevant literature and clear criteria to assess the relevance and quality of the literature reviewed, in order to synthesise findings to inform decision making. A full description of the review process is provided in Appendix 1.

An important first step in a systematic review is to clearly define the inclusion criteria for the review. For the purpose of this review, we defined CLE (the intervention of interest) as education which has a primary legal focus and is provided face to face, either one-on-one or to a group (e.g. lecture style or as interactive workshops, role plays, legal theatre), either in person or via teleconference. The intervention had to be legal education as distinct from legal advice (i.e. tailored to the particular person’s situation) (Gander 2003, p.4). The CLE could have a primary focus on one or multiple legal issues. To separate CLE from school based education, we only included studies of education programs for adult community members or workers who support community members.

The search was also limited to published and unpublished Australian and international studies reported in English between 2000 and 2012. The studies were then further assessed to select only studies that evaluated the outcomes of community legal education (as defined above) and included outcome related data.

The 30 studies that met all of the above inclusion criteria regarding relevance to the scope of the review were then appraised for their methodological appropriateness. Studies were assessed regarding the following questions:

• Was the data/evidence collection methodology appropriate given the (apparent) questions or topics?

• Was the data/evidence collection methodology appropriately executed given the (apparent) questions or topics?

• Were the data reporting and data analysis procedures appropriate for an effectiveness study, and adequate to avoid significant bias?

Studies that did not have a sufficient description of either the CLE program, research methodology or the results were also excluded, as relevance and appropriateness could not be assessed. Table 1 summarises the process for selecting the CLE studies to be included in the review (see Appendix 1 for more detail).

Appraisal of the health education literature

Given the limited amount of literature relating to the effectiveness of CLE, and the greater culture of evaluation in the health field, we also collected a body of literature relating to health education in which effectiveness evaluation is more routinely practiced. We focused on systematic reviews and meta-analyses, since these studies had already assessed the appropriateness of the methodology employed. It was felt that there could be useful lessons for the CLE field from another sector, although it is acknowledged that there may be differences between the sectors which could have an influence on the impact of education.

The search process for this literature is set out in Appendix 1. The 21 systematic reviews of literature were appraised for relevance, and then for methodological appropriateness. Cochrane systematic reviews1 were assumed to be of sufficient methodological appropriateness for inclusion. Non-Cochrane reviews were assessed using the AMSTAR assessment tool for systematic reviews. The papers included scored a minimum of 5 points out of 11 using this tool, in all cases meeting what were assessed here as the most critical criteria relating to methodological quality.2

Fourteen papers were included in the study.

THE SMALL NUMBER OF STUDIES INCLUDED IN THE REVIEW

Across all sectors, it is common for systematic reviews to include very few studies. The small number of included studies results from both tightly defined research questions and limitations in the research identified. This is particularly the case for research undertaken of strategies in complex social settings, where rigorous outcome focused research is extremely challenging. The fact that this review only includes two studies on the effectiveness of CLE, both from North America, must be understood in this context.

More specifically, this review firstly only focused on studies which had assessed effectiveness using the tight definition of this term set out in Digiusto (2012), that is, studies which sought and were able to demonstrate that CLE caused an outcome. To demonstrate effectiveness in these terms takes a particular methodology that, due to the expertise and expense required, is not frequently used in the legal assistance sector.

Second, of the few studies that examined the outcomes of CLE, many fell short of the methodological standard required to establish effectiveness (as defined above) or did not provide enough information to assess whether or not the appropriate method had been used. Further, some studies were methodologically sound in some ways, but not others. For instance, they may have been able to identify outcomes, but not link them in a causal way to the CLE program, over and above other influences. Again, we stress the very real difficulties in evaluating the effectiveness of CLE due to factors such as the often short-lived nature of the intervention and challenges developing appropriate measures to adequately capture impacts on effectiveness (see Allmark et. al., 2012). Finally, it is important to note that a lack of definitive evidence demonstrating that CLE is effective does not necessarily mean that CLE is not effective, but rather that the quality of research studies have not allowed for a conclusion at this time.

Cochrane Reviews are systematic reviews of primary research in human health care and health policy, and are internationally recognised as the highest standard for assessing outcomes in evidence-based health care.

These criteria were: was an `a priori` design provided, was a comprehensive literature search performed

1 Cochrane Reviews are systematic reviews of primary research in human health care and health policy, and are internationally recognised as the highest standard for assessing outcomes in evidence-based health care.

2 These criteria were: was an `a priori` design provided, was a comprehensive literature search performed