http://dx.doi.org/10.1590/S1414-32832007000100011

IResearcher, Department of Health
Administration and Planning, National School of Public Health, Oswaldo Cruz
Foundation, Rio de Janeiro, Brazil. <virginia@ensp.fiocruz.br>
IIProfessor, Department of Health Planning, Institute of Community
Health, Fluminense Federal University (UFF), Niterói, Rio de Janeiro, Brazil.
<lilian@vm.uff.br>

ABSTRACT

This article analyzes the uniqueness of methodologies
used in the quality assessment of graduate studies programs in Public Health
in two Latin American countries: Argentina and Brazil. The authors conducted
a search in websites that addressed themes such as "public health training and
education" and "quality assessment methodologies", besides reexamining documents
and bibliography on the theme. The analysis took the following dimensions into
account: the Latin American political and economic context during the previous
ten years (1994-2004), reforms in the educational systems (emphasizing graduate
studies), and the quality assessment methodologies implemented. The authors
found similarities in the evaluation systems in effect in both countries, as
well as shortcomings in these processes, namely, evaluation driven by supervision
and control rather than redirection and reorientation of teaching, and the mandatory
adjustment of courses to a standard model defined by experts.

In 2004, during a presentation in a Seminar2,
Dr. Mirta Roses Periago, Director of the Pan-American Health Organization (PAHO),
summarized the macro health trends in Latin America and the principal challenges
for the 21st century. Among the trends, she highlighted the population
increase and population aging, leading to a more complex epidemiological profile,
and the population shift to urban areas, along with increased poverty and socioeconomic
inequalities. According to Dr. Roses Periago, the two central tendencies in
globalization  democratization and decentralization (of power, information,
technology, and development)  are leading to profound political, economic,
and social changes. She further pointed out that the fundamental challenges
for health systems are to guarantee social protection in health for all citizens,
contribute to the elimination of inequalities in access, guarantee quality services,
provide the opportunity for comprehensive care for excluded social groups, meet
the population's health needs and demands, and eliminate the ability to pay
as a restrictive factor.

In the scenario described above, the tendency
among graduate programs in public health3, including graduate studies
stricto sensu (Master's and PhD programs) and lato sensu (specialization
courses) is to focus on health and life (and not only disease and death), to
concentrate more on reflection concerning the social subject as a fundamental
category for understanding reality, and to complete one's scientific knowledge
in the graduate program and return to the health service and further develop
evidence-based themes. The quality assessment of this training (as a strategic
component) has still not been fully developed.

There are currently at least four types of quality
assessment models being used in higher education: (1) the American model, in
which assessment is focused on reaching institutional goals and pre-established
standards; (2) the continental European model, conducted by peers and whose
focus is the academic program; (3) the British model, which conducts peer assessment
and uses performance indicators; and (4) the Scandinavian model, a variant of
the continental European model, which includes self-assessment and external
assessment processes (Royero, 2002).

Considering the recent context in which the discussion
of quality assessment is carried out, the current article aims to analyze the
uniqueness of methodologies used in quality assessment of graduate studies in
Public Health, with a focus on graduate programs stricto sensu in two
Latin American countries, Argentina and Brazil.

The choice of these two countries was not random,
but based on the observation that they show convergences at the socioeconomic
and political levels. Both are developing countries, with their pasts marked
by democratic instability, and who have been cooperating in various fields for
more than 15 years, including in higher education, consolidated recently with
the Educational Mercosur4.
They are also the countries in which quality assessment of graduate studies
programs (Master's and PhD) has been conducted regularly. Since the 1990s, the
replacement of the so-called welfare state with a state that evaluates services
provision  while decentralizing the administration of education 
has had a major impact on the dynamics of economic and political power, leading
to profound changes and reforms in the systems of higher education in Latin
America (Mollis, 1999).

The period of neo-liberal reforms in Latin America,
beginning with the educational reform in Chile in the 1980s (González, 2003),
expanded steadily in nearly all of the countries in the region, and beginning
with the re-democratization of these countries, some public universities (Mexico,
Colombia, and Argentina) adapted to the demands of an international agenda promoted
by the World Bank (WB) and later by the Inter-American Development Bank (IDB),
although some countries (Bolivia and Brazil) put up serious institutional resistance
(Mollis & Bensimon, 1999).

To achieve this study's objective, we proceeded
to a non-exhaustive search in the websites that approached the themes "public
health training" and "quality assessment methodologies", in addition to a literature
review of these themes. As for the documents, for Argentina we prioritized those
referring to the main quality assessment agency, the National Commission for
University Assessment and Accreditation (CONEAU), and for Brazil, the documents
from the Evaluation Division of the Coordinating Body for the Enhancement of
Graduate Studies (CAPES). Since this study adopted a qualitative approach with
a strong descriptive component, the aim was not to make it reproducible on a
large scale, but to highlight its comparative relevance for other studies in
Latin America. In this sense, the analysis focused on the following dimensions:
the political and economic context in Latin America in the previous ten years
(1994-2004), reforms in the educational systems (with an emphasis on graduate
studies, stricto sensu), and the methodologies used to implement quality
assessment.

The purpose is thus to contribute to the discussion
on the implementation of quality assessment processes for graduate studies courses
or programs stricto sensu in Public Health, whose methodological procedures
are in keeping with the local and regional realities.

Reforms in higher educations: the impacts
on training in health and the relationship to quality assessment of graduate
studies in Public Health

In the 1990s, educational assessment became one
of the strategic thrusts of educational reforms in Latin America. In the field
of Public Health, the quality assessment systems have become essential components
of the management systems in graduate training.

In a study conducted in 2002 for PAHO (Davini
et al., 2002) to analyze the limits and reach of training processes (lato
sensu) for Public Health professionals, based on the permanent education
focus, it was observed that many of the programs only used the educational assessment
procedures partially; in other words, they concentrated on process assessment,
but the latter was not followed by an evaluation of the results. Meanwhile,
other programs were using assessment instruments that were not properly linked
to the proposed educational objectives, thus turning assessment into a mere
bureaucratic tool. The study concluded that educational assessment (whether
normative or strategic) was given secondary priority, despite great progress
in the field of educational assessment in the last twenty years.

Before proceeding to an analysis of the methodological
components of the quality assessment proposals in Argentina and Brazil, we will
present a brief background on the creation of the two university systems, seeking
to draw a parallel between the respective reforms in the areas of health and
higher education and the quality assessment processes for graduate studies.

The Argentine university system was created in
the 18th century, with a Jesuit orientation and an emphasis on the
humanities. Beginning in the 19th century, the French model was introduced
for the transmission of knowledge by means of theoretical and erudite teaching.
Meanwhile, Brazil was one of the last countries in Latin America to create its
university, and its higher education system dates to the 19th century.
The Brazilian model was also inspired by the French model, and the concept of
integration between teaching and research only materialized after the university
reform of 1968 (Morosini, 1994), when the graduate studies programs were implemented
based on the U.S. model. In Argentina, it was only in 1992, with the Federal
Education Act, that there was reference to the educational system as a whole,
including graduate studies, whose university institutions had already been offering
courses since the late 1980s (Fernández Lamarra, 2002).

In Latin America, during the 1990s, proposals
for reform in health and higher education were part of an overall regional arrangement
for public policies in this area, with financial support and strategic monitoring
by institutions like the WB and IDB (Hortale et al., 2004). The quality
assessment models implemented in higher education varied from country to country,
but adopting similar logics to the four models described previously.

In Argentina, the 1990s witnessed a major expansion
of graduate studies courses (both lato sensu and stricto sensu),
due to such factors as the increase in the number of public and private universities,
the consolidation of the research field, and the establishment of a new "professional
market", with the demand for greater competitiveness and the incorporation of
new technologies. From the point of view of pedagogical processes, this field,
although less bound to the traditional institutional and teaching practices
of the universities, maintains unaltered the course objectives, didactic/pedagogical
contents, and assessment processes (Fernández Lamarra, 2002).

Until 1995, when the Higher Education Act was
passed, there was no agency for the control or assessment of undergraduate and
graduate studies courses. The creation of CONEAU in 1996 launched a process
focused on establishing a system for the assessment and accreditation5
of these courses. Its objectives are: to conduct institutional assessment in
all the universities; to accredit undergraduate and graduate courses; to issue
recommendations on projects for new universities; and to recognize private institutions
(Koifman, 2004).

In Brazil, from the 1960s to the 1980s, local
experiences with assessment were conducted in the universities, with limited
integration; with the country's re-democratization in the mid-1980s, this process
became institutionalized in two watersheds in the Ministry of Education: institutional
assessment, and quality of the courses offered. However, the area of graduate
studies stricto sensu was already a step ahead, and had been conducting
assessments since 1976. Despite the limits, assessments slowly became mandatory
and were incorporated into the agenda of institutions of higher learning. The
year 2001 witnessed the creation of the National Education Plan, defined for
a ten-year period, including the National Assessment System and mechanisms for
monitoring its targets (Brasil, 2001). This proposal contains the first explicit
reference to the concern for expanding higher education with quality. Among
its 23 goals, five refer to the institutionalization of an internal and external
assessment system, with the aim of (re)accreditation and promotion, encompassing
the public and private sectors in the institutional and course-related dimensions.

In 2004, the National Graduate Studies Plan (PNPG
2005-2010) (CAPES, 2005) was launched, consolidating what had been presented
in the previous plans (I, II, III, and IV). This plan proposed that assessment
be based on the quality and excellence of the results, the specificity of the
areas of knowledge, and the impact of the results on the academic and business
communities and society as a whole. Each area of knowledge was expected to show,
using comparable indicators, its scientific and social relevance in the national
and international context, although the plan does not define the indicators.

The field of Public Health, which is by nature
multi-professional and interdisciplinary, established itself in this scenario
with graduate studies (both lato sensu and stricto sensu), and
over the years, the teaching institutions adjusted their pedagogical projects
to a health reality which is now much more complex, as described in the introduction
to this article. The competencies of Public Health professionals were progressively
restructured, requiring new skills, capacities, and attitudes. According to
Rovere (2003, p.7):

The quality of a product or service  in
this case, education in Public Health  materializes in the value that
this training adds to professional practices in the field, assessed from the
perspective of the needs and expectations of employers, professionals, and beneficiaries:
the students, health services and the population. If this added value is not
stable  it can increase or decrease according to what is offered by the
course as a function of the students' expectations and needs, always in transformation
 then quality becomes a relative concept, a comparative value, always
socially and historically situated. In the final analysis, the value judgment
of the product or service results from a comparison with other alternatives,
and can always improve. That is why it is said that quality is the true target,
which moves with the horizon, and cannot stop due to the lack of an on-going
effort to improve its proposals.

In Brazil, if the Health Reform process consolidated
in 1988 influenced the curriculum reform implemented in all the health professions
in the late 1990s with the Curricular Guidelines, reflecting on graduate studies
(lato sensu and stricto sensu), in Argentina the proposals for
curriculum reform, especially in undergraduate training in Medicine, are still
not integrated with the proposals for health sector reform, focused on primary
care.

Taking as an example the School of Medicine at
the University of Buenos Aires, the activities are developed mainly in the hospitals
(university and non-university). This suggests that there is not a strong link
between the course and the public health system, and that medical training excludes
knowledge about the system in which the future physicians will work (Koifman,
2004). This characteristic differs considerably from the current trend in the
Brazilian undergraduate health courses, based on the Curricular
Guidelines6.

The Case of Argentina: standards compatible
with what?

In Argentina, together with the Higher Education
Act, CONEAU was created in 1995 and implemented beginning in 1996, conducting
assessment and accreditation of graduate studies programs (stricto sensu)
in the field of Public Health. CONEAU is in charge of requiring the courses
to submit a self-assessment, based on a questionnaire, previously prepared and
supplied to them. Subsequently, a committee of peers (academics and professionals
with teaching and administrative experience) meets to examine the documents
of various courses in the same specialty, prior to consulting the universities
on possible critiques from the peers.

CONEAU takes the peers' opinions into account
in elaborating a report, and classifies the course or program in categories
A, B, or C, according to its quality. Ruling 1.168 of the Ministry of Education
sets the minimum quality criteria, and in the fields of Medicine and Public
Health, the overall standards for any given type of graduate studies program
were complemented by the Report of the Advisory Committee on Graduate Studies
in Health Sciences.

The principal responsibilities of CONEAU are
summarized in Table 1. The Act that established
CONEAU also provides for the creation of private agencies to conduct such procedures.
According to Fernández Lamarra (2002), such agencies differ considerably from
other institutions or agencies to the extent that they act both in the assessment
and accreditation of government and private institutions, and undergraduate
and graduate careers7.

In the definition of standards and criteria,
adaptations were made from methodologies used in other countries like the United
States, France, England, Netherlands, Spain, Chile, and Brazil. The following
institutional assessment principles were adopted: to learn, understand, and
explain how the universities function in order to improve them; to contribute
to the enhancement of their practices, especially those related to decision-making
and improvement in the stakeholders' understanding of their institutions, encouraging
reflection on the meaning of their activities (Fernández Lamarra, 2002).

Table 1 also
summarizes the standards used in the assessments. Each standard is subdivided
into various criteria.

For the standard "institutional framework of
the course", four criteria are defined: (a) the institution's proposal in relation
to prevailing regulations, resolutions, or rulings in the graduate studies system;
(b) regulation of the specific functioning of the career for which the accreditation
is requested; (c) definition of the career's objectives, academic program foundations,
and activities for the theme at hand; (d) consideration for the presentation
of joint or inter-institutional careers, with the objective of taking advantage
of the academic, scientific, and technological potential of the country's university
institutions, in association among themselves or with foreign institutions,
which, in a joint effort to improve the educational supply, combine the sufficient
human and material resources. The requirements for the career's accreditation
can be complemented by means of cooperation among the respective institutions.
For this to happen, a specific agreement is indispensable between the respective
academic units.

The standard "academic program" includes three
criteria: (a) project (background, academic and/or professional relevance, objectives,
admissions requirements, academic activities program, thesis rules, student
follow-up and grading methodology, and conditions for granting degrees); (b)
personalized Master's and PhD programs (the institutions may offer a Master's
or PhD modality in which the program is presented by the thesis or dissertation
supervisor as a function of the proposed theme); (c) course load (distribution
of course load in units with different durations and formats, like courses,
seminars, and workshops).

The "student body" standard involves two criteria:
(a) number of regular and visiting students and (b) number of thesis/dissertation
supervisors.

The "faculty" standard includes three criteria:
(a) admissions policy, processes, and conditions, grading, passing, and awarding
of degrees; (b) adequate thesis supervision vis-à-vis the specific academic
program and degree; (c) collection and systematic organization of data on trends
in enrollment, passing and retention rates, and degrees, and of all other important
information in this regard.

The "infrastructure" standard includes four criteria:
(a) installations and equipment (adequate access, laboratories, staff, and teaching
materials for the respective activities, in keeping with the needs generated
in performing such activities); (b) library (access, collection, and updating);
(c) information technology (access to adequate computer equipment and information
and communication networks for the needs of the respective activities); and
(d) availability for professional research and practice.

In a meeting in 2003 (Borrel, 2004), the representatives
of graduate studies courses (strictro sensu) in Public Health concluded
that the criteria used by CONEAU were insufficient, since the assessments failed
to take into consideration the distinctive elements of training in the field
of Public Health, which has a triple foundation (teaching/research/management),
thus failing to determine whether the course had the necessary quality for the
proposed level of training (lato sensu versus stricto sensu).

The Case of Brazil: do the standards meet
specificities?

In Brazil, the government agency that standardizes
and monitors graduate studies programs and courses (stricto sensu) is
CAPES. As of the last triennial assessment, in 2004, a standard instrument was
used according to the sub-area of knowledge, with weighted criteria according
to the specific area of assessment. For the sub-area of "health"  whose assessment
areas are: Physical Education, Nursing, Pharmacy, Medicine I, Medicine II, Medicine
III, Dentistry, and Collective Health - the instrument has seven items, including
the following criteria, with the respective weights.

For the item "program proposal", five criteria:
(a) coherence and consistency; (b) adequacy and scope of the course majors;
(c) adequacy and scope of the research lines; (d) proportion of faculty, researchers,
student authors, and other participants; and (e) infrastructure (laboratories
and computer and financial resources).

For the item "faculty", four criteria: (a) composition
and activity, institutional affiliation, and work contract; (b) size of the
so-called faculty reference nucleus 6 (NRD6) in relation to the faculty as a
whole, and activity by the NRD6; (c) scope and specialization of the NRD6 in
relation to the course majors and research lines; and (d) faculty exchange or
turnover, and participation by other faculty members.

In the item "research activities", six criteria:
(a) adequacy and scope of the research lines and projects in relation to the
course majors; (b) link between research lines and projects; (c) adequacy of
the number of research lines and projects under way in relation to the size
and qualifications of the NRD6; (d) student participation in the research projects;
(e) financing, including participation by institutional research promotion agencies
and other sources; (f) development of collaborative and inter-institutional
research lines and projects.

For the item "training activities", four criteria:
(a) adequacy and scope of the curricular structure in relation to the program's
proposal and its course majors or research lines. Adequacy and scope of the
various disciplines in relation to the research lines and projects; (b) distribution
of course load and average course hours, compatible with the size of the NRD6,
participation by other faculty; (c) number of thesis/dissertation supervisors
in the NRD6 in relation to the size of the faculty, and distribution of thesis
supervision among the faculty and average number of supervisees per faculty
supervisor; and (d) course activities and supervision in the undergraduate courses.

For the item "student body", five criteria: (a)
size of student body in relation to size of the NRD6; (b) number of supervisees
in relation to size of student body; (c) number of degrees awarded and dropout
rate in relation to size of student body; (d) number of authorial students in
the graduate studies program in relation to size of student body; (e) activities
involving integration between graduate and undergraduate studies.

For the item "theses and dissertations", four
criteria: (a) link between theses/dissertations and course majors and research
lines and projects, and adequacy vis-à-vis course level; (b) average time-to-degree
for scholarship students, average scholarship time, ratio between average time-to-degree
comparing scholarship and non-scholarship students; (c) number of degrees in
relation to size of NRD6, and participation by other faculty; and (d) qualifications
of thesis/dissertation review panels, and participation by members from outside
institutions.

In the item "academic output", three criteria:
(a) adequacy of types of output vis-à-vis the program's proposal, and link to
the course majors, research lines and projects, or theses and dissertations;
(b) adequacy of the channels or means for publication/dissemination, amount
and regularity in relation to size of NRD68,
and distribution of authorship among faculty members; and (c) student authorship
and co-authorship.

In 2001, foreign observers present at the triennial
assessment conducted by CAPES made some insightful comments on the assessment
criteria. A recent article by Hortale (2003) discusses the characteristics of
these criteria and the issue of quality of teaching (an aspect that has received
little attention under the current CAPES assessment model), identifying some
present trends, for example, that "transforming these assessments, which
have been predominantly quantitative and insufficient for verifying quality
improvement in the proposed education, can contribute to better organization
of efforts at the institutional, faculty, and student levels" (p. 1840).

Although Brazil passed a law in 2004 that established
the National Assessment System for Higher Education (Brasil, 2004), it is limited
to undergraduate courses, leaving CAPES with the responsibility of assessing
the country's graduate studies programs.

Meanwhile, the current National Graduate Studies
Plan (PNPG), in referring to assessment, states that it should be based "on
the quality and excellence of the results, the specificity of the areas of knowledge,
and the impact of the results on the academic and business communities and society
as a whole" (CAPES, 2005, p. 63). Atlhough the PNPG reaffirms the need for
the indicators to reflect "the relevance of the new knowledge, its importance
in the social context, and the impact of the technological innovation in the
global and competitive world" (p. 63), it does not suggest changes in the
criteria in the current assessment system implemented by CAPES.

Final remarks

The current study, although limited due to its
predominantly descriptive component, analyzed the uniqueness of methodologies
used in the quality assessment processes for graduate studies in Public Health
in two countries of Latin America: Argentina and Brazil.

Both countries are currently reformulating their
educational systems, including the implementation or consolidation of assessment
processes. Although there are similarities in various aspects, examination of
the Brazilian and Argentine realities shows differences that should not be overlooked
(Koifman, 2004).

One similarity relates to the nature of the agencies
working in the graduate studies area in the two countries, CONEAU and CAPES,
which is not to assess the quality of courses and programs. In a previous article
(Hortale et al., 2004), quality was interpreted, as proposed by Royero
(2003, p. 2), as "a fundamental category for conducting the assessment of
the world's institutions of higher learning", highlighting its attributes
as: (a) multidimensional and complex, encompassing educational and social processes;
(b) a socially determined category, related to specific socioeconomic contexts;
(c) associated with the on-going transformation and adaptation of educational
systems; and (d) integrated with the process of social efficiency in institutions
of higher learning.

Based on the above definition, we observed that
in the Argentine case, CONEAU has the responsibility of both assessing and accrediting
the courses, using similar criteria to those of quality attributes. Meanwhile,
in Brazil, CAPES, which was originally an accreditation agency, conducts the
assessment of re-accreditation processes without specifying the quality criteria
it utilizes, as demonstrated in a previous study (Hortale, 2003). The fact that
CAPES uses a more improved assessment instrument can be explained by its more
than 30 years of experience, as compared to CONEAU, which was created in 1995.

We also identified some weaknesses in these processes.
Both cases display ambiguity in the concept of quality in higher education.
In Brazil, the quality of higher education is only defined in terms of academic
excellence, that is, according to the number of research projects and publications.
Besides being insufficiently defined, the criteria for quality in higher education
are not associated with the social use of the knowledge produced by scientific
research or acquired by students. Spagnolo & Calhau (2002) point out that
the CAPES assessment system focuses more on research than on quality of teaching.
In its assessment instrument, there are no indicators for specifically assessing
the teaching methods, and quality is inferred from an analysis of the number
of publications, faculty qualifications, number of theses and dissertations
supervised, and the faculty's workload in the program.

In both countries, assessment is still associated
with supervision and control, rather than with redirecting and reorienting the
teaching. One can identify an isomorphism in the two countries' assessment systems,
namely, that the courses must adjust to a standard model defined by experts.

Finally, accumulated experience with the assessment
of higher education in both countries, although still insufficient in terms
of its quality attributes, allows further improvement of the procedures, as
proposed in the National Assessment System in Brazil (although, as mentioned
previously, it is limited to undergraduate courses), and by CONEAU in Argentina.
Such progress in itself would be a major contribution to improving the quality
of higher education in the two countries.

1 Rua Leopoldo Bulhões, 1480, Rio de Janeiro,
RJ 21.041-210, Brazil.2 "The Future of Public Health and the Millennium
Development Goals", September 15, 2004, National School of Public Health, Oswaldo
Cruz Foundation (ENSP-FIOCRUZ). 3 This article uses the term "graduate
studies programs, stricto sensu", commonly used in Brazil to refer to
Master's and PhD programs, but excluding shorter specialization courses (lato
sensu), although many other countries do not make this distinction. 4 Although it is predominantly trade-related,
the Southern Cone Common Market or Mercosur operates an educational department
with the objective of incorporating the educational dimension into the economic
integration process. This department, called Educational Mercosur, has held
periodic meetings since 1991, and the ministers of education from the four countries
signed a Protocol of Intent that includes the following areas: "formation
of a citizens' awareness in favor of the integration process; human resources
training to contribute to development; harmonization of the educational systems " (Piñon, 1997, p. 187). 5 In the references in Spanish, the term acreditación is used to define course accreditation procedures, although
the meaning we adopt is that of a "strategic practice to monitor and guarantee
academic and pedagogical quality" (Hortale et al ., 2002). For purposes
of discussing the Brazilian and Argentine experiences in assessment, in this
article we chose to use the term credenciamento in Portuguese, which
gives a more exact idea of the content of the respective experiences. 6 In Brazil, the development of curriculum
guidelines in the health field occurred from 1999 to 2001, in a process involving
broad mobilization of the health schools and other stakeholders, who succeeded
in submitting the proposals that were contrary to an initial version that had
been presented and which they considered conservative. As a result, the guidelines
submitted for the courses in Medicine reflect the proposals by the movements
for change in the field (ABEM, CINAEM etc.). 7 Until the 1990s, in Argentina there were
no private medical courses. The opening of private medical schools was accompanied
by a concern for some level of monitoring and regulation. From this point on,
projects began to be implemented for the assessment programs, accompanied fundamentally
by the work of CONEAU. Furthermore, the discussions in Educational Mercosur
emphasized the need to establish curriculum guidelines and standards for the
courses in Medicine.8 NRD6 refers to the faculty reference group for assessment, and consists of
faculty members in charge of defining and consolidating the Program's proposal
and research lines and projects.