Federal Research In Vocational Ed.Said 'Inadequate'

Panelists conducting the first "outside" assessment of the
government's six-year-old, $25-million center for research in
vocational education have concluded that it "lacks research objectives"
and has failed to make the "significant improvements" in the nation's
vocational systems envisioned by the Congress when it established the
entity.

The panel found that the National Center for Research in Vocational
Education (ncrve) has been well managed and has complied with federal
mandates. But in a report submitted to the Education Department's
office of planning, budget, and evaluation (opbe), the panelists
described publications produced by the center's staff as "uncritical,"
"superficial," and "too narrow" in focus.

"Specifically, the panel felt that ncrve's [research publications]
simply have not provided the information that is needed to isolate the
factors which result in vocational-education programs that are
effective and equitable," the report asserts.

The eight-member panel, assembled last June by a Washington, D.C.,
area consulting firm that contracted with the Education Department's
evaluation office to conduct a review of the center's publications
program, included state and local educators and representatives from
higher education, foundations, and industry.

Created by Congress in 1976, the ncrve is administered under
contract by Ohio State University and is responsible for conducting
research on programmatic and policy issues of "national significance,"
collecting statistics, and providing technical assistance and inservice
training. Since 1978, the center has received more than $25 million in
federal funding; its activities are monitored by the Education
Department's office of vocational and adult education (ovae).

As the national clearinghouse for the dissemination of information,
the center is part of a $6-billion vocational-education enterprise that
serves more than 16 million students attending secondary and
postsecondary institutions nationwide.

In the evaluation report, the panel criticized the vocational
office's handling of the center's current five-year contract and
concluded that the responsibility for the center's problems should be
shared by both agencies.

In general, according to the report, "the overall relationship
between ncrve and ovae is too fraternal and not conducive to an
aggressive, change-oriented program of research and development."

New Five-Year Contract

In January, the Education Department awarded Ohio State a new
five-year contract, worth $4 million per year and renewable
annually.

A member of the evaluation panel said the group was told at its
initial briefing last spring that the assessment of the center was
being undertaken because "a funding decision" would have to be made "in
the near future." The panelist said he surmised that the department
expected the panel to make a "harsh but fair" appraisal of ncrve's
performance that could be used to "open up the selection process and
avoid an automatic renewal of the contract with Ohio State."

Only two organizations, Ohio State and the University of Tennessee,
bid on the contract. Sources said a "pro forma" letter from Senator
Howard H. Baker Jr., the majority leader, supporting the Tennessee
university's bid touched off an extensive mail campaign from supporters
of Ohio State.

In addition to receiving "stacks of" letters from researchers and
the vocational-education community, the department received letters
from members of the both the House and Senate concerning allegations of
unfairness, the sources said. Ohio's Democratic Senators, along with
leaders of the House Education and Labor Committee, wrote the Secretary
of Education that they were "concerned that factors other than quality
and capability are entering into the award process." (See Education
Week, Nov. 17, 1982.)

Faced with "intense pressure to keep the contract at Ohio State,"
said a source, who asked not to be identified, the department decided
not to make the evaluation panel's critical assessment of ncrve (which
is dated October 1982) available to the department employees and
another group of outside experts, who were evaluating the proposals for
the new contract, even though the assessment was available to
department officials at that point.

Robert M. Worthington, assistant secretary for vocational and adult
education, said last week that Ohio State was awarded the federal
contract after a thorough analysis of the two proposals. He said the
department's decision was based on the quality of Ohio State's proposal
and its past experience in operating the center.

Ohio State's bid for the contract was higher, Mr. Worthington
explained, because "Tennessee did not bid on all the functions"
specified in the department's request for proposals. The department's
decision, he said, was "not a question of cost, but quality."

Mr. Worthington said his office "formally" received the panel's
report three weeks ago.The report is being reviewed by ovae's research
staff, members of the national advisory council to the Ohio State
center, and the center's staff, he said.

"We don't have any intention of moving immediately on [the panel's
findings]," Mr. Worthington said. The panel was "given the contract to
evaluate the center's printed products; they did not talk to us about
our relationship to the center or the center's relationship to us.

"Not a single member [of the panel] visited the center or conferred
with the center's staff," Mr. Worthington added. "My understanding of
the report is that it was a small contract to look at the center's
publications; they were not asked to comment on the center."

"There have been very few criticisms of the center's work," Mr.
Worthington said, adding that he is "confident" that the center will
continue to produce "the kind of quality work they have done" in the
past.

Robert E. Taylor, executive director of the ncrve, said the panel's
report is "disturbing" because only 10 percent of the center's
publications were evaluated and those publications reflect three of six
functions mandated under its contract. "Those functions constitute only
37 percent of the budget during that period," he said.

"We have been audited to death and all the evaluations have been
positive," Mr. Taylor said, adding that many of the findings in the
recent evaulation "do not represent a consensus on the panel."

Mr. Taylor would not comment on the political nature of the report,
but he said that the lobbying efforts of several professional
organizations "assured an outside peer-review panel."

Acknowledging that the center has "real problems of resource
contraints," Mr. Taylor said the panel's recommendations would be
difficult to implement unless changes are made in the center's contract
requirements.

"In some respects, the report did not deal with the differences of
opinion over what the center ought to be within the department
itself,'' Mr. Taylor said.

The panel's conclusions were based on a review of work produced by
the ncrve between January 1978 and January 1982, according to the
report, which was prepared by Technassociates Inc., of Rockville, Md.
Although the center has been evaluated in the past, the report notes
that the panel's efforts represent the first time an intensive
eval-uation has been conducted by a group that has no direct interest
in the outcome.

The panel's conclusions are based only on ncrve's publications and
do not include evaluations of other activities administered by the
center, such as technical assistance, conferences, seminars, advanced
study, or leadership training.

The panel nonetheless offered six recommendations intended to assure
that the center is more productive in the next five years than it has
been in the past. In addition to continued federal support, the panel
urged that the Education Department take steps to provide the center
with "greater guidance and flexibility"; narrow the center's research
agenda; and allow the center to concentrate on a limited number of
activities.

The panel also recommended that the center improve the quality of
its staff and that it discontinue inhouse research projects and
contract most of its work through a competitive process.

Compliance. In concluding that the ncrve had not met Congressional
intent, the panel explained that the reports reviewed did not "ask
searching questions about the performance of the present
vocational-education system" and failed to concentrate sufficiently on
identifying the organizational, structural, and behavioral
characteristics of effective and equitable" programs.

One panelist called the center's approach "birdshot," according to
the report. "There have been enough hits to please its diverse
constituencies, but no solid hits on anything," the report argues. "The
system has not been sufficiently directed. The system lacks clear
research objectives."

The report adds that there was "a strong consensus" that the center
put "too little effort" into involving the private sector in its
research process.

Target audience. The report found that the majority of the center's
work has been aimed at vocational educators at the expense of groups
outside of the profession.

Only about 20 percent to 30 percent of the center's work was geared
toward members of Congress, state legislators, governors, and mayors.
About 10 percent to 20 percent addressed the interests of corporate
executives, owners of small businesses, and labor officials, according
to the report.

Usefulness. The panel members unanimously agreed that the bulk of
the center's products would only be helpful "in solving routine
operational problems." But they concluded that the center has done an
"inadequate job" on work targeted for certain audiences, such as
policymakers.

The report contends that those looking for ways "to modify
vocational education or to link it to other aspects of education and
training in order to respond to demands for social reform, to changing
technology, and to changing economic conditions are likely to find the
center's products least useful."

Knowledge. The report noted that most of the panel members concluded
that the center's work "added little to the existing knowledge" about
vocational education.Those panelists with strong research backgrounds
were "very critical"; those with nonresearch backgrounds, however, had
a more positive view of the center's contributions to the field.

Quality. There was "widespread agreement that the quality of the
ncrve's written works has varied greatly," according to the report. The
panel members were particularly dissatisfied with the center's work in
the areas of applied research, policy information, and evaluation.

The report explains that the center's research "has been
conceptually and methodologically inadequate," and that it "has lagged
far behind standards in almost all other fields of social science."

Improvement. The report acknowledges that much of the center's work
could be helpful with routine matters, but contends that they "are
unlikely to bring about significant change in the effectiveness and
equity of established programs." Most of the panel members said that
the chances of expanding vocational opportunities for girls and women,
the handicapped, economically disadvantaged youth, and adults were
either ''poor" or "fair."

"One after another, the evaluators argued that we are no closer
today to knowing what makes vocational-education programs work
effectively than we were when the Congress approved" the center seven
years ago, the report asserts.

Management. The report contends that the Education Department's
vocational office failed to establish effective compliance standards
and to support "multi-year targeting of resources" on critical problems
in vocational-education research and development.

"There needs to be a greater emphasis on quality and less on
quantity," the report asserts.

Vol. 02, Issue 27

Notice: We recently upgraded our comments. (Learn more here.) If you are logged in as a subscriber or registered user and already have a Display Name on edweek.org, you can post comments. If you do not already have a Display Name, please create one here.

Ground Rules for Posting
We encourage lively debate, but please be respectful of others. Profanity and personal attacks are prohibited. By commenting, you are agreeing to abide by our user agreement.
All comments are public.