Legislative Analyst's Office

Analysis of the 2001-02 Budget Bill

School Accountability

The Public Schools Accountability Act (PSAA) (Chapter 3x , Statutes of 1999, [SB 1x, Alpert]) created a statewide school-level accountability system which
(1) rewards schools for academic improvement, (2) provides external assistance for lower-performing schools, and (3) potentially sanctions schools which
continue to fail after receiving external assistance. The Academic Performance Index (API) is the cornerstone of the PSAA. Its purpose is to measure a school's
academic performance and the growth in performance over time.

Recently, the second annual set of API scores was released, allowing the first API growth scores to be calculated. Based on the API growth data, the State
Department of Education (SDE) will be distributing $677 million in rewards to schools, teachers, and other school personnel in the current year. Figure 1 shows
the various rewards programs tied to API results.

Figure 1

Rewards Programs Based On API a

1999-00 Through 2001-02
(Proposition 98 Appropriations, In Millions)

Program

1999-00

2000-01

2001-02
Proposed

Governor's Performance Awards

$96b

$131b

$350

Teacher Performance Awards



350



Certificated Staff Performance
Incentives

50

100

100

Totals

$146

$581

$450

aAcademic Performance Index.

b
The Public Schools Accountability Act appropriated $96 million in the
1999-00 fiscal year. Those funds were carried over and are being used for
rewards in 2000-01.

As shown in Figure 2, the budget also proposes significant resources for various programs targeted at low-performing schools, based upon the API. As Figures
1 and 2 illustrate, the state is allocating a significant amount of moneyalmost $1.7 billion in 1999-00 through 2001-02for programs using this index. The box
nearby provides some background information on how the API is calculated and how the measure is used.

Figure 2

Programs Targeted at Low-Performing
Schools, Based on the API a

2000-01 and 2001-02
(In Millions)

Program

2000-01

2001-02

Immediate Intervention/Underperforming
Schools Program

$52.1b

$139.1b

Teaching as a Priority

118.7

118.7

Governor's Teaching Fellows

3.5

17.5

National Board Certification

10.0

10.0

Cal Grant T

10.0

10.0

Assumption Program of Loans for
Education (APLE)

5.6c

14.6c

Totals

$199.9

$309.9

aAcademic Performance Index.

b
Includes both General Fund and federal funds.

c
The APLE provided 6,500 warrants in 2000-01 with the same number proposed
for 2001-02. There is no immediate cost to issuing a warrant, but the
state will incur out-year costs of up to $19,000 each. In the proposed
budget, the state would commit to out-year costs of up to $123.5 million
for 6,500 warrants.

How Does the Academic Performance Index Work?

An API Score

The State Department of Education (SDE) calculates a score ranging
from 200 to 1,000 for each school based on its students' standardized
test scores (Stanford-9). The calculation is relatively complex and
involves combining student scores across the different subject areas.
The SDE ranks the schools and provides a decile ranking (one to ten),
which is generally used for its ease of understanding. A decile ranking
of ten is best.

Subgroup Population API Score

The SDE performs a similar calculation for each "numerically significant"
subgroup of students. These subgroups include: African American, American
Indian, Asian, Filipino, Pacific Islander, white not Hispanic, Hispanic,
and socioeconomic disadvantaged. Numerically significant means either
(1) at least 30 pupils and at least 15 percent of a school's enrollment
or (2) at least 100 students in a school.

Growth Target

Annually, a school receives a growth target for the next school year
along with its API score. The State Board of Education set an interim
statewide performance target of 800 points. Schools with API scores
below 800 must close the gap between their current score and the state
performance target by at least 5 percent to meet their growth target.
For example, if a school's 1999 API score was 500, the school's growth
target would be (800 - 500) * 5 percent = 15 points. Schools with API
scores at the 800 point level or above must increase their scores by
1 point to meet their growth target.

Subgroup Growth Targets

Each numerically significant subgroup at a school has a growth target
of 80 percent of the schoolwide growth target. In the example used above,
the schoolwide growth target was 15 points. Under that example, each
numerically significant subgroup at the school must improve by at least
80 percent of 15 points - thus, by at least 12 points to meet the target.

Reward Programs

Schools are eligible for rewards based on meeting their assigned annual
growth targets for their API scores and subgroup API scores. The reward
programs are described below.

Same subgroup and participation rate requirements as Governor's
Performance Awards.

Who Receives the Funding?

School site councils decide uses.

Half decided by school site councils, other half distributed
among all school site staff.

Certificated staff only.

1,000 across state receive $25,000 each.

3,750 receive $10,000 each.

7,500 receive $5,000 each.

The API Is a Work in Progress

The state's accountability system is still a work in progress. The API is currently based on only one measure, namely the Stanford-9 test results. The Stanford-9
is an "off the shelf" student assessment produced by Harcourt Brace and administered to California students in grades
2 through 11. The PSAA requires that the API include at least five additional measures: graduation rates, student attendance rate, teacher attendance rate,
Standards-Aligned Standardized Testing and Recording (STAR), and the High School Exit Exam (HSEE). The Legislature included this broad set of outcome
measures to ensure that increases in the API were based upon real gains in achievement and not temporary fluctuations in a single measure.

The SDE has not included graduation rates, student attendance nor teacher attendance in the API because it is currently not able to collect accurate school-level
data on these outcome measures. The SDE has not included Standards-Aligned STAR and HSEE results because the assessments are not yet fully operational.
While California's current system is a long way from the full implementation, many efforts are under way to improve the quality of the information used in
calculating a school's API (see shaded box on Improving the API).

Improving the API

Since the state rewards schools for their growth in academic performance index (API) scores, the State Department of Education (SDE) must be able to collect
data on performance measures for two years before it can be included in the API measure. The timetable for possibly including additional measures in the API
is as follows:

Measure to Include in API

PotentialAvailability

Standards-Aligned STAR

2002

High School Exit Exam

2003

Graduation rate

Unknown

Student attendance rate

Unknown

Teacher attendance rate

Unknown

Standards-Aligned Standardized Testing and Recording (STAR). As part of the spring 2001 STAR test, the SDE will administer four subject tests (math,
language arts, science, and social science) aligned with California's content standards. Additionally, SDE will provide performance standards for the first time
in spring 2001. A performance standard translates the results of a student's test into a scale ranging from mastery of the subject area to below basic. Assuming
there are no problems with the validity of the assessments or performance standards, results from the Standards-Aligned STAR could be included in the API as
early as 2002.

High School Exit Exam (HSEE). Based upon the current State Board of Education plan, all 10th grade students will take the HSEE in spring 2002.
Potentially, SDE could include the results from the spring 2002 and spring 2003 administrations in the 2003 API.

Graduation Rate, Student and Teacher Attendance Rate. The Public Schools Accountability Act required the Superintendent of Public Instruction (SPI) to
report by September 1999 on the accuracy of the measures to be included in the API, and to recommend necessary actions to implement an accurate reporting
system for these measures. The SPI's report recommended use of the California School Information Services to collect data on graduation rates and student
and teacher attendance rates. The administration rejected the SPI proposal, and $500,000 was included in the 2000-01 Budget Act for the Office of the
Secretary for Education (OSE) to conduct a study of data collection alternatives for expanding the number of measures in the API. According to OSE, it will
release a Request for Proposal some time this spring for this study. The findings of the study will determine the timetable for including these measures in the
API.

In addition to the measures described above, the Governor's budget provides $5 million to create a longitudinal student assessment database. The main policy
goal of this proposal is to enable the state to begin to construct "value-added" measures of student performance, the advantages of which we discuss at length
in the "K-12 Databases" section of this chapter.

Problems With the Stanford-9

The Stanford-9 provides California schools with a score reflecting how students in California perform relative to a national norm created from the results of
administering the same test to a nationally "representative" group of students. We see three major problems with the Stanford-9:

The Stanford-9 is not aligned to California's academic content standards.

Experts question whether Stanford-9 test scores for many of the state's English Language Learners (ELLs) are meaningful.

The test questions do not change from one year to the next.

We discuss each of these problems in more detail below.

Stanford-9 Not Aligned to Standards. The current
version of the API is solely dependent on the Stanford-9, an assessment tool
that is not aligned to the state's academic content standards. The California
Academic Content Standards adopted by the State Board of Education (SBE) define
what the state wants students to learn in each grade and subject (see shaded
box for actions related to the academic content standards). Since the Stanford-9
tests a different set of information than the content standards, the state is
sending conflicting messages about what schools should be teaching. Should teachers
and administrators focus instruction on areas covered by the Stanford-9? The
state is offering bonuses that can exceed $25,000 per individual for success
measured by that test. Or should they teach what is expected under the academic
content standards for which the state has invested $1 billion for new textbooks,
the Governor is proposing $335 million for staff development, and upon which
the HSEE is based?

State Academic Content Standards

The California Assessment of Academic Achievement Act (Chapter 975, Statutes
of 1995 [AB 265, Alpert]), required the State Board of Education (SBE)
to adopt academically rigorous content standards in the five core curriculum
areasmath, English-language arts, science, social science, and English
language development. The standards provide a framework for what a student
should learn in five core subject areas in each grade.

Since adoption of the standards, the state has taken several actions
to emphasize the importance of the standards to schools and teachers.

Instructional Materials. The Schiff-Bustamante Standards-Based
Instructional Materials Program (Chapter 312, Statutes of 1998,
AB 2041) is provid-ing school districts with $1 billion over four
years to purchase instructional materials aligned with the academic
content standards.

The Standardized Testing and Recording (STAR) Augmentation. In
1998-99, the SBE added an augmentation to the STAR test in math
and English-language arts, consisting of questions specifically
aligned to state content standards. In the current year, the board
will develop student performance standards around the STAR augmentation
to determine which students have met basic, proficient, and advanced
levels of performance.

High School Exit Exam (HSEE). Chapter 1x, Statutes of 1999 (SB
2x, O'Connell), authorized the HSEE. This legislation requires the
test to be aligned to the state academic standards.

Teacher Staff Development. The Governorís budget proposes $490
million including a $335 million expansionfor teacher staff development
that trains teachers in using state standards.

Other Actions to Emphasize Standards. The state also has aligned
the Golden State Exams, and developed sample curriculum for each
subject based on the standards.

How Meaningful Are Stanford-9 Scores for ELLs?
The second problem with the Stanford-9 is the validity of Stanford-9 scores
for many ELL students. The advisory committee of experts appointed by the SBE
for development of the API advised the board to exclude Stanford-9 test scores
for ELL students from the API calculation. Their recommendation was based upon
analysis of Stanford-9 test results for ELL students. A school may be doing
an excellent job of helping a student transition to English fluency, but that
student's Stanford-9 score may not reflect the improvements the student has
made. The SBE did not follow the recommendation. Although not implemented by
the SBE, the advisory committee's recommendation puts into question the comparability
of API scores between schools with many ELL students and schools with few or
no ELL students.

Test Questions Don't Change From Year to Year. The final problem with the Stanford-9 is that the set of actual questions on the test does not change from
year to year. Particularly with a high stakes test, it is important to vary test questions from year to year in order to minimize possibilities for literal "teaching to
the test" and outright cheating. The SDE did not stipulate the rotation of questions from year to year in its contract with the publisher of the Stanford-9. So for
the five-year duration of the contractwhich expires after the administration of the spring 2002 testthe test given in each grade is exactly the same in each of
its administrations. Teachers know the exact questions that their students will be asked and upon which their school will be evaluated. Providing $677 million
in rewards based solely upon this test creates incentives for schools to misuse their knowledge of Stanford-9 test questions.

Reduce Funding for Governor's Performance Awards

We recommend the Legislature reduce the funding proposed for the Governor's Performance Awards by $219 million until the Academic Performance
Index is based upon a broader set of reliable performance indicators. (Reduce Item 6110-123-0001 by $219 million.)

Funding Too High for Untested Measure. As discussed above, the API is a work in progress and will take several years to develop fully. The administration is
making progress on increasing the number of outcome measures used to calculate the API, and is planning to switch to a value-added approach, which will
improve the use of the existing achievement measure. Until then, however, there are numerous problems with the Stanford-9 being used as the sole measure for
the API as discussed above.

Consequently, we believe it is not only premature to attach such high stakes (and such large funding amounts) to the API but it may be counterproductive to
improving learning outcomes for California's public school students by redirecting the focus of teaching from the state's broad content standards to the set of
material tested on the Stanford-9. Once the Governor and SDE have been able to develop the data collection systems and assessment tools needed to support a
quality accountability system, the Legislature then should determine the appropriate level of funding needed for school incentives. Accordingly, we recommend
reducing funding for the Governor's Performance Awards to $131 million, the same level appropriated in the 2000-01 Budget Act. This recommendation would
create an ongoing $219 million of Proposition 98 savings which the Legislature could direct more effectively to other K-12 education needs.
(Reduce Item 6110-123-0001 by $219 million.)

Inadequate Funding to Evaluate PSAA

We recommend the Legislature (1) increase funding to the State Department of Education (SDE) to contract for an evaluation of the Public Schools
Accountability Act from $250,000 to $500,000, and (2) reappropriate $250,000 included in Item 6110-001-0001 of the 2000-01 Budget Act for the same
purpose. We further recommend SDE report at budget hearings on the possibility of using federal funds for this evaluation.

The PSAA requires SDE to contract for an external evaluation of the PSAAincluding the Governor's Performance Awards program, Immediate
Intervention/Underperforming Schools Program (II/USP), and the impact of these programs on student achievement. The 2000-01 Budget Act provided SDE
with $250,000 for the evaluation. In the summer of 2000, the SDE released a Request for Proposal for the evaluation but did not receive a single qualified bid.
As a result, the state has not begun to evaluate this major program. The Governor proposes an additional $250,000 from the General Fund (non-Proposition 98)
for the PSAA evaluation this year.

We believe that the $500,000 that would be appropriated between the two budgets is still inadequate given the complexity of issues that the evaluation must
address. For example, under this evaluation, the SDE is required to contract for a comprehensive evaluation of the implementation, impact, costs, and benefits
of II/USP and Governor's Performance Awards program. A preliminary report is due March 31, 2002. The state may distribute as much as $1.7 billion related to
these programs in 1999-00 through 2001-02. Potentially more important, as early as fall 2002, schools and employees of schools participating in the II/USP
could face sanctions from the state under the terms of the PSAA. Sanctions could include the Superintendent of Public Instruction reassigning principals,
reorganizing schools, reassigning certificated staff, contracting out the management of schools or even closing schools. The state needs to make sure that the
measure upon which such high-stakes decisions are made truly tracks changes in student achievement. Given the importance of this program and the extensive
list of legislative requirements for the evaluation, we recommend that the Legislature (1) increase the appropriation for the evaluation from $250,000 to
$500,000 in the budget year, and (2) reappropriate $250,000 included in the 2000-01 Budget Act for this purpose. (This would make $750,000 available to the
study.)

As discussed below, the II/USP is partially funded with federal funds. The federal grant which provides funding for II/USP allows the state to use up to
5 percent of the grant amount to pay the costs of administering the program. We believe that evaluation of the program is an essential part of the program's
administration. To the extent feasible, we believe it is preferable to use federal funds instead of the General Fund to pay for at least part of the PSAA
evaluation. Federal funds could not be used for all of the evaluation costs because the PSAA evaluation must evaluate programs beyond II/USP. We
recommend SDE report at budget hearings on the feasibility of using federal funds to pay for at least part of the PSAA evaluation.

Certificated Staff Performance Incentives Overfunded

We recommend that the Legislature reduce funding for the Certificated Staff Performance Incentive Awards by $50 million on a one-time basis to take
advantage of carryover funding available for this program. (Reduce Item 6110-133-0001 by $50 million.)

Figure 3 shows the appropriations and expenditures that have been made or are proposed for this program. The 1999-00 Budget Act provided
$50 million for the program. No funds were distributed in 1999-00, however, because only one year of API scores was available prior to summer 2000,
therefore making it impossible to determine API growth targets. The 2000-01 Budget Act and the 2001-02 Governor's Budget each include $100 million for this
program. The administration's spending plan for 2000-01 is to use the $50 million from the 1999-00 Budget Act and $50 million from the 2000-01 Budget Act.
As a result, $50 million from the 2000-01 Budget Act will still be available for rewards in the budget year. We recommend the Legislature use this $50 million
in carryover funds for the certificated staff incentives and reduce Item 6110-133-0001 by $50 million. This action would provide the total of $100 million for
budget-year expenditures proposed for this program, yet also free up $50 million in Proposition 98 funds for other K-14 education priorities of a one-time
nature.

Figure 3

Certificated Staff Incentive
Program Funding

1999-00 Through 2001-02
(In Millions)

Appropriation in Budget
Act

Proposed
Expenditure Level

1999-00

$50



2000-01

100

$100

2001-02a

100

100

Totals

$250

$200

a Amount proposed in
Governor's budget.

Immediate Intervention/Underperforming Schools Program

We recommend the Legislature enact legislation to broaden the eligibility criteria for the Immediate Intervention/Underperforming School Program to
include all schools in the lowest two deciles of the Academic Performance Index.

The II/USP provides state and federal support for school-wide reform efforts at low-performing schools. The 2001-02 Governor's Budget proposes
$113.8 million from the General Fund (Proposition 98) for the program. The budget would provide planning grants for a third cohort of 430 schools, and
implementation grants for the first and second cohorts (782 schools in total). Figure 4 show the schools and number of students in each cohort for 2001-02. The
Governor's budget continues to fund 78 schools with federal funds, which we discuss below.

Figure 4

Participation in II/USP a in 2001-02

Cohort 1 (participation started
1999-00)

Grant:
Implementation.

Number
of Schools: 352b

Number
of Students: 285,000c

Cohort 2 (participation started
2000-01)

Grant:
Implementation

Number
of Schools: 430d

Number
of Students: 430,500

Cohort 3 (participation starts 2001-02)

Grant:
Planning

Number
of Schools: 430

Number
of Students: Unknown

a Immediate Intervention/Underperforming
Schools Program.

b An additional 78 schools
are funded with federal funds.

c An additional 89,000
students are funded with federal funds.

d Some schools in Cohort 2
may apply to be funded with federal funds.

In their first year in the program, participating schools are provided $50,000 planning grants to develop a comprehensive school reform plan. As part of the
planning phase, the schools must hire qualified external evaluators to assist in developing the reform plans. Once a school's plan is approved by the SBE, the
school receives annual implementation grants of up to $200 per enrolled student. Schools will receive the implementation grants for two years, and may be
granted a third year of funding by the SBE if they continue to struggle to meet their API growth targets. The SBE can decide to impose sanctions after either the
second or third year of funding for schools that continue to struggle. The first cohort will reach the end of their two-year implementation grant in July 2002 and
potentially could face sanctions as early as fall of 2002.

Program Eligibility Has Changed. Any school whose average student score was below the 50th percentile on the Stanford-9 was eligible to participate in the
first II/USP cohort in 1999-00. Under the original criteria, over half of the schools in the state were eligible to apply for the program, and 1,419 schools applied
for the 430 slots available. Chapter 695, Statutes of 2000 (SB 1552, Alpert), amended the PSAA to change the eligibility criteria to be based upon the API. As
revised by Chapter 695, schools in the lower half of the API which did not meet their API growth targets were eligible. Based upon 1999-00 API scores, 938
schools were eligible in 2000. Of the 938 schools, 532 applied for the 430 slots available.

Legislature Should Broaden II/USP Eligibility to Include Lowest-Achieving Schools. The policy change of Chapter 695 to focus on schools not making their
growth targets makes sense to ensure that schools that are struggling to improve receive additional resources. However, only focusing on schools not achieving
their growth targets may be too narrow. For example, under the eligibility criteria in Chapter 695, the school with the lowest API score in the entire state is not
eligible for this immediate assistance program since it reached its annual growth target. Even though such a low-performing school is showing signs of
improvement, we believe that its improvement could potentially be expedited by participation in this program. Such a low-performing school is likely to be in
greater need of external assistance than a school in the fifth decile (close to the state average) which did not meet its annual API growth target. Accordingly, we
recommend the Legislature expand the eligibility for the II/USP program to all schools in the lowest two deciles of API ranking.

Extra Federal Funds Available for II/USP

We recommend the State Department of Education report at budget hearings on the extent that General Fund costs for the Immediate
Intervention/Underperforming Schools Program can be reduced by the use of federal funds.

California has received $92 million in federal funding since 1998-99 which can only be used for II/USP implementation grants (see Figure 5). As mentioned
above, 78 schools have received federal funds for II/USP. To date, SDE has provided $32 million in federal funds to those schools. An additional $16.3 million
is proposed for those schools in the budget year. After funding these 78 schools, SDE will have an additional $44 million remaining to provide implementation
grants to new schools.

Figure 5

Federal Funding for II/USP a

(In Millions)

Federal
Appropriation

Spending

1998-99

$16.2



1999-00

16.1

$15.2

2000-01

26.5

16.3

2001-02

32.9

16.3

Totals

$91.9

$47.9

aImmediate Intervention/Underperforming
Schools Program.

The 430 schools in the second cohort will have the option of participating in II/USP using either General Fund monies or the available federal funding. Schools
in the second cohort have until May 15, 2001 to apply for federal funds for their II/USP implementation grants. Federal funding has the following trade-offs:

Guaranteed Funding Level. Federally funded schools receive a guaranteed $200 per student for three years, while state-funded schools have no per student
guarantee and only two guaranteed years of funding.

More Rigorous Requirements. Federally funded schools apply on a competitive basis, and must complete a more rigorous planning, reporting, and
evaluation process.

The SDE has provided a rough estimate of the number of schools that it expects to receive federal funds. We estimate that $10 million in federal funds will be
used to fund schools selected by SDE to receive federal funds, leaving approximately $34 million in federal funds available for future II/USP schools. Such low
usage of federal funds for II/USP would result in greater demand for state funds for II/USP, which we discuss below. We recommend the SDE report at budget
hearings on steps the Legislature can take to increase the usage of available federal funds to pay the costs of II/USP.

General Fund Appropriation for II/USP May Be Low

The amount provided in the Governor's budget for the Immediate Intervention/Underperforming Schools Program is inadequate to maintain the
current-year per-pupil grant level.

In 2000-01, schools in the first cohort received $168 per-pupil implementation grants. The Department of Finance informs us that the Governor did not have a
target per-pupil implementation grant for II/USP in developing the 2001-02 budget. The budget provides $92 million for implementation grants. Based upon
our estimate of federal fund usage for II/USP, we estimate that the per-pupil funding level provided in the Governor's budget would result in implementation
grants of $146 per pupil.

Figure 6 shows the options that the Legislature has (based upon our cost estimates) if it wants to (1) maintain the Governor's proposed level, (2) maintain last
year's per-pupil grant amount of $168 per pupil, or (3) provide schools the maximum statutory grant amount of $200 per pupil. If the Legislature wants to either
maintain the current-year's per-pupil funding level or increase the per-pupil funding level to the maximum, it would require additional General Fund support of
$15 million or $36 million, respectively.