Education
Assessment and Accountability Review Subcommittee

Minutes
of the<MeetNo1>5th Meeting

of the 2006 Interim

<MeetMDY1>December 5, 2006

The<MeetNo2>fifth meeting of the Education Assessment
and Accountability Review Subcommittee was held on<Day>Tuesday,<MeetMDY2>December
5, 2006, at<MeetTime>1:00 PM, in<Room>Room 169 of the Capitol Annex. Representative
Harry Moberly Jr., Co-Chair, called the meeting to order, and the secretary
called the roll.

Representative Moberly asked for a motion for the approval
of the minutes from the meeting of October 6, 2006. Senator Winters made the
motion to approve the minutes, and Representative Rasche seconded the motion.
The motion was approved by voice vote.

Representative Moberly introduced Dr. James Catterall,
Chair, and Dr. David Miller, member, National Technical Advisory Panel for
Assessment and Accountability (NTAPAA), to review the current status of
Kentucky's testing and assessment system. Representative Moberly said that
NTAPAA was created in 1998 by statute to provide the legislature with technical
expertise on assessment and accountability issues. He said the Education
Assessment and Accountability Review Subcommittee (EAARS) would like to work
closer with the NTAPPA in the future. He also said he and Senator Westwood
requested that NTAPAA provide the EAARS with a paper outlining the most current
issues and projected changes in assessment, including a review of the steps
that need to be taken to implement Senate Bill 130 and the inclusion of the ACT
in the accountability system. He said a discussion of the issue of longitudinal
accountability for students was also requested. The paper is in the meeting
folder.

Dr. Catterall said every six or seven years, large-scale
assessment systems need an overhaul. He said a packet has been provided to members
that show the inclusion of the ACT, PLAN, and WorkKeys in the assessment system.
He said Senate Bill 130 signifies some long standing and pointed concerns about
assessment at the high school level. He said issues such as readiness of the
students for college, acceptance into college, and performance during college
are of primary concern.

Dr. Catterall said another concern is to have a way to
identify students in the eighth and tenth grades who may need extra help and
provide an intervention, as well as encourage students who are breezing through
high school to take a program of more challenging standards.

Dr. Catterall said there are long standing issues regarding
the motivation of high school students in Kentucky for taking the Kentucky Core
Content Tests (KCCT). He said implementing the ACT at the high school level
attempts to alleviate the motivational concerns for students.

Dr. Catterall said the National Technical Panel for Accountability
and Assessment (NTAPAA) has seen no reason that the ACT cannot be used in the
various ways that it has been proposed in Senate Bill 130. It is possible to
augment the ACT with KCCT type items, and have it behave as the KCCT test for
eleventh graders. He said the only problem remaining is that NTAPAA does not
know how long it will take for the ACT test to work in the intended manner.

Dr. Catterall explained the basic design and the scoring
system of the ACT test. He said the alignment of the ACT test with the core
content in Kentucky is an outstanding issue, but NTAPAA supports that there is
some alignment between the ACT and high school standards in Kentucky, but does
not know how much and of what kind. He said it is greater than a 30 or 35 percent
range, but less than 70 percent.

Dr. Catterall said it will be interesting to see how many
more students end up attending college with an "ACT for all" policy.
He said some students will realize they are ready to attend college when they
may have previously assumed they were not college ready.

Senator Kelly said another point of the "ACT for
all" policy is for students to learn at a very early age that they are not
college material when they may believe they are. He said at that time,
counseling and intervention strategies would be implemented to get the student
enrolled into a more rigorous curriculum in order to overcome the deficiencies.

Dr. Catterall explained the basics of the Commonwealth
Accountability Testing System (CATS). He discussed standards settings and said
formal standards are set for each component test in CATS. He said CATS could
also still be used in the high school to meet the requirement for the No Child
Left Behind (NCLB) program, but alignment is again an issue. He noted that the
augmented items in CATS would have to be open response questions as all the
questions on the ACT are multiple-choice questions.

Representative Moberly asked if an alignment is performed
before the development of the augmented questions, or is the entire process
part of the alignment. Dr. Catterall said the process itself is part of the
alignment. He said the big vision would include administering the ACT once
without consequences within the accountability system, but disseminating the
scores to all students, and then administering the ACT within the
accountability system the following year.

Dr. Miller said the alignment issue becomes an on-going
concern when forms are being created by outside vendors, rather than the
current KCCT test that is used and created in Kentucky. Dr. Catterall said some
of the process would be out of Kentucky's control because if the ACT makes
changes to its test, then Kentucky would have to make similar changes if it was
to continue to use it in order to keep the integrity of the test.

Representative Draud asked if alignment is realistic or not.
Dr. Miller said studies can be conducted to determine how much is already
aligned, but it almost certainly is going to have to include an augmented test
that is given in conjunction with the ACT. Representative Draud expressed
concern about giving another test in addition to the ACT, and Dr. Miller said
the ACT would at least have to incorporate some augmented items in order to
match the Kentucky core content. Dr. Catterall said NTAPAA supports the notion
that the alignment can be done if the appropriate effort is devoted to the
task.

Representative Moberly asked if NTAPAA's position was the
same as the Kentucky Department of Education's (KDE) that the ACT could be used
as a substitute for the current norm-referenced test (NRT) in the 2007-2008
school year, and the augmented test be used in place of the core contest
Kentucky that is currently administered the following year. Dr. Catterall said
that was correct.

Senator Kelly said he is very concerned about the delay in
incorporating the ACT in the accountability system primarily because of adding
layers of testing. He said there is significant resistance from some people,
but he is a strong advocate of just implementing the ACT into the accountability
system right away. He asked Dr. Catterall what the real concerns or negatives
in student performance would be in implementing this in the 2007-2008 school
year.

Dr. Catterall said the only thing that would harm student
achievement would be testing students too much. Senator Kelly clarified the
question by asking if the potential problems with the integrity of the test
were potential threats to the achievement of the students. Dr. Catterall said
it was not a direct threat to student achievement, however, results can be
obtained from tests that are not necessarily matched to the items that the
teachers taught during the school year, which can cause a problem within the
system. On the technical side, a redesigned test, such as an augmented ACT, would
have to receive approval from Washington D.C. to ensure standard alignment protocols.

Dr. Miller said it would not harm the student individually
to implement the augmented ACT in 2007-2008, but it could hurt the NCLB
requirements. Secondly, the validity of the information may be absent that
year.

Senator Kelly said those are technical problems that will
have to be dealt with whether it is implemented in 2007-2008 or 2008-2009. He
said it will be a struggle to align the test to meet NCLB requirements, but it
is a struggle the KDE faces now under the current system. He thinks the
benefits of incorporating the ACT outweigh the technical problems.

Dr. Miller said there are some advantages for students to
take the ACT right away, but requirements for NCLB and ensuring a fair
accountability system for the school that year, are both technical issues that
have to be addressed prior to implementation. He said the ACT could be
implemented next year, but he would not drop the KCCT so quickly until the
outstanding issues are resolved.

Senator Kelly said he wants the ACT incorporated as fully as
possible next year to eliminate duplication in testing. He said Kentucky can
work through the hurdles as they arise, as you would with any revamping of an
accountability system.

Representative Moberly disagreed. He said the integrity of
the accountability system depends upon the alignment. The ACT will still be a
part of the accountability system by replacing the NRT, but the core content
cannot be replaced because it ruins the integrity of the accountability system.
He said the bill states that the augmented ACT will not replace the core
content before alignment has occurred, which will take at least a year. Senator
Kelly says he is ready for the change to occur now.

Representative Rasche asked if the discussion was about
implementation of the augmented ACT in the 2007-2008 school year and the panel
said yes. He asked if it would be possible to administer a smaller version of
the ACT in 2007, to a smaller number of people on the same basis as it will be
given universally in 2008, for the purpose of beginning the alignment process.
Dr. Catterall said yes. Representative Rasche asked if this was a practical
approach. Dr. Catterall said a group of 10,000 would be a good size to sample
in 2007.

Senator Winters asked if a year and half was ample time to
make these testing adjustments. Dr. Catterall said live operational data is
missing from students because there is no previous data to see how students
perform on the augmented ACT. He said a large scale experiment for one year
would provide ample data before implementing the change statewide.

Senator Kelly said the experimental process had been started
by administering the EXPLORER test to tenth graders, and asked if most students
perform similarly on the ACT as they do on the EXPLORER. Dr. Catterall said
that is the belief. Senator Kelly said the data is available now and there is
no need to do a large scale experiment and wait another year.

Dr. Miller said there are many things that are just still
unknown, such as how large the augmentation will have to be to the ACT. He said
if many items already match, it becomes much more doable on a quicker schedule.

Senator Kelly said it is known that there is at least 35
percent alignment already, and possibly as much as 70 percent. He wants to move
forward as quickly as possible to reduce the test taking time and duplication,
while continuing to study and work towards the goal of alignment.

Dr. Catterall said there are cross purposes surfacing. If
Kentucky moves forward in 2007-2008 by administering the augmented ACT, it
threatens the integrity of the accountability system. He said if the rest of
the NTAPAA panel were present at the meeting, the majority would feel it would
be best to test it a year, and then go live the next year.

Dr. Catterall said House Bill 58 in 1998 said a longitudinal
component should be added within the assessment system if it could be rendered
valid and reliable. He said the CATS test did not test mathematics and reading
every year in the elementary grades so it was very difficult to build a solid
longitudinal component.

Dr. Catterall responded to the proposal of adding a
longitudinal assessment when two years of data is available. NTAPAA has had
multiple discussion of possible models and in-use models from other states and
is gravitating toward a preferred approach to longitudinal scaling and
reporting. NTAPAA has suggested constructing longitudinal scales from grade
three to grade eight in reading and math. He said using "vertically
moderated" standards to assign performance levels at all grade levels
attends to the distributions of performance levels within grade levels; thus
students are not simply placed on a continuous linear scale. Vendors, KDE, and
NTAPAA are in close communication on this element of the new system. He said
NTAPAA is reasonably confident that longitudinal assessment will be supported
and implemented in the new testing system design.

Senator Kelly said he thought the longitudinal issue would
be addressed by administering the PLAN in the eighth grade, the EXPLORER in
tenth grade, and the ACT in the eleventh grade. He thought the tests were
predictive of each other and could be tracked to see how students are doing
over time. He asked if that was an erroneous assumption on his part or was it
part of one of the moderated vertical scales.

Dr. Miller said it is not something that NTAPAA had
specifically talked about. If it were to be done, it would have to in conjunction
with the ACT, and he does not know if ACT would allow that. He said it would
have to be discussed with the ACT group.

Senator Kelly asked if the longitudinal component was
legislatively mandated. Representative Moberly said it was discussed, and
Senator Kelly said there is no reason Kentucky cannot move ahead in that
direction if it waits for one more year to get data. Dr. Miller said it is two
different situations. Senator Kelly said he understands that the longitudinal
technical challenges are huge, but he feels the alignment issues are small from
a technical standpoint.

Dr. Miller said it took so long to create a vertical scale
because NTAPAA did not know that it could technically be done when students
were tested in subject areas every third or fourth year. He said consecutive
years of testing did not occur until this year.

Representative Draud asked if Kentucky's core content is
in-line with the ACT. Dr. Catterall said it is not known because the alignment study
has not been conducted. Representative Draud said that could create a lot of
anxiety among the teachers throughout the Commonwealth if Kentucky utilizes an
accountability index that is not known to align with the core content. Dr.
Catterall said NTAPAA is working to see if the accountability elements are aligned
with the core content. Representative Draud asked when the information from the
alignment of the core content to the ACT would be known. Dr. Catterall said it
would certainly be known after an administration of the test. The administration
of the test provides student level data of all items on the test for comparison
purposes. Representative Draud asked if the KCCT and the ACT test had ever been
compared.

Senator Winters said he has samples from various people that
have compared the two tests with one comparison showing an 80 percent
alignment. He said this does not include student performance, but just
comparisons of the two tests. He feels that NTAPPA should look at the two tests
for comparisons and give their advice with their specific testing expertise.

Senator Kelly responded to Representative Draud that data
can be used from the EXPLORER test administered to tenth graders and by other
methods of comparison. He said the current test is not 100 percent aligned. Dr.
Miller said a sample of items are used from the core content so there is never
100 percent alignment. Dr. Catterall said every item on the test is aligned to
the core content for assessment, but the entire content for assessment is not
represented on the test.

Senator Kelly reiterated that it is better to start
implementing the test now and continue to work on the alignment or it could
drag out another 18 months. Dr. Miller said depending upon how much alignment
is there, he is concerned about creating the test in 12 months time. He also
said the supplemental materials will need to be field tested.

Senator Kelly said if the alignment is pretty good then it
will take much less time to prepare. Dr. Miller agreed. Senator Kelly asked if
the study of alignment was being delayed. Dr. Miller was not sure, and Senator
Kelly said it was a question for the KDE. Representative Moberly asked Ms. Pam
Rogers, Associate Commissioner of Education and Accountability, KDE, to come to
the table.

Ms. Rogers said the KDE has submitted a Request for Proposal
(RFP) to have an outside vendor look at the alignment of the ACT with the core
content, and a vendor will be selected this month. Ms. Rogers said the
responses to the RFP had been received and interviews would begin in a week
with a member of NTAPAA participating as well.

Representative Rasche asked to what degree the ACT is a
criterion-based test versus a norm-based test. Dr. Miller said all tests are
never completely one or the other, and Representative Rasche said to what
degree. Dr. Miller said the ACT is a norm-referenced test for the most part and
that is why supplemental materials will be so important to cover the
criterion-based needs.

Representative Rasche said he feels the ACT may discriminate
toward the traditional college-bound students and the questions could be
oriented to the expertise in that area. He asked how the test would distinguish
between a high and medium novice, and some questions need to be geared toward
the lower end students because Kentucky needs that assessment information.

Dr. Miller said he thinks the ACT covers a wide range of items,
but it will be an issue that is looked at as the tests are studied.
Representative Rasche said it was an intuition totally unfounded, but sometimes
it is the level within the core content that needs to be assessed.

Representative Rasche asked about using a smaller sampling
size for a pilot such as 3,000 or 5,000 students. Dr. Miller said he is not
sure about the exact number, but the bigger issue is if there is an adequate
sampling to get a broad range of Kentucky students.

Representative Moberly recognized the state board members in
the audience. He communicated to them that they cannot get a clear sense of the
implementation of Senate Bill 130 because the subcommittee is divided on the
issue. He agrees with NTAPPA and the KDE on the implementation of the core
content in the ACT, and said moving ahead aggressively is reckless and goes
against the integrity of the accountability system.

Senator Kelly said if KDE had expressed an opinion to
Representative Moberly he wished they had communicated it to him. He has not
seen a response or communication from the KDE on their position of the
implementation of Senate Bill 130. He said it was communicated in the meeting
that the implementation could occur right away, although it may be preferred to
have more data to preserve the integrity of the test and address other
technical issues. He said advantages of implementing the ACT in 2007-2008
include encouraging student motivation and avoiding duplication of testing,
while decreasing the time spent testing. He hopes it does not take eight years
to get implemented, and Representative Moberly concurred.

Representative Moberly introduced Ms. Lu S. Young,
Superintendent, Jessamine County Public School System and a member of the
School Curriculum, Assessment, and Accountability Council (SCAAC) to discuss
Senate Bill 130. She said she is supportive of all the juniors and seniors
taking the ACT and believes that students will perform better on the ACT
resulting in a higher college going rate in Kentucky.

Ms. Young said schools should be informed in a prudent way
about changes in the accountability index to give them time to respond to new
demands. She said it takes time for schools and teachers to adjust to major
changes in the accountability system, and what seems like small technical
changes can have large impacts when they are actually implemented in the
schools. She said schools are well on the way to reaching proficiency by 2014,
and changes in the accountability system could throw some schools off course.

Ms. Young said once it is determined how much the augmented
ACT will weigh in the accountability system, schools will have to allocate
resources, such as time and money, to prepare their students to do well on the ACT.
She also said that 2008 is the second year of the Kentucky CATS biennium, and
this is a vulnerable biennium as all these changes are occurring. She urged the
committee to follow the bill language and administer the ACT to all students in
2008, and use this live data in the second year of the biennium to make some
significant and important decisions.

Senator Kelly said he appreciated her optimism about the
forthcoming changes. He understands why people in the field are also
apprehensive about making these changes. He said there is a concern on the part
of the legislators that over half of the Kentucky schools are not meeting their
performance goals and are not on their way to proficiency by 2014. He also said
the ACT test does not test unknown information, but helps to prepare students
for college, which is one of the ultimate goals for students.

Representative Draud asked Ms. Young her opinion on students
having too much testing. Ms. Young said she agreed with Representative Moberly
that Kentucky can go ahead and include the ACT and PLAN as the NRT without
making any dramatic changes. She also said 65 percent of public school students
in Kentucky take the ACT now, so for the additional 35 percent of the students,
the ACT will be an additional testing burden. She said the augmentation will
work, but she believes open-response questions is a fundamental component of
CATS that truly makes a difference.

Representative Draud asked if Ms. Young had any suggestions
on tests that can be eliminated or consolidated. Ms. Young said the new KCCT
package has done a good job of shortening areas not tested by open-response
such as practical living, vocational studies, and arts and humanities. She does
believe the augmentation of the ACT can result in a shortened level of testing,
but she is not supportive of this in the 2007-2008 biennium.

Representative Rasche asked at what point the proposal
became ACT "augmented" instead of ACT "supplemented". He
believes that the KCCT should be the core and the ACT should be used to off-set
some parts of it. Ms. Young said she appreciates that fundamental notion of
KCCT supplemented as opposed to augmented. She cautioned not to produce a final
product with just an open-response augmentation or supplementation because
students who do not test well tend to do better starting out with
multiple-choice questions.

Representative Moberly introduced Ms. Marcia Seiler,
Director, Dr. JoAnn Ewalt, Research Division Manager, and Mr. W. Bryan Jones,
Investigations Division Manager, Office of Education Accountability (OEA) to
give their 2006 annual report, 2006 school finance report, and the proposed
research agenda. Ms. Seiler said the General Assembly in 2006 modified KRS
7.410 that outlined the duties of OEA resulting in important changes. One is
that OEA prepares a research agenda to be approved by the EAARS subcommittee.
She said five research items have been developed for 2007 as OEA sought input
from members of EAARS in developing the research topics.

Dr. Ewalt gave a Power Point presentation on the 2006 school
finance report. She also told members that a complete 2006 school finance
report was in their meeting folders.

Dr. Ewalt said OEA presented the 2005 school finance report
in June, 2006. It contained unaudited, preliminary data for fiscal year 2005,
and concluded that the equity gap has been improving in the past three years,
and is narrower in all of the years since the implementation of the Kentucky
Education Reform Act (KERA). From 1990 to 2005, the equity gap was reduced by 4
percent. She also said from 1990 to 2005, the inflation-adjusted equity gap was
reduced by 36 percent.

Dr. Ewalt said the 2006 school finance report shows the
equity gap is wider than initially reported. It was based on updated data for
local, state, and federal school district revenue and contains audited, final
data for fiscal year 2005. It also contains minor revisions for fiscal year
2003 and 2004.

Dr. Ewalt said wealth Quintiles are established by ranking
districts from lowest to highest, divided into five groups, each with one-fifth
of the state's students. She said Quintile 1 represents the lowest property
wealth districts and Quintile 5 represents the highest property wealth districts.
She explained the calculations for defining and analyzing the equity gap. A
detailed handout is in the meeting folder located in the Legislative Research
Commission.

Senator Kelly asked if the gap was referring to the gap
between the total amount of funds received state and locally per pupil. Dr.
Ewalt said yes, the Quintiles were established using wealthier and less
wealthier districts in order to establish if they are improving over time.

Senator Kelly asked if the receipts included construction
funds as well as operational expenses. Dr. Ewalt said the School Facility
Construction Commission (SFCC) funds were not included in determining the
equity gap, only operational expenses.

Dr. Ewalt said when comparing fiscal year 2005 audited to
unaudited per-pupil revenue amounts, the districts in Quintile 5 (highest
wealth) experienced greater increases in local and state revenues than the
districts in the other quintiles. When comparing fiscal year 2005 audited
per-pupil revenue amounts to fiscal year 2004 levels, the districts in Quintile
5 experienced greater increases in local revenues than districts in other quintiles.
Quintiles 1-4 had more state revenue, but not enough to offset the gains Quintile
5 made in local revenue.

Representative Draud said the equity issue in Kentucky is
good compared to the nation. Dr. Ewalt said Kentucky was in the forefront
addressing equity issues, but was not sure if it is now. She will have to
research to see how Kentucky compares to other states regarding equity issues
currently.

Representative Draud asked if the revenue behind each pupil
included the general fund and the building fund. Dr. Ewalt said it does not
include the building fund, only the general fund.

Senator Kelly said he knows the figures were adjusted for
inflation, but what happens if it were adjusted for cost of living. Dr. Ewalt
said it would be possible to do a regional adjustment as well, but it has not
been done thus far because there is no commonly accepted regional adjuster for
Kentucky. Senator Kelly said he would like to see some figures of this nature,
and Dr. Ewalt said she would get some figures for him. Ms. Seiler said the
finance report for next year will include some different scenarios.

Senator Kelly said the districts that are growing, primarily
located in the higher Quintiles, have money to hire teachers, but are
struggling to keep up with the cost of building new facilities. Dr. Ewalt said
growth districts were looked at in the SFCC report last year, but in general,
the growth nickels and other things available to rapidly growing districts,
have assisted these districts and they have lowered their unmet need faster
than other districts. In some cases, districts have reported that once the
building is open it is very hard to maintain, while other districts report that
their problem is the initial construction costs.

Senator Westwood said the federal revenue impact on these
issues was not discussed. He would like the impact of federal funding included
in order to get a full picture. He also said the larger report indicates that
the gap gets larger when federal funds are factored in for Quintiles 4 and 5.
He asked why this because it would seem that Quintiles 1, 2, and 3 would
receive the most federal dollars.

Dr. Ewalt said Fayette and Jefferson Counties really drive
those figures because they receive so much in federal funds. In fact, Jefferson
County had $3 million more in federal dollars in 2005 than in 2004. She said
the smaller districts and lower quintile districts proportionally received more
federal dollars, but in terms of actual dollars, they get less.

Dr. Ewalt discussed the 2007 research agenda for OEA. The
topics include: 1) review of the flexible focus fund program; 2) review of the
extended school services program; 3) understanding how tax provisions interact
with the Support Educational Excellence in Kentucky (SEEK) formula; 4) 50-state
education rankings compendium; and 5) review of the assessment and
accountability of special needs students.

Senator Kelly made the motion to accept OEA's 2007 research
agenda, and Representative Draud seconded the motion. The motion was approved
by voice vote. Dr. Ewalt asked the members to approve the other studies that
were presented in the past, such as the finance reports as well as the School
District Efficiency and Effectiveness Report. Representative Moberly said the
amendment to the motion was accepted, and no members had an objection.

Ms. Seiler briefly discussed the 2006 annual report. She
said the OEA 2006 annual report is included in the members' folders located and
a copy is located in the LRC library. Some highlights included: a background
summary of investigations carried out by the OEA; summaries of research
conducted in 2006, including a review of the SFCC, indicators of efficiency and
effectiveness in primary and secondary education spending, and the 2005 school
finance report.

Representative Moberly made a motion to accept the previous
reports presented by the OEA. Senator Kelly made the motion to accept, and
Representative Draud seconded the motion. The motion was approved by voice
vote.

Senator Kelly would like OEA's annual report on the agenda
for the next EAARS meeting. Representative Moberly agreed, and commended Ms.
Seiler's leadership in getting the EAARS committee and the OEA in sync. He said
OEA is following the direction that EAARS wanted and doing very good work.

With no further business before the committee, the meeting
adjourned at 3:20 p.m.