Promise to Practice Overview

Equity

Equity could be more apparent in Idaho’s school improvement plan. It can be inferred in Idaho’s Accountability Framework as a Gap Closing component, but equity is not explicitly apparent in the State Technical Assistance Team (STAT) model. The state’s planning rubric also includes equity-oriented questions, but it could be more explicit.

Strengths

Idaho’s plan details a meaningful partnership between the state and local teams. Districts are also encouraged to develop comprehensive turnaround plans through guiding questions provided by the state.

Improvements

Check-ins between the state and districts should be more structured around the content of strategy implementation. It’s also unclear if school improvement plan evaluators are still required to use the School Wide Improvement Plans rubric, which would provide guidance on strategic resource allocation for the neediest students. It’s also unclear is resources are being actively matched to districts and schools based on need and capacity.

Coherent and Aligned Vision for Improving Outcomes:

Strategic Use of Funding and Alignment of Resources:

Rigorous Review Process:

Continuous Improvement, Monitoring and Evaluation:

Evidence-Based Interventions:

Capacity Building and Autonomy:

Engagement:

Sustainability:

Click through the tabs on the left to see how Idaho scored in each category.

Equity

How well does the state’s approach to school improvement include focused attention on supporting underserved students and closing the achievement gap? Does the state require LEAs to maintain an equity focus in their school improvement plans, activities and resource allocations?

Equity can be inferred from the materials in the Idaho Accountability Framework as a Gap Closing component of the measure, but is not explicitly apparent in the State Technical Assistance Team (STAT) model. The state’s planning rubric includes equity-oriented questions, but the equity thread is not as clear as it could be. Improvement plans are focused on key areas of need of the school as determined by the needs assessment but do not seem to call out subgroup achievement gaps as a priority area. The school improvement process will only happen if underserved students are supported properly.

Strengths

How is the state thoughtfully leveraging ESSA’s flexibility to put in place the necessary policies and procedures that create an enabling environment for effective and sustained school improvement, and that consider state/local lessons learned from past efforts? What parts of the state’s turnaround strategy or guidance to LEAs were strongest or exemplary?

Interactions between state and local teams are consistent and appear to have shifted from being compliance oriented to meaningful partnerships with an ongoing robust exchange. Guiding questions on the school improvement plan encourage districts to develop comprehensive turnaround plans that can maximize effectiveness. Although the lack of focus on equity and closing the achievement gap requires improvement, the format of the application encourages thorough planning. The state has given some thought to sustaining improvement, through high-touch interactions between the state’s STAT team and districts, and appears committed to stakeholder involvement at each step of the process.

Improvements

How can the state improve its turnaround efforts? What parts of the state’s strategy or guidance to LEAs were unclear? What risks and challenges might the state face with its current approach?

The mutual input from both the state and districts during periodic check-ins is essential for progress. The autonomy of check-ins should be more structured around the content of the strategies implemented for school improvement. Meaningful discourse and guidance about strategic resource allocation for the neediest students and schools will target efforts towards equity in School Wide Improvement Plans (SWIPs). The SWIP rubric provides clear expectations for a plan, but it appears the rubric is no longer required for plan evaluators. The state has made available several resources through state-sponsored networks and external partners yet it is not clear that those resources are being actively matched to districts and schools based on needs and capacity.

Coherent and Aligned Vision for Improving Outcomes

How well does the state articulate a coherent vision or theory of action that drives their school improvement efforts? Is this vision aligned with the state’s accountability system and goals for closing the achievement gap?

While Idaho outlines a vision for improvement and sustainability, this vision is not well articulated throughout their school improvement work such as their applications or aligned with their continuous improvement systems. That said, Idaho has produced a short document which describes how the state, districts, and schools are mutually accountable for school improvement. The document outlines what support districts and schools can expect, including the roles of the state versus the districts and funding implications of being identified for improvement. Both the stakeholder checklists and logic model document indicate accountability at various levels of participation.

Collectively, a statewide goal is to have 90% of schools identified for comprehensive support make progress on at least two of their goals. Idaho developed a logic model that outlines the roles and responsibilities for each of the partners including state, district, schools, as well as external partners and how they can support improving student outcomes. The logic model breaks down the “outcomes – impact” for the state, external partners, and districts and schools. Districts, for example, should see increased student success by 2020-21 as measured by student attendance, behavior, achievement, engagement and graduation rate.

It appears that subgroup performance as an indicator for school improvement is the primary method by which Idaho is aiming to addressing the achievement gap. There is little evidence in the materials provided that demonstrates that the state is calling upon leaders to examine the gap within their school or district, unless Idaho is ensuring that districts and schools are using the data from that indicator to inform the development of their schoolwide improvement plan. The state should be commended for inclusion of external partners in its logic model and expectations about how those partners perform against the model.

Strategic Use of Funding and Alignment of Resources

Is the state allocating funding in a way that is strategic and maximizes resources? Are LEAs expected to prioritize improvement efforts that address the underlying performance issues?

Idaho adopted a formula allocation model to fund schools more efficiently, but this could hinder prioritizing districts with the highest need. States utilizing formula grants may bear a slightly higher burden to help districts develop creative ways to allocate resources across programs to fulfill their goals. It can also reduce incentives for districts to focus on improving school outcomes. Idaho’s school improvement application requires an annual budget for just comprehensive support and improvement schools, broken down by categories such as salaries, supplies and travel. Within the school wide improvement plan, districts must lay out a “few” key needs in instruction and the school program and the process by which they identified the key needs. The plan’s scoring rubric implies that the district “meets standard” if the budget is complete and the budget aligns with prioritized needs. Based on Idaho’s ESSA plan, the state will prioritize targeted schools to receive resources first such as funding, expertise, math and English language arts coaches, leadership training, and assessment development. States utilizing formula grants, like Idaho, may have a higher burden to help districts develop creative ways to allocate resources across programs to fulfill their goals.

Rigorous Review Process

Is the state applying rigorous criteria and review processes to ensure resources will be used to support effective school improvement efforts? Is the state prioritizing funding to LEAs who demonstrate the greatest need for school improvement funding (including LEAs with a high percentage of CSI and TSI schools) and the strongest commitment to school improvement?

After evaluating the impact of the state’s 2017-2018 school improvement pilot, Idaho purposefully shifted away from using a rubric, given the emphasis on compliance serving functions, and instead focuses on forming a partnership between the districts and the state. As a result, the rubric used to evaluate school improvement plans is considered to be a supplemental tool rather than a requirement. For instance, the scoring rubric appears to focus on completion of each plan component, rather than quality.

The rubric does prompt the identified schools to describe the extent to which stakeholders, including tribal leaders, were involved in the development of the improvement plan. It also outlines a comprehensive needs assessment prompting the schools to review evaluate the curriculum and distribution of highly effective teachers, and requires the districts to describe their process for monitoring their progress, among many other elements. While these are key components to a review, it is difficult to gauge the steps that Idaho will take should a district submit a plan that is not comprehensive or rigorous.

Continuous Improvement, Monitoring and Evaluation

Does the state have a robust, data-driven process to monitor LEAs’ implementation of the school improvement plans within their district? Did the state establish clear milestones to ensure improvement over time, and within four years?

Idaho’s system of monitoring school improvement relies on check ins, or self-reporting, as a method to track progress. The monthly check-in process described by the state is intended to foster the state-district relationship as that of a partnership rather than evaluator. Although district and state check-ins appear to be robust and comprehensive, they may risk subjectivity and weakness in prioritizing highest need. For instance, the regular check ins include questions from the state to the district such as “share some feelings and thoughts about the work that has been done to date.” Fortunately, the state also asks what additional supports the district need as they develop their school improvement plans. Idaho also outlines the topics to be laid out during each progress check in.

In addition, Idaho developed a logic model and timeline that outlines annual outputs delineated by state, external partners, and districts and schools. Milestones or expectations of improvement over time are not clearly defined. Aside from regular check ins that suggest highlighting small success, it is unclear how the monitoring tool is connected to the theory of action to evaluate school wide improvement plans. It is also unclear what data the districts are using during their check in conversations. If the state is relying on the districts to evaluate success during these periodic meetings, is there sufficient capacity to sustain this?

Evidence-Based Interventions

To what extent is the state mandating LEAs use evidence-based strategies in their improvement efforts? Does the state provide guidance and supports to LEAs to help them identify and implement the most effective strategies based upon their needs?

There is clear indication within Idaho’s ESSA plan, logic model, and application that evidenced-based strategies should be used to drive improvement. The state has provided multiple resources for districts including evidenceforessa.org, U.S. Department of Education’s What Works Clearinghouse, as well as the Northwest Comprehensive Center and Regional Education Lab Northwest. Idaho also requires districts to provide evidence-based interventions for each prioritized need, identify the level of evidence and give a description of how the intervention meets the definition of evidence based on the state application.

Districts are required to detail the extent to which academic intervention programs demonstrate efficacy with the target population and how the interventions support the needs of students from diverse cultural and linguistic backgrounds. In addition, they must also detail what processes will be used to monitor and evaluate the plan for effectiveness. There is time dedicated specifically to discussing intervention strategies on the state timeline involving all members of State Technical Assistance Team (STAT), but available evidence does not provide any indication that there will be specific guidance and support for implementation for targeted populations. The STAT could conduct comprehensive needs assessments, define evidenced-based interventions, and conduct or evaluate periodic data collection.

Capacity Building and Autonomy

How well does the state articulate, delineate or set parameters around which interventions and responsibilities belong to the state, LEA and/or school? Does the state provide support or guidance to help LEAs identify and reduce barriers to school improvement? Does the state have a framework or process to support and monitor outside entities who partner with the state, LEAs or schools in school improvement efforts?

Idaho noted that each school is assigned a capacity builder improvement coach shortly after being identified for comprehensive support. The state describes capacity builders as the heart of the state’s school improvement system. They play an active role in supporting identified schools with their improvement efforts including supporting the school in developing an approvable SWIP, attending the check in phone calls with the state for feedback, and working to establish a school-based leadership team to sustain the work. Idaho should be commended for their efforts to support the capacity of the districts.

Idaho’s State Technical Assistance Team (STAT) logic model and timeline clearly illustrates the delineation of inputs, outputs and outcomes expected of the state, external partners and districts and schools. External partners are provided for districts and schools including the Superintendents Network, Building Capacity Project, Principals Network, Northwest Rural Innovation Student Engagement, Response to Intervention, Technical Assistance and Professional Development.

Idaho has not provided a framework or monitoring tool to measure the quality of outside providers, and has left it up to districts to describe the “rigorous review process.” External resources are also optional for districts and would be utilized best if STAT identifies and matches district and school leaders with the available supports that best meet the key priority areas identified in the needs assessment. This approach would reduce barriers to school improvement.

Engagement

Does the state require LEAs to engage with stakeholders such as parents and community members in the development and implementation of their school improvement plans? Does the state provide sufficient guidance and resources to LEAs to effectively do so, helping them foster local buy-in and promote sustainability?

Stakeholder engagement is evident in Idaho’s ESSA plan, needs assessment for a School Wide Improvement Plan, and rubric that outlines expectations The states ESSA plan encourages stakeholder engagement in the School Wide Improvement Plan processes and instructs districts with low-performing schools to include the principal, teachers, and parents in the development and implementation. The needs assessment requires stakeholder positions be listed along with description and utilization within the School Wide Improvement Plan. Specific engagement plans and strategies must be outlined to identify the roles of each stakeholder. Although Idaho has a robust plan on encouraging stakeholders, there is no detailed guidance on how to effectively promote sustainability, which keep this area from an exemplary rating.

Engagement

Does the state have a plan in place to review the school improvement efforts statewide and evaluate the impact and effectiveness? Does the state have a process in place to support LEAs and schools by enhancing their capacity to maintain their improvement efforts upon exiting identification and intervention?

Idaho’s ESSA plan indicates that the state will require a written plan for sustaining improved student achievement to be approved by the State Technical Assistance Team (STAT). This sustainability plan will include goals, strategies and a monitoring plan, as well as an explanation of how the school will sustain improvement after no longer receiving improvement funds. The state asserts that state-sponsored professional development opportunities required of identified schools will still be available to schools that have exited from improvement status, and the logic model describes that STAT will monitor implementation and progress while a school is in improvement. Little could be found to indicate that this sustainability work has progressed yet, which makes it uncertain to what extent the state will continue supporting and enhancing the capacity of districts and schools once they have exited improvement.

The only documentation of statewide evaluation of improvement efforts was by progress monitoring at the district and school level through high-touch interactions. That said, the state does report that it has reached an agreement with their regional comprehensive center to conduct a third party, mixed-methods evaluation of the state’s capacity builder program, which should lead to useful information for the state to use.

Overview

Strengths

Idaho’s plan is built on high-quality standards and assessments and a straightforward set of indicators, and the plan intends to celebrate its highest-performing schools. Notably, the state attempts to provide multiple avenues to success through both current performance and improvement over time.

Idaho appears to have invested considerable effort into its school improvement plan. The plan includes detailed descriptions of the types of resources that will be available to districts, such as an on-site review process and a comprehensive list of supports, but it will be up to districts to take advantage of them.

Weaknesses

Idaho’s decision not to use a summative rating for schools, and the fact that schools will be held accountable for either achievement or growth on each indicator, could make it extremely challenging for parents to compare performance across schools. Moreover, Idaho’s chosen method for measuring growth does not actually track individual students over time, and a school could be given credit for making “growth” even if no individual students were actually making progress.

Idaho missed opportunities to broaden the lens of what makes for a successful school. For example, while Idaho will test students in science and pays for all high school students to take the SAT, it does not plan to add those results to its school rating system.

Finally, although the plan includes a comprehensive list of optional supports, it does not appear that Idaho will employ the kind of transformative interventions that are likely necessary to turn around the very lowest-performing schools.

Click through the tabs on the left to see how Idaho scored in each category.

GOALS

While the state’s aspirations seem reasonable and attainable, Idaho’s approach to goals raises some concerns.

Idaho aims to reduce the percentage of students who are not proficient by 33 percent over six years. The state proposes the same methodology for English language proficiency. For graduation rates, Idaho aims to reduce non-graduates by 75 percent over six years, with a 95 percent target within six years. It applies the same methodologies to other groups of students, resulting in more ambitious goals for groups that are currently lower performing.

Absent historical data, it is unclear whether these goals are ambitious or attainable. The state neither provides historical performance on the state assessment nor ties its goals to an objective bar. The state also says it will disaggregate performance for individual subgroups of students, but it has only provided goals for a large combined group of “minority” students, rather than breaking out targets for individual groups, such as African-American or Latino students.

STANDARDS AND ASSESSMENTS

Idaho has a high-quality assessment system in grades 3-8 and 10, developed by the Smarter Balanced Assessment Consortium.

The Idaho Standards Achievement Test (ISAT) includes both interim and summative assessments using a computer-adaptive process that results in a coherent and comprehensive system of assessment. It has already passed federal standards for validity and reliability. To measure English language proficiency, Idaho uses the WIDA ACCESS test, which is a popular and high-quality choice.

At the high school level, the state also pays for every student to take the SAT, but does not include this as part of its accountability system (it uses only the 10th-grade state assessment). In addition, the state administers a science assessment in grades 5 and 7, but this is not included as part of accountability. Idaho could consider incorporating both the SAT and science assessments as a way to broaden the scope of its accountability system.

Idaho could strengthen its plan by providing more information about its alternate achievement standards and aligned assessments for students with the most severe cognitive disabilities, as well as ensuring that it has a process in place to meet the 1 percent cap on alternate assessments.

INDICATORS

Idaho has chosen a straightforward list of indicators to include in its accountability system, but it’s unclear how those indicators will be measured and whether they are all valid, reliable, and comparable statewide.

Elementary and middle schools will be held accountable for student proficiency rates in English language arts and math, a measure of growth on those assessments, English language proficiency, and a student satisfaction and engagement survey. As discussed in more detail below, the “growth” measure Idaho is proposing does not actually measure individual student progress over time. And while Idaho’s student engagement survey may be useful as a diagnostic tool, it’s not clear from the information provided whether it has sufficient validity, reliability, and comparability to belong in a high-stakes accountability system.

At the high school level, Idaho is proposing a similar list of indicators: proficiency in reading and math, English language proficiency, four-year graduation rate, “growth” in graduation rate, and college- and career-readiness indicators. While this is a relatively straightforward list, it may pose some unintended consequences for schools. For example, the graduation rate “growth” indicator may disproportionately reward lower-performing schools, since it appears to be equally weighted with the four-year graduation rate.

Similarly, while it is encouraging to see Idaho reward high schools for students who demonstrate college and career readiness through participating in advanced opportunities, earning industry-recognized certification, and/or participation in recognized high school apprenticeship programs, it’s unclear how these variables will be defined or if they’re all equally rigorous. Additionally, the plan indicates the denominator for this variable will be graduating students, omitting students who did not graduate. Idaho’s plan would be stronger if the denominator was based on all students.

ACADEMIC PROGRESS

Idaho plans to measure student achievement as the percentage of students performing on grade level.

This is a clean, easy-to-understand measure of student achievement. However, Idaho’s proposed growth measure only tracks year-over-year changes at the school level. This method is susceptible to differences in the student population enrolled in a given school in a given year. As such, a school’s performance on the growth indicator could improve (or decline) simply because the population served by the school is changing.

Even if the state did have an individual growth model, the state’s methodology does not incentivize schools to care about both proficiency and growth.

Idaho is planning to give full weight in its school identification system to either proficiency or growth. That is, schools will be ranked on both their achievement and growth scores, and then the measure used for their accountability rating will be the higher of the two rankings. This is particularly problematic given the way Idaho has structured its growth measure, because a school could be given credit for making “growth” even if no individual students were actually making annual progress. The opposite is true as well—schools with a high proficiency rating could maintain a high score even if their students failed to make progress over time.

ALL STUDENTS

To consider the performance of subgroups, Idaho will identify any school with a subgroup gap of 35 percentage points as compared to their non-group peers.

However, it’s not clear if this comparison is within schools or a statewide comparison, and using gap closure as the definition runs the risk of keeping the bar low for historically underserved groups. Instead of setting the bar at a high level of achievement, it sets the bar based on comparison with their peers, who may not be achieving at high levels.

Idaho is planning to use a combined subgroup of “minority” students to run these calculations, and it has not provided a rationale for why it chose this 35 percent threshold. Similarly, the state will identify “additional targeted support” schools where any individual subgroup, on its own, would be identified for comprehensive support, but it has not yet provided data on how many schools this would capture.

Idaho has set its minimum n-size at 10 students, meaning any group with at least 10 students will be considered for accountability purposes.

The state provides helpful data showing why it made this decision. This allows the state to capture more subgroups for more data elements, and thus creates a more statistically sound identification system. Idaho is also planning to use three years of data, which will further boost the inclusion of subgroups and help make the system less subject to one-year swings.

Idaho has also placed a substantial weight—30 percent in elementary and 22.5 percent in high school—on the performance of English learners.

While the state’s desire to focus on English learners is understandable given their historically low achievement in Idaho, placing so much weight on this indicator could create perverse incentives. This is particularly problematic given that the state is still waiting on its latest results in order to finalize how the indicator would be measured or incorporated into the accountability system. Moreover, the state mentions just 6 percent of its students are English learners, and schools with concentrations of English learners will, in effect, have a very different accountability system than the majority of schools in the state.

Idaho’s approach to the 95 percent participation requirement is concerning.

If a school fails to meet the threshold in a given year, participation can also be calculated with a three-year average. Moreover, the sanction for not meeting the requirement—the development of an outreach plan—may be insufficient.

IDENTIFYING SCHOOLS

Idaho will identify “schools for improvement only if they are both the lowest performing in the state and not improving.”

Idaho will turn each of its variables into statewide rankings of both current-year performance and improvement over time, and then use only the higher of the two ranks. That is, if a school scores well overall, but exhibits low growth on the measure, it will get credit for its overall high score and not be penalized for a lack of growth. On the other end, a school with large improvements, but low absolute performance, will be scored based on its improvements.

Idaho has elected to use a dashboard rather than a summative rating, which will be difficult for parents to determine how a given school is performing, and how its performance compares with other schools’ performance.

Idaho should consider what steps it will take to ensure that the dashboards are parent-friendly and provide actionable information for all stakeholders. In addition, allowing schools to choose between either achievement or growth may run counter to ESSA, which requires states to design accountability systems that are comparable statewide.

Idaho’s plan for identifying schools for comprehensive support is based on a relative ranking of all schools across the state.

It will identify the bottom 5 percent of schools and will identify K-8, high schools, and alternative schools separately (which will protect against certain grade spans having systematically higher or lower scores). The state will also identify for comprehensive support high schools with graduation rates below 67 percent based on four-year cohort graduation rates, averaged over three years. Other than the graduation rate rule, schools will be ranked relative to one another, and it would be possible for a school to escape scrutiny simply because other schools perform worse.

Idaho’s approach to identifying schools with low-performing subgroups appears to be oriented around schools with large within-school gaps.

This method should help identify places with large within-school gaps, but it may not identify schools with lower overall performance and especially low-performing subgroups.

SUPPORTING SCHOOLS

Idaho’s plan describes a comprehensive statewide system of supports for identified schools.

However, it’s not clear whether the list of supports will be sufficiently rigorous to turn around the lowest-performing schools. The plan includes detailed descriptions of the types of resources that will be available to districts, but it will be up to districts to take advantage of them.

All comprehensive improvement schools will conduct a needs assessment and develop an improvement plan based on the results.

Those plans must “address the why, who, what, when, and resource allocation for making improvement changes” and articulate short- and long-term goals. The plan must include external stakeholders in the development process and during implementation. Districts must also address how they will monitor and oversee the improvement plan’s implementation, as well as how its effectiveness will be evaluated.

For schools that fail to improve for three years after being flagged as in need of improvement, Idaho will conduct an on-site visit using a structured protocol that results in a set of recommendations. The state may also direct some of the district’s spending and assign a leadership coach to the school district. Idaho’s statewide system of support includes diagnostic needs assessments, school improvement coaching, mentoring and support for principals, training on the state’s content standards, support for English learners, and technical assistance on usage of school time and engagement with families and the community.

These first steps, particularly the on-site review and comprehensive supports, are likely necessary—but not sufficient—to turn around the lowest-performing schools and subgroups. Idaho mentions that its school improvement team will work with school districts to ensure their interventions are evidence-based, but it does not articulate what this would mean in practice.

The state plans to distribute the 7 percent of its Title I funds dedicated to school improvement through a formula.

This will limit the ability of the state to push districts toward more rigorous interventions. To further strengthen its plan, Idaho should indicate if and how it intends to provide direct student services using the optional 3 percent set-aside.

EXITING IMPROVEMENT STATUS

Idaho’s plan calls for a three-year timeline for identified schools.

Specifically, a school in support status can exit if (1) it no longer meets the criteria that resulted in initial identification, (2) it shows a “consistent growth trajectory,” and (3) it submits a written plan as to how the school will sustain success without additional funding. A school can exit early (but forfeit funds) if it is on track to hit its three-year target.

A school that performs marginally better than those in the state’s bottom 5 percent could lose necessary attention and supports to be able to escape scrutiny. Instead, Idaho could strengthen its plan by requiring each low-performing school to meet a rigorous, objective bar, such as meeting its interim goals for two-to-three years, before exiting to ensure that schools do not bounce in and out of status.

CONTINUOUS IMPROVEMENT

The state deserves credit for embedding stakeholder feedback in its school improvement planning processes.

Idaho’s plan often cites stakeholder engagement and buy-in in the initial development, including how to define an appropriate subgroup size, how to set performance targets, and in its school identification approach. Districts with low-performing schools must include the principal, teachers, and parents, in the development and implementation of school improvement plans. Additionally, districts must articulate how they plan to monitor and oversee the plan’s implementation, and how they will evaluate the effectiveness of the plan over time.

However, it’s not entirely clear who the state consulted in the plan-writing process or what happens moving forward with regard to learning and adjusting as the plan is implemented.

At the state level, while Idaho is transitioning to more of a hands-off approach toward schools, it will be important for the state to remain vigilant in monitoring and supporting the schools that have historically struggled. It would be helpful for the state to develop a list of key stakeholder groups and organizations that can be part of an ongoing feedback cycle and inform implementation of this plan