Promise to Practice Overview

Equity

Massachusetts does not include a component in its school improvement plan that focuses squarely on equity. The state’s improvement targets under ESSA set higher expectations for underserved groups as required, but these are targets, not explicit strategies for approaches for achieving them.

Strengths

The state does a good job of leveraging local autonomy and accountability by providing a range of quality resources, documents, processes, and plans to districts and schools, as well as clearly defining district/school and state roles. Massachusetts’ emphasis on interim benchmarks and long-term goals that are derived from available data should be emulated by other states.

Improvements

Massachusetts needs to create more urgency around closing opportunity and achievement gaps by designating or prioritizing more concrete strategies specifically describing how it will do so. Massachusetts should also adapt its current accountability system to ESSA—providing guidance on how and when states will extend supports to low-performing schools under ESSA, and how schools can exit from comprehensive and targeted support.

Coherent and Aligned Vision for Improving Outcomes:

Strategic Use of Funding and Alignment of Resources:

Rigorous Review Process:

Continuous Improvement, Monitoring and Evaluation:

Evidence-Based Interventions:

Capacity Building and Autonomy:

Engagement:

Sustainability:

Click through the tabs on the left to see how Massachusetts scored in each category.

Equity

How well does the state’s approach to school improvement include focused attention on supporting underserved students and closing the achievement gap? Does the state require LEAs to maintain an equity focus in their school improvement plans, activities and resource allocations?

While Massachusetts has created a well-defined structure for school improvement, it does not include a component that focuses squarely on equity. There are references within various tools and resources to underserved students and achievement gaps, and sometimes prompts to describe how they will they be addressed through state, district, and school efforts, but the equity thread through the school improvement system is hard to find.

The state’s improvement targets under ESSA set higher expectations for underserved groups as required, but these are targets, not explicit strategies for approaches for achieving them. Given that the needs assessments and root cause analyses in underperforming schools will inevitably uncover areas of need among underserved student populations, it is incumbent on the state to create a stronger, more coherent equity focus throughout all of its school improvement efforts.

Strengths

How is the state thoughtfully leveraging ESSA’s flexibility to put in place the necessary policies and procedures that create an enabling environment for effective and sustained school improvement, and that consider state/local lessons learned from past efforts? What parts of the state’s turnaround strategy or guidance to LEAs were strongest or exemplary?

Massachusetts has been consistent in its approach to school improvement, specifically for its lowest performing schools, for several years, and that approach is regarded by many as the best currently utilized by any state. For those schools, the state does a good job helping districts and schools understand what their roles are and what the state’s involvement will be.

The state does a good job of leveraging local autonomy and accountability by providing a range of quality resources, documents, processes, and plans to districts and schools. Massachusetts’ emphasis on interim benchmarks and long-term goals that are derived from available data should be emulated by other states.

Improvements

How can the state improve its turnaround efforts? What parts of the state’s strategy or guidance to LEAs were unclear? What risks and challenges might the state face with its current approach?

Massachusetts needs to create more urgency around closing opportunity and achievement gaps by designating or prioritizing more concrete strategies specifically describing how it will do so. The state is clear that it will use its authority if a district or school is failing to meet accountability targets. However, it must be just as clear how a district with a Level 1 school that shows a large achievement gap will be held accountable, even if it is a relatively small student population.

The state has more work to do to adapt its established accountability system to the ESSA era. It is not clear how and when the state will extend supports to low-performing schools going forward under its ESSA plan or when schools can be exited from comprehensive or targeted status.

Massachusetts would also benefit from a more explicit focus on feedback loops: how these frameworks, processes, procedures, and needs assessments working for all schools, districts, and the state.

Coherent and Aligned Vision for Improving Outcomes

How well does the state articulate a coherent vision or theory of action that drives their school improvement efforts? Is this vision aligned with the state’s accountability system and goals for closing the achievement gap?

Massachusetts has identified four strategies for school improvement: establish a school-wide community of practice through leadership, shared responsibility, and professional collaboration; employ intentional practices for improving instruction; provide student-specific supports and instruction to all students; and provide appropriate social, emotional, and behavioral supports in order to create a safe, orderly, and respectful learning environment for students and teachers. The theory of action is grounded in “strong authority” and “sufficient autonomy and flexibility to empower school leadership within a context of accountability.” These strategies seem relatively well established in the state.

Likewise, Massachusetts has a strong and consistent track record of setting goals and expectations for schools and districts, then giving authority and autonomy to schools and districts who are not low performing to meet those expectations.

Even so, the state can do more to connect the state’s improvement strategies and theory of action to school improvement under ESSA in order to bring coherence to state expectations under the law, especially through an equity lens. For example, in the improvement and turnaround documentation, there is no mention of comprehensive or targeted schools. Level 3, 4, and 5 schools are in the lowest 20% of schools based on a combination of data and other evidence, which appears to mean that there will be both comprehensive and targeted schools in Level 4 and could lead to confusion. It would benefit schools, districts, and the state to ensure common language around status and expectations in the ESSA era.

Strategic Use of Funding and Alignment of Resources

Is the state allocating funding in a way that is strategic and maximizes resources? Are LEAs expected to prioritize improvement efforts that address the underlying performance issues?

Massachusetts expects districts to align funding and resources to the lowest performing schools, and the state’s ESSA plan details how it will allocate federal school improvement funding to comprehensive and targeted schools based on need and strength of plan. However, further details are sparse at this point, and it is not clear what criteria the state will use to make to make fund allocation decisions based on the plans.

The state has set a clear expectation that districts use their autonomy and funding flexibility to make decisions about how funding is being maximized to target the needs of students who are struggling. Again, however, the framework for doing so is not clear. In their improvement plans, schools and districts are required to identify turnaround strategies specifically for low-income students, English learners, and students with disabilities, yet the rubric focuses on the overall budget proposal and its scope.

Rigorous Review Process

Is the state applying rigorous criteria and review processes to ensure resources will be used to support effective school improvement efforts? Is the state prioritizing funding to LEAs who demonstrate the greatest need for school improvement funding (including LEAs with a high percentage of CSI and TSI schools) and the strongest commitment to school improvement?

It is evident that the state employs, and expects districts to use, rigorous criteria and review processes to ensure effective school improvement efforts are employed. Massachusetts has a number of rubrics and processes documented, but there appears to be a missed opportunity to create cross-district learning, especially those that have demonstrated the ability to meet accountability targets or have made strong progress in closing opportunity and achievement gaps.

Funds have been identified for districts that are in need of priority assistance, such as Level 3 grant funding and Level 4 technical assistance. The rubric reviewed for Level 3 indicates the expectation that districts describe how resources will be used to support effective school improvement, including the capacity of school principal and the utilization of time to improve teaching and learning. However, there is a lack of detail around the review for Level 4 plans and, as stated above, while Massachusetts intends to focus resources on those high need schools with the clearest vision for success, it is not clear how that will happen in reality.

Continuous Improvement, Monitoring and Evaluation

Does the state have a robust, data-driven process to monitor LEAs’ implementation of the school improvement plans within their district? Did the state establish clear milestones to ensure improvement over time, and within four years?

In Massachusetts, it is clear that Level 1 and 2 schools and districts are entrusted to take action and responsibility for improvement with very little involvement from the state. The state becomes increasingly involved from Levels 3 — 5, but there is not a lot of documentation provided beyond Level 3. Massachusetts expects that districts with Level 3-5 schools develop measurable annual goals, including those that target English learners, students with disabilities, and low-income students. Districts are also required to have interim benchmarks that must be submitted to the state periodically. Guidance is provided on particular priority areas that annual goals should address and sample goals, and the turnaround plan for Lawrence on the state website provides good evidence of the state’s role with at least these districts and schools.

That aside, there are hints and indirect evidence of the state’s active involvement in monitoring and ensuring improvement in Levels 1 – 3, but there is very little specific detail available. There does not appear to be a mechanism through which the state engages formally with schools or districts about their school improvement plans if they are not identified for intervention as part of the underperforming criteria identified in state statute. There is evidence of resources for district and school improvement planning called Planning for Success, which was piloted by districts in Massachusetts and is based on the Massachusetts Planning and Implementation Framework, but it does not seem to dive deeply into closing opportunity and achievement gaps.

Evidence-Based Interventions

To what extent is the state mandating LEAs use evidence-based strategies in their improvement efforts? Does the state provide guidance and supports to LEAs to help them identify and implement the most effective strategies based upon their needs?

Massachusetts has clearly established the four strategies for improvement in its theory of action, which the state asserts are evidence-based, and has invested in providing a repository of information and resources for schools and districts to use in implementing those strategies. The four strategies are part and parcel with Level 3 turnaround guidance. The state’s “How Do We Know” initiative is a solid example of how a state can provide resources on evidence-based practice to its districts. For Level 4 and Level 5 districts, the state plays a well-established, hands-on role in the improvement process.

Massachusetts stresses local autonomy and flexibility for schools and districts not in need of improvement, and it is unclear where the state’s role ends and the local district’s role begins in establishing and maintaining evidence-based interventions in those cases. The state provides resources, guidance, and training from the state level and, to at least some degree, professional development services through its District and School Assistance Centers. However, whether schools in Levels 1 — 3 are expected to take advantage of these services is not evident.

Capacity Building and Autonomy

How well does the state articulate, delineate or set parameters around which interventions and responsibilities belong to the state, LEA and/or school? Does the state provide support or guidance to help LEAs identify and reduce barriers to school improvement? Does the state have a framework or process to support and monitor outside entities who partner with the state, LEAs or schools in school improvement efforts?

Massachusetts has an established history of taking state-level action to improve the lowest performing schools and districts. For Level 5 schools, the state has the authority to do full takeovers through receivership, to assign third-party organizations to manage a school on behalf of the state or the district, or to support districts in establishing alternative governance structures for specific schools or clusters of schools. Each of these authorities have employed at least once.

For districts not in the highest levels of intervention, resources are made available but expectations can be unclear. It is clear that autonomy is a key lever utilized in the school improvement strategy, and for this reason little guidance is provided to schools and districts not in turn around status (Level 3 — 5). Discrepancies in student outcomes within districts not identified for improvement appear to receive little attention from the state, and there is concern that district planning tools do not adequately prompt districts to analyze these outcome discrepancies for themselves.

Engagement

Does the state require LEAs to engage with stakeholders such as parents and community members in the development and implementation of their school improvement plans? Does the state provide sufficient guidance and resources to LEAs to effectively do so, helping them foster local buy-in and promote sustainability?

For schools in Levels 3 – 5, Massachusetts has clear requirements for stakeholder engagement. It is less clear what the state expects from schools in Levels 1 – 2, although the state’s “Planning for Success” initiative may address this.

Even for schools in Levels 3 – 5, the state could do more to provide guidance and resources to districts and schools to effectively engage stakeholders. Guidance does not take into account the cultural and linguistic diversity of students and families. For example, translation and interpretation removes barriers to strong and meaningful participation of parents and community stakeholders, but that expectation is missing. There is also no description in how the state assists the districts and schools in fostering local buy-in and promoting sustainability. For example, the state could introduce or broker relationships with high impact opportunities between districts and local businesses, universities, and city entities.

Engagement

Does the state have a plan in place to review the school improvement efforts statewide and evaluate the impact and effectiveness? Does the state have a process in place to support LEAs and schools by enhancing their capacity to maintain their improvement efforts upon exiting identification and intervention?

Massachusetts is clearly committed to evaluating its approach and has completed multiple studies of its improvement and turnaround strategies. That said, the extent to which those studies have informed changes to the state’s approach is not apparent.

The state offers a sustainability toolkit for districts and schools online, and the sheer amount of self-help and directed resources, tools, frameworks, grant funding, and other supports lends significant strength to sustainability of improvement efforts. It’s not clear, though, how all of this work and support aligns to schools in comprehensive and targeted status and when those schools exit those identifications. Again, alignment between Massachusetts’ existing approach and ESSA would bring more coherence.

Overview

Strengths

The Massachusetts plan begins with a compelling and coherent vision of the state’s successes and challenges — resulting in the articulation of five overarching strategies that are meant to drive the state’s plan. The state’s Executive Summary is clear, understandable, and accessible for all audiences.

Massachusetts is also noteworthy for its strong history of taking actions that have resulted in improvements in the state’s lowest-performing schools and districts. The plan offers useful research, experiences, and insights into what the state’s approaches are, and how and why they’ve worked in the past.

The state’s plan for Supporting Excellent Educators is also a model worthy of study by other states. It emphasizes how educator supports for early literacy and middle school math, for example, will push the thinking and learning of teachers and principals.

Weaknesses

Massachusetts’ plan does not connect the dots between its long-term vision and how that vision will be embodied in its accountability system. As a result, it misses multiple opportunities to tie these together in meaningful and actionable ways.

Massachusetts clearly spent a lot of time rethinking its performance index, indicators, and determinations. But due to the lack of data, simulations, and examples, this is still somewhat abstract rather than an actionable plan.

The state proposes to include stakeholders as it continues refining and implementing its ESSA plan, but the state has not proposed a clear stakeholder engagement plan, and it lacks timelines around many critical decisions.

Click through the tabs on the left to see how Massachusetts scored in each category.

GOALS

Massachusetts aims to reduce the proficiency gap by one-third over the next six years.

Although the state is transitioning to new assessments and the results will not be available until the summer of 2017, the state has provided a sample set of performance targets based on its prior state assessments. Similarly, Massachusetts is waiting for more current data on English language proficiency, and in the meantime has provided placeholder goals that it plans to update in the near future.

Massachusetts’ proposed “long-term goal” for graduation rates is set for 2020 — but it would still leave large gaps among different student groups.

Encouragingly, the state has set gap-closing graduation rate improvement goals for lower-performing subgroups, and it says the goals are based on historical improvement rates. However, the state does not explain how it derived its 2020 goals, nor provide historical data to demonstrate that they are truly ambitious.

Massachusetts does not provide a comprehensive vision or set long-term goals for the future.

While Massachusetts does highlight five overarching strategies to support all students, based on clear gaps that the state is currently experiencing, these strategies are not cohesively pulled through the plan as submitted.

STANDARDS AND ASSESSMENTS

Massachusetts has had strong standards that align with college readiness, but the state is in the midst of revising those standards.

The state has been using the Partnership for Assessment of Readiness for College and Careers (PARCC) assessment, but is in the process of developing and administering a new assessment (derived from PARCC) called the Next-Generation MCAS. Due to the transitions taking place and the lack of transparency around those, it is not certain at this time that Massachusetts’ new standards and assessments will be as strong as they have been historically.

Massachusetts does not assess students in any language other than English.

This policy is due to a state law passed in 2002 — and excludes the high school mathematics assessment, which is offered in Spanish. Despite this legal obstacle, research indicates that English learners are best served by being permitted to take content-area assessments in their native language.

Lastly, Massachusetts should strengthen its plan by ensuring that it has a process in place to meet the 1 percent cap on alternate assessments for students with the most significant cognitive disabilities.

INDICATORS

Massachusetts’ plan includes a relatively simple list of high-quality indicators.

Massachusetts plans to revisit its indicators when it has its new assessment results in summer 2017, but for now the state’s plan includes: academic achievement, academic progress, English language proficiency, graduation rates, and chronic absenteeism.

Within those categories, its school quality indicators appear to be well constructed. The state’s proposed use of “Success in grade 9 courses” is notable, as it will serve as an early warning indicator to help schools and districts identify which students are most likely to drop out, and position them to better respond. To produce this measure, the state proposes to use prior-year student performance data to determine if a student failed one or more courses in high school.

Massachusetts also deserves credit for including its science assessment in its accountability system.

That decision will help lessen concerns about curriculum narrowing and will force schools to pay attention to student performance beyond reading and math.

The plan also includes a measure for the successful completion of “challenging coursework,” but it does not describe how that would be defined. The state may want to consider combining this indicator with access to the courses as an incentive for schools to pay attention to both access and success.

The state should place greater weight on its four-year graduation rate.

At the high school level, it’s not clear if the state’s proposed inclusion of two separate measures of graduation rates (in addition to an annual dropout rate) are sufficiently discrete to add extra information to the system. The state would have a stronger plan, and more closely align its accountability system with its long-term goals, if it placed greater weight on the four-year graduation rate.

ACADEMIC PROGRESS

Massachusetts’ accountability system plans to include both student achievement and growth.

It is planning to place a strong weight on both, but neither of its proposed measures is aligned to the state’s grade-level standards for students.

The state’s proposed achievement measure is an average scale score.

This method combines all student scores into one number, which avoids setting rigid cut points for students, but it can obscure information about how many students are meeting various performance thresholds. Similarly, the state’s growth measure, Student Growth Percentiles, compares students with each other, not with an objective standard. While the state plans to add another growth measure based on whether students are growing at a pace sufficient to meet state standards, the plan provides very little information about how that measure will be used or when it will be phased in.

ALL STUDENTS

Massachusetts is clearly wrestling with the question of how to incentivize schools to address the needs of every child with a focus on traditionally disadvantaged students.

But with few specifics and little backup data to support its outlined approach, it is difficult to predict its impact.

Massachusetts plans to identify any school with a subgroup in the lowest performing 10 percent “of all eligible subgroups” for targeted support.

However, it’s not entirely clear from the state’s plan if that applies to each subgroup or if the state is envisioning one composite group based on all subgroups. The former would be a stronger plan and would allow for more tailored interventions than one composite group of “achievement gap” schools, but the state could provide greater clarity about how this identification process will work and how many schools might be identified.

Massachusetts plans to include “gap-closing for high-needs students” as part of its accountability system.

However, more detail is needed about how it would be weighted in its overall school rating system. The state provides data showing that its “high-needs” subgroup would capture virtually all underserved students in schools where the number of English language learner, economically disadvantaged, and special education students is not high enough to be disaggregated alone. This high-needs subgroup could be a useful addition to subgroup analysis and performance, with the stated impact of adding an additional 150 schools to be held accountable for subgroup performance.

The state has not committed to how it will address students with disabilities or English learners.

Massachusetts has also not committed either way to including or excluding former students with disabilities within the students-with-disabilities subgroup. Similarly, the state does not commit to how it will approach the inclusion of English learners for either recently arrived English learners or for recently exited English learners for a time frame of two to four years. Nor does it say when it will make a decision.

IDENTIFYING SCHOOLS

Massachusetts has the outline of a plan for how it will rate schools.

The state has proposed weights of its indicators in its accountability system, but it plans to revisit those decisions in the coming months. Once the indicators are combined, each school will be assigned to a tier, qualifying it for specific levels of support or intervention.

The state has also committed to identifying a number of schools with low-performing subgroups for targeted support.

Massachusetts states that it will model out its plan once it has additional data, but for now it’s difficult to know what that will look like.

Educators, parents, and communities deserve — and indeed, need — to understand the state’s thinking behind its final weighting system. Further, the state explains that it plans to reset the index ranges each year, but it does not explain how the state would make those changes, why it would do so, or how any changes would be communicated to educators and parents. To drive improvement behaviors, all of this must be transparent.

SUPPORTING SCHOOLS

Massachusetts has a strong track record in this area that could serve as a model for other states.

Massachusetts has commissioned studies to evaluate the effectiveness of its approaches to school turnaround: a 2014 study called Turnaround Practices in Action and a 2016 quasi-experimental study called Evaluation of Level 4 Turnaround Efforts in Massachusetts. Together, these studies form the evidentiary foundation for the state’s current practices and Massachusetts will continue to embed those principles into competitive grant funds dedicated for school-improvement activities.

Massachusetts cites its actions that have resulted in improving the state’s lowest-performing schools and districts.

The state has the authority to do full takeovers through receivership, to assign third-party organizations to manage a school on behalf of the department or the district, or to support districts in establishing alternative governance structures for specific schools or clusters of schools such as Empowerment Zones. It has used each of these authorities in different circumstances across the state — all to good effect.

For schools and districts that do not require the highest level of intervention, Massachusetts provides resources focused on four evidence-based turnaround practices; it provides direct support to the state’s largest districts through the Commissioner’s Districts programs; it supports small and midsized districts through the District and School Assistance Centers; and three additional offices provide specialized support in such areas as social/emotional learning, partner vetting/selection, and managing/evaluating/monitoring those in receivership.

It’s not clear how Massachusetts will extend supports to low-performing schools going forward.

Information on this issue wasn’t readily available in the state’s plan. In addition, there is a lack of specificity around exactly how, at what time, or in what stages support will be provided to low-performing schools. There are no examples given of how this impressive array of support will be tailored to specific needs.

Exiting Improvement Status

Massachusetts has not yet finalized its exit its criteria for low-performing schools to exit improvement status.

However, it has outlined what those might be once the state has more current data. To exit comprehensive support status, schools must meet achievement, growth, and gap closing targets, and submit “exit assurances” that provide a school’s plan to sustain its efforts going forward.

For schools identified for targeted support, they can exit improvement status when they rise above the 20th percentile within the group that led to the school’s identification and if that group meet its annual targets. Although these criteria are still in development, they represent a potentially strong combination of requirements that would demonstrate schools were on track for sustained improvements.

Continuous Improvement

Massachusetts emphasizes its commitment to continuous improvement.

The state says it will continuously engage stakeholders to develop improvements. It is less clear in the plan where specific stakeholder feedback may have influenced a decision or will influence future decisions.

Massachusetts also asserts that it has established intentional processes that allow the state to study the effectiveness of initiatives.

The state, however, does not identify the frequency or “the how” behind the ongoing conversations and processes and improvements/adjustments to the overall state plan. It is positive that the plan references that the state agency will focus on implementation, including dedicating some staff to the effort. It is also positive that the state plan will be the basis of the report to the commissioner.

What is less clear is who is making decisions and then communicating out broadly on things like annual performance targets or the final decisions on index measures and weighting.

Massachusetts could strengthen its plan by articulating a more refined monitoring approach.

The state’s theory of improvement is based on the belief that it will set targets for each school and district that requires it to “stretch and continually improve,” but it offers little insight as to how any larger continuous improvement framework will be defined, operated, or resourced.