From the window in Dave Hart’s corner office, the University of Tennessee athletic director can see Neyland Stadium above all else, a fortress along the Tennessee River that instantly identifies the Volunteers as a member of college football’s elite.

Its enormity is a testament to the sport’s incredible growth during the past two decades and the power of the Tennessee brand. It also effectively conceals an athletic department that built enormous debt while trying to maintain its place among the richest and most powerful football programs in the Southeastern Conference.

Now, after staggering to losing football seasons in four of the last five years and seeing attendance drop to levels last seen in the 1970s, the Vols find themselves mired in more than $200 million of debt, the most in the SEC, with reserves of just $1.95 million, the least in the conference.

The athletic department spends a startling $21 million a year on debt payments, $13.5 million of which comes from the school’s stressed $99.5 million athletic budget and the rest from donations.

It’s an ugly financial picture for one of the nation’s strongest football brands, the kind of financial hole that SEC football giants aren’t supposed to be in, brought about by expensive coaching buyouts and costly improvements to Neyland Stadium and other athletic facilities, just as the losing seasons set in.

“We’ve got to get football healthy,” Hart said from his office in the new $50 million Brenda Lawson Athletic Center, just weeks after hiring Butch Jones from Cincinnati to be the Vols’ fourth football coach in the last six years. “That’s our economic engine. When that program is successful, everybody wins.”

Hart speaks with an urgency about the task at hand. Formerly the No. 2 at Alabama and the longtime AD at Florida State, he took the job in Knoxville 18 months ago fully aware of the upside and the pitfalls of big-time college football. After replacing former AD Mike Hamilton, who resigned under pressure in June 2011, Hart began with a lengthy evaluation process that revealed mounting debt, much more than he had realized.

“Our financial position was probably the biggest surprise of the assessment,” said Hart, bucking the perception that Tennessee athletics is flush with cash. “We’ve got to be better stewards of our finances.”

In his first year at Tennessee, as Hart began to fully understand the financial obstacles in his path, the athletic department posted a $3.98 million deficit, which reduced the reserves to a dangerously low level. So Hart put together the first steps of a plan that would help the Vols balance their annual budget and begin to climb out from under their crushing debt.

A cost-savings merger of the men’s and women’s programs was a logical step that will save $5 million this year, but it came at a cost. Seventeen jobs were eliminated. Tennessee used to be one of the few schools with completely separate departments for men and women.

Hart also negotiated with Chancellor Jimmy Cheek to at least temporarily halt an annual payment of $7 million from athletics to the university’s general fund that typically is made at the first of the fiscal year. Hart hopes to resume the annual transfer at some point, but preferably at the end of the fiscal year, not the beginning, so that athletics doesn’t start each year in a deficit.

Cheek pledged to keep that $7 million in the athletic department for the next three years to help the Vols get back on their feet.

“The bottom line is that, for SEC schools with extraordinary revenues, the profit margin is still very thin,” said Bill Carr, a former AD at Florida who now consults with athletic departments on strategy and searches. “Whether it’s Tennessee or any other school, if you’re not selling tickets at full bore and getting contributions to go with them, and that revenue tapers, it becomes very hard to put away the dollars you need. And then you have some undesired expenses like buyouts, and you can wind up in a negative position. The margin is razor thin for most schools.”

With those savings, Tennessee projects a balanced budget in 2012-13, which is a necessity for a program with just $1.95 million in reserves. Building reserves into the $50 million range or more is a priority, said Hart, who added that most SEC schools have reserves ranging from $50 million to $100 million.

Tennessee’s reserves were close to $30 million about five years ago, but they’ve been depleted by those $21 million in transfers back to the university over the last three years, and $11.4 million in buyouts to fired coaches in football, basketball and baseball, as well as administrators. Hamilton walked away in 2011 with a $1.335 million buyout.

None of that includes a $5 million buyout owed Derek Dooley, who was fired as Tennessee’s football coach in November, and $2 million to his assistants. That $7 million will have to be found in this year’s budget.

The Vols incur another expense that costs them $1.6 million a year in city and county taxes. It’s called the “amusement tax,” and it charges the school 5 percent on each ticket sold in football, men’s basketball and women’s basketball, in addition to the 9.25 percent state sales tax. The amusement tax, according to Tennessee athletics Chief Financial Officer Bill Myers, doesn’t pertain to concerts—just the Vols’ major sporting events. Cities and counties that are home to other Tennessee schools and the NFL Titans don’t impose such a tax, he said.

“There’s a perception that we’re sitting on a ton of money,” Myers said, “and that’s just not the case.”

The school is lobbying to have the decades-old tax repealed.

The financial cavalry is on the way, though, in the form of additional TV revenue from the SEC’s contracts with ESPN and CBS. Those deals average $205 million a year for the league and they are expected to jump to about $300 million annually when they are updated. The conference is renegotiating the contracts to account for the growth to 14 teams with the addition of Missouri and Texas A&M.

Hart said SEC athletic directors have not been told exactly how much new revenue that will mean, but projections are $5 million to $10 million a year for each school. Nowhere in the SEC will that be more welcome than at Tennessee.

“We need to stabilize,” Hart said.

He also must make sure the Vols don’t follow any of the patterns that put them in such a financial mess. While college football was booming throughout the 1990s and 2000s, Tennessee was one of the schools leading the charge.

Peyton Manning, a 1998 national title and a burning rivalry with Florida had Big Orange fans feverish for their team and turning out in record numbers on fall Saturdays. Multiple expansion projects increased Neyland Stadium’s capacity to nearly 105,000 as Tennessee fought Michigan and Penn State for attendance supremacy.

Much of the department’s debt came from a series of expansions to Neyland, followed by a series of improvements in the last seven years that actually reduced capacity and created more premium spaces. Built in three phases between 2006 and 2010, those changes cost more than $130 million. The athletic department continues to investigate ways to upgrade Neyland, whether it’s more premium spaces or more chair-back seats, measures aimed at improving the fan experience and driving more revenue, while reducing capacity. Tennessee also brought in IMG Learfield Ticket Solutions to help with sales beginning in 2011.

Facility improvements for basketball, softball and soccer added to the debt, which doesn’t have a state-mandated cap.

These days, however, Tennessee football has lapsed into irrelevance. Only a season-finale win over hapless Kentucky enabled the Vols to avoid a winless 2012 SEC season.

While no one in orange would ever call Neyland Stadium a giant albatross around the neck of the athletic department—that would be sacrilege here in the mountains of east Tennessee—it simply isn’t doing its job as one of the program’s chief sources of revenue.

Attendance dropped to an average of 89,965 in 2012, the lowest since 1979. A couple of late-season games against Troy and Kentucky reportedly drew about 60,000 actual fans in the stands, even though the announced attendance—or tickets sold—was a little more than 81,000.

“We’ve been smart about our projections,” Hart said, adding that the decline in 2012 football ticket revenue won’t significantly hurt the Vols’ efforts to balance the budget in 2012-13.

But he knows that Tennessee can’t stay on its current financial path.

“What you’re seeing is the cost of keeping up with the competition,” Carr said. “You’re trying to keep up and then you wind up paying coaches not to coach. That can add up in a hurry.”