CAMP LEJEUNE, N.C. — Motion sensors and razor-wire coils ring the ammunition depot on this vast Marine Corps base. Sentries stand watch in the lobby of the headquarters complex. Military police officers patrol the barracks every few hours. But no building here boasts the defenses of the giant, government-run supermarket, whose bright, wide aisles are stocked with seemingly every brand of every food product available in America — Heinz ketchup, Oscar Mayer bacon, Lay’s chips — all sold at close to wholesale prices.

The cost of ordering the goods, filling the shelves and checking out customers is all borne by the American taxpayer.

Three summers ago, Richard V. Spencer, a retired investment banker who serves on a Pentagon advisory board, proposed shutting down the commissary at Camp Lejeune and every other domestic military base, a step that would save taxpayers about $1 billion a year.

He called several large retailers to see if they would be willing to take over the markets. None were, but Wal-Mart, which has stores within 10 miles of most U.S. bases, proposed offering equivalent discounts to troops, their spouses and their retired brethren. He figured other national chains would follow suit.

When the Defense Department bureaucracy that runs the commissaries learned of Spencer’s plan, it sounded an alarm among allies in industry and in Congress. A trade group whose mission is to represent companies that sell goods in military stores fired off a letter to Defense Secretary Robert M. Gates, warning him it would be “ill-advised” to make major changes. Senators and representatives dispatched similar missives. So did veterans groups. As the correspondence stacked up in his inbox, Gates summoned Spencer and other members of the Defense Business Board.

The commissary fight was an opening salvo — and a glimpse of the opposition — in the next big war facing the U.S. military: confronting the enormous cost of pay raises, benefits programs and other taxpayer-subsidized services, which have increased almost 90 percent since 2001 and have become the fastest-growing part of the Defense Department’s budget. With Pentagon spending set to shrink over the next five years, senior military leaders warn that they will be forced to order reductions in troop strength, training and equipment purchases if personnel expenditures are not curtailed.

“We are on an unsustainable course,” said Arnold Punaro, a retired Marine major general who chairs the Reserve Forces Policy Board. “We are trading off active-duty combat readiness to protect all of these benefits.”

Personnel costs now account for more than a quarter of all defense spending, which consumes the largest single slice of the federal budget after Social Security. Although no part of the Pentagon’s allotment is easy to cut — it is impossible to eliminate a weapons system or close a base without running into a welter of opposition — military pay and benefits will be among the most difficult to scale back, pitting growing demands for fiscal austerity against promises to support those who serve to defend the country.

In an era when private employers are reducing health care and pensions, the military continues to offer generous retirement benefits, including to service members who have never spent a day in combat. For troops who remain in uniform for 20 years or more, the military provides an annual pension immediately upon retirement — even if the retiree is 38 years old — equivalent to at least half of their final-years salary. Enrollment for an entire family in a military health-care plan that operates much like a private health maintenance organization will cost a retiree just $539 this year, about one-ninth of what the average non-military family will pay out of pocket in HMO premiums.

Those on active-duty also have bucked national trends. Over the past decade, military salaries have grown at a faster rate than those of civilian workers. The average enlisted soldier now earns more than 90 percent of Americans who have less than two years of college. Most Army captains — the third-most-junior rank of officer — will take home more than $90,000 this year.

That compensation does not take into account a raft of other services subsidized by taxpayers: commissaries, child care, schools on domestic bases, and morale and recreation programs. The tab for those will reach about $5 billion this year.

Many active-duty personnel and retirees contend that their compensation and benefits are well-deserved, given the risks and strain associated with a military career. “We’re not overpaying these folks by any measure. We’re not giving them a sweetheart deal on health care,” said Norb Ryan, a retired vice admiral who runs the Military Officers Association of America, a 380,000-member organization that lobbies on behalf of active and retired personnel. “We owe it to them because they’ve put in decades of extraordinary sacrifice on behalf of our nation.”

But military leaders argue that they have no choice. Leon E. Panetta, who served as defense secretary until February, warned last year that if the growth in personnel costs is not brought under control, “then we won’t be able to afford the training and equipment our troops need in order to succeed on the battlefield.”

At current trends, the military’s health-care spending will grow from $51 billion this year to $95 billion in 2030, according to the Congressional Budget Office. The military’s retirement system costs the federal government more than $100 billion a year. In two decades, the annual sum is projected to reach $217 billion.

The benefits crisis is the result of more than shrinking budgets. Retirees are living longer and health care is becoming more expensive. But the biggest reason, according to military leaders and independent analysts, is Congress, which has sweetened pay and benefits packages over the past decade, often against the Pentagon’s wishes.

“Nobody up here wants to be tarred as anti-military, and they don’t want to say no to the veterans groups,” said a senior aide to a Republican representative on the House Armed Services Committee who spoke on the condition of anonymity to discuss the subject candidly. “Those guys are organized — and they vote.”

The commissary proposal, as envisaged by Spencer, actually would have provided a greater benefit to retirees. They would have received the same commissary discount at large retailers closer to their homes, saving them from driving onto bases. “Nobody was going to have to pay more,” he said. “It was just a more efficient way to deliver the service.”

The story of how the plan was developed — and how it died — is instructive, Punaro said, because “it shows just how broken the system has become.”

From surplus to supermarket

When Spencer was a young Marine officer in the 1970s, commissaries sold basic staples on steel racks, much of it leftovers from the mess halls. Extra bags of pork chops. Loaves of sliced white bread. “Subsistence food,” he said.

The commissary at Lejeune now rivals the nicest American supermarkets. There’s filet mignon in the meat rack, wedges of Camembert in the cheese section and fresh baguettes in the bakery. Gone is the surplus feel. Every item is stocked in almost every flavor and size. There are two dozen varieties of Ocean Spray cranberry juice and 15 types of ketchup.

“This is crazy,” exclaimed Michael Bayer, a corporate strategist in Washington who served as chairman of the Defense Business Board from 2007 to 2011. The 20-member advisory board, composed of private-sector executives, is charged with providing advice on business and management matters to the defense secretary.

“Why,” Bayer mused aloud as he walked down the aisles of the Fort Myer commissary in Arlington, “is the federal government in the business of selling so many sizes of ketchup?”

The Army opened the first commissaries in the 1820s to provide food and other basic goods to troops in frontier outposts. Items were sold to soldiers and their families at the same price it cost the government to acquire them. For decades, the stores remained a vital means of sustenance for those stationed on rural bases.

Transformations in the military and civilian society over the following decades — including the expansion of large discount retailers near remote bases — obviated the original reason for commissaries. But political leaders shied away from changes. Military families didn’t want to lose their own supermarkets.

Over the past 12 years, Congress has repeatedly approved increases in the allotment for commissaries. This year’s subsidy — $1.4 billion — was authorized without a single question from legislators.

‘A tremendous deal’

The gentrification of the commissaries began in the name of efficiency. In 1990, the Pentagon decided to consolidate the separate grocery stores run by the Army, Air Force, Navy and Marines into a joint enterprise, reasoning that combining buying power and sharing overhead would reduce costs. A new bureaucracy, the Defense Commissary Agency, was created to run the operation.

The new management set out to transform the base stores into modern supermarkets, with the same ambience as a Safeway or a Giant. Pharmacy items were added, as were fresh baked goods and organic foods. Commissaries now stock an average of 11,000 items.

Because commissaries continue to sell goods at cost and everything is tax exempt, prices are, on average, about 30 percent less than at grocery stores off base. Customers do pay a 5 percent surcharge to fund the construction and renovation of stores, but none of that covers labor and overhead costs.

“It’s a tremendous deal,” said Jeffrey O’Brien, a retired Marine master gunnery sergeant, who works at Camp Lejeune as a contractor. “There’s no way you’ll find stuff in town this cheap.”

The discount attracts more than active-duty troops and their spouses. Legions of retirees, who also are eligible to shop, pour onto Lejeune and other bases, often on weekends, to fill up carts. Some drive more than two hours each way to avail themselves of the privilege.

Unlike with ordinary retailers, high volumes of traffic make the commissary system more costly to operate. The sprawling store on Fort Belvoir in Northern Virginia had more sales last year than any other commissary — in excess of $100 million — but because none of that offset operating costs, the facility was among the most expensive for the military.

Many bases also have their own department stores, known as post exchanges, which operate on a different business model. Exchanges on domestic bases set prices at levels that allow them to pay for labor and other costs without a government subsidy.

Because there is no pressure to turn a profit, commissaries have more personnel per square foot of store space than other major grocers. The commissary agency funds more than 14,000 full-time-equivalent employees at 247 commissaries worldwide. They are paid the government’s general schedule of wages, and they are provided health-care and retirement benefits.

In 2009, when the business board was asked by Gates to find ways for the Pentagon to operate more efficiently, Bayer, the group’s chairman, turned to Spencer. Bayer needed to say only one word: “commissaries.”

Bayer and Spencer agreed that the project would not start as an official Defense Business Board venture — at least not until they came up with a credible alternative. They regarded themselves as informal consultants, two retired businessmen, both of whom served in the military and then went on to successful careers in the private sector, working for free to help the government. All he wanted in return, Spencer often joked, was “stock options in the Pentagon.”

He limited his focus to the 179 commissaries that are in the United States. He had no interest in closing grocery stores on overseas bases, where personnel do not always have access to private shops that sell low-priced American products.

Spencer began by asking the commissary agency for financial statements. He didn’t get much. He sought the items it uses to support its 30 percent savings claim. He got no response.

What little that arrived stunned him. The agency’s antiquated financial systems, he learned, are not compliant with the federal government’s accounting standards. Its workforce was bloated compared with other retailers. It refused to collaborate with the massive Defense Logistics Agency, which buys food for mess halls, to obtain better prices from manufacturers.

Spencer also discovered that the agency’s annual subsidy did not include other hidden costs. Commissaries don’t have to pay rent. Security services, when needed, are provided by military police.

It didn’t take Spencer long to come to a basic conclusion: “Running a chain of grocery stores is not a core competency of the Defense Department.”

He thought about proposing that a private company be hired to run the stores. But when he called up several large national retailers, including Wal-Mart, Costco and three grocery chains, he got the same response. “We don’t want this,” he recalled being told. Too many employees, they said, and they would be unable to lure non-military customers onto access-controlled bases.

Two days later, an executive at Wal-Mart reached out to Spencer with a proposal: The massive retailer would provide active-duty troops, their spouses and retirees with a discount equivalent to the commissary’s prices on the same goods in its stores. The company would apply the discount by scanning a shopper’s military identification card at the cash register.

The Wal-Mart executive also sent Spencer data showing that the company has stores less than 10 miles from 70 percent of domestic commissaries. The other 30 percent didn’t worry him. Except for a few remote locations — such as the White Sands Missile Range in New Mexico — he figured other retailers close to those facilities would agree to establish similar cash-register discount programs. The federal government, after all, couldn’t just hand over an exclusive concession to Wal-Mart; other stores would have to be allowed to participate.

They would, Spencer figured, because offering discounts to service members and veterans would make them look patriotic. It also would make business sense: He calculated that the difference in prices between large discount stores and the commissaries is not that significant — far less than 30 percent difference with ordinary supermarkets — and whatever losses the stores took on groceries would be outweighed by other items military shoppers would buy.

“It was a win-win-win plan,” he said. “The government would save more than a billion dollars, these companies would make money and the people who shop at the commissaries would pay the same prices.”

But Spencer never had a chance to make that case to Gates. Before he could work out all of the details, defenders of the status quo pounced.

Battle lines are drawn

Spencer’s queries to the commissary agency for financial data aroused suspicion. Fearing attack, the agency discreetly got the word out to members of Congress whose districts are home to large numbers of veterans — and commissaries. They, in turn, sent letters to Gates, telling him to leave the commissaries alone.

The commissary agency also got help from the companies with which it does business. The American Logistics Association, a Washington-based trade group representing 250 firms that sell goods on base stores, wrote to Gates. Its letter was co-signed by the leaders of two other organizations with similar purposes, the Armed Forces Marketing Council and the Coalition of Military Distributors.

“Commissaries are core to the high standards of military quality of life,” the groups wrote. “They represent a reciprocal commitment from the taxpayer to the military people for their dedication and sacrifice.”

The organizations noted that 12,000 military family members are employed in commissaries and that the low food prices are regarded by many military families as their second-most-important benefit after health care. The groups also lavished praise on the commissary agency, saying that customer service in commissaries exceeds “all commercial supermarket rankings.”

Patrick B. Nixon, the president of the logistics association, contends that military personnel should not be forced to shop for groceries in civilian stores. “The military community is different from the rest of the nation,” Nixon said in an interview. “These stores are manned by people who are familiar with their situation, who appreciate what they’re doing for our country.”

For large food producers who support the logistics association — they include Unilever, Procter & Gamble and Kraft Foods Group — commissaries are an attractive market, Nixon said, because they do not require the firms to compete against house-label generic goods. The shelves are stocked only with brand-name items. And the aisles are filled not just with retirees but also with young families.

“A company can form brand loyalties to folks who are new to the shopping experience,” Nixon said.

Nixon, who is a registered lobbyist, served as the director of the commissary agency before joining the logistics association. His prior employment, he said, provided him with “first-hand knowledge of how valuable commissaries are.”

Although Spencer’s plan intrigued Gates’s senior advisers, the incoming correspondence revealed the contours of an arduous fight: They’d have to take on Congress, powerful veterans groups and some of the largest American companies.

“It was just going to be too hard,” said a former Gates adviser, who spoke on the condition of anonymity to discuss internal deliberations. “It would have subsumed our whole agenda.”

Spencer understood. “He had to pick his fights,” he said of Gates.

Gates retired the following summer. His successor, Panetta, had been in office for less than a month when he received a letter signed by four dozen members of Congress.

It put him on notice.

“We recognize military resale benefits have been targeted with cavalier rhetoric by budget-minded advocates questioning the need for the military to operate grocery and department stores,” they wrote. “We would ask you to look past the rhetoric.”

Looking for cuts elsewhere

This April, the new secretary of defense, Chuck Hagel, sent his proposed budget for the 2014 fiscal year to Congress. It envisions no substantial changes to the commissaries. Once again, the Pentagon wants $1.4 billion to subsidize them.

Defense officials have concluded that, for now, commissaries are worth the cost — or, at least, closing them is not worth the fight. The benefit, which the Pentagon values at $4,400 a year per family, “helps us retain the quality men and women we need,” said Frederick Vollrath, an assistant secretary of defense.

Instead, Hagel is seeking to constrain the growth of personnel costs by limiting this year’s pay raise for active-duty troops to 1 percent and increasing health-care fees for retirees. Military salaries were supposed to rise 1.8 percent this year because of a law that requires troop pay to keep pace with changes in private-sector compensation, but Pentagon leaders contend that is too generous at a time when the budget is shrinking.

Unlike the commissary discount, they don’t think limiting the pay increase will lead troops to quit.

“Slowing the growth isn’t going to adversely affect our ability to recruit and retain people,” said Robert F. Hale, the Pentagon’s comptroller.

Nor will raising health-care fees for retirees, he insisted. Hagel on Friday described the additional payments the Pentagon is seeking as “not very much” — less than $20 per year for the next five years to raise the annual fee to just 4 percent of a retiree’s yearly military pension.

In his first months on the job, Hagel has made addressing the growth of personnel costs a key element of his overall effort to reduce defense spending. Speaking earlier this year, he cited a warning from the former chief of naval operations, Adm. Gary Roughead, who said the Defense Department risked transforming from “an agency protecting the nation to an agency administering benefit programs, capable of buying only limited quantities of irrelevant and overpriced equipment.”

But Hagel, who faced a difficult confirmation battle, may find upcoming fights over pay and benefit reductions to be even tougher. Major veterans organizations — the American Legion, the Veterans of Foreign Wars and the Military Officers Association of America — all slammed his budget. “These proposals cross the line,” said Ryan, the officers’ association president.

Their opposition resonated on Capitol Hill. Last month, a House subcommittee rejected the 1 percent pay cap and the health-care fee increases. The panel’s chairman, Rep. Joe Wilson (R-S.C.), said he was “choosing to keep the faith with our armed forces, who have been serving in combat for nearly 12 years.”

Veterans organizations want the Defense Department to address inefficiencies within the military health system before raising fees.

This year’s Pentagon proposals were only an opening bid. Those on all sides of the issue agree that keeping personnel costs from overwhelming the overall budget will require far more aggressive action, but there is no consensus on where to trim costs.

To Spencer, reducing the commissary subsidy seemed the logical place to start.

“If we can’t fix something this straightforward,” he said, “how are we going to tackle everything else?”

Rajiv Chandrasekaran is a senior correspondent and associate editor. He has served as national editor and as bureau chief in Baghdad, Cairo and Southeast Asia.

Comments our editors find particularly useful or relevant are displayed in Top Comments, as are comments by users with these badges: . Replies to those posts appear here, as well as posts by staff writers.

To pause and restart automatic updates, click "Live" or "Paused". If paused, you'll be notified of the number of additional comments that have come in.

Comments our editors find particularly useful or relevant are displayed in Top Comments, as are comments by users with these badges: . Replies to those posts appear here, as well as posts by staff writers.