Last week, I reported on the convoluted and sometimes baffling war over the fate of the Westlake Subdivision Improvement Association, the homeowners association (HOA) for 6,500 homes in Daly City. At the heart of the struggle was U. Deutsch, an elderly German-born woman who has spent a good portion of the last 30 years working to dismantle an organization she considers fraudulent and unjust.

Although the facts were on her side, I couldn't help but wondering whether Deutsch -- after carrying the torch for so long, engaging in countless lawsuits and spending thousands of dollars of her own money to rid her neighborhood of an intrusive HOA -- had not become a little, well, too focused. This is the temptation with such stories about homeowners-association wars: that one will interpret them as tales of personalities (be they high minded or mean spirited), of people who have a little too much time on their hands.

But Evan McKenzie, author of "Privatopia: Homeowner Associations and the Rise of Residential Private Government," has heard too many such stories to reduce them to quaint battles between obsessed busybodies. As an attorney who spent six years representing HOAs in California, he had a privileged insider's view of a burgeoning and little-understood industry. And, as an associate professor of political science at the University of Illinois at Chicago, he was one of the first academics to grapple with how the rise of such governing organizations is reshaping our neighborhoods.

His scholarly book, which became the equivalent of "Silent Spring" for homeowners disgruntled with their HOAs, attracted a popular following. Since its publication in 1996, he's received scores of letters from homeowners driven to the brink. "You think, 'Wow, these people are eccentric,'" he says. "They get wrapped up in some petty struggle. It drives them nuts. But it's these issues that make people crazy."

What precisely are "these issues"?

Freedom of expression, the right to privacy, the right to keep the home you worked a lifetime to buy. In other words, the loss of that prize most of these people thought they had already achieved: the American dream.

Now that property values have risen so much that many hardworking families can't afford a traditional domicile, they buy the closest approximation they can pay for: a condo, a townhouse or a "single-family home" in a new "planned development." If the complex has any shared property, such as landscaped grounds, a pool or even a small strip of trees, then it's by definition a common-interest development (CID).

Sometimes, these homes look no different from the homes down the street, but they exist in a different legal framework: Instead of being governed by local county or city laws enforced by government employees, they follow the laws of their "Covenants, Conditions & Restrictions," which are enforced by their HOAs. Thus, the association may write laws about the sort of flowers you can plant in your front lawn and whether you can remodel your house. According to McKenzie, the rise of HOA-governed CIDs is changing private-property rights, civil rights and the responsibilities of local government everywhere.

What's more ominous, now that more than 80 percent of new housing in the country is being built within CIDs, is the professional industries that have grown around them -- complete with their own lobbying groups -- eager to make sure government oversight remains at a minimum.

We in the Bay Area like to think of ourselves as immune to the afflictions that trouble the rest of America, but McKenzie says that even here, the vast majority of new housing are CIDs. "Anywhere where land prices are high, there will be lots of CIDs," he says. The gated communities of Harbor Bay Island in Alameda, the live/work lofts of San Francisco and the tract-home prairies of Tracy: All are common-interest developments.

To help me understand how homeowners associations have evolved, McKenzie offers a thumbnail history. "The original reason for all HOAs was to enforce race-restrictive covenants," he says, explaining that such organizations predate the invention of the CID, with its shared responsibility to maintain property. "But in the 1960s and '70s, as land prices were getting more expensive, developers started creating [CIDs] in order to create more density and therefore increase profitability."

In devising CIDs, developers managed to build more houses on less land -- avoiding local density codes -- through creation of "private communities" with their own roads and utilities infrastructure. "They build a shared infrastructure with common utility lines so the physical infrastructure is a lot cheaper," explains McKenzie. "Then they build private streets, which are very narrow and not as deep, so they're cheaper streets. So they need a lot of repairs."

But who will take care of the infrastructure and the roads? Not the city, of course, because these are private communities. Instead, the HOA, run by volunteer homeowners with help from profession property managers and lawyers, must levy assessments from homeowners to pay for the cost of maintaining the property. With such an arrangement, developers get the best of both worlds: an escape from city planning ordinances that restrict their profits, and an entity that inherits the results of their cost cutting.

Sometimes, what looks like a developer's enlightened benevolence reveals itself to be egregious self-interest. Developers maximize useable land, then donate "open space" to the HOA to maintain, marketing it as one of the benefits to the community. Such open space can turn out to be more of a liability than an asset, however. In a famous case in Hillsborough, "parkland" turned out to be an unstable hillside, requiring expensive maintenance to prevent it from destroying homes. And McKenzie recalls one case in which the "open space" turned out to be, as he describes it, "almost a Superfund site."

With these conditions, why do Americans keep flocking to CIDs? In part, it's because if you want a new home, there is little else to choose from. But it's also because these developments offer a taste of the American dream that's ideal for a fraction of the cost of a discrete single-family home. "Instead of everyone getting a big yard with a swimming pool, developers figured out how to do the same thing," explains McKenzie. "They build amenity-rich developments with golf courses, swimming pools, game rooms -- all exclusively for the homeowners."

"They can still sell exclusion without being formally racially exclusive," he says. "It's become an important marketing feature for a very lucrative form of housing."

The problems in these private communities tend to erupt a few years down the line, though, after the developers have sold off the last of the houses and the HOA has taken over. Once the statute of limitations for suing the developer -- usually one to three years -- has expired, the association is on its own. Suddenly, roofs begin to leak, streets break down and the HOA realizes it needs to begin serious repairs.

"When the developers leave, the incoming association directors often find that they're 'underreserved'," says McKenzie. "The developers have almost always set the fees too low -- they set them low to sell homes -- so the HOA begins raising fees. And sometimes the assessments are huge -- $600 a month."

At other times, an expense is so big and so urgent that the HOA must levy a "special assessment" to pay for a new roof or a rotting deck. "So the board votes for a special assessment of, say, $10,000 [per homeowner]. And if you can't pay, they'll slap a lien on your home. The HOA has a fiduciary duty to maintain the property."

Why would cities allow developers to circumvent planning ordinances and create communities that may not be up to the task of maintaining their own infrastructure?

"The cities get something for nothing," says McKenzie. "They still get the property taxes without building or maintaining the infrastructure, the pipes, the streets, the parks. Essentially, it's a form of double taxation." Although New Jersey recently passed a law compelling cities to provide homeowners in private communities with a snow-removal and leaf-removal rebate (since they don't use those city services), most states have been slow to enact legislation acknowledging the double-taxation issue.

In framing the dangers of CIDs, McKenzie is careful not to vilify HOA directors. "Most of them are salt-of-the-earth people. They do it because they know it needs to be done," he says. "But all you need is one or two power-hungry control freaks on a board to create a lot of problems." Such individuals can use a HOA for a number of ulterior reasons: monetary gains (by providing business to close associates and then receiving kickbacks), political power (using their position as a launching pad for positions in city politics) or personal vendettas (suddenly, you have a supposedly valid reason to spy on your neighbors.)

He says that, unlike with city governments, which ostensibly must follow democratic laws of governance, HOAs are woefully unregulated. Sometimes the boards simply vote themselves into perpetual power, since they can prevent opponents from voting or running for election by suggesting that the upstarts are not in good standing with the HOA. Or, as McKenzie put it, "These elections can make Broward County look like the epitome of fair voting."

Even meeting protocol is fraught with improprieties. "I've seen hilarious things," says McKenzie, such as "where the meeting minutes say that the director thanked so and so for his comments, but what actually happened was that the director said, 'Sit down and shut up.' They are banana republics."

Even when an HOA is well run and filled with great people, adds McKenzie, "it's only one election away from a disaster."

Sometimes, living in one of these communities means losing certain free-speech rights. Some HOAs prohibit putting political signs in your window or flying the flag. Others curtail privacy rights with ordinances about improper behavior. One book of HOA horror stories tells of an elderly woman being fined for kissing her elderly boyfriend in the car outside her home.

Such petty disputes over flag flying and senior necking can turn serious if the transgression collides with a vindictive HOA member or, worse, an avaricious lawyer. It can even lead to people losing their homes.

According to McKenzie, there is now an entire cottage industry of lawyers and property managers who feed off such liens and fees, offering their services to the generally inexperienced HOAs. As an HOA lawyer, McKenzie watched the industry grow from a small group of specialists to big business. Over the years, he saw that HOA law seemed to attract two kinds of lawyers.

"The first are those that offer general counsel," he says. "They're really good lawyers. The HOAs need the service, and it's a good service.

However, McKenzie adds, "the second type is the collections lawyers. They are brought in to collect unpaid assessments. They say to the HOA, 'I will represent you, and you don't have to pay me, just so long as I'm given a free hand in how I do my work.'"

These lawyers take a "collection-agency posture," he says, putting liens on property when homeowners are 10 days late paying an assessment. "Every letter has a price tag -- and if the homeowners don't pay, you slap them with a lien." Although the assessment is perhaps for only a few hundred dollars, the lien may total $5,000 or $10,000 -- and, in order to clear their title, the homeowners must pay not only the assessment but the lien as well. If they cannot do so, McKenzie says, they can lose their home through a form of foreclosure unmediated by the courts or any local government.

"These lawyers are so rapacious that it's just shocking," says McKenzie, adding that no laws govern their fees -- they can basically charge whatever they want. "It's up to the homeowner to file an action with the court, and if you don't file a lawsuit, you are out of your house before you can say boo," he adds.

The worst part about the whole process, says McKenzie, it that it's legal, a fully institutionalized practice: "The bar even offers workshops on the process."

California's state government is reviewing the laws governing CIDs, and McKenzie remains hopeful that states will begin legislating reforms, regulating such issues as how HOAs are governed and how much their lawyers can be paid. But, already, there is a powerful trade association, the Community Association Institute, which tends to resist laws that increase government oversight.

Government might also help prevent some of the limitations on civil rights with a simple declaration. McKenzie says, "We need a more general statement from the states saying that you have basic rights as a homeowner -- a bill of rights for owners -- that even if we have privatized our neighborhoods, we can't privatize our constitutional rights."

But, for now, many homeowners continue to buy new homes in CIDs without knowing hide nor hair about exactly what they are gaining and what they are losing.

"A lot of times, when people buy homes [with HOAs], they don't know what they are signing away," says McKenzie. "It's all justified by contract. You signed, so you consented to it. But people have not really meaningfully consented to be governed by HOAs. Most haven't read the covenants -- they don't know what they mean."

Carol Lloyd is currently at work on a book about Bay Area real estate. She teaches a class on buying your first home in the Bay Area, and another class based on her best-selling career counseling book for creative people, "Creating a Life Worth Living." For more information, email her at surreal@sfgate.com.