It is owned by the Wilzig brothers–Alan, 33, and Ivan, 43–two wild and crazy bankers who, in the last few years, have become the subject of juicy tabloid items. But behind the funny little gossip items (girls from Scores stripping by the pool, etc.) lies something more: While the Wilzig brothers have been enjoying themselves, something has nagged at them–the fact that their father, an Auschwitz survivor and self-made multimillionaire, Siggi Wilzig, 72, is not all that impressed. It’s hard, after all, to impress a man who survived the death camps and a death march and then managed to turn himself into a wealthy banker in another country.

When the Wilzig brothers became boldface names in the gossip columns for their wild parties, there was some trouble in the family over the Hamptons castle. “My father was disgusted,” said Alan, the younger brother. “He was inches away from blowing up the house with a bazooka.”

During the week, the brothers live in separate apartments in the City Spire building on West 56th Street. They went in on the castle in the seaside town of Watermill, L.I., partly as an investment. Alan is the one who got it built, with the help of his longtime girlfriend, Karin Koenig. “I told my mother, my sister and my girlfriend, ‘If you see something you like, buy it. We’ll find a place for it,'” Alan said. With so many hands decorating the place, the décor ended up eclectic, kind of like an everything bagel.

Alan has mixed feelings about what the castle has become. “People didn’t understand,” he said. “I just wanted to do something to make my father proud.” He laid blame for all the commotion at the castle on Ivan’s “laissez-faire” attitude: “My brother felt bad that the people who worked in the clubs never got to come over because they were working,” Alan said. “So he said if they want to come over at 3:45, O.K. All of a sudden, people are ringing the buzzer in the middle of the night.”

As the owner of the Trustcompany Bank of New Jersey, a founder of the Holocaust Museum in Washington, D.C., and the first Holocaust survivor to lecture the cadets at West Point, Siggi Wilzig is not interested in spending much time there. The whole notion of a summer home doesn’t interest him, in fact. If he takes a vacation, he goes to Kutsher’s Resort Hotel and Country Club in the Catskills.

Still, he admires certain things about the castle. “The quality is excellent,” Siggi Wilzig said. “It’s a credit to Alan, because Ivan likes design, but Alan built it. It is built like a fortress. Someone doesn’t have to worry in a bad storm or hurricane. But I am a simple guy. I don’t drive two and a half hours to a place.”

He has made only two visits to his sons’ castle. The first time, he climbed a ladder on the property in his business shoes. “I wanted to make sure you could see the ocean over the tops of the trees,” Mr. Wilzig said. After inspecting the place, he told Alan to change four things. “And I made the changes,” Alan said. “He saw the problems on just a quick walk-through.”

In the cluster of buildings known as Auschwitz, Siggi Wilzig could not have imagined the Hamptons. There, he watched as 59 members of his family were killed over a three-year period. When he arrived in America, he had nothing and knew no one. His credo is carved in marble above a fireplace in one of his offices at the Trustcompany bank: “Free men who forget their bitter past do not deserve a bright future.”

The 12-story Trustcompany building is the only moderately tall structure in this part of Jersey City. Mr. Wilzig greeted me in its executive offices and led me to a modest dining room. He’s 5 feet 5 inches tall, with a shock of coarse fly-away hair sprouting out of his scalp. “I remember every single day I was there,” he said. His eyes were black and deep.

The Nazis sent the Wilzig family to Auschwitz in 1943. Siggi was 16 years old at the time and already hardened from two years of forced labor. Just after he reached Auschwitz, his brother was beaten to death. His mother was murdered immediately upon her arrival at the death camp. His father was killed in front of him. Siggi Wilzig knows the date: April 8, 1943. Two other family members were killed two days before the liberation of Auschwitz–and buried two days after the war was over.

Siggi Wilzig was nearly sent to the gas chamber many times: “I went through 18 to 20 selections,” he said. “Standing naked with a bundle under my arms. But I tell you this: I never thought I was going to die.… It was such a will to survive, it would have been impossible to me not to survive!”

His last four months in Auschwitz were spent toiling in a laundry. There, the workers were charged with washing the clothes of murdered Jews–clothes that were later redistributed to the Germans. Toward the war’s end, he came upon clothing belonging to his mother’s relatives. “They all fled to Holland,” he said. “They got caught and came with the last transport in the beginning of August, two months after D-Day, from Holland to Auschwitz–and I found the laundry marks on the clothes.”

And something else, which he will include in the memoir he’s working on now: “In 1943 and ’44, they took blood from the stinking Jewish people and gave it to the wounded soldiers on the Russian front. No one ever recorded that. I did it twice. They gave me an extra piece of bread and one time a bone. Like special soup from horse meat.” He paused. “I never told the children that.”

His tenorish voice rose to shrillness from time to time. “Are you getting this down ?” he cried. “Is that thing taping ?” Mr. Wilzig’s forearm bears the tattooed number the Nazis branded him with, 104732, in addition to a triangle denoting his nationality. He was asked to tell his worst memory from those years.

“Night shooting of people, Dutch and Greek Jews,” he said, almost in an incantation. “Raining all day and having no clothes. They don’t teach you this at Harvard: Do you sleep in wet clothes so that the warmth of the body dries them, or do you take it off and freeze to death?”

And an odd, stray memory: “There was a wedding in Auschwitz–did you know that? In a bordello. A Spanish girl was engaged to a German socialist that was in the army, they emptied the bordello and he got married there. No one reported it.”

In January 1945, he left Auschwitz on a forced death march. He was rescued on May 8, 1945, in Mauthausen, Austria, by the U.S. Army.

The first years in America were not so easy: After emigrating in 1947, he worked as a bow-tie presser, then sold school notebooks to reluctant university bookstore managers. “I was the original Death of a Salesman,” he said. “My fingers got arthritis from holding the cases.” In 1954, he married Naomi Sisselman, the daughter of a New Jersey real estate mogul. Her parents did not approve of Mr. Wilzig–so the couple eloped to New York.

In the early 60’s, he started playing the stock market. He saw something in Canadian oil and gas stocks that looked undervalued. He found one, Wilshire Oil, particularly attractive. “Wilshire Oil was half-American, half-Canadian,” he said. “I was so happy when I bought it. But when I came home my wife said to me: ‘You bought more stock?’ And I’d say, ‘They must have had an open order from me.’ I lied.”

So began his slow takeover of the company. His days as a traveling salesman were over. But even after he had managed to acquire a major share, he was not accepted by the people at Wilshire Oil. “For five years,” he said, “they wouldn’t give me a directorship. I started a proxy fight and they wound up giving me four seats on the board.”

Picture Tevye as J.R. Ewing–that’s what Siggi Wilzig became. In 1968, he set his sights on the Trust Company of New Jersey, as it was then called, as a way to offset the risks in oil exploration. The bank had been founded by a military man with a German-sounding name. Mr. Wilzig heard stories about what went on at the Trust Company during the war: “When two officers in this bank heard the Nazis took Paris, they played Nazi songs and danced in the main branch,” Mr. Wilzig said. “That’s how German it was here.”

But he persevered, much the way he had with the oil company. “Half the board called me behind my back little Jew bastard, and I didn’t fire one when the time came,” he said. “I’m not a fighter. I did all my fighting in Auschwitz.”

Over 30 years, Mr. Wilzig took the Trust Company from a $170 million business to just under $3 billion. He gained a reputation for working 14-hour days and for knowing all his customers. Forced to spin the oil company off to comply with the Bank Holding Company Act of 1980, Mr. Wilzig and his family cashed in: For every 1,000 shares of Wilshire stock, shareholders received 111 shares of the bank. Mr. Wilzig officially became a $75,000-a-year consultant to the oil company, while remaining in control of the bank. His daughter Sherry, a Brown graduate, now serves as the titular president of Wilshire.

Until recently, he never used the president’s office at the bank, preferring instead to roam from office to office, conducting his business on the go. “I didn’t like the confinement of being behind a desk,” he said.

His sons remember having to keep out of crowds when they went out with their father. “Even something happy, like a movie, was a problem,” Alan said. The Holocaust was with them during their upbringing in Clifton, N.J. Ivan, who grew up in the late 60’s and early 70’s, said: “My friends used to go crazy when they’d come over–every single TV was tuned to The World at War .”

“We could be on a trip to Colonial Williamsburg,” said Alan, “and my mother would say, ‘Don’t buy a pewter cup. No metal cups in the house for your father.'”

But both sons joined the family business.

“From the time I was 8, sitting on my dad’s knee, I knew I wanted to be a banker and work with my dad,” said Alan, who is a graduate of the University of Pennsylvania’s Wharton School, undergraduate division. “I have a passion for it.”

After 15 years with the bank, Ivan seems less committed. Also a University of Pennsylvania graduate, he once planned on becoming a psychologist, but got a law degree from Benjamin N. Cardozo School of Law at Yeshiva University because his father thought it was more valuable. Still, Ivan cut a deal recently–to open 40 new bank branches in A.&P. supermarkets–that impressed his father. “It’s the most important event in the bank’s history,” Siggi Wilzig said.

Both brothers said they will probably not marry as long as their father lives. Siggi Wilzig’s requirement for purely Jewish wives is more stringent than that of the Council of Orthodox Rabbis. “My father can be very a powerful and demanding figure to contend with,” said Alan. “He rules with an iron hand.” Indeed, Siggi Wilzig has little flexibility, more than 50 years after the war, about certain issues. He explained why he does not drive a Mercedes: “It’s not because of the quality of the Mercedes truck. Because it never broke down once when it took the children to the gas chambers!”

After the bad press in the gossip columns for the Wilzig brothers (and, no, those girls by the pool weren’t Scores dancers, Alan claimed, but “guests or dates of our guests–and they were sunbathing topless, not naked or in G-strings”), Alan is trying to improve his image, for the sake of the bank and the family name. He recently appeared on the front page of a Jersey City Coptic newspaper with Pope Shenouda III–a nice Jewish boy from New Jersey receiving a large silver cross from the Orthodox Copts’ bearded patriarch. He’s also the bank’s point man in a $7 million restoration of Jersey City’s Journal Square.

Now, if he can only get that castle sold before the weather gets warm and the trouble starts all over again …