Nat Holman came a long way from his days growing up on the Lower East Side. He achieved great recognition as player, coach and author. But along the way he had obstacles to overcome. The basketball scandal of 1951 brought him and the City College basketball team into the middle of controversy. The scandal—four players shaved points to accommodate a bookmaker's odds—broke less than a year after Holman led the College to its greatest heights—a double championship (the only team ever to win both the NCAA and NIT post-season tournaments in the same year). Holman was at first under suspicion. His reputation as a master of basketball was held against him. How could he have not seen what was going on?

As suspicion of Holman having knowledge of point shaving seemed to fade, Dr. Buell Gallagher, a newly appointed president of City College, took a new interest in it. President Gallagher suspended Holman for "conduct unbecoming a teacher" until an investigation could be completed. Holman was deeply hurt by the scandal. He was offered early retirement but decided not to take it. The investigation was completed a year later and proved Holman had no knowledge of the point shaving, but the damage was done. Two players served minor jail time and the other two received suspended sentences. Holman's reputation suffered. Articles in the press accused Holman of being of being "ambitious" and "vain" and contributed to an atmosphere of public hostility and doubt. Holman lost a year of coaching, only to return a year later to coach four more years, 1952-1954 and 1958-1960, before retiring to become an ambassador for the game of basketball.

The most devastating blow was to City College. The teams were banned from playing at Madison Square Garden and were moved from Division I to Division III. The impact of these penalties prevented City College from getting the exposure necessary to compete for top student athletes throughout the country. However, the greatest impact was the loss of revenue from not being able to play at Madison Square Garden. Unable to compete at MSG and generate the revenue that helped to pay for other sports at City College, the Athletics Department had to discontinue a majority of the sports programs. In 1978 City College returned to MSG to play in an exhibition game against long time rival NYU. Today City College is still recovering, and basketball has been revived.