While he was successfully defending Virginia from George McClellan's Yankees, John Magruder was also preparing a position in case his Army of the Peninsula had to fall back. Between two tributaries, one feeding to Queen's Creek and one to College Creek, Fort Magruder lay roughly in the center of a three mile strip of land. Chosen to defend the fort in a rear-guard action was Georgian James Longstreet. Opposing him were two divisions under the command of three of McClellan's corps commanders, Edwin Vose Sumner, Erasmus Keyes and Samuel Heinztelman. As overall commander, Sumner is frequently cited for his "limited capacity."

"Fighting Joe" Hooker neared the outer defenses of Fort Magruder in a cold, steady rain as General Longstreet's artillery opened fire. Hooker's men unlimbered and returned the fire, but the effectiveness of the cannon is debatable. Hooker waited for W. F. "Baldy" Smith, of Keyes' Fourth Corps, to join his line in an advance, but Sumner had ordered Smith to take no action. In the meantime, Longstreet, who had begun the day with just about two brigades of men at Fort Magruder, moved three more brigades forward.

Without Smith in support on his right, Hooker's flank was badly exposed. Using his field telegraph Hooker asked for more support, but Sumner responded that Hooker would have to wait for the other division from his own Third Corps to arrive. Although close, Phil Kearny was mired in the muddy roads of the Peninsula.

The long and loud battle sounds drew General Joseph E. Johnston back to the Confederate line. Longstreet offered Johnston command, but after viewing the situation, Johnston left, leaving the management of the battle in the Georgian's capable hands. "Baldy" Smith asked Sumner if he could push a brigade forward at Lee's Mill, where a slave had told Union officers that an earthen fort was unmanned. Sumner agreed and Smith ordered Winfield Scott Hancock forward after reinforcing his brigade.

Then something unexpected happened. Longstreet's men found Hooker's weak flank and attacked, driving the Yankees back. Hooker and his corps commander, Samuel Heintzelman, tried to rally his troops to no avail. Heintzelman even had a band unit begin playing in an attempt to strengthen the men's resolve.

As luck would have it, Phil Kearny had been moving his men towards the sound of battle, and as the Confederates broke though the Union lines, his forward echelons began arriving on the scene. He boldly rode ahead of his men, drawing Rebel fire, then returned to his line to direct his men towards the Rebels. He drove them back into some thick woods.

Hancock arrived on Longstreet's left about 3 p.m. after completing a two mile march led by Lieutenant George Armstrong Custer. Jubal Early, who was camped at William and Mary College, received orders to advance and push Hancock back. Early, with permission from Longstreet and his direct commander, Daniel Harvey Hill, tried to outflank Hancock's position but ended up in front of the Union line. He wheeled his men around and charged the Union position only to be bloodily repulsed.

As darkness fell over the battlefield George McClellan arrived. He had been busily supervising the loading of William B. Franklin's troops on transports in the York River, listening to the sounds of battle throughout the day. Throughout the day commanders had been telegraphing him requesting his presence. At 1:00 pm he left, taking 4 hours to make a ride that took his aide and hour and a half. McClellan pronounce the Battle of Williamsburg a great victory, mostly because of Hancock driving Early back and gave Hancock an enduring nickname following the battle: "Superb." Longstreet and Hill withdrew that night.