He urged upon the Coal & Navigation Company the advantage of a steam railway as a coal carrier, but the project was not then considered feasible. In 1851, the majority of the stock of the Delaware, Lehigh, Schuylkill & Susquehanna Railroad Company (incorporated in 1846), which became the Lehigh Valley Railroad Company in January 1853, came into his control, and between November 1852 and September 1855 a railway line was built for the Company, largely by Packer's personal credit, from Mauch Chunk to Easton.[1] He built railways connecting the main line with coal mines in Luzerne and Schuylkill counties; and he planned and built the extension (completed in 1868) of the line into the Susquehanna Valley and thence into New York state to connect at Waverly with the Erie railway. Among his clerks and associates during this period was future businessman and soldier George Washington Helme.

Packer also took an active part in politics. In 1842-1843, he was a member of the Pennsylvania House of Representatives. In 1843-1844, he was county judge of Carbon County under Governor David R. Porter. He served two terms as a Democratic member of the U.S. House of Representatives beginning in 1853. Packer made an unsuccessful bid for the Democratic Party's Presidential nomination in 1868. He got the party's nod for the 1869 Pennsylvania Governor's race, but lost the campaign to John W. Geary by 4,596 votes, one of the closest statewide races in Pennsylvania history.

In 1865, he gave $500,000 and 60 acres (243,000 m²), later increased to 115 acres (465,000 m²) in Bethlehem, Pennsylvania, for a technical school for the professions represented in the development of the Lehigh Valley. Lehigh University was chartered in 1866, and its main building, Packer Hall, was completed in 1869. With Packer's generous gift, Lehigh was consequently able to offer education tuition free for 20 years, 1871 - 1891, before the economic troubles of the 1890's forced the University to reverse this policy.

Mr. Packer's residence, the Asa Packer Mansion became a museum, opened for tours in 1956, and was named a National Historic Landmark in 1985. Asa Packer was a member of St. Mark's Episcopal Church and contributed large amounts of money to this beautiful Gothic Revival Church that is located in downtown Jim Thorpe. St. Mark's was declared a National Historic Landmark in 1987. There is also an elementary school in Bethlehem, Pennsylvania, named after Asa Packer.