With a large family to support, Joe Jackson surrendered his dreams of becoming a boxer and secured a job as a crane operator for U.S. Steel.

He and his brother Luther also formed a band in the mid-1950s called The Falcons, intent on booking gigs for extra money.

The band only lasted a few years, but Jackson had developed an ear for music and believed he had found some talent in his children. He formed The Jackson Brothers in 1963 -- with sons Tito, Jackie and Jermaine -- and began entering them in local talent shows. With the addition of Marlon and Michael, The Jackson 5 was born in 1966. Two years later, they signed with Motown Records.

They went on to become one of the most successful R&B groups in history, with their father initially acting as their manager.

At the height of their stardom, The Jackson 5 sold millions of records and had their own CBS variety show.

"Joseph's role as manager dwindled however as Motown CEO Berry Gordy began to take more charge on his act, a role that reverted back to Joseph when he began managing the entire family for performances in Las Vegas," according to Jackson's official site. "Joseph also helped his sons seal a deal with CBS after leaving Motown."

The success of The Jackson 5 led to Michael Jackson going solo, becoming such a major star that he was later dubbed the King of Pop. Youngest daughter Janet also became a hugely successful recording artist.

The elder Jackson managed daughters Rebbie, La Toya, and Janet in the early 1980s until they, like their brothers before, struck out on their own.

Joe Jackson was criticized at times for being a harsh task master. His children told stories about their father being hard on them growing up.

In 2013 interview with CNN, Jackson was asked about his daughter Janet's complaint that the children were not allowed to call him "Dad," instead referring to him as "Joe."

"You had all those kids running hollering around," Jackson said. "They're hollering, 'Dad, Dad, Dad,' you know, and it gets to be -- it sounds kind of funny to me. But I didn't care too much about what they called me, just as long as they (were) able to listen to me and what I had to tell them, you know, in order to make their lives successful. This was the main thing."

Jackson admitted that he disciplined his children physically but said he had no regrets.

Joe Jackson on physically disciplining his kids: 'I'm glad I was tough'

"I'm glad I was tough, because look what I came out with," he said. "I came out with some kids that everybody loved all over the world. And they treated everybody right."

Jackson also weathered some controversy after his wife documented his alleged extramarital affairs in her book, "My Family, The Jacksons."

The couple split more than once and lived apart for decades, but they reportedly never divorced.

The couple presented a united front when their son Michael died in 2009 from an overdose of Propofol.

The elder Jackson told CNN his son had tried to reach him before his death, but they didn't connect.

"He says, 'Call my father.' This was before he passed. 'He would know how to get me out of this,'" Joe Jackson said. "But they didn't get in touch with me. They said they couldn't find me, but I was right there."