But Ma was just as happy — perhaps even happier — when he was barely making any money right out of college.

After graduating in 1988, Ma worked as an English teacher at a local university in his hometown of Hangzhou, China. He only made $12 a month, according to the documentary about his life called "Crocodile in the Yangtze."

When speaking at a luncheon with the Economics Club of New York on Tuesday, Ma referred to this period as the "best life I had."

When you don't have much money, you know how to spend it, Ma explained. But once you become a billionaire, you have a lot of responsibility.

"If you have less than $1 million, you know how to spend the money," he said during Tuesday's speech. " [At] $1 billion, that's not your money...The money I have today is a responsibility. It's the trust of people on me."

Ma says he feels a need to spend his money "on behalf of the society."

"I spend it our way," he said. "It's a trust."

This isn't the first time Ma has spoken about the burden of being a billionaire. When speaking at a panel at the Clinton Global Initiative in New York, he referred to his days as an English teacher as "fantastic," according to CNN Money. He said anyone with $1 million is "lucky," but when you reach $10 million, "you've got troubles."