The number of non-EU immigrants to Spain fell in 2010, for the first time since the country's construction boom began luring record numbers of immigrants. The tide has turned as jobs dried up and many immigrants decided to go home.

Madrid's El Rastro market is where street vendors have been hawking wares for more than 500 years. Paul Shoyoye first came here 35 years ago, an immigrant from Nigeria. Back then he was a customer in a new country with a bright future. Now, he sells cheap clothing on a street corner, unable to find higher-paying work.

"This is the worst moment," said Shoyoye. "Before, there were jobs for everybody. People work, they enjoy that, and they're safe - as somebody who has worked. But now the economy is bad. I have to tell you, it's bad."

Immigrants like Shoyoye came to Spain for a better life, college degrees, health care.

"I worked in the central bank of Nigeria before, and then I worked at one of the biggest insurance (companies) in Africa," said Shoyoye. "I came here because I needed to study again, and then I studied at university here in Madrid. After that, there were no jobs. The way they pay, the salaries, are just too small. That's why I have to change my mind."

He estimates his profits are a third of what they were three years ago. And after 35 years in Spain, he's considering going back to Nigeria. He explains why some of his friends from Morocco and Ecuador are doing the same.

"Some of them are going back home, because they say they don't have jobs," said Shoyoye. "After the construction rise, they don't have many jobs and they're going back home. They return home because they have jobs there, or at least they're nearer to their families."

The trend Shoyoye describes is backed up by government figures. The Labor Department says fewer non-Europeans moved to Spain in 2010 than any year in the past decade. The biggest drop has been here in Madrid, with nearly 14 percent fewer immigrants last year alone.

Economist Josep Oliver, who co-authored Spain's Yearbook of Immigration, said part of the issue is how fast immigrants rushed to Spain in the boom years, when work was abundant. "From half a million in the mid-90s to close to 5 million in 2008. It's really the most important inflow of immigrants in any European country in the last decades," said Oliver.

Oliver said during that time, 40 percent of new jobs in Europe were in Spain - a country that represents only 10 percent of the continent's total economy.

On one hand, falling immigration numbers now will alleviate some of Spain's unemployment, which remains above 20 percent. But on the other hand, Spain has such a low birth rate that it needs foreign workers to keep its economy alive. That problem has led to some controversial austerity measures.

"The increasing number of years before retirement, it's really a hot potato," said Oliver. "The proposal of our government is to increase it from 65 to 67. That is because of our demography. At some moment in the next decade, our baby boomers will start to retire and then we will have problems to maintain our pension system, and generally speaking, our welfare system."

Oliver acknowledges that the opportunities for immigrants have dried up in Spain. But he thinks another immigration wave is inevitable.

"I can't imagine Spain in the next 30 years, seeing our population shrinking in absolute terms, as is predicted if we don't have another immigration wave, or if we don't have an increasing fertility rate, and that seems not possible," said Oliver. "It's difficult to imagine not only a pension system, but also a labor market, working properly without a correct amount of young workers."

Shoyoye said that if he decides to go home to Nigeria, he probably won't come back. After half of lifetime in Spain, he said it's a tough call.