Pope Francis has hit out at prejudice against Gypsies, urging people to stem their suspicion and calling on authorities to help those "at the margins of society" find greater means of integration.

In an address at the start of a five-day meeting in the Vatican on the pastoral care of migrants and itinerant people, the pope noted that Gypsies were sometimes seen with hostility and suspicion and were "barely involved in local political, economic or social dynamics".

In off-the-cuff remarks, he added that he had seen for himself the way in which parts of Italian society regarded them.

"I remember many occasions here in Rome where I would get on the bus and some Gypsies would get on too, and the driver would say: 'Keep an eye on your wallets'. This is scorn; it might be true, but it's scorn," he said, according to quotes reported by the Ansa news agency.

Hostility towards Roma remains widespread in Italy. In March, a bakery in Rome was criticised for posting a sign in its window warning that access was "strictly forbidden to Gypsies".

In his remarks, Francis acknowledged that the situation presented "a complex reality". But he emphasised social factors which he said led to many Gypsies becoming susceptible to "new forms of slavery" in society such as forced begging.

"Among the factors which create situations of poverty for a section of the population in today's society, we can identify the lack of educational structures, both cultural and professional; the fraught access to healthcare, discrimination in the workplace, and a lack of dignified housing," he said. "If these social scourges can affect everyone indiscriminately, the weakest groups are those who most easily fall victim to new forms of slavery. It is the least protected people who fall into the trap of exploitation, forced begging and various forms of abuse."

Gypsies, Roma and Travellers were among the most vulnerable, he added.

"Regarding the situation of Gypsies across the world, it is ever more necessary to elaborate new approaches in the civil, cultural and social sphere, as indeed in the church's pastoral strategy, to take on the challenges that emerge from modern forms of persecution, oppression and, sometimes, slavery too," Francis said.

The meeting in the Vatican is being organised by the Pontifical Council for the Pastoral Care of Migrants and Itinerant People.

Zeljko Jovanovic, director of the Open Society Foundations' Roma initiative office, said: "The pope's statements challenging the exclusion of Roma are hugely important, particularly in Italy where we see the EU and state-sponsored segregation forcing Roma to live in special camps – the worst in Europe – and an acceptance of anti-Roma public discourse and sometimes violence which usually goes unpunished.

"Italian society treats Roma today in the same way Italians were treated a century ago in America, as 'lazy beggars and violent criminals'. In July, Italy will take on the rotating presidency of the European Union and this offers an opportunity for Italy to show member countries that it's possible to enact meaningful policies on improving the life of Roma communities."