In the heart of Jerusalem's ancient Old City, the Via Dolorosa – the route that Jesus took, burdened by a wooden cross, on the way to his public execution almost two millennia ago – straddles the busy El-Wad thoroughfare. This is where the Najib family encounters an obstruction almost every time one of them climbs the worn stone steps to their home of three generations.

The obstruction is an Israeli security guard with a weapon slung across his body, a baseball cap shadowing his face and an uncompromising attitude written on his features. He stands, according to members of the Palestinian family, in the middle of the gloomy staircase, his body almost filling the narrow passage to the upper floors.

He doesn't move, they say. Sometimes they edge past, eyes averted, wanting to avoid any confrontation. Sometimes they stand their ground, argue even: "Let us pass, this is our home, move out of our way." Such encounters may bring a fleeting triumph, but rarely do they last longer than a second or two. In reality, the Najibs fear that they and others like them are fighting a battle that may already be lost.

The setting for this battle is the historic Old City: a small walled enclave of less than one square kilometre within the sprawling city that is Jerusalem, divided into loose quarters for Muslims, Jews, Christians and Armenians. It is the heart of the decades-old Israeli-Palestinian conflict, the centre for the world's three great monotheistic religions, and a magnet for pilgrims and tourists from all over the world. In this crucible of faith, priests, rabbis and imams brush past bare-limbed backpackers as they make their way over the treacherously smooth flagstones of its narrow alleyways. Gaggles of pilgrims from Eastern Europe, West Africa and Latin America jostle with ultra-orthodox Jews and devout Muslims on their way to pray at the Church of the Holy Sepulchre, the Western Wall or the Al-Aqsa mosque.

But away from the souvenir shops selling religious trinkets, olive-wood chessboards and belly-dancing outfits, a religious and nationalistic struggle is ratcheting up tensions. Palestinians say a programme of "Judaisation" of the Old City is accelerating; ideologically driven and biblically inspired Jewish settlers insist they are simply redeeming land gifted to them by God.

Around 1,000 Jewish settlers now live among 31,000 Palestinians in the Muslim Quarter of the Old City, taking over homes that have been inhabited by Muslim families for decades or even centuries, and flying Israeli flags from the walls and rooftops of their properties. They are the frontline fighters in a broader battle – backed by the Israeli government, city authorities and security services – to ensure Jewish control of Jerusalem and to drive its Palestinian population down to a minimum.

Twelve members of the Najib family – eight adults and four children – live in the three rooms of their first-floor apartment on El-Wad street. Ebtahaj Najib, 58, moved into the house on the day she married her cousin in 1973 and all nine of her children were born and brought up in the house, including her 38-year-old son Youssef. Her husband died eight years ago.

By custom, Palestinian extended families live together or close by, but there is insufficient space at the Najibs' house, and some of her sons have been forced to move out since getting married and starting families of their own.

‘You think everyone has a room?’: a photo of the Najib family patriarch, who bought the first-floor apartment on El-Wad street decades ago. Photograph: Tanya Habjouqa for the Observer Tanya Habjouqa/Guardian

"You think everyone has a room?" laughs Ebtahaj when I ask where the remaining family members sleep. The answer is: crowded together, with the couches in the living room becoming beds when night falls.

Even so, the Najibs' home is spacious compared to many in the Muslim Quarter. Light streams through large windows into a high-ceilinged living room, the walls of which are adorned with 1950s portraits of Youssef's dapper, moustachioed grandfather and glamorous, lipsticked grandmother. Outside the sparsely furnished room, a balcony overlooks shops and cafés – the Old City's best-known hummus place, Abu Shukri, is almost below.

Immediately above the balcony, five large Israeli flags hang from the second floor. To casual passers-by, these symbols of the Jewish state, along with the Hebrew sign over the arched entrance announcing the Synagogue of the Union of the Fighters of Jerusalem in the Old City, send a clear message: this building is in Jewish hands. The Najib family's presence is rendered almost invisible.

For the past 30 years, a yeshiva – a place for religious study – has been based in the floors above the Najibs' home. According to the Najibs, the students, teachers and round-the-clock armed security guards make noise, throw garbage down the stairwell and intimidate the children. "Every minute – midnight, midday, evening, morning – they are singing, praying, playing music, slamming doors, coming up and down the stairs. But they never speak to us," says Youssef.

Nobody at the yeshiva is willing to speak to the Observer, either. As I leave the Najibs' home, under the watchful eyes of a security guard stationed in a sentry box almost opposite the family's front door, a group comes down the stairs. I ask to hear their side of the story. They push past without making eye contact. Daniel Luria, the spokesman for Ateret Cohanim, the organisation behind the yeshiva, later tells me that none of the settlers – a term he rejects – in the Muslim Quarter would be willing to be interviewed. "It's never advantageous. We are always seen as the occupier – the Palestinians are always seen as the residents," he says.

According to Ateret Cohanim's website, ateret.org.il, the yeshiva is the "spiritual epicentre of a community of almost 1,000 residents in the heart of the Old City in the so-called Muslim Quarter". It now refers to the area as the "Renewed Jewish Quarter".

But Ateret Cohanim is much more than a promoter of religious studies. It is dedicated to helping Jews buy up Arab properties in the Old City and East Jerusalem in furtherance of what Luria calls the "physical and spiritual redemption" of the city. Ateret Cohanim has assisted in the purchase of at least 50 properties in the Muslim Quarter, and plans to build on that number.

The complex and violent history of this city has filled countless books. There is no dispute that Jews were its earliest inhabitants, but the presence of Muslims and Christians also stretches back over multiple centuries.

‘Unfortunately some Arabs have not come to terms with having Jews live next door to them’: Arab women are watched by Israeli soldiers and, left, El-Wad street in the Old City. Photograph: Tanya Habjouqa for the Observer Tanya Habjouqa/Guardian

More recently, at the end of the war following the declaration of the State of Israel in 1948, Jerusalem was divided, with the Old City on the Jordanian-controlled eastern side of the armistice line, known as the Green Line. The Jewish population within the ancient stone walls sank to zero.

Nineteen years later, Israel captured East Jerusalem in the 1967 Six Day war, "liberating" – in its terminology – the Old City. Jews returned to live close to their revered site of the Western Wall and Israel declared the "reunified, indivisible" city of Jerusalem to be its "eternal" capital. Israel's annexation of East Jerusalem has never been recognised by the international community. The Palestinians want Arab-dominated East Jerusalem as the capital of a future state, but Israel is determined to resist any division or sharing of the city; hence the state's policy of establishing Jewish "neighbourhoods" – settlements, to the rest of the world – in areas across the pre-1967 Green Line. Some of these settlements are big developments, housing thousands of Israeli Jews in modern apartment blocks. Others are tiny pockets of hard-liners in the heart of Palestinian communities, where the presence of settlers and their security guards causes friction and animosity.

With little prospect in sight of a peace deal involving a shared Jerusalem, Ateret Cohanim, one of the key drivers of religiously motivated settler pockets, is increasing and consolidating the Jewish presence in the Muslim, Christian and Armenian quarters of the Old City. According to Luria, the organisation "facilitates purchases" but does not buy property itself. This is disputed by its critics, who say it runs a network of front companies in an attempt to disguise its involvement in acquisitions.

A report, Jerusalem, The Old City, published in 2009 by the International Peace and Cooperation Centre (IPCC) – a Palestinian civil society organisation – said Ateret Cohanim was "taking the lead in the process of Judaising the Old City". Properties were acquired using three different methods, it said: claiming historic Jewish ownership and securing a court order to evict Palestinian residents; taking over "absentee property", or using underhand transactions, in which the identity of the buyer is concealed.

Luria denies that Ateret Cohanim uses front companies but concedes that buyers sometimes use Palestinian intermediaries. "Arab law says that an Arab should be killed if he sells property to a Jew," he says. "It's a disgrace in a modern, democratic country, but Arabs sometimes have to be protected. He can't openly be a party to the sale in some circumstances. So Arab middlemen are sometimes used, and legal somersaults are performed. Not in every deal, but when necessary."

The organisation's current targets include properties near Herod's Gate, a Palestinian community centre near Lion's Gate, and homes near the Little Western Wall, just below the Haram al-Sharif or Temple Mount, say Palestinian activists.

"This is the heartland of the Jewish people. Why shouldn't we return – especially if we're paying good money?" says Luria. "We're not kicking people out. Jews should be able to buy here, just like in London or New York. We are the indigenous people in this land." He adds: "The Arabs are illegal squatters occupying this land." If an Arab feels "uncomfortable" with Jews living in the Muslim Quarter, "that's a shame. But if he doesn't like it, there is no shortage of other countries with Muslim majorities. If they can't accept us, that's their problem. Why should I apologise or feel bad?"

Despite being loquacious on the rights of Jews to the land, Luria is reticent on the funding of Ateret Cohanim. I ask him if Irving Moskowitz, an octogenarian US bingo tycoon whose eponymous foundation funds settler activities in East Jerusalem and who is widely reported as giving millions of dollars to Ateret Cohanim, is one of his backers. "We receive donations from here and abroad, but we don't discuss any individual who supports the organisation," is all Luria is willing to say. Support also comes from the state of Israel, not least in the form of security for the settlers.

A few metres from the Najibs' home, at the junction of El-Wad street and Via Dolorosa, the Israeli border police maintain a daily presence, routinely demanding to see Palestinians' identity papers, where they live and where they are going. "They never stop the Jews," says Youssef Najib. "They are here to help the Jews."

'This is the heartland of the Jewish people. Why shouldn’t we return – especially if we’re paying good money?': El-Wad street in the Old City. Photograph: Tanya Habjouqa for the Observer Tanya Habjouqa/Guardian

Meanwhile, in the narrow alleys and hidden courtyards of the Muslim Quarter, the daily grind of life is worsening little by little. In the past 30 years its population has doubled, exacerbating already-high levels of overcrowding and poverty. A report on the Palestinian economy published earlier this year by the United Nations said housing density in the Muslim Quarter was almost three times as high as in the Jewish Quarter, and many Palestinian homes lacked running water and a proper sewage system. More than 80% of dwellings require major rehabilitation or urgent maintenance, according to the IPCC.

Three out of four children in the Muslim Quarter live below the poverty line, and unemployment is more than 30%. Garbage collection is sporadic in these back streets, and there are almost no open spaces for children to play in. The use of child labour is widespread; the dropout rates from schools are high. Domestic violence and drug abuse is on the rise.

A major reason for the migration into the Old City is an Israeli requirement for Palestinians to prove that Jerusalem is their "centre of life" in order for them to keep their valued residency rights in the city, giving greater access to jobs, education and healthcare. More than 7,000 Palestinians had Jerusalem residency rights revoked between 2006 and 2011; faced with such a threat, thousands more moved from suburbs and villages outside Jerusalem back into the city – including the Old City – to secure their identity papers. Others, who found themselves cut off from the city centre by the vast concrete separation wall, moved into the Old City to avoid daily checkpoint ordeals.

On top of this, says the UN, Palestinians in the Old City are "caught between the frontlines of interaction with Israeli settlers and authorities on a daily basis and the frontlines of a struggle to preserve and assert Palestinian cultural and political identity and its Islamic and Christian roots. This has entailed a growing sense of siege and conflict for indigenous Palestinian residents, who perceive their lifestyles, livelihoods and social cohesion to be at risk in the discordant climate reigning in the Old City, with religious fervour easily degenerating into communal tensions."

Luria dismisses such a picture. "Jewish families are living in the Old City side by side with Arabs, in some cases even in the same courtyard. OK, it is not necessarily beautiful puppy love, but it is basic coexistence, which is the best you can hope for in a place as volcanic as Jerusalem. Unfortunately some Arabs have not come to terms with having Jews live next door to them – Arabs who in general have a problem living in a national Jewish homeland. But the land of Israel belongs to the Jewish people."

In the house on El-Wad street – beneath the fluttering Israeli flags, along from the armed security guards and Israeli police officers, and where the sound of the muezzin from the mosque across the street sometimes competes with the chant of Jewish prayers from the upstairs yeshiva – Youssef Najib shrugs when I ask if he thinks the Jews are here to stay in the Muslim Quarter.

"They won't even give us the West Bank for a state, so you think they'll give back East Jerusalem?" he says. But he has created his personal frontline in the battle for the Old City. Many times settlers have knocked on the Najibs' door to offer the family money to leave the property. But Youssef says: "If you give me the whole wealth of Israel, I will not give you my home."