Next week, Tate Modern in London is unveiling a complete retrospective by a director who I can only describe as the Samuel Beckett of world cinema: and even that comparison doesn't quite convey how severe and how uncompromisingly difficult his movies have latterly become. This is the Portugese film-maker Pedro Costa – a cult master, a figure who is widely considered on the festival circuit to be for hardcore auteur followers only. A Pedro Costa film does not get a "release". It does not "do business" – any more than a piece by Edgar Varèse rules the iTunes chart. I myself have seen critics and writers at festivals gird their loins reasonably happily for a Béla Tarr film. But at the words "Pedro Costa", they flinch. A haunted look comes into their eyes.

Now, I can understand this. But considering the arc represented by Costa's major features O Sangue, or Blood (1989), Ossos, or Bones (1997), No Quarto Do Vanda, or In Vanda's Room (2000) and Juventude Em Marcha, or Onward Youth (2007), I now believe that his career arc is one of the most fascinating in modern cinema. Following this career is not, however, easy and Costa does not make it easy, increasingly setting his films in the collapsing rubble of Fonthainas, Lisbon's grimmest slum. He favours interminably long shots, long silences, long aimless semi-audible conversations between semi-comatose drug addicts: like watching a Big Brother live feed direct from some of the most poverty-stricken places in Europe. When Onward Youth was briefly shown in the UK last year, it appeared under the title Colossal Youth, and the trade press cheerfully dubbed it "Colossal Bore". My colleague Cath Clarke wrote about this film last year with great insight.

This director increasingly contrives scenes in cramped rooms in semi-darkness, shot from below, with perhaps one light-source in the form of a window in the top right corner of the frame, which glows without illuminating the scene. His most recent film is Ne Change Rien, which applies this technique, eccentrically, to a film about the singing career of the French film star Jeanne Balibar, who is shown rehearsing, performing and recording in a weird crepuscular darkness.

The retrospective shows Pedro Costa's work evolving from conventional dramatic movie-making into an experimental docu-installation form, which is something between a real-time "reportage" cinema and an exhibition of animated portrait images. However difficult and punishing his films are, I am becoming weirdly hooked on them. They deserve a hearing from people who are open-minded about cinema as an art form, and particularly as an experimental art form. In scheduling the Pedro Costa series the Tate's curator Stuart Comer is effectively challenging movie writers to re-examine the criteria on which they discuss cinema.

Blood (1989), is Costa's first film, made in black-and-white, about the relation of two brothers and their father. It looks beautiful, and is clearly influenced by Truffaut's The 400 Blows, and I think also has something of Buñuel's Los Olvidados. It could have been made 40 or even 50 years ago, and Costa contrives a Nouvelle Vague feel, along with a Hollywood-ised reference to Nicholas Ray and maybe Charles Laughton's Night Of The Hunter.

Bones (1997) is, I think, Costa's best film. Maybe it's even some sort of masterpiece: a dark, mysterious and mesmeric movie, shot in colour and set in Lisbon, among the urban poor. The faces that Costa captures are compelling, and disquieting, the kind of faces you would see in an unsettling dream: particularly the young woman at the very beginning, and also the face of Tina, who has just had a baby, and whose partner, played by Nuno Vaz, takes it away to try to get rid of it, trying first to give it to the hospital nurse and then to a prostitute. They are like the faces of ghosts, or faces of the dead. The atmosphere of Bones is extraordinary, like a horror film without the horror, or like a social-realist version of David Lynch's Eraserhead.

Like many of Costa's films, it has been wearily or amiably dismissed as miserabilism – and yet this isn't how I responded to it. The film is about poverty, yes, and this is a subject which some people in both the movie business or the journalism business think is bad taste, as if poverty doesn't really exist all that much and to emphasise it is a callow pose or crass exaggeration of style. Costa's film takes poor people seriously and does not apologise for the moral and even spiritual seriousness of what it is doing. Watching this, I remembered a resonant line from Dickens's Bleak House: "What the poor are to the poor is known only to themselves and to God."

With In Vanda's Room (2000) and Onward Youth (2007), Costa's work moves to a dauntingly austere mode, a mode in which traditional cinephilic references are much less useful. To continue the Beckett analogy, Blood represents his "Molloy/Watt" phase, the early phase in which his work is at its most traditional and accessible. Bones is the "Godot" phase, in which his identity becomes strongest and most distinctive, and yet still accessible in normal cinematic terms, and In Vanda's Room and Onward Youth are his opaque and difficult, later phase, his "Imagination Dead Imagine" phase.

In Vanda, we see the return of two personae from Bones. They are the sisters Zita and Vanda Duarte, now overtly playing themselves, and in fact the film now sheds the ostensible fictional mode of Bones, and now becomes far more of a documentary portrait. Zita – whose face in closeup was so haunting at the beginning of Bones – and Vanda are now fully paid-up heroin addicts. Long, long scenes show them in their dark, cramped, squalid room in the Fontainhas slums of Lisbon, doing smack and talking inconsequentially about nothing much. The movie itself, with its series of fixed camera positions, is closer in spirit to an exhibition of photography, a succession of cinematic tableaux. The vivid, ghostly close-ups of Bones do not feature.

The people, living fragmented and embattled lives, are shown in tenements which are in the very process of being torn down. Yet there is no positive spirit attached to this, no sense that the Portugese state is moving them to better quarters: just a grim feeling that these houses are awful and even these are being destroyed. In one scene, taking place in crepuscular gloom, one man complains of a fear of death, and another says to him: "The bad never die. It is the innocent who die" – a very Beckettian line, especially when you realise that it is meant to be reassuring, and that the speaker considers himself and his companion to be one of the "bad".

The Pedro Costa retrospective isn't for everyone; but if you want to see challenging art on screen then it is a must. Go and see Bones – one of the most enigmatic and haunting films of modern European cinema.