Igor Stravinsky, symphonist. Not exactly the first thing that comes to mind, and yet Stravinsky wrote five pieces that include "symphony" or "symphonies" in the title. Four are masterpieces (Symphonies of Wind Instruments, the Symphony of Psalms, Symphony in C and Symphony in Three Movements), and one – the early Symphony in E Flat Major he wrote as a student – is a piece of juvenilia but is still scandalously neglected in the concert hall. If you don't know it, listen to it here – top tunes; clear, brazen orchestration; and some firebrandish over-ambition, it's all there; and is proof of how much Stravinsky owed to his teacher and mentor, Rimsky-Korsakov.

But this week's piece is one of those later works of Stravinsky's that does strange things with the symphonic form. The Symphony of Psalms is, for me, one of the most deeply moving and genuinely spiritual pieces Stravinsky ever wrote, but to understand why, we need to examine, and quite possibly overturn, some conventional ideas about the music Stravinsky was writing at this period of his life.

The Symphony of Psalms was a commission to celebrate the 50th anniversary of the Boston Symphony Orchestra in 1930. The orchestra wanted something symphonic, and Stravinsky's publisher wanted something popular. Stravinsky gave them both – but not in the way either party could possibly have envisaged. "I took the word, not in the publisher's meaning of 'adapting to the understanding of the people', but in the sense of 'something universally admired'," he said. That's how Stravinsky thought of the Psalms he sets in each of the Symphony's three movements, and especially Psalm 150 in the final movement, which is by far the longest of the three, with its vision of a world celebrating God through music.

Stravinsky had apparently already had the idea of composing a piece that would include the psalms in an orchestral context, but he didn't want to follow in anyone's footsteps. "I even chose Psalm 150 in part for its popularity, though another and equally compelling reason was my eagerness to counter the many composers who had abused these magisterial verses as pegs for their own lyrico-sentimental 'feelings'. The Psalms are poems of exaltation, but also of anger and judgement, and even of curses." The two earlier movements set texts of human penitence , sinfulness and longed-for salvation: the sinner's cry to be heard in the first movement, the "new song" forged after the Lord has hauled the psalmist "out of a horrible pit, out of the miry clay" in the second.

The secret of this 23-minute work's symphonism is its relationship between chorus and orchestra. Stravinsky composed for them so that "the two elements are on an equal footing, neither outweighing the other". He said he wanted "to create an organic whole without conforming to the various models adopted by [symphonic] custom, but still retaining the periodic order by which the symphony is distinguished from the suite."

The music of this period of Stravinsky's life is called neo-classical, which suggests an often ironic or coldly knowing refraction of past forms and manners through the prism of Stravinsky's way of hearing, seeing and imagining. But that's the idea I think you need to chuck out when you're listening to the Symphony of Psalms. There are references to previous styles and modes of musical discourse in the piece – especially the double fugue of the second movement, with a dense contrapuntal texture in the choir and the orchestra and a first melody that flirts with a memory of the main subject of Bach's Musical Offering – and there are moments that resonate with Stravinsky's recent music, such as his opera-oratorio Oedipus Rex. But it's the new sounds, forms and shapes of the Symphony of Psalms that define the piece as a profoundly unironic, essentially sincere expression of Stravinsky's unique approach to the psalms, the symphony and even his faith (he was an observant Orthodox believer at this time in his life).

The first noise the Symphony makes crystallises the sharp directness of its soundworld: an E minor chord played by the orchestra of two pianos, harp, woodwind, brass and low strings (there's no room for violins or violas in this symphony, those conventional carriers of symphonic line and melody – the choir, in a sense, takes their place). There's an octatonic skirl in the oboe and bassoon (a way of splitting the 12 notes of the chromatic scale into alternating intervals of semitone and tone), and the symphony is propelled into life.

The choir's first entry sounds like a stern lament with the altos' severe yet keening line before the whole choir cries out in desperation. The whole of this short movement is laceratingly unsentimental, but there's a vivid emotional power to its final, major-key cadence, as the choral voices long for their non-existence, a time when they will "be no more".

It's music that offers the opposite of easy spiritual comfort, and the second movement is no different. After a long instrumental exposition of the orchestra's fugue ("altogether too obvious, too regular and too long", Stravinsky felt, later in his life), there are moments of depictive as well as structural power in the choir's music: the way the voices build their lines on top of one another when the words talk of "setting my feet upon a rock", and above all, the massive and shocking fortissimo of the "new song" the psalmist wants the Lord to put in their mouth. Again, it's the unflinching power of Stravinsky's setting that's so striking: it may not be "lyrico-sentimental" but it is, definitively, expressive – to use a word Stravinsky didn't like much.

The setting of Psalm 150 in the finale is the most original movement in the symphony. It's a contrast between two kinds of time, and two wildly differentiated sorts of music: the slow, circling sighs and breathtaking evocation of static, infinite timelessness that you hear at the start of the movement, and fast, violent shock of the music that comes next. There's an animal ferocity in this faster music: Stravinsky's vision of praising God takes in desperation and even savagery.

One section of this music is also inspired by a specific image Stravinsky had, "a vision of Elijah's chariot climbing the heavens. Never before had I written anything quite so literal as the triplets for horn and piano to suggest the horses and chariot." And it's the contrast between the two kinds of music that defines this final movement, and makes its coda – which returns to and extends the slow music you've heard at the start and briefly, shockingly, in the middle of the movement – so moving.

Moving? Absolutely. This final music of the Symphony of Psalms, as Stravinsky creates repeating cycles of different metre and phrase, gives us a glimpse of a kind of musical eternity in the choir's praise of the Lord. And it's precisely in that alchemical combination of the music's objective, hieratic construction and its clarity and directness that its expressivity lies. Stravinsky famously said music was "powerless to express anything at all", but that's only one side of his aesthetics. In his series of lectures at Harvard, he suggested that the real nature of his music was to do with sublimated, "Dionysiac" feeling in combination with "Apollonian" order. "What is important for the lucid ordering of the work – for its crystallisation – is that all the Dionysiac elements which set the imagination of the artist in motion and make the life-sap rise must be properly subjugated before they intoxicate us, and must finally be made to submit to the law: Apollo demands it." Contained by the Apollonian clockwork of the cycles of time and the new-fangled symphonic discourse of this final movement, Dionysus – paradoxically – dances, prays, and curses all the more vividly and powerfully.

Five key recordings

Columbia Symphony Orchestra/Igor Stravinsky: Stravinsky's own recording gives the lie to the idea that this music is cool, arch, or ironic: this is a raw, visceral performance.

Forget, first of all, its mis-translated moniker. Tchaikovsky’s final symphony might be about death, but it’s the piece he termed ‘the best thing I have composed’ and is a confident and supremely energetic work