Steve Brookstein, winner of the very first X Factor in 2004, and his wife Eileen Hunter are busying themselves one early afternoon in the 606 club in Chelsea, west London. It's a venue they often perform at, and it's where they're going to sing for Showing Off... the title track of Eileen's recently finished soon to be released self-financed album, I Miss the Sky.

The scene at the club aches with the sort of run-down sadness there often is when you are inside a jazz club at midday and certain things are missing, like the night, alcohol, conversation and anticipation, and the sense that outside, everything has stopped. All this after-midnight atmosphere seems to be replaced by peeling paint, dirty floors and a desolate bar area, and a sense that this is not a place of entertainment but a space where dreams have long ago run out.

It must be said that this sadness cannot help but be enhanced by the presence of Brookstein, simply because even when he was winning the X Factor and having a grinning, triumphant Simon Cowell put his arm around him, he just happened to have the shape and demeanour of someone for whom things always go wrong. Brookstein winning the X Factor as an over 25 was surely the moment when this hard-working, music-loving luckless bloke in his mid-30s with bit of a decent white soul voice, previously doomed to obscurity, had actually made it in what we can quite comfortably describe as the music business.

As soon as he had made it, though, give or take a brief honeymoon where his Sony/Cowell-organised suited and booted covers music designed to sell in Tesco and Woolies suggested the illusion of success, it became clear that he had not really made it. Cruelly, he had been told officially in a public arena that he was really, really wanted, and then, in a way, after a few months of automatic attention that was ultimately really for the programme itself, he was told that really there had been a huge mistake, and no one wanted him at all.

The millions of people who voted for him, and the hundreds of thousands who had bought his records, turned out to not really exist, or at least they weren't Steve Brookstein fans. They were fans of the process of apparently being part of the creation of a new star, but they were not at all interested in nurturing and permanently adoring the star they had created. As soon as the show was over, as soon as they'd bought his records to confirm that they had chosen the right person, because after all he was No 1 in the charts, they lost interest.

They were more interested in the next star they might create. They had invested in Steve Brookstein not as someone who was a proper pop star, but simply as the momentary end product of their own gullibility towards the programme and how the programme slickly encouraged their participation. Steve returned to obscurity, but this obscurity now came connected to this new understanding he had of what it was like to be a star – he was now trapped on the outside of the mainstream music industry but cursed with the fact that he had been given this glimpse of what it felt like to be operating in the absolute dead centre. Plain everyday obscurity had been replaced with a kind of agonised unique obscurity. He would now always be known for not being known, for the fact that his second album reached a high of 167, and there was still further to fall.

The show carried on, because it could, because Cowell wanted it to, and it could even prove that even when its winners in fact seemed not to have the sacred, elusive X factor, the show itself exploded with possibility, and was still built and sold in the name of success, and marketed as the ultimate example of how dreams could come true. In the 606 club, Steve comes across like so many earnest people who work in the margins of the music world – clearly the wrong kind of person to enter a talent competition where a real love of the idea of music as something transporting is actually a disadvantage.

Desperation to turn his love for music into a way of making a living had driven him to X Factor, which always noted, and at times viciously mocked, his lack of desire to be famous – this was interpreted as a kind of insulting apathy, as opposed to a desire just to make music and sing songs and earn an adequate living. It wasn't fame he wanted. It was, genuinely, music, and the ability to make music, in his way, in his own time.

The desperation remains, and ultimately, as is often the case, it's the kind of desperation that actually gets in the way of any kind of relative success. For someone who as soon as he auditioned for Louis, Sharon and Simon caused suspicion because he didn't repeat the all important mantra "I want it so much," he actually really does want it so much, too much, and still believes it can happen because he loves music, can hold a tune and make a pleasant noise – and he may yet be able to work out how to turn the fact he was an X Factor to his advantage, finally overcoming the embarrassing fact he really is who he is, the very first X Factor winner, who struggles to deal with the chaotic, cruel and unforgiving tabloid world that suddenly and randomly, and inappropriately, embraced him.

Eileen sings her song with a nice friendly jazz accompaniment, standing up well to the lack of atmosphere in the daylight club, and Steve sings backing vocals with the kind of level-headed restraint that is the absolute opposite of his shaken, shaky desperation to be respected or at least noticed as a real musician. The competence of the performance, the controlled neatness of Eileen's delivery, the authenticity of swing, the concentration of Steve, somehow it all adds to the overall sadness.

Once, perhaps, if Steve had not been called back after being rejected by the judges, and had escaped this particular adventure, you might have come across such a performance in a jazz club as a support act, and been gently surprised. You wouldn't have been dismayed to see the outfit support Diana Krall or Maddy Peyroux.

Now alas, the performance exists in the writhing, degenerate shadows of the X Factor, and it somehow reminds you of how cold-blooded and spiteful the music business can be – how if you don't belong to what various forces have decided is the only way things fit together, you can be profoundly exiled. And, to be honest, there's a kind of wounded bravery about the fact that Steve, and Eileen, for better or worse – after the beating they took in and around X Factor, and the impact it had on their lives, and the fact, as proved here, that you cannot separate him from the show, and that whenever he tries to talk about his gruelling experiences he ends up sounding bitter – just keep on trying.