I grew up on a Glasgow council estate in the 1960s and 70s. I received nothing but support from my family in my ambition to go to university. I also had the good fortune to attend local schools that, despite limited resources, strongly valued academic aspiration and achievement. My path to university was a straight one, even if it was a path followed only by a minority.

So I have taken particular interest in the debate triggered by David Willetts's exhortation to universities to target white working-class males as they are under-represented in higher education.

There is an issue here. Ucas figures for last autumn's intake show a downturn in applications from men last year of 22,000 – while the decrease in the entry rate for men to higher education was four times that for women. The minister called this "the culmination of a decades-old trend in our education system which seems to make it harder for boys and men to face down the obstacles in the way of learning."

Commentators appear to have taken the minister's injunction to treat male under-representation similarly to ethnic minority under-representation too literally: by treating it, too, as a proxy for an ideological debate on educational entitlement, social engineering and mobility.

Despite the generalisations of this access debate, the shape of and reasons for it and the remedies needed are complex, multifaceted, historically contingent and contested. Research at the University of East London's Continuum Research Unit indicates that working-class male participation can vary enormously even within the same borough, from 35% to 58%. One underlying factor is that males enjoy post-16 education less. Both males and females are worried about support finances beyond fees.

This is not just a challenge in England. North of the border, NUS Scotland has supported tougher access targets to "Unlock Scotland's potential", including a call for uncapped places for students from poorer areas. The geographical distribution of such areas creates a difficult challenge for some universities.

What, then, can and should be done? First, I would argue that many of the policies or interventions that would help white working-class males would help boost access for all under-represented groups, albeit to a greater or lesser extent. We must continue to prioritise multiple routes to higher education through A-level, BTec and apprenticeships, too. At the University of East London our own New Beginnings access programme is an important route for male mature students. At UEL we have already increased working-class male participation in our access agreement.

Second, we need to move beyond the often polarised debate between blaming universities and blaming schools. It is true that according to the General Teaching Council, one in four primary schools has no male teachers. Similarly, according to the Centre for Economic Performance at the LSE, boys at secondary schools believe female teachers will give them lower grades – and invest less effort as a consequence. Equally, despite efforts to attract more working-class students, some universities have fallen short – some significantly – of targets they themselves felt were achievable. More selective universities remain divided on the use of contextual data (information about students' backgrounds). Between universities and schools we need stronger partnerships rather than deeper polarisations.

Third, we need to recover the ground lost through the abolition of the education maintenance allowance and AimHigher, and maintain institutional widening-participation funding. There is scant evidence that trying to tackle under-representation through funding from fees under Office for Fair Access-approved access agreements alone would work. It is necessary, but not sufficient.

Fourth, we need to give more attention to national advisory schemes and publicity focusing on how to understand and benefit from the opportunities afforded by a university education per se, as well as schemes that advise students of the advantages of one university over another.

Fifth, too often we ask applicants of different intellectual shapes and sizes to twist and turn to fit the places available. This access "Tetris" has to stop. We need to recognise that white working-class males, like many under-represented groups, do not go to university straight from school on the back of three A-levels. If we define this as the problem, we overlook potentially productive interventions with the 23-30 age group via further education college and university access programmes. At UEL our network of 19 FE colleges and associated bursary package was instrumental in ensuring we hit our recruitment targets.

Sixth, we need to make sure there are enough places available in the higher education system generally to ensure that universities are not forced to choose between early birds and late bloomers. Indeed, the universities best placed to attract and support working-class white males, modern universities, are the ones losing the most places through the policy of unlimited places for students getting ABB A-levels.

Seventh, we need to rethink the rhetoric on apprenticeships and the false dichotomy between academic and practical courses. Too much rhetoric about apprenticeships is a thinly veiled discourse about males and manual labour targeted, of course, less at Neets (Not in Education, Employment or Training) and more at OPCs (Other People's Children).

Overall, then, we need to rebuild the student finance infrastructure, the partnership infrastructure and the advisory infrastructure to provide support for those who do not have a straight route to university and who, despite their talents, are pushed away or who wander away. Unless we act urgently, and collectively, we will hit a point of no return and their path to university will be blocked for ever.

• Professor Patrick McGhee is chair of Million+ thinktank and vice-chancellor, University of East London