Science-fiction writing used to be the preserve of spotty teenagers and cranks. It was a small, incestuous subculture, regarded by most people with amusement and disdain. Both were sometimes deserved. The culture was plagued by a misogyny so intense it sometimes crossed the line into psychosis. There were many talented authors, but also an abundance of shameless hacks. The quality of the writing varied wildly not only between writers, but in the works of an individual writer – often within a single book. Several eminent authors prided themselves on being able to write a novel in a couple of weeks. Samuel R Delany, Gene Wolfe and Michael Moorcock have all written both unreadable garbage and books regarded as literature even by non-geeks.

In that wild west era, plots could go anywhere or nowhere. A typical plot development, in Philip K Dick's Clans of the Alphane Moon, has a hero crushed by divorce and failure, contemplating suicide in his crappy apartment. At the last moment, he's interrupted by his neighbour, a telepathic slime mould. Having rudely flowed under the door, it says, "I couldn't help overhearing …" Then it offers the man a job and says it will find him a replacement wife. Off it goes, and soon a teenage girl arrives at the door. She is completely content to be fixed up with a much-older suicidal loser by an alien slime mould. Her breasts are exhaustively described.

This takes three pages, and is not buoyed by any particular grace or style. It's ridiculous; it's offensive. And it has an effect you simply cannot produce with a book that is well written.

Authors also did not feel constrained to create sympathetic characters. Protagonists often despised the human race, whether or not they belonged to it. Fans might be rooting for a hero, only to have him rape a girl because she hurt his feelings, as in The Stars My Destination by Alfred Bester. James Tiptree Jr.'s Love Is the Plan The Plan Is Death is told from the point of view of an insectoid alien besotted with a larva – ingeniously combining gross bugs and paedophilia. Here is a typical passage:

(Spoiler alert: In the consummation of their love, the larva eats the ecstatic protagonist.)

Perhaps the highest insanity of all was achieved in Cordwainer Smith's A Planet Named Shayol. Shayol is a prison planet where inmates are colonised by local life forms called dromozoa. The dromozoa confer immortality, but also cause people to sprout extra organs and limbs – noses, heads, strings of torsos trailing like railroad cars – which are harvested for transplant. The largest feature in the landscape is "an enormous human foot, the height of a six-storey building". As the prison warden, an artificially evolved cow, explains, the foot is Go-Captain Alvarez, the man who discovered the planet. "After 600 years, he's still in fine shape." It gets crazy from there.

These stories share a blithe disregard for the comfort zone of the reader. Jarring elements are introduced so carelessly they're often hard to follow. But this makes them, for lack of a better word, mindblowing. It's sometimes difficult to believe that they were written by a human, and scary to contemplate that that human was at large in the community. The goofball plots, the disjointed writing styles, even the candour of the sexism are things older fans miss, much the way New Yorkers miss the post-apocalyptic landscape of New York of the 70s. It could be nasty, but it was different from everything else, unpredictable and often frightening – and it was ours.

Nowadays, we all live in a science-fictional world. Computers are our boon companions; our food is invented in a laboratory. So it's unsurprising that science fiction has moved into the mainstream. Respectable literary authors dabble in it. Science-fiction novels are nominated for major awards. The average reader is no longer a mind-blown teen who will accept any unpleasantness in exchange for cool ideas. The average reader is the average reader. So editors are acquiring books according to criteria that were formerly incidental to the genre – quality, readability, plots that make sense. The twisted misogyny is gone, and with it the bracing misanthropy. The cool ideas are still there, but a certain anarchic power has been lost.

As a literary author who's just written a work of crossover SF myself, I'm not hypocrite enough to claim that these developments are entirely negative. But I hope fans of the new SF will look back to some of the weirder, more socially unacceptable books of the past. A crafted work of literature is a beautiful thing; a genius's unedited ravings, however, is a thing both beautiful and rare.

• Sandra Newman's new novel, The Country of Ice Cream Star, is published by Chatto & Windus.