The English artist Nathaniel Mellors seems unsure of himself. Is it humor he’s after? That’s the initial impression of his exhibition “Progressive Rocks” at the New Museum, his first New York solo show. Just past the entrance to the single gallery presentation—which includes four large-scale videos, a photograph, some sculptures, and a couple trifling paintings—there’s a film projected onto a screen in front of a bench. The film, like the others in the show, was written and directed by the artist, and this one was shot in the California desert. It tells the silly story of a boyish man named Truson (played by David Birkin), who has stumbled upon a Neanderthal (Patrick Kennedy) outside a cave and cannot believe his strange good luck. He pulls out a video camera to record his discovery and peppers his new friend with questions. What’s the meaning of cave painting? How did he make those pictures of animals? What was he trying to say?

The caveman, at first, is articulate and open to conversation. But the questions are naive and grating, and Truson—who wears a baby blue knit hat styled for a newborn—repeats himself incessantly. The dialogue is aimless, and before long, the Neanderthal senses that we are going nowhere. He throws his hands into the air, calls the whole conversation “a big bag of cocks,” and then apologizes when he feels he’s gone too far.

The first few minutes of the work, titled The Sophisticated Neanderthal Interview, are slightly charming, but the gag never develops beyond the initial conceit that it would be funny if a caveman were more refined than a childish Homo sapiens. In two related films, which again star the Neanderthal, now in entirely new situations, the idea slackens even further. In one of them, titled Ourhouse—Episode 1, he tries to prove his cunning by inventing a time-traveling toilet. How Mellors came up with the idea is anyone’s guess; he seems to like non sequiturs, and many of his artistic decisions seem unmotivated. Still, there is a genuinely droll moment in the film, in which the Neanderthal, now wearing pleated gray slacks and a dandyish scarf, says, “All of my toilets are made with Aztec porcelain and have a brutalist trim.”

But humor is not Mellors’ strong suit. He is much better with his frightful and impressive sculptures of lifelike animatronic limbs that look like the severed remains of cyborgs. One of them replicates the head of the caveman and spins around ominously while his jaw flaps up and down. Another sculpture is the mutated, melted head of a man with an eye and an ear on his forehead. These works move in disconcerting ways, as does the best sculpture in the show, The Vomiter, where a bald man has a plastic tube coming out of his throat through which he throws up into an enlarged red solo cup.

"The Vomiter (Ourhouse)" 2010Alexander Newton

These genuinely unsettling objects are unfortunately few in this exhibition, but Mellors, who was born in 1974 in Doncaster, England, has made more in the past. Sometimes, they come in pairs, like Hippy Dialectics (which he showed in London in 2011), where two heads (one covered in an alien blue skin) are connected by a long, scraggly beard. Works like these aren’t completely humorless; they have some of the grotesque fun that’s so prevalent in Bruce Nauman’s obscene neon light sculptures, one of which shows a hanged cartoon man with a giant erection. So it’s conceivable that Mellors could mine his distressing work for comedic effect, but he needs to strike a better balance.

That’s most obvious in the show’s biggest installation, The Aalto Natives (Floored Version), which is adapted from his joint presentation with the Finnish artist Erkka Nissinen at the Finnish Pavilion in the most recent Venice Biennale. Across four separate screens, a group of muppets devised by Mellors and Nissinen tell a convoluted story about the history of life. One of the characters, Atum, has a cardboard box for a head; the other, his father, Geb, is a giant talking egg.

In the Venice presentation, the emphasis was on whimsy. When one of the muppets sang a song in the Finnish Pavilion about the history of the country, his sweetness rose above the story’s overall lack of direction. But in New York, the work is shown in a room where the centerpiece is a newly made animatronic sculpture of severed human body parts, which poisons the joy. On their own, the separate parts might work well: the strangeness of the sculpture could deepen if it weren’t forced to fit into a larger narrative, and the film could offer a small dose of irreverent amusement. As it stands, Mellors is trying to do too much at once, and some simplifications could do him a large favor.