R.A. Lafferty(click on names to see more mathematical fiction
by the same author)

...

A wacky sci-fi adventure comedy featuring space pirates. There is not much math in the book, but the central plot revolves around a mathematical ``doomsday equation'' and the goal of preventing the horrible fate it predicts for humanity:

(quoted from Annals of Klepsis)

The humanly inhabited universe, according to the best -- or at least the newest -- mathematical theory, does have a tertiary focus, and it is there that it is vulnerable. The humanly inhabited universe, with its four suns and its seventeen planets, is an unstable closed system of human orientation and precarious balance, a kinetic three-dimensional ellipse in form, with its third focus always approaching extinction. As with any similar unstable premise-system, the entire construct must follow its third focus into extinction. This is known as the `Doomsday Equation.'

The equation has been bad-mouthed because it originated on an asteroid and not a planet; but must we forever believe that planetary mathematics is always superior to asteroid mathematics?...

As a result, attempts to understand or alter the equation and characters with expertise in mathematics play a key role in the plot.