Mathematics is extremely difficult to carry off in fiction, as few readers will take the time to do the math required to understand the processes involved. "The Martian" was an exception, in that it was a wonderfully involving book, and the math never felt overwhelming. More importantly, you could follow the story without following the math--though it added greatly to appreciating the story.

Focusing a story on math is similar to including maps of website references in a story. The problem is, few readers will remove themselves from the story long enough to do their research. Readers like to lose themselves in a book, and don't want to be pulled away. The easy solution, is to make the story stand on its own (without relying on external knowledge). That's why such stories are rare, and successful ones are even rarer.

Science fiction is a good example. Science fiction is incredibly popular, but most of that is actually science fantasy, imaginary worlds not based on science. Hard science fiction is harder to write, as it requires an in-depth knowledge of the science principals involved, and you've got to convey the information to readers largely uninterested in such trivia.