For many years he worked on mathematical physics and probability theory, and he retained a residual interest in these fields, particularly in connection with possible extensions of stochastic mechanics to field theory.

In 1950, Nelson formulated a popular variant of the four color problem: What is the chromatic number, denoted χ{\displaystyle \chi }, of the plane? In more detail, what is the smallest number of colors sufficient for coloring the points of the Euclidean plane in such a way that no two points of the same color are unit distance apart?[3] We know by simple arguments that 4 ≤ χ ≤ 7. The problem was introduced to a wide mathematical audience by Martin Gardner in his October 1960 Mathematical Games column. The chromatic number problem, also now known as the Hadwiger–Nelson problem, was a favorite of Paul Erdős, who mentioned it frequently in his problems lectures.

In the later part of his career, he worked on mathematical logic and the foundations of mathematics. One of his goals was to extend IST (Internal Set Theory—a version of a portion of Abraham Robinson's non-standard analysis) in a natural way to include external functions and sets, in a way that provides an external function with specified properties unless there is a finitary obstacle to its existence. Other work centered on fragments of arithmetic, studying the divide between those theories interpretable in Raphael Robinson's Arithmetic and those that are not; computational complexity, including the problem of whether P is equal to NP or not; and automated proof checking.

In September 2011, Nelson announced that he had proved that Peano arithmetic was logically inconsistent.[4] An error was found in the proof, and he retracted the claim.[5]