Oftentimes open problems will have some evidence which leads to a prevailing opinion that a certain proposition, $P$, is true. However, more evidence is discovered, which might lead to a consensus that $\neg P$ is true. In both cases the evidence is not simply a "gut" feeling but is grounded in some heuristic justification.

Some examples that come to mind:

Because many decision problems, such as graph non-isomorphism, have nice probabilistic protocols, i.e. they are in $\mathsf{AM}$, but are not known to have certificates in $\mathsf{NP}$, a reasonable conjecture was that $\mathsf{NP}\subset\mathsf{AM}$. However, based on the conjectured existence of strong-enough pseudorandom number generators, a reasonable statement nowadays is that $\mathsf{NP}=\mathsf{AM}$, etc.

I learned from Andrew Booker that opinions of the number of solutions of $x^3+y^3+z^3=k$ with $(x,y,z)\in \mathbb{Z}^3$ have varied, especially after some heuristics from Heath-Brown. It is reasonable to state that most$k$ have an infinite number of solutions.

Numerical evidence suggests that for all $x$, $y$, we have $\pi(x+y)\leq \pi(x)+\pi(y)$. This is commonly known as the "second Hardy-Littlewood Conjecture". See also this MSF question. However, a 1974 paper showed that this conjecture is incompatible with the other, more likely first conjecture of Hardy and Littlewood.

Number theory may also be littered with other such examples.

I'm interested if it has ever happened whether the process has ever repeated itself. That is:

Have there ever been situations wherein it is reasonable to suppose $P$, then, after some heuristic analysis, it is reasonable to supposed $\neg P$, then, after further consideration, it is reasonable to suppose $P$?

I have read that Cantor thought the Continuum Hypothesis is true, then he thought it was false, then he gave up.

$\begingroup$I have fixed some of what I think are typos, if I've got anything wrong feel free to edit back.$\endgroup$
– WojowuMar 16 at 13:22

10

$\begingroup$Not exactly an "open problem", but infinitesimals were fundamental in the initial development of calculus by Leibniz, then after Cauchy's rigorous development of analysis their use was considered non-rigorous, but then later in the 20th century a completely rigorous theory of infinitesimals was developed by Robinson.$\endgroup$
– Sam HopkinsMar 16 at 20:33

1

$\begingroup$Regarding Booker's 1950s computing evidence: Wasn't it Swinnerton-Dyer who in the late 60s said something like "most number theory calculations to date do nothing more than pile up integers in the manner of a magpie"? (his work with Birch being an exception of course) $$ $$ For that matter, I don't see why the "prevailing opinion" should be for finiteness, indeed the 1955 Miller/Woollett paper (§14) perhaps suggests the opposite. OTOH, the later Gardiner/Lazarus Stein paper (1965) does opine that not all missing values would be represented, though they caution the reader about their evidence.$\endgroup$
– literature-searcherMar 16 at 23:02

1

$\begingroup$@MarkS I think you could include examples where conjectures turned out to be wrong (once). In number theory, besides the Hardy-Littlewood example you mention there is the inequality $\pi(x) < {\rm Li}(x)$ (disproved by Littlewood with no example). For sums of 3 cubes, initially doubt was cast in writing on "$k \not\equiv 4, 5 \bmod 9$" being sufficient for solvability (in the paper of Gardiner/Lazarus/Stein). So few thought about the problem in the 1950s that maybe those who did the numerical testing back then could be regarded as having the "prevailing opinion". :)$\endgroup$
– KConradMar 17 at 18:51

5 Answers
5

I think that originally there was a belief (at least on the part of some mathematicians) that for an elliptic curve $E/\mathbb{Q}$, both the size of the torsion subgroup of $E(\mathbb Q)$ and its rank were bounded independently of $E$. The former is true, and a famous theorem of Mazur. But then as curves of higher and higher rank were constructed (cf. What heuristic evidence is there concerning the unboundedness or boundedness of Mordell-Weil ranks of elliptic curves over $\Bbb Q$?), the consensus became that there was no bound for the rank. But recently there have been heuristic arguments that have convinced many people that the correct conjecture is that there is a uniform bound for the rank. Indeed, something like: Conjecture There are only finitely many $E/\mathbb{Q}$ for which the rank of $E(\mathbb{Q})$ exceed 21. (Although there is one example of an elliptic curve of rank 28 due to Elkies.)

$\begingroup$As is also the case for some other examples discussed, this instance could eventually become an example of a tide that turned (at least) thrice :)$\endgroup$
– NellMar 16 at 22:55

1

$\begingroup$@StanleyYaoXiao You're correct, I'm just being my usual imprecise self. At present, there seems to be no hope of doing any sort of descent to show that the rank is exactly 28.$\endgroup$
– Joe SilvermanMar 16 at 23:30

Specifically, the link says “By the late 1960s, many were doubtful of the Calabi conjecture”, then Yau did “produce a "counterexample" to the conjecture. The "counterexample" was soon discovered to be flawed”, finally in 1976 Yau proved the conjecture. More details in Yau’s autobiography:

(pp. 76–77, 1971): Calabi had proposed a systematic strategy for constructing a vast number of manifolds endowed with special geometrical properties of which we’d never seen a single example. (...) Hitchin and I, along with many others, considered Calabi’s conjecture “too good to be true.”

(p. 86, 1972–73): with the French mathematician Jean-Pierre Bourguignon [we] tried out various approaches that might lead to the identification of a counterexample to the Calabi conjecture.

(p. 90, August 1973): I mentioned that I’d come up with a seemingly robust counterexample or two. Word got around, and I was asked to give an informal presentation (...) By the end of this session, most people left the room with the sense that I had proved Calabi wrong.

(p. 95, Fall 1973): Calabi (...) upon reflection had found some aspects of it puzzling. (...) When I went through the (...) counterexamples I’d been considering, one by one, they fell apart (...) So now I had to reverse my course 180 degrees and pour my efforts into proving that Calabi had been right all along.

(p. 104, Fall 1975): I was making steady progress (...) The proof, as I structured it, rested on four separate estimates to the critical complex Monge–Ampère equation.

(pp. 109–110, Fall 1976): I holed up in my study for as long as I could, pouring all of my energy into the Calabi conjecture. Within a week or two, the zeroth-order estimate had been completed and, consequently, the problem as a whole had been completed too.

$\begingroup$(Francois Ziegler is the OP. I merely incorporated a comment he made into the answer.)$\endgroup$
– Todd Trimble♦Mar 17 at 12:16

$\begingroup$The "Yau's autobiographical account" links to wikipedia, and you have to click the link which brings you to wayback machine to download a .ppt. Is there a more stable link to this? It seems weird to link to wikipedia to link to an outside website, as the wikipedia page could change or be reorganized.$\endgroup$
– Paul PlummerMar 18 at 18:45

It's a geometric conjecture (well, a "question" but I think it's easier to follow if I call it a conjecture) that is true in 2 dimensions and everyone expected it to hold in all higher dimensions, but things turned out differently with a surprise ending. Over time counterexamples were found to the conjecture in every dimension above 5, then it was proved to be true in dimension 3, then a counterexample was found in dimension 4, then the counterexample in dimension 4 was shown to be an example rather than a counterexample, and then it was proved to be true in dimension 4 by the same person who earlier gave the "counterexample" in dimension 4, with both of his papers appearing in the Annals.

In 1923 Dulac "proved" that every polynomial vector field in the plane has finitely many cycles [D]. In 1955-57 Petrovskii and Landis "gave" bounds for the number of such cycles depending only on the degree of the polynomial [PL1], [PL2].

Coming from Hilbert, and being so central to Dynamical Systems developments, this work certainly "built a small industry". However, Novikov and Ilyashenko disproved [PL1] in the 60's, and later, in 1982, Ilyashenko found a serious gap in [D]. Thus, after 60 years the stat-of-the-art in that area was back almost to zero (except of course, people now had new tools and conjectures, and a better understanding of the problem!).

Abel first thought that he had solved the general quintic by radicals. Then he found a mistake and subsequently he proved that it was impossible to solve the quintic algebraically. The famous and notoriously difficult problem about the pointwise convergence almost everywhere of L2-functions, which Lusin formulated in 1913 and actually goes back to Fourier in 1807, was solved by you in the mid-1960s. We understand that the prehistory of that result was converse to that of Abel’s, in the sense that you first tried to disprove it. Could you comment on that story?

Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).