Abaye said: If one throws a bin into the street, [even] if it is ten [handbreadths] high but not six
broad, he is liable; if six broad, he is exempt. Raba said: Even if it is not six broad, he is [still]
exempt. What is the reason? It is impossible for a piece of cane not to project above ten.

Why does Abaye want six tefahim of width to exempt the thrower? Rashi (d"h gavohah) explains that a circle of six tefahim contains a 4-by-4 square, which creates its own reshut:

In other words, to calculate the width of the circle, we need to fit a circle around the square's diagonals, like this:

But Rashi notes that Abaye's figure of 6 is not exact. The Gemara gives the diagonal of a square as 1.4 times the length of a side, which would mean a width of 4 × 1.4 = 5.6 for the circle.

So why did Abaye say 6, not 5.6?

Rashi answers that Abaye was approximating, and rounded up in order to "distance one from a prohibition of Shabbat." Tosafot (the first d"h rehavah) quote Rabbeinu Hannanel as answering that Abaye specifically meant 6.0, and the extra 0.4 above 5.6 is for the walls of the basket. Tosafot reject this because they don't think the walls should be included.

One nitpick: the circle in fact should be 5.65685... wide. That's 4 × sqrt(2), since the diagonal of a square is actually sqrt(2) times a side. The square root of 2 is an irrational number that starts 1.41421..., not the clean fraction 1.4.

It seems that Rashi takes the Gemara's figure of 1.4 for the diagonal at face value. Tosafot were aware that it's "slightly more" than 1.4, as they prove in Sukkah 8a, d"h kol.

Sukkah 8a. It's a fun amud.

The irrationality of a diagonal's length was proven in the 5th century BCE by Hippus. That discovery was a big disappointment for the Greeks, who thought all numbers must be rational, and legend says Hippus was murdered for it. That discovery was also a major milestone in number theory. So it's a shame that the hakhmei Tsarfat seem to not have heard about it.

But the Steinsaltz commentary on today's daf notes that the Rambam was aware of irrational numbers. The Rambam's commentary on Eruvin 1:5, he describes the irrationality of π, or 3.14159..., the ratio of a circle's circumference to its diameter:

It's a number that "we can never say exactly what it is—not by our lack of knowledge, as the fools believe, but because the number is unknown by its nature; it is essentially unknowable. But it can be approximated closely...by the ratio of 1 to 3 and 1/7." So anytime π is needed in the Talmud it's rounded to 3, since it's "only approximate" anyway.

That's a pretty clear understanding that π is irrational. Then, on Eruvin 2:5, the Rambam explains that the square root of 5000 is also irrational:

I can't find anywhere that the Rambam describes a square's diagonal as irrational, but chances are he knew that too. The diagonal of a square does get a shout-out in The Guide for the Perplexed 2:13, but only regarding the diagonal being longer than a side.