2 Answers
2

Yes, suppose we have in polar coordinates $f(r,\theta) = (r,\theta + r/(1-r))$. In effect the disk is rotated by increasing amounts as $r$ tends to 1. It is still a homeomorphism on the interior of the unit disk, but it's impossible to extend continuously to the boundary.

Added: This map cannot be extended continuously to any point of the boundary. If it could be, then for any "target" open neighborhood of the image of that point, we should be able to find an open neighborhood of the point whose image is contained in the "target". But the open neighborhood of any boundary point will be mapped to points arbitrarily close to every point on the boundary (and so cannot be contained in any sufficiently small "target").

In a similar vein, take polar coordinates where $r\in [0,1)$ and $\theta \in (-1,1]$ (so re-scale the angle by $\pi$). Then the map $\theta \to \mathop{sign}(\theta)|\theta|^{1/(1-r)}$ is a continuous map from $(-1,1]$ to itself, except when $r = 1$. The singularity is slightly different: here the entire boundary, except for $(-1,0)$ gets bunched up at $(1,0)$.
–
Willie WongDec 15 '10 at 15:08

Let $H=\{(x,y)\in\mathbb R^2:y>0\}$ be the upper half plane in $\mathbb R^2$ and let $\bar H$ be its closure. Let me give you an homeo $f:H\to H$ which does not extend to an homeo $\bar f:\bar H\to\bar H$, and which fixes the point of infinity on the $x$-axis. Then you can conjugate with an homeo $D\to H$ from the open unit disk which extends to the boundary (apart from one point...)

Let $\phi:\mathbb R\times(0,\infty)\to\mathbb R$ be given by $\phi(x,t)=\frac1{\sqrt{4\pi t}}\exp\left(-\frac{x^2}{4t}\right)$, and let $h:\mathbb R\to\mathbb R$ be a positive smooth function with support on $[-2,2]$ and such that it is constantly $1$ on $[-1,1]$. Define now $g:H\to H$ so that when $x\in\mathbb R$ and $y>0$ we have $$
g(x,y)=\int_{-\infty}^\infty\phi(x-\xi,y)(1-h(\xi))\,\mathrm d\xi.
$$ This is a solution of the one-dimensional head equation. Properties of that equation (which one can easily show in this case!) imply that $f$ extends continuously to a map $\bar g:\bar H\to\mathbb H$.

Now consider the map $f:(x,t)\in H\mapsto (xg(x,t),t)\in H$. This is an homeomorphism, and it extends to the map $\bar f:(x,t)\in\bar H\mapsto (x\bar g(x,t),t)\in\bar H$, which is not an homeo.