1 Answer
1

To see what is going on, consider some examples, such as $F(x)=x(1-x)$ and $h(x)=\sin x$. What is $\lim_{t\to 0} t^{-1}(F(f+th)-F(f))$ in this case? I think it's $\sin (1/2)$. In general, the limit is $h(x_0)$, not $\max h$.

The proof goes like this: given $\epsilon>0$, pick $\delta$ such that $|h(x)-h(x_0)|<\epsilon$ whenever $|x-x_0|<\delta$. Let $\gamma=f(x_0)-\max\{f(x):|x-x_0|\ge \delta\}$, which is a positive number. For sufficiently small $t$
$$\max\{f(x)+th(x):|x-x_0|\ge \delta\}<f(x_0)-\gamma/2$$
while for $|x-x_0|<\delta$ we have
$$f(x_0)+th(x_0)-t\epsilon < f(x)+th(x) \le f(x_0)+th(x_0)+t\epsilon$$
Hence, $$th(x_0)-t\epsilon\le F(f+th)-F(f)\le th(x_0)+t\epsilon$$ and the conclusion follows.

On the other hand, suppose $f$ attains its maximum at exactly two distinct points $x_1,x_2$. Take $h$ such that $h(x_1)<h(x_2)$. Then the above argument shows that the limit as $t\to 0^+$ is $h(x_2)$ while the limit as $t\to 0^-$ is $h(x_1)$. Hence, the derivative does not exist. The case of general set of maxima $M$ is similar: the one-sided limits are $\max_M h $ and $\min_M h$, which are not equal in general.