The key relation for functional differentiation is
$$\frac{\delta}{\delta f(y)}f(x)=\delta(x-y),
$$
where $\delta(x-y)$ is the Dirac delta function, and the usual properties of differentiation (e.g. linearity, chain rule) still hold.

I think you need to use Cases[expr, f[x_] :> x, {0, Infinity}] // Union in order to retain only the distinct instances of f[x_]. For instance, FunctionalD[f[x] + g[f[x]], f[y]] gives twice the expected result doing it your way.
–
Stephen LuttrellNov 13 '12 at 21:22

The other thing I completely forgot -- it's been a while since I've had to functional differentiate -- is that one should have e.g. FunctionalD[f'[y],f[x]] = -DiracDelta'[y-x]. That could probably be fixed by supplementing the pattern in the Cases to include derivatives of f, and include an corresponding appropriate derivative of the DiracDelta in the Map.
–
Jeremy MichelsonNov 14 '12 at 2:03

I had to downvote this because it returns incorrect results as you notice yourself in the previous comment.
–
JensNov 15 '12 at 4:45

@Jeremy: As far as I'm concerned the rule for the derivative is not overly important (in many cases one would simply get rid of the derivative through a Fourier transform).
–
EmersonNov 15 '12 at 21:46

Mathematica already knows quite a lot about functional derivatives. In particular, you can do variational derivatives. That is, you have to give it the functional and the function (I would strongly suspect that your problem can be written so as to use the VariationalD function). To get started, have a look at the tutorial for the Variational Methods package.

Edit:
Originally I had a definition based on the Variational Methods package here, but the following method is more transparent, so I removed the original attempt (which also used a different syntax).

Edit 2

I just came across another approach to defining the functional derivative without recourse to the VariationalMethods package. It's directly from the documentation for DiracDelta (under "Applications"):

@Emerson We can easily handle multiple arguments as soon as you understand what they are supposed to mean in terms of the variational approach. The new rule you added in your edit simply means you're varying f at y in a product of two functionals, one with x1 and the other with x2 as the integration variable. After applying the regular product rule to that, you're back to the FunctionalD in my answer. Of course one could try to automate the regular differentiation parts too - but the core of the calculation is always what I described already.
–
JensNov 16 '12 at 4:30

@chris Sure, Mathematica probably can't do the limit in that case. But that's still better than getting a wrong result for that case, as you would with the accepted answer. Anyway, if you have an integral, are you sure you don't want the variational derivative instead of the functional derivative?
–
JensAug 24 '14 at 16:51

Mathematica is a registered trademark of Wolfram Research, Inc. While the mark is used herein with the limited permission of Wolfram Research, Stack Exchange and this site disclaim all affiliation therewith.