This question could be vastly improved. Please add more background and details about what you're interested in and what you already know and what you'd like to know. Please also put a version of the question in the title. In fact, "What mathematical treatment is there on the renormalization group flow in a space of Lagrangians?" would make for a perfectly good title, and then you could have a few paragraphs of question. See mathoverflow.net/howtoask .
–
Theo Johnson-FreydApr 23 '11 at 21:07

5 Answers
5

The answer to your question depends on whether you are interested in the perturbative RG
or the nonperturbative one.

Typically one starts with a Gaussian measure $d\mu_{0,\infty}$ on a space of fields $\phi$
given by a covariance
$$C_{0,\infty}(x,y)=\int \phi(x)\phi(y)\ d\mu_{0,\infty}(\phi)\ .
$$
For instance one can take for the covariance $\frac{1}{\xi^2}$ in Fourier space.
Then one introduces a UV regularization at length scale $l$ by multiplying
for instance by $\exp(-l^2 \xi^2)$ which cuts-off momenta $\xi$ which are larger than $l^{-1}$. This defines
$$
C_{l,\infty}(x,y)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}
\frac{e^{-l^2\xi^2}}{\xi^2} e^{i\xi(x-y)}\ d^d\xi\ .
$$
The RG is used in order to study quantities of the form
$$
\int e^{-V(\phi)}\ d\mu_{l,\infty}(\phi)\ .
$$
The idea is to use a ``rescaling to unit lattice'', i.e., a scaling change of variable
so one has an integral as before with $l=1$ (with a different $V$ that I will still call
$V$ to keep notations simple).
Then one uses a decomposition of Gaussian measures
$$
\int e^{-V(\phi)}\ d\mu_{1,\infty}(\phi)
=\int \int e^{-V(\psi+\zeta)} d\mu_{1,L}(\zeta)d\mu_{L,\infty}(\psi)
$$
where $d\mu_{1,L}$ is the Gaussian measure corresponding to the covariance
$C_{1,L}=C_{1,\infty}-C_{L,\infty}$ and $L$ is some number $>1$.
If one defines the constant $[\phi]=\frac{d-2}{2}$, called the scaling dimension
of the field, then
the law of the field $\psi(x)$ is the same as
that of $\phi_L(x)=L^{-[\phi]}\phi(L^{-1}x)$
where $\phi$ is sampled according to the original measure
$d\mu_{1,\infty}$. Hence
$$
\int e^{-V(\phi)}\ d\mu_{1,\infty}(\phi)
=\int \left(\int e^{-V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)\right)d\mu_{1,\infty}(\phi)
$$
$$
=\int e^{-V'(\phi)}\ d\mu_{1,\infty}(\phi)
$$
where
$$
V'(\phi)=-\log\left(
\int e^{-V(\phi_L+\zeta)} d\mu_{1,L}(\zeta)
\right)\ .
$$
The renormalization group transformation on the space of Lagrangians
is the map $V\rightarrow V'$.
One can also do this infinitesimally by taking $L\rightarrow 1$, in which case
one talks about an RG flow rather than a transformation.
In the perturbative RG one writes the dynamical variable $V$ which is a complicated
functional of the field as a formal power series in some variable which you can think of as Planck's constant.
In the nonperturbative RG one essentially wants to use analysis to control the sum of this series. There are rigorous ways to study both RGs. The perturbative one is of course
much simpler.

What you will find in Costello's book is only the perturbative RG. He does treat
Yang-Mills in flat space using the Batalin-Vilkovisky formalism, which is quite
remarkable for an introductory
book. For the curved case, see the paper
http://arxiv.org/abs/0705.3340
by S. Hollands which appeared in J. Math. Phys.

If you would be happy learning about the RG flow on $\phi^4$ instead of Yang-Mills,
then much simpler perfectly rigorous presentations are available:

the book "Renormalization: an introduction" by Manfred Salmhofer, Springer, 1999.

As for the rigorous nonperturbative RG, the Park City lectures by Brydges mentioned by jc is definitely the best place to start.
The issue here is that for Bosons one cannot really take the log in the definition
of $V'$. This is called the large field problem, and one algebraic way around
it is to use a so-called polymer representation. All this is explained by Brydges.
Another nice introduction to the nonperturbative RG for Bosons is a set of (preliminary) lecture
notes by Antti Kupiainen (you can find them on Google
if you search for "Introduction to the renormalization group kupiainen").

For Fermions,
taking the log is not a problem and good mathematical presentation can be found, e.g.,
in:

the book "Non-perturbative renormalization" by Vieri Mastropietro, World Sci. 2008.

the book "Renormalization group" by Giuseppe Benfatto and Giovanni Gallavotti, Princeton University Press, 1995.

If you would like a very short account of the kind of theorems one would like to prove
in the nonperturbative RG setting you can also look up my recent Oberwolfach extended abstract: http://arxiv.org/abs/1104.2937

A small complement to Abdelmalek Abdesselam's answer: on the rigorous, non-perturbative side, there is also a recent (originally two-part, now turned into three-part) exposition by Jonathan Dimock, available in the arXiv's. He uses the $\phi^4$ scalar field theory in 3 dimensions in finite volume as a model for his discussion - the three parts are listed below:

Tadeusz Balaban refined the method of block-spin renormalization group employed by Gallavotti, Kupiainen and many others for lattice field models in order to analyse "large field" regions, aiming at the treatment of the continuum limit of pure Yang-Mills models in finite volume and 4 dimensions. His long series of papers on the subject from the 80's remain essentially the state of the art towards the rigorous construction of realistic models in quantum field theory in 4 dimensions (see, for instance, the latest of the series), together with the paper of Magnen, Rivasseau and Sénéor, which was motivated by Balaban's work. The third part of Dimock's exposé is meant to establish the convergence of the expansion scheme laid down in Parts I and II.

David C Brydges and his collaborators have been using renormalization inspired ideas to prove theorems about statistical mechanical systems for a few years now. From his page:

A large part of theoretical physics is built around the “functional integral” formulation of quantum field theory. These functional integrals are defined in the sense of formal power series (renormalised perturbation theory). It is widely, but wrongly believed, by mathematicians, that no precise definition that is useful for rigorous analysis is within sight. The renormalization group (RG), as pioneered by Ken Wilson (Nobel prize in Physics, 1982), provides a clear roadmap for defining functional integrals and studying the remarkable range of phenomena contained within them, in particular, renormalisation, scaling limits and the phase transitions of statistical mechanics. In these cases one can work with integrals based on measures on spaces of functions as opposed to complex valued "measures" on spaces of functions. The complex valued case (Feynman functional integrals) is indeed further toward the horizon of difficulty. Without facing the difficulties of the complex valued case, there is already an enormous range of possible applications. My interests in recent years have been in applications to self-avoiding walk in four dimensions. Functional integrals combine with supersymmetry to generate combinatoric identities so whenever I need a rest from the RG I like to think about that aspect as well. The papers below are a mixture of themes involving supersymmetry and analysis by RG. My colleague Joel Feldman is using closely related ideas to prove results in the context of condensed matter physics.

A good place to start reading about this line of research is his lecture notes from a 2007 PCMI summer conference on Statistical Mechanics. They are available for download from his webpage above, or in printed form as part of the proceedings from said conference.

Langlands has thought about field theories for some time and has written a paper about renormalization group fixed points. The paper can be found among his collected works at his IAS website. The section on mathematical physics is located at this URL
http://publications.ias.edu/rpl/section/28.