7 Answers
7

Gauss sums are not an ad-hoc construction! I know two ways to motivate the definition, one of which requires that you know a little Galois theory and the other which is totally mysterious to me.

Here is the Galois-theoretic explanation. Let $\zeta_p$ be a primitive $p^{th}$ root of unity, for $p$ prime. The cyclotomic field $\mathbb{Q}(\zeta_p)$ is Galois, so one can define its Galois group, the group of all field automorphisms which preserve $\mathbb{Q}$. Such an automorphism is determined by what it does to $\zeta_p$, and it must send $\zeta_p$ to another primitive $p^{th}$ root of unity. It follows that the Galois group $G = \text{Gal}(\mathbb{Q}(\zeta_p)/\mathbb{Q})$ is isomorphic to $(\mathbb{Z}/p\mathbb{Z})^{\times}$, which is cyclic of order $p-1$.

Now suppose $p$ is odd. As a cyclic group of even order, $G$ has a unique subgroup $H$ of index two given precisely by the multiplicative group of quadratic residues $\bmod p$, so by the fundamental theorem of Galois theory the fixed field $\mathbb{Q}(\zeta_p)^H$ is the unique quadratic subextension of $\mathbb{Q}(\zeta_p)$. And it's not hard to see that this unique quadratic subextension must be generated by

which you will of course recognize as a Gauss sum! So the Gauss sum generates a quadratic subextension, and any of various methods will tell you that this subextension is precisely $\mathbb{Q}(\sqrt{p^{\ast}})$ where $p^{\ast} = (-1)^{ \frac{p-1}{2} } p$. (This does not actually require any computation: if you know enough algebraic number theory, it follows from a consideration of which primes ramify in cyclotomic extensions.)

The totally mysterious explanation is that Gauss sums naturally appear when you start thinking about the discrete Fourier transform. For example, the trace of the DFT matrix is a Gauss sum. But more mysteriously, Gauss sums are eigenfunctions of the DFT in a certain sense. (I sketch how this works here.) There is a sort of mysterious connection here to the Gaussian distribution, which is an eigenfunction of the continuous Fourier transform; see this MO question. Again, I don't know what to make of this. There is a book by Berg called The Fourier-analytic proof of quadratic reciprocity and it may or may not be about this construction.

Not just quadratic reciprocity, one can use them to prove higher
reciprocity laws: see Ireland and Rosen's A Classical Introduction
to Modern Number Theory. They also turn up in the functional
equation for Dirichlet L-functions (and are massively generalized in the
topic of root numbers).

For example, if one considers $x = (x_1, \dots , x_d) \in \mathbb{F}_q^d$ and defines $f(x) = x_1^2 + \dots + x_d^2$, then $A_t$ would be some finite field analogue of a sphere. Bounding such a set would then be equivalent to bounding

A small additional note, in line with an earlier answer: Gauss sums are, literally, the Lagrange resolvents obtained in the course of expressing roots of unity in terms of radicals. (Yes, then the Kummer-Stickelberger business can be used to effectively obtain the actual radical expressions...: here .)