It is not clear what you mean by "formula". For example, the quadratic form $x^2+y^2+z^2$ is also alone in its genus, but there is no simple way to determine the number of representations $x^2+y^2+z^2=k$. There is a formula involving class numbers and the square part of $k$, but it only connects two subtle quantities, neither of which is simpler than the other.
–
GH from MOJan 23 '12 at 17:30

3

@GH: True, but here the number of variables is even, so the theta function is a modular form has integral weight and the formula for its coefficients should be reasonably simple.
–
Noam D. ElkiesJan 23 '12 at 17:42

Thank Noam, I had not been able to explain it. Do you know some reference in where this formula "could be"?
–
emiliocbaJan 23 '12 at 18:15

1

@emiliocba: As usual I don't know a reference, and it would be easier to (re)construct the formula than to locate it in the literature. If you already know the formula it's just a matter of checking that it gives rise to a modular form in the appropriate space and that this space is small enough that there a unique candidate form. If not, itshould be possible to surmise the formula from the first few dozen coefficients of the theta function $\left(\sum_{x,y\in\bf Z} q^{x^2+xy+y^2}\right)^3$.
–
Noam D. ElkiesJan 23 '12 at 22:49

1

@GH There are no cusp forms here. In any case I think if the form is unique in its genus the theta function must be in the Eisenstein subspace.
–
Noam D. ElkiesJan 24 '12 at 15:51

3 Answers
3

The formula that emiliocba seeks seems to be as follows.
Let $\chi$ be the Dirichlet character mod $3$. For $k>0$
write $k = 3^e n$ with $n \equiv \pm 1 \bmod 3$. Then
the number of representations of $k$ by this quadratic form $A_2^3$ is
$$
s(k) :=
9 (3^{2e+1}-\chi(n)) \phantom. \sum_{d|n} \phantom. \chi(n/d)\phantom. d^2.
$$
I append gp code that verifies that this holds for $k \leq 432$.

To prove it in general it will be enough to check that
$$
\varphi := 1 + \sum_{k=1}^\infty \phantom. s(k) q^k
$$
is a modular form of weight $3$ and character $\chi$ for $\Gamma_0(3)$,
and to match a few coefficients with the theta function $\theta_{A_2^3}$.
In principle, it is enough to match only the $q^0$ coefficient:
the dual of $A_2^3$ is isomorphic with the scaling of $A_2^3$ by $1/3$,
so by Poisson summation $\theta_{A_2^3}$ is modular also for
the normalizer $\Gamma_0^+(3)$ of $\Gamma_0(3)$ (generated by $\Gamma_0(3)$ and
the involution $w_3 : \tau \longleftrightarrow -1/3\tau\phantom.$);
and $\Gamma_0^+(3)$ has only one cusp,
and no cusp forms of weight less than $6$ (the weight of
$\eta(\tau)^6 \eta(3\tau)^6$), so the normalized Eisenstein series $\varphi$
is the only candidate for $\theta_{A_2^3}$.

This is a supplement to Noam Elkies' nice answer. The coefficients $s(k)$ can be expressed as
$$ s(k)=27\sum_{d\mid k}\chi(k/d)d^2-9\sum_{d\mid k}\chi(d)d^2, $$
hence the function $\varphi$ is a linear combination of
$$E_1:=\sum_{k=1}^\infty\sum_{d\mid k}\chi(k/d)d^2q^k
\quad\text{and}\quad
E_2:=1-9\sum_{k=1}^\infty\sum_{d\mid k}\chi(d)d^2q^k.$$
The latter functions are proportional to the standard Eisenstein series
$$ E_1':=\sum'_{m,n\in\mathbb{Z}}\chi(m)(mz+n)^{-3}
\quad\text{and}\quad
E_2':=\sum'_{m,n\in\mathbb{Z}}\chi(n)(mz+n)^{-3},$$
which form a basis of the space of modular forms $M_3(\Gamma_0(3),\chi)$, hence indeed $\varphi$ lies in this space. For more details see Section 7.1 in Miyake: Modular Forms, especially Lemma 7.1.1 and Theorem 7.1.3.

Let's recall that the number $R(k)$ of representations of $k$ as $x^2+y^2$ can be written as follows: write $k=2^\alpha bc$ where $b$ is composed entirely of primes congruent to 1 (mod 4) and $c$ is composed entirely of primes congruent to 3 (mod 4). Then $R(k)=0$ unless $c$ is a square, in which case $R(k) = 4\tau(b)$, where $\tau(b)$ is the number of divisors of $b$.

A very similar proof would surely address the number $S(k)$ of representations of $k$ as $x^2+xy+y^2$: write $k=3^\alpha bc$ where $b$ is composed entirely of primes congruent to 1 (mod 3) and $c$ is composed entirely of primes congruent to 2 (mod 3). Then I believe that $S(k) = 0$ unless $c$ is a square, in which case $S(k) = 4\tau(b)$. (Or maybe it's $6\tau(b)$.) I guess we should also mention $S(0)=1$.

In your original question, the number of representations of $k = x^t Qx$ where $x\in{\mathbb Z}^6$ will be exactly the triple convolution $\sum_{m=0}^k \sum_{n=0}^{k-m} S(m)S(n)S(k-m-n)$. ($Q$ is positive definite so we needn't worry about negative integers.) This probably leads to a rather different-looking formula than one would get from modular forms.