Use the fact that the product of the roots is $\displaystyle \frac{k-6}{3}$ and the sum of the roots is $\displaystyle k$ (which you showed), together with $\displaystyle (\alpha +\beta )^2 = \alpha^2 + 2\alpha \beta + \beta^2$.

You can check yourself: from that last equation solve for the numerical value of k, substitute back into the original equation and solve to find the roots and check that the sum of the squares of the roots is 20/3.

And the answer is no, $\displaystyle \alpha^2+\beta^2 \ne (\alpha+\beta)(\alpha+\beta)$, you are told: