It turns out that Groot didn't get along with his fellow saplings back on his homeworld, preferring the company of their squirrel-like "Maintenance Mammals" instead. He even attacked a sapling who was threatening one of these animals.