There are good trees and there are bad trees.
If you haven't made your way through the sphynx pyramid and into the Lost Sea area you haven't yet met any of the bad trees. Bad trees won't like you. Bad trees will attack. Bad trees will leave a seed behind when they die.