I agree with you. Without religion, Homo sapiens are still territorial primates, narcissistic and full of lust.

And given the opportunity, both genders and all races can be pretty brutal. Fortunately, we are moving into an era when women have the chance to be just as mean and exploitational as men, and people of all colors can cheat and steal and exploit other people, ravenously. Just a matter of time.

Without religion I would guess less brutal but also less productive. Religion can be a uniting force among its adherents and the ability for a few people to more easily control large groups of people could have given ancient religious man an advantage.

Once you get to the stage of having developed basic writing and medicine, I think the non religious society would gain the advantage.