Neighbors' kids come over EVERYDAY. Is this common in the States?

We have some relatively new neighbors move in on our cul-de-sac. They have 2 older kids (4 & 9). Mine is 2. They like to come over and play with my kid all the time. I really don't mind as they play really well with my son and they are nice children. But lately, they have been coming EVERYDAY! I have had to send them home a few times because they were eating into our family time. What kind of parents allow their children to go to a neighbor's house everyday and stay as long as they want? Is this common in the States? Where I come from, it's not. And if we want to go some place, we had to call ahead to get permission, etc. In this case, the parents never call. The kids just come at all times of the day and ring the doorbell. I like these kids but hey...I don't run a free babysitting service, you know. Once, I even caught the kids playing in our yard with all my son's toys while we were away!! I know that cos' when I came home, all my son's toys were scattered everywhere. I told the kids they can't come over and play if we are not home. I also mentioned it to the parents. I think it's rude for these parents to keep doing this to us everyday. I feel bad about sending these kids home. I've even suggested to the parents that maybe they only come over a few times a week but that seemed to have fallen on deaf ears. Am I being petty? Should I even be concerned or say anything else to these parents?