The parent directory has the set group id flag, thus the sambashare group owner. The idea is that both samba users and server users belong to the sambashare group, and thus are to be able to edit, delete, and create files and directories. However, since created folders doesn't have the write flag for group set, server users cant create new files or folders in those folders without sudo.

i have tested adding and removing the directory mask, force directory mode, directory security mode, and the force directory security mode, but the behaviors still remains. Newly created files and folders doesn't get intended 774 permission, but rather 764 and 754 respectively.

Yes, that seems to be roughly the same manpage as samba.org/samba/docs/man/manpages-3/smb.conf.5.html which I looked at, the problem is that it doesn't seem to matter what octal values I set, I still get the same permissions set on the created file or folder.
–
ZazJan 23 '12 at 11:14

Did you set all the options I gave you ? If so, please update your first post and at that point I would suggest you file a bug report.
–
bodhi.zazenJan 23 '12 at 16:10

It already said.. but on closer inspection and testing, adding the 2 to the directory masks did fix the problem. Many thanks. : D
–
ZazJan 23 '12 at 18:37

Fantastic, thank you for marking this as the accepted answer, it help others with a similar problem.
–
bodhi.zazenJan 23 '12 at 19:14

After a lot of trial and error, this is the correct code to share samba dir using SGID and unix groups. If user connects anonymously he gets r/o, if he logs in and is a member of assigned group he gets r/w.

I have group named 'admin' set as primary group to users with write privileges, everyone else gets read only rights.

I force user to nobody, so different people working on same files don't interfere with each other.

I set chmod 2755 on shared directory, so it inherits created directories with the same group 'admin'