You need to set the ZFS property 'com.apple.devdisk=on' on the ZFS dataset you want to share for TimeMachine. This setting means that a /dev/disk* entry gets created for that particular dataset, which seems to be enough for OSX to then enable SMB FullSync.

None that I'm aware of! @lundman can surely tell you more, but my theory is that normally a ZFS dataset has no device entry associated with it, so the SMB server doesn't know what device to send the appropriate sync ioctl to. By enabling the devdisk option, it now sees a device and can send the appropriate ioctl. I may of course be totally wrong

Oh, and when you first set the 'com.apple.devdisk=on' property, you won't get a /dev/disk* entry created until the pool has been exported/imported (or the machine rebooted).

The only drawback with devdisk on is that its a bit more complicated, and eats a tiny bit more memory. But we didn't want to leave the default "on" so for those with 100s of dataset, would get 100s of /dev/disk entries.