The problem seems to be that there is a rep:policy node in the target language tree, which is correct, because we don't want local editors to edit this part of the content. If I removed that node (which wasn't that easy before I found out that I could rename it, and then remove it) the rollout was successful. But we really want to set access rights here, so this is not a solution. Is this a bug in CQ5?

Yes, it looks like a permission issue, but since I'm doing the rollout as admin, which has full permissions to all pages in both source and target languages, it shouldn't be an issue, so I think MSM is doing something wrong here. It doesn't help if I remove all Deny-nodes under the rep:policy node, it still won't work. Only if I remove the rep:policy node itself the rollout will work. But this means that we can't set any detailed permissions at all in the target languages content trees (because that creates rep:policy nodes which breaks the rollout).

"Resource-based ACLs are stored per resource/node in a special child node rep:policy... Note that you can read/browse these nodes using the JCR API, but cannot modify them. This must always happen through the JCR access control API."