I've committed this patch to loosen up our access control slightly. If a module wants to deny view or editing access they can implement hook_field_access() and set the same permissions through that hook. FileField shouldn't be making assumptions about which permissions allow users to edit/view a field.