Access to the file manager is controlled by the Site Settings > Security > Permissions > Roles That Can Delete Files in the Editor.
The idea is that since the file manager allows deleting files access to it is determined by delete permissions.
I will change it so that the default is to include Content Administrators as an allowed role there, but the change will only affect new installations.
In any case you can easily add Content Administrators role (and any others you want to grant access to) to the allowed delete roles.
Hope it helps,