It is a little confusing but it does work as intended.
Users in roles that can upload files can upload files in the editor while editing content, such as through the image dialog or the link dialog which allows browsing the server and uploading files.
In order to have access to the File Manager a user must also be in Roles Allowed to Delete Files in the Editor because the File Manager also allows deleting files.
You can get some extra protection by adding users to Roles that can upload and browse but only in a user specific folder instead of adding them to Roles that Can Browse and Upload, because then at least they are limited to only being able to delete files they uploaded themselves.
Hope it helps,