This problem seems to be happening to some users with Medium Trust installations. I have a solution to prevent this problem for the next release.
For the current release I think you could work past it if you add this to Web.config:
<add key="InitialSitePasswordFormat" value="0" />
then you will need to clean out all objects from the db and then visit /Setup/Default.aspx again.
This will make the initial password format clear text and avoid calling the crypto helper when it creates the admin user. I think this error is preventing the creation of the admin user.
Hope it helps,