Sounds like the machine key changed since it previously encrypted smtp settings or else you changed from using the legacy crypto helper.
The only way to fix it is to clear those settings out from the database and then you can enter them again correctly from the UI.
SET KeyValue = ''
WHERE SiteID = yoursiteid
AND (KeyName = 'SMTPPassword' OR KeyName = 'SMTPUser')
Hope that helps,