We are basically using the built in ASP.NET CreateUserWizard and Login, and PasswordRecovery controls. These controls interact with the mojoMembershipProvider which is our implementation of ASP.NET MembershipProvider.
For a custom solution you would be bypassing the membership provider completely. Basic steps you would need to do is.
1. Validate the credentials according to your own methods, a web service or whatever.
2. If valid you would ensure that there is an existing mojoPortal SiteUser corresponding to the login, if not you would create a SiteUser. There is exampel code that does this check and creates users when using Windows Authentication if you look in Web/Components/AuthHandlerHttpModule.cs
3. To log the user in you basically would call FormsAuthentication.SetAuthCookie or FormsAuthentciation.RedirectFromLogin (google for info on those methods.)
Hope it helps,