1. It should open in a colorbox dialog. If you've upgraded from previous versions you may be missing the css for colorbox which would cause the problem you describe. See the sticky thread about important skin changes, specifically the part about colorbox.
2. no, nothing about mojoportal can affect the default document configuration in IIS. However no page should have /Default.aspx configured as its url in Page Settings because that is a real physical page that handles all requests for cms pages. So your home page should be like /home.aspx but because it is the first page in the site hierarchy it should also show that page when the url is /Default.aspx. See How mojoPortal Works for more detail.
Hope that helps,