Most likely cause of this problem is if the Data folder is not writable. We have cache depndency files we create there and then we touch tyhe file to clear the cache, so if it can't create or touch the fie it can't clear the cache so then only recycle the app can clear the cache.
SiteSettings is cached in memory and has the skin seting for the site. When you change skins or save site settings it tries to touch the dependency file. If the file is touched it will clear the cache and therefore site settigns will be reloaded with the new changes.
Hope it helps,