Hey Everyone,

I'm hoping this is the correct area to post this... I'm running into a problem were the pages will not serve up and I get a page from my hosting company (Rackspace) that there are "No Suitable Nodes to serve up request", I opened a ticket up with them and below is what I got back.

I replyed back but have not received an answer yet.  So, I'm asking to see if someone knows why I may be getting this, it started to happen more with the latest release of MojoPortal ( 2.3.5.8b MySQL).  Here's the location of the latest log file:  http://test.dsatallahassee.org/logs/mojoportal-log-20110125193841.txt

I have a great deal of sites that I switch over to MojoPortal and the latest code and they are all having this problem.  Please help if you can.

Below is the Reply:

Hello Mark,

"No suitable nodes" is an error that occurs as a result of a script exceeding the maximum timeout value. If a client connection does not receive a response from the server after approximately 30 to 60 seconds the load balancer will close the connection and the client will immediately receive the error message.

In most cases, the script will continue to execute until it reaches completion, throws an error, or times out on the server, but the client will not see the page load as expected and will instead receive this error.

For longer running scripts in PHP you can use the 'exec' function to run scripts in the background and have the script write its status to a database or a file. Then you could use AJAX to display a loading bar and check the script's status.
Once it's completed you can then remove the loading bar and proceed to a completion page. This is just an example, but the concept is good for anything. Another option is to run the script as a cron job using PHP or Perl instead of HTTP, which circumvents the load balancer's timeout.

If you can tell us when the error is occurring we can do our best to evaluate whether there was a problem on our end that caused the error. If you're seeing timeouts intermittently I would recommend auditing your code, because that likely means it could use optimization or needs to be broken down into smaller pieces. For instance, if you were inserting several records into a database, try inserting a few at a time and then sending back a response to the client's browser.

The bottom line when it comes to code causing this error is that data must be sent back to the client's browser to keep the connection alive So regardless of what sort of processing a page is doing it must return data frequently or the load balancer will assume the connection has been dropped and will forcefully terminate it.

Another common cause is that your site is trying to load files that don't exist (404 errors). This dramatically slows a site down and in rare cases can cause a timeout. More commonly a site that is loading data from an external location can see load issues causing a timeout.

For instance, if your site relies on Google Analytics, Authorize.net, or PayPal and for some reason these corresponding services go down or begin responding slowly, your site will see a performance hit that in some cases can cause the page not to load or to load intermittently. Keep in mind that all sorts of plugins for content management systems like Wordpress and Joomla use third-party services behind the scenes, such as bit.ly and Twitter.

It's also a possibility that the script simply uses a heavy amount of resources. If this is a PHP site you may need to try increasing the memory limit--which defaults to 32M--to something more appropriate for your site, such as 64M or even 128M in extreme cases.
There is a knowledge base article on that here:
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#How_do_I_change_the_PHP_memory_limit_value.3F

If you're not familiar with using an .htaccess file--as the article above requires--you may take a look at these articles:
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#What_is_a_.htaccess_file.3F
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#How_do_I_use_a_.htaccess_file.3F

In extremely rare cases you can see this message due to an invalid cookie being stored by your browser, which will cause you to see the error message on pages that initiate a session on your site (such as login or member pages--or sometimes even your homepage). All though rare, it's a good idea to clear your browser cache and try the page again to verify that this is not the case.

Please let us know if there's anything else we can assist you with!

Regards,
Matt Wheeler
Technical Support
The Rackspace Hosting Cloud

Hey Joe,

More information from Rackspace below.  I'm installing the MS SQL version of the code right now, I'll post on how that goes... I'm hoping that all this information helps other folks with mojoPortal if they are with Rackspace or a different hosting company that may have the same issues.

Hello again,

No suitable Nodes message is the result of a timeout error. With hundreds of thousands of websites on our system and this message being in place for over a year or two, It's not really a surprise.

However NSN's in no way point out a particular problem. It could be anything that happens when you make a request through the load balancer and your website does not respond fast enough. After 30 seconds you will get that message regardless of the actual problem. And the not knowing, and frustration of not knowing how to look into it is what spurs people to make so many comments about this.

But I have exhausted everything I could to help you out here. For test.dsatallahassee.org:

Average Response Time across the nodes in your IIS cluster is 0.03 seconds  after hitting the front page multiple times across each node I was not able to reproduce the error. (eliminating the possibility that only one node was bad) 

It did seem at first that the website had not been accessed in a while. IIS will unload the application and this can take several seconds to reload. So those first few hits will take a long time (sometimes up to 30 seconds causing NSN's) But I assume you know about this and you are talking about after you have been browsing around for a while(without switching browsers). If you switch browsers you will often hit a new node however if not then you will almost always be talking to only one node.

Currently when doing operations against your mysql node it only takes: 0.182 seconds for mysql50-58 for us to
- Connect to your MySQL server successfully.
- Connect to your database successfully.
- Creat table
- Insert row
- Select row
- Drop table

I know asp and .net is working because your site is showing up regularly along with the hundreds of other websites on that IIS cluster. I am pretty sure this does not have to do with the physical connection to your mysql database because we have monitoring setup for that which is reporting nothing outrageous going on. However we can put up a script to test that connection if you so desire to see it for yourself.

I would love to investigate this further with you, I just don't have anything to go off of without your help here. I am literally out of ideas because I don't know how the software on your website works. So you may have to get a developer to take a look at the site to see if you can catch more information about the error or locate the cause of the slowness.

If you get any more information or steps to replicate the issue let us know here. We can take that information and probably find out what's going on here. I can even send you over to our L2 Windows tech here. But right now i don't have anything to give him.

Gene A. Guido III
Linux Administrator 1
Rackspace Cloud

Hey Joe,

I did, however I just added your questions again.

Just as a side note, I just installed the MS SQL version of mojo and so far I have not ran into those problems... I have a tester testing it now to see if that is really true.

Now, one thing I have noticed... When loading sql.dsatallahassee.org in IE 7, 8 or 9 it never completely loads, it always says there one more thing to load.  But in Chrome and FireFox it says it loads just fine.  So, I have no clue what that is.

I'll keep you posted.

Hey Joe,

By marking the 'DisableSearchIndex' to true has helped a great deal.  I'm still getting a few NSN errors when login as admin and doing updates.

Question for you, in the web.config there is the following lines:

    <!-- comment this out for Mono -->
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />

When it says for Mono, does that mean when running only one site?  Also, do you think this may create an issue in a medium trust cloud hosting?

Thanks,

Mark

Hey Joe,

Some new information from Rackspace:  When I get the No Suitable Nodes message they have found that the code is locking one of the worker processes.  They are still researching it, but wanted you to know.

Mark

Hey Joe,

Wanted to give you an update, I turned off the keepalive in the config and added the web service you talked about.  However after I removed the keepalive and added the web service I started to receive the following on three of the pages:  (it has stopped, just does not make since)

It was happening on the contact-us, and the sign in page.

Server Error in '/' Application.
------------------------------------

--------------------------------------------

Execution permission cannot be acquired.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Security.Policy.PolicyException: Execution permission cannot be acquired.

Source Error:

The source code that generated this unhandled exception can only be shown when compiled in debug mode. To enable this, please follow one of the below steps, then request the URL:

1. Add a "Debug=true" directive at the top of the file that generated the error. Example:

  <%@ Page Language="C#" Debug="true" %>

or:

2) Add the following section to the configuration file of your application:

<configuration>
   <system.web>
       <compilation debug="true"/>
   </system.web>
</configuration>

Note that this second technique will cause all files within a given application to be compiled in debug mode. The first technique will cause only that particular file to be compiled in debug mode.

Important: Running applications in debug mode does incur a memory/performance overhead. You should make sure that an application has debugging disabled before deploying into production scenario. 

Stack Trace:

[PolicyException: Execution permission cannot be acquired.]
   System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Boolean checkExecutionPermission) +7606504
   System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Int32& securitySpecialFlags, Boolean checkExecutionPermission) +57

[FileLoadException: Could not load file or assembly 'Brettle.Web.NeatHtmlTools, Version=1.0.2979.14781, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Failed to grant permission to execute. (Exception from HRESULT: 0x80131418)]
   System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection) +0
   System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection) +43
   System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) +127
   System.Reflection.Assembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) +142
   System.Reflection.Assembly.Load(String assemblyString) +28
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +46

[ConfigurationErrorsException: Could not load file or assembly 'Brettle.Web.NeatHtmlTools, Version=1.0.2979.14781, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Failed to grant permission to execute. (Exception from HRESULT: 0x80131418)]
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +613
   System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory() +203
   System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai) +105
   System.Web.Configuration.AssemblyInfo.get_AssemblyInternal() +29
   System.Web.Compilation.CompilationUtil.GetTypeFromAssemblies(AssemblyCollection assembliesCollection, String typeName, Boolean ignoreCase) +132
   System.Web.Compilation.BuildManager.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase) +291
   System.Web.UI.Page.get_ScriptManagerType() +44
   System.Web.UI.Page.get_IsPartialRenderingSupported() +61
   System.Web.UI.WebControls.ValidationSummary.OnPreRender(EventArgs e) +257
   System.Web.UI.Control.PreRenderRecursiveInternal() +80
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Control.PreRenderRecursiveInternal() +171
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +6741
   System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +242
   System.Web.UI.Page.ProcessRequest() +80
   System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context) +21
   System.Web.UI.Page.ProcessRequest(HttpContext context) +49
   ASP.default_aspx.ProcessRequest(HttpContext context) +4
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +181
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75

--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:2.0.50727.4206; ASP.NET Version:2.0.50727.4209 

First let me address Rick... I was not saying mojoPortal was the issue, Joe and I have been working back and forth on this issue and he asked that I supply him with all information that came out of my research with Rackspace.  I agree with you Rick, Joe is the best at address something that is truly a mojoPortal issue, if one is found.

Second,  Joe...

Thank you for your information about the web farms, I like being on the cutting edge, ok may be not, but right now Rackspace is what I have, currently there is no place that can bet the pricing I have with them for what I receive, and I do get great support with them, they have not closed the ticket on the issues and are still working with me on a daily bases to find a solution.

Thanks again and like always will keep you posted on the updates.

Mark