No Suitable Nodes - Rackspace Hosting

Post here for help with installing or upgrading mojoPortal pre-compiled release packages. When posting in this forum, please provide all relevant details. You may also want to review the installation or upgrading documentation.

If you have questions about using the source code or working with mojoPortal in Visual Studio, please post in the Developer forum.

Post here for help with installation of mojoPortal pre-compiled release packages

When posting in this forum, please try to provide as many relevant details as possible. Particularly the following:

  • What operating system were you running when the bug appeared?
  • What database platform is your site using?
  • What version of mojoPortal are you running?
  • What version of .NET do you use?
  • What steps are necessary to reproduce the issue? Compare expected results vs actual results.

You may also want to review the installation or upgrading documentation.

If you have questions about using the source code or working with mojoPortal in Visual Studio, please post in the Developer forum.

This thread is closed to new posts. You must sign in to post in the forums.
1/25/2011 7:42:03 PM
Gravatar
Total Posts 165

No Suitable Nodes - Rackspace Hosting

Hey Everyone,

I'm hoping this is the correct area to post this... I'm running into a problem were the pages will not serve up and I get a page from my hosting company (Rackspace) that there are "No Suitable Nodes to serve up request", I opened a ticket up with them and below is what I got back.

I replyed back but have not received an answer yet.  So, I'm asking to see if someone knows why I may be getting this, it started to happen more with the latest release of MojoPortal ( 2.3.5.8b MySQL).  Here's the location of the latest log file:  http://test.dsatallahassee.org/logs/mojoportal-log-20110125193841.txt

I have a great deal of sites that I switch over to MojoPortal and the latest code and they are all having this problem.  Please help if you can.

 

 

Below is the Reply:

Hello Mark,

"No suitable nodes" is an error that occurs as a result of a script exceeding the maximum timeout value. If a client connection does not receive a response from the server after approximately 30 to 60 seconds the load balancer will close the connection and the client will immediately receive the error message.

In most cases, the script will continue to execute until it reaches completion, throws an error, or times out on the server, but the client will not see the page load as expected and will instead receive this error.

For longer running scripts in PHP you can use the 'exec' function to run scripts in the background and have the script write its status to a database or a file. Then you could use AJAX to display a loading bar and check the script's status.
Once it's completed you can then remove the loading bar and proceed to a completion page. This is just an example, but the concept is good for anything. Another option is to run the script as a cron job using PHP or Perl instead of HTTP, which circumvents the load balancer's timeout.

If you can tell us when the error is occurring we can do our best to evaluate whether there was a problem on our end that caused the error. If you're seeing timeouts intermittently I would recommend auditing your code, because that likely means it could use optimization or needs to be broken down into smaller pieces. For instance, if you were inserting several records into a database, try inserting a few at a time and then sending back a response to the client's browser.

The bottom line when it comes to code causing this error is that data must be sent back to the client's browser to keep the connection alive So regardless of what sort of processing a page is doing it must return data frequently or the load balancer will assume the connection has been dropped and will forcefully terminate it.

Another common cause is that your site is trying to load files that don't exist (404 errors). This dramatically slows a site down and in rare cases can cause a timeout. More commonly a site that is loading data from an external location can see load issues causing a timeout.

For instance, if your site relies on Google Analytics, Authorize.net, or PayPal and for some reason these corresponding services go down or begin responding slowly, your site will see a performance hit that in some cases can cause the page not to load or to load intermittently. Keep in mind that all sorts of plugins for content management systems like Wordpress and Joomla use third-party services behind the scenes, such as bit.ly and Twitter.

It's also a possibility that the script simply uses a heavy amount of resources. If this is a PHP site you may need to try increasing the memory limit--which defaults to 32M--to something more appropriate for your site, such as 64M or even 128M in extreme cases.
There is a knowledge base article on that here:
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#How_do_I_change_the_PHP_memory_limit_value.3F

If you're not familiar with using an .htaccess file--as the article above requires--you may take a look at these articles:
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#What_is_a_.htaccess_file.3F
http://cloudsites.rackspacecloud.com/index.php/.htaccess_Frequently_Asked_Questions#How_do_I_use_a_.htaccess_file.3F

In extremely rare cases you can see this message due to an invalid cookie being stored by your browser, which will cause you to see the error message on pages that initiate a session on your site (such as login or member pages--or sometimes even your homepage). All though rare, it's a good idea to clear your browser cache and try the page again to verify that this is not the case.

Please let us know if there's anything else we can assist you with!

Regards,
Matt Wheeler
Technical Support
The Rackspace Hosting Cloud

1/25/2011 11:12:49 PM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Well, Rackspace got back with me and their suggestion is below... However their suggestion is already in the web.config. So, I'm still looking for solutions.

 

 

 

Hello Mark,

 

We Typically find this issue occurs when the load balancers hit that 30 second time out. So if it's making an external call  it very well could be a timing issue with getting data back from the database as you suggested.

The following link has examples of how to modify the web.config file to extend the execution timeout
http://www.websupergoo.com/helpupload50/source/2-tech_notes/3-web.config.htm

 

  <httpRuntime
    maxRequestLength="1048576"
    executionTimeout="3600"
  />

Please modify this in the web.config to resolve the issue.

 

Thank you,
Tony Martinez
Tech Support
Rackspace Cloud

1/26/2011 7:39:10 AM
Gravatar
Total Posts 18439

Re: No Suitable Nodes - Rackspace Hosting

I moved this thread to the Installation Forum it has nothing to do with Accessibility or Usability.

The main problem I see in your log is that the application is restarting over and over multiple times per minute. That security error is expected to be logged in medium trust whenever the app restarts but yours is restarting way too frequently and only your host can help solve that or figure out why that is happening. Typically in shared hosting when I see that kind of problem my first guess would be that too low a memory limit has been set on the application pool and it reaches this level right away and restarts over and over.

Hope it helps,

Joe

1/26/2011 6:42:24 PM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Hey Joe,

I forward your comments over to Rackspace and the reply is below.  I just can't afford to switch hosting companies at this time, I'm going to try (at more cost to me) to use the MSSQL database engine to see if that helps.

 

Thanks,

At the end of the day we don't really have the skills or resources necessary to find out which part of mojoportal would be causing the problem. As we are not developers ourselves. But if we can find out which part is causing slow load times then we can look into possible hardware causes for it. Which is what we are experts in here. In particular if you can narrow it down to a database call or another section of code we can replicate that with a test script and really diagnose the problem at hand.

It may be a bit frustrating but we have several monitoring programs and system administrators watching over these systems. All of our resources here are indicating the system is working as it should. Please be assured if we had any indication of something going wrong we would notify you, and fix it ourselves.

I would love to have solved this problem for you myself, if i had a fix it button I would certainly have pushed it for you. You have been great in helping us out so far. Just let us know how we can assist you here. If you have any questions about how our system works or what No Suitable Nodes means I would be happy to elaborate for you here.

Gene A. Guido III
System Administrator 1
Rackspace Cloud

1/27/2011 9:15:49 AM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Hey Joe,

More information from Rackspace below.  I'm installing the MS SQL version of the code right now, I'll post on how that goes... I'm hoping that all this information helps other folks with mojoPortal if they are with Rackspace or a different hosting company that may have the same issues.

 

Hello again,

No suitable Nodes message is the result of a timeout error. With hundreds of thousands of websites on our system and this message being in place for over a year or two, It's not really a surprise.

However NSN's in no way point out a particular problem. It could be anything that happens when you make a request through the load balancer and your website does not respond fast enough. After 30 seconds you will get that message regardless of the actual problem. And the not knowing, and frustration of not knowing how to look into it is what spurs people to make so many comments about this.

But I have exhausted everything I could to help you out here. For test.dsatallahassee.org:

Average Response Time across the nodes in your IIS cluster is 0.03 seconds  after hitting the front page multiple times across each node I was not able to reproduce the error. (eliminating the possibility that only one node was bad) 

It did seem at first that the website had not been accessed in a while. IIS will unload the application and this can take several seconds to reload. So those first few hits will take a long time (sometimes up to 30 seconds causing NSN's) But I assume you know about this and you are talking about after you have been browsing around for a while(without switching browsers). If you switch browsers you will often hit a new node however if not then you will almost always be talking to only one node.

Currently when doing operations against your mysql node it only takes: 0.182 seconds for mysql50-58 for us to
- Connect to your MySQL server successfully.
- Connect to your database successfully.
- Creat table
- Insert row
- Select row
- Drop table

I know asp and .net is working because your site is showing up regularly along with the hundreds of other websites on that IIS cluster. I am pretty sure this does not have to do with the physical connection to your mysql database because we have monitoring setup for that which is reporting nothing outrageous going on. However we can put up a script to test that connection if you so desire to see it for yourself.

I would love to investigate this further with you, I just don't have anything to go off of without your help here. I am literally out of ideas because I don't know how the software on your website works. So you may have to get a developer to take a look at the site to see if you can catch more information about the error or locate the cause of the slowness.

If you get any more information or steps to replicate the issue let us know here. We can take that information and probably find out what's going on here. I can even send you over to our L2 Windows tech here. But right now i don't have anything to give him.


Gene A. Guido III
Linux Administrator 1
Rackspace Cloud

1/27/2011 9:27:17 AM
Gravatar
Total Posts 18439

Re: No Suitable Nodes - Rackspace Hosting

Did you point out to them that the app is recycling over and over multiple times per minute according to your log?

Did you ask them to look at the memory limits on your app pool and determine if that could be causing the app pool to recycle frequently?

There are no errors of concern in your log, but it is very concerning that your app pool recycles continuously and that is likely a configuration issue on the app pool forcing it to recycle.

Hope it helps,

Joe

1/27/2011 1:44:31 PM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Hey Joe,

I did, however I just added your questions again.

Just as a side note, I just installed the MS SQL version of mojo and so far I have not ran into those problems... I have a tester testing it now to see if that is really true.

Now, one thing I have noticed... When loading sql.dsatallahassee.org in IE 7, 8 or 9 it never completely loads, it always says there one more thing to load.  But in Chrome and FireFox it says it loads just fine.  So, I have no clue what that is.

I'll keep you posted.

2/3/2011 8:59:16 AM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Joe,

Below are items from Rackspace, they have really been trying to find the issue, and this is what they have found: 

 

Hello Mark,

I'm appending more data to what was posted above. We investigated the worker process on the server while clicking on a few links on
http://test.dsatallahassee.org.asp1-16.websitetestlink.com/

At some point, the site causes a dw20.exe to run against your app pool and your app pool restarts again. Once a new worker process starts, the site has a slow initialization due to recompile and it is possible for you to get a no suitable node messge due to time out. This can be caused by an error in the site application.  Unfortunately, the site is not showing the error. You sould take a look at your site code to see if something is loading, but not closing correctly.

I have attached some failed request tracing for your site to show the requests in its entire transit through IIS for your analysis.

Another option of troubleshooting your code is to load the code locally and attempt to crash the application. Something that your code may be doing is similar to this blog post:
http://improve.dk/blog/2008/04/07/spawning-threads-in-aspnet-can-be-dangerous

Feel free to let us know how your troubleshooting goes.

Kind Regards,

David Tsai
Rackspace Cloud Sites

The file from Rackspace is freb1.zip (the name is the link to the file)

Hi Mark,

We are having a weird issue with your site test.dsatallahassee.org. Your code seems to trigger a kernel call that is waiting on a resource and never completes.  Since it never completes, the process cannot be killed off or recycled. This then starts a chain of new worker processes, ending up consuming all the memory on the server(s). I looked at the stack which I will paste

RetAddr           : Args to Child                                                           : Call Site
00000000`7557e9d4 : 00000000`1c43f5f0 00000000`fff00000 00000000`00000000 00000000`7556e334 : ntdll!NtQueryFullAttributesFile+0xa
00000000`7556a996 : 00000000`1c43eb24 00000000`1c43f5dc 00000000`1c43ea48 00000000`fff00000 : wow64!whNtQueryFullAttributesFile+0x3c
00000000`75873688 : 00000000`016d6474 fffff680`c0000011 00000000`00416c98 00000000`00000000 : wow64!Wow64SystemServiceEx+0xca
00000000`7556ab46 : 00000000`77860000 00000000`1c11fd20 00000000`2f774c5a 00000000`1c11f060 : wow64cpu!ServiceNoTurbo+0x28
00000000`7556a14c : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wow64!RunCpuSimulation+0xa
00000000`778d05a8 : 00000000`00000000 00000000`00000000 00000000`fffdf000 00000000`00000000 : wow64!Wow64LdrpInitialize+0x4b4
00000000`778868de : 00000000`1c11f060 00000000`00000000 00000000`fffdf000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string'+0x20aa1
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

The last call is ntqueryfullattributes and it never completes. I have verified this as I looked at the threads and they are in a waitstate. Can you please take a  look at your code to see if anything pops out at you. We need to find a resolution, else you can consume all the memory. If you want a .dmp of your worker process, feel free to ask and we can also supply you it.

2/3/2011 9:37:10 AM
Gravatar
Total Posts 18439

Re: No Suitable Nodes - Rackspace Hosting

Hi Mark,

That doesn't really give me much to go on given that mojoPortal is installed in lots of hosting environments and this issue is not happening elsewhere and no specific error is being logged.

Only suggestions of things to try that come to mind are:

  • Don't use any AppKeepAlive task if you are using one disable it
  • try commenting out <add name="UrlRoutingModule" type="mojoPortal.Web.Routing.RoutingModule, mojoPortal.Web"/> in Web.config in the httpModules section and in the modules section. This is new and not really being used yet. I have no reason to suspect it is part of the problem other than it is a new thing in mojoPortal.
  • Building the search index is done on a background thread, you could try disabling it <add key="DisableSearchIndex" value="true"/> and just using google or bing for site search but as I've said it works fine in lots of other hosting environments.

Wish I could be more help but that is all I have right now.

Hope it helps,

Joe

2/3/2011 10:52:32 AM
Gravatar
Total Posts 165

Re: No Suitable Nodes - Rackspace Hosting

Hey Joe,

Based on your comments (cloned below) I commented those two areas out and I'm not sure which one fixed it, but the sites are no longer giving the NSN errors.

I'm going to do some additional testing by putting the UrlRouting back in and watching it and then switch it and turn search index back on and see what happens.

I'll keep you posted.  Thanks for all your help.

try commenting out <add name="UrlRoutingModule" type="mojoPortal.Web.Routing.RoutingModule, mojoPortal.Web"/> in Web.config in the httpModules section and in the modules section. This is new and not really being used yet. I have no reason to suspect it is part of the problem other than it is a new thing in mojoPortal.
Building the search index is done on a background thread, you could try disabling it <add key="DisableSearchIndex" value="true"/> and just using google or bing for site search but as I've said it works fine in lots of other hosting environments.
 

2/6/2011 8:45:08 AM
Gravatar
Total Posts 18439

Re: No Suitable Nodes - Rackspace Hosting

It would be good to know which suggestion solved it.

Thanks,

Joe

You must sign in to post in the forums. This thread is closed to new posts.