Image gallery bulk upload failing

7/14/2012 11:23:04 AM

Crispin

Total Posts 537

View Posts

feet planted firmly on the ground

I have an image gallery where I can upload single images, and sometimes two at a time with "bulk upload", but any more than two doesn't work. It appears to succeed, in that the "progress" popup says completed then disappears. But no image thumbnails appear and I have this in the system log (from one episode trying to upload four files):

2012-07-14 11:18:50,970 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-14 11:18:51,422 INFO (null) - (null) - (null) - mojoPortal.Web.WebTaskManager - deserialized WebTaskManager task
2012-07-14 11:18:51,422 INFO (null) - (null) - (null) - mojoPortal.Web.WebTaskManager - Queued WebTaskManager on a new thread
2012-07-14 11:18:52,935 ERROR (null) - (null) - (null) - mojoPortal.Web.Global - Referrer(none) useragent Shockwave Flash
System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
   at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] unused, String encryptionAlgorithm, String validationAlgorithm, Deserializer deserializer, SignatureChecker sigChecker)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] validationKey, String encryptionAlgorithm, String validationAlgorithm)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString)
   at Brettle.Web.NeatUpload.UploadHttpModule.GetCookiesFromQueryString(String qs)
   at Brettle.Web.NeatUpload.UploadHttpModule.Application_BeginRequest(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
2012-07-14 11:18:52,998 INFO (null) - (null) - (null) - mojoPortal.Web.Global - crypto error trapped and returned a 404 instead of 500 for security reasons
2012-07-14 11:18:53,076 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-14 11:18:55,774 ERROR (null) - (null) - (null) - mojoPortal.Web.Global - Referrer(none) useragent Shockwave Flash
System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
   at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] unused, String encryptionAlgorithm, String validationAlgorithm, Deserializer deserializer, SignatureChecker sigChecker)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] validationKey, String encryptionAlgorithm, String validationAlgorithm)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString)
   at Brettle.Web.NeatUpload.UploadHttpModule.GetCookiesFromQueryString(String qs)
   at Brettle.Web.NeatUpload.UploadHttpModule.Application_BeginRequest(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
2012-07-14 11:18:55,977 INFO (null) - (null) - (null) - mojoPortal.Web.Global - crypto error trapped and returned a 404 instead of 500 for security reasons
2012-07-14 11:18:59,269 ERROR (null) - (null) - (null) - mojoPortal.Web.Global - Referrer(none) useragent Shockwave Flash
System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
   at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast)
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] unused, String encryptionAlgorithm, String validationAlgorithm, Deserializer deserializer, SignatureChecker sigChecker)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString, Byte[] encryptionKey, Byte[] validationKey, String encryptionAlgorithm, String validationAlgorithm)
   at Brettle.Web.NeatUpload.ObjectProtector.Unprotect(String secureString)
   at Brettle.Web.NeatUpload.UploadHttpModule.GetCookiesFromQueryString(String qs)
   at Brettle.Web.NeatUpload.UploadHttpModule.Application_BeginRequest(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
2012-07-14 11:18:59,362 INFO (null) - (null) - (null) - mojoPortal.Web.Global - crypto error trapped and returned a 404 instead of 500 for security reasons

mojoPortal Version 2.3.8.5 MSSQL

Operating System Microsoft Windows NT 6.0.6002 Service Pack 2

ASP.NET Info v4.0.30319 Running in Full Trust

7/14/2012 11:47:51 AM

Joe Audette

Total Posts 18439

View Posts

Re: Image gallery bulk upload failing

It looks to me like the application pool is recycling during the upload, this could happen in a low memory environment, often hosts configure the application pool to recycle when a certain memory limit is reached, if this memory limit is configured too low it causes performance and other problems because the application recycles constantly. In this case it seems that if you are uploading more than one or two images at a time the memory needed to upload the images is tipping it over that limit and when the app pool recycles during the upload it causes failure to complete the upload.

You'll notice the application start event logged along with the other events all within the same minute and within a few seconds of each other.

Better hosting or a higher memory limit on the app pool should solve it.

Hope that helps,

Joe

7/14/2012 11:50:59 AM

Crispin

Total Posts 537

View Posts

feet planted firmly on the ground

Re: Image gallery bulk upload failing

OK thanks Joe that makes sense. It must be a very low limit because these are images of around 200-300K each! I'll have a word with Arvixe.

7/14/2012 11:54:19 AM

Crispin

Total Posts 537

View Posts

feet planted firmly on the ground

Re: Image gallery bulk upload failing

Question - can I upload images into an image gallery feature by FTP? I'm guessing not as they appear to be processed on upload into full/web/thumb folders.

I have a few hundred, and the prospect of uploading them two at a time is not filling me with joy.

7/14/2012 1:08:23 PM

Joe Audette

Total Posts 18439

View Posts

Re: Image gallery bulk upload failing

You can upload by ftp for the folder gallery but not the image gallery.

About the memory, the thing is it isn't just the images its the aggregate of memory used to run your site and I'm pretty sure Arvixe gives about 200 MB memory limit which is about standard for any decent shared hosting with ASP.NET, its the same amount that Discount ASP provides. I'm not sure you'll be able to get more out of them, but of course you can always ask.

The way .NET manages memory with garbage collection tends to keep things in memory even after they are not being used but eventually when it needs more memory it garbage collects the memory from the no longer used objects.

The images just seem to be tipping it over the top of the limit when you upload more than a few at a time and that causes the recycle of the app.

Best,

Joe

7/14/2012 6:01:46 PM

IndigoTea

Total Posts 355

View Posts

Re: Image gallery bulk upload failing

Crispin,

What you're experiencing might also be a combination of shared hosting limitations, and a shared server that's close to full capacity. When you touch bases with Arvixe, ask if it's possible to move your hosting account to another server; that's helped others here with shared hosting on Arvixe.

If you'd like, I can assist in escalating your issue; it's one of the things I do as the Arvixe Community Liaison.

7/15/2012 12:42:38 AM

Crispin

Total Posts 537

View Posts

feet planted firmly on the ground

Re: Image gallery bulk upload failing

Thanks Joe and Indigo Tea

Arvixe have a 250MB stated limit for Personal ASP hosting, which is what my site is on.

https://support.arvixe.com/index.php?/Knowledgebase/Article/View/138/4/imposed-memory-limit-in-windows-servers

I'm accustomed to seeing mojo sites using perhaps 120-150MB RAM when running on our own servers, but I've never seen one over 200. And my site only had 375 visits yesterday. So it's still hard to see how uploading 600K as 3 images would push it over 250MB.

I went on Arvixe "chat" support, and the chatter went to check how much memory my site was actually using while I tried the uploads, but instead of telling me the answer, at that point he decided that it needed further investigation and converted it into a support ticket. To me this suggests he found it was only using (e.g.) 150MB but was still recycling like a hippie.

I'll certainly ask about a server move, because this is unusable. Particular when I can push 200 images at a time up to SkyDrive, Flickr, etc.

Perhaps this also raises a design question about the Image Gallery feature... is it handling the multiple uploads as nicely as possible? I've a feeling that (at work) we've been using something else in preference to Neat Upload lately, but I'll need to check this.

7/15/2012 12:20:11 PM

Joe Audette

Total Posts 18439

View Posts

Re: Image gallery bulk upload failing

With any shared hosting your site is on a server with lots of other sites and the overall server performance can be affected by the performance or load characteristics of the other sites, sometimes there can be one rogue site causing lots of problems. So asking to move to a different server is not a bad idea if you are having trouble.

I don't think there is anything inefficient about the way we are processing uploads or images.

Long term I have my eye out for something to replace NeatUpload but at the moment I still believe its the best option available. There are a few specific things I have an eye on that are still evolving and not quite mature enough yet but that show some promise. My main beef with NeatUpload is that we can only use it in full trust hosting. Once we get to a point where the vast majority of users have browsers with full html 5 support there will be better ways of doing things such as upload progress bars but until then its complicated by all kinds of needed workarounds for older browsers.

Best,

Joe

7/18/2012 1:45:58 PM

Crispin

Total Posts 537

View Posts

feet planted firmly on the ground

Re: Image gallery bulk upload failing

I've been talking to Arvixe tech support and they say there is adequate headroom on the server I'm on, and that the app should be able to use its full 250MB. Arvixe are asking me how much memory my application is demanding... can you help me answer that one?

I now find that I can upload about 5 images at one go right now (250K each) - but not more - clearly showing that the behaviour does depend on server load, because when I started this thread it failed on 3.

How many files of this size should I be able to upload with NeatUpload? How much memory is this demanding?

And now I'm seeing lots of sequences like this in my server log:

2012-07-18 12:01:57,716 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:02:14,050 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:09:57,806 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:12:14,057 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:20:59,137 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:22:14,095 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:28:12,427 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:31:52,060 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:35:10,991 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:35:14,563 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:36:12,205 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:36:15,419 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:37:11,532 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 12:37:57,958 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 12:57:11,578 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:01:37,433 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:05:11,668 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:05:12,838 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:06:12,851 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:06:40,104 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:08:11,988 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:08:48,258 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:10:12,061 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:11:50,341 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:12:12,166 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:12:19,030 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:13:12,366 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:16:48,457 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.
2012-07-18 13:17:12,450 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Global.asax.cs Application_End
2012-07-18 13:17:16,522 INFO (null) - (null) - (null) - mojoPortal.Web.Global - Application Started.

suggesting that something is causing my app pool to recycle even when I'm not doing anything. Should I be concerned? I think yes!?

7/18/2012 2:05:19 PM

Joe Audette

Total Posts 18439

View Posts

Re: Image gallery bulk upload failing

Yes, you should be concerned about that, it does seem to indicate that your app pool is recycling continuously and the only thing I know of that causes that is reaching the memory limit so it seems your site is reaching the limit immediately after starting and continuously recycling.

I have no way of knowing how much memory your site is actually using. In .NET we are using higher level programming languages where we don't manage memory directly from application code it is handled by the runtime, ie we don't allocate or de-allocate memory from application code like low level languages such as C and C++.

Arvixe should be able to determine how much memory your app is actually using.

By default we are using very minimal caching but if you have configured to use more caching or running multiple sites from one installation that will require more memory. I know there are lots of people using mojoPortal on Arvixe and not having frequent app pool recycling so there is something peculiar to your installation or it doesn't really have enough memory available.

It also matters whether you have any custom code of your own involved? It is definitely possible to write code that leaks memory but I'm not aware of any memory leaks in the code we ship.

Also how many sites are you running in your package? Do they allocate you more memory per site or you get a fixed amount for all sites? Some hosts will let you run as many sites as you want in a package but it doesn't mean you get more server resources for each site, you get what you pay for.

You may need to escalate this to a higher level support at Arvixe.

7/18/2012 2:37:10 PM

Joe Davis

Total Posts 2239

View Posts

Re: Image gallery bulk upload failing

Hi Crispin,

If you want to setup your site on alternative hosting to see and see if the problem persists, I'll give you a month of free hosting @ i7MEDIA to troubleshoot it.

Just send us a note here and reference this thread if you are interested.

Thanks,
Joe D.