New upload control - how hard would it be to "plug it in"?

This is a forum to suggest new features for mojoPortal. 

This thread is closed to new posts. You must sign in to post in the forums.
8/27/2011 5:41:38 PM
Gravatar
Total Posts 6

New upload control - how hard would it be to "plug it in"?

Hi Joe.

I'll probably regret asking this, but anyway:

I have made my own upload control that is fully MS Ajax compatible (it can be put inside an UpdatePanel), it can be styled with custom button designs, is based on Open Source JavaScript rather than Flash, etc. etc.. My source code is available here: www.kindbergs.dk/ajax-safe-uploader.aspx

I found out that once I set NeatUpload's useHttpModule="false" (or totally disable NeatUpload) my uploader works on my web host under mojoPortal - and my web host is one of those who require Medium Trust compatibility, but on the other hand they allow me to control the max size of uploads with httpRuntime's   httpmaxRequestLength.

So great, I can use it in my own modules in mojoPortal - but I would really like the shared files module and the text editors to also be able to use it...

So my question is:

What would it take to make an alternate upload 'provider' for mojoPortal?

I mean, for single or multiple file selection, I could make my upload control give properties or methods similar to NeatUpload so it works without changing the code behind of mojoPortal, but I don't know if there is any smooth way to introduce it as a provider so I (and others) can choose which uploader to use?

And the integration between the text editors and the uploader + file browser, is that provided by NeatUpload or mojoPortal code?

(Just looked at the NeatUpload on CodePlex and saw that you are also maintaining it now - and I must admit my control is very simple and basic compared to the big work put into that project. Probably I should let mine mature some months before trying to "take over the world" wink - but anyway I would like to get an estimate of the work ahead of me if I try to pursue this route of integration.)

Best wishes

Allan K. Nielsen

8/28/2011 10:56:01 AM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

Your control looks interesting and useful. However, I think trying to make a provider model around the uploader control would be quite challenging. I think the only easy integration strategy would be to just include both uploaders and show/hide one or the other based on configuration. I "might" be willing to do that at some point, but right now I have a lot of other pressing concerns to work on.

I'd be inclined to include it in mojoPortal so people could use it in those cases where they need an uploader to work inside an UpdatePanel, but the biggest concern I have after reading the documentation is that it has a database component supporting only MS SQL and MySql where we would need to support other db platforms as well. Really I don't understand why it needs to be coupled with a database, can it not just upload to disk?

The main benefits of NeatUpload are

  • it streams files to disk without having to keep the entire file stream in memory on the server
  • has a progress bar (does yours have a progress indicator? if so how does it work?)
  • if Flash is available then when using the multi file uploaded users can control select multiple files at once from the dialog, if not users can still upload multiple files but have to select each one separately.
  • can be disabled and degrades gracefully under medium trust

The File browser in mojoPortal is not part of NeatUpload but it uses NeatUpload currently for uploading files.

Best Regards,

Joe

8/28/2011 2:34:53 PM
Gravatar
Total Posts 6

Re: New upload control - how hard would it be to "plug it in"?

Hi Joe.

I understand it is quite some work to add a new plug in option, and I also need to do some other projects first - and maybe I'll have it rewritten to work differently later on.

The problem with NeatUpload is that is goes back to basic functionality in medium/partial trust, while mine works with full functionality in partial trust. (And NeatUpload just gave me a hell of time because I did not realize there was wrong access permissions on the server - it seamed to work but the file was not stored and the system log did not contain any new entries of errors, so it look me a long time to realize the potential cause of the error... Yes, not your fault. I'll probably feel better about NeatUpload tomorrow once my hosting provider has fixed the permissions.)

Of course the reason why my control work in partial trust is because I don't care about the memory use or spooling the file. For huge files NeatUpload is definetly better, but for uploading images for use in a blog, or uploading a PDF file for others to download, that might not be needed.

The reason for the database coupling is that my control works opposite of yours: you gather all files on the client and first uploads them when the "Submit" button is pressed. In mine, the JavaScript (made by Valum) uploads the files as soon as they are selected, and I use the database to keep track of the files between (partial) postbacks, and since the server does not know if the user abandons the page, the database allows me to easily delete all the abandoned uploads. The database is also used for some security information, so the control can tell the upload handler to expect a file of a certain kind, so it can be validated by the upload handler. (It was designed for being on publicly available pages, like for unregistered users to upload a CV as part of a job application, or for uploading images to a post on a social site.)

I guess I could make a purely file based version by storing the extra details in XML format or something, and loop over all files in a temp directory to delete all outdated uploads... I forgot how amazingly many databases mojoPortal supports, but by designing my control to support 2 different databases, I always have the possibility to add more, and I understand that in order for my control to be of interest to you, I need to either support all those databases or provide an alternative.

In Internet Explorer I only have a spinning gif animation and a "Wait" text while the file is uploaded, but Valum's JavaScript uses 'new' features of Firefox (3+), Chrome and Safari to upload the file in an alternate way, whereby a standard JavaScript event is fired by the browser to indicate upload progress. I don't remember if this new way has been implemented in IE 10 yet or not... Anyway, I guess your need for the popup progress indicator is because the files are uploaded at the end when the whole page is posted back, while in mine there is no files being transferred when you press "submit", so it is just a standard quick (partial) pageload.

The multiple-files selection thing is exactly the same in mine as in NeatUpload, although here the limitation is again that IE only allows one file selected at a time, while the other browsers allow multiple selected at once. But in IE it is possible to open and select the next file while the first is being uploaded. (Your Flash version probably has the advantage on IE.)

Well, maybe I have at least planted the idea that a control exist for doing uploads on pages with partial postbacks, but writing this I can also see that it might have a different purpose (smaller file uploads on publicly visible pages where validation controls might prevent the "submit" button from saving the file).

I'll keep thinking about ways to improve and expand my control, and if you get to the point where you would like to offer a choice between upload controls, then please don't hesitate to yell for me, and I'll customize it so it can work with mojoPortal.

Ok? smiley

8/29/2011 8:03:32 AM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

I'm definitely interested in your control and will keep my eye on your work.

If it were not for the database issue I'd at minimum include it with mojoPortal right away and would consider implementing a way to use it in some of the key places where we have upload functionality as time provides.

What about making a simple object for tracking the extra data and storing that object in a session variable corresponding to each file instead of the database?

Best,

Joe

8/29/2011 7:17:53 PM
Gravatar
Total Posts 355

Re: New upload control - how hard would it be to "plug it in"?

One approach would be to create a System.Cache instance, and convert the file to a byte array, store it in the cache with an assigned GUID, with a short expiration date. That way any "abandoned" files would be disposed of by the expiration setting, but remove the reliance on the DB for temp info storage. 

8/30/2011 5:40:02 AM
Gravatar
Total Posts 6

Re: New upload control - how hard would it be to "plug it in"?

Hi,

The cache would not work for web gardens/farms - and I am currently working on a custom website that runs in a web garden where I also need my upload control.

But I checked and the session state cookie seams to be transferred correct in both post methods (if cookies are enabled), so I could store information there, but I would still need to check for abandoned files based on file dates.

I'll work on rewriting / abstracting my data layer more, as well as my own utilities to handle temporary files, so that the big files could be stored on disk and only key information in either memory (session) or database, depending on configuration choices.

Joe: If somebody wants to use both NeatUpload and my AjaxSafeUpload, can NeatUpload be configured to not intercept uploads to the specific ~/controls/upload/ folder? Or would it still do its streaming magic despite maxNormalRequestLength being set to something much higher?

Best,

Allan K.

8/30/2011 1:20:21 PM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

The maxNormalRequestLength is a neatUpload specific setting that would only affect non-upload requests, so I don't think it would be a factor. I don't know of a way to configure NeatUpload to not be involved in an upload request because it has an http module that would be involved in every request if it is enabled. That said, when it is enabled it does not seem to interfere with uploads not implemented with the NeatUpload controls. For example in the File Manager (the jquery one as opposed to the alternate file manager) it does not use NeatUpload, it uploads files using javascript and an iframe as I recall and NeatUpload has never seemed to interfere with it but it may still be involved in handling the upload request.

Actually I may have overstated the benefit of NeatUpload a little. Dean Brettle initially implemented NeatUpload back in the .NET 1.1 days and at that time it was a big advantage to be able to not load the whole file into memory on the server, but later I found out from this discussion that as of .NET 2 there has always been a way to make it flush to disk rather than load very large files into memory with a new setting for requestLengthDiskThreshold on the <httpRuntime element. I'm really not sure if Dean knew about that improvement that came in .NET 2.0 or not.

Best,

Joe

8/30/2011 11:54:50 PM
Gravatar
Total Posts 355

Re: New upload control - how hard would it be to "plug it in"?

Allan,

Just curious - is your web garden utilizing a distributed cache solution such as nCache or AppFabric? Depending on how it's configured to share session state, it may already have a solution built-in.

Joe - I know one of the things you've been working on lately is web farm support for mojo - how are you implementing that?

9/11/2011 9:01:15 PM
Gravatar
Total Posts 6

Re: New upload control - how hard would it be to "plug it in"?

Hi Joe,

I have been busy with other work/stuff for a little while, but now I am testing a new version of my upload control that works just with session state and files instead of database.

I'll do my integration tests during the next week and let you know the result smiley (I have not yet uploaded a new version to my website - I want to test it in practice first, since I restructured quite a bit, and want to know how stable it is.)

And I found my answer to configuring NeatUpload to not interfere with alternate upload handlers in the NeatUpload manual, but I still need to test that too...

 

IndigoTea: In my work projects I am extremely conservative in which add-ons I use, and I have not even been looking for any distributed cache. The only reason it is a web garden is because I added some special actions for new users that would take approx. 1-4 seconds to compute, and I did not want the other users to be stuck meanwhile. (It is only about 500-700 users online every week, although each are editing content on pages that are unique for them, and I think our database would need scaling up before the website.)

So for Session state I just use the asp.net session state process for in memory sharing. ViewState is another story - as users are editing long lists the ViewState can get over 100.000 bytes, so that I store in a separate local database instead of transferring it back and forth, including a history of ViewState for previous pages so that the back button still works. In a way, the database versions of my upload control was modeled after the same concept of a special table/database for handling the "garbage collection" of lost history - and in my session+files based version of the upload control, I still have to resort to looping over all the files in a temp directory in order to delete old/forgotten files.

Maybe I could write some handler that gets notified when the session ends/times out and then delete the files, but I am not sure if that event would be stable enough, so I anyway need to check the file creation dates to catch forgotten files.

 

Best wishes

Allan K.

9/16/2011 4:29:51 PM
Gravatar
Total Posts 6

Re: New upload control - how hard would it be to "plug it in"?

Hi Joe.

Version 0.10.0 of my AjaxSafeUpload control is now posted at my website and on sourceforge, which includes the Session state + file version that does not need a database. In some respects it even mimics NeatUpload's properties, such as a Files array of an UploadedFile class - although my definition is very different.

If you have time, check it out and let me know what you think?

 

Bests

Allan K.

 

9/20/2011 7:03:25 AM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

I'm going to download your control today and take a look at it.

There is one idea I'd like to put forth in hopes you might like to tackle it. I've done a lot of work lately in mojoPortal for web farm support and one of the things I implemented was a file system abstraction layer and I also implemented a working file system provider for Windows Azure Blob storage and one of the issues I ran into is that my current solution involves uploading to the web node and then from there the file is uploaded to blob storage. It works ok but this 2 hop process takes longer especially for larger files. I think a better solution is to upload directly from the client machine to Azure Blob storage, which as I understand it can be done using a Shared Access Signature which provides a temporary ticket to allow the upload. But since this is client side stuff it would require implementing something directly in the upload control where the access signature would be set server side but then the upload would go directly to blob storage instead of first to the web node. So far I've only seen example implementations that use Silverlight, I'm not even sure if it can be done without using something like that, but it would be really great if it is possible to do it without requiring the silverlight plugin.

Anyway, just food for thought.

Best,

Joe

9/20/2011 7:18:12 AM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

I just downloaded and tried it out, I built the solution as Release - Session and launched the Default.aspx of the web demo project in the browser and I got a security error:

Security Exception
Description: The application attempted to perform an operation not allowed by the security policy.  To grant this application the required permission please contact your system administrator or change the application's trust level in the configuration file.

Exception Details: System.Security.SecurityException: Request for the permission of type 'System.Web.AspNetHostingPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.
 

and I'm not even running it in medium trust. So I get the sense it won't run in medium trust?

Best,

Joe

9/20/2011 4:07:48 PM
Gravatar
Total Posts 6

Re: New upload control - how hard would it be to "plug it in"?

Hi Joe.

Very odd - the web.config has trust level set to "High", but I have tried setting it to "Medium" and it still works.

A quick google said something about network/file permissions - are you using a local/remote IIS or the one build into Visual Studio? Or something about file permissions where the solution is?

It works with the VS build in hosting-thingy (forgot its name) and I have it running on IIS 7.0 in full trust at work and at a hosting provider with my own website (which says it uses medium trust, but is closer to High trust.) But I must admit I have not tried it using the local IIS (my machine is a bit underpowered for that), so maybe it requires some extra settings for that? I just tested the unzipped solution with VS on my work computer, and it runs fine there too (without using IIS).

!?

Edit: Most likely you put the unzipped solution on a network or external drive. Putting it on a local drive should solve it?

 

Regarding Azure, it is an interesting new platform which I unfortunately have not yet had a reason to study - so thank you for giving me a good reason to study it! :-)

But is Azure not always based on SQL Server? So wouldn't the version of my upload control which stores uploaded files in the database instead of the file system be perfect for that?

Remember, my upload control does not actually try to save any files to any public part of your website - it just holds them and passes them to your server code (in a partial postback event), so you can decide what you want to do with them. And you might as well store them a totally different place than the database or the file system, if you have such a place available. This of course also give you "2 hops", but in each their own postback.

But I could take my session-based version and see how it could store the temporary files in the Azure blob storage instead of the file system - if the normal website structure is available so that my two upload handlers can receive the files?

(Honestly, I have no idea how a website without a file system is structured... And I also got two paid projects I need to finish now, so it will probably take a few weeks before I get back to studying Azure. Sorry.)

Edit: JavaScript in older browsers (like Internet Explorer) can postback to special URLs, but the file content of the postback cannot be manipulated by JavaScript, so it requires a special upload handler to extract the file upload (which can be attached to the postback in two different ways) and store it. For security reasons I would also recommend having an upload handler to accept or reject uploads, as even though Silverlight or Flash could do what you want client side, their actions can be seen and misused by a hacker if there is the slightest hole in the security rules (for example you should not allow reading an uploaded file before your code have verified it is safe, and have removed upload rights so it cannot be replaced).

 

Bests

Allan K.

 

10/6/2011 2:20:06 PM
Gravatar
Total Posts 18439

Re: New upload control - how hard would it be to "plug it in"?

Hi Allan,

Sorry I haven't got back to you sooner. I did unzip your control on a network share so that is probably the problem, I just haven't had a chance to try it again.

I have created a document with an overview of Windows Azure, SqlAzure is the database part of Azure but there are many other parts. The Azure Blob Storage is what would be used for file uploads. I've implemented a file system layer for Azure Blob Storage but it runs on the web role, so the file gets uploaded to the web role and then from there up to Azure Blob storage, but there are ways to do it by generating temporary tickets that allow directly uploading from the client to the blob storage using a REST api. So I think it would involve generating the ticket server side and then rendering into javascript on the page. I "think" that what the Silverlight uploaders do is somehow upload it in chunks which makes handling larger files easier, but I'm not all that up to speed on it. Posting the file to the rest api from javascript may work well enough for most files.

When I get a chance I'll give your control another try and make sure to unzip it on a local drive.

Best,

Joe

You must sign in to post in the forums. This thread is closed to new posts.