Thoughts from the team at Mission Data

AJAX file upload progress for Java using commons fileupload and prototype

This has been done before with PHP (AJAX upload progress meter for PHP) etc but I needed something a little different because I wanted to upload a file and then have it loaded into a database. I looked around and found that someone had already made something that used the commons file upload package to do the upload part (AJAX Upload progress monitor for Commons-FileUpload Example). It wasn’t exactly what I was looking for but it a good start.

To understand the way this works I think it is easiest to break it down into parts:

  1. A file upload extention that counts bytes as they are uploaded
  2. An interface that monitors the progress of something running on the server
  3. AJAX to pull the monitoring into the current screen

Counting bytes when files are uploaded

This was taken from the example listed above. It extends and wraps parts of the commons File Upload classes so that you can count the bytes as they are uploaded to the server. You can download the source with build file or the binary. You will also need the commons file upload, commons io and commons logging. If you download the source put the commons jars in the lib directory before building.

The code is fairly simple to follow. MonitoredDiskFileItemFactory replaces DiskFileItemFactory and the construction of a MonitoredDiskFileItemFactory takes a OutputStreamListener that will be passed on down the chain. The new factory creates MonitoredDiskFileItems instead of DiskFileItems for each file uploaded. When the file needs to be written to disk a MonitoredOutputStream is given back instead of a normal OutputStream. The MonitoredOutputStream calls the OutputStreamListener methods as the bytes are written and with that you now have a way to monitor the byte count as the file is created on the server.

Now to test this all out we can just have an OutputStreamListener that writes its progress out to a logfile or something.

public class FileUploadListener implements OutputStreamListener
{
  private long totalFileSize;
  private long currentFileRead;

  public FileUploadListener(long totalFileSize)
  {
    this.totalFileSize = totalFileSize;
    this.currentFileRead = 0;
  }

  public void start()
  {
    log.debug("Upload started. Total file size: " + totalFileSize);
  }

  public void bytesRead(int byteCount)
  {
    log.debug("Read bytes. Currently " + byteCount + " out of " + totalFileSize + " bytes.");
    currentFileRead+=byteCount;
  }

  public void error(String error)
  {
    log.debug("Hit an error: " + error);
  }

  public void done()
  {
    log.debug("Upload done.");
  }

  public long getTotalRead()
  {
    return currentFileRead;
  }

  public long getTotalSize()
  {
    return totalFileSize;
  }
}

Now we try it out. You can put this in a servlet or jsp so I’m only going to list the parts that matter.

  FileUploadListener listener = new FileUploadListener(request.getContentLength());
  session.setAttribute("LISTENER", listener);
  FileItemFactory factory = new MonitoredDiskFileItemFactory(listener);
  ServletFileUpload upload = new ServletFileUpload(factory);
  List items = upload.parseRequest(request);
  for (Iterator i = items.iterator(); i.hasNext();)
  {
    FileItem fileItem = (FileItem) i.next();
    if (!fileItem.isFormField())
    {
       // code here to process the file
     }
   }

I’m going to assume you can find the correct way to do the actual form upload part.

Note: One issue that you will face at some point is where the upload post goes to becuase when you get to the AJAXy part of things you want the post to stay on the same page. You can use a hidden iframe and the form’s “target” parameter to do this (I have an example later). This is one thing the Java examples I found didn’t have but the PHP examples did and I’m not sure exactly how the Java examples work without it.

Monitoring progress on the server

The next step is to monitor the progress of the upload on the server. What you are monitoring on the server doesn’t even need to be the upload. For the work I was doing the upload goes fairly quickly but what happens to the file after the upload takes a little longer. I wanted to monitor both and that is one reason I think it helps to break this up into parts because you aren’t limited to just monitoring file uploads.

The main thing to keep in mind here is that the application server is multithreaded and you can make more than one request to the server at the same time. You probably know that you can open a tab in firefox or another window in ie and use the same session from the current webapp you are using. Knowing that you can create a page that monitors the status of things as they are running on the server.

From the example above you could toss the listener into the users session. Then insead of logging you just add a couple variables to keep track of the number of bytes that have been uploaded. Then create a simple jsp that pulls the Listener out of the session and dumps its data to a page. Open two windows, one to the upload page and another one to the status page. Start the upload and then start refreshing the status. You should see that the values change as the file is uploaded.

<%@page%>
<%
  FileUploadListener listener = (FileUploadListener)session.getAttribute("LISTENER");
%>
Total size: <%=listener.getTotalSize()%><br/>
Read count: <%=listener.getTotalRead()%><br/>

Of course you will probably want more than just the total size and bytes read as well as more formating like a little progress bar or something but I’ll leave that up to you.

AJAX integration with prototype

You have the major parts to the upload progress done and now all you need is the AJAX part. To do this I chose to use prototype because it cuts right to what you want to do. One call is all you need to use: Ajax.PeriodicalUpdater.

The Ajax.PeriodicalUpdater call will update a container (in my case a div) on a set interval. Here is an example of how to have it update a div with an id of “status” every second.

    new Ajax.PeriodicalUpdater(
                                'status',
                                'status.jsp',
                                {asynchronous:true, frequency:1, method:'get'});

The first argument is the id of the div, the second is the jsp that contains the data to stick into the div every second and the 3rd arguement is a set of options. There are more options availabe if you need them.

You would want to kick the update off whenever the form is posted. When the post is complete the iframe used as a place to post to will load with the results of the servlet or jsp that you posted to. If you return some javascript as a result for the iframe you will be able to create a final “finished” message on the page to let the user know the upload has completed and stop the processing of the AJAX updater.

So there you have it. The basics of setting up an upload progress bar using java and AJAX. I have left out a good bit but you should have enough to at least get you started.

By request I have created a simple example that pulls everything together. The source contains everything you need to create a war file including all source and an ant build file.

Example Source

38 Responses to “AJAX file upload progress for Java using commons fileupload and prototype”

  1. April 10th, 2006 @ 12:47 am Daniel responded:

    Does your version use DWR? I’m new to JSP and trying to piece your example together. Do you have a link to the full source?

  2. April 13th, 2006 @ 9:10 am carsonm responded:

    I didn’t use DWR mainly because I think it is overkill for this. The fileupload status stuff is fairly simple to do and one of the points here was that. I have created a complete example of the above with source and a build file. I’m going to edit the post to include it at the bottom.

  3. April 17th, 2006 @ 1:17 pm Jeff responded:

    I noticed that you did not include the source to the classes in fileupload-ext.jar. If you didn’t have a reason not to, I wouldn’t mind seeing them. I couldn’t tell where the monitoring actually started, but I was assuming that it was in the constructor of the MonitoredDiskFileItemFactory.

  4. April 17th, 2006 @ 1:36 pm carsonm responded:

    If you look at the first paragraph in “Counting bytes when files are uploaded” the source and an ant build file is availble in a zip file there. If you want to build the source yourself you will ned the external libraries listed in that sam paragraph. Here is another link to the source zip: Source

  5. April 17th, 2006 @ 2:27 pm Jeff responded:

    Thanks, I missed that link earlier. This looks to be exactly what I need (file upload from JSP to Servlet to Database with progress monitoring), so I appreciate the very well written walkthrough.

  6. April 18th, 2006 @ 9:58 am Jeff responded:

    I’m in the process of integrating this with my system and it’s gone fairly smoothly so far, but I’m having trouble with two things.

    1. I can’t get my upload form page to stop polling after the download is complete. I am calling sendCompleteResponse in a similar manner to your example, but I am not seeing anything as a result of the call.

    2. I can’t figure out how to throttle the upload so that I can see progress when the server and client are on the same box. It looks like the commons code is just reading the file off of the InputStream and writing it to a temp file (which then calls the monitoring methods) as fast as it can.

  7. April 18th, 2006 @ 11:07 am Jeff responded:

    Scratch my first error – that was just because I was trying to do a redirect in my servlet (duh). Still not sure if there’s a great way to throttle the upload, but it’s not a huge deal.

    Also, if you are uploading over SSL in IE, you will see the following message:

    This page contains both secure and nonsecure items.
    Do you want to display the nonsecure items?

    This is a bug in IE referenced here: http://support.microsoft.com/default.aspx?scid=kb;en-us;261188. It is due to the fact that the target iframe for the POST has an empty src attribute. The work-around is easy – just set the src to a dummy page.

  8. April 18th, 2006 @ 12:18 pm carsonm responded:

    To answer your second issue you can toss a Thread.sleep(500); in the “public void bytesRead(int bytesRead)” of your OutputStreamListener implementation and that will pause the upload a little.

  9. April 18th, 2006 @ 12:44 pm Jeff responded:

    Thanks – actually just found that out myself a minute ago. I am noticing what looks like a bug though. I saw that my progress was always starting at 100% and going to 200%. I saw that the done() method in FileUploadListener sets the bytesRead to the total size of the upload. Of course, this is a perfectly reasonable behavior for that method…however, it appears that done() is being called once per input in the upload form. My guess is that this is a result of the commons code closing the InputStream of each FileItem it parses.

    Not sure what the cleanest solution is, but I am not sure the current one will suffice due to the way that commons-fileupload works.

  10. April 18th, 2006 @ 2:08 pm jon077 responded:

    Excellent example. Have you heard of anyone using XmlHttpRequest to do the file upload instead of using an IFRAME?

  11. April 18th, 2006 @ 3:34 pm Mark responded:

    This looks very nice, just tested and works really well. Thank you very much for the example, I’m going to try and use it with the web application that I’m developing!

    Has it been tested with Spring by chance? I’m using Spring and Commons-fileupload so I hope it will all behave nicely together…

    Mark

  12. April 18th, 2006 @ 4:05 pm Jeff responded:

    Just doing the final polish on integrating this with our system and it looks fantastic. The only major issue I had was with the FileUploadListener improperly reporting the number of bytes read. The problem is as follows:

    When you parse the request using commons-fileupload, it will create a FileItem for each input field present in the HttpServletRequest, using whatever factory was specified (in this case MonitoredDiskFileItemFactory). As these fields are read in and flushed out, MonitoredOutputStream calls the done() method on its FileUploadListener. However, since all of the FileItems share the same FileUploadListener, done() may be called many times, which resets the bytesRead to the total size of the request.

    The easiest way I found to get around this was to add a simple check to my done() method, only allowing done() to change the bytesRead if the bytesRead were already greater than 0. In my initial unit testing this solution has worked quite well. It’s not exactly elegant, but it didn’t involve changing basic assumptions in commons or carson’s code either.

  13. April 19th, 2006 @ 10:09 am Mark responded:

    Jeff,

    Will that fix work when multiple files from the same or from multiple clients are uploaded simultaneously?

    Thanks,

    Mark

  14. April 19th, 2006 @ 10:29 am Jeff responded:

    Mark,

    Multiple clients shouldn’t be an issue since the progress data is kept in a session variable. Multiple files from the same client shouldn’t be a problem either since the progress is based on the contentLength of the HttpServletRequest, which would include all files and form inputs in the request.

    One thing that would probably an issue though is multiple forms received from the same session (i.e. two browser windows sharing the session). I think it could be worked around…it’s just not a use case I’m worried about at the moment. :)

    However, I have found some cases where my previous solution did not work, so I am still searching for a better one. If anyone has any ideas I’d be very happy to hear them.

  15. April 19th, 2006 @ 10:43 am Mark responded:

    OK, thanks Jeff.

    I’m going to be working on this today, so I’ll let you know if I dig anything up.

  16. April 19th, 2006 @ 3:59 pm Mark responded:

    Hi Jeff,

    I tried a few files today and it always worked fine, progress always started at 0 and went to 100%. When didn’t it work for you? Can you post a test case somewhere?

    Thanks,

    Mark

  17. April 19th, 2006 @ 4:17 pm Jeff responded:

    Mark,

    The problem occurs when your request contains more than just a file input. The test case is pretty simple: just add an additional form field, make sure to actually input a value, then submit. If the form field is parsed before the file, it will screw up the progress bar.

    I have since implemented a more robust solution that extends carson’s code and treats each form field individually. The only catch as of now is that it depends on each form field name being unique. If anyone is interested I could probably get it in a more publish-able form.

  18. April 28th, 2006 @ 8:04 am Mark responded:

    Hi Jeff,

    Sorry, haven’t been around. If you would like to post/send me your updated version, it would be much appreciated! Just add gmail.com to the below:

    mwaschkowski

    to email me. I’m planning on revisting this code on the weekend.

    Best,

    Mark

  19. May 1st, 2006 @ 10:58 am Brian OConnor responded:

    Hi Mark & Jeff,

    I’m not even sure of the point of the “done” method. In fact I’m not even sure why there is maintenance of a “currentStatus” in the FileUploadListener, it is not used anywhere. Thus, the start(), done() and error() methods can be removed (they serve no purpose).
    The UploadServlet can be changed to handle the final status, by calling something like sendCompleteResponse, that creates HTML that calls a done() method on the html page. This done method can stop the updater and set the final status (upload complete or whatever). This method fixes all the other issues addressed.

    One more nice thing to add, would be for the ability of the listener to know what field/file is currently being “worked on”. This method could be called by MonitoredDiskFileItemFactory, during createItem.

  20. May 2nd, 2006 @ 10:35 am Anand responded:

    its pretty nice
    but it running in my system but progress bar not showin up
    only time is increasing .
    need help.

    Thanks

  21. May 2nd, 2006 @ 11:00 am Anand responded:

    I down loaded the code compiled it ,ive made no changes ,
    to the code .

    showing message :Uploade o out of 24242 bytes(0%) of 0 Kbs

    thanks

  22. May 3rd, 2006 @ 5:33 am Adam responded:

    I’ve been lookign for a progress monitor for uploads for a long while and this is by far the clearest explanation, cheers.

    One quick question, when I return to the upload page and start a second upload the page displays the completion message for the previous upload, then after a couple of seconds starts to display the upload bar correctly.

    Could this be due to testing it on the same server? I have throttled the upload somewhat and tried to remove any information that was being stored in the session after completion, but haven’t been able to stop this problem.

  23. May 3rd, 2006 @ 1:25 pm carsonm responded:

    If you are re-entering the page again my guess would be that something is still in the session. I would double check to make sure the LISTENER session object is gone when the upload is complete. You may want to check by creating a temporary jsp or something that checks it after you do your upload just to make sure that isn’t the problem.

  24. May 9th, 2006 @ 2:55 pm steven responded:

    nice job man i didnt get it too work yet but i like your ideas.

  25. May 18th, 2006 @ 10:39 am visu responded:

    Nice Job! I incorporated the code given in my web application and it works wonderful.
    Thanks,
    Visu

  26. June 12th, 2006 @ 3:21 pm srik responded:

    looks interesting. Will try to incorporate in my app.

  27. June 12th, 2006 @ 5:08 pm Misael responded:

    Really good example. I would like to know if the multiple file problem was solved, because i could not do it.
    Thanks!
    Misael

  28. June 12th, 2006 @ 7:13 pm Gena responded:

    Thank you very much for the component. Works great!

  29. July 12th, 2006 @ 1:09 pm Rui Felix responded:

    Hello,

    I’ve been looking for something like this for a while! I’ve downloaded the code and get it working on my project but got a problem:
    When I File to upload the return message are the shown below
    Upload Status:
    Uploaded: 0 out of 60760 bytes (0%) 0 Kbs
    Runtime: 51 seconds out of 1687777hours 46minutes 39seconds 1687777hours 45minutes 48seconds remaining

    After this I’ve put some debug In my code an I get to this issue, I don’t get to the for where you say that’s where I can process the uploaded file.
    I’ve put debug after each line and it stops processing when it get’s to this line ServletFileUpload upload = new ServletFileUpload(factory); doesn’t reach the for cycle.

    Thanks,

    Waiting for some reply on this.
    Rui Felix

  30. July 14th, 2006 @ 9:26 am Jay responded:

    Is there a way to implement a cancel on a download? If so, can you provide an example on where to implement that?

    Thanks,
    Jay

  31. July 14th, 2006 @ 9:27 am Jay responded:

    Sorry I meant upload.

    Jay

  32. July 19th, 2006 @ 9:16 am Dan responded:

    Hi,

    I can this working really well with the example code but am having trouble integrating it into an existing application that uses a jsp file (which I’d rather not have to re-write as a servlet) to do the data upload. Any pointers to get the submission to go to a jsp. Probably straightforward but I can’t seem to work it out.

    Thanks.

  33. August 31st, 2006 @ 4:00 am Michael Amster responded:

    I was wondering if there was a way to eliminate the posting to an IFRAME. I want the progress to work, but when complete, I want to refresh to a page like normal non-AJAX web behavior. I guess I could always refresh after the completion, but it would be great if I did not have to target the IFRAME. Anyone see a working example of this?

  34. September 9th, 2006 @ 8:40 am Free PHP Ajax Upload Script responded:

    There is a way that you can cancel the file upload – and it’s very easy. All that you have to do is set the SRC of the IFRAME to another file, such as the common blank.html (a file with nothing in it). You can view an example of this at http://www.seemysites.net/projFolder/uploader

  35. September 9th, 2006 @ 8:44 am Free PHP Ajax Upload Script responded:

    Oh, I forgot – you need to be sure to set the SRC back to the fie that actually does the uploading before you try to upload another file.

  36. October 4th, 2006 @ 9:06 pm jim responded:

    EXCELLENT ! EXCELLENT! and again EXCELLENT!

    Just finish implementing your code in a project i am working on. Fit in was easy.
    However are you aware that the code only work with 80% of the current browsers?

    The code goes into a loop under Mozillia Firefox 1.5.0.4 (the latest release).

    Yes, i know it seems strange considering that you are using Prototype and firefox plays nicely with Prototype

    Seems the JS function killUpdate is not being dispatched when you reply with your stats data. The main indicators of this are: The submit query button remains disabled and if you have the firebug JS debugger you will see the code continually making HTTP get calls to the upload module.

    Stranger still is that the code runs fine with IE 6.0.2 ( the 80% market killer )

    Any suggestions for a work around or patch?

    Again kudos to you for making an Apache project output easy to use and incorporate by using a simple example which is usable in its original state,

  37. March 1st, 2007 @ 1:47 am Tremend Tech Blog » AJAX file upload monitoring - monitor your file upload with DWR and commons-fileupload responded:

    […] Monitoring your file upload using ajax is a must have for any web 2.0 application. There are already some good examples developed by prosson and carsonm. […]

  38. September 20th, 2007 @ 2:40 am Script instalation service responded:

    I’ve been lookign for a progress monitor for uploads for a long while and this is by far the clearest explanation, cheers.

blog comments powered by Disqus