AJAX upload progress bars with jQuery, Django and nginx

john, 21 October 2008
ajax, django, jquery

previous: Immutable Django model fields

next: Django WSGI handler and SSL proxies

Update: the JavaScript originally used for the client side has been replaced with jquery-upload-progress, which works with Safari. I also took some time to clean up the Django upload handler and progress update view, so if you downloaded the demo app before December 4, 2008, you may want to get the latest version.

I've included upload progress bars on a couple of Django sites. Thanks to the new file handling in Django 1.0, a code snippet from djangosnippets.org, and jquery-upload-progress, it's really not difficult, but I haven't seen a complete example, so I put together a demo app showing how the pieces fit together. I'll also cover the case where you want a proxy to provide the upload progress tracking, without involving Django.

If you want to skip the lecture and download example code, go right ahead.

Background

The basic technique is simple: on the page containing your upload form is some JavaScript to generate a unique identifier for the upload request. When the form is submitted, that identifier is passed along with the rest of the data.

When the server starts receiving the POST request, it starts logging the bytes received for the identifier. While the file's being uploaded, the JavaScript on the upload page makes periodic requests asking for the upload progress, and updates a progress widget accordingly.

Client side

The client-side JavaScript is a straightforward application of jquery-upload-progress:

<script type="text/javascript" src="{{MEDIA_URL}}js/jquery.js"></script>
<script type="text/javascript" src="{{MEDIA_URL}}js/jquery.uploadProgress.js"></script>
<script type="text/javascript" charset="utf-8">
//<![CDATA[
$(document).ready(function() { 
    $(function() {
        $('#upload_form').uploadProgress({
            jqueryPath: "{{MEDIA_URL}}js/jquery.js",
            progressBar: '#progress_indicator',
            progressUrl: '{% url upload_progress %}',
            start: function() {
                $("#upload_form").hide();
                filename = $("#id_file").val().split(/[\/\\]/).pop();
                $("#progress_filename").html('Uploading ' + filename + "...");
                $("#progress_container").show();
            },
            uploadProgressPath: "{{MEDIA_URL}}js/jquery.uploadProgress.js",
            uploading: function(upload) {
                if (upload.percents == 100) {
                    window.clearTimeout(this.timer);
                    $("#progress_filename").html('Processing ' + filename + "...");
                } else {
                    $("#progress_filename").html('Uploading ' + filename + ': ' + upload.percents + '%');
                }
            },
            interval: 1000
        });
    });
});
//]]>
</script>

Add that to the bottom of your form page. For the actual progress bar, include this in your form page:

<div id="progress_container">
    <div id="progress_filename"></div>
    <div id="progress_bar">
        <div id="progress_indicator"></div>
    </div>
</div>

In your CSS, add something like this:

#progress_container {
    font-size: .9em;
    width: 100%;
    height: 1.25em;
    position: relative;
    margin: 3em 0;
    display: none;
}

#progress_filename {
    font-size: .9em;
    width: 100%;
}

#progress_bar {
    width: 100%;
    border: 1px solid #999;
}

#progress_indicator {
    background: #8a9;
    width: 0;
    height: 4px;
}

That should give you the progress bar. Now you just need progress reports....

Server side (Django)

On the server, you need two views: one to handle the upload form, and one to respond to progress requests. The first is completely standard; you don't have to do anything special for upload progress. Check the demo app, if you don't believe me. :^)

The magic happens in a special file upload handler, made possible by the new upload processing in Django 1.0. Again, the version here is derived from this snippet, with modifications to allow switching upload tracking back and forth between Django and nginx.

class UploadProgressCachedHandler(MemoryFileUploadHandler):
    """
    Tracks progress for file uploads.
    The http post request must contain a query parameter, 'X-Progress-ID',
    which should contain a unique string to identify the upload to be tracked.
    """

    def __init__(self, request=None):
        super(UploadProgressCachedHandler, self).__init__(request)
        self.progress_id = None
        self.cache_key = None

    def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.handle_raw_input')
        self.content_length = content_length
        if 'X-Progress-ID' in self.request.GET:
            self.progress_id = self.request.GET['X-Progress-ID']
        if self.progress_id:
            self.cache_key = "%s_%s" % (self.request.META['REMOTE_ADDR'], self.progress_id )
            cache.set(self.cache_key, {
                'state': 'uploading',
                'size': self.content_length,
                'received': 0
            })
            if settings.DEBUG:
                logger.debug('Initialized cache with %s' % cache.get(self.cache_key))
        else:
            logging.getLogger('UploadProgressCachedHandler').error("No progress ID.")

    def new_file(self, field_name, file_name, content_type, content_length, charset=None):
        pass

    def receive_data_chunk(self, raw_data, start):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.receive_data_chunk')
        if self.cache_key:
            data = cache.get(self.cache_key)
            if data:
                data['received'] += self.chunk_size
                cache.set(self.cache_key, data)
                if settings.DEBUG:
                    logger.debug('Updated cache with %s' % data)
        return raw_data

    def file_complete(self, file_size):
        pass

    def upload_complete(self):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.upload_complete')
        if settings.DEBUG:
            logger.debug('Upload complete for %s' % self.cache_key)
        if self.cache_key:
            cache.delete(self.cache_key)

What it does is take the generated tracking ID and as it's receiving the file, update the Django cache with the number of bytes received. The progress view just has to check the cache for that data:

def upload_progress(request):
    """
    Return JSON object with information about the progress of an upload.
    """
    if 'HTTP_X_PROGRESS_ID' in request.META:
        progress_id = request.META['HTTP_X_PROGRESS_ID']
        from django.utils import simplejson
        cache_key = "%s_%s" % (request.META['REMOTE_ADDR'], progress_id)
        data = cache.get(cache_key)
        json = simplejson.dumps(data)
        return HttpResponse(json)
    else:
        logging.error("Received progress report request without X-Progress-ID header. request.META: %s" % request.META)
        return HttpResponseBadRequest('Server Error: You must provide X-Progress-ID header or query param.')

The JSON returned is what the JavaScript uses to render the progress bar.

Server side (nginx)

While the Django upload handling got a lot better with the 1.0 release — the memory consumption is now controlled, and you have a lot more flexibility in storing the uploads — it may be better to let your web server shoulder the bulk of the work. That way it can deal with slow clients, and your hefty Django processes can be put to better use.

I use nginx and its mod_uploadprogress to do this. The technique is the same; the module tracks the upload progress and responds to AJAX requests for it. The difference is that your Django app doesn't get involved until the entire file is on the server, so the time it spends processing the upload is much less.

Here's the nginx config from the demo app in its entirety:

upload_progress uploaddemo 1m;

server {
    listen 80;
    server_name example.com;

    client_max_body_size 1000m;

    location ^~ /upload/progress {
        report_uploads uploaddemo;
    }

    location /media/ {
        alias /usr/local/django/test/uploaddemo/media/;
    }

    location / {
        fastcgi_pass                127.0.0.1:8080;
        fastcgi_pass_header         Authorization;          
        fastcgi_hide_header         X-Accel-Redirect;
        fastcgi_hide_header         X-Sendfile;
        fastcgi_intercept_errors    off;
        fastcgi_param               CONTENT_LENGTH          $content_length;
        fastcgi_param               CONTENT_TYPE            $content_type;
        fastcgi_param               PATH_INFO               $fastcgi_script_name;
        fastcgi_param               QUERY_STRING            $query_string;
        fastcgi_param               REMOTE_ADDR             $remote_addr;
        fastcgi_param               REQUEST_METHOD          $request_method;
        fastcgi_param               REQUEST_URI             $request_uri;
        fastcgi_param               SERVER_NAME             $server_name;
        fastcgi_param               SERVER_PORT             $server_port;
        fastcgi_param               SERVER_PROTOCOL         $server_protocol;
        track_uploads uploaddemo 30s;
    }
}

Last bit of blather

That gets you started with upload progress tracking. Of course, in a real application you're probably going to have more complex security concerns, and you'll probably flesh this out with better delivery of the uploaded content.

I handle all that with custom views that store uploads in a protected directory and delegate the actual file delivery to nginx via its X-Accel-Redirect feature; this could also be done with Apache via mod_xsendfile, or with lighttpd's X-Sendfile support.

Again, credit for the original Django upload progress handler goes to the snippet posted by ebartels on djangosnippets.org. All I've done here is put things together in the context of a complete app.

related files

comments (9)

Liza Daly

31 October 2008