The Challenge
I’m currently working on an application in Django which I hope to deploy in the upcoming weeks, it’s just that I fix one bug and another one appears, then tests pass, I add something new, suddenly nothing is working so I have to redesign the whole thing again. This has also prompted me to explore the topic of debugging a bit more recently.
I guess that’s how we learn.
The application is probably nothing groundbreaking or unique, however, it is challenging enough for my current skill level.
Basic functionality and the main goal is to process an input from a user (this might be numerous files with various sizes), compress it, and let the user download the files.
Since processing a large number of files can be considered a long-running operation, I didn’t want it to block the whole request-response cycle as that is not very user-friendly and nowadays if the user has to wait even 5 seconds for something (without letting him know), he turns the browser off.
That’s when asynchronous tasks become very useful. Asynchronous operations are very important nowadays, and it’s a topic I need to explore a lot more.
An asynchronous task is a task that runs independently of the main program flow and does not block the execution of other tasks. In other words, it allows the program to continue to execute other code while the task is running in the background.
Celery & RabbitMQ
Popular tool(s) to handle asynchronous tasks in Django app is currently Celery with some kind of message broker, RabbitMQ in my case. Redis is another popular message broker.
I am very new to both of these and these are complex tools, so I will just quickly sum up my understanding of them at this point.
How I really used and applied them in my app is perhaps for another topic, once I go live with the actual app.
Celery
Celery is a task queue library that allows you to execute tasks asynchronously in the background, while your Django application continues to handle incoming requests. Celery can be used to handle a wide range of tasks, from sending emails to processing images, or processing files, like in my case, and it supports a variety of message brokers, including RabbitMQ.
Another important aspect of using Celery in an app is that it allows you to scale your application horizontally, by adding more worker processes to handle an increasing number of tasks.
RabbitMQ
RabbitMQ is a message broker that is often used in combination with Celery. It allows your Django application to send tasks to a queue, where they can be executed by a separate worker process.
This separation of concerns allows your application to handle incoming requests quickly and efficiently, while the worker process handles the heavy lifting of the task execution.
The Outcome
What does it mean in practice?
Now that processing of the files is taken care of asynchronously, the user doesn’t have to wait, he can even close the browser, do something else and come back later and the files will be prepared for him.
This leads to a better user experience.