How to know that file was fully downloaded or not with Python and Nginx

Introduction

I have been working at Go Wombat as Python Developer & Machine Learning Engineer. Once I got an interesting task on my project: to find out if a file was completely downloaded from a server and write information about this event in the database by using API. I started looking for a solution and found one on StackOverflow. It suggested using the Nginx server. For API service, I used Django and Django Rest Framework. After starting implementing the described solution I realized it had many disadvantages. I’m gonna show my solution to the task.

First of all, you need to create a few microservices:

• web (for managing API requests);

• nginx (as you understand it will be an Nginx web-server);

• logger (a part responsible for gathering and sending logs to web-part)

Also, it is good practice to use docker containers for this purpose.

Nginx part

Now it is time to write the nginx.conf file. We will store all the information about a request, so specify the name of the label, in our case, ‘download’. Then, describe the format of your future log file: an IP, time of a request, a request, a status code, a number of bytes which were sent, and the name of the file. To know more, check Nginx documentation. Traffic is the main parameter because, in the API, you must compare its value with the actual file size and find out if the file was downloaded completely (a little trick).

/media is a directory in the Django app which stores files, so we add an alias to get files from it, that’s why you need to add a proxy_pass to the web application. After this, just add a location for the log file.

Everything else is like in an ordinary Nginx config. Full the config:

In the same directory, add the simple Dockerfile for the Nginx:

logger part

Now, the turn of the logger service. Create in the root the logger directory, add to it logs directory, and download.py file inside. You need to read data in real-time from the log file. Use the Popen class from the subprocess python module:

send_logs function sends data about a request to a web application, that’s why you need to install the requests module for this. All example is uploaded below:

And docker file. Nothing ordinary here, but it deserves special attention because it uses the slim variant of Python image, which saves significant resources.