How to know that file was fully downloaded or not with Python and Nginx

Updated on September 21, 2021

Read — 3 minutes

How to know that file was fully downloaded or not with Python and Nginx

Introduction

I have been working at Go Wombat as a Python Developer & Machine Learning Engineer. On one project I had to find out whether a file had been completely downloaded from a server and record that event in the database through an API. I found a solution on StackOverflow that uses Nginx. For the API service I used Django and Django REST Framework. When I started implementing that solution, I found several drawbacks, so below is my own approach.

First, create three microservices:

web (for managing API requests);
nginx (the Nginx web server);
logger (collects the logs and sends them to web).

It is good practice to run each of them in a Docker container.

Nginx part

Now write the nginx.conf file. We store information about each request, so give the log format a name, in our case ‘download’. Then describe the format of the log file: the IP address, the request time, the request, the status code, the number of bytes sent and the file name. For details, see the Nginx documentation. Traffic (bytes sent) is the key value: the API compares it with the actual file size to tell whether the file was downloaded completely. This check does not cover partial (Range) requests from resumed downloads or download managers. Nginx also has a built-in $request_completion variable, set to “OK” when a request has completed, which you can add to the log format.

log_format download '{"remote_addr" : "$remote_addr",'
       ' "time":"$time_local",'
       ' "request":"$request", '
       ' "status":"$status",  '
       ' "traffic":$body_bytes_sent, '
       ' "uri": "$uri" }';

/media is the Django app directory that stores the files, so its location gets an alias to serve files from it and a proxy_pass to the web application. Then add an access_log directive that writes to the log file in the ‘download’ format.

location /media {
     proxy_pass http://web:8000;
     default_type application/octet-stream;
     alias /web/media/;
     access_log /var/log/nginx/download.checker.log download;
}

Everything else is a standard Nginx config. The full config:

worker_processes auto;

events {}

http {

  log_format download '{"remote_addr" : "$remote_addr",'
       ' "time":"$time_local",'
       ' "request":"$request", '
       ' "status":"$status",  '
       ' "traffic":$body_bytes_sent, '
       ' "uri": "$uri" }';

  server {
    listen 80; # the port your site will be served on
    charset     utf-8;

    client_max_body_size 700M;  # max upload size

    # configs for the Django app
    location / {
      proxy_pass  http://web:8000;
      proxy_redirect  off;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /media {
      proxy_pass http://web:8000;
      default_type application/octet-stream;
      alias /web/media/;
      access_log /var/log/nginx/download.checker.log download;
    }
  }
}

In the same directory, add a simple Dockerfile for Nginx:

FROM nginx

WORKDIR /nginx_logs
COPY nginx.conf /etc/nginx

EXPOSE 80

Logger part

Next comes the logger service. In the project root, create a logger directory, add a logs directory to it and put a download.py file inside. The service has to read the log file in real time, so use the Popen class from Python’s subprocess module:

log = subprocess.Popen(
    ['tail', '-F', 'download.checker.log'],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

while True:
    send_logs(log)

The send_logs function sends the request data to the web application, so you need to install the requests module. The full example:

import subprocess
import json
import requests
import logging


def send_stats(log: subprocess.Popen):
    data = json.loads(log.stdout.readline(), encoding="utf-8")
    payload = {
        "traffic": data["traffic"],
        "file": data["uri"].split("/")[-1],
        # you can add your fields
    }

    try:
        response = requests.post(
           url="http://web:8000/api/v1/stats",
           json=payload
        )
        response.raise_for_status()
    except Exception as e:
        logging.error(e)
    else:
        logging.info(f"Log was sent to web API {payload}")


if __name__ == "__main__":
    log = subprocess.Popen(
        ['tail', '-F', 'download.checker.log'],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE
    )

    while True:
        send_stats(log)

And the Dockerfile. It uses the slim variant of the Python image, which keeps the image small because it ships only the minimal Debian packages needed to run Python.

FROM python:3.8-slim-buster

WORKDIR /logs
RUN pip install requests
COPY /logs /logs

CMD ["python", "download.py"]

docker-compose file

Finally, write a docker-compose file with a shared volume. The log file is written in the Nginx container, while the code that sends the data runs in the logger container, so both need the same volume. Create a shared_logs volume and mount it in both containers.

version: '3.7'
services:

  web:
    # your application with API

  nginx:
    build:
      context: ./nginx
    links:
      - web
    volumes:
      - "shared_logs:/var/log/nginx/"
    ports:
      - "80:80"

  logger:
    build:
      context: ./logger
    links:
      - web
    volumes:
      - "shared_logs:/logs"

volumes:
  shared_logs: