[SOLVED] Why is handling POST requests with Django REST Framework getting slow for body sizes greater than 1024 characters?

Issue

I’m using DRF. I found that receiving a POST with more than 1024 characters causes a ~1 second penalty, while anything less than that is effectively free. I’ve simplified this into this trivial example:

views.py:

import time
from rest_framework.decorators import api_view
from django.http import HttpResponse

@api_view(['POST'])
def test_endpoint(request):
    t = time.time()
    data = request.body
    total_time = time.time() - t

    print('got post data:', round(total_time, 3))
    return HttpResponse('body size:{} time:{}'.format(len(data), round(total_time, 3)))

url.py:

urlpatterns = [
    url(r'^test_endpoint', test_endpoint),
]

You can see that all I’m doing is reading the request.body and measuring the time it takes to do so. Then I respond with that time, and the len of the request.body (to prove I accessed it).

Then I execute these curls:

$ time curl -X POST http://127.0.0.1:8000/test_endpoint -d 0123456782345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567901234567890123456789012345678901234567
body size:1024 time:0.0
real    0m0.045s
user    0m0.006s
sys 0m0.009s
$ time curl -X POST http://127.0.0.1:8000/test_endpoint -d 01234567823456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345679012345678901234567890123456789012345678
body size:1025 time:0.999
real    0m1.020s
user    0m0.006s
sys 0m0.006s

You can see the second one has one extra character, and that causes a ~1 second penalty to ready request.body.

Why is that? How can I prevent this?

I made this as vanilla as I could. I created an new project with django-admin startproject helloworld_project .. I put the code above into it. And running it locally with python manage.py runserver. I don’t have a webserver in front of it; I’m accessing it directly with my browser. That’s all I’m doing.

Also, I’m doing this on Django==1.11. This problem seems to go away on Django==2.0. I cannot easily upgrade to 2.0. Is there a workaround for this issue on 1.11?

Solution

Let’s go deeper and see where the real slowness is.

This function is called from body property.

def read(self, *args, **kwargs):
    self._read_started = True
    try:
        import time
        t = time.time()
        data = self._stream.read(*args, **kwargs)  # HERE(!)
        total_time = time.time() - t
        print('TOTAL TIME: ', total_time)
        return data
    except IOError as e:
        six.reraise(UnreadablePostError, UnreadablePostError(*e.args), sys.exc_info()[2])

For 1024 characters:
TOTAL TIME: 3.123283386230469e-05

For 1025 characters: TOTAL TIME: 0.991084098815918

---

Hmm… This problem is gone in Django 2.0, let’s see release notes:

Requests and Responses

The runserver Web server supports HTTP 1.1.

HTTP 1.1 Supports chunked transfers while HTTP 1.0 does not. I think this can explain the slow behavior.

---

Is there a workaround for this issue on 1.11?

With default runserver I believe that the answer is No. But you won’t use it in production, right? Any other production server like gunicorn, uWSGI supports HTTP 1.1.

Answered By – Satevg

Answer Checked By – Mildred Charles (BugsFixing Admin)