Jahongir Rahmonov

How to write a Python web framework. Part II.

2019-02-23T20:10:00+05:00

Learn how to write your own framework in Python!

I have created a whole course out of this series. If you enjoyed this blog post, you will love the course. It is much more polished and contains many more parts. Check it out!

In the first part, we started writing our own Python framework and implemented the following features:

WSGI compatibility
Request Handlers
Routing: simple and parameterized

Make sure to read Part I of these series before this one.

This part will be no less exciting and we will add the following features in it:

Check for duplicate routes
Class Based Handlers
Unit tests

Ready? Let's get started.

Duplicate routes

Right now, our framework allows to add the same route any number of times. So, the following will work:

@app.route("/home")
def home(request, response):
    response.text = "Hello from the HOME page"


@app.route("/home")
def home2(request, response):
    response.text = "Hello from the SECOND HOME page"

The framework will not complain and because we use a Python dictionary to store routes, only the last one will work if you go to http://localhost:8000/home/. Obviously, this is not good. We want to make sure that the framework complains if the user tries to add an existing route. As you can imagine, it is not very difficult to implement. Because we are using a Python dict to store routes, we can simply check if the given path already exists in the dictionary. If it does, we throw an exception, if it does not we let it add a route. Before we write any code, let's remember our main API class:

# api.py
from parse import parse
from webob import Request, Response


class API:
    def __init__(self):
        self.routes = {}

    def route(self, path):
        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    def __call__(self, environ, start_response):
        request = Request(environ)

        response = self.handle_request(request)

        return response(environ, start_response)

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            parse_result = parse(path, request_path)
            if parse_result is not None:
                return handler, parse_result.named

        return None, None

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    def default_response(self, response):
        response.status_code = 404
        response.text = "Not found."

We need to change the route function so that it throws an exception if an existing route is being added again:

# api.py


class API:
    ...

    def route(self, path):
        if path in self.routes:
            raise AssertionError("Such route already exists.")

        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    ...

Now, try adding the same route twice and restart your gunicorn. You should see the following exception thrown:

Traceback (most recent call last):
...
AssertionError: Such route already exists.

We can refactor it to decrease it to one line:

# api.py

class API:
    ...

    def route(self, path):
        assert path not in self.routes, "Such route already exists."

    ...

Voilà! Onto the next feature.

Class Based Handlers

If you know Django, you know that it supports both function based and class based views (our handlers). We already have function based handlers. Now we will add class based ones which are more suitable if the handler is more complicated and bigger. Our class based handlers will look like this:

# app.py

...

@app.route("/book")
class BooksHandler:
    def get(self, req, resp):
        resp.text = "Books Page"

    def post(self, req, resp):
        resp.text = "Endpoint to create a book"

...

It means that our dict where we store routes self.routes can contain both classes and functions as values. Thus, when we find a handler in the handle_request() method, we need to check if the handler is a function or if it is a class. If it is a function, it should work just like now. If it is a class, depending on the request method, we should call the appropriate method of the class. That is, if the request method is GET, we should call the get() method of the class, if it is POST we should call the post method and etc. Here is how the handle_request() method looks like now:

# api.py

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    ...

The first thing we will do is check if the found handler is a class. For that, we use the inspect module like this:

# api.py
...
import inspect

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            if inspect.isclass(handler):
                pass   # class based handler is being used
            else:
                handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    ...

Now, if a class based handler is being used, we need to find the appropriate method of the class depending on the request method. For that we can use the built-in getattr function:

# api.py

...

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            if inspect.isclass(handler):
                handler_function = getattr(handler(), request.method.lower(), None)
                pass
            else:
                handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    ...

getattr accepts an object instance as the first param and the attribute name to get as the second. The third argument is the value to return if nothing is found. So, GET will return get, POST will return post and some_other_attribute will return None. If the handler_function is None, it means that such function was not implemented in the class and that this request method is not allowed:

if inspect.isclass(handler):
    handler_function = getattr(handler(), request.method.lower(), None)
    if handler_function is None:
        raise AttributeError("Method now allowed", request.method)

If the handler_function was actually found, then we simply call it:

if inspect.isclass(handler):
    handler_function = getattr(handler(), request.method.lower(), None)
    if handler_function is None:
        raise AttributeError("Method now allowed", request.method)
    handler_function(request, response, **kwargs)

Now the whole method looks like this:

...

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            if inspect.isclass(handler):
                handler_function = getattr(handler(), request.method.lower(), None)
                if handler_function is None:
                    raise AttributeError("Method now allowed", request.method)
                handler_function(request, response, **kwargs)
            else:
                handler(request, response, **kwargs)
        else:
            self.default_response(response)

    ...

I don't like that we have both handler_function and handler. We can refactor them to make it more elegant:

# api.py
...

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            if inspect.isclass(handler):
                handler = getattr(handler(), request.method.lower(), None)
                if handler is None:
                    raise AttributeError("Method now allowed", request.method)

            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    ...

And that's it. We can now test the support for class based handlers. First, if you haven't already, add this handler to app.py:

# app.py

...

@app.route("/book")
class BooksResource:
    def get(self, req, resp):
        resp.text = "Books Page"

Now, restart your gunicorn and go to the page http://localhost:8000/book and you should see the message Books Page. And there you go. We have added support for class based handlers. Play with them a little bit by implementing other methods such as post and delete as well.

Onto the next feature!

Unit Tests

What project is reliable if it has no unit tests, right? So let's add a couple. I like using pytest, so let's install it:

pip install pytest

and create a file where we will write our tests:

touch test_bumbo.py

Just to remind you, bumbo is the name of the framework. You may have named it differently. Also, if you don't know what pytest is, I strongly recommend you look at it to understand how unit tests are written below.

First of all, let's create a fixture for our API class that we can use in every test:

# test_bumbo.py
import pytest

from api import API


@pytest.fixture
def api():
    return API()

Now, for our first unit test, let's start with something simple. Let's test if we can add a route. If it doesn't throw an exception, it means that the test passes successfully:

...

def test_basic_route(api):
    @api.route("/home")
    def home(req, resp):
        resp.text = "YOLO"

Run the test like this: pytest test_bumbo.py and you should see something like the following:

collected 1 item

test_bumbo.py .                                                                                                                                                            [100%]

====== 1 passed in 0.09 seconds ======

Now, let's test that it throws an exception if we try to add an existing route:

# test_bumbo.py

...

def test_route_overlap_throws_exception(api):
    @api.route("/home")
    def home(req, resp):
        resp.text = "YOLO"

    with pytest.raises(AssertionError):
        @api.route("/home")
        def home2(req, resp):
            resp.text = "YOLO"

Run the tests again and you will see that both of them pass.

We can add a lot more tests such as the default response, parameterized routing, status codes and etc. However, all of them require that we send an HTTP request to our handlers. For that we need to have a test client. But I think this post will become too big if we do it here. We will do it in the next post in this series. We will also add support for templates and a couple of other interesting stuff. So, stay tuned.

As usual, if you want to see some feature implemented please let me know in the comments section.

A little reminder that this series is based on the Alcazar framework that I am writing for learning purposes. If you liked this series, show some love by starring the repo.

Fight on!

Check out Part I here
Check out Part III here

How to write a Python web framework. Part I.

2019-02-09T20:10:00+05:00

Learn how to write your own framework in Python!

I have created a whole course out of this series. If you enjoyed this blog post, you will love the course. It is much more polished and contains many more parts. Check it out!

"Don't reinvent the wheel" is one of the most frequent mantras we hear every day. But what if I want to learn more about the wheel? What if I want to learn how to make this damn wheel? I think it is a great idea to reinvent it for the purpose of learning. Thus, in this series, we will write our own Python web framework to see how all that magic is done in Flask, Django and other frameworks.

In this first part of the series, we will build the most important parts of the framework. At the end of it, we will have request handlers (think Django views) and routing: both simple (like /books/) and parameterized (like /greet/{name}). If you like it after reading, please let me know in the comments what other features we should implement next.

Before I start doing something new, I like to think about the end result. In this case, at the end of the day, we want to be able to use this framework in production and thus we want our framework to be served by a fast, lightweight, production-level application server. I have been using gunicorn in all of my projects in the last few years and I am very satisfied with the results. So, let's go with gunicorn.

Gunicorn is a WSGI HTTP Server, so it expects a specific entrypoint to our application. If you don't know what WSGI is go find out, I will wait. Otherwise, you will not understand a huge chunk of this blog post.

Have you learnt what WSGI is? Good. Let's continue.

To be WSGI-compatible, we need a callable object (a function or a class) that expects two parameters (environ and start_response) and returns a WSGI-compatible response. Don't worry if it doesn't make sense yet. Hopefully it will "click" for you while writing the actual code. So, let's get started with the code.

Think of a name for your framework and create a folder with that name. I named it bumbo:

mkdir bumbo

Go into this folder, create a virtual env and activate it:

cd bumbo
python3.6 -m venv venv
source venv/bin/activate

Now, create the file named app.py where we will store our entrypoint for gunicorn:

touch app.py

Inside this app.py, let's write a simple function to see if it works with gunicorn:

# app.py

def app(environ, start_response):
    response_body = b"Hello, World!"
    status = "200 OK"
    start_response(status, headers=[])
    return iter([response_body])

As mentioned above, this entrypoint callable receives two params. One of them is environ where all kinds of info about request is stored such as a request method, url, query params and the like. The second is start_response which starts the response as the name suggests. Now, let's try to run this code with gunicorn. For that install gunicorn and run it like so:

pip install gunicorn
gunicorn app:app

The first app is the file which we created and the second app is the name of the function we just wrote. If all is good, you will see something like the following in the output:

[2019-02-09 17:58:56 +0500] [30962] [INFO] Starting gunicorn 19.9.0
[2019-02-09 17:58:56 +0500] [30962] [INFO] Listening at: http://127.0.0.1:8000 (30962)
[2019-02-09 17:58:56 +0500] [30962] [INFO] Using worker: sync
[2019-02-09 17:58:56 +0500] [30966] [INFO] Booting worker with pid: 30966

If you see this, open your browser and go to http://localhost:8000. You should see our good old friend: the Hello, World! message. Awesome! We will build off of this.

Now, let's turn this function into a class because we will need quite a few helper methods and they are much easier to write inside a class. Create an api.py file:

touch api.py

Inside this file, create the following API class. I will explain what it does in a bit:

# api.py

class API:
    def __call__(self, environ, start_response):
        response_body = b"Hello, World!"
        status = "200 OK"
        start_response(status, headers=[])
        return iter([response_body])

Now, delete everything inside app.py and write the following:

# app.py
from api import API

app = API()

Restart your gunicorn and check the result in the browser. It should be the same as before because we simply converted our function named app to a class called API and overrode its __call__ method which is called when you call the instances of this class:

app = API()
app()   #  this is where __call__ is called

Now that we created our class, I want to make the code more elegant because all those bytes (b"Hello World") and start_response seem confusing to me. Thankfully, there is a cool package called WebOb that provides objects for HTTP requests and responses by wrapping the WSGI request environment and response status, headers and body. By using this package, we can pass the environ and start_response to the classes provided by this package and not have to deal with them ourselves. Before we continue, I suggest you take a look at the documentation of WebOb to understand what I am talking about and the API of WebOb more.

Here is how we will go about refactoring this code. First, install WebOb:

pip install webob

Import the Request and Response classes at the beginning of the api.py file:

# api.py
from webob import Request, Response

...

and now we can use them inside the __call__ method:

# api.py
from webob import Request, Response

class API:
    def __call__(self, environ, start_response):
        request = Request(environ)

        response = Response()
        response.text = "Hello, World!"

        return response(environ, start_response)

Looks much better! Restart the gunicorn and you should see the same result as before. And the best part is I don't have to explain what is being done here. It is all self-explanatory. We are creating a request, a response and then returning that response. Awesome! I do have to note that request is not being used here yet because we are not doing anything with it. So, let's use this chance and use the request object as well. Also, let's refactor the response creation into its own method. We will see why it is better later:

# api.py
from webob import Request, Response

class API:
    def __call__(self, environ, start_response):
        request = Request(environ)

        response = self.handle_request(request)

        return response(environ, start_response)

    def handle_request(self, request):
        user_agent = request.environ.get("HTTP_USER_AGENT", "No User Agent Found")

        response = Response()
        response.text = f"Hello, my friend with this user agent: {user_agent}"

        return response

Restart your gunicorn and you should see this new message in the browser. Did you see it? Cool. Let's go on.

At this point, we handle all the requests in the same way. Whatever request we receive, we simply return the same response which is created in the handle_request method. Ultimately, we want it to be dynamic. That is, we want to serve the request coming from /home/ differently than the one coming from /about/.

To that end, inside app.py, let's create two methods that will handle those two requests:

# app.py
from api.py import API

app = API()


def home(request, response):
    response.text = "Hello from the HOME page"


def about(request, response):
    response.text = "Hello from the ABOUT page"

Now, we need to somehow associate these two methods with the above mentioned paths: /home/ and /about/. I like the Flask way of doing it that would look like this:

# app.py
from api.py import API

app = API()


@app.route("/home")
def home(request, response):
    response.text = "Hello from the HOME page"


@app.route("/about")
def about(request, response):
    response.text = "Hello from the ABOUT page"

What do you think? Looks good? Then let's implement this bad boy!

As you can see, the route method is a decorator, accepts a path and wraps the methods. It shouldn't be too difficult to implement:

# api.py

class API:
    def __init__(self):
        self.routes = {}

    def route(self, path):
        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    ...

Here is what we did here. In the __init__ method, we simply defined a dict called self.routes where we will be storing paths as keys and handlers as values. It can look like this:

print(self.routes)

{
    "/home": <function home at 0x1100a70c8>,
    "/about": <function about at 0x1101a80c3>
}

In the route method, we took path as an argument and in the wrapper method simply put this path in the self.routes dictionary as a key and the handler as a value.

At this point, we have all the pieces of the puzzle. We have the handlers and the paths associated with them. Now, when a request comes in, we need to check its path, find an appropriate handler, call that handler and return an appropriate response. Let's do that:

# api.py
from webob import Request, Response

class API:
    ...

    def handle_request(self, request):
        response = Response()

        for path, handler in self.routes.items():
            if path == request.path:
                handler(request, response)
                return response

    ...

Wasn't too difficult, was it? We simply iterated over self.routes, compared paths with the path of the request, if there is a match, called the handler associated with that path.

Restart the gunicorn and try those paths in the browser. First, go to http://localhost:8000/home/ and then go to http://localhost:8000/about/. You should see the corresponding messages. Pretty cool, right?

As the next step, we can answer the question of "What happens if the path is not found?". Let's create a method that returns a simple HTTP response of "Not found." with the status code of 404:

# api.py
from webob import Request, Response

class API:
    ...

    def default_response(self, response):
        response.status_code = 404
        response.text = "Not found."

    ...

Now, let's use it in our handle_request method:

# api.py
from webob import Request, Response

class API:
    ...

    def handle_request(self, request):
        response = Response()

        for path, handler in self.routes.items():
            if path == request.path:
                handler(request, response)
                return response

        self.default_response(response)
        return response

    ...

Restart the gunicorn and try some nonexistent routes. You should see this lovely "Not found." page. Now, let's refactor out finding a handler to its own method for the sake of readability:

# api.py
from webob import Request, Response

class API:
    ...

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            if path == request_path:
                return handler

    ...

Just like before, it is simply iterating over self.route, comparing paths with the request path and returning the handler if paths are the same. It returns None if no handler was found. Now, we can use it in our handle_request method:

# api.py
from webob import Request, Response

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response)
        else:
            self.default_response(response)

        return response

    ...

I think it looks much better and is pretty self explanatory. Restart your gunicorn to see that everything is working just like before.

At this point, we have routes and handlers. It is pretty awesome but our routes are simple. They don't support keyword parameters in the url path. What if we want to have this route of @app.route("/hello/{person_name}") and be able to use this person_name inside our handlers like this:

def say_hello(request, response, person_name):
    resp.text = f"Hello, {person_name}"

For that, if someone goes to the /hello/Matthew/, we need to be able to match this path with the registered /hello/{person_name}/ and find the appropriate handler. Thankfully, there is already a package called parse that does exactly that for us. Let's go ahead and install it:

pip install parse

Let's test it out:

>>> from parse import parse
>>> result = parse("Hello, {name}", "Hello, Matthew")
>>> print(result.named)
{'name': 'Matthew'}

As you can see, it parsed the string Hello, Matthew and was able to identify that Matthew corresponds to the {name} that we provided.

Let's use it in our find_handler method to find not only the method that corresponds to the path but also the keyword params that were provided:

# api.py
from webob import Request, Response
from parse import parse

class API:
    ...

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            parse_result = parse(path, request_path)
            if parse_result is not None:
                return handler, parse_result.named

        return None, None

    ...

We are still iterating over self.routes and now instead of comparing the path to the request path, we are trying to parse it and if there is a result, we are returning both the handler and keyword params as a dictionary. Now, we can use this inside handle_request to send those params to the handlers like this:

# api.py
from webob import Request, Response
from parse import parse

class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

    ...

The only changes are, we are getting both handler and kwargs from self.find_handler, and passing that kwargs to the handler like this **kwargs.

Let's write a handler with this type of route and try it out:

# app.py
...

@app.route("/hello/{name}")
def greeting(request, response, name):
    response.text = f"Hello, {name}"

...

Restart your gunicorn and go to http://localhost:8000/hello/Matthew/. You should the wonderful message of Hello, Matthew. Awesome, right? Add a couple more such handlers of yours. You can also indicate the type of the given params. For example you can do @app.route("/tell/{age:d}") so that you have the param age inside the handler as a digit.

Conclusion

This was a long ride but I think it was great. I personally learned a lot while writing this. If you liked this blog post, please let me know in the comments what other features we should implement in our framework. I am thinking of class based handlers, support for templates and static files.

Fight on!

Check out Part II here

Django Static Files

2018-06-07T20:10:00+05:00

Have you ever been frustrated when your Django application did not find the static files you were using? Then, you read the documentation and played around with all those settings variables and after a couple of hours it was finally fixed. However, you had no idea what you just did. I have been in this situation so many times that I decided to write this post where I explain everything you need to configure your static files in a Django application.

The first thing you need to know is that static files are stored at $APP/static/ where $APP is an application name:

These are usually static files such as css and javascript files that are related to that $APP.

You can also tell Django to look for static files in some arbitrary locations of your choice. You use STATICFILES_DIRS for that:

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "unrelatedstatic")
]

These are usually static files that are not related to any app that you have in your Django application.

Now, keep in mind that these different folders are where you store your static assets DURING DEVELOPMENT. When you deploy your app to the production, you want to configure your web server (e.g. NGINX) to look for static files in one single location rather than multiple paths. How do you collect all your static files from different apps to a single folder? As simple as this:

python manage.py collectstatic

However, this will throw an exception because it does not know where to collect the files yet. You need to tell it by setting the STATIC_ROOT setting:

STATIC_ROOT = os.path.join(BASE_DIR, "allstaticfiles")

Now run that command and all you static files will be collected to allstaticfiles folder in your project root:

Again, you DON'T PUT any static file in STATIC_ROOT. That's where files get collected automatically after you run collectstatic.

Now, in production, you can configure your web server to serve static files from that location. In NGINX, it would look like this:

location /static/ {
    root   /my/project/allstaticfiles/;
}

And the last piece is to set STATIC_URL to your STATIC_ROOT so that when you do {% static 'main.css' %} in a django template, it uses the allstaticfiles folder in production.

Again, STATIC_URL is set to static during development and to STATIC_ROOT in production:

I learned this while attending DjangoCon Europe 2018 and thus all credits go to Curtis Maloney who presented this in a lighting talk. It made my life easier and I hope it does the same for everybody else.

Fight on!

Branch by Abstraction

2018-04-08T20:10:00+05:00

Agile, devops, continuous integration, continuous delivery, scrum, kanban, automation... How many times a day do you hear one of these words? I come across these words every single day whether it be in a book or during a conversation with a co-worker. There is a reason for it though. Everybody is trying to improve their team's performance so that they deliver their products to the market faster than their competition. And these words/techniques/practices, if adopted well, help you achieve this. The best book I can recommend to understand their benefit is The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

Make sure to read this book if you are at all interested in improving your own and your team's performance.

If you have read it, you know that one of the biggest and most important pillars of a DevOps team is continuous integration. It means that every commit is integrated to the mainline (master/trunk) branch all the time.

Now imagine that you have a big improvement to do in your project. Your team decided to switch from Peewee ORM to SQLAlchemy. How would one normally go about doing this task? He would open a new branch and do everything there. As this is a big task, it would take him days, if not weeks, to finish it. At the end, he would merge this branch with the master. He would probably have lots of conflicts. He would eventually merge everything and deploy it to production. Users would probably complain that some things are not working at all. Something went wrong while merging the conflicts. It would take him a couple more days to fix everything.

You can't blame this guy. Merging is difficult and scary. You should blame the process. The team was not continually integrating their code. If they were, they would not have had to merge all the things at the end. They would integrate every commit and as commits are small, it would not be difficult to merge.

"But this is a big task. How can you integrate all your commits to the master all the time and deploy to production? Users would see the incomplete work in progress" I hear you say. Let me show you how you can accomplish this feat painlessly both for you and your users.

Feature Toggles

The first thing we need to understand is Feature Toggles. This is a simple but powerful concept. Imagine you developed a big feature. But you are too afraid to open it up for thousands of users. You just want to test it with a couple hundred users and gradually increase this number. You can do something like this:

if feature_is_turned_on_for_this_user(feature_name, user):
    return the_new_interface
else:
    return the_old_interface

That is it. Feature Toggles are simply an if statement with a fancy name :)

Branch by Abstraction

This one is more interesting and is implemented in several steps.

Step I is a situation described above. There are some parts of you code that use Peewee ORM code that you want to replace:

Flawed Code

In Step II, we create an abstraction layer for the Peewee code and make one of the clients to work with this abstraction layer using Feature Toggles:

Abstraction Layer

In Step III, we move all the clients to work with this abstraction layer only:

All clients working with abstraction layer

In Step IV, we add the new SQLAlchemy code and make one of the clients to work with the new code using Feature Toggles:

One client working with SQLAlchemy

In Step V, we move all the clients to use the new code:

All clients working with the new code

At this point you have successfully and undetectably moved your old Peewee code to the new SQLAlchemy code. If something goes wrong, you can easily bring back the old code until you fix the bug in the new code. How? Feature Toggles, baby!

If everything is working okay, you may even delete the abstraction layer:

No abstraction layer

That is it. No branches, no merges needed. Everything has been done in the master branch.

"This is all good and dandy but some code would be great" I hear you say. Cool. Let's go. But remember that this is an imaginary case and I will be using Python-like pseudo-code.

Step I

Some parts of your software uses Peewee code that you want to replace:

def client_one():
    ...
    peewee = Peewee('my_database')
    users = peewee.get_users()
    ...

def client_two():
    ...
    peewee = Peewee('my_database')
    new_user = peewee.create_user(username='rahmonov', password='rockstar')
    ...

def client_three():
    ...
    peewee = Peewee('my_database')
    user = peewee.get_user(username='rahmonov')

Step II

We add an abstraction layer and use it in one of the clients:

class DatabaseManager:
    def __init__(self, database_url, use_sqlalchemy=False)
        self.use_sqlalchemy = use_sqlalchemy
        self.peewee = Peewee('database_url')

    def get_users(self):
        users = self.peewee.get_users()
        return users

    def get_user(self, username):
        user = self.peewee.get_user(username=username)
        return user

    def create_user(self, username, password):
        new_user = self.peewee.create_user(username=username, password=password)
        return new_user


def client_one():
    ...
    db = DatabaseManager('my_database')
    users = db.get_users()
    ...

Nothing really changed. We have DatabaseManager that acts just like Pewee. At this point, you should make sure that all your unit tests are passing.

Step III

We use the new DatabaseManager in all of our clients:

def client_one():
    ...
    db = DatabaseManager('my_database')
    users = db.get_users()
    ...

def client_two():
    ...
    db = DatabaseManager('my_database')
    users = db.create_user(username='rahmonov', password='rockstar')
    ...

def client_three():
    ...
    db = DatabaseManager('my_database')
    user = db.get_user(username='rahmonov')

Step IV

We add the new SQLAlchemy code to the DatabaseManager class and use it in one of the clients:

class DatabaseManager:
    def __init__(self, database_url, use_sqlalchemy=False)
        self.use_sqlalchemy = use_sqlalchemy
        self.peewee = Peewee('database_url')
        self.sqlalchemy = SQLAlchemy('database_url')

    def get_users(self):
        if self.use_sqlalchemy:
            users = self.sqlalchemy.retrieve_all_users()
        else:
            users = self.peewee.get_users()

        return users

    def get_user(self, username):
        if self.use_sqlalchemy:
            user = self.sqlalchemy.fetch_user(username=username)
        else:
            user = self.peewee.get_user(username=username)

        return user

    def create_user(self, username, password):
        if self.use_sqlalchemy:
            new_user = self.sqlalchemy.create_user(username=username, password=password)
        else:
            new_user = self.peewee.create_user(username=username, password=password)

        return new_user


features = {
    'sqlalchemy': {'client_one': True, 'client_two': False, 'client_three': False}
}


def client_one():
    ...
    db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_one'])
    users = db.get_users()
    ...

Note the features dictionary which we are using to show how feature-toggles work. Ideally, those configurations should be stored in a database or something else that you can change without deploying your code. Imagine something goes wrong. You will just have to set the value to False and it will start using the old code automatically without any deployment.

Step V

We use the new code in all of our client codes:

features = {
    'sqlalchemy': {'client_one': True, 'client_two': True, 'client_three': True}
}

def client_one():
    ...
    db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_one'])
    users = db.get_users()
    ...

def client_two():
    ...
    db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_two'])
    users = db.create_user(username='rahmonov', password='rockstar')
    ...

def client_three():
    ...
    db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_three'])
    user = db.get_user(username='rahmonov')

Make sure that all unit tests pass. If something goes wrong, turn off the feature for everybody, fix the bug and turn it back on.

When everything is good, go on to the next step:

Step V (optional)

You can optionally delete the abstraction layer and the dead Peewee code as well.

def client_one():
    ...
    sqlalchemy = SQLAlchemy('database_url')
    users = sqlalchemy.retrieve_all_users()
    ...

def client_two():
    ...
    sqlalchemy = SQLAlchemy('my_database')
    new_user = sqlalchemy.create_user(username='rahmonov', password='rockstar')
    ...

def client_three():
    ...
    sqlalchemy = SQLAlchemy('my_database')
    user = sqlalchemy.fetch_user(username='rahmonov')

That's it! You have successfully worked on a big improvement without using branches, i.e. directly on the master branch. Add small commits and unit tests to this mix and you will have the famous Trunk Based Development implemented, which is a theme for another post.

Let me know in the comments if you have any questions.

Fight on!

Static Site Generator in Python

2018-03-31T20:10:00+05:00

Static site generators are all the rage lately. This very website that you are reading is also powered by a static site generator and I love it. But why have they become so popular? What advantages do they have in comparison to some content management system such as Wordpress?

First of all, Wordpress uses server-side code. It means that each time a user enters a website, Wordpress fetches the content from the database and re-creates the pages for you. That is, content is created only after you request it. With static site generators, you pre-create everything and serve the ready content to all users. This is much faster than dynamically recreating the content each time.

Second of all, you need to host your CMS somewhere and take care of your infrastructure. Although, it is pretty cheap now, it is still a headache. With static site generators, you have GitHub Pages which takes care of everything for you absolutely for free. It is extremely easy as well.

Additionally, as CMSs rely on a programming language and a database, it is by definition less secure than a static web site.

How it works

The way static site generators work is pretty simple. You write your blog posts in a markdown file and when you run your static site generator, it will compile your posts into pre-created templates which then can be served to your readers.

For example, let's say that you have a markdown file introduction-to-kubernetes.md in which you have your post content:

title: Introduction to Kubernetes
date: 2018-02-03 20:10

Looking back, 2017 was the year Kubernetes conquered the container orchestration space. For years, Kubernetes' rivals
such as Docker Swarm and Mesos have been offering their own container orchestration tools and now they both added
support for Kubernetes within their ecosystems. The largest cloud providers such as AWS, Microsoft Azure and Oracle
Cloud announced Kubernetes integrations into their respective cloud platforms, not mentioning Google where Kubernetes
came from originally. So, every developer would benefit from at least learning the basics of Kubernetes. That's exactly
what we are going to do in this post.

Also, you have this template:

<html>
<head>

<title>{{ title }}</title>

</head>

<p>Published at: {{ date }}</p>

<body>
{{ content }}
</body>

</html>

And when you run your static site generator, you will have something like this as a result:

<html>
<head>

<title>Introduction to Kubernetes</title>

</head>

<p>Published at: 2018-02-03 20:10</p>

<body>
Looking back, 2017 was the year Kubernetes conquered the container orchestration space. For years, Kubernetes' rivals
such as Docker Swarm and Mesos have been offering their own container orchestration tools and now they both added
support for Kubernetes within their ecosystems. The largest cloud providers such as AWS, Microsoft Azure and Oracle
Cloud announced Kubernetes integrations into their respective cloud platforms, not mentioning Google where Kubernetes
came from originally. So, every developer would benefit from at least learning the basics of Kubernetes. That's exactly
what we are going to do in this post.
</body>

</html>

What happened is that your generator parsed the markdown and put the details in your template where they belong. Easy!

The next step is you deploy these ready files and your readers will enjoy your fast and secure content. And after a year or so you become famous and be buried with thousands of offers for a job or partnership.

Such a simple but at the same time very powerful idea.

Seems like a pretty good idea for a weekend project, doesn't it? Why don't we try to create our own version of a static site generator, you and I? What do you say? A very basic one. Agree? Cool! Let's get started.

Let's reinvent static site generators

Here is the plan. We will do two pages. On the first main page, we are going to show the list of all posts, sorted by date, with their titles, published date and their summary, which will look something like this:

Index Page

Then, if the user clicks on the title of any post, he is taken to its detail page where he can read the whole content:

Detail Page

Okay? Let's create our project folder then:

mkdir custom-ssg

ssg stands for Static Site Generator, by the way.

In this folder, create the content folder where we will write our markdown files, that is, our blog posts:

cd custom-ssg & mkdir content

In the content folder, create your first blog post file how-to-become-senior.md and copy the contents of this file.

In the project root, create ssg.py which will be our main file where we will write our script to generate static pages.

Now, we need a way to turn markdown files such as the one we created that looks like this:

title: How to become a senior software developer?
date: 2019-03-10 20:10
slug: how-to-become-senior-software-developer
summary: You have been a junior or a middle developer for too long now. You feel like you deserve more. You feel like it is time to grab life by the balls and rise up the career ladder.

You have been a junior or a middle developer for too long now. You feel like you deserve more. You feel like it is time
to grab life by the balls and rise up the career ladder. Well, congratulations! You are so lucky to be reading this blog post
because I am about to reveal a secret ingredient I used to become a senior software developer. This is a unique ingredient
which no one else is aware of and which will skyrocket your career in no time. But I warn you! This is not an easy feat
to accomplish. Ready? Let's go.
...

into something like this:

{
    'date': '2019-03-10 20:10',
    'slug': 'how-to-become-senior-software-developer',
    'summary': 'You have been a junior or a middle developer for too long now. You feel like you deserve more. You feel like it is time to grab life by the balls and rise up the career ladder.',
    'title': 'How to become a senior software developer?'
    'content': '<p>You have been a junior or a middle developer for too long now. You feel like you deserve more.</p>...'
 }

Fortunately, the coolest Python community has already done the work for us and written this awesome library. Before we install it, please note that the best practice is to use a virtualenv for packages. Let's install the package:

pip install markdown2

Now, let's try this out in our ssg.py:

from markdown2 import markdown

with open('content/how-to-become-senior.md', 'r') as file:
    parsed_md = markdown(file.read(), extras=['metadata'])

    print('Metadata: ', parsed_md.metadata)
    print('Content: ', parsed_md)

The output should be the following:

Metadata: {'title': 'How to become a senior software developer?', 'date': '2019-03-10 20:10', 'slug': 'how-to-become-senior-software-developer', 'summary': 'You have been a junior or a middle developer for too long now. You feel like you deserve more. You feel like it is time to grab life by the balls and rise up the career ladder.'}

Content: <p>You have been a junior or a middle developer for too long now. You feel like you deserve more. You feel like it is time...

Great. Exactly what we needed! Now we can use these variables to create pages. But before that we need templates and not just ordinary html files but ones with placeholders for our variables.

For example, this:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>{{post.title}}</title>
</head>
<body>

<h1>{{post.title}}</h1>

<small>{{post.date}}</small>

{{post.content}}

</body>
</html>

If we had this template file, we could use the variables we generated above and insert them into this template. Fortunately, that's easy too thanks to Jinja2, a modern templating language for Python. Install it:

pip install jinja2

To try Jinja2 out, create a folder templates in the project root and create a post-detail.html and put the above html inside this file.

Now, in your ssg.py, append the following code:

from jinja2 import Environment, PackageLoader

env = Environment(loader=PackageLoader('custom-ssg', 'templates'))
post_detail_template = env.get_template('post-detail.html')

data = {
    'content': parsed_md,
    'title': parsed_md.metadata['title'],
    'date': parsed_md.metadata['date']
}

print(post_detail_template.render(post=data))

After this, your ssg.py should look like this:

from jinja2 import Environment, PackageLoader
from markdown2 import markdown

with open('content/how-to-become-senior.md', 'r') as file:
    parsed_md = markdown(file.read(), extras=['metadata'])

    env = Environment(loader=PackageLoader('ssg', 'templates'))
    post_detail_template = env.get_template('post-detail.html')

    data = {
        'content': parsed_md,
        'title': parsed_md.metadata['title'],
        'date': parsed_md.metadata['date']
    }

    print(post_detail_template.render(post=data))

As you can see here, the first thing we need to do is to create an Environment by showing where the templates folder is located. Then, with this environment we get the template we need and at the end we call render() function and pass it the data it needs.

If you run your ssg.py, you will see that the code parsed the markdown and put all the data in its own place in the above template. Great!

But before we can reinvent static site generators, we should create the index.html where we show the list of all posts. So, create index.html inside the templates folder and put the following in this file:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>List of Posts</title>
</head>
<body>

<h1>My Blog Posts</h1>

{% for post in posts %}
  <p>
      <h2>{{loop.index}}: <a href="posts/{{ post.slug }}/">{{post.title}}</a> <small>{{post.date}}</small></h2>
      {{post.summary}}
  </p>
{% endfor %}

</body>
</html>

We are simply looping through the posts list and showing all the data except the main content.

We have all the pieces now and we can get started with building our static site generator.

We will do it in several steps. First, delete the contents of ssg.py. We will write the real code now.

Now, let's import all the packages that we will need along the way:

import os
from datetime import datetime
from jinja2 import Environment, PackageLoader
from markdown2 import markdown

What we will do next is parse all markdown files and collect them in a dictionary so that we can use them later:

POSTS = {}
for markdown_post in os.listdir('content'):
    file_path = os.path.join('content', markdown_post)

    with open(file_path, 'r') as file:
        POSTS[markdown_post] = markdown(file.read(), extras=['metadata'])

Fairly easy. Next we sort these posts by their published date in a reversed order so that we show the newest ones first:

POSTS = {
    post: POSTS[post] for post in sorted(POSTS, key=lambda post: datetime.strptime(POSTS[post].metadata['date'], '%Y-%m-%d  %H:%M'), reverse=True)
}

One thing to note here is we need to convert the date to datetime first in order to properly sort the posts as comparing strings would not work correctly.

Next, we get the html template files with Jinja2:

env = Environment(loader=PackageLoader('ssg', 'templates'))
index_template = env.get_template('index.html')
post_template = env.get_template('post-detail.html')

Now, we can render the index.html by passing the data it needs:

index_posts_metadata = [POSTS[post].metadata for post in POSTS]

index_html_content = index_template.render(posts=index_posts_metadata)

Now we can create write the contents of index_html_content into a real file inside the output folder that we can serve to our users:

with open('output/index.html', 'w') as file:
    file.write(index_html_content)

We don't have the output folder yet, so create it:

mkdir output

If you run your ssg.py at this point, you will have index.html inside your output folder and if you open it in the browser, it will look like this:

Index Page with one post

Try adding these top-books-i-read.md and nginx-ingress.md into your content folder and then run ssg.py again.

You will now see that your index.html has three posts listed:

Index Page with three posts

However, if you click on the title, there will be a not found error as we haven't generated the detail pages yet.

It is also not difficult to do. Append the following code to ssg.py:

# render each post and write it to output/posts/<post.slug>/index.html
for post in POSTS:
    post_metadata = POSTS[post].metadata

    post_data = {
        'content': POSTS[post],
        'title': post_metadata['title'],
        'date': post_metadata['date'],
    }

    post_html_content = post_template.render(post=post_data)

    post_file_path = 'output/posts/{slug}/index.html'.format(slug=post_metadata['slug'])

    os.makedirs(os.path.dirname(post_file_path), exist_ok=True)
    with open(post_file_path, 'w') as file:
        file.write(post_html_content)

As you can see, we loop through the posts, get the needed data, render the template by passing it the data and then create a file out of this rendered data in an appropriate folder.

Now run your ssg.py and then click on the title of a post. You will now see the whole content of the post you clicked:

Detail Page

Congratulations, you have just built your own static site generator. Cool, right? If you want to challenge yourself, go ahead and add other features such as tagging, pagination and deploy your blog in Github Pages.

If something is not clear, let me know in the comments. I may have made some mental shortcuts along the way :)

Fight on!

Python GIL

2017-12-08T20:10:00+05:00

In this blog post, we will look at Python GIL, Threads, Processes and AsyncIO.

Let's say that we want to write a function that takes a number as an argument and simply counts down. Pretty easy:

def count_down(n):
    while n > 0:
        n -= 1

Let's call this function with a big number and measure how much it takes:

from time import time

before = time()
count_down(100000000)
after = time()

print(after-before)

On my machine, it takes 8.54 seconds. Now, let's call it twice and measure that:

from time import time

before = time()
count_down(100000000)
count_down(100000000)
after = time()

print(after-before)

On my machine, it takes 17.38 seconds.

It is working fine but our boss is not happy. He wants us to make it faster. How do we make it faster? We use threads so that these two function calls run in parallel, right? In theory, running a function twice in parallel should take as much as running it once. Because, well, those two function calls run in parallel. Let's try this out by calling the function twice but in different threads:

from threading import Thread
from time import time

before = time()

thread1 = Thread(target=count_down, args=(100000000,))
thread2 = Thread(target=count_down, args=(100000000,))

thread1.start()
thread2.start()

thread1.join()
thread2.join()

after = time()
print(after-before)

It takes around 18.4 seconds on my machine. Wait, WHAT? It not only did not take as much as calling the function once but in fact it was even slower than calling the function twice without Threads. Try that with Python2.7 and you will get even worse results. A prominent software developer, David Beazly reported that it even took almost 2x time in his talk Inside the Python GIL.

But, WHY?

Please, welcome the notorious GIL - Global Interpreter Lock. This guy is to blame for all the problems in the world. GIL allows only one thread to run at a time. In the above example, when we thought we were using two threads, we were only using just one, because the big boss GIL did not allow us to. That's why, it did not run faster. But, why was it even slower? That's because every now and then Python tries to switch Threads as we asked him to use two. GIL does not allow him to do so. However, when he tries to switch to no avail, there is some overhead associated with it. Thus, the end result is even slower.

But, WHY?

We all know that Threads can be a good thing and help us make our programs run much faster. Why does the GIL exist, then? Damn you, GIL! You are like Negan from the Walking Dead.

Negan - GIL

Well, it turns out GIL is not the villain after all. Actually, GIL is the good guy. It turns out that Python's memory management is not thread-safe! That is, if you run multiple threads, your program may behave in weird and even catastrophic ways. GIL is there to help us from such scenarios.

Superman - GIL

But, wouldn't it be easier to remove the Thread class instead?

Excellent question! Instead of writing an additional tool that would prevent Threads from doing bad things to our programs, why not just delete the Thread classes so that programmers simply won't be able to use them? We are not allowed to use them anyways.

Turns out, there are some cases when Thread can be of use. All above examples were CPU bound, meaning that they need only CPU to run, they wait on CPU. However, if your code is IO bound or image processing or NumPy number crunching, GIL won't be in your way as these operations happen outside the GIL. That's where you can use the Thread class safely and effectively.

Let's write some IO bound function. A function that requests a web url and returns its content as a text:

import requests

def get_content(url):
    response = requests.get(url)
    return response.text

Calling this function once with the argument https://google.com takes 0.88 seconds on my machine. Calling it twice - 1.72 seconds, of course. Now, let's call it twice in different threads and see the Thread's effect:

before = time()

thread1 = Thread(target=get_content, args=('https://google.com',))
thread2 = Thread(target=get_content, args=('https://google.com',))

thread1.start()
thread2.start()

thread1.join()
thread2.join()

after = time()

print(after - before)

It takes 0.89 seconds. Hooray! Calling the function twice in different threads took as much as calling it just once. Once again, it worked because our function was IO bound. The same thing did not work above when the function was CPU bound.

There is a catch

Each thread takes some additional memory and thread switching takes some time. Although it is not a lot, it adds up when you have thousands of threads running. Think gigabytes of extra RAM and at least 5% of CPU time spent only for context switching.

In order to solve this, Python developers came up with the asyncio library. Basically, it has its own event loop to control the execution of functions in an asynchronous way inside a single thread. If Threads were controlled by the underlying OS, asyncio knows when to switch tasks with the help of some keywords written by developers themselves. That's right! You get to decide when to switch. Let's convert the above example to use asyncio:

import asyncio
import aiohttp

loop = asyncio.get_event_loop()
session = aiohttp.ClientSession(loop=loop)


async def get_content(pid, url):
    async with session.get(url) as response:
        content = await response.read()
        print(pid, content)

loop.create_task(get_content(1, 'http://asyncio.readthedocs.io/'))
loop.create_task(get_content(2, 'http://asyncio.readthedocs.io/'))
loop.create_task(get_content(3, 'http://asyncio.readthedocs.io/'))
loop.create_task(get_content(4, 'http://asyncio.readthedocs.io/'))
loop.create_task(get_content(5, 'http://asyncio.readthedocs.io/'))

loop.run_forever()

Note that we are using aiohttp instead of requests because it is an asynchronous http library. Without going into the details, what this code does is define an async function (aka coroutine) and call it 5 times with some id (just to illustrate asynchronicity). Output would be something like this:

3 b'<!DOCTYPE html PUBLIC...
4 b'<!DOCTYPE html PUBLIC...
2 b'<!DOCTYPE html PUBLIC...
1 b'<!DOCTYPE html PUBLIC...
5 b'<!DOCTYPE html PUBLIC...

Note that ids are not in the order they were called, meaning that the tasks were switched when they encountered the keyword await. We get the same results as using Threads but asyncio uses less overhead and you control the switching process.

Although this does not explain the details of asyncio, I hope it at least explains why it is useful.

Okay, now that we dealt with IO bound tasks, let's go back to our first CPU bound problem.

There is still hope for CPU bound tasks!

Turns out there is multiprocessing.Process class in Python which offers similar functionality and interface to the Thread class. The difference is that it uses sub-processes instead of threads. That's why, it won't be blocked by the GIL. Awesome! Let's try that out with the above examples. As I said, it offers a similar interface to the Thread class and thus we will just have to replace Thread with Process and we are done:

from multiprocessing import Process
from time import time

before = time()

process1 = Process(target=count_down, args=(100000000,))
process2 = Process(target=count_down, args=(100000000,))

process1.start()
process2.start()

process1.join()
process2.join()

after = time()
print(after-before)

It took ~8 seconds on my machine. That is as much as it took when we called the function just once. Cool! Try adding one more process and it will still take the same time. That is awesome! However, do keep in mind that processes run in a separate memory space thus can't share data with each other, whereas, threads can.

Also keep in mind that we are talking about the CPython here. Other implementations such Jython and IronPython don't have GIL so they can use Threads. If you don't know what Jython or IronPython are, refer to this post of mine.

Thanks for reading and make sure to leave a comment if you have any questions.

Fight on!

What is Python?

2017-10-23T20:10:00+05:00

Would you be surprised if I said that Python is not a programming language at all?

Then, let me tell you that Python is actually this guy:

This is the actual Python

Just kidding . However, even in a programming sense Python is not a language in itself. It is an interface. It is basically a description of what a language should do and how it should do it. Don't believe me? Here is the official specification of this interface. Take if statements as an example. Here is what this document says about them:

The if statement is used for conditional execution:

if_stmt ::=  "if" expression ":" suite
             ( "elif" expression ":" suite )*
             ["else" ":" suite]

It selects exactly one of the suites by evaluating the expressions one by one until one is found to be 
true (see section Boolean operations for the definition of true and false); then that suite is executed 
(and no other part of the if statement is executed or evaluated). If all expressions are false, the 
suite of the else clause, if present, is executed.

"But what have I been using all my life then ?" I hear you ask. You have been using the default implementation of Python, which is actually called CPython. That means that by Python most people mean CPython. Read on to understand what that is.

As with any interface, there are its implementations. And the most popular Python implementations are the following:

CPython - The default and most widely used implementation written in C
Jython - Written in Java and for the JVM
IronPython - Written in C# and tightly integrated with the .NET framework.
PyPy - Alternative implementation of Python written in (surprise!) Python.

And here is the most interesting part: as these implementations are written in different languages, they allow you to use them inside your Python program. Specifically, CPython allows to use C and C++, Jython allows to use Java and IronPath allows to use C# in your Python program. How crazy is that?

Let's see some examples. Let's say that I have this function called example written in C:

static PyObject * example(PyObject *self)
{
    // do something
    return Py_BuildValue("i", result);
}

In CPython, I can import it and call it like this:

import example

result = example.do_something()

These are called extensions. Read more about them here.

The same thing with Jython. You can import Java classes and use them inside your Python:

C:\jython>jython
Jython 2.0 on java1.2.1
Type "copyright", "credits" or "license" for more information.
>>> from java.util import Random
>>> r = Random()
>>> r.nextInt()
-790940041

And the same thing with IronPython. Import .NET code and use it as you like:

>>> from System.DateTime import Now
>>> Now #doctest: +ELLIPSIS
<System.DateTime object at ...>
>>> # Let's make it even more obvious that "Now" is evaluated only once
>>> a_second_ago = Now
>>> import time
>>> time.sleep(1)
>>> a_second_ago is Now
True
>>> a_second_ago is System.DateTime.Now
False

I don't know about you but mixing up languages like this is fascinating to me, although I haven't had a chance to use any of them in a real project except for CPython, of course.

However, that might be not too far away. Hailed as the future of Python, PyPy has been showing great progress. Even the creator of Python, Guido van Rossum said some nice words about it:

If you want your code to run faster, you should probably just use PyPy.

Its distinctive features, apart from being fast thanks to its Just-in-Time compiler, include optimized memory usage, support for stackless which makes massive concurrency possible and most importantly its compatibility. That means that you can swap your Python implementation with PyPy without changing your codebase and you will have all those advantages that PyPy offers. You can read more about its features here.

Conclusion

Python is actually an interface and there are several different implementations, with the most commonly used being CPython and most promising being PyPy. Thank you for reading!

Fight on!

Custom ChainMap in Python

2017-10-21T20:10:00+05:00

In the previous post, we learned what ChainMap is in Python and how/why to use it. In this blog post, we will awaken our inner hacker and try to implement the same thing ourselves. Just for fun :) Please, check out this post if you don't know what ChainMap is.

As software developers, we often hear the words "Don't reinvent the wheel" which basically means "Don't recreate basic stuff which has been created before for you". However, I believe that occasional reinventions of some "wheels" help us understand some things from the inside out and grow as software developers.

So, let's go ahead and reinvent ChainMap in Python.

First things first, let's do the initialization process. The original version takes an undefined number of dictionaries upon initialization and store them inside the maps property:

>>> from collections import ChainMap

>>> a = {'name': 'hell', 'value': 'yeah'}
>>> b = {'name': 'paradise', 'value': 'no'}

>>> first_chain = ChainMap(a)
>>> print(first_chain.maps)
[{'value': 'yeah', 'name': 'hell'}]
>>> second_chain = ChainMap(a, b)
>>> print(second_chain.maps)
[{'value': 'yeah', 'name': 'hell'}, {'value': 'no', 'name': 'paradise'}]
>>> third_chain = ChainMap()
>>> print(first_chain.maps)
[{}]

Based on these observations, we can conclude that we should override the __init__ method and it should take a number of dictionaries and store them in maps property which is a list. If no dictionary is given, maps should be equal to a list of one empty dict:

class MyChainMap:
    def __init__(self, *maps):
        self.maps = list(maps) or [{}]

    def get(self, key, default=None):
        for map in self.maps:
            if key in map:
                return map[key]

        return default

maps should be converted to list because *maps is a tuple. Let's try it out:

>>> my_chain = MyChainMap(a, b)
>>> print(my_chain.maps)
[{'name': 'hell', 'value': 'yeah'}, {'name': 'paradise', 'value': 'no'}]

Great! Next, let's do the get() method. In the original version, get takes two parameters: key and default=None. It tries to find the key and returns the value if found. Otherwise, it returns the default (whose default is None):

>>> print(first_chain.get('name'))
'hell'
>>> print(first_chain.get('nonexistent_key') is None)
True
>>> print(first_chain.get('nonexistent_key', 'default_value'))
'default_value'

We will try to find the given key in our maps and return the first found result, otherwise we will return the default value:

class MyChainMap:
    ...

    def get(self, key, default=None):
        for map in self.maps:
            if key in map:
                return map[key]

        return default

Let's try it out:

>>> my_chain = MyChainMap(a, b)
>>> print(my_chain.get('name'))
'hell'
>>> print(my_chain.get('nonexistent_key') is None)
True
>>> print(my_chain.get('nonexistent_key', 'default_value'))
'default_value'

Works and looks the same. Great! We know that in the original ChainMap, we can retrieve values just like dicts:

>>> first_chain['name']
'hell'
>>> first_chain['nonexistent_key']
KeyError: 'nonexistent_key'

It returns the value if found, otherwise throws KeyError. From Python's data model, we know that the special __getitem__ method is called when accessing a key like a dict: first_chain['name']. So, we will override it accordingly:

class MyChainMap:
    ...

    def __getitem__(self, key):
        for map in self.maps:
            if key in map:
                return map[key]

        raise KeyError(key)

Let's try it out:

>>> my_chain = MyChainMap(a, b)
>>> my_chain['name']
'hell'
>>> my_chain['nonexistent_key']
KeyError: 'nonexistent_key'

Very nice!

Next, let's implement the new_child method of ChainMap. As we know, it takes one dictionary as a parameter and returns a new object with whose maps consists of this new dictionary and its other dictionaries:

class MyChainMap:
    ... 

    def new_child(self, new_map):
        return MyChainMap(new_map, *self.maps)

So, we just initialized a new object with the new map and old ones. Easy! Let's try it out:

>>> my_chain = MyChainMap(a)
>>> my_new_chain = my_chain.new_child(b)
>>> my_new_chain.maps
[{'name': 'paradise', 'value': 'no'}, {'name': 'hell', 'value': 'yeah'}]

Great! And lastly, let's implement the parents property of the ChainMap. We know that it returns a new ChainMap with all the maps except for the first one. So, it should be pretty easy as well:

class MyChainMap:
    ... 

    @property
    def parents(self):
        return MyChainMap(*self.maps[1:])

and there you go! Please note that it is a property not a callable function, just like in the original implementation. Let's try it out:

>>> a = {'name': 'hell', 'value': 'yeah'}
>>> b = {'name': 'paradise', 'value': 'no'}

>>> my_chain = MyChainMap(a, b)
>>> print(my_chain.maps)
[{'value': 'yeah', 'name': 'hell'}, {'value': 'no', 'name': 'paradise'}]
>>> chain_parents = my_chain.parents
>>> print(chain_parents.maps)
[{'value': 'no', 'name': 'paradise'}]

Awesome! And there you have it. A custom version of ChainMap. But it is not yet complete. I will leave the rest of it up to you .

Go ahead and try to implement pop, clear, __setitem__ and __delitem__ methods of ChainMap and let me know in the comments if you did or have any questions.

Be a hacker and fight on!

Python ChainMap

2017-10-17T20:10:00+05:00

In Part I of these series about Python's collections module, we talked about the Counter class and its usage. In this blog post, we will be looking at another class in this module: ChainMap.

Let's say that we are building this awesome web app which is expected to bring us billions of dollars. In this app, we have two environments: development and production. Each of these environments has its own configurations and we store those configs in dictionaries, like this:

development = {'app_name': 'Billions', 'database': '132.123.33.2', 'db_username': 'development_user'}

production = {'database': '222.44.55.16', 'db_username': 'production_user'}

When deploying our app in production, we first look up the production dictionary for a value. If nothing is found, then we look it up from the development dictionary. Nothing new, classic config; production overrides development.

To do that, let's create a function called get_config, which takes name as a parameter and returns the result if it finds it, or else None:

>>> def get_config(name):
...     if name in production:
...         return production[name]
...        
...     if name in development:
...         return development[name]
...        
...     return None

Let's test it:

>>> get_config('database')
'222.44.55.16'
>>> get_config('db_username')
'production_user'
>>> get_config('app_name')
'Billions'
>>> get_config('api_key') is None
True

Looks like it is working well. Please note that this method is for production only. In development, we would first search in the development dict and only then production.

This is working well, but I don't like it. There has to be a better way. What if I combine them in one dict? Like this:

>>> development = {'app_name': 'Billions', 'database': '132.123.33.2', 'db_username': 'development_user'}
>>> production = {'database': '222.44.55.16', 'db_username': 'production_user'}
>>>
>>> common = dict()
>>> common.update(development)
>>> common.update(production)

Looks good. Let's test it:

>>> common.get('database')
'222.44.55.16'
>>> common.get('db_username')
'production_user'
>>> common.get('app_name')
'Billions'
>>> common.get('api_key') is None
True

Cool! It is behaving in the same way as the method we wrote above. However, we did it without having to write a function.

Well, it turns out that there is even better way. Welcome ChainMap!

Basically, what it does is to group multiple dicts into one, updateable view which has the same interface as the ordinary dict (with some additions, of course). Let's see it in action:

>>> from collections import ChainMap
>>>
>>> development = {'app_name': 'Billions', 'database': '132.123.33.2', 'db_username': 'development_user'}
>>> production = {'database': '222.44.55.16', 'db_username': 'production_user'}
>>>
>>> chain = ChainMap(production, development)
>>>
>>> chain.get('database')
'222.44.55.16'
>>> chain.get('db_username')
'production_user'
>>> chain.get('app_name')
'Billions'
>>> chain.get('api_key') is None
True

Awesome! Look how much cleaner it got. With one single line, we accomplished all those things that we did above.

Other Features

As I mentioned above, it looks and behaves just like an ordinary dict. However, it has some extra functionality.

You can see the list of comprising dictionaries:

>>> chain.maps
[{'database': '222.44.55.16', 'db_username': 'production_user'}, {'database': '132.123.33.2', 'db_username': 'development_user', 'app_name': 'Billions'}]

You can reverse this order:

>>> chain.maps = list(reversed(chain.maps))
>>> chain.maps
[{'database': '132.123.33.2', 'db_username': 'development_user', 'app_name': 'Billions'}, {'database': '222.44.55.16', 'db_username': 'production_user'}]

This means that now when look something up, it will first loop up the development dictionary because we reversed the order:

>>> chain.get('db_username')
'development_user'

Cool!

You can also add another dictionary to the group:

>>> most_important_config = {'db_username': 'I am the king'}
>>> chain = chain.new_child(most_important_config)
>>> chain.get('db_username')
'I am the king'

As you can see, the one we just added was added as the first child and thus it will look it up first.

And lastly, you can see parents of this chain which basically means see all comprising dictionaries except the first one:

>>> >>> chain.parents
ChainMap({'database': '132.123.33.2', 'db_username': 'jahongirr', 'app_name': 'Billions'}, {'database': '222.44.55.16', 'db_username': 'production_user'})

As you can see here, contents of most_important_config is missing as it is the first element of the ChainMap.

Conclusion

In this blog post, we saw what ChainMap is, how to use it and most importantly why to use it. Always learn why something should be used. Otherwise, it is easy to forget or misuse it.

Fight on!

Python Generators

2017-09-02T20:10:00+05:00

This is a tale of how one programmer learned the basics of generators in Python and is told from his point of view.

The Tale

So, my boss told me to write a function that accepts n as a parameter and generates this many random numbers and returns the result. That should not be too difficult:

import random

def random_nums(n):
    nums = []
    for i in range(n):
        nums.append(random.randint(0, 1000))   # random number is between 0 and 1000
    return nums

Let me test it out:

>>> random_nums(10)
[886, 348, 763, 493, 401, 219, 911, 789, 834, 940]

>>> random_nums(5)
[366, 160, 951, 682, 153]

It seems to be working well. However, I have a few concerns about this function. First of all, what if the n param is really large? Because the function stores the random numbers in a list (nums = []), that list will get bigger and bigger, and eventually the machine may run out of memory.

Second of all, the function is eager which means that it will go from the start till the end without any pause. This, in turn, means that if I want to do something like this:

for i in random_nums(10000000):
    print(i)

It will first execute random_nums(10000000) part and then start the loop. It means that even if I needed only the first element of the list, I would have to wait for all 10000000 numbers to be generated. That sucks

Waiting for the first number...

Well, there must be a better way. There must be a solution which solves both memory and eagerness issues. So, in order to achieve that I must get rid of storing the numbers and make the function lazy so that it generates one number at a time. What if I create such a class? That should not be too difficult. I could use __iter__ and __next__ functions to make my class iterable and in that __next__ I will return one number at a time and remember the last index:

class BetterRandomNums:
    def __init__(self, n):
        self.n = n
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        random_number = random.randint(0, 1000)

        if self.index >= self.n:
            raise StopIteration('No more numbers')

        self.index += 1

        return random_number

Let me try it out now:

Better random number generator

Now, it is better! Basically, it is generating one random number, giving me that number and letting me print it and only then going on to create the next random number. Exactly what I wanted

Now, if I try to access the first number, it will happen immediately:

Fast access to the first number

There you go. This is what I wanted to do. But boy does it look ugly and I can't believe that I had to do all these things to get this done. This is Python, there must be a Pythonic way of doing it.

After doing a little research...

Hooray! It turns out there is such a thing and it is called generators

Using a generator, our new function will look like this vs the old one:

# the new function                                            # the old one
def random_nums2(n):                                |         def random_nums(n):
    for i in range(n):                              |            nums = []    
        rand_num = random.randint(0, 1000)          |            for i in range(n):
                                                    |                nums.append(random.randint(0, 1000))
        yield rand_num                              |            return nums

So, what changed? They look very similar. The are two small differences, though. First, the new function does not store the results in a list. That's why, it will not use any memory for that. Second, instead of return we have yield. This is the keyword that turns a function into a generator. As a result, it will generate a random number, give a user that random number, and then that's it; it will remember that it produced one random number and wait for the user to continue. Only when the user calls it again, it will generate the second random number.

Let's try it out:

Generator at work

Basically, it is working just like the BetterRandomNums class above. But it is much more compact and does not require knowledge of all those magic methods that the class had. As PewDiePie would say:

Very nice

Conclusion

There you have it. We had problems of memory and an eager function. We solved them with the help of generators. This tale was intended to show not only how to use generators but also why we need to use them. I hope it is more or less clear. If not, contact me and I will do my best to explain things further.

Thanks for reading and fight on!

Python Decorators

2017-08-30T20:10:00+05:00

The following is a tale of how one programmer learned the basics of decorators in Python and is told from his point of view.

The Tale

So, my boss told me to write two functions both of which return whatever they accept. Well, that seems pretty easy:

def first_func(arg):
    return arg

def second_func(arg):
    return arg

When I asked him what the difference was between these two functions he said that the first one should only accept integers and the second one should only accept strings, otherwise they both should raise an AssertionError with a helpful message.

Okay, that should not be too difficult:

def first_func(arg):
    assert isinstance(arg, int), "{arg} is not an instance of int".format(arg=arg)
    return arg

def second_func(arg):
    assert isinstance(arg, str), "{arg} is not an instance of str".format(arg=arg)
    return arg

Let's try them out:

>>> first_func(123)
>>> 123

>>> first_func('some text')
---------------------------------------------------------------------------
AssertionError: some text is not an instance of int

>>> second_func('text')
>>> 'text'

>>> second_func(123)
---------------------------------------------------------------------------
AssertionError: 123 is not an instance of str

Cool, they seem to work pretty well. But it bothers me that those assertion lines seem exactly the same and I am not following the DRY principle. Uncle Bob will kill me. I think I should probably factor them out to a separate function and call where needed. The function should accept an argument and a type and check if that argument is an instance of that type. If not, raise an assertion error:

def assert_type(arg, type):
    assert isinstance(arg, type), "{arg} is not an instance of {type}".format(arg=arg, type=type)

Now I can call this in the two functions that I wrote earlier:

def first_func(arg):
    assert_type(arg, int)
    return arg

def second_func(arg):
    assert_type(arg, str)
    return arg

Looks much better now and they are producing the same results. Hmmm, I have a strange feeling that I could improve this further. It is Python after all. There should be something more pythonic. As Raymond Hettinger would say:

There must be a better way!

I need to do a little research on how I can accomplish this feat.

...after 3 days...

Oh wow! I watched this amazing talk by James Powell and learned that there is a feature in Python that does exactly what I want to do with my functions and it is called decorators

Basically, what they allow me to do is add an additional functionality to my existing functions. That's exactly what I want to do: I want to add type-checking functionality to my existing first_func and second_func. Let's see an example:

def p_decorate(func):
   def func_wrapper(name):
       return "<p>{0}</p>".format(func(name))
   return func_wrapper

@p_decorate
def get_text(name):
   return "lorem ipsum, {0} dolor sit amet".format(name)

So, in this example, there is a get_text function which accepts a parameter name and returns a string of that name inside a random text. There is another method called p_decorate. It accepts a function and returns a function which is declared inside this p_decorate. It accepts name as a parameter. It then surrounds it with <p></p> tags and returns the result. And one more thing to note is that there is @p_decorate on top of def get_text(name).

This is all very mysterious to me but let's see what result it will yield:

>>> get_text('John')
>>> '<p>lorem ipsum, John dolor sit amet</p>'

Okay, the expected result. Let's try to understand what is going on here:

First, when I call get_text, it will actually call p_decorate(get_text) because of @p_decorate on top of the function declaration. And what will p_decorate(get_text) return? It will return another function called func_wrapper. So, basically get_text(name) will be replaced by func_wrapper(name) which will return p tag surrounded string that we saw.

Now, it is much clearer to me. Back to my own functions.

Using these decorators, I want my end result to look like this:

@assert_type(int)
def first_func(arg):
    return arg

It means that our decorator will have one more layer which will accept the type as parameter and will have two inner functions. Let's try to write that decorator:

def assert_type(type):
    def wrapper(func):
        def decorated_func(arg):
            assert isinstance(arg, type), "{arg} is not an instance of {type}".format(arg=arg, type=type)
            return func(arg)

        return decorated_func

    return wrapper

Cool! And now I can use this decorator as I wanted:

@assert_type(int)
def first_func(arg):
    return arg

@assert_type(str)
def second_func(arg):
    return arg

and run some tests:

>>> first_func(123)
>>> 123
>>> first_func('123')
---------------------------------------------------------------------------
AssertionError: 123 is not an instance of <class 'int'>

Voila! It is working as before but now the functions are much better-looking. They are even hot

Wow! I could not even imagine that I would learn so much more than just the decorators.

Along the way, I learned that functions are first-class objects in Python which means that I can use a function as arguments to another function:

>>> def add(x, y):
        return x + y

>>> def apply(func, x, y):
        return func(x, y)

I also learned that functions can have inner functions:

def assert_type(type):
    def wrapper(func):
        def decorated_func(arg):
            assert isinstance(arg, type), "{arg} is not an instance of {type}".format(arg=arg, type=type)
            return func(arg)

        return decorated_func

    return wrapper

And most importantly those inner functions can use the variables from the outer functions and can remember them even when they go out of scope (clojure), like in the following line:

assert isinstance(arg, type), "{arg} is not an instance of {type}".format(arg=arg, type=type)

where type comes from the function above.

Overall, it was a productive day

Conclusion

Thanks for reading thus far. This post was not written to illustrate what exactly decorators are or how they work. But rather, it was written in order to show why we need decorators and how they can improve our code. That's why, in order to make your decorators knowledge comprehensive, please go ahead and read the these blog posts for details. They are awesome!

P.S. There is the complementary video that I have made on Python Decorators :)

Fight on!

What the hell is WSGI anyway and what do you eat it with?

2017-07-25T20:11:00+05:00

Before we dive into the details of WSGI, why don't we take a bird-eye's view on what happens when a user uses our web application.

Part I: the world through a web server's eyes

Imagine for a moment that you are a web server (e.g. a gunicorn). Your job consists of the following parts:

You sit around and wait patiently for a request from some kind of a client
When a client comes to you with a request, you receive this request
Then, you take this request to a guy called PythonApp and say to him: "Hey dude, wake up! Here is a request from a very important client. Please, do something about it"
You get a response from this PythonApp guy
You then take this response back to your client

This is the only thing you do. You just serve your clients. You know nothing about content or anything. That's why you are so good at it. You can even scale up and down processing depending on the demand from the clients. You are so focused on this task.

Part II: PythonApp guy

PythonApp guy is your software (duh!). Whereas a web server should exist and wait for an incoming request all the time, your software exists only at the execution time:

A web server wakes it up and gives him the request
It takes the request and executes some commands on it
It returns a response to the web server
It goes to sleep
Web server takes this response back to his client

The only thing it does is execute, not sit around and wait.

The Problem

The scenario above is all good and roses. However, a web server's conversation with the PythonApp guy could have gone a little differently. Instead of:

Hey dude, wake up! Here is a request from a very important client. Please, do something about it

it could have been like this:

Эй, чувак, проснись! Вот запрос от очень важного клиента. Пожалуйста, сделай что нибудь

or it could have been like this:

Ehi amico, svegliati! Ecco una richiesta da un cliente molto importante. Si prega, fare qualcosa al riguardo

or even like this:

嘿，伙计，醒醒吧！这里是一个非常重要的客户端的请求。请做点什么

Do you get it? The web server could have behaved in a number of different ways and the PythonApp guy had to learn all these languages to understand what it is saying and behaving accordingly.

What this means is that, in the past you had to adapt your software to fit the requirements of a web server. Moreover, you had to write different kinds of wrappers in order to make it suitable across different web servers. What developer wants to deal with such things instead of writing code?

WSGI to the rescue

Here is where the WSGI comes in! Understand it as a SET OF RULES for a web server and a web application. The rules for a web server look like this:

Okay, if you want to talk to that PythonApp guy, speak these words and sentences. Also, learn these words as well which he will speak to you. Furthermore, if something goes wrong, here are the curse words that the PythonApp guy will be saying and here is how you should react to them

And the rules for a web application look like this:

Okay, if you want to talk to a web server, learn these words because a web server will be using them when addressing you. Also, you use the following words and be sure that a web server understands them. Furthermore, if something goes wrong, use these curse words and behave in this way

Enough talk, let's fight

Let's take a look at the WSGI application interface to see how it should behave. According to PEP 333, the document which specifies the details of the WSGI, the application interface is implemented as a callable object such as a function, a method, a class or an instance with a __call__ method. This object should accept two positional arguments and return the response body as strings in an iterable.

The two arguments are:

a dictionary with environment variables
a callback function that will be used to send HTTP status and HTTP headers to the server

Now that we know the basics why don't we create a web framework which will definitely take away some market share from Django itself :) Our web framework will do something that no one is doing right now: IT WILL PRINT ALL ENVIRONMENT VARIABLES IT RECEIVES. Genius!

(have been watching to much Pewdiepie. Goddammit)

Okay, let's create that callable object which receives to arguments:

def application(environ, start_response):
        pass

Easy enough. Now, let's prepare the response body that we want to return to the server:

def application(environ, start_response):
    response_body = [
        '{key}: {value}'.format(key=key, value=value) for key, value in sorted(environ.items())
    ]
    response_body = '\n'.join(response_body)

Easy as well. Now, let's prepare the status and headers, and then call that callback function:

def application(environ, start_response):
    response_body = [
        '{key}: {value}'.format(key=key, value=value) for key, value in sorted(environ.items())
    ]
    response_body = '\n'.join(response_body)

    status = '200 OK'

    response_headers = [
        ('Content-type', 'text/plain'),
    ]

    start_response(status, response_headers)

And finally, let's return the response body in an iterable:

def application(environ, start_response):
   response_body = [
       '{key}: {value}'.format(key=key, value=value) for key, value in sorted(environ.items())
   ]
   response_body = '\n'.join(response_body)

   status = '200 OK'

   response_headers = [
       ('Content-type', 'text/plain'),
   ]

   start_response(status, response_headers)

   return [response_body.encode('utf-8')]

That's it. Our genius web framework is ready. Of course, we need a web server to serve our application and here we will be using Python's bundled WSGI server. But if you want to learn the WSGI server interface, take a look at here.

Now, let's serve our application:

from wsgiref.simple_server import make_server

def application(environ, start_response):
    response_body = [
        '{key}: {value}'.format(key=key, value=value) for key, value in sorted(environ.items())
    ]
    response_body = '\n'.join(response_body)

    status = '200 OK'

    response_headers = [
        ('Content-type', 'text/plain'),
    ]

    start_response(status, response_headers)

    return [response_body.encode('utf-8')]

server = make_server('localhost', 8000, app=application)
server.serve_forever()

Save this file as wsgi_demo.py and run it python wsgi_demo.py. Then, go to localhost:8000 and you will see all the variables listed:

All headers

YES! This framework is going to get very popular!

Now that we know about the WSGI application interface, let's talk about something that we deliberately missed earlier: Middleware.

With middleware, the above scenario will look like this:

Web server gets a request
Now, it won't directly talk to the PythonApp guy. It will send it through a postman (middleware)
The postman delivers the request to the PythonApp guy
After the PythonApp guy does his job, gives the response to the postman
The postman delivers the response to the web server

The only thing to note is that while the postman is delivering the request/response, he may tweak it a little bit.

Let's see it in action. We will now write a middleware that reverses the response from our application:

class Reverseware:
    def __init__(self, app):
        self.wrapped_app = app

    def __call__(self, environ, start_response, *args, **kwargs):
        return [data[::-1] for data in self.wrapped_app(environ, start_response)]

Simple enough. If we insert this code to the example above, the full code will look like this:

from wsgiref.simple_server import make_server


class Reverseware:
    def __init__(self, app):
        self.wrapped_app = app

    def __call__(self, environ, start_response, *args, **kwargs):
        return [data[::-1] for data in self.wrapped_app(environ, start_response)]


def application(environ, start_response):
    response_body = [
        '{key}: {value}'.format(key=key, value=value) for key, value in sorted(environ.items())
    ]
    response_body = '\n'.join(response_body)

    status = '200 OK'

    response_headers = [
        ('Content-type', 'text/plain'),
    ]

    start_response(status, response_headers)

    return [response_body.encode('utf-8')]

server = make_server('localhost', 8000, app=Reverseware(application))
server.serve_forever()

Now, if you run it, you will see something like this:

All headers are displayed in reverse

Beautiful!

Alright, that's it from me today. If you want to learn more about the WSGI, please see the updated PEP 3333. Thanks for reading!

Fight on!

Python Collections - Counter

2017-07-21T20:10:00+05:00

Python has the following general purpose built-in containers: dict, list, set, and tuple. However, apart from them, there are specialized alternative container data types in Python's collections module. In this blog post, we will take a look at the Counter class from this module.

Counter

A Counter is a child class of dict which, as its name suggests, counts hashable objects. Basically, it stores elements as dictionary keys and their counts as dictionary values:

In [1]: from collections import Counter

In [2]: my_list = ['a', 'b', 'c', 'c', 'a', 'd', 'b', 'e', 'a']

In [3]: print(Counter(my_list))
Counter({'a': 3, 'c': 2, 'b': 2, 'e': 1, 'd': 1})

As you can see that it is unordered and is basically saying there are 3 of 'a' in my_list and etc.

Besides initializing from an iterable like we saw in the previous example, a Counter can also be initialized from another mapping:

In [4]: print(Counter({'car': 6, 'house': 10}))
Counter({'house': 10, 'car': 6}) 

In [5]: print(Counter(car=6, house=10))
Counter({'house': 10, 'car': 6})

In [6]: print(Counter('sibling'))    # another iterable
Counter({'i': 2, 'l': 1, 'b': 1, 'g': 1, 's': 1, 'n': 1})

As a Counter is a child of the dict class, it has dict's interface:

In [7]: c = Counter(my_list)

In [8]: c.items()
Out[8]: dict_items([('e', 1), ('c', 2), ('b', 2), ('a', 3), ('d', 1)]) 

In [9]: c.keys()
Out[9]: dict_keys(['e', 'c', 'b', 'a', 'd'])

In [10]: c.values()
Out[10]: dict_values([1, 2, 2, 3, 1]

The only difference is that if you try to access a missing item, a Counter will return zero whereas a dict would raise a KeyError:

In [11]: c['t']
Out[11]: 0

Other than those standard dict methods, a Counter has 3 more specific ones.

most_common(n) - returns a list of n most common elements and their counts in a tuple, ordered from the most common to the least. If n is None, then the method will return all of the elements:

In [12]: c = Counter('hallelujah')

In [13]: c.most_common(3)
Out[13]: [('l', 3), ('h', 2), ('a', 2)]

In [14]: c.most_common()
Out[14]: [('l', 3), ('h', 2), ('a', 2), ('j', 1), ('u', 1), ('e', 1)]

elements() - returns an iterator which repeats each element as many times as its count:

In [15]: list(c.elements())
Out[15]: ['l', 'l', 'l', 'j', 'u', 'h', 'h', 'e', 'a', 'a']

subtract(iterable-or-mapping) - Counts of common elements are subtracted from each other

In [16]: d = Counter('hollar')

In [17]: d
Out[17]: Counter({'a': 1, 'h': 1, 'l': 2, 'o': 1, 'r': 1})

In [18]: c
Out[18]: Counter({'a': 2, 'e': 1, 'h': 2, 'j': 1, 'l': 3, 'u': 1})

In [19]: c.subtract(d)

In [20]: c
Out[20]: Counter({'a': 1, 'e': 1, 'h': 1, 'j': 1, 'l': 1, 'o': -1, 'r': -1, 'u': 1})

Also, some mathematical operations can be applied to combine Counter objects:

Adding(+) two Counters together will perform the following on the elements: c[x] + d[x]:

In [21]: c = Counter(a=3, b=1)

In [21]: d = Counter(a=1, b=2)

In [22]: c + d
Out[22]: Counter({'a': 4, 'b': 3})

Subtracting(-) is the same as the subtract() method (keeps only positive counts):

In [23]: c - d
Out[23]: Counter({'a': 2})

Intersaction(&) will keep only the minimum of corresponding counts: min(c[x], d[x]):

In [24]: c & d
Out[24]: Counter({'a': 1, 'b': 1})

Union(|) will keep only the maximum of corresponding counts: max(c[x], d[x]):

In [25]: c | d
Out[25]: Counter({'a': 3, 'b': 2})

And finally, there are shortcuts for adding an empty counter and subtracting from an empty counter:

In [26]: c = Counter(a=2, b=-4)   

In [27]: +c                     # the same as: c + Counter()
Out[27]: Counter({'a': 2})   

In [28]: -c                     # the same as: Counter() - c
Out[28]: Counter({'b': 4})

Now, with all this theoretical knowledge learned, let's try to apply it to solve a real problem. Let's try to tackle this problem in HackerRank, shall we? Before proceeding further, try to solve it yourself and then compare your solution with mine. Better yet, comment your solution here to discuss.

Task:

Raghu is a shoe shop owner. His shop has X number of shoes. 
He has a list containing the size of each shoe he has in his shop. 
There are N number of customers who are willing to pay x amount of money only if they get the shoe of their desired size.

Your task is to compute how much money Raghu earned.

Input Format

The first line contains X, the number of shoes. 
The second line contains the space separated list of all the shoe sizes in the shop.
The third line contains N, the number of customers. 
The next N lines contain the space separated values of the shoe size desired by the customer and x, the price of the shoe.

Output Format

Print the amount of money earned by Raghu.

My solution:

from collections import Counter

X = int(input())
shoes = [int(val) for val in input().split()]
N = int(input())

shoe_collection = Counter(shoes)
total_money = 0

for i in range(N):
    size, money = [int(val) for val in input().split()]

    if shoe_collection.get(size):
        total_money += money
        shoe_collection[size] -= 1

print(total_money)

It is pretty easy to understand but if you have any questions make sure to ask in the comments!

Fight on!

TestCase vs TransactionTestCase in Django

2017-06-11T20:10:00+05:00

Based on my observation, a lot of developers don't seem to understand the difference between TestCase and TransactionTestCase in Django and how to use them. In this post, I will try to put the puzzle pieces together and make things clear.

TestCase class

Here is what the documentation has to say about the TestCase class:

Wraps the tests within two nested atomic() blocks: one for the whole class and one for each test.

Now imagine that you have a method that must be executed inside a transaction or else it raises an error. You could write a test similar to this:

class SomeTestCase(TestCase):
    def test_your_method_raises_error_without_atomic_block(self):
        with self.assertRaises(SomeError):
            your_method()

In this test, your_method() is called without any transaction and the test is asserting that it raises SomeError because of that.

However, this test will unexpectedly fail! The reason is, you guessed it, TestCase wraps the tests with atomic() blocks ALL THE TIME. Thus, your_method() will not raise an error, which is why this test will fail.

TransactionTestCase to the rescue

This is where TransactionTestCase should be used. It does not wrap the tests with atomic() block and thus you can test your special methods that require a transaction without any problem. The above test will pass with TransactionTestCase now:

class SomeTestCase(TransactionTestCase):
    def test_your_method_raises_error_without_atomic_block(self):
        with self.assertRaises(SomeError):
            your_method()

Real Life example

Let's see a real example now. A queryset method called select_for_update() is one of those methods that require to be inside a transaction. If you call it without any transaction, it raises an error.

Let's say you have a model called Item and you are calling select_for_update():

Item.objects.select_for_update()

It will immediately raise the following error:

TransactionManagementError: select_for_update cannot be used outside of a transaction.

Now, let's try to write tests for it with both TestCase and TransactionTestCase:

class ItemTestCase(TestCase):
    def setUp(self):
        self.item = Item.objects.create(name='hat')

    def test_select_for_update_raises_an_error_without_transaction(self):
        with self.assertRaises(TransactionManagementError):
            items = Items.objects.select_for_update().filter()
            print(items)  # needed to actually execute the query because they are lazy

Try to run the test and you will get the following:

AssertionError: TransactionManagementError not raised

The reason? TestCase wraps the tests with atomic() blocks ALL THE TIME. Good. Glad you remember this.

Now, let's make this test pass with TransactionTestCase:

class ItemTestCase(TransactionTestCase):
    def setUp(self):
        self.item = Item.objects.create(name='hat')

    def test_select_for_update_raises_an_error_without_transaction(self):
        with self.assertRaises(TransactionManagementError):
            items = Items.objects.select_for_update().filter()
            print(items)  # needed to actually execute the query because they are lazy

and voila! The test passes! Great!

I hope it will clear things out for some people. Let me know in the comments if something is still not clear.

Fight on!

Set up Continuous Integration and Continuous Deployment for a Django app with Jenkins

2017-04-02T20:10:00+05:00

In this post, we are going to do something really cool. We are going to set up a CI and CD server for a django app with the help of Jenkins. But first, let's see why we need such a server.

In the previous post, we deployed our application to Digital Ocean and now it is up and running:

Donald Trump and one of his quotes

Now, let's try to update it.

By the way, to follow along this tutorial you should fork this repo if you didn't do so in the previous post. The reason is that we will play around with the app to see our CI and CD server in play later on.

OK, now let's go ahead and change something in the template file djtrump/templates/djtrump/index.html. Find the input element whose placeholder says Your name and change it to Type your name:

# Change this
<input type="text" placeholder="Your name" name="name" value="{{ name }}">

# to this
<input type="text" placeholder="Type your name" name="name" value="{{ name }}">

Commit the change and push it to the remote repo:

git commit -am 'Change input placeholder'
git push origin master

Now, ssh into your server:

ssh root@104.236.57.112   // change this to the IP address of your own server

Go to the project folder and pull the new code from the repo:

cd djtrump
git pull origin master

Make sure origin refers to your fork of the app. Now, open the browser and type the server IP and you will see that the placeholder changed. Now, this is a very simple scenario. When we make changes to models or requirements, we will also have to run migrations, install dependencies and restart the nginx. Besides, we are not considering tests here. In the real world, you should run tests in the prodcution server as well and make sure everything is working.

Although, this is simple, it is very clumsy and error prone to do it manually. Wouldn't be great if you as a developer just pushed your code into the repository, leave the rest to be done by itself and be notified when everything is done?

This is exactly what we will do now. At the end, we will make some changes to the code and push it to the remote repo. Then, our jenkins server will pull the code, run the tests and if they pass successfully deploy the changes to the app server. After all this, it will also notify us through Slack.

Cool right?!

Enough talk, let's fight!

Create a new server and install Jenkins

Go through exact same steps as in the previous post to create a droplet with the only difference of name. Name it djtrump-jenkins. Wait a few seconds and when your server is ready, ssh into it:

ssh root@192.241.166.116  // replace this IP address with yours

Install Jenkins:

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install jenkins

Now, if you go to JENKINS_IP_ADDRESS:8080 on your browser, you should see Jenkins up and running:

Jenkins setup page

Open to /var/lib/jenkins/secrets/initialAdminPassword, copy your password and paste it on the form. Click on Continue.

After a moment, Customize Jenkins page opens. There, click on Select plugins to install.

On this page, search for Github and Pipeline plugins and check them. Click install:

Jenkins plugins page

After everything installs, on the next page, fill in the details and create a user. Click on Start using Jenkins.

Set up a Jenkins Job

Click on New Item or create new jobs to get started.

On the new job page, name the item DJTrump and choose Multibranch pipeline:

Jenkins new job

On the configuration page, click on Add source and choose Git. Fill in the details of your repository in Github. Then, check Periodically if not otherwise run and leave the interval of 1 minute:

Jenkins configure job

Click Save. You will see that Jenkins will immediately start to build the project. As we chose Multibranch mode, it will start to build for all branches. However, you will see that both branches will fail:

Jenkins failed

To see what is going on, click on master and on the next page, click on #1 in the Build History list. Then, on the next page, click on Console Output in the left sidebar. You will see the logs which show that Jenkins cloned the project and then encountered an error that says: No such DSL method 'slackSend' found among steps. So, it tried to notify us through slack that a Jenkins build started and it got this error:

Jenkins logs

But wait! How did Jenkins know what to do in the first place? How did it know that it should send a slack message saying that a build started?

If you go to the root folder of the project you will see a file named Jenkinsfile:

#!groovy

node {

    try {
        stage 'Checkout'
            checkout scm

            sh 'git log HEAD^..HEAD --pretty="%h %an - %s" > GIT_CHANGES'
            def lastChanges = readFile('GIT_CHANGES')
            slackSend color: "warning", message: "Started `${env.JOB_NAME}#${env.BUILD_NUMBER}`\n\n_The changes:_\n${lastChanges}"

        stage 'Test'
            sh 'virtualenv env -p python3.5'
            sh '. env/bin/activate'
            sh 'env/bin/pip install -r requirements.txt'
            sh 'env/bin/python3.5 manage.py test --testrunner=djtrump.tests.test_runners.NoDbTestRunner'

        stage 'Deploy'
            sh './deployment/deploy_prod.sh'

        stage 'Publish results'
            slackSend color: "good", message: "Build successful: `${env.JOB_NAME}#${env.BUILD_NUMBER}` <${env.BUILD_URL}|Open in Jenkins>"
    }

    catch (err) {
        slackSend color: "danger", message: "Build failed :face_with_head_bandage: \n`${env.JOB_NAME}#${env.BUILD_NUMBER}` <${env.BUILD_URL}|Open in Jenkins>"

        throw err
    }

}

This is where all the steps that Jenkins should follow are described. Let's break it down.

The first element you will notice is node. Node is a very crucial player which allocates an executor and workspace for the pipeline. Think of it as a worker that does the job. Inside node, there is one try/catch block. It is very simple. It tries to do something and if it fails it notifies us in Slack and throws error. Inside the try block, there are four stages. First, it pulls the code from the source control and notifies us in Slack that a build has started and what has changed in the code. We will see the end result later. Then, it runs tests which are located in the djtrump/tests folder. They are just a couple of simple tests which are intended for demonstration purposes only. Then, it runs the deployment script deployment/deploy_prod.sh. And finally, it will notify us that the build has been successful in Slack.

Now, let's see what is inside that deployment script:

#!/bin/sh

ssh root@104.236.57.112 <<EOF
  cd djtrump
  git pull
  source /opt/envs/djtrump/bin/activate
  pip install -r requirements.txt
  ./manage.py migrate
  sudo supervisorctl restart djtrump
  exit
EOF

It is also very simple. We are sshing into our server, going to the project folder, pulling the code from the repository, activating the virtual environment, installing the requirements, running migrations and restarting the djtrump project with supervisor. Just like what we did when we manually updated our app in the server.

Now, let's get back to what the Jenkins error message was saying. It said that it could not find slackSend method. The reason is that we have not installed the Slack plugin. This is how Jenkins works, with the help of plugins. You want to work with Github, you install Github plugin. You want to work with Slack, you install the Slack plugin and etc. This is how it is done: Go to the main page and click on Manage Jenkins. There, choose Manage Plugins. Click on the Available tab and search for the Slack Notification Plugin. Check it and click on Install without restart. After it gets installed, click on Restart Jenkins when installation is complete and no jobs are running.

After it restarts, log in.

We also need to install Jenkins plugin on the Slack side. Assuming you already have a slack team, go to https://my.slack.com/services/new/jenkins-ci. There, choose #general channel and click on Add Jenkins CI integration:

Jenkins Slack integration

On the bottom of the next page, you will find some settings such as your Jenkins integration token. Save settings. Now, open another browser window. Go to your Jenkins server, navigate to Manage Jenkins -> Configure System and find Global Slack Notifier Settings there. Fill in team subdomain, integration token and channel:

Jenkins Slack config

Save!

If you did everything correctly, slack messaging will work now. However, there are a couple more things we should do to avoid more errors. First of all, we should install virtualenv for python:

ssh root@104.236.57.112  // replace this with the IP of your own Jenkins server
apt install python3-pip
pip3 install virtualenv

Now, we should install postgresql and create a database so that it will not complain while running tests. If you go to djtrump/settings/base.py, you will see that our database configuration looks like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.environ.get('DATABASE_NAME', 'djtrump'),
        'USER': 'djtrumpuser',
        'PASSWORD': 'password',
        'PORT': '5432',
    }
}

So, we should create a database with name djtrump whose owner is djtrumpuser with password password.

First, install postgresql:

apt-get install postgresql postgresql-contrib

Log into the postgres interactive shell:

sudo -u postgres psql

Create the database, the user and make the user owner of that database:

postgres=# CREATE DATABASE djtrump;
postgres=# CREATE USER djtrumpuser WITH PASSWORD 'password';
postgres=# GRANT ALL PRIVILEGES ON DATABASE djtrump TO djtrumpuser;
postgres=# \q

Now open /etc/postgresql/9.5/main/pg_hba.conf, find the line that says the following:

# "local" is for Unix domain socket connections only
local   all             all                        peer

and change peer to md5 because we will be using username/password combination to connect to postgresql:

# "local" is for Unix domain socket connections only
local   all             all                        md5

Now, restart postgresql:

sudo service postgresql restart

And there is one last thing that we should do. As we saw in the deployment script, we are sshing from the Jenkins server to the DJTrump app server. However, it is not allowed yet. To allow this, we need add the public key of the Jenkins server to the authorized keys of the app server. To do that, log in as jenkins user coz this is the user that will be running the jobs:

sudo su - jenkins

Type this and do not enter passphrase:

ssh-keygen -t rsa

Open ~/.ssh/id_rsa.pub and copy the content. Now, in another terminal tab ssh into the app server:

ssh root@104.236.57.112   // replace the IP

Open ~/.ssh/authorized_keys and paste the public key you copied here. Save!

Now, check if you can ssh into the app server from the Jenkins server as the user jenkins. From the previous tab, type this:

ssh root@104.236.57.112   // replace the IP

Type yes and you should see that you are connected without any password. Congrats! We should be all set.

Time to check!

Navigate to Jenkins root -> DJTrump -> master and once there click on Build now to start a build manually:

And you will see that it will start a build and go through all 4 stages we describe at the beginning:

Jenkins successful build

If you go to your Slack, you should see some messages in #general channel:

Slack notification

Awesome, right?! But that is not so good if we have to make build manually, right?

Fortunately, that's also taken care of. Go to your djtrump and make some kind of change. For example, change the placeholder again to read NNAAAMMMEEE like we did at the beginning of this tutorial. Then, stage and commit the changes:

git commit -am 'Change placeholder to read NNAAAMMMEEE'

In a minute, in your jenkins server you will see that the next build has started and ended successfully:

Jenkins next build

And if you visit your app, you will see that changes have taken place:

New changes

Hooray! Isn't awesome? You just pushed your code and everything else happened by "itself". Ther is even more. Check your slack. You have been notified about the start, what has changed and the end of the build. I love this kind of automation.

Now, go ahead and play around with it. Try to make one of the tests fail and you will see that the failing code will not be deployed! It will stop in the Test stage:

Test stage failed

In Slack, you will see this fun message:

Slack message about the failed Jenkins build

If you click on Open in Jenkins and you will taken to that exact failed build.

Once again, I love this kind of automation! And you should too!

Huh, that was quite a ride! We have learned a lot!

If you encountered a problem along the way, feel free to leave a commend or email me. Will be glad to help!

Fight on!

Customize django admin templates

2015-10-18T21:14:00+05:00

Quite often, we need to customize the django admin template: to add additional functionality or just change its look and feel.

Recently, I had the task of adding an ajax request on admin page load(specifically, change_form.html) and adding two buttons, one of which would bring a modal up and the other would delete the selected items which, in their turn, were brought by that ajax request.

Here is what I did and went through:

Template

First, I created a change_form.html file in /templates/admin/ folder. As I wanted this file to extend, not replace, the default change_form.html, I wrote this at the beginning of the file:

{% extends 'admin/change_form.html' %}

This, naturally, led to maximum recursin depth exceeded error as it was trying to extend itself.

Then, I read somewhere that I could put this file in /templates/admin/app_name folder, so that it will change the templates of only this app. Voila! Recursion problem solved.

Then, I added all those buttons and modals I wanted to(more on this later). Everything was working fine until I found out that the buttons I added appear on the admin pages of all the models in the app. But I wanted them to appear only on the admin pages of, let's say, product model.

Turned out, the same works here. I just needed to put the file inside /templates/admin/app_name/model_name. In my case, it the file was /templates/admin/enterprise/product.

Styles, Scripts and Buttons

Where do I put them? Taking a look at admin/base.html did the trick. It has special blocks for extra styles and scripts. So I put my css and javascript files in the following way:

{% block extrastyle %}
    {{ block.super }}

    <link rel="stylesheet" href="{% static 'css/enterprise/main.css'%}">
{% endblock %}

{% block extrahead %}
    {{ block.super }}

    <script type="text/javascript" src="{% static 'js/enterprise/modal.js'%}"></script>
{% endblock %}

As for buttons, I thought a good place would be above the default buttons. So, I thought out the block of those buttons and put mine right above them:

{% block submit_buttons_bottom %}
    <div class="submit-row">
        <button class="btn btn-danger">Custom Button</button>
    </div>

    {{ block.super }}
{% endblock %}

As I was using django-admin-bootstrapped, submit-row gave the div nice and natural look.

Additional

In the ajax request, I had to send the id of the product being changed. So I thought I could get it with {{ product.id }} but I was wrong. Then, I learned that I could get it like so:

{{ original.id }}

So, the model instance being dealt with can be accessed with the word original. How original, isn't it?

Wrap up

Either I am bad at reading the documentation or it could be improved a little further. Probably first option.

I hope it will help somebody save some time in the future.

Fight on!

Introduction to Python Social Auth

2015-09-12T21:14:00+05:00

Python Social Auth aims to be an easy to setup social authentication and authorization mechanism for Python projects supporting protocols like OAuth (1 and 2), OpenId and others.

Written by omab, this library helps a great deal in integrating social authentication to your web apps. Why am I writing this when there is a whole documentation on the subject? This post is by no means intended to replace the documentation. It is intended to serve as an introduction to the library itself and concepts used in it, such as pipeline, partial pipeline, extending and etc. understanding of which would have saved me a lot of time when I was learning the library.

I will not talk about small things like installation and configuration but rather try to give you a bigger picture on PSA.

Pipeline

PSA uses a mechanism called Pipeline to do the autentication. Pipeline is like a stack of functions. These functions get executed one by one and return some result to the next function. The default pipeline looks like this:

SOCIAL_AUTH_PIPELINE = (
    'social.pipeline.social_auth.social_details',
    'social.pipeline.social_auth.social_uid',
    'social.pipeline.social_auth.auth_allowed',
    'social.pipeline.social_auth.social_user',
    'social.pipeline.user.get_username',
    # 'social.pipeline.mail.mail_validation',
    # 'social.pipeline.social_auth.associate_by_email',
    'social.pipeline.user.create_user',
    'social.pipeline.social_auth.associate_user',
    'social.pipeline.social_auth.load_extra_data',
    'social.pipeline.user.user_details'
)

This is what happens when a user clicks a login button: social_details function gets executed first. It gets the information it can from, let's say, Facebook and returns it to the method social_uid in a simple format. social_uid method does the same thing: does something with the information it got from social_details and returns the result to auth_allowed. So on and so forth until the end of the pipeline when user gets returned to the url you specified. This is the authentication pipeline. There is also an disconnection pipeline, i.e. pipeline for when a user logs out. The same principles apply to that too. More info on pipeline can be found here.

Now, you can do whatever you like with this set of functions for the pipeline and customize it however you like. You can remove any of the methods, for example to create a pipeline that won't create users, just accepts already registered ones:

SOCIAL_AUTH_PIPELINE = (
    'social.pipeline.social_auth.social_details',
    'social.pipeline.social_auth.social_uid',
    'social.pipeline.social_auth.auth_allowed',
    'social.pipeline.social_auth.social_user',
    'social.pipeline.social_auth.associate_user',
    'social.pipeline.social_auth.load_extra_data',
    'social.pipeline.user.user_details'
)

Customize any of the methods:

SOCIAL_AUTH_PIPELINE = (
    'path.to.custom.social_details',    # custom method
    'social.pipeline.social_auth.social_uid',
    'social.pipeline.social_auth.auth_allowed',
    'social.pipeline.social_auth.social_user',
    'social.pipeline.social_auth.associate_user',
    'social.pipeline.social_auth.load_extra_data',
    'social.pipeline.user.user_details'
)

or just create a function and add it to the pipeline. Good example of extending a pipeline can be found here.

Partial Pipeline

It is also possible to cut the pipeline to ask the user for more information and resume the proccess later. For example, to ensure that the user provides his email, you can write the following partial pipeline:

@partial
def require_email(strategy, details, user=None, is_new=False, *args, **kwargs):
    if user and user.email:
        return
    elif is_new and not details.get('email'):
        email = strategy.request_data().get('email')
        if email:
            details['email'] = email
        else:
            return redirect('require_email')

This method first checks whether email exists. If so, continues the pipeline. Otherwise, it will redirect to require_email view, which looks like this:

def require_email(request):
    return render(request, 'path/to/template.html', args)

So, it will render the template with a form. Attention: this form must be submitted to /complete/<backend>/ to continue the pipeline:

<form action="/complete/instagram" method="post">

After a user submits the form, the pipeline comes to this part of the partial pipeline code:

email = strategy.request_data().get('email')
if email:
    details['email'] = email

and if user entered his email, the pipeline continues and you will have access to his email through details['email'].

Important use case

Provide login/registration with PSA for two types of users.

So we have two types of users, SimpleUser and Shop(SimpleUser) which extends from SimpleUser. To provide different registration proccess for them we can do the following:

Provide two different links and mark one of them with a get parameter(user_type):

<a href="{% url 'social:begin' 'facebook' %}">Login as SimpleUser</a>
<a href="{% url 'social:begin' 'facebook' %}?user_type=shop">Login as Shop</a>

In order to access this get parameter, we will have to do this in our settings file:
```
FIELDS_STORED_IN_SESSION = ['user_type']
```
This will ensure that the value of this paramter is saved in a session.

Then in create_user method of the pipeline, create different users depending on this parameter:

def create_user(strategy, backend, details, user=None, *args, **kwargs):
    user_type = strategy.session_get('type')
    if user_type == 'shop':
          # create a shop
    else:
          # create a simple user

and don't forget to include it in the pipeline settings.

We are done! Hooorrayy =)

Okay fellas, I hope you now have at least a little idea about how things work in PSA and can easily read the documentation. These were only the most important and basic parts. Checkout the documentation for the details.

Fight on!