Let’s get down to business:

1. Download the compressed pre-ajax Django Project from the repo
2. Activate a virtualenv
3. Install the requirements
4. Sync the database
5. Fire up the server

Once logged in, test out the form. What we have here is a simple communication app with just create rights. It looks nice, but there’s one annoying issue: The page refresh. How do we get rid of it? Or, how do we update just a portion of a webpage without having to refresh the entire page? Enter AJAX. AJAX is a client-side technology used for making asynchronous requests to the server-side – i.e., requesting or submitting data – where the subsequent responses do not cause an entire page refresh.

This tutorial assumes you have working knowledge of Django as well as some experience with JavaScript/jQuery. You should also be familiar with the basic HTTP methods, particularly GET and POST. Need to get up to speed? Get Real Python.

Use Protection

Regardless of whether you’re using AJAX or not, forms are at risk for Cross Site Request Forgeries (CSRF) attacks.

Read more about CSRF attacks on the Coding Horror blog. They’ve got a great article.

To prevent such attacks, you must add the {% csrf_token %} template tag to the form, which adds a hidden input field containing a token that gets sent with each POST request.

If you look at the talk/index.html template, you can see that we have already included this token. However, when it comes to AJAX requests, we need to add a bit more code, because we cannot pass that token using a JavaScript object since the scripts are static.

To get around this, we need to create a custom header that includes the token to watch our back. Simply grab the code here and add it to the end of the main.js file. Yes, it’s a lot of code. We could go through it line-by-line, but that’s not the point of this post. Just trust us that it works.

Moving on…

Handling Events

Before we touch the AJAX code, we need to add an event handler to our JavaScript file using jQuery.

Keep in mind that jQuery is JavaScript. It’s simply a JavaScript library used to reduce the amount of code you need to write. This is a common area of confusion so just be mindful of this as you go through the remainder of this tutorial.

Which event(s) do we need to “handle”? Since we’re just working with creating a post at this point, we just need to add one handler to main.js:

1
2
3
4
5
6

// Submit post on submit
$('#post-form').on('submit', function(event){
    event.preventDefault();
    console.log("form submitted!")  // sanity check
    create_post();
});

Here, when a user submits the form this function fires, which-

1. Prevents the default browser behavior for a form submission,
2. Logs “form submitted!” to the console, and
3. Calls a function called create_post() where the AJAX code will live.

Make sure to add an id of post-form to the form on the index.html file:

1

<form action="/create_post/" method="POST" id="post-form">

And add a link to the JavaScript file to the bottom of the template:

1

<script src="static/scripts/main.js"></script>

Test this out. Fire up the server, then open your JavaScript console. You should see the following when you submit the form:

1
2

form submitted!
Uncaught ReferenceError: create_post is not defined

This is exactly what we should see: The form submission is handled correctly, since “form submitted!” is displayed and the create_post function is called. Now we just need to add that function.

Adding AJAX

Let’s develop one last iteration before we add the actual AJAX code.

Update main.js:

Add the create_post function:

1
2
3
4
5

// AJAX for posting
function create_post() {
    console.log("create post is working!") // sanity check
    console.log($('#post-text').val())
};

Again, we ran a sanity check to ensure the function is called correctly, then we grab the input value of the form. For this to work correctly we need to add an id to the form field:

Update forms.py:

1
2
3
4
5
6
7
8
9
10

class PostForm(forms.ModelForm):
    class Meta:
        model = Post
        # exclude = ['author', 'updated', 'created', ]
        fields = ['text']
        widgets = {
            'text': forms.TextInput(
                attrs={'id': 'post-text', 'required': True, 'placeholder': 'Say something...'}
            ),
        }

Notice how we also added a placeholder to the field and made it required along with the id. We could add some error handlers to the form template or simply let HTML5 handle it. Let’s use the latter.

Test again. Submit the form with the word “test”. You should see the following in your console:

1
2
3

form submitted!
create post is working!
test

Sweet. So, we’ve confirmed that we’re calling the create_post() function correctly as well as grabbing the value of the form input. Now let’s wire in some AJAX to submit the POST request.

Update main.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// AJAX for posting
function create_post() {
    console.log("create post is working!") // sanity check
    $.ajax({
        url : "create_post/", // the endpoint
        type : "POST", // http method
        data : { the_post : $('#post-text').val() }, // data sent with the post request

        // handle a successful response
        success : function(json) {
            $('#post-text').val(''); // remove the value from the input
            console.log(json); // log the returned json to the console
            console.log("success"); // another sanity check
        },

        // handle a non-successful response
        error : function(xhr,errmsg,err) {
            $('#results').html("<div class='alert-box alert radius' data-alert>Oops! We have encountered an error: "+errmsg+
                " <a href='#' class='close'>&times;</a></div>"); // add the error to the dom
            console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console
        }
    });
};

What’s happening? Well, we submit form data to the create_post/ endpoint, then wait for one of two responses – either a success or a failure…. Follow the code comments for a more detailed explanation.

Update the views

Now let’s update our views to handle the POST request correctly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def create_post(request):
    if request.method == 'POST':
        post_text = request.POST.get('the_post')
        response_data = {}

        post = Post(text=post_text, author=request.user)
        post.save()

        response_data['result'] = 'Create post successful!'
        response_data['postpk'] = post.pk
        response_data['text'] = post.text
        response_data['created'] = post.created.strftime('%B %d, %Y %I:%M %p')
        response_data['author'] = post.author.username

        return HttpResponse(
            json.dumps(response_data),
            content_type="application/json"
        )
    else:
        return HttpResponse(
            json.dumps({"nothing to see": "this isn't happening"}),
            content_type="application/json"
        )

Here we grab the post text along with the author and update the database. Then we create a response dict, serialize it into JSON, and then send it as the response – which gets logged to the console in the success handler: console.log(json), as you saw in the create_post() function in the JavaScript file above.

Test this again.

You should see the object in the console:

1
2
3
4

form submitted!
create post is working!
Object {text: "hey!", author: "michael", postpk: 15, result: "Create post successful!", created: "August 22, 2014 10:55 PM"}
success

How about we add the JSON to the DOM!

Updating the DOM

Update the template

Simply add an id of “talk” to the <ul>:

1

<ul id="talk">

Then update the form so that errors will be added:

1
2
3
4
5
6
7
8

<form method="POST" id="post-form">
    {% csrf_token %}
    <div class="fieldWrapper" id="the_post">
        {{ form.text }}
    </div>
    <div id="results"></div> <!-- errors go here -->
    <input type="submit" value="Post" class="tiny button">
</form>

Update main.js

Now we can add the JSON to the DOM where that new “talk” id is:

1
2
3
4
5
6

success : function(json) {
    $('#post-text').val(''); // remove the value from the input
    console.log(json); // log the returned json to the console
    $("#talk").prepend("<li><strong>"+json.text+"</strong> - <em> "+json.author+"</em> - <span> "+json.created+"</span></li>");
    console.log("success"); // another sanity check
},

Ready to see this in action? Test it out!

If you’d like to see what an error looks like, then comment out all the CSRF Javascript in main.js and then try to submit the form.

Rinse, Repeat

Your turn. We need to handle some more events. With your new found knowledge of jQuery and AJAX, you get to put these into place. I added code to the final app – which you can download here – that includes a delete link. You just need to add an event to handle the click, which then calls a function that uses AJAX to send a POST request to the back-end to delete the post from the database. Follow the same workflow as I did in this tutorial. We’ll post the answer to this next time.

If you get stuck, and can’t debug the errors, follow this workflow –

1. Use the “Google-it-first” algorithm
2. Struggle. Spin your wheels. Set the code aside. Run around the block. Then come back to it.
3. Still stuck? Comment below, stating first the problem and then detailing the steps you’ve taken to solve the problem thus far

Be sure to try troubleshooting on your own before asking for help. Spinning your wheels, hacking away at a solution will benefit you in the long run. It’s the process that matters, not so much the solution. It’s part of what separates poor developers from great developers. Good luck.

Conclusion

How does your app look? Ready for more?

1. AJAX is so yesterday. We can do a lot more with much less code using AngularJS.
2. In most cases, it’s a standard to couple the client-side JavaScript, whether it’s AJAX or Angular or some other framework, with a server-side RESTful API.
3. Where’s the tests?

What would you like to see next? Comment below. Cheers!

Happy coding!

Link to repo.

In this section, we’re going scrape the contents of a webpage that the user enters and then process the text to display the number of times a word occurs.

Remember, here’s what we’re building: A Flask app with a simple text box. When you enter text into the box, the app processes then displays a count of how many times each word appears on the page.

1. Part One: Setup a local development environment and then deploy both a staging environment and a production environment on Heroku.
2. Part Two: Setup a PostgreSQL database along with SQLAlchemy and Alembic to handle migrations.
3. Part Three: Add in the back-end logic to scrape and then process the counting of words from a webpage using the requests, BeautifulSoup, and Natural Language Toolkit (NLTK) libraries. (current)
4. Part Four: Implement a request queue backed by Redis to handle the actual text processing.
5. Part Five: Setup Angular on the front-end to continuously poll the back-end to see if the request is done.
6. Part Six: Add the D3 library into the mix to graph a frequency distribution and histogram.

Need the code? Grab it from the repo.

Install Requirements

Tools we’ll use in this part:

• requests – http://docs.python-requests.org/en/latest/
• BeautifulSoup – http://www.crummy.com/software/BeautifulSoup/
• NLTK – http://www.nltk.org/

1
2
3

$ workon wordcounts
$ pip install requests nltk beautifulsoup4
$ pip freeze > requirements.txt

Refactor the Index Route

To get started let’s get rid of the “hello world” part of the index route in our app.py file and set up the route to render a form to accept URLs. First, add a templates folder to hold our templates and add an index.html file to it.

1
2
3

$ mkdir templates
$ cd templates
$ touch index.html

Set up a very basic HTML page:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<!DOCTYPE html>
<html>
  <head>
    <title>Wordcount</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen">
    <style>
      .container {
        max-width: 1000px;
      }
    </style>
  </head>
  <body>
    <div class="container">
      <h1>Wordcount 3000</h1>
      <form role="form" method='POST' action='/'>
        <div class="form-group">
          <input type="text" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">
        </div>
        <button type="submit" class="btn btn-default">Submit</button>
      </form>
      <br>
      {% for error in errors %}
        <h4>{{ error }}</h4>
      {% endfor %}
    </div>
    <script src="http://code.jquery.com/jquery-1.11.0.min.js"></script>
    <script src="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
  </body>
</html>

We use Bootstrap to add a bit of style so our page isn’t completely hideous. Then add a form with a text box for users to enter a URL. Additionally, we utilize a Jinja for loop to iterate through a list of errors, displaying each one.

Update the app.py file to serve the template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from flask import Flask, render_template
from flask.ext.sqlalchemy import SQLAlchemy
import os

app = Flask(__name__)
app.config.from_object(os.environ['APP_SETTINGS'])
db = SQLAlchemy(app)


@app.route('/', methods=['GET', 'POST'])
def index():
    return render_template('index.html')

if __name__ == '__main__':
    app.run()

Why both HTTP methods? Well, we will eventually use that same route for both GET and POST requests – to serve the index.html page and handle form submissions, respectively.

Fire up the app to test it out:

1

$ python manage.py runserver

Navigate to http://localhost:5000/ and you should see the form staring back at you.

Requests

Now let’s use the requests library to grab the HTML page from the submitted URL.

Change your index route to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

@app.route('/', methods=['GET', 'POST'])
def index():
    errors = []
    results = {}
    if request.method == "POST":
        # get url that the user has entered
        try:
            url = request.form['url']
            print url[:7]
            if 'http://' not in url[:7]:
                url = 'http://' + url
            r = requests.get(url)
            print r.text
        except:
            errors.append(
                "Unable to get URL. Please make sure it's valid and try again."
            )
    return render_template('index.html', errors=errors, results=results)

Make sure to update the imports as well:

1
2
3
4

from flask import Flask, render_template, request
from flask.ext.sqlalchemy import SQLAlchemy
import os
import requests

1. Here, we import the requests library as well as the request object from Flask. The former is used to scrape the specific user-provided URL, while the latter is used to handle GET and POST requests in Flask.
2. Next we add variables to capture both errors and results, which are passed into the template.

3. Within the view itself, we check if the request is a POST-

   • If POST: Grab the value from the form and assign it to the url variable. Then we do some clean-up on the URL to add ‘http://’ if it’s not part of the submitted URL. Finally, we add an exception to handle any errors and append a generic error message to the errors list. We then render the template, including the errors list and results dictionary.
   • If GET: We simply render the template.

Let’s test this out:

1

$ python manage.py runserver

You should be able to type in a web page and in the console you’ll see the text of that webpage returned (as long as it’s a valid page, of course).

Text Processing

With the HTML in hand, let’s now count the frequency of the words that are on the page and display them to the end user. Update your code to the following and we’ll walk through what’s happening:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

from flask import Flask, render_template, request
from flask.ext.sqlalchemy import SQLAlchemy
from stop_words import stops
from collections import Counter
from bs4 import BeautifulSoup
import operator
import os
import requests
import re
import nltk

#######################
#### configuration ####
#######################

app = Flask(__name__)
app.config.from_object(os.environ['APP_SETTINGS'])
db = SQLAlchemy(app)

from models import Result


################
#### routes ####
################


@app.route('/', methods=['GET', 'POST'])
def index():
    errors = []
    results = {}
    if request.method == "POST":
        # get url that the person has entered
        try:
            url = request.form['url']
            if 'http://' not in url[:7]:
                url = 'http://' + url
            r = requests.get(url)
        except:
            errors.append(
                "Unable to get URL. Please make sure it's valid and try again."
            )
            return render_template('index.html', errors=errors)
        if r:

            # text processing
            raw = BeautifulSoup(r.text).get_text()
            nltk.data.path.append('./nltk_data/')  # set the path
            tokens = nltk.word_tokenize(raw)
            text = nltk.Text(tokens)

            # remove punctuation, count raw words
            nonPunct = re.compile('.*[A-Za-z].*')
            raw_words = [w for w in text if nonPunct.match(w)]
            raw_word_count = Counter(raw_words)

            # stop words
            no_stop_words = [w for w in raw_words if w.lower() not in stops]
            no_stop_words_count = Counter(no_stop_words)

            # save the results
            results = sorted(
                no_stop_words_count.items(),
                key=operator.itemgetter(1),
                reverse=True
            )
            try:
                result = Result(
                    url=url,
                    result_all=raw_word_count,
                    result_no_stop_words=no_stop_words_count
                )
                db.session.add(result)
                db.session.commit()
            except:
                errors.append("Unable to add item to database.")
    return render_template('index.html', errors=errors, results=results)

if __name__ == '__main__':
    app.run()

Create a new file called stop_words.py and add the following list:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

stops = [
    'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you',  'your',
    'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she',
    'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their',
    'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that',
    'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
    'have', 'has', 'had',  'having', 'do', 'does', 'did', 'doing', 'a', 'an',
    'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of',
    'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through',
    'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',
    'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then',
    'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any',
    'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor',
    'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can',
    'will', 'just', 'don', 'should', 'now', 'id', 'var', 'function', 'js', 'd',
    'script', '\'script', 'fjs', 'document'
]

What’s happening?

Text Processing

1. In our index route we use beautifulsoup to clean the text, by removing the HTML tags, that we get back from the URL as well as nltk to-

   • Tokenize the raw text (break up the text into individual words), and
   • Turn the tokens into an nltk text object.

2. In order for nltk to work right, you need to download the correct tokenizers. First create a new directory – mkdir nltk_data – then run – python -m nltk.downloader.

   When the installation window appears, update the ‘Download Directory’ to whatever_the_absolute_path_to_your_app_is/nltk_data/.

   Then click the ‘Models’ tab and select ‘punkt’ from under the ‘Identifier’ column. Click ‘Download’. Check the official documentation for more information if you need help.

Remove Punctuation, Count Raw Words

1. Since we don’t want punctuation counted in the final results, we create a regular expression that matches anything not in the standard alphabet.
2. Then, using a list comprehension, we create a list of words without punctuation or numbers.
3. Finally, we count the number of times each word appears in the list using Counter – which is a really useful tool for tallying the number of times something appears in a list.

Stop Words

Our current output contains a lot of words that we likely don’t want to count – i.e., “I”, “me”, “the”, and so forth. These are called stop words.

1. With the stops list, we again use a list comprehension to create a list of words that do not include those stop words.
2. Next, we create a dictionary with the words (as keys) and their associated counts (as values).
3. And finally we use the sorted tool to get a sorted representation of our dictionary. We then can use this to display the words with the highest count at the top of the list, which means that we won’t have to do that sorting in our Jinja template.

For a more robust stop word list, use the NLTK stopwords corpus.

Save the Results

Finally we use a try/except to save the results of our search and counts to the database.

Display Results

Now let’s update index.html in order to display the results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

<!DOCTYPE html>
<html>
  <head>
    <title>Wordcount</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen">
    <style>
      .container {
        max-width: 1000px;
      }
    </style>
  </head>
  <body>
    <div class="container">
      <div class="row">
        <div class="col-sm-5 col-sm-offset-1">
          <h1>Wordcount 3000</h1>
          <br>
          <form role="form" method="POST" action="/">
            <div class="form-group">
              <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">
            </div>
            <button type="submit" class="btn btn-default">Submit</button>
          </form>
          <br>
          {% for error in errors %}
            <h4>{{ error }}</h4>
          {% endfor %}
          <br>
        </div>
        <div class="col-sm-5 col-sm-offset-1">
          {% if results %}
            <h2>Frequencies</h2>
            <br>
            <div id="results">
              <table class="table table-striped" style="max-width: 300px;">
                <thead>
                  <tr>
                    <th>Word</th>
                    <th>Count</th>
                  </tr>
                </thead>
                {% for result in results%}
                  <tr>
                    <td>{{ result[0] }}</td>
                    <td>{{ result[1] }}</td>
                  </tr>
                {% endfor %}
              </table>
            </div>
          {% endif %}
        </div>
      </div>
    </div>
    <br><br>
    <script src="http://code.jquery.com/jquery-1.11.0.min.js"></script>
    <script src="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
  </body>
</html>

Here, we added an if statement to see if our results dictionary has anything in it and then added a for loop to iterate over the results and display them in a table. Run your app and you should be able to enter a URL and get back the count of the words on the page.

1

$ python manage.py runserver

What if we wanted to display the first ten keywords from the dictionary? We can simply limit the dictionary to the first 10 results:

1
2
3
4
5

results = sorted(
    no_stop_words_count.items(),
    key=operator.itemgetter(1),
    reverse=True
)[:10]

Test it out.

Summary

Okay great. Given a URL we can count the words that are on the page. If you use a site without a massive amount of words, like http://realpython.com for instance, the processing should happen fairly quickly. What happens if the site has a lot of words, though? For example, try out http://gutenberg.ca. You’ll notice that this takes longer to process. If you have a number of users all hitting your site at once to get word counts, and some of them are trying to count larger pages, this can become a problem. Or perhaps you decide to change the functionality so that when a user inputs a URL, we recursively scrape the entire web site and calculate word frequencies based on each individual page. Again, with enough traffic, this will significantly slow down the site.

What’s the solution? Instead of counting the words after each user makes a request, we need to use a queue to process this in the backend – which is exactly where will start next time. For now, commit your code and push it up to staging only since this new text processing feature is only half finished.

Before you push to Heroku, I recommend removing all language tokenizers except for English along with the zip file:

1
2
3
4
5
6

.
└── tokenizers
    └── punkt
        ├── PY3
        │   └── english.pickle
        └── english.pickle

This will significantly reduce the size of the commit. Keep in mind though that if you do process a non-English site, it will only process English words.

1

$ git push stage master

Check it out here – http://wordcounts-stage.herokuapp.com/

Test it out on staging. Comment if you have questions. See you next time!

In this article, we’ll look at how to setup Sublime Text for full stack Python development (from the front to back), enhance the basic functionality with custom themes and packages, and use many of the commands, features, and keyword shortcuts that make ST3 so powerful.

This tutorial assumes you’re using a Mac and are comfortable with the terminal. If you’re using Windows or Linux, many of the commands will vary, but you should be able to use Google to find the answers quickly given the info in this tutorial.

Before we start, let’s address what I mean exactly by “full stack”.

In today’s world of HTML5 and mobile development, Javascript is literally everywhere. EVERYWHERE. Python coupled with a framework such as Django or Flask is not enough. To really develop a website from end-to-end, you must be familiar with Javascript (and the various Javascript frameworks), REST APIs, responsive design, and of course HTML and CSS, among others. And let’s face it: Like any craftsman, in order to be the best programmer you can be, your tools must be sharp. Your development environment must be setup for full stack development – which is exactly what we are going to do right now…

Features

Let’s start by looking at a few of the default features of Sublime Text 3…

1. Split Layouts allow you to arrange your files in various split-screens. This is useful for test driven development (Python code on one screen, test scripts on another) or when working on the front end (HTML on one screen, CSS and/or Javascript on another).

   

2. Vintage Mode provides you with vi commands for use within ST3.

3. Chrome-like Tabs make navigating and editing several files much simpler.
4. Automatic loading of the last session re-opens all files and folders you had open when you closed the editor the last time. I leave ST3 open all the time, with various projects open – so if I reset the computer, it opens the files and folders right back up.

5. Code Snippets increase your productivity by giving you the ability to create common pieces of code with a single keyword. There are a number of default snippets. For example, open a new file and type in “lorem” then press tab. You should get a paragraph of lorem ipsum text. Also, if you type “defs” then press tab in a Python file it will setup a generic function.

   You can also create your own snippets: Tools > New Snippet. Refer to the documentation for help, and also check out some of my snippets here.

Customizing Sublime Text 3

After downloading ST3 …

Install the subl command line tool

Like the mate command for TextMate, Sublime Text includes a command line tool called subl that allows you to open one file, or an entire directory of files and folders, from the terminal.

To enable this command, create a symbolic link to the subl binary:

1

$ ln -s "/Applications/Sublime Text 3.app/Contents/SharedSupport/bin/subl" ~/bin/subl

Ensure that the link works by opening Sublime:

1

$ subl

If that didn’t work, you probably need to add /bin to your Path:

1

$ echo "export PATH=~/bin:$PATH" >> ~/.profile

Then repeat step one.

If you are still having trouble, check out this article for help. Also, here are links for help on creating the symbolic links in Windows and Linux.

Now you can open a file or directory using the following commands:

1
2
3
4
5
6
7
8

# open the current directory
$ subl .

# open a directory called tests
$ subl ~/Documents/test

# open a file called text.txt
$ subl test.txt

If there are spaces in the path, you must surround the entire path in double quotes:

1

$ subl "~/Documents/test/my test file.txt"

To view all the commands, open up the help file:

1

$ subl --help

Install Package Control

To begin taking advantage of the various packages for extending Sublime’s functionality, you need to install the package manager called Package Control – which you must install manually. Once installed, you can use Package Control to install/remove/upgrade all other ST3 packages.

1. To install, copy the Python code for Sublime Text 3 found here. Click View > Show Console to open the ST3 console. Paste the code into the console. Press enter. Reboot ST3.

2. You can now install packages by using the keyboard shortcut cmd+shift+P. Start typing install until Package Control: Install Package appears. Press enter and search for available packages.

3. Some other relevant commands are:

   • List Packages shows all your installed packages
   • Remove Packages removes a specific package
   • Upgrade Package upgrades a specific package
   • Upgrade/Overwrite All Packages upgrades all your installed packages

   
   

   Check out the official documentation to view more commands.

   

Create a Custom Settings File

You can fully configure Sublime Text using JSON-based settings files, making it easy to transfer, or synchronize, your customized settings to another system. First, we need to create our customized settings. It’s best to create a base file for all environments as well as language-specific settings files.

To set up a base file click Sublime Text > Preferences > Settings – User. Add an empty JSON object to the file and add your settings like so:

1
2
3
4
5
6
7

{
  // base settings
  "auto_complete": false,
  "sublimelinter": false,
  "tab_size": 2,
  "word_wrap": true
}

1. For language specific settings click Sublime Text > Preferences > Settings – More > Syntax Specific – User. Then save the file using the following format: LANGUAGE.sublime-settings. So, for Python-specific settings, save the file as Python.sublime-settings.

2. You can obviously configure your settings to your liking; however, I highly recommend starting with my base and Python-specific settings – then making changes as you see fit.

3. Optional: You can use Dropbox to sync all your settings. Simply upload your settings files to Dropbox and load them from there to sync the Sublime environments on all your machines.

4. A good reference for settings can be found at the Sublime Text Unofficial Documentation.

Themes

ST3 also gives you the option to change the overall theme to better suit your personality. Design your own. Or, if you’re not artistically inclined, you can download one of the various custom themes designed by the Sublime community through Package Control. Check out ColorSublime to preview themes before installing them.

The ever popular Soda Dark Theme and the minimal Flatland are two of my personal favorites.

After installing a theme, make sure to update your base settings, Sublime Text > Preferences > Settings – User:

1
2
3
4

{
  "theme": "Flatland Dark.sublime-theme",
  "color_scheme": "Packages/Theme - Flatland/Flatland Dark.tmTheme"
}

Packages

Besides the packaged themes, I take advantage of the following packages to speed up my workflow.

SideBarEnhancements

SideBarEnhancements extends the number of menu options in the sidebar, speeding up your overall workflow. Options such as “New file” and “Duplicate” are essential and should be part of ST3 out of the box. The “Delete” option alone makes it worth downloading. This feature simply sends files to the Trash, which may seem trivial but if you delete a file without it, it’s very difficult to recover unless you’re using a version control system.

Download this now!

Anaconda

Anaconda is the ultimate Python package; it adds a number of IDE-like features to ST3 including:

• Autocompletion works by default, but there are a number of configuration options.
• Code linting uses either PyLint or PyFlakes with pep8. I personally use a different linting package, as I will explain shortly, so I disable linting altogether within the user-defined Anaconda settings file, Anaconda.sublime-settings, via the file menu: Sublime > Preferences > Package Settings > Anaconda > Settings – User: {"anaconda_linting": false}
• McCabe code complexity checker runs the McCabe complexity checker tool within a specific file. If you’re unfamiliar with a complexity checker, be sure to visit the link above.
• Goto Definitions finds and displays the definition of any variable, function, or class throughout your entire project.
• Find Usage quickly searches where a variable, function, or class has been used in a specific file.

• Show Documentation: shows the Docstring for functions or classes (if defined, of course).

  

You can view all of the features here, or within the README file in ST3’s Package Settings: Sublime Text > Preferences > Package Settings > Anaconda > README.

SublimeCodeIntel is another popular package, which has many of the same features as Anaconda. I suggest testing them both out.

Djaneiro

Djaneiro supports Django templating and keyword highlighting and provides useful code snippets (tab completions) for Sublime Text. The snippet system is an incredible timesaver. You can create common Django blocks with only a few keystrokes for templates, models, forms, and views. Check out the official documentation to see a list of snippets.

My personal favorites are for templating: var creates {{ }} and tag creates {% %}

requirementstxt

Requirementstxt provides autocompletion and syntax highlight-lighting as well as a nice version management system for your requirements.txt files.

SublimeLinter

SublimeLinter is a framework for ST3 linters. The package itself does not include any actual linters; those must be installed separately via Package Control using the SublimeLinter-[linter_name] naming syntax. You can view official linters here. There are also a number of third party linters, which can be viewed in Package Control. Check out the installation instructions here.

For Python linting, I recommend using SublimeLinter-pyflakes and SublimeLinter-pep8.

I also use SublimeLinter-jshint, SublimeLinter-pyyaml, SublimeLinter-csslint, SublimeLinter-html-tidy, and SublimeLinter-json.

Most of these linters have dependencies associated with them, so please read the installation instructions before installing.

You can customize each linter in the user-defined SublimeLinter.sublime-settings file: Sublime Text > Preferences > Package Settings > SublimeLinter > Settings – User. For example, I ignore the following pep8 errors and warnings:

1
2
3
4
5
6
7
8

"pep8": {
    "@disable": false,
    "args": [],
    "excludes": [],
    "ignore": "E501,C0301,W0142,W0402,R0201,E1101,E1102,C0103,R0901,R0903,R0904,C1001,W0223,W0232,W0201,E1103,R0801,C0111",
    "max-line-length": 100,
    "select": ""
},

GitGutter

GitGutter shows little icons in ST3’s gutter area that indicate whether a line has been inserted, modified, or deleted since the last commit.

If you want support for a number of distributed version control systems (Git, SVN, Bazaar and Mercurial), check out Modific

FTPSync

FTPSync syncs your project with your remote files. Simply open the file to download it (if the remote file is newer than your local file) and upload it to your remote server with every save. Great way to keep your local and remote(s) in sync. You’ll want to make sure to add at least one remote connection by clicking Sublime Text > Preferences > Package Settings > FTPSync > Setup FTPSync.

Sample settings:

1
2
3
4
5
6
7
8
9
10
11

{
  'primary': {
    host: 'ftp.mywebsite.com',
    username: 'johnsmith',
    password: 'secretpassword',
    path: '/www/',

    upload_on_save: true,
    tls: true
  }
}

I personally set the password to null because I don’t want it visible in that file. FTPSync just asks for my password after each save.

AdvancedNewFile

AdvancedNewFile is used to create a new folder or file from within ST3 with key bindings alone:

Simply bring up the AdvancedNewFile input through the appropriate key binding. Then, enter the path, along with the file name into the input field. Upon pressing enter, the file will be created. In addition, if the directories specified do not yet exist, they will also be created. By default, the path to the file being created will be filled shown in the status bar as you enter the path information.

For a more detailed explanation on its usage, check out the documentation on Github. Be sure to read about Tab Completion as well as Predefined Aliases.

I replaced the normal “cmd+n” command to create a new file with AdvancedNewFile by adding the following code to the Key Bindings – User file: Sublime Text > Preferences > Package Settings > AdvancedNewFile > Key Bindings – User:

1
2
3

[
  { "keys": ["cmd+n"], "command": "advanced_new_file_new"}
]

You can also setup a default directory to start with: Sublime Text > Preferences > Package Settings > AdvancedNewFile > Settings – User

1

{"default_initial": "/Users/michaelherman/Documents/repos"}

Now when I create a new file, the /Users/michaelherman/Documents/repos string is automatically inserted first, since 99% of the time I store all my scripts in that directory.

Emmet

Emmet, previously known as Zen Coding, uses simple abbreviations to generate HTML or CSS code snippets.

For example, if you type a bang, !, then press tab in an HTML file ,the HTML5 doctype and a few basic tags are generated:

1
2
3
4
5
6
7
8
9
10

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Document</title>
</head>
<body>

</body>
</html>

Check out the official documentation as well as this handy cheat sheet for more info.

Markdown Preview

Markdown Preview is used for previewing and building markdown files.

To use, open the Package Manager then type Markdown Preview to show the available commands:

• Markdown Preview: Python Markdown: Preview in Browser
• Markdown Preview: Python Markdown: Export HTML in Sublime Text
• Markdown Preview: Python Markdown: Copy to Clipboard
• Markdown Preview: Github Flavored Markdown: Preview in Browser
• Markdown Preview: Github Flavored Markdown: Export HTML in Sublime Text
• Markdown Preview: Github Flavored Markdown: Copy to Clipboard
• Markdown Preview: Open Markdown Cheat sheet

Once converted, the output file is updated on each subsequent save.

Keyboard Shortcuts

1. Goto Anything (“cmd+p”) is used for quickly finding and opening files. Just type in a part of a path and filename within a project and you can easily open that file. This is great for quickly opening files in large Django projects.
2. Goto Line Number (“ctrl+g”) takes you to a specific line number in an active file.
3. Goto Symbol (“cmd+r”) lists all functions and classes within a file to make them easier to find. Simply start typing the one you want.
4. Go to beginning of line (cmd+left-arrow-key) and Go to end of line (cmd+right-arrow-key)
5. Delete current line (ctrl+shift+k)
6. Multi-Edit is by far my favorite shortcut
   1. Select a word, press “cmd+d” to select the next same word, then press “cmd+d” again to select the next same word…
   2. Press “cmd+click” to create a cursor for editing every where you click
7. Block select (“option+left-mouse-click) is used to select a block of text. Perfect for removing blank space when formatting a CSV file.

For more shortcuts, take a look at this article.

Custom Commands

It’s easy to write your own custom commands and key bindings with Python. I currently use …

1. Copy the path of the current file to the clipboard – link
2. Close all tabs except the active one – link

Install these by adding the Python files to your “/Sublime Text 3/Packages/User” directory via the file menu (Sublime > Preferences > Browse Packages), then opening the User directory. To complete the setup, bind them from the Key Bindings – User file (Sublime Text > Preferences > Package Settings > AdvancedNewFile > Key Bindings – User).

1
2
3
4
5
6
7
8
9
10
11
12

[
  // Copy file name
  {
    "keys": ["cmd+shift+c"],
    "command": "copy_path_to_clipboard"
  },
  // Close all other tabs
  {
    "keys": ["cmd+alt+w"],
    "command": "close_tabs"
  }
]

Additional Resources

1. Community-maintained documentation
2. Package Manager documentation
3. Unofficial documentation reference
4. Pimp my Editor – Presentation

Conclusion

I hope this article was helpful to you and that you were able to integrate some of the above packages and custom settings along with your own based on your personal preferences to improve your workflow. If you have any comments or suggestions of your own, please let me know in the comments below. Finally, check out the dotfiles folder in this repo to view all resources that I created.

Cheers!

Edits made by Derrick Kearney. Thank you!

Remember: In Python, a function is a “first-class” citizen, meaning they are on par with any other object (i.e., integers, strings, lists, modules, etc.). You can dynamically create or destroy them, pass them to other functions, return them as values, and so forth.

This tutorial utilizes Python version 3.4.1.

1. Encapsulation

You use inner functions to protect them from anything happening outside of the function, meaning that they are hidden from the global scope.

Here’s a simple example that highlights that concept:

1
2
3
4
5
6
7
8

def outer(num1):
    def inner_increment(num1):  # hidden from outer code
        return num1 + 1
    num2 = inner_increment(num1)
    print(num1, num2)

inner_increment(10)
# outer(10)

Try calling the inner_increment() function:

1
2
3
4

Traceback (most recent call last):
  File "inner.py", line 7, in <module>
    inner_increment()
NameError: name 'inner_increment' is not defined

Now comment out the inner_increment call and uncomment the outer function call, outer(10), passing in 10 as the argument:

1

10 11

Keep in mind that this is just an example. Although this code does achieve the desired result, it’s probably better to make the inner_increment() function a top-level “private” function using a leading underscore: _inner_increment().

The following recursive example is a slightly better use case for a nested function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def factorial(number):

    # error handling
    if not isinstance(number, int):
        raise TypeError("Sorry. 'number' must be an integer.")
    if not number >= 0:
        raise ValueError("Sorry. 'number' must be zero or positive.")

    def inner_factorial(number):
        if number <= 1:
            return 1
        return number*inner_factorial(number-1)
    return inner_factorial(number)

# call the outer function
print(factorial(4))

Test this out as well. One main advantage of using this design pattern is that by performing all argument checking in the outer function, you can safely skip error checking altogether in the inner function.

For a more detailed discussion of recursion see, Problem Solving with Algorithms and Data Structures.

2. Keepin’ it DRY

Perhaps you have a giant function that performs the same chunk of code in numerous places. For example, you might write a function which processes a file, and you want to accept either an open file object or a file name:

1
2
3
4
5
6
7
8
9

def process(file_name):
    def do_stuff(file_process):
        for line in file_process:
            print(line)
    if isinstance(file_name, str):
        with open(file_name, 'r') as f:
            do_stuff(f)
    else:
        do_stuff(file_name)

Again, it is common to just make do_stuff() a private top-level function, but if you want to hide it away as an internal function, you can.

How about a practical example?

Let’s say you want to know the number of WiFi hot spots in New York City. And yes the city has the raw data to tell us: datasource. Visit the site and download the CSV.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def process(file_name):

    def do_stuff(file_process):
        wifi_locations = {}

        for line in file_process:
            values = line.split(',')
            # Build the dict, and increment values
            wifi_locations[values[1]] = wifi_locations.get(values[1], 0) + 1

        max_key = 0
        for name, key in wifi_locations.items():
            all_locations = sum(wifi_locations.values())
            if key > max_key:
                max_key = key
                business = name
        print('There are {0} WiFi hot spots in NYC and {1} has the most with {2}.'.format(
            all_locations, business, max_key))

    if isinstance(file_name, str):
        with open(file_name, 'r') as f:
            do_stuff(f)
    else:
        do_stuff(file_name)


process("NAME_OF_THE.csv")

Run the function:

1

There are 1251 WiFi hot spots in NYC and Starbucks has the most with 212.

3. Closures and Factory Functions

Now we come to the most important reason to use inner functions. All of the inner function examples we’ve seen so far have been ordinary functions that merely happened to be nested inside another function. In other words, we could have defined these functions in another way (as discussed); there is no specific reason for why they should be nested.

But when it comes to closure, that is not the case: You must utilize nested functions.

What’s a closure?

A closure simply causes the inner function to remember the state of its environment when called. Beginners often think that a closure is the inner function, when it’s really caused by the inner function. The closure “closes” the local variable on the stack and this stays around after the the stack creation has finished executing.

An example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def generate_power(number):
    """
    Examples of use:

    >>> raise_two = generate_power(2)
    >>> raise_three = generate_power(3)
    >>> print(raise_two(7))
    128
    >>> print(raise_three(5))
    243
    """

    # define the inner function ...
    def nth_power(power):
        return number ** power
    # ... which is returned by the factory function

    return nth_power

What’s happening here?

1. The ‘generate_power()’ function is a factory function – which simply means that it creates a new function each time it is called and then returns the newly created function. Thus, raise_two and raise_three are the newly created functions.
2. What does this new, inner function do? It takes a single argument, power, and returns number**power.

3. Where does the inner function get the value of number from? This is where the closure comes into play: nth_power() gets the value of power from the outer function, the factory function. Let’s step through this process:

   • Call the outer function: generate_power(2)
   • Build the nth_power() function which takes a single argument power
   • Take a snapshot of the state of nth_power() which includes power=2
   • Pass that snapshot into the generate_power() function
   • Return the nth_power() function

   Put another way, the closure functions to “initialize” the number bar in the nth_power() function and then returns it. Now, whenever you call that newly returned function, it will always see its own private snapshot that includes power=2.

Real World

How about a real world example?

1
2
3
4
5
6
7
8
9
10
11
12
13
14

def has_permission(page):
    def inner(username):
        if username == 'Admin':
            return "'{0}' does have access to {1}.".format(username, page)
        else:
            return "'{0}' does NOT have access to {1}.".format(username, page)
    return inner


current_user = has_permission('Admin Area')
print(current_user('Admin'))

random_user = has_permission('Admin Area')
print(current_user('Not Admin'))

This is a simplified function to check if a certain user has the correct permissions to access a certain page. You could easily modify this to grab the user in session to check if they have the correct credentials to access a certain route. Instead of checking if the user is just equal to ‘Admin’, you could query the database to check the permission then return the correct view depending on whether the credentials are correct or not.

Conclusion

The use of closures and factory functions is the most common, and powerful, use for inner functions. In most cases, when you see a decorated function, the decorator is a factory function which takes a function as argument, and returns a new function which includes the old function inside the closure. Stop. Take a deep breath. Grab a coffee. Read that again.

Put another way, a decorator is just syntactic sugar for implementing the process outlined in the generate_power() example.

I’ll leave you with an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def generate_power(exponent):
    def decorator(f):
        def inner(*args):
            result = f(*args)
            return exponent**result
        return inner
    return decorator


@generate_power(2)
def raise_two(n):
    return n

print(raise_two(7))


@generate_power(3)
def raise_three(n):
    return n

print(raise_two(5))

If your code editor allows it, view the generate_power(exponent) and generate_power(number) functions side-by-side to illustrate the concepts discussed. (Sublime Text has Column View, for example).

If you have not coded the two functions, open the code editor and start coding. For new programmers, coding is a hands on activity, like riding a bike you just have to do it – and do it solo. So back to the task at hand. After you typed the code, you can now clearly see that the code is similar in that it produces the same results but there are differences. For those who have never used decorators, noting these differences will be the first step in understanding them if you venture down that path.

If you’d like to know more about this syntax and decorators in general, check out our Primer on Python Decorators. Comment below with questions.

Edits made by Derrick Kearney. Thank you!

This is the final article in our Django migrations series:

• Part 1: Django Migrations – A Primer
• Part 2: Digging Deeper into Migrations
• Part 3: Data Migrations (current article)
• Video: Django 1.7 Migrations – primer

Back again.

Migrations are mainly for keeping the data model of you database up-to-date, but a database is more than just a data model. Most notably, it’s also a large collection of data. So any discussion of database migrations wouldn’t be complete without also talking about data migrations.

Data Migrations Defined

Data migrations are used in a number of scenarios. Two very popular ones are:

1. When you would like to load “system data” that your application depends upon being present to operate successfully.
2. When a change to a data model forces the need to change the existing data.

Do note that loading dummy data for testing is not in the above list. You could use migrations to do that, but migrations are often run on production servers, so you probably don’t want to be creating a bunch of dummy test data on your production server.

Examples

Continuing from the previous Django Project, as an example of creating some “system data”, let’s create some historical bitcoin prices. Django migrations will help us out, by creating an empty migration file and putting it in the right place if we type:

1

$ ./manage.py makemigrations --empty historical_data

This should create a file called historical_data/migrations/003_auto<date_time_stamp>.py. Let’s change the name to 003_load_historical_data.py and then open it up. You’ll have a default structure which looks like:

1
2
3
4
5
6
7
8
9
10
11
12

# encoding: utf8
from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('historical_data', '0002_auto_20140710_0810'),
    ]

    operations = [
    ]

You can see it’s created a base structure for us, and even inserted the dependencies. That’s helpful. Now to do some data migrations, use the RunPython migration operation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# encoding: utf8
from django.db import models, migrations
from historical_data.models import PriceHistory
from datetime import date

def load_data(apps, schema_editor):
    PriceHistory(date=date(2013,11,29),
         price=1234.00,
         volume=354564,
         total_btc=12054375,
         ).save()
    PriceHistory(date=date(2012,11,29),
         price=12.15,
         volume=187947,
         total_btc=10504650,
         ).save()


class Migration(migrations.Migration):

    dependencies = [
        ('historical_data', '0002_auto_20140710_0810'),
    ]

    operations = [
        migrations.RunPython(load_data)
    ]

We start off by defining the function load_data which – you guessed it – loads data.

For a real app we might want to go out to blockchain.info and grab the complete list of historic prices, but we just put a couple in there to show how the migration works.

Once we have the function we can call it from our RunPython operation and then this function will be executed when we run ./manage.py migrate from the command line.

This is arguably more work than running syncdb and having it load a fixture. In fact, migrations don’t respect fixtures – meaning they won’t automatically load them for you like syncdb would.

This is mainly due to philosophy.

While you could use migrations to load data, they are mainly about migrating data and/or data models. We’ve shown an example of loading system data, mainly because it’s a simple explanation of how you would set up a data migration, but often times, data migrations are used for more complex actions like transforming your data to match the new data model.

An example might be if we decided to start storing prices from multiple exchanges instead of just one, so we could add fields like price_gox, price_btc, etc, then we could use a migration to move all data from the price column to the price_btc column.

In general when dealing with migrations in Django 1.7, it’s best to think of loading data as a separate exercise from migrating the database. If you do want to continue to use/load fixtures, you can use a command like:

1

$ ./manage.py loaddata historical_data/fixtures/initial_data.json

This will load data from the fixture into the database.

This doesn’t happen automatically as with a data migration (which is probably a good thing), but the functionality is still there; it hasn’t been lost, so feel free to continue to use fixtures if you have a need. The difference is that now you load data with fixtures when you need it. This is something to keep in mind if you are using fixtures to load test data for your unit tests.

Conclusion

This, along with the previous two articles, covers the most common scenarios you’ll encounter when using migrations. There are plenty more scenarios, and if you’re curious and really want to dive into migrations, the best place to go (other than the code itself) is the official docs. It’s the most up-to-date and does a pretty good job of explaining how things work. If there is a more complex scenario that you would like to see an example of, please let us know by commenting below.

Remember that in the general case, you are dealing with either:

1. Schema Migrations – a change to the structure of the database or tables with no change to the data. This is the most common type, and Django can generally create these migrations for you automatically.

2. Data Migrations – a change to the data, or loading new data. Django cannot generate these for you. They must be created manually using the RunPython migration.

So pick the migration that is correct for you, run makemigrations and then just be sure to update your migration files every time you update your model – and that’s more or less it. That will allow you to keep your migrations stored with your code in git and ensure that you can update your database structure without having to lose data.

Happy migrating!

This is the second article in our Django migrations series:

• Part 1: Django Migrations – A Primer
• Part 2: Digging Deeper into Migrations (current article)
• Part 3: Data Migrations
• Video: Django 1.7 Migrations – primer

We’re back!

Last time we went over the basics of using the new Django migrations system.

Jumping right back in to where we left off, what happens when things don’t work as they should? Well, that’s when you may have to go searching through your previous migrations to try to figure out what’s going on. To help with that let’s dig a bit deeper to get a better understanding of how migrations work.

How Migrations Know What to Migrate

Try this. From the bitcoin_tracker app run the migration again (./manage.py migrate). What happens? Nothing. And that’s exactly the point.

By default Django will never run a migration more than once on the same database. This is managed by a table called django_migrations that is created in your database the first time migrations are ran. For each migration that is ran or faked, a new row is inserted into the table.

For example, here is what the table might look like after running our initial migration:

Not very interesting because there is just one migration, but there will be new rows added for each subsequent migration.

The next time migrations are run, it will skip the migration files listed in the database table. This means that even if you change the migration file manually, it will be skipped if there is already an entry for it in the database.

This makes sense as you generally don’t want to run migrations twice. But if for whatever reason you do, one way to get it to run again is to first delete the corresponding row from the database (Do note this is not an “officially recommended way”, but it will work). In the case of upgrading from South the first time you run migrations, Django will first check the database structure, and if it is the same as the migration (i.e., the migration doesn’t apply any new changes) then the migration will be “faked” meaning not really ran, but the django_migrations table will still be updated.

Conversely, if you want to “undo” all the migrations for a particular app, you can migrate to a special migration called zero.

For example if you type:

1

./manage.py migrate historical_data zero

It will undo/reverse all the migrations for the historical_data app. In addition to using zero; you can also use any arbitrary migration, and if that migration is in the past then the database will be rolled back to the state of that migration, or rolled forward if the migration hasn’t yet ran. Pretty powerful stuff!

The Migration File

What about creating the actual migration file? In other words, what exactly happens when you run ./manage.py makemigrations <appname>?

Django migrations are actually creating a migration file that describes how to create the appropriate tables in the database. In fact, you can look at the migration file that was created. Don’t worry: It’s just Python.

Don’t forget to git add the new migrations directory so it is version-controlled.

The historical_prices app will now have a sub-directory called /migrations where all the migration files for that app will reside. Let’s look at historical_data/migrations/0001_initial.py, as this is the file where the initial migration code is created. It should look similar to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# encoding: utf8
from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='PriceHistory',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, primary_key=True, auto_created=True)),
                ('date', models.DateTimeField(auto_now_add=True)),
                ('price', models.DecimalField(decimal_places=2, max_digits=5)),
                ('volume', models.PositiveIntegerField()),
                ('total_btc', models.PositiveIntegerField()),
            ],
            options={
            },
            bases=(models.Model,),
        ),
    ]

For a migration to work, you must create a class called Migration() that inherits from django.db.migrations.Migration. This is the class that the migration framework will look for and execute when you ask it to run migrations (which we will do later).

The Migration class contains two main lists, dependencies and operations.

Migration dependencies

dependencies is a list of migrations that must be ran prior to this migration being run.

In the case above nothing has to be run prior so there are no dependencies. But if you have foreign key relationships, for example, then you will have to ensure a model is created before you can add a foreign key to it. So let’s assume we had another app called main that defined the table we wanted to reference in our foreign key. Then our dependencies list might look like this:

1
2
3

dependencies = [
   ('main', '__first__'),
]

The dependency above says that migrations for the main app must be run first.

Be sure check out the video to see examples of utilizing foreign keys and how they affect the dependencies within the migration files.

You can also have a dependency on a specific file, like so:

1
2
3

dependencies = [
    ('main', '0001_initial'),
]

This is a dependency for the file called 0001_initial from the main app.

Dependencies can also be combined so you can have multiple dependencies. This functionality provides a lot of flexibility, as you can accommodate foreign keys that depend upon models from different apps. It also means that the numbering of the migrations (usually 0001, 0002, 0003, …) doesn’t strictly have to be in the order they are applied. You can add any dependency you want and thus control the order without having to re-number all the migrations.

Migration operations

The second list in the Migration() class is the operations list. This is a list of operations to be applied as part of the migration. Generally the operations can fall under one of the following types:

• CreateModel: You guessed it: this creates a new model. See the migration above for an example.
• DeleteModel: removes a table from the database; just pass in the name of the model.
• RenameModel: Given the old_name and new_name, this renames the model.
• AlterModelTable: changes the name of the table associated with a model. Same as the db_table option.
• AlterUniqueTogether: changes unique constraints.
• AlteIndexTogether: changes the set of custom indexes for the model.

• AddField: Just like it sounds. Here is an example:

  python migrations.AddField( model_name='PriceHistory', name='market_cap', field=models.PositiveIntegerField(), ),

• RemoveField: We don’t want that field anymore… just drop it.

• RenameField: Given model_name, old_name and new_name, this changes the field with old_name to new_name.

There are also a few “special” operations:

• RunSQL: This allows you to pass in raw SQL and execute it as part of your model.
• RunPython: passes in a callable to be executed; useful for things like data loading as part of the migration.

You can even write your own operations. Generally when you run makemigrations, Django will create the necessary migrations with the appropriate dependencies and operations that you need. However, understanding the migration files themselves and how they work give you more flexibility.

Example

Let’s make a few more changes to our model to see the effect on the migrations:

1
2
3
4
5
6
7

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    #bitcoin to the moon (we need more digits)
    price = models.DecimalField(max_digits=8, decimal_places=2)
    volume = models.PositiveIntegerField()
    total_btc = models.PositiveIntegerField()
    market_cap = models.PositiveIntegerField(null=True)

Being bullish on bitcoin we have decided we need a larger number for the price field, and we have also decided to keep track of the market capitalization. Notice how we made the market_cap field nullable. If we didn’t, migrations will ask to supply a value for all the existing rows (just like South does):

1
2
3
4
5

You are trying to add a non-nullable field 'market_cap' to PriceHistory without a default;
we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows)
 2) Quit, and let me add a default in models.py

Now running ./manage.py makemigrations again will produce a new migration file 0002_auto_<date_time_stamp>.

That file should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

class Migration(migrations.Migration):

    dependencies = [
        ('historical_data', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='PriceHistory',
            name='market_cap',
            field=models.PositiveIntegerField(null=True),
            preserve_default=True,
        ),
        migrations.AlterField(
            model_name='PriceHistory',
            name='price',
            field=models.DecimalField(max_digits=8, decimal_places=2),
        ),
    ]

Notice the ‘dependencies’ list which declares that we have to run our initial migration prior to running this one. Also this migration has two operations – AddField, which creates our newly added market_cap, and AlterField, which updates the max_digits of our price field.

It’s important to understand that these operations just call the migrations framework, which handles performing the various operations against the database that is defined in your settings.py file.

Out of the box migrations have support for all the standard databases that Django supports. So if you stick to the primitives listed in dependencies section you can manually create whatever migration you want, without having to worry about the underlying SQL. That’s all done for you.

Conclusion

We’ve come to another end, but there’s one more beginning. In the last post, we’ll look at Data Migrations. Cheers!

He is now sharing his love.

This blog post is a follow-up to Face Recognition in Python, so make sure you’ve gone through that first post.

As I mentioned in the first post, it’s quite easy to move from detecting faces in images to detecting them in video via a webcam – which is exactly what we will detail in this post.

Pre-requisites:

1. OpenCV installed (see the previous blog post for details)
2. A working webcam

The Code

Let’s dive straight into the code, taken from this repository.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import cv2
import sys

cascPath = sys.argv[1]
faceCascade = cv2.CascadeClassifier(cascPath)

video_capture = cv2.VideoCapture(0)

while True:
    # Capture frame-by-frame
    ret, frame = video_capture.read()

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = faceCascade.detectMultiScale(
        gray,
        scaleFactor=1.1,
        minNeighbors=5,
        minSize=(30, 30),
        flags=cv2.cv.CV_HAAR_SCALE_IMAGE
    )

    # Draw a rectangle around the faces
    for (x, y, w, h) in faces:
        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

    # Display the resulting frame
    cv2.imshow('Video', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything is done, release the capture
video_capture.release()
cv2.destroyAllWindows()

Now let’s break it down…

1
2
3
4
5

import cv2
import sys

cascPath = sys.argv[1]
faceCascade = cv2.CascadeClassifier(cascPath)

This should be familiar to you. We are creating a face cascade, as we did in the image example.

1

video_capture = cv2.VideoCapture(0)

This line sets the video source to the default webcam, which OpenCV can easily capture.

NOTE: You can also provide a filename here, and Python will read in the video file. However, you need to have ffmpeg installed for that since OpenCV itself cannot decode compressed video. Ffmpeg acts as the front end for OpenCV, and, ideally, it should be compiled directly into OpenCV. This is not easy to do, especially on Windows.

1
2
3

while True:
    # Capture frame-by-frame
    ret, frame = video_capture.read()

Here, we capture the video. The read() function reads one frame from the video source, which in this example is the webcam. This returns:

1. The actual video frame read (one frame on each loop)
2. A return code

The return code tells us if we have run out of frames, which will happen if we are reading from a file. This doesn’t matter when reading from the webcam, since we can record forever, so we will ignore it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# Capture frame-by-frame
ret, frame = video_capture.read()

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

faces = faceCascade.detectMultiScale(
    gray,
    scaleFactor=1.1,
    minNeighbors=5,
    minSize=(30, 30),
    flags=cv2.cv.CV_HAAR_SCALE_IMAGE
)

# Draw a rectangle around the faces
for (x, y, w, h) in faces:
    cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

# Display the resulting frame
cv2.imshow('Video', frame)

Again, this code should be familiar. We are merely searching for the face in our captured frame.

1
2

if cv2.waitKey(1) & 0xFF == ord('q'):
    break

We wait for the ‘q’ key to be pressed. If it is, we exit the script.

1
2
3

# When everything is done, release the capture
video_capture.release()
cv2.destroyAllWindows()

Here, we are just cleaning up.

Test!

So, that’s me with my driver’s license in my hand. And you can see that the algorithm tracks both the real me and the photo me. Note that when I move slowly, the algorithm can keep up. When I move my hand up to my face a bit faster, though, it gets confused and mistakes my wrist for a face.

Like I said in the last post, machine learning based algorithms are rarely 100% accurate. We aren’t at the stage where Robocop driving his motorcycle at 100 mph can track criminals using low quality CCTV cameras… yet.

The code searches for the face frame by frame, so it will take a fair amount of processing power. For example, on my five year old laptop, it took almost 90% of the CPU.

Next Steps

Okay, so you know how to detect faces. But what if you want to detect your own object, like your car or your TV or your favorite toy?

OpenCV allows you to create your own cascades, but the process isn’t well documented. Here is a blog post that shows you how to train your own cascade to detect a banana.

If you want to take it one step further and recognize individual faces – perhaps to detect and recognize your face amongst many strangers – the task is surprisingly difficult. This is mainly due to the large amount of image pre-processing involved. But if you are willing to tackle the challenge, it is possible by using machine learning algorithms as described here.

Want to know more?

This will be covered in much greater detail along with a number of computational science and machine learning topics and more in my upcoming course. The course is based on a highly successful Kickstarter.

The Kickstarter is over, but you can still order the course at Python for Engineers. Visit to find out more.

Also, post links to your videos below to get direct feedback from me. Comment if you have questions.

Oh – and next time we’ll be tackling some motion detection. Stay tuned!

Cheers!

What’s new in Django 1.7? Basically migrations. While there are some other nice features, the new migrations system is the big one.

In the past you probably used South to handle database changes. However, in Django 1.7, migrations are now integrated into the Django Core thanks to Andrew Godwin, who ran this Kickstarter. He is also the original creator of South.

In honor of this momentous update, we are going to cover migrations, how they work and how to get the most out of them across three blog posts and one video:

• Part 1: Django Migrations – A Primer (current article)
• Part 2: Digging Deeper into Migrations
• Part 3: Data Migrations
• Video: Django 1.7 Migrations – primer

Let’s begin…

The problems that Migrations Solve

Migrations:

1. Speed up the notoriously slow process of changing a database schema.
2. Make it easy to use git to track your database schema and its associated changes. Databases simply aren’t aware of git or other version control systems. Git is awesome for code, but not so much for database schemas.
3. Provide an easy way to maintain fixture data linked to the appropriate schema.
4. Keep the code and schema in sync.

Have you ever had to make a change to an existing table (like re-naming a field) and you didn’t want to mess with dropping the table and re-adding it?

Migrations solve that.

Or perhaps you needed to update the schema on a live application with millions of rows of data that simply cannot be lost.

Migrations make this much easier.

In general, migrations allow you to manage and work with your database schema in the same way you would with your Django code. You can store versions of it in git, you can update it from the command line, and you don’t have to worry about creating large complex SQL queries to keep everything up to date – although you still can if you love pain… I mean SQL.

Getting Started with Migrations

For this blog post we are going to create a simple bitcoin tracker app (actually we are just going to create the model).

Be sure check out the video to see examples of the migrations workflow.

Setup the project

With Django 1.7 installed you can create the project using the following commands:

1
2
3

$ django-admin.py startproject bitcoin_tracker
$ cd bitcoin_tracker
$ ./manage.py startapp historical_data

That should give you a simple project with an app called historical_data. Now to create a model edit historical_data/models.py:

1
2
3
4
5

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    price = models.DecimalField(max_digits=5, decimal_places=2)
    volume = models.PositiveIntegerField()
    total_btc = models.PositiveIntegerField()

Also don’t forget to add the newly created app to settings.INSTALLED_APPS.

This is a basic model to keep track of bitcoin prices.

Create the migrations

With the model created, the first thing we need to do is “initialize” our migrations. We can do this with the following command:

1

./manage.py makemigrations historical_data

Note: Specifying the name of the application, historical_data, is optional. Leaving it off will create migrations for all apps.

Upon running the above command Django migrations will create the migrations for you and output an informational message such as:

1
2
3

Migrations for 'historical_data':
  0001_initial.py:
    - Create model PriceHistory

This creates the migrations files which instruct Django on how to create the models that you need.

Apply migrations

Then to apply the models (e.g., update the database) just run:

1

$ ./manage.py migrate

Not only will this apply the migrations we just created for PriceHistory but since this is a new app, it will create all the Django tables for us as well (just like syncdb did in Django versions prior to 1.7).

Workflow

So the basic process for using migrations looks like this:

1. Create or update a model.
2. Run ./manage.py makemigrations <app_name>.
3. Run ./manage.py migrate to migrate everything or ./manage.py migrate <app_name> to migrate an individual app.
4. Repeat as necessary.

That’s it! Pretty straight-forward for the basic use-case. Fortunately, this basic use case will work the majority of the time!

Apply Migrations to an Existing Project

That’s all well and good if you’re starting from scratch but many people will be upgrading from a previous version of Django and migrating from South.

To do that, Django recommends to just start using the new migrations system and everything should work. The recommended upgrade path from South to Django 1.7 migrations, paraphrased from here is basically:

1. Delete all your South migration files (yup – just blow ‘em away).
2. Run ./manage.py makemigrations. Django will just make initial migrations files based upon your current models.
3. Run ./manage.py migrate. Django will see the tables that already exist for your migrations and just mark them as completed without running them.

Now, if you have circular dependencies and are getting errors when going through the above process, you may have to fake the migration with:

1

$ ./manage.py migrate --fake <appname>

Also, if you have a need to support both South and Django Migrations at the same time then upgrade to South 1.0 and read this article.

South vs Django migrations

For those familiar with South this should feel pretty familiar and probably a little bit cleaner. For easy reference the following table compares the old South workflow to the new Django Migrations workflow:

So from the table we can see that Django Migrations, basically follows the same process as South (at least for the standard migration process) – it just simplifies things a bit.

Conclusion

That’s it. It’s all meant to be very straight forward. Of course there will always be edge cases when things don’t work out exactly as intended – but we’ll save that for next time.

Cheers!

Video

In this post we’ll look at a surprisingly simple way to get started with face recognition using Python and the open source library OpenCV.

OpenCV

OpenCV is the most popular library for computer vision. Originally written in C/C++, it now provides bindings for Python.

OpenCV uses machine learning algorithms to search for faces within a picture. For something as complicated as a face, there isn’t one simple test that will tell you if it found a face or not. Instead, there are thousands of small patterns/features that must be matched. The algorithms break the task of identifying the face into thousands of smaller, bite-sized tasks, each of which is easy to solve. These tasks are also called classifiers.

For something like a face, you might have 6,000 or more classifiers, all of which must match for a face to be detected (within error limits, of course). But therein lies the problem: For face detection, the algorithm starts at the top left of a picture and moves down across small blocks of data, looking at each block, constantly asking, “Is this a face? … Is this a face? … Is this a face?” Since there are 6,000 or more tests per block, you might have millions of calculations to do, which will grind your computer to a halt.

To get around this, OpenCV uses cascades. What’s a cascade? The best answer can be found from the dictionary: A waterfall or series of waterfalls

Like a series of waterfalls, the OpenCV cascade breaks the problem of detecting faces into multiple stages. For each block, it does a very rough and quick test. If that passes, it does a slightly more detailed test, and so on. The algorithm may have 30-50 of these stages or cascades, and it will only detect a face if all stages pass. The advantage is that the majority of the pictures will return negative during the first few stages, which means the algorithm won’t waste time testing all 6,000 features on it. Instead of taking hours, face detection can now be done in real time.

Cascades in practice

Though the theory may sound complicated, in practice it is quite easy. The cascades themselves are just a bunch of XML files that contain OpenCV data used to detect objects. You initialize your code with the cascade you want, and then it does the work for you.

Since face detection is such a common case, OpenCV comes with a number of built-in cascades for detecting everything from faces to eyes to hands and legs. There are even cascades for non-human things. For example, if you run a banana shop and want to track people stealing bananas, this guy has built one for that!

Installing OpenCV

First, you need to find the correct setup file for your operating system.

I found that installing OpenCV was the hardest part of the task. If you get strange unexplainable errors, it could be due to library clashes, 32/64 bit differences, etc. I found it easiest to just use a Linux virtual machine and install OpenCV from scratch.

Once installed, you can test whether or not it works by firing up a Python session and typing:

1
2
3

$ python
>>> import cv2
>>>

If you don’t get any errors, you can move on to the next part.

Understanding the code

Let’s break down the actual code, which you can download from the repo. Grab the face_detect.py script, the abba.png pic, and the haarcascade_frontalface_default.xml.

1
2
3

# Get user supplied values
imagePath = sys.argv[1]
cascPath = sys.argv[2]

You first pass in the image and cascade names as command-line arguments. We’ll use the Abba image as well as the default cascade for detecting faces provided by OpenCV.

1
2

# Create the haar cascade
faceCascade = cv2.CascadeClassifier(cascPath)

Now we create the cascade and initialize it with our face cascade. This loads the face cascade into memory so it’s ready for use. Remember, the cascade is just an XML file that contains the data to detect faces.

1
2
3

# Read the image
image = cv2.imread(imagePath)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Here we read the image and convert it to grayscale. Many operations in OpenCv are done in grayscale.

1
2
3
4
5
6
7
8

# Detect faces in the image
faces = faceCascade.detectMultiScale(
    gray,
    scaleFactor=1.1,
    minNeighbors=5,
    minSize=(30, 30),
    flags = cv2.cv.CV_HAAR_SCALE_IMAGE
)

This function detects the actual face – and is the key part of our code, so let’s go over the options.

1. The detectMultiScale function is a general function that detects objects. Since we are calling it on the face cascade, that’s what it detects. The first option is the grayscale image.

2. The second is the scaleFactor. Since some faces may be closer to the camera, they would appear bigger than those faces in the back. The scale factor compensates for this.

3. The detection algorithm uses a moving window to detect objects. minNeighbors defines how many objects are detected near the current one before it declares the face found. minSize, meanwhile, gives the size of each window.

   I took commonly used values for these fields. In real life, you would experiment with different values for the window size, scale factor, etc., until you find one that best works for you.

The function returns a list of rectangles where it believes it found a face. Next, we will loop over where it thinks it found something.

1
2
3
4
5

print "Found {0} faces!".format(len(faces))

# Draw a rectangle around the faces
for (x, y, w, h) in faces:
    cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2)

This function returns 4 values: the x and y location of the rectangle, and the rectangle’s width and height (w , h).

We use these values to draw a rectangle using the built-in rectangle() function.

1
2

cv2.imshow("Faces found" ,image)
cv2.waitKey(0)

In the end, we display the image, and wait for the user to press a key.

Checking the results

Let’s test against the Abba photo:

1

$ python face_detect.py abba.png haarcascade_frontalface_default.xml

That worked. How about another photo:

That … is not a face. Let’s try again. I changed the parameters and found that setting the scaleFactor to 1.2 got rid of the wrong face.

What happened? Well, the first photo was taken fairly close up with a high quality camera. The second one seems to have been taken from afar and possibly from a mobile phone. This is why the scaleFactor had to be modified. As I said, you’ll have to setup the algorithm on a case by case basis to avoid false positives.

Be warned though that since this is based on machine learning, the results will never be 100% accurate. You will get good enough results in most cases, but occasionally the algorithm will identify incorrect objects as faces.

The final code can be found here.

Extending to a webcam

So what if you want to use a webcam? OpenCV grabs each frame from the webcam and you can then detect faces by processing each frame. You will need a powerful computer, though my five year old laptop seems to cope fine, as long as I don’t dance around too much.

UPDATED The next blog post is live: Face Detection in Python Using a Webcam. Check it out!

Want to know more?

I will be covering this and more in my upcoming book Python for Science and Engineering, which is currently on Kickstarter. I will also cover machine learning, for those who are interested in it.

Thanks!

http://discoverflask.com

Series Overview

Last time we went over how to set up a basic Flask structure and then developed a static site, styled with Bootstrap. In this second part of the series, we’ll be adding a login page for end users to, well, login to.

Building on the code from the previous tutorial, we need to-

• Add a route to handle requests to the login URL, and
• Add a template for the login page

Add a route to handle requests to the login URL

Make sure your virtualenv is activated. Open app.py in your code editor and add the following route:

1
2
3
4
5
6
7
8
9
10

# route for handling the login page logic
@app.route('/login', methods=['GET', 'POST'])
def login():
    error = None
    if request.method == 'POST':
        if request.form['username'] != 'admin' or request.form['password'] != 'admin':
            error = 'Invalid Credentials. Please try again.'
        else:
            return redirect(url_for('home'))
    return render_template('login.html', error=error)

Make sure you also update the imports:

1

from flask import Flask, render_template, redirect, url_for, request

So, what’s going on?

1. First, notice that we specified the applicable HTTP methods for the route, GET and POST, as an argument in the route decorator.

2. GET is the default method. So, if no methods are explicitly defined, Flask assumes that the only available method is GET, as is the case for the previous two routes, / and /welcome.

3. For the new /login route we need to specifiy the POST method as well as GET so that end users can send a POST request with their login credentials to that /login endpoint.

4. The logic within the login() function tests to see if the credentials are correct. If they are correct, then the user is redirected to the main route, /, and if the credentials are incorrect, an error populates. Where do these credentials come from? The POST request, which you’ll see in just a minute.

5. In the case of a GET request, the login page is simply rendered.

NOTE: The url_for() function generates an endpoint for the provided method.

Add a template for the login page

Create a new file called login.html, adding it to the “templates” directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<html>
  <head>
    <title>Flask Intro - login page</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="static/bootstrap.min.css" rel="stylesheet" media="screen">
  </head>
  <body>
    <div class="container">
      <h1>Please login</h1>
      <br>
      <form action="" method="post">
        <input type="text" placeholder="Username" name="username" value="{{
          request.form.username }}">
         <input type="password" placeholder="Password" name="password" value="{{
          request.form.password }}">
        <input class="btn btn-default" type="submit" value="Login">
      </form>
      {% if error %}
        <p class="error"><strong>Error:</strong> {{ error }}
      {% endif %}
    </div>
  </body>
</html>

Time for a quick test …

1. Fire up the server. Navigate to http://localhost:5000/login.

2. Enter the incorrect credentials, then press login. You should get this response: “Error: Invalid Credentials. Please try again.”

3. Now use “admin” for both the username and password and you should be redirected to the / URL.

4. Can you tell what’s happening here? When the form is submitted, a POST request is sent along with the form data, value="{{request.form.username }}" and value="{{request.form.password }}", to the controller, app.py – which then handles the request and either responds with an error message or redirects the user to the / URL. Be sure to check out the accompanying video to dig deeper into this with Chrome Developer Tools!

5. Finally, we have some logic in our templates. Originally, we passed in None for the error. Well, if the error is not None, then we display the actual error message, which gets passed to the template from the views: <p class="error"><strong>Error:</strong> {{ error }}</p>. For info on how this works, check out this blog post to learn more about the Jinja2 templating engine.

Conclusion

What do you think? Simple, right? Don’t get too excited yet, as we still have much more to do with reguard to user management …

Now that users have the ability to login, we need to protect that URL / from unauthorized access. In other words, when an end user hits that endpoint, unless they are already logged in, then they should be immediately sent to the login page. Next time. Until then, go practice some jQuery.

Be sure to grab the code and watch the video.

Video

http://discoverflask.com

Series Overview

Flask is a micro web framework powered by Python. It’s API is fairly small, making it easy to learn and simple to use. But don’t let this fool you, as it’s powerful enough to support enterprise-level applications handling large amounts of traffic. You can start small with an app contained entirely in one file, then slowly scale up to multiple files and folders in a well-structured manner as your site becomes more and more complex.

It’s an excellent framework to start with, which you’ll be learning in true “Real Python-style”: with hands-on, practical examples that are interesting and fun.

Note: This tutorial was original posted on 01/29/2013. We revamped it and due to the number of changes made, it was decided to “retire” the old tutorial and create an entirely new one. If you are interested in viewing the code and video for the old tutorial, please visit this repo.

Check out the accompanying video.

Requirements

This tutorial assumes you have Python 2.7.x, pip, and virtualenv installed.

Ideally, you should have basic knowledge of the command line or terminal and Python in general. If not, you will learn enough to get by, then as you continue working with Flask, your development skills will advanced as well. If you do want additional help, check out the Real Python series to learn Python and web development from the ground up.

You also need a code editor or IDE such as Sublime Text, gedit, Notepad++, or VIM, etc. If you’re ensure what to use, check out Sublime Text, which is a lightweight, yet powerful cross-platform code editor.

Conventions

1. All examples in this tutorial utilize a Unix-style prompt: $ python hello-world.py. Remember that the dollar sign is not part of the command, and the equivalent command in Windows is: C:\Sites> python hello-world.py.

2. All examples are coded in Sublime Text 3.

3. All examples utilize Python 2.7.7. You can use any version of 2.7.x, though.

4. Additional requirements and dependency versions are listed in the requirements.txt file found within the Github repo.

Setup

1. Navigate to a convenient directory, such as the “Desktop” or “Documents” folder
2. Create a new directory called “flask-intro” to house your project
3. Activate a virtualenv
4. Install Flask with Pip $ pip install Flask

Structure

If you’re familiar with Django, web2py, or any other high-level (or full-stack) framework, then you know that each impose a specific structure. Due to it’s minimalist nature, however, Flask does not provide a set structure, which can be difficult for beginners. Fortunately, it’s fairly easy to figure out, especially if you use a single file for the Flask components.

Create the following project structure within the “flask-intro” folder:

1
2
3
4

.
├── app.py
├── static
└── templates

Here, we simply have a single file for our Flask app called app.py and then two folders, “static” and “templates”. The former houses our stylesheets, Javascript files, and images, while the latter is for HTML files. This is a good base to start with. We’re already thinking in terms of front vs. back-end. The app.py will utilize the Model-View-Controller (MVC) design pattern in the back-end to handle requests and dish out responses to the end user.

Put simply, when a request comes in, the controller, which handles our app’s business logic, decides how to handle it.

For example, the controller could communicate directly with a database (like MySQL, SQLite, PostgreSQL, MongoDB, etc.) to obtain the requested data and return a response, via the views, with the appropriate data in the appropriate format (like HTML or JSON). Or perhaps the end user’s request is for a resource that does not exist – in which case the controller will respond with a 404 error.

Starting with this structure will help with scaling your app out into separate files and folders, since there’s already a logical separation between the front and back-end. If you’re unfamiliar with the MVC pattern, read more about it here. Get used to it as almost every web framework utilizes some form of MVC.

Routes

Open your favorite editor and add the following code to your app.py file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# import the Flask class from the flask module
from flask import Flask, render_template

# create the application object
app = Flask(__name__)

# use decorators to link the function to a url
@app.route('/')
def home():
    return "Hello, World!"  # return a string

@app.route('/welcome')
def welcome():
    return render_template('welcome.html')  # render a template

# start the server with the 'run()' method
if __name__ == '__main__':
    app.run(debug=True)

This is fairly straightforward.

After importing the Flask class, we create (or instantiate) the application object, define the views to respond to requests, then start the server.

The route decorators are used to associate (or map) a URL to a function. The URL / is associated with the home() function, so when the end user requests that URL, the view will respond with a string. Similarly, when the /welcome URL is requested, the view will render the welcome.html template.

In short, the main application object is instantiated which is then used to map URLs to functions.

For a more detailed explanation, read Flask’s quick-start tutorial.

Test

Time for a sanity check. Fire up your development server:

1

$ python app.py

Navigate to http://localhost:5000/. You should see “Hello, World!” staring back at you. Then request the next URL, http://localhost:5000/welcome. You should see, a “TemplateNotFound” error. Why? Because we have not set up our template, welcome.html. Flask is looking for it, but it’s not there. Let’s do that. First, kill the server by pressing CTRL+C from your terminal.

For more on the actual response, check out the video accompanying this post.

Templates

Create a new file in your templates directory called welcome.html. Open this file in your code editor, then add the following HTML:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<!DOCTYPE html>
<html>
  <head>
    <title>Flask Intro</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
  </head>
  <body>
    <div class="container">
      <h1>Welcome to Flask!</h2>
      <br>
      <p>Click <a href="/">here</a> to go home.</p>
    </div>
  </body>
</html>

Save. Run your server again. What do you see now when you request http://localhost:5000/welcome? Test the link. It works but it’s not very pretty. Let’s change that. This time leave the server running as we make the changes.

Bootstrap

Okay. Let’s utilize those static folders by adding a stylesheet. Have you heard of Bootstrap? If your answer is no, then check out this blog post for details.

Download Bootstrap, then add the bootstrap.min.css and bootstrap.min.js files to your “static” folder.

Update the template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<!DOCTYPE html>
<html>
  <head>
    <title>Flask Intro</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="static/bootstrap.min.css" rel="stylesheet" media="screen">
  </head>
  <body>
    <div class="container">
      <h1>Welcome to Flask!</h2>
      <br>
      <p>Click <a href="/">here</a> to go home.</p>
    </div>
  </body>
</html>

We just included the CSS stylesheet; we’ll add the Javascript file in a latter tutorial.

Return to your browser.

Remember that we left the server running? Well, when Flask is in debug mode, app.run(debug=True), there’s an auto-reload mechanism that kicks in on code changes. Thus, we can just press refresh in our browser and we should see the new template staring right back at us.

Nice.

Conclusion

What are your initial thoughts? Comment below. Grab the code. Watch the video.

In under thirty minutes, you learned the Flask basics and set the ground work for a larger app. If you’ve worked with Django before, you probably noticed immediately that Flask stays out of your way as you develop, leaving you free to structure and design your app how you see fit. Because of the lack of structure, true beginners may struggle a bit, however it’s an invaluable learning experience that will benefit you in the long run, regardless of whether you continue working with Flask or move on to a higher-level framework.

In the next tutorial we’ll look at adding some dynamic content.

Cheers!

Video

Problem

1. Football: The football.csv file contains the results from the English Premier League. The columns labeled ‘Goals’ and ‘Goals Allowed’ contain the total number of goals scored for and against each team in that season (so Arsenal scored 79 goals against opponents, and had 36 goals scored against them). Write a program to read the file, then print the name of the team with the smallest difference in ‘for’ and ‘against’ goals.
2. Weather: In weather.csv you’ll find daily weather data. Write a program to read the file, then output the day number (column one) with the smallest temperature spread (the maximum temperature is the second column, the minimum the third column).
3. See if you can write the same program to solve both questions.
4. Test Driven Development!

You can grab all the files here.

Try this on your own before you look at my answer below.

Note: I have not solved this problem before, so I will be going through a number of iterations. Hopefully, this will give you insight into the process/workflow I went through. Once finished, compare your workflow with mine. What did you do differently? Finally, I will be using a true TDD approach for this. In other words, after I write a test, I will only write the bare minimum amount of code to get it to pass. That said, there will most likely be plenty of refactoring throughout. Bare with me. Patience.

Part 1: Football

Write your tests

In true TDD style, we start by writing our tests that will fail. Since we must first read in the CSV data, let’s ensure that we can do that, by testing that the data read in is what we think it should be.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import unittest
from parse_csv import read_data


class ParseCSVTest(unittest.TestCase):

    def setUp(self):
        self.data = 'football.csv'

    def test_csv_read_data_headers(self):
        self.assertEqual(
            read_data(self.data)[0],
            ['Team', 'Games', 'Wins', 'Losses', 'Draws', 'Goals', 'Goals Allowed', 'Points']
            )

    def test_csv_read_data_team_name(self):
        self.assertEqual(read_data(self.data)[1][0], 'Arsenal')

    def test_csv_read_data_points(self):
        self.assertEqual(read_data(self.data)[1][7], '87')


if __name__ == '__main__':
    unittest.main()

Here, we are using a function imported from parse_csv.py called read_data(), which we haven’t written yet, to read in the data. Then we have three unit tests to check the values of the data read in.

Run your tests:

1
2
3
4
5

$ python parse_csv_test.py
Traceback (most recent call last):
  File "parse_csv_test.py", line 2, in <module>
    from parse_csv import read_data
ImportError: cannot import name read_data

Write the read_data() function

Next, let’s write just enough code in parse_csv.py to get our tests to pass.

1
2
3
4
5
6
7
8
9
10
11
12
13

import csv


def read_data(data):
    with open(data, 'r') as f:
        data = [row for row in csv.reader(f.read().splitlines())]
    return data


# ---- run code ---- #

data = "football.csv"
read_data(data)

Run the tests again:

1
2
3
4
5
6
7
8
9

$ python parse_csv_test.py -v
test_csv_read_data_headers (__main__.ParseCSVTest) ... ok
test_csv_read_data_points (__main__.ParseCSVTest) ... ok
test_csv_read_data_team_name (__main__.ParseCSVTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

Test Redux

Back to the test file. This time, we want to find the “smallest difference in ‘for’ and ‘against’ goals”. So, add this import and function:

1
2
3
4
5
6

from parse_csv import read_data, get_min_score_difference

...

def test_get_min_score_difference(self):
    self.assertEqual(get_min_score_difference(parsed_data), "no idea")

Since, we don’t know off hand what the smallest difference will be, we can just add the string “no idea”. You could update that when you know what the answer is after we add the get_min_score_difference() function to parse_csv.py – but how do we know if that answer is correct?

Keep in mind, that since this is a small amount of data, you can get away with calculating them all by hand and finding the smallest difference. Or you could also use test data.:

1
2
3
4
5
6
7

def test_get_min_score_difference(self):
    parsed_data = [
        ['Team', 'Games', 'Wins', 'Losses', 'Draws', 'Goals', 'Goals Allowed', 'Points'],
        ['Arsenal', '38', '26', '9', '3', '79', '36', '87'],
        ['Liverpool', '38', '24', '8', '6', '67', '30', '80']
    ]
    self.assertEqual(get_min_score_difference(parsed_data), '37')