This test should not concern itself with any functionality triggered by the event, only that the event gets triggered. The event should have separate unit tests in which the event is triggered, and then the test verifies it carried out the required actions.

Technically, these don’t quite qualify as being unit tests because they hit the database, but they should cover the controller adequately. To make them true unit tests, you’d need to implement the repository pattern for the database queries rather than using Eloquent directly, and mock the repository, so you can assert that the mocked repository receive the right data and have it return the expected response.

As long as your controllers are kept as small as possible, it’s generally not too hard to test them. Unfortunately, fat controllers become almost impossible to test, which is another good reason to avoid them.

]]>https://matthewdaly.co.uk/blog/2018/01/07/more-tricks-for-speeding-up-your-laravel-test-suite/
https://matthewdaly.co.uk/blog/2018/01/07/more-tricks-for-speeding-up-your-laravel-test-suite/Sun, 07 Jan 2018 16:32:03 GMTWhen you first start doing test-driven development with Laravel, it can be quite hard to produce a test suite that runs quickly enough. The first time I used Laravel for a large project, I had a test suite that at one time, took over seven minutes to run, which was pretty awful considering that the ideal time for a test suite to take to run is no more than ten seconds.

Fortunately, with experience you can pick up some techniques which can quite drastically speed up your test suite. Here are some of the ones I’ve learned that can be useful.

Note that some of these are contradictory, and what works for one use case won’t necessarily work for another, so my advice is to try these and see what makes a difference for your use case.

Reduce the cost of hashing

Inside the createApplication() method of tests\CreatesApplication.php, place the following statement:

Hash::setRounds(4);

This makes hashing passwords quicker and less demanding, and since you don’t care about the security of a password in a test, you’re not losing out in any way by doing so.

This, by itself, can massively reduce the time taken by your test suite - your mileage may vary, but I’ve personally seen it cut to a third of the previous time by using this. In fact, it’s recently been added to Laravel by default.

If you’re creating a lot of fixtures for tests, do so in a transaction

Sometimes, your application requires a lot of data to be added to the database just to be usable, and it’s quite common to use seeders for this purpose. However, it can take some time to insert a lot of data, especially if it has to be re-run for every test. If you do have to insert a lot of data before a test, you can cut down the time substantially by wrapping the seeder calls in a transaction:

I’ve personally seen this trick cut the insert time by over half, every single time the database is seeded. If you don’t have much data to insert, it may not help, but for large amounts of data it can make a big difference.

If a lot of tests need the same data, migrate and seed it first, then wrap the tests in transactions and roll them back afterwards

If multiple tests need to work with the same dataset, you should consider running the migrations and seeders before the first test, and then wrapping each test inside a transaction. That way the data will only be inserted once, and will be rolled back to that initial good state after each test.

Using something like this instead of one of the existing testing traits may be a better fit under those circumstances. However, if your application uses transactions for some functionality this might cause problems.

Don’t create a full instance of the Laravel application unless you have to

Not every test requires that you instantiate the full Laravel application, and doing so slows your tests down. If you don’t absolutely need the full application instantiated in the test, consider having your test inherit from the below simple test case class instead:

Use the new Refresh Database trait

use RefreshDatabase;

This testing trait is generally more efficient than migrating down and up, because it empties the database afterwards rather than stepping through the changes of each migration. If you have a non-trivial number of migrations, it will almost certainly be quicker than migrating down, then back up for the next test.

Mock what you can’t control

You should never, ever be making calls to external APIs in your test suite, because you can’t control whether those external API’s work - if a third-party API goes down, you may get a failed test run even if your application is working perfectly, not to mention it will add the time taken to send the request and receive a response to the test time. Instead, mock the calls to the third-party API.

For large applications, consider moving parts into a separate package

If you have a particularly large application, it’s worth considering moving parts of it out into standalone packages and requiring them using Composer’s support for private Git repositories. That way, those packages can have their own test suites, and the main application’s test suite can cover the remaining functionality.

For instance, it’s fairly straightforward to pull out your models and migrations and put them in a separate package, and the tests for them can go with them to that package.

You should also consider whether parts of your application would be useful as standalone packages, and if so pull them out along with their tests. That way, not only are you making your test suite quicker, but you’re also saving yourself work by creating a reusable solution for a problem you might encounter again in the future.

Turn off XDebug

XDebug has a horrendous effect on the performance of the test suite. Turn it off unless you need it to generate test coverage. Better yet, set up continuous integration and have that generate the coverage for you.

Summary

When you first start using Laravel, it can be hard to keep your test suite lean, and the longer a test suite takes to run, the less likely it is to actually get run regularly. To practice TDD properly, your test suite should not take long enough that your mind starts to wander, and ten seconds is a good target to aim for in this regard - you need to be able to run it several times a minute without problem. Obviously things like having a faster computer or an SSD will help, but there’s a lot you can do to make your test suite more efficient, even when running on a quite basic machine.

]]>https://matthewdaly.co.uk/blog/2017/11/28/building-a-postcode-lookup-client-with-httplug-and-phpspec/
https://matthewdaly.co.uk/blog/2017/11/28/building-a-postcode-lookup-client-with-httplug-and-phpspec/Tue, 28 Nov 2017 11:40:39 GMTWhile PHPUnit is my normal go-to PHP testing framework, for some applications I find PHPSpec superior, in particular REST API clients. I’ve found that it makes for a better flow when doing test-driven development, because it makes it very natural to write a test first, then run it, then make the test pass.

In this tutorial I’ll show you how to build a lookup API client for UK postcodes. In the process of doing so, we’ll use PHPSpec to drive our development process. We’ll also use HTTPlug as our underlying HTTP library. The advantage of this over using something like Guzzle is that we give library users the freedom to choose the HTTP library they feel is most appropriate to their situation.

Background

If you’re unfamiliar with it, the UK postcode system is our equivalent of a zip code in the US, but with two major differences:

The codes themselves are alphanumeric instead of numeric, with the first part including one or two letters usually (but not always) derived from the nearest large town or city (eg L for Liverpool, B for Birmingham, OX for Oxford), or for London, based on the part of the city (eg NW for the north-west of London)

A full postcode is in two parts (eg NW1 8TQ), and the first part narrows the location down to a similar area to a US-style zip code, while the second part usually narrows it down to a street (although sometimes large organisations that receive a lot of mail will have a postcode to themselves).

This means that if you have someone’s postcode and house name or address, you can use those details to look up the rest of their address details. This obviously makes it easier for users to fill out a form, such as when placing an order on an e-commerce site - you can just request those two details and then autofill the rest from them.

Unfortunately, it’s not quite that simple. The data is owned by Royal Mail, and they charge through the nose for access to the raw data, which places this data well outside the budgets of many web app developers. Fortunately, Ideal Postcodes offer a REST API for querying this data. It’s not free, but at 2.5p per request it’s not going to break the bank unless used excessively, and they offer some dummy postcodes that are free to query, which is perfectly fine for testing.

For those of you outside the UK, this may not be of much immediate use, but the underlying principles will still be useful, and you can probably build a similar client for your own nation’s postal code system. For instance, there’s a Zipcode API that those of you in the US can use, and if you understand what’s going on here it shouldn’t be hard to adapt it to work with that. If you do produce a similar client for your country’s postal code system, submit a pull request to update the README with a link to it and I’ll include it.

Note that we don’t install an actual HTTPlug client, other than the mock one, which is only useful for testing. This is deliberate - we’re giving developers working with this library the choice of working with whatever HTTP client they see fit. We do use the Guzzle PSR7 library, but that’s just for the PSR7 library.

Then we install our dependencies:

$ composer install

We also need to tell PHPSpec what our namespace will be. Save this as phpspec.yml:

Don’t forget to update the namespace in both files to whatever you’re using, which should have a vendor name and a package name.

With that done, it’s time to introduce the next component.

Introducing HTTPlug

In the past I’ve usually used either Curl or Guzzle to carry out HTTP requests. However, the problem with this approach is that you’re forcing whoever uses your library to use whatever HTTP client, and whatever version of that client, that you deem appropriate. If they’re also using another library that someone else has written and they made different choices, you could have problems.

HTTPlug is an excellent way of solving this problem. By requiring only an interface and not a concrete implementation, using HTTPlug means that you can specify that the consumer of the library must provide a suitable implementation of that library, but leave the choice of implementation up to them. This means that they can choose whatever implementation best fits their use case. There are adapters for many different clients, so it’s unlikely that they won’t be able to find one that meets their needs.

In addition, HTTPlug provides the means to automatically determine what HTTP client to use, so that if one is not explicitly provided, it can be resolved without any action on the part of the developer. As long as a suitable HTTP adapter is installed, it will be used.

Getting started

One advantage of PHPSpec is that it will automatically generate much of the boilerplate for our client and specs. To create our client spec, run this command:

$ vendor/bin/phpspec desc Matthewbdaly/Postcode/Client
Specification for Matthewbdaly\Postcode\Client created in /home/matthew/Projects/postcode-client/spec/ClientSpec.php.

Now that we have a spec for our client, we can generate the client itself:

Note the use of the let() method here. This lets us specify how the object is constructed, with the beConstructedWith() method. Also, note that $this refers not to the test, but to the object being tested - this takes a bit of getting used to if you’re used to working with PHPUnit.

Also, note that the objects passed through are not actual instances of those objects - instead they are mocks created automatically by PHPSpec. This makes mocking extremely easy, and you can easily set up your own expectations on those mock objects in the test. If you want to use a real object, you can instantiate it in the spec as usual. If we need any other mocks, we can typehint them in our method in exactly the same way.

If we once again use vendor/bin/phpspec run we can now generate a constructor:

Note that we expect $this->setKey('foo') to return $this. This is an example of a fluent interface - by returning an instance of the object, it enables methods to be chained, eg $client->setKey('foo')->get(). Obviously it won’t work for anything that has to return a value, but it’s a useful way of making your classes more intuitive to use.

This test is by far the biggest so far, so it merits some degree of explanation.

Note that we don’t make a real HTTP request against the API. This may sound strange, but bear with me. We have no control whatsoever over that API, and it could in theory become inaccessible or be subject to breaking changes at any time. We also don’t want to be shelling out for a paid service just to test our API client works. All we can do is test that our implementation will send the request we expect it to send - we don’t want our test suite reporting a bug when the API goes down.

We therefore typehint not just the dependencies for the constructor, but a request, response and stream instance. We mock our our responses from those instances using the willReturn() method, so we have complete control over what we pass to our client. That way we can return any appropriate response or throw any exception we deem fit to test the behaviour under those circumstances. For the message factory, we specify what arguments it should receive to create the request, and return our mocked-out request object.

Also, note we use shouldBeLike() to verify the response - this is effectively using the == operator, whereas shouldBe() uses the === operator, making it stricter.

We first build up our URL, before using the message factory to create a request object. We then pass the built request to our client to send, before decoding the response into the format we want.

This should make our tests pass:

$ vendor/bin/phpspec run
100% 4
1 specs
4 examples (4 passed)
307ms

Our client now works, but there are a couple of situations we need to account for. First, the API will raise a 402 if you make a request for a real postcode without having paid. We need to catch this and throw an exception. Add this to spec/ClientSpec.php:

And our API client is feature complete! You can find the source code of the finished client here.

Summary

Personally, I find that while PHPSpec isn’t appropriate for every use case, it’s particularly handy for API clients and it’s generally my go-to testing solution for them. It handles producing a lot of the boilerplate for me, and it results in a much better workflow for test-driven development as it makes it very natural to write the test first, then make it pass.

HTTPlug has been a revelation for me. While it takes a bit of getting used to if you’re used to something like Guzzle, it means that you’re giving consumers of your library the freedom to choose the HTTP client of their choice, meaning they don’t have to fight with several different libraries requiring different versions of Guzzle. It also allows for easy resolution of the HTTP client, rather than having to explicitly pass through an instance when instantiating your client. I’m planning to use it extensively in the future.

]]>https://matthewdaly.co.uk/blog/2017/11/16/creating-custom-assertions-with-phpunit/
https://matthewdaly.co.uk/blog/2017/11/16/creating-custom-assertions-with-phpunit/Thu, 16 Nov 2017 15:15:50 GMTToday I’ve been working on a library I’m building for making it easier to build RESTful API’s with Laravel. It uses an abstract RESTful controller, which inherits from the default Laravel controller, and I wanted to verify that the instantiated controller includes all the traits from the base controller.

However, there was a problem. The only practical way to verify that a class includes a trait is with the class_uses() function, but this doesn’t work if the class inherits from a parent that includes these traits. As the class is abstract, it can’t be instantiated directly, so you must either create a dummy class just for testing that extends it, or mock the class, and that means that class_uses() won’t work. As a result, I needed to first get the parent class, then call class_uses() on that, which is possible, but a bit verbose to do repeatedly for several tests.

Fortunately it’s quite easy to create your own custom assertions in PHPUnit. I started out by setting up the test with the assertion I wanted to have:

Actually implementing the assertion is fairly straightforward. You simply add the assertion as a method on the base test case you’re using. and accept whatever arguments are required, plus a final argument of $message = ''. Then you call self::assertThat(), as demonstrated below:

In this case we’re asserting that the specified trait appears in the list of traits on the parent class. Note the use of self::isTrue() - this just verifies that the response is truthy.

Using this method it’s quite easy to create custom assertions, which can help make your tests less verbose and easier to read.

]]>https://matthewdaly.co.uk/blog/2017/08/19/run-your-tests-locally-with-sismo/
https://matthewdaly.co.uk/blog/2017/08/19/run-your-tests-locally-with-sismo/Sat, 19 Aug 2017 14:40:07 GMTContinuous integration is a veritable boon when working on any large software project. However, the popularity of distributed version control systems like Git over the older, more centralised ones like Subversion means that when you commit your changes, they don’t necessarily get pushed up to a remote repository immediately. While this is a good thing because it means you can commit at any stage without worrying about pushing up changes that break everyone else’s build, it has the downside that the tests aren’t automatically run on every commit, just every push, so if you get sloppy about running your tests before every commit you can more easily get caught out. In addition, a full CI server like Jenkins is a rather large piece of software that you don’t really want to run locally if you can help it, and has a lot of functionality you don’t need.

Sismo is a small, simple continuous integration server, implemented in PHP, that’s ideal for running locally. You can set it up to run your tests on every commit, and it has an easy-to-use web interface. Although it’s a PHP application, there’s no reason why you couldn’t use it to run tests for projects in other languages, and because it’s focused solely on running your test suite without many of the other features of more advanced CI solutions, it’s a good fit for local use. Here I’ll show you how I use it.

Setting up Sismo

Nowadays I don’t generally install a web server on a computer directly, preferring to use Vagrant or the dev server as appropriate, so Sismo generally doesn’t have to coexist with anything else. I normally install PHP7’s FastCGI implementation and Nginx, along with the SQLite bindings (which Sismo needs):

$ sudo apt-get install nginx php7.0-fpm php7.0-sqlite3

Then we can set up our Nginx config at /etc/nginx/sites-available/default:

Hopefully this shouldn’t be too difficult to understand. We create an array of projects, then specify a notifier (this is Linux-specific - refer to the documentation for using Growl on Mac OS). Next, we specify that by default the tests should run composer install followed by vendor/bin/phpunit. We then specify this project is a Github project - it also supports Bitbucket, or plain SSH, or the default Project, but in general it shouldn’t be a problem to use it with any repository as you can just run it against the local copy. Finally we return the list of projects.

That should be working, but it doesn’t get us anything we don’t get by running the tests ourselves. To trigger the build, we need to set up a post-commit hook for our project in .git/hooks/post-commit:

You should now be able to view your project in the Sismo web interface at http://localhost:

Clicking on the project should take you through to its build history:

From here on, it should be straightforward to add new projects as and when necessary. Because you can change the command on a per-project basis, you can quite happily use it to run tests for Python or Node.js projects as well as PHP ones, and it’s not hard to configure it.

I personally find it very useful to have something in place to run my tests on every commit like this, and while you could just use a post-commit hook for that, this approach is less obtrusive because it doesn’t force you to wait around for your test suite to finish.

]]>https://matthewdaly.co.uk/blog/2017/06/17/snapshot-test-your-vue-components-with-jest/
https://matthewdaly.co.uk/blog/2017/06/17/snapshot-test-your-vue-components-with-jest/Sat, 17 Jun 2017 13:12:02 GMTAt work I’ve recently started using Vue as my main front-end framework instead of Angular 1. It has a relatively shallow learning curve and has enough similarities with both React and Angular 1 that if you’re familiar with one or both of them it feels quite familiar. We’re a Laravel shop and Laravel comes out of the box with a basic scaffolding for using Vue, so not only is it the path of least resistance, but many of my colleagues knew it already and it’s used on some existing projects (one of which I’ve been helping out on this week), so it made sense to learn it. Add to that the fact that the main alternative is Angular 2, which I vehemently dislike, and learning Vue was a no-brainer.

Snapshot tests are a really useful way of making sure your user interface doesn’t change unexpectedly. Facebook introduced them to their Jest testing framework last year, and they’ve started to appear in other testing frameworks too. In their words…

A typical snapshot test case for a mobile app renders a UI component, takes a screenshot, then compares it to a reference image stored alongside the test. The test will fail if the two images do not match: either the change is unexpected, or the screenshot needs to be updated to the new version of the UI component.

This makes it easy to make sure than a UI component, such as a React or Vue component, does not unexpectedly change how it is rendered. In the event that it does change, it will fail the test, and it’s up to the developer to confirm whether or not that’s expected - if so they can generate a new version of the snapshot and be on their way. Without it, you’re stuck manually testing that the right HTML tags get generated, which is a chore.

Jest’s documentation is aimed pretty squarely at React, but it’s not hard to adapt it to work with Vue components. Here I’ll show you how I got it working with Vue.

Setting up a new project

I used the Vue CLI boilerplate generator to set up my initial dependencies for this project. I then had to install some further packages:

$ npm install --save-dev jest babel-jest jest-vue-preprocessor

After that, I had to configure Jest to work with Vue. The finished package.json looked like this:

I won’t include things like the Webpack config, because that’s all generated by Vue CLI. Note that we need to tell Jest what file extensions it should work with, including .vue, and we need to specify the appropriate transforms for different types of files. We use jest-vue-preprocessor for .vue files and babel-jest for .js files.

With that done, we can create a basic component. We’ll assume we’re writing a simple issue tracker here, and our first component will be at src/components/Issue.vue:

Constructor is what creates our Vue component, while vm is our actual newly-mounted Vue component. We can refer to the HTML inside the component through vm.$el, so we can then work with the virtual DOM easily.

In the first test we use the more traditional method of verifying our UI component has worked as expected - we fetch an HTML tag inside it and verify that the content inside is what we expect. This is fine for a small component, but as the components get larger we’ll find it more of a chore.

The second test is much simpler and more concise. We simply assert that it matches the snapshot. Not only is that easier, but it can scale to components of any size because we don’t have to check every little element.

Jest is telling us that our snapshot has changed, but if we expect that, we can just run npm test -- -u to replace the existing one with our new one. Then, our tests will pass again.

Now, this component is pretty useless. It doesn’t accept any external input whatsoever, so the response is always going to be the same. How do we test a more dynamic component? Amend the component to look like this:

Jest has picked up on our changes and thrown an error. However, because we know the UI has changed, we’re happy with this situation, so we can tell Jest to replace the prior snapshot with npm test -- -u as mentioned earlier:

Great, we now have a passing test suite again! That’s all we need to make sure that any regressions in the generated HTML of a component get caught.

Of course, this won’t help with the actual functionality of the component. However, Jest is pretty easy to use to write tests for the actual functionality of the application. If you prefer another testing framework, it’s possible to do the same with them, although I will leave setting them up as an exercise for the reader.

]]>https://matthewdaly.co.uk/blog/2017/02/18/integrating-behat-with-laravel/
https://matthewdaly.co.uk/blog/2017/02/18/integrating-behat-with-laravel/Sat, 18 Feb 2017 21:25:57 GMTThe Gherkin format used by tools like Cucumber is a really great way of specifying how your application will work. It’s easy for even non-technical stakeholders to understand, it makes it natural to break your tests into easily reusable steps, and it encourages you to think about the application from an end-user’s perspective. It’s also one of the easiest ways to get started writing automated tests when you first start out - it’s much more intuitive to a junior developer than lower-level unit tests, and is easier to add to a legacy project that may not have been built with testability in mind - if you can drive a browser, you can test it.

Behat is a PHP equivalent. Combined with Mink, it allows for easy automated acceptance tests of a PHP application. However, out of the box it doesn’t integrate well with Laravel. There is Jeffrey Way’s Behat Laravel extension, but it doesn’t seem to be actively maintained and seems to be overkill for this purpose. I wanted something that I could use to run integration tests using PHPUnit’s assertions and Laravel’s testing utilities, and crucially, I wanted to do so as quickly as possible. That meant running a web server and using an automated web browser wasn’t an option. Also, I often work on REST API’s, and browser testing isn’t appropriate for those - in API tests I’m more interested in setting up the fixtures, making a single request, and verifying that it does what it’s meant to do, as quickly as possible.

As it turns out, integrating Behat and Laravel isn’t that hard. When using Behat, your FeatureContext.php file must implement the Behat\Behat\Context\Context interface, but as this interface does not implement any methods, you can extend any existing class and declare that it implements that interface. That means we can just extend the existing Tests\TestCase class in Laravel 5.4 and gain access to all the same testing utilities we have in our regular Laravel tests.

Then, in the constructor we can set environment variables using putenv() so that we can set it up to use an in-memory SQLite database for faster tests. We also use the @BeforeScenario hook to migrate the database before each scenario, and the @AfterScenario hook to roll it back afterwards.

Note that I’ve added a few basic example methods for our tests. As you can see, we can call the same methods we normally use in Laravel tests to make assertions and HTTP requests. If you’re using Dusk, you can also call that in the same way you usually would.

We might then write the following feature file to demonstrate our application at work:

Feature: Login
Background:
Given a user called "Alan" exists
And a user called "Bob" exists
And a user called "Clare" exists
And a user called "Derek" exists
And a user called "Eric" exists
Scenario: Log in as Alan
Given I am logged in as "Alan"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Bob
Given I am logged in as "Bob"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Clare
Given I am logged in as "Clare"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Derek
Given I am logged in as "Derek"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Eric
Given I am logged in as "Eric"
And I visit the path "/"
Then I should see the text "Laravel"

We can then run these tests with vendor/bin/behat:

$ vendor/bin/behat
Feature: Login
Background: # features/auth.feature:3
Given a user called "Alan" exists # FeatureContext::aUserCalledExists()
And a user called "Bob" exists # FeatureContext::aUserCalledExists()
And a user called "Clare" exists # FeatureContext::aUserCalledExists()
And a user called "Derek" exists # FeatureContext::aUserCalledExists()
And a user called "Eric" exists # FeatureContext::aUserCalledExists()
Scenario: Log in as Alan # features/auth.feature:10
Given I am logged in as "Alan" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Bob # features/auth.feature:15
Given I am logged in as "Bob" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Clare # features/auth.feature:20
Given I am logged in as "Clare" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Derek # features/auth.feature:25
Given I am logged in as "Derek" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Eric # features/auth.feature:30
Given I am logged in as "Eric" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
5 scenarios (5 passed)
40 steps (40 passed)
0m0.50s (19.87Mb)

Higher level tests can get very tedious if you’re not careful - you wind up setting up the same fixtures and making the same requests many times over. By using Behat in this way, not only are you writing your tests in a way that is easy to understand, but you’re also breaking it down into logical, repeatable steps, and by passing arguments in each step you limit the amount of repetition. It’s also fast if you aren’t running browser-based tests, making it particularly well-suited to API testing.

]]>https://matthewdaly.co.uk/blog/2016/04/04/writing-faster-laravel-tests/
https://matthewdaly.co.uk/blog/2016/04/04/writing-faster-laravel-tests/Mon, 04 Apr 2016 19:55:15 GMTNowadays, Laravel tends to be my go-to PHP framework, to the point that we use it as our default framework at work. A big part of this is that Laravel is relatively easy to test, making practicing TDD a lot easier.

Out of the box running Laravel tests can be quite slow, which is a big issue - if your test suite takes several minutes to run, that’s a huge disruption. Also, Laravel doesn’t create a dedicated test database - instead it runs the tests against the same database you’re using normally, which is almost always not what you want. I’ll show you how to set up a dedicated test database, and how to use an in-memory SQLite database for faster tests. This results in cleaner and easier-to-maintain tests, since you can be sure the test database is restored to a clean state at the end of every test.

Setup

Our first step is to make sure that when a new test begins, the following should happen:

We should create a new transaction

We should empty and migrate our database

Then, at the end of each test:

We should roll back our transaction to restore the database to its prior state

To do so, we can create custom setUp() and tearDown() methods for our base TestCase class. Save this in tests/TestCase.php:

That takes care of building up and tearing down our database for each test.

EDIT: Turns out there’s actually a much easier way of doing this already included in Laravel. Just import and add either use DatabaseMigrations; or use DatabaseTransactions; to the TestCase class. The first will roll back the database and migrate it again after each test, while the second wraps each test in a transaction.

Using an in-memory SQLite database for testing purposes

It’s not always practical to do this, especially if you rely on database features in PostgreSQL that aren’t available in SQLite, but if it is, it’s probably worth using an in-memory SQLite database for your tests. If you want to do so, here’s some example settings you might want to use in phpunit.xml:

I would still recommend that you test against your production database, but this can be easily handed off to a continuous integration server such as Jenkins, since that way it won’t disrupt your workflow.

During TDD, you’ll typically run your tests several times for any change you make, so if they’re too slow it can have a disastrous effect on your productivity. But with a few simple changes like this, you can ensure your tests run as quickly as possible. This approach should also be viable for Lumen apps.

]]>https://matthewdaly.co.uk/blog/2016/03/26/building-a-location-aware-web-app-with-geodjango/
https://matthewdaly.co.uk/blog/2016/03/26/building-a-location-aware-web-app-with-geodjango/Sat, 26 Mar 2016 21:30:29 GMTPostgreSQL has excellent support for geographical data thanks to the PostGIS extension, and Django allows you to take full advantage of it thanks to GeoDjango. In this tutorial, I’ll show you how to use GeoDjango to build a web app that allows users to search for gigs and events near them.

Requirements

I’ve made the jump to Python 3, and if you haven’t done so yet, I highly recommend it - it’s not hard, and there’s very few modules left that haven’t been ported across. As such, this tutorial assumes you’re using Python 3. You’ll also need to have Git, PostgreSQL and PostGIS installed - I’ll leave the details of doing so up to you as it varies by platform, but you can generally do so easily with a package manager on most Linux distros. On Mac OS X I recommend using Homebrew. If you’re on Windows I think your best bet is probably to use a Vagrant VM.

We’ll be using Django 1.9 - if by the time you read this a newer version of Django is out, it’s quite possible that some things may have changed and you’ll need to work around any problems caused. Generally search engines are the best place to look for this, and I’ll endeavour to keep the resulting Github repository as up to date as I can, so try those if you get stuck.

Getting started

First of all, let’s create our database. Make sure you’re running as a user that has the required privileges to create users and databases for PostgreSQL and run the following command:

$ createdb gigfinder

This creates the database. Next, we create the user:

$ createuser -s giguser -P

You’ll be prompted to enter a password for the new user. Next, we want to use the psql command-line client to interact with our new database:

$ psql gigfinder

This connects to the database. Run these commands to set up access to the database and install the PostGIS extension:

Our first model

At this point, it’s worth thinking about the models we plan for our app to have. First we’ll have a Venue model that contains details of an individual venue, which will include a name and a geographical location. We’ll also have an Event model that will represent an individual gig or event at a venue, and will include a name, date/time and a venue as a foreign key.

Before we start writing our first model, we need to write a test for it, but we also need to be able to create objects easily in our tests. We also want to be able to easily examine our objects, so we’ll install iPDB and Factory Boy:

Setting up the admin

For an application like this, you’d expect the curators of the site to maintain the gigs and venues stored in the database, and that’s an obvious use case for the Django admin. So let’s set our models up to be available in the admin. Open up gigs/admin.py and amend it as follows:

Now, if you start up the dev server as usual with python manage.py runserver and visit http://127.0.0.1:8000/admin/, you can see that our Event and Venue models are now available. However, the string representations of them are pretty useless. Let’s fix that. First, we amend our tests:

Our models are now in place, so you may want to log into the admin and create a few venues and events so you can see it in action. Note that the location field for the Venue model creates a map widget that allows you to select a geographical location. It is a bit basic, however, so let’s make it better. Let’s install django-floppyforms:

$ pip install django-floppyforms

And add it to our requirements:

$ pip freeze -r requirements.txt

Then add it to INSTALLED_APPS in gigfinder/setttings.py:

INSTALLED_APPS = [
...
'django.contrib.gis',
'gigs',
'floppyforms',
]

Now we create a custom point widget for our admin, a custom form for the venues, and a custom venue admin:

Note in particular that we define the media for our widget so we can include some required Javascript. If you run the dev server again, you should see that the map widget in the admin is now provided by Google Maps, making it much easier to identify the correct location of the venue.

With our admin ready, it’s time to move on to the user-facing part of the web app.

Creating our views

We will keep the front end for this app as simple as possible for the purposes of this tutorial, but of course you should feel free to expand upon this as you see fit. What we’ll do is create a form that uses HTML5 geolocation to get the user’s current geographical coordinates. It will then return events in the next week, ordered by how close the venue is. Please note that there are plans afoot in some browsers to prevent HTML5 geolocation from working unless content is server over HTTPS, so that may complicate things.

How do we query the database to get this data? It’s not too difficult, as shown in this example:

I’ve set up a number of venues using the admin, one round the corner, two in Norwich, and one in London. I then imported the models, the Point class, and the Distance function, and created a Point object. Note that the Point is passed three fields - the first and second are the latitude and longitude, respectively, while the srid field takes a value of 4326. This field represents the Spatial Reference System Identifier used for this query - we’ve gone for WGS 84, which is a common choice and is referred to with the SRID 4326.

Now, we want the user to be able to submit the form and get the 5 nearest events in the next week. We can get the date for this time next week as follows:

With that in mind, let’s write the test for our view. The view should contain a single form that accepts a user’s geographical coordinates - for convenience we’ll autocomplete this with HTML5 geolocation. On submit, the user should see a list of the five closest events in the next week.

Note that we’re switching from a View to a FormView so that it can more easily handle our form. We could render the form using this as well, but as it’s a simple form I decided it wasn’t worth the bother. Also, note that the longitude goes first - this caught me out as I expected the latitude to be the first argument.

Now, if we run our tests, they should complain about our missing template:

Here we’re adding the request context so that the CSRF token is available.

If you run the dev server, add a few events and venues via the admin, and submit a search, you’ll see that you’re returning events closest to you first.

Now that we can submit searches, we’re ready to commit:

$ git add gigs/
$ git commit -m 'Can now retrieve search results'

And we’re done! Of course, you may want to expand on this by plotting each gig venue on a map, or something like that, in which case there’s plenty of methods of doing so - you can retrieve the latitude and longitude in the template and use Google Maps to display them. I’ll leave doing so as an exercise for the reader.

I can’t say that working with GeoDjango isn’t a bit of a struggle at times, but being able to make spatial queries in this fashion is very useful. With more and more people carrying smartphones, you’re more likely than ever to be asked to build applications that return data based on someone’s geographical location, and GeoDjango is a great way to do this with a Django application. You can find the source on Github.

]]>https://matthewdaly.co.uk/blog/2016/01/26/mocking-external-apis-in-python/
https://matthewdaly.co.uk/blog/2016/01/26/mocking-external-apis-in-python/Tue, 26 Jan 2016 23:40:25 GMTIt’s quite common to have to integrate an external API into your web app for some of your functionality. However, it’s a really bad idea to have requests be sent to the remote API when running your tests. At best, it means your tests may fail due to unexpected circumstances, such as a network outage. At worst, you could wind up making requests to paid services that will cost you money, or sending push notifications to clients. It’s therefore a good idea to mock these requests in some way, but it can be fiddly.

In this post I’ll show you several ways you can mock an external API so as to prevent requests being sent when running your test suite. I’m sure there are many others, but these have worked for me recently.

Mocking the client library

Nowadays many third-party services realise that providing developers with client libraries in a variety of languages is a good idea, so it’s quite common to find a library for interfacing with a third-party service. Under these circumstances, the library itself is usually already thoroughly tested, so there’s no point in you writing additional tests for that functionality. Instead, you can just mock the client library so that the request is never sent, and if you need a response, then you can specify one that will remain constant.

I recently had to integrate Stripe with a mobile app backend, and I used their client library. I needed to ensure that I got the right result back. In this case I only needed to use the Token object’s create() method. I therefore created a new MockToken class that inherited from Token, and overrode its create() method so that it only accepted one card number and returned a hard-coded response for it:

This replaced stripe.Token with MockToken so that in this test, the response from the client library was always going to be the expected one.

If the response doesn’t matter and all you need to do is be sure that the right method would have been called, this is easier. You can just mock the method in question using MagicMock and assert that it has been called afterwards, as in this example:

Mocking lower-level requests

Sometimes, no client library is available, or it’s not worth using one as you only have to make one or two requests. Under these circumstances, there are ways to mock the actual request to the external API. If you’re using the requests module, then there’s a responses module that’s ideal for mocking the API request.

Note the use of the `@responses.activatedecorator. We useresponses.add()` to set up each URL we want to be able to mock, and pass through details of the response we want to return. We then make the request, and check that it was made as expected.

Summary

I’m pretty certain that there are other ways you can mock an external API in Python, but these ones have worked for me recently. If you use another method, please feel free to share it in the comments.

]]>https://matthewdaly.co.uk/blog/2015/08/02/testing-django-views-in-isolation/
https://matthewdaly.co.uk/blog/2015/08/02/testing-django-views-in-isolation/Sun, 02 Aug 2015 16:58:45 GMTOne thing you may hear said often about test-driven development is that as far as possible, you should test everything in isolation. However, it’s not always immediately clear how you actually go about doing this. In Django, it’s fairly easy to get your head around testing models in isolation because they’re single objects that you can just create, save, and then check their attributes. Forms are also quite easy to test, because you can just set the parameters with the appropriate values and check that the validation works as expected. With views, it’s much harder to imagine how you’d go about testing them in isolation, and often people just settle for writing higher-level functional tests instead. While functional tests are important, they’re also slower than unit tests, which makes it less likely they’ll be run often. So I thought I’d show you a quick and simple example of testing a Django view in isolation.

One of the little projects I’ve written in the past to help get my head around certain aspects of Django is a code-snippet sharing Django application which I named Snippetr. The index route of this application is a form for submitting a brand-new code snippet and I’ll show you how we would write a test for that.

Testing a GET request

Before now, you may well have used the Django test client to test views. That is fine for higher-level tests, but if you want to test a view in isolation, it’s no use because it emulates a real web server and all of the middleware and authentication, which we want to keep out of the way. Instead, we need to use RequestFactory:

from django.test import RequestFactory

RequestFactory actually implements a subset of the functionality of the Django test client, so while it will feel somewhat familiar, it won’t have all the same functionality. For instance, it doesn’t support middleware, so rather than logging in using the test client’s login() method, you instead attach a user directly to the request, as in this example:

request = RequestFactory()
request.user = user

You have to specify the URL in the request, but you also have to explicitly pass the request through to the view you want to test, which can be a bit confusing. Let’s see it in context. First of all, we want to write a test for making a GET request:

First of all, we define a setUp() method that creates a user and an instance of RequestFactory() for use in the test. Note that I’m using Factory Boy to define UserFactory in order to make it easier to work with. Also, if you have more than one view to test, you should create a base class containing the setUp() method that your view tests inherit from.

Next, we have our test for making a GET request. Note that we’re using the reverse() method to get the route for the view named snippet_create. You’ll need to import this as follows if you’re not yet using it:

from django.core.urlresolvers import reverse

We then attach our user object to the request manually, and fetch the response by passing the request to the view as follows:

response = SnippetCreateView.as_view()(request)

Note that this is the syntax used for class-based views - we call the view’s as_view() method. For a function-based view, the syntax is a bit simpler:

response = my_view(request)

We then test our response as usual. In this case, the view adds some additional context data, and we check that we can access that, as well as checking the status code.

Testing a POST request

Testing a POST request is a little more challenging in this case because submitting the form will create a new Snippet object and we don’t want to interact with the model layer at all if we can help it. We want to test the view in isolation, partly because it will be faster, and partly because it’s a good idea. We can do this by mocking the Snippet model’s save() method.

To do so, we need to import two things from the mock library. If you’re using Python 3.4 or later, then mock is part of unittest as unittest.mock. Otherwise, it’s a separate library you need to install with pip. Here’s the import statement for those on Python 3.4 or later:

Here we’re saying that in this test, when the save() method of the Snippet model is called, it should instead call a mocked version, which lacks the functionality and only registers that it has been called and a few details about it.

Next, we put together the data to be passed through and create a POST request for it. As before, we attach the user to the request. We then pass the request through in the same way as for the GET request. We also check that the response code was 302, meaning that the user would be redirected elsewhere after the form was submitted correctly.

Finally, we assert that Snippet.save.called is true. called is a Boolean value, representing whether the method was called or not. We also check the value of Snippet.save.call_count, which is a count of the number of times the method was called - here we check that it’s set to 1.

As you can see, while the request factory is a little harder than the Django test client to figure out, it’s not too difficult once you get the hang of it. By combining it with judicious use of mock, you can easily test your views in isolation, and without having to interact with the database or set up any middleware, these tests will be much faster than those using the Django test client.

]]>https://matthewdaly.co.uk/blog/2014/09/28/django-blog-tutorial-the-next-generation-part-9/
https://matthewdaly.co.uk/blog/2014/09/28/django-blog-tutorial-the-next-generation-part-9/Sun, 28 Sep 2014 19:51:02 GMTYes, I know the eight instalment was meant to be the last one! Within 24 hours of that post going live, Django 1.7 was released, so naturally I’d like to show you how to upgrade to it.

The biggest change is that Django 1.7 introduces its own migration system, which means South is now surplus to requirements. We therefore need to switch from South to Django’s native migrations. Fortunately, this is fairly straightforward.

Next, remove South from INSTALLED_APPS in django_tutorial_blog_ng/settings.py.

You now need to delete all of the numbered migration files in blogengine/migrations/, and the relevant .pyc files, but NOT the directory or the __init__.py file. You can do so with this command on Linux or OS X:

That resolves the error in serving static files, but not the error with the admin. If you run the dev server, you’ll be able to see that the admin actually works fine. The problem is caused by the test client not following redirects in the admin. We can easily run just the admin tests with the following command:

There are two main issues here. The first is that when we try to edit or delete an existing item, or refer to it when creating something else, we can no longer rely on the number representing the primary key being set to 1. So we need to specifically obtain this, rather than hard-coding it to 1. Therefore, whenever we pass through a number to represent an item (with the exception of the site, but including tags, categories and posts), we need to instead fetch its primary key and return it. So, above where we try to delete a post, we replace 1 with str(post.pk). This will solve a lot of the problems. As there’s a lot of them, I won’t go through each one, but you can see the entire class above for reference, and if you’ve followed along so far, you shouldn’t have any problems.

The other issue we need to fix is the login and logout tests. We simply add follow=True to these to ensure that the test client follows the redirects.

I’ll also cover development tools and practices that can make using Django easier. But first there’s a few housekeeping tasks that need doing…

Don’t forget to activate your virtualenv - you should know how to do this off by heart by now!

Upgrading Django

At the time of writing, Django 1.7 is due any day now, but it’s not out yet so I won’t cover it. The biggest change is the addition of a built-in migration system, but switching from South to this is well-documented. When Django 1.7 comes out, it shouldn’t be difficult to upgrade to it - because we have good test coverage, we shouldn’t have much trouble catching errors.

However, Django 1.6.6 was recently released, and we need to upgrade to it. Just enter the following command to upgrade:

$ pip install Django --upgrade

Then add it to your requirements.txt:

$ pip freeze > requirements.txt

Then commit your changes:

$ git add requirements.txt
$ git commit -m 'Upgraded Django version'

Implementing a sitemap

Creating a sitemap for your blog is a good idea - it can be submitted to search engines, so that they can easily find your content. With Django, it’s pretty straightforward too.

First, let’s create a test for our sitemap. Add the following code at the end of tests.py:

Now, let’s implement our sitemap. The sitemap application comes with Django, and needs to be activated in your settings file, under INSTALLED_APPS:

'django.contrib.sitemaps',

Next, let’s think about what content we want to include in the sitemap. We want to index our flat pages and our blog posts, so our sitemap should reflect that. Create a new file at blogengine/sitemap.py and enter the following text:

Under the items function, we check to see if the tag exists. However, under get_object we can see that if the object didn’t exist, it would already have returned a 404 error. We can therefore safely amend items to not check, since that try statement will never fail:

If you open htmlcov/index.html in your browser, you should see that the test coverage is back up to 100%. With that done, it’s time to commit again:

$ git add blogengine/
$ git commit -m 'Fixed gaps in coverage'

Remember, it’s not always possible to achieve 100% test coverage, and you shouldn’t worry too much about it if it’s not possible - it’s possible to ignore code if necessary. However, it’s a good idea to aim for 100%.

Using Fabric for deployment

Next we’ll cover using Fabric, a handy tool for deploying your changes (any pretty much any other task you want to automate). First, you need to install it:

$ pip install Fabric

If you have any problems installing it, you should be able to resolve them via Google - most of them are likely to be absent libraries that Fabric depends upon. Once it’s installed, add it to your requirements.tzt:

Now, all this file does is push our changes to Github (or wherever else your repository is hosted) and to Heroku, and runs your migrations. It’s not a terribly big task anyway, but it’s handy to have it in place. Let’s commit our changes:

There, wasn’t that more convenient? Fabric is much more powerful than this simple demonstration indicates, and can run tasks on remote servers via SSH easily. I recommend you take a look at the documentation to see what else you can do with it. If you’re hosting your site on a VPS, you will probably find Fabric indispensable, as you will need to restart the application every time you push up a new revision.

Tidying up

We want our blog application to play nicely with other Django apps. For instance, say you’re working on a new site that includes a blogging engine. Wouldn’t it make sense to just be able to drop in this blogging engine and have it work immediately? At the moment, some of our URL’s are hard-coded, so we may have problems in doing so. Let’s fix that.

First we’ll amend our tests. Add this at the top of the tests file:

from django.core.urlresolvers import reverse

Next, replace every instance of this:

response = self.client.get('/')

with this:

response = self.client.get(reverse('blogengine:index'))

Then, rewrite the calls to the search route. For instance, this:

response = self.client.get('/search?q=first')

should become this:

response = self.client.get(reverse('blogengine:search') + '?q=first')

I’ll leave changing these as an exercise for the reader, but check the repository if you get stuck.

Debugging Django

There are a number of handy ways to debug Django applications. One of the simplest is to use the Python debugger. To use it, just enter the following lines at the point you want to break at:

import pdb
pdb.set_trace()

Now, whenever that line of code is run, you’ll be dropped into an interactive shell that lets you play around to find out what’s going wrong. However, it doesn’t offer autocompletion, so we’ll install ipdb, which is an improved version:

$ pip install ipdb
$ pip freeze > requirements.txt

Now you can use ipdb in much the same way as you would use pdb:

import ipdb
ipdb.set_trace()

Now, ipdb is very useful, but it isn’t much help for profiling your application. For that you need the Django Debug Toolbar. Run the following commands:

$ pip install django-debug-toolbar
$ pip freeze > requirements.txt

Then add the following line to INSTALLED_APPS in your settings file:

'debug_toolbar',

Then, try running the development server, and you’ll see a toolbar on the right-hand side of the screen that allows you to view some useful data about your page. For instance, you’ll notice a field called SQL - this contains details of the queries carried out when building the page. To actually see the queries carried out, you’ll want to disable caching in your settings file by commenting out all the constants that start with CACHE.

We won’t go into using the toolbar to optimise queries, but using this, you can easily see what queries are being executed on a specific page, how long they take, and the values they return. Sometimes, you may need to optimise a slow query - in this case, Django allows you to drop down to writing raw SQL if necessary.

Note that if you’re running Django in production, you should set DEBUG to False as otherwise it gives rather too much information to potential attackers, and with Django Debug Toolbar installed, that’s even more important.

Please also note that when you disable debug mode, Django no longer handles static files automatically, so you’ll need to run python manage.py collectstatic and commit the staticfiles directory.

Optimising static files

We want our blog to get the best SEO results it can, so making it fast is essential. One of the simplest things you can do is to concatenate and minify static assets such as CSS and JavaScript. There are numerous ways to do this, but I generally use Grunt. Let’s set up a Grunt config to concatenate and minify our CSS and JavaScript.

You’ll need to have Node.js installed on your development machine for this. Then, you need to install the Grunt command-line interface:

$ sudo npm install -g grunt-cli

With that done, we need to create a package.json file. You can create one using the command npm init. Here’s mine:

Memcached

If you frequent (or used to frequent) social media sites like Reddit, Slashdot or Digg, you may be familiar with something called variously the Digg or Slashdot effect, whereby if a page gets submitted to a social media site, and subsequently becomes popular, it can be hit by a huge number of HTTP requests in a very short period of time, slowing it down or even taking the server down completely.

Now, as a general rule of thumb, for most dynamic websites such as blogs, the bottleneck is not the web server or the programming language, but the database. If you have a lot of people hitting the same page over and over again in quick succession, then you’re essentially running the same query over and over again and getting the same result each time, which is expensive in terms of processing power. What you need to be able to do is cache the results of the query in memory for a given period of time so that the number of queries is reduced.

That’s where Memcached comes in. It’s a simple key-value store that allows you to store values in memory for a given period of time so that they can be retrieved without having to query the database. Memcached is a very common choice for caching, and is by far the fastest and most efficient type of cache available for Django. It’s also available on Heroku’s free tier.

Django has a very powerful caching framework that supports numerous types of cache in addition to Memcached, such as:

Database caching

Filesystem caching

Local memory caching

There are also third-party backends for using other caching systems such as Redis.

Now, the cache can be used in a number of different ways. You can cache only certain parts of your site if you wish. However, because our site is heavily content-driven, we should be pretty safe to use the per-site cache, which is the simplest way to set up caching.

In order to set up Memcached, there’s a couple of Python libraries we’ll need. If you want to install them locally, however, you’ll need to install both memcached and libmemcached (on Ubuntu, the packages you need are called memcached and libmemcached-dev) on your development machine. If you don’t want to do this, then just copy and paste these lines into requirements.txt instead:

django-pylibmc-sasl==0.2.4
pylibmc==1.3.0

If you are happy to install these dependencies locally, then run this command once memcached and libmemcached are installed:

$ pip install pylibmc django-pylibmc-sasl

With that done let’s configure Memcached. Open up the settings file and add the following at the bottom:

That’s it! The first section configures the application to use Memcached to cache the content when running on Heroku, and sets some configuration parameters, while the second section tells Django to use the per-site cache in order to cache all the site content.

Now, Heroku doesn’t include Memcached by default - instead it’s available as an add-on called Memcachier. To use add-ons you need to set up a credit card for billing. We will set it up to use the free developer plan, but if you outgrow this you can easily switch to a paid plan. To add Memcachier, run this command:

$ heroku addons:add memcachier:dev

Please note that Memcachier can take a few minutes to get set up, so you may want to leave it a little while between adding it and pushing up your changes. Now we’ll commit our changes:

And that’s all you need to do to set up Memcached. In addition to storing your query results in Memcached, enabling the caching framework in Django will also set various HTTP headers to enable web proxies and browsers to cache content for an appropriate length of time. If you open up your browser’s developer tools and compare the response headers for your homepage on the latest version of the code with the previous version, you’ll notice that a number of additional headers appear, including Cache-Control, Expires and Last-Modified. These tell web browsers and web proxies how often to request the latest version of the HTML document, in order to help you reduce the bandwidth used.

As you can see, for a site like this where you are the only person adding content, it’s really easy to implement caching with Django, and for a blog there’s very little reason not to do it. If you’re not using Heroku and are instead hosting your site on a VPS, then the configuration will be somewhat different - see here for details. You can also find information on using other cache backends on the same page.

That isn’t all you can do to speed up your site. Heroku doesn’t seem to be very good for serving static files, and if your site is attracting a lot of traffic you might want to host your static files elsewhere, such as on Amazon’s S3 service. Doing so is outside the scope of this tutorial, but for that use case, you should check out django-storages.

Clearing the cache automatically

There is one issue with this implementation. As it is right now, if you view the home page, add a post, then reload the page, you may not see the new post immediately because the cache will continue serving the old version until it has expired. That behaviour is less than ideal - we would like the cache to be cleared automatically when a new post gets added so that users will see the new version immediately. That response will still be cached afterwards, so it only means one extra query.

This is the ideal place to introduce signals. Signals are a way to carry out a given action when an event takes place. In our case, we plan to clear the cache when a post is saved (either created or updated).

Note that as we’ll be testing the behaviour of the cache at this point, you’ll need to install Memcached on your local machine, and we’ll need to change the settings to fall back to our local Memcached instance:

This should be fairly self-explanatory. We create one post, and request the index page. We then add a second post, request the index page again, and check for the second post. The test should fail because the cached version is returned, rather than the version in the database.

Now we have a test in place, we can implement a fix. First, add this to the top of your models.py:

This is fairly straightforward. What we’re doing is first defining a function called new_post that is called when a new post is created. We then connect it to the post_save signal. When a post is saved, it calls new_post, which clears the cache, making sure users are seeing the latest and greatest version of your site immediately.

There are a number of signals available, and when you create one, you have access to the created object via the instance parameter. Using signals you can implement all kinds of functionality. For instance, you could implement the functionality to send an email when a new post is published.

If you’re using Travis CI, you’ll also need to update the config file:

Formatting for RSS feeds

Now, we want to offer more than one option for RSS feeds. For instance, if your blog is aggregated on a site such as Planet Python, but you also blog about JavaScript, you may want to be able to provide a feed for posts in the python category only.

If you have written any posts that use any of Markdown’s custom formatting, you may notice that if you load your RSS feed in a reader, it isn’t formatted as Markdown. Let’s fix that. First we’ll amend our test:

Refactoring our tests

Now, before we get into implementing the feed, our tests are a bit verbose. We create a lot of items over and over again - let’s sort that out. Factory Boy is a handy Python module that allows you to create easy-to-use factories for creating objects over and over again in tests. Let’s install it:

Next we’ll create a factory for adding users. Note that the factory name doesn’t have to match the object name, so you can create factories for different types of users. Here we create a factory for authors - you could, for instance, create a separate factory for subscribers if you wanted:

This factory is a little bit different. Because our Post model depends on several others, we need to be able to create those additional objects on demand. By designating them as subfactories, we can easily create the associated objects for our Post object.

That means that not only can we get rid of our Post() calls, but we can also get rid of the factory calls to create the associated objects for Post models. Again, I’ll leave actually doing this as an exercise for the reader, but you can always refer to the GitHub repository if you’re not too sure.

Using Factory Boy made a big difference to the size of the test file - I was able to cut it down by over 200 lines of code. As your application gets bigger, it gets harder to maintain, so do what you can to keep the size down.

Here we create two posts in different categories (note that we create a new category and override the post category for it). We then fetch /feeds/posts/category/python/ and assert that it contains only one post, with the content of the first post and not the content of the second.

Note that the category RSS feed route is similar to the post RSS feed route, but accepts a slug parameter. We will use this to pass through the slug for the category in question. Also note we import the CategoryPostsFeed view. Now, we need to create that view. Fortunately, because it’s written as a Python class, we can extend the existing PostsFeed class. Open up your views file and amend it to look like this:

Note that many of our fields don’t have to be explicitly defined as they are inherited from PostsFeed. We can’t hard-code the title, link or description because they depend on the category, so we instead define methods to return the appropriate text.

Also note get_object() - we define this so that we can ensure the category exists. If it doesn’t exist, then it returns a 404 error rather than showing an empty feed.

We also override items() to filter it to just those posts that are in the given category.

Now, we can get our category RSS feed, but how do we navigate to it? Let’s add a link to each category page that directs a user to its RSS feed. To do so, we’ll need to create a new template for category pages. First, let’s add some code to our tests to ensure that the right template is used at all times. Add the following to the end of the test_index method of PostViewTest:

# Check the correct template was used
self.assertTemplateUsed(response, 'blogengine/post_list.html')

Then, add this to the end of test_post_page:

# Check the correct template was used
self.assertTemplateUsed(response, 'blogengine/post_detail.html')

Finally, add this to the end of test_category_page:

# Check the correct template was used
self.assertTemplateUsed(response, 'blogengine/category_post_list.html')

These assertions confirm which template was used to generate which request.

Note that we first of all change the template used by this view. Then, we override get_context_data to add in additional data. What we’re doing is getting the slug that was passed through, looking up any category for which it is the slug, and returning it as additional context data. Using this method, you can easily add additional data that you may wish to render in your Django templates.

Moving our templates

Before we crack on with implementing search, there’s one more piece of housekeeping. In Django, templates can be applied at project level or at app level. So far, we’ve been storing them in the project, but we would like our app to be as self-contained as possible so it can just be dropped into future projects where we need a blog. That way, it can be easily overridden for specific projects. You can move the folders and update the Git repository at the same time with this command:

Note that git mv updates Git and moves the files, so you don’t need to call git add.

Implementing search

For our final task today, we will be implementing a very simple search engine. Our requirements are:

It should be in the header, to allow for easy access from anywhere in the front end.

It should search the title and text of posts.

First, we’ll write our tests:

class SearchViewTest(BaseAcceptanceTest):
def test_search(self):
# Create a post
post = PostFactory()
# Create another post
post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post')
# Search for first post
response = self.client.get('/search?q=first')
self.assertEquals(response.status_code, 200)
# Check the first post is contained in the results
self.assertTrue('My first post' in response.content)
# Check the second post is not contained in the results
self.assertTrue('My second post' not in response.content)
# Search for second post
response = self.client.get('/search?q=second')
self.assertEquals(response.status_code, 200)
# Check the first post is not contained in the results
self.assertTrue('My first post' not in response.content)
# Check the second post is contained in the results
self.assertTrue('My second post' in response.content)

Now we’ll actually implement our search. Implementing search using Django’s generic views can be fiddly, so we’ll write our search view as a function instead. First, amend the imports at the top of your view file to look like this:

As this is the first time we’ve written a view without using generic views, a little explanation is called for. First we get the values of the q and page parameters passed to the view. q contains the query text and page contains the page number. Note also that our page defaults to 1 if not set.

We then use the Q object to perform a query. The Django ORM will AND together keyword argument queries, but that’s not the behaviour we want here. Instead we want to be able to search for content in the title or text, so we need to use a query with an OR statement, which necessitates using the Q object.

Next, we use the Paginator object to manually paginate the results, and if it raises an EmptyPage exception, to just show the last page instead. Finally we render the template blogengine/search_post_list.html, and pass through the parameters page_obj for the returned page, object_list for the objects, and search for the query.

Don’t forget to do a quick sense check to make sure it’s all working as expected. Then it’s time to commit:

$ git add blogengine/
$ git commit -m 'Implemented search'

And push up your changes:

$ git push origin master
$ git push heroku master

And that’s the end of this instalment. Please note this particular search solution is quite basic, and if you want something more powerful, you may want to look at Haystack.

As usual, you can get this lesson with git checkout lesson-7 - if you have any problems, the repository should be the first place you look for answers as this is the working code base for the application, and with judicious use of a tool like diff, it’s generally pretty easy to track down most issues.

Apologies, but I’m holding over implementing the search and additional feeds for a future instalment - in the past I’ve tried to cover too much in one post and that has led to me putting them off for much too long. So this instalment and future ones are going to be shorter so I can get them out the door quicker.

Ready? Let’s get started!

Fixing bugs

When someone reports a bug, it’s tempting to just dive straight in and start fixing it. But TDD cautions us against this practice. If we have a bug, then our tests should have caught it. If they don’t, then before we fix the bug, we have to make sure we can catch it if it crops up in future by implementing a test for it, and ensuring that it fails. Once that test is in place, we can then go ahead and fix the bug, safe in the knowledge that if it should reappear in future, we will be warned when our tests run.

As it happens, we have a bug in our web app. If you activate your virtualenv in the usual way and run the development server, and then try to create a new post without adding a tag, you’ll see that it fails as the tag is empty. Now, it’s pretty obvious that this is because the tags attribute of the Post model cannot be blank, so we have a good idea of what we need to do to fix this. But to make sure it never occurs again, we need to implement a test first.

If you compare category and tags, you’ll immediately see that category has the additional parameters blank and null both set to True. So that’s what we need to do for tags as well. Amend the model to look like this:

Remember: Always make the effort to create a test to reproduce your bug before fixing it. That way, you know you can catch it in future.

Syntax highlighting

This is one feature that not everyone will want to implement. If you want to be able to show code snippets on your blog, then implementing syntax highlighting is well worth your time. However, if that’s not what you want to use your blog for, feel free to skip over this section.

Now, earlier in the series we implemented Markdown support. In Markdown there are two main ways to denote a code block. One is to indent the code by four spaces, while the other is to use fenced code blocks with syntax highlighting. There are many flavours of Markdown available for Python, and unfortunately the one we’ve been using so far doesn’t support fenced code blocks, so we’ll be switching to one that does.

We also need to be able to generate a suitable stylesheet to highlight the code appropriately. For that, we’ll be using Pygments. So let’s first uninstall our existing implementation of Markdown:

$ pip uninstall Markdown

And install the new modules:

$ pip install markdown2 Pygments

Don’t forget to record the changes:

$ pip freeze > requirements.txt

Now, we need to amend our Markdown template tags to use the new version of Markdown:

All we do here is change the Markdown module that gets imported, and amend how it is called. Note that we pass through the parameter fenced-code-blocks to enable this functionality.

If you now run the development server and create a post with some code in it (just copy the Ruby example from here), then view it on the site, you should be able to see that it’s in a <code> block. However, it’s not highlighted yet. Let’s commit our changes:

Now, if you examine the markup for your code blocks using your browser’s developer tools, you’ll notice that the code is wrapped in many different spans with various classes. Pygments can generate a CSS file that uses those classes to highlight your code.

First, let’s create a folder to store our CSS files in:

$ mkdir blogengine/static/css

Next, we’ll add a blank CSS file for any other styling we may want to apply:

Now, if you run the development server and reload the page, your code will be highlighted using the default Pygments style. If you don’t like it, there are plenty to choose from. Run the following command:

$ pygmentize -L styles

That will list the various styles available. For instance, let’s say we want to try the Tango style:

$ pygmentize -S tango -f html > blogengine/static/css/code.css

If you like the Monokai theme in Sublime Text, there’s a Pygments version of that:

$ pygmentize -S monokai -f html > blogengine/static/css/code.css

If you like the Solarized theme, that’s not bundled with Pygments, but can be installed separately:

Pick one that you like - I’m going to go for the dark version of Solarized.

Note that this doesn’t actually change the background colour of the code blocks. You will therefore need to set this manually using the CSS file we created earlier. If you’re using Solarized Dark like I am, then this should set it correctly:

div.codehilite pre {
background-color: #002b36;
}

If you’re using Solarized Light, then this should be more appropriate:

Whoops! We introduced an error here. If we take a look, we can see that the problem is on line 7, where we import the Markdown module. That makes sense, as we now use a different implementation of Markdown. Fortunately, in Python you can import modules with a different name, which makes this a breeze to fix. Change the line to:

import markdown2 as markdown

Now, if you run your tests, they should pass. It’s important that if a test breaks, you fix it as soon as possible and don’t put it off. Let’s commit these changes:

$ git add blogengine/tests.py
$ git commit -m 'Fixed a broken test'

Our syntax highlighting is now done! If you want to see it in action using the Solarized Dark theme, check out the copy of this blogging engine hosted here.

Tidying up

Now that our blog is up and running on Heroku, it could do with a bit of work on the front end to make it look a bit nicer. If you recall, we’re using Bootstrap for our front end, so you may want to refer to the documentation for that to give you some ideas on how you want to style your blog.

Bootstrap has a nifty pager class for Next and Previous links, so let’s apply that to our post list template:

And we’re done! Don’t forget, you can grab this lesson with git checkout lesson-6.

Next time, we’ll cover search (I promise!).

]]>https://matthewdaly.co.uk/blog/2014/05/24/django-blog-tutorial-the-next-generation-part-5/
https://matthewdaly.co.uk/blog/2014/05/24/django-blog-tutorial-the-next-generation-part-5/Sat, 24 May 2014 19:15:54 GMTHello again! I was originally planning to cover implementing a search system, adding more feeds, and tidying up the front end in this instalment. However, I felt it was time for a change of pace, so instead, we’re going to look at:

Checking code coverage and getting it to 100%

Using continuous integration

Deploying to Heroku

Don’t worry, the original lesson isn’t going anywhere. We’ll still be implementing all of that later on, but today is the day we get your Django blog up and running on the web. That way, you can get a better idea of how Django works in the wild, and you have something concrete to show for your efforts.

Continuous integration

If you’re not familiar with continuous integration, it’s basically a process that carries out some tasks automatically when you push a new commit to the repository. These tasks may include running unit tests, linting your code, and checking it to see what percentage of the code base is covered by the tests (after all, if your tests don’t actually cover every scenario, then there’s more of a chance that something might slip through the net. It’s also possible to implement hooks to automatically deploy your application only if the tests pass.

Typically, you will have a continuous integration server running somewhere that regularly polls your Git repository for changes, and when it finds a new commit, will check it out and run the tests (or whatever other task you configure it to). One of the most popular continuous integration servers around is Jenkins - I use it at work and can highly recommend it. However, we aren’t going to cover using Jenkins here because setting it up is quite a big deal and it really is best kept on a server of its own (although feel free to use it if you prefer). Instead, we’re going to use Travis CI, which integrates nicely with GitHub and is free for open source projects. If you don’t mind your code being publicly available on GitHub, then Travis is a really great way to dip your toe into continuous integration.

NB: You don’t have to use continuous integration at all if you don’t want to - this isn’t a big project so you don’t really need it. If you don’t want to put your code on GitHub, then feel free to just follow along and not bother pushing your code to GitHub and configuring Travis.

Code coverage

As mentioned above, code coverage is a measure of the percentage of the code that is covered by tests. While not infallible, it’s a fairly good guide to how comprehensive your tests are. If you have 100% code coverage, you can be fairly confident that your tests are comprehensive enough to catch most errors, so it’s a good rule of thumb to aim for 100% test coverage on a project.

So how do we check our test coverage? The coverage Python module is the most common tool for this. There’s also a handy Django module called django-jenkins, which is designed to work with Jenkins, but can be used with any continuous integration server, that can not only run your tests, but also check code coverage at the same time. So, make sure your virtualenv is up and running:

$ source venv/bin/activate

Then, run the following command:

$ pip install coverage django-jenkins

Once that’s done, add these to our requirements file:

$ pip freeze > requirements.txt

We now need to configure our Django project to use django-jenkins. Add the following to the bottom of the settings file:

This adds django-jenkins to our installed apps and tells it to include two additional tasks, besides running the tests. The first task runs Pylint to check our code quality (but we aren’t really concerned about that at this point). The second checks the coverage. Finally, we tell django-jenkins that the blogengine app is the only one to be tested.

You’ll also want to add the following lines to your .gitignore:

reports/
htmlcov/

These are the reports generated by django-jenkins, and should not be kept under version control. With that done, it’s time to commit:

Now, let’s run our tests. From now on, you’ll use the following command to run your tests:

$ python manage.py jenkins

This ensures we check the coverage at the same time. Now, you’ll notice that the reports folder has been created, and it will contain three files, including one called coverage.xml. However, XML isn’t a very friendly format. Happily, we can easily generate reports in HTML instead:

$ python manage.py jenkins --coverage-html-report=htmlcov

Running this command will create another folder called htmlcov/, and in here you will find your report, nicely formatted as HTML. Open up index.html in your web browser and you should see a file-by-file breakdown of your code coverage. Nice, huh?

Now, if your code so far is largely identical to mine, you’ll notice that the model and view files don’t have 100% coverage yet. If you click on each one, you’ll see a handy line-by-line breakdown of the test coverage for each file. You’ll notice that in the views file, the areas of the code for when tags and categories don’t exist are highlighted in pink - this tells you that these lines of code are never executed during the tests. So let’s fix that.

Now, our models still don’t have 100% coverage yet. If you look at the breakdown for models.py, you’ll see that the line if not self.slug: has only partial coverage, because we missed out setting a slug for the categories and tags in our test. So, let’s fix that. In PostTest(), amend the test_create_category() and test_create_tag() methods as follows:

Then refresh your coverage page, and we should have hit 100% coverage. Excellent news! This means that we can be confident that if any problems get introduced in future, we can pick them up easily. To demonstrate this, let’s upgrade our Django install to the latest version and check everything still works as expected:

$ pip install Django --upgrade

This will upgrade the copy of Django in our virtualenv to the latest version. Then we can run our tests again:

$ python manage.py jenkins --coverage-html-report=htmlcov

Our tests should still pass, indicating that the upgrade to our Django version does not appear to have broken any functionality. Let’s update our requirements.txt:

Preparing our web app for deployment to Heroku

As mentioned previously, I’m going to assume you plan to deploy your site on Heroku. It has good Django support, and you can quite happily host a blog there using their free tariff. If you’d prefer to use another hosting provider, then you should be able to adapt these instructions accordingly.

Now, so far we’ve used SQLite as our database for development purposes. However, SQLite isn’t really suitable for production purposes. Heroku provide a Postgresql database for each web app, so we will use that. To configure it, open up settings.py and amend the database configuration section to look like this:

A little explanation is called for. Remember when we first started out, we installed the django-toolbelt package, which included dj-database-url? Well, here we use the dj_database_url module to get the database from an environment variable set on Heroku. We set a default value so as to fall back to SQLite when that variable is not set. The other settings are required by Heroku.

You may want to run your tests again to ensure everything is still working before committing:

Setting up Continuous Integration and coverage

Now, as mentioned previously, I’ll be demonstrating how to set up Travis CI for our project. Travis CI is only free for open-source projects, so if you don’t want to make your code publicly accessible, you may want to use Jenkins instead. I’ll leave setting that up and running it as an exercise for the reader, but I recommend a plugin called Cobertura which allows you to publish details of your code’s test coverage.

Unfortunately, Travis CI doesn’t have the capability to publish your coverage results. Fortunately, you can use Coveralls.io to pick up the slack. Like Travis, it’s free for open-source projects, and if you host your code on GitHub, it’s pretty easy to use.

You can find instructions on setting up a project on Travis CI here. Once you’ve configured it, you’ll need to set up your .travis.yml file. This is a simple text file that tells Travis CI how to run your tests. Put the following content in the file:

Now, this tells Travis that this is a Python application, and we should be using Python 2.7. Please note you can test against multiple versions of Python if you wish - just add another line with the Python version you want to test against.

Then, we see that we use Pip to install our requirements. Finally we run our tests with Coverage, in order to generate coverage data, and afterwards call coveralls to pass the coverage data to Coveralls.io.

With that done, assuming you have Travis CI and Coveralls configured, and your code is already hosted on GitHub, then you should be able to just push your code up to trigger the build:

$ git push origin master

If you keep an eye on Travis in your browser, you can watch what happens as your tests are run. If for any reason the build fails, then it shouldn’t be too hard to figure out what has gone wrong using the documentation for Travis and Coveralls - both services are pretty easy to use.

Congratulations - you’re now using Continuous Integration! That wasn’t so hard, was it? Now, every time you push to GitHub, Travis will run your tests, and you’ll get an email if they fail, giving you early warning of any problems, and you can check your coverage at the same time. Both Travis and Coveralls offer badges that you can place in your README file on GitHub to show off your coverage and build status - feel free to add these to your repo.

You may want to try making a change that breaks your tests and committing it, then pushing it up, so that you can see what happens when the build breaks.

Deploying to Heroku

Our final task today is deploying our blog to Heroku so we can see it in action. First of all, if you don’t already have an account with Heroku, you’ll need to sign up here. You should also be prompted to install the Heroku toolbelt. Once that’s done, run the following command:

$ heroku login

You’ll be prompted for your credentials - enter these and you should be able to log in successfully.

Now, in order to run our Django app on Heroku, we’ll need to add a Procfile to tell Heroku what command to run in order to start your app. In this case, we will be using Gunicorn as our web server. Assuming our project is called django_tutorial_blog_ng, this is what you need to put in this file:

web: gunicorn django_tutorial_blog_ng.wsgi

That tells Heroku that the file we need to run for this application is django_tutorial_blog_ng/wsgi.py. To test it, run the following command:

$ foreman start

That will start our web server on port 5000, using Gunicorn rather than the Django development server. You should be able to see it in action here, but you’ll notice a very serious issue - namely that the static files aren’t being served. Now, Django has a command called collectstatic that collects all the static files and drops them into one convenient folder. Heroku will run this command automatically, so we need to ensure our static files are available. Amend your .gitignore file so it no longer excludes our static files:

Deployment

Every Heroku app needs a unique name. If you don’t specify one, then Heroku will generate one for you. Your app will have the domain name appname.herokuapp.com - however, if you already have a domain name lined up for your blog, you can point that domain name at the Heroku app if you wish. I’m going to deploy mine with the name blog-shellshocked-info.

You also need to consider where you want to deploy it. Heroku has two regions - North America and EU. By default it will deploy to North America, but as I’m in the EU, that’s where I want to deploy my app to - obviously if you’re in the Americas, you may be better off sticking with North America.

So, let’s create our app. Here’s the command you need to run:

$ heroku apps:create blog-shellshocked-info --region eu

You will want to change the app name, or remove it entirely if you’re happy for Heroku to generate one for you. If you want to host it in North America, drop the --region eu section.

Once completed, this will have created your new app, and added a Git remote for it, but will not have deployed it. To deploy your app, run this command:

$ git push heroku master

That will push your code up to Heroku. Please note that building the app may take a little while. Once it’s done, you can run heroku open to open it in your web browser. You should see an error message stating that the relation blogengine_post does not exist. That’s because we need to create our database structure. Heroku allows you to easily run commands on your app with heroku run, so let’s create our database and run our migrations:

These are exactly the same commands you would run locally to create your database, but prefaced with heroku run so that they get run by Heroku. As usual, you will be prompted to create a superuser - you’ll want to do this so you can log into the admin.

That’s all for today! We’ve finally got our site up and running on Heroku, and set up continuous integration so our tests will get run for us. You can see an example of the site working here. As usual, you can check out the latest version of the code with git checkout lesson-5. If you’d like a homework assignment, then take a look at automating deployment to Heroku on successful builds and see if you can get it set up successfully.

Next time around, we’ll get back to implementing our search and tidying up the front end. See you then!

]]>https://matthewdaly.co.uk/blog/2014/02/15/django-blog-tutorial-the-next-generation-part-4/
https://matthewdaly.co.uk/blog/2014/02/15/django-blog-tutorial-the-next-generation-part-4/Sat, 15 Feb 2014 17:45:00 GMTHello again! As promised, in this instalment we’ll implement categories and tags, as well as an RSS feed.

As usual, we need to switch into our virtualenv:

$ source venv/bin/activate

Categories

It’s worth taking a little time at this point to set out what we mean by categories and tags in this case, as the two can be very similar. In this case, we’ll use the following criteria:

A post can have only one category, or none, but a category can be applied to any number of posts

A post can have any number of tags, and a tag can be applied to any number of posts

If you’re not too familiar with relational database theory, the significance of this may not be apparent, so here’s a quick explanation. Because the categories are limited to one per post, the relationship between a post and a category is known as one-to-many. In other words, one post can only have one category, but one category can have many posts. You can therefore define the categories in one table in your database, and refer to them by their ID (the reference to the category in the post table is referred to as a foreign key).

As usual, we will write a test first. Open up blogengine/tests.py and edit the line importing the Post model as follows:

This test checks that we can create categories. But categories aren’t much use unless we can actually apply them to our posts. So we need to edit our Post model as well, and to do so we need to have a test in place first. Edit the test_create_post method as follows:

This is very similar to the prior code for the posts, and just checks we can create categories via the admin. We also need to check we can apply these categories to posts, and that they don’t break the existing tests:

Note that we add Category before Post - this is because Category is a foreign key in Post, and must be defined in order to be used. Also, note that we add the category attribute as a ForeignKey field, like User and Site, indicating that it is an item in another table being references.

We also allow for category to be blank or null, so the user does not have to apply a category if they don’t wish to.

All we do here is assert that for both the post pages and the index, the text from the category name is shown in the response. We also need to check the category-specific route works. Add this method to PostViewTest:

Let’s take a look at why they failed. test_category_page failed because the Category object had no method get_absolute_url. So we need to implement one. To do so, we really need to add a slug field, like the posts already have. Ideally, we want this to be populated automatically, but with the option to create one manually. So, edit the models as follows:

We’re adding the slug attribute to the Category model here. However, we’re also overriding the save method to detect if the slug is set, and if not, to create a slug using the slugify function, and set it as the category’s slug. We also define an absolute URL for the category.

Now, if you run the tests, they will fail because we haven’t made the changes to the database. So, we use South again:

$ python manage.py schemamigration --auto blogengine

Then run the migration:

$ python manage.py migrate

Now, running our tests will show that the tables are in place, but we still have some work to do. The index and post pages don’t show our categories, so we’ll fix that. First, we’ll fix our post list:

Now, we should only have one failing test outstanding - the category page. For this, generic views aren’t sufficient as we need to limit the queryset to only show those posts with a specific category. Fortunately, we can extend Django’s generic views to add this functionality. First, we edit our URLconfs:

This is quite simple. We import the ListView, as well as our models. Then we extend ListView by getting the slug from the request, fetching the appropriate category, and returning only those posts that have that category. If the category does not exist, we return the empty Post object list. We haven’t had to set the template manually as it is inherited from ListView.

Tags

Tags are fairly similar to categories, but more complex. The relationship they have is called many-to-many - in other words, a tag can be applied to many posts, and one post can have many tags. This is more difficult to model with a relational database. The usual way to do so is to create an intermediate table between the posts and tags, to identify mappings between the two. Fortunately, Django makes this quite easy.

Let’s write the tests for our tagging system. As with the categories, we’ll write the tests for creating and editing them first, and add in tests for them being visible later. First we’ll create a test for creating a new tag object:

Note the difference in how we apply the tags. Because a post can have more than one tag, we can’t just define post.tag in the same way. Instead, we have post.tags, which you can think of as a list, and we use the add method to add a new tag. Note also that the post must already exist before we can add a tag.

We also need to create acceptance tests for creating, editing and deleting tags:

These tests are virtually identical to those for the Category objects, as we plan for our Tag objects to be very similar. Finally, we need to amend the acceptance tests for Post objects to include a tag:

Note that tags is a ManyToManyField, and we pass through the model we wish to use, much like we did with the categories. The difference is that one tag can be applied to many posts and a post can have many tags, so we need an intermediate database table to handle the relationship between the two. With Django’s ORM we can handle this quickly and easily.

Now, like with the categories beforehand, we want to be able to show the tags applied to a post at the base of it, and list all posts for a specific tag. So, first of all, we’ll amend our PostViewTest class to check for the tags:

We create a tag near the top, and check for the text in the page (note that to avoid false positives from the categories, we set the name of the tags to something different). We do this on both the index and post pages.

RSS Feed

For the final task today, we’ll implement an RSS feed for our posts. Django ships with a handy syndication framework that makes it easy to implement this kind of functionality.

As usual, we’ll create some tests first. In this case, we won’t be adding any new models, so we don’t need to test them. Instead we can jump straight into creating acceptance tests for our feed. For now we’ll just create one type of feed: a feed of all the blog posts. In a later instalment we’ll add feeds for categories and tags.

First of all, we’ll implement our test. Now, in order to test our feed, we need to have a solution in place for parsing an RSS feed. Django won’t do this natively, so we’ll install the feedparser Python module. Run the following commands:

$ pip install feedparser
$ pip freeze > requirements.txt

Once that’s done, feedparser should be available. You may wish to refer to the documentation as we go to help.

Let’s write our test for the RSS feed. First, we import feedparser near the top of the file:

import feedparser

Then we define a new class for our feed tests. Put this towards the end of the file - I put it just before the flat page tests:

We’re getting a 404 error because the post feed isn’t implemented. So let’s implement it. We’re going to use Django’s syndication framework, which will make it easy, but we need to enable it. Open up django_tutorial_blog_ng/settings/py and add the following under INSTALLED_APPS:

'django.contrib.syndication',

Next, we need to enable the URLconf for this RSS feed. Open up blogengine/urls.py and amend the import fromblogengine.views` near the top:

from blogengine.views import CategoryListView, TagListView, PostsFeed

Further down, add in the following code to define the URL for the feed:

# Post RSS feed
url(r'^feeds/posts/$', PostsFeed()),

Note that we imported the PostsFeed class, but that hasn’t yet been defined. So we need to do that. First of all, add this line near the top:

from django.contrib.syndication.views import Feed

This imports the syndication views - yes, they’re another generic view! Our PostsFeed class is going to inherit from Feed. Next, we define the class:

This is fairly straightforward. We define our title, link, and description for the feed inside the class definition. We define the items method which sets what items are returned by the RSS feed. We also define the item_title and item_description methods.

Flat pages

Django ships with a number of useful apps - we’ve already used the admin interface. The flat pages app is another very handy app that comes with Django, and we’ll use it to allow the blog author to create a handful of flat pages.

First of all, you’ll need to install the flatpages app. Edit the INSTALLED_APPS setting as follows:

As mentioned previously, all models in Django have an id attribute by default. Each flat page also has a URL, title, and content.

Also note the separate django_flatpage_sites table, which maps sites to flat pages. Django can run multiple sites from the same web app, and so flat pages must be allocated to a specific site. This relationship is a many-to-many relationship, so one flat page can appear on more than one site.

The other fields are hidden by default in the admin and can be ignored. Let’s have a go with Django’s handy shell to explore the flatpage. Run python manage.py shell and you’ll be able to interact with your Django application interactively:

As you can see, flatpages is a Django app similar to the blogengine one, with its own models, as is sites. You can see that the FlatPage class is a model. We can create an instance of it and save it interactively:

This command is often handy for debugging problems with your models interactively. If you now run the server and visit the admin, you should notice that the Flatpages app is now visible there, and the ‘About me’ flat page is now shown in there.

Let’s also take a look at the SQL required for the Site model. Run python manage.py sqlall sites:

So, now that we have a good idea of how the flat page system works, we can write a test for it. We don’t need to write unit tests for the model because Django already does that, but we do need to write an acceptance test to ensure we can create flat pages and they will be where we expect them to be. Add the following to the top of the test file:

from django.contrib.flatpages.models import FlatPage
from django.contrib.sites.models import Site

Now, before we write this test, there’s some duplication to resolve. We have two tests that subclass LiveServerTestCase, and both have the same method, setUp. We can save ourselves some hassle by creating a new class containing this method and having both these tests inherit from it. We’ll do that now because the flat page test can also be based on it. Create the following class just after PostTest:

This template is based on the blog post one, and just changes a handful of variable names. Note that it can still inherit from the blogengine base template, and in this case we’re using that for the sake of consistency.

If you run your tests, you should now see that they pass, so we’ll commit our changes:

Here we create a User object to represent the author. Note the create_user convenience method for creating new users quickly and easily.

We’re going to exclude the author field from the admin - instead it’s going to be automatically populated based on the session data, so that when a user creates a post they are automatically set as the author. We therefore don’t need to make any changes for the acceptance tests for posts - our changes to the unit tests for the Post model are sufficient.

Now, if you want to use the Django comment system, you can do so, and it shouldn’t be too hard to puzzle out how to implement it using the documentation and my prior post. However, in my humble opinion, using a third-party comment system is the way to go for blog comments - they make it extremely easy for people to log in with multiple services without you having to write lots of additional code. They also make it significantly easier to moderate comments, and they’re generally pretty good at handling comment spam.

Now, the Facebook comment system requires that you pass through the absolute page URL when initialising the comments. At present we can’t do that without hard-coding the domain name in our template, which we want to avoid. So, we need to add a site field to each post to identify the site it’s associated with.

All we’ve done here is to add the site attribute when creating a new post using the Django database API, and when we create one via the admin, we add an additional site aparameter to the HTTP POST request with a value of 1. Run the tests and they should fail:

Now create and run the migrations - you’ll be prompted to create a default value for the site attribute as well:

(venv)Smith:django_tutorial_blog_ng matthewdaly$ python manage.py schemamigration --auto blogengine
? The field 'Post.site' does not have a default specified, yet is NOT NULL.
? Since you are adding this field, you MUST specify a default
? value to use for existing rows. Would you like to:
? 1. Quit now, and add a default to the field in models.py
? 2. Specify a one-off value to use for existing columns now
? Please select a choice: 2
? Please enter Python code for your one-off default value.
? The datetime module is available, so you can do e.g. datetime.date.today()
>>> 1
+ Added field site on blogengine.Post
Created 0005_auto__add_field_post_site.py. You can now apply this migration with: ./manage.py migrate blogengine
(venv)Smith:django_tutorial_blog_ng matthewdaly$ python manage.py migrate
Running migrations for blogengine:
- Migrating forwards to 0005_auto__add_field_post_site.
> blogengine:0005_auto__add_field_post_site
- Loading initial data for blogengine.
Installed 0 object(s) from 0 fixture(s)

Now, before we get started, don’t forget to switch into your virtualenv. From within the directory for the project, run the following command:

$ source venv/bin/activate

If you haven’t used Bootstrap before, you’re in for a treat. With Bootstrap, it’s easy to make a good-looking website quickly that’s responsive and mobile-friendly. We’ll also use HTML5 Boilerplate to get a basic HTML template in place.

Now, to install these easily, we’ll use Bower, which requires Node.js. Install Node.js first. On most Linux distros, you’ll also need to set NODE_PATH, which can be done by pasting the following into your .bashrc:

NODE_PATH="/usr/local/lib/node_modules"

With that done, run the following command to install Bower:

$ sudo npm install -g bower

Next we need to create a Bower config. First, create the folder blogengine/static. Then create a new file called .bowerrc and paste in the following content:

{
"directory": "blogengine/static/bower_components"
}

This tells Bower where it should put downloaded libraries. Next, run the following command to gener Bower:

$ bower init

Answer all the questions it asks you - for those with defaults, these should be fine, and everything else should be easy enough. Next, run the following command to install Bootstrap and HTML5 Boilerplate:

$ bower install bootstrap html5-boilerplate --save

Note that as jQuery is a dependency of Bootstrap, it will also be installed automatically. Now, we need to keep our Bower-installed files out of version control - the bower.json file keeps track of them for us. So add the following to your .gitignore file:

Now, let’s make our template nicer. Django’s templating system is very powerful and lets one template inherit from another. We’re going to create a base template, using HTML5 Boilerplate as a starting point, that all of our web-facing pages will use. First, create a directory to hold the base template:

$ mkdir templates/blogengine/includes

Then copy the index.html file from HTML5 Boilerplate to this directory as base.html:

We need to use {% load staticfiles %} to be able to load any static files.

We use the {% static %} template tag to load static files such as CSS and HTML

We define blocks called title and content. Any template that extends this one can override whatever is inside this template.

Please note that HTML5 Boilerplate may conceivable change in future, so bear in mind that all you really need to do is load the staticfiles app, use the static tag for any static files that need to be loaded, and define the blocks in the appropriate places.

Now fire up the server with python manage.py runserver and check everything is working OK. You should see that your new base template is now in use and the CSS and JS files are being loaded correctly. Let’s commit again:

Formatting our content

As it stands right now, we can’t do much to format our posts. It is possible to include HTML in our posts with Django, but by default it will strip it out. Also, we don’t want users to have to write HTML manually - we want to make our blog user friendly!

There are two possible approaches. One is to embed a rich text editor like TinyMCE in the admin and use that for editing the files, but I’ve found things like that to be cumbersome. The alternative is to use some other form of lightweight markup, and that’s the approach we’ll take here. We’re going to use Markdown for editing our posts.

Django has actually dropped support for Markdown, but it’s not hard to implement your own version. First, install Markdown and add it to your requirements.txt:

$ pip install markdown
$ pip freeze > requirements.txt

Now, we shouldn’t write any production code before writing a test, so let’s amend our existing post test to check to see that Markdown is working as expected:

What we do here is we convert our post text to include a link using Markdown. We also need to render that post in markdown within the test so that what we have in the test matches what will be produced - otherwise our test will be broken. We also check that the link is marked up correctly.

Save the file and run the tests - they should fail. Now, create the following directory and file:

Pagination

As at right now, all of our posts are displayed on the index page. We want to fix that by implementing pagination. Fortunately, that’s very easy for us because we’re using Django’s generic views. Go into blogengine/urls.py and amend it as follows:

Try adding a few more blog posts, and you’ll see the pagination links. But give them a try, and they won’t work. Why not? Well, as it turns out there was a bug in the project-wide urls.py file (my bad!). Let’s fix that:

Viewing individual posts

As our last task for today, we’ll implement individual pages for each post. We want each post to have a nice, friendly URL that is as human-readable as possible, and also includes the date the post was created.

Add this method to the PostViewTest class, after test_index. It’s very similar to test_index, since it’s testing much the same content. However, not that we fetch the post-specific URL using the method get_absolute_url, and we then fetch that page.

Now, if you run the test, it will fail because get_absolute_url isn’t implemented. It’s often a good idea to have a get_absolute_url method for your models, which defines a single URL scheme for that type of object. So let’s create one. However, to implement our URL scheme we need to make some changes. Right now we have the date, but we don’t have a text string we can use, known in Django as a slug. So we’ll add a slug field, which will be prepopulated based on the post title. Edit your model as follows:

Here we’ve added a slug field to the model, as well as implementing our get_absolute_url method. Note we’ve limited the date to year and month, but you can include days if you wish.

While we’re in here, we’ve also implemented the __unicode__ method. Essentially, this sets how Django describes the object in the admin - in this case, the post title is a logical way of describing that Post object, so it returns the post title.

We’ve also added the class Meta, with the ordering field. This tells Django that by default any list of posts should return them ordered by pub_date in reverse - in other words, latest first.

To have the slug filled in automatically, we need to customise the admin interface a little as well:

Now, I recommend at this stage going into the admin and deleting all of your posts, because otherwise you’ll have problems in migrating them. The issue is that each slug is compulsory and must be unique, and it’s not practical to use South to automatically generate new slugs from the title on the fly, so by deleting them at this stage you’ll avoid problems. Once that’s done, run this command:

$ python manage.py schemamigration --auto blogengine

You’ll be prompted to specify a one-off default value - enter any string you like, such as “blah”. Then run the migration:

Whoops! Our tests are broken, because the slug field isn’t being filled in. If you take a look at the page for adding a post, you’ll notice that the slug is filled in using JavaScript, so our test fails because the test client doesn’t interpret JavaScript. So in the tests we have to fill in the slug field manually.

Also, for the unit tests, the slug attribute isn’t being created at all, so it can’t be saved. Let’s remedy that. First, edit the test_create_post method of PostTest:

What we’re doing here is that every time we create a Post object programmatically, we add the post.slug atttribute to it. Also, when submitting a post via the admin, we pass the slug parameter via HTTP POST, thus emulating how a form would submit this data.

If you run the tests again, you’ll see that test_post_page still fails. This is because we haven’t yet up the URLs, templates and views to do so. Let’s fix that. We’ll use another generic view, called a DetailView, to display the posts. Amend blogengine/urls.py as follows:

As before, I’ve tagged the final commit with ‘lesson-2’, so if you’re following along, you can switch to this point with git checkout lesson-2.

Next time we’ll add support for flat pages and multiple authors, as well as adding support for comments via a third-party commenting system.

]]>https://matthewdaly.co.uk/blog/2013/12/28/django-blog-tutorial-the-next-generation-part-1/
https://matthewdaly.co.uk/blog/2013/12/28/django-blog-tutorial-the-next-generation-part-1/Sat, 28 Dec 2013 15:00:32 GMTMy series of Django tutorials for building a blogging engine are by far the most popular posts I’ve ever written on here. I’ve had a lot of people contact me with questions or just to express their thanks, for which I’m very grateful!

However, these tutorials haven’t really aged well. I’ve since had the opportunity to use Django in a professional capacity, which has significantly improved my understanding of the framework and the whole Python ecosystem, and there’s a lot of best practices that I didn’t follow and now wish I had. There’s also been a few gotchas that have hindered a few people in the past that I’d like to have the opportunity to correct.

So, I’m therefore creating a brand new series of tutorials to remedy this situation. This series will cover exactly the same basic idea of using Django to build a blogging engine, but will expand on what the original series did in many ways. We will cover such additional topics as:

Using Twitter Bootstrap to make your blog look good without too much hassle

Using Virtualenv to sandbox your blog application

Using South to effectively manage changes to your database structure

Writing some simple unit tests

Deploying the finished application to Heroku

Ready? Let’s get started!

Getting everything set up

Now, first of all, I’m going to assume you’re using some variant of Unix, such as Linux or Mac OS X. I’m not saying you can’t follow this tutorial with Windows, but you’ll have a harder time, because Windows just isn’t as developer-friendly as Unix in general. A modern Linux distro like Ubuntu is generally pretty easy to use, and you can easily run it in Virtualbox, so if you use Windows I would recommend you use that to make things easier.

You should also have at least a basic grasp of the command line, such as how to create and navigate directories. You don’t have to be a wizard with it, though.

You also need a proper programmer’s text editor. I use Vim, but I freely admit that Vim has a very steep learning curve and you may have trouble picking it up at the same time as following this tutorial. Emacs is also a very powerful text editor, and if you like it, feel free to use it. If you haven’t yet found a programmer’s text editor you like, I suggest you check out Sublime Text, which is easy to get started with, but also very powerful, and can be used without purchasing a license. Don’t worry too much about your text editor - it’s not vitally import that you use what I use, just find one that works for you. That said, I will say one thing - DON’T use an IDE. IDE’s hide too many details from new users and make it harder to figure out what’s going on.

You will also need to ensure you have the following installed:

Python. I recommend installing Python 2.7, because you may have issues with Python 2.6, and Python 3 isn’t universally supported yet so you might have some issues with that

Virtualenv

Pip

Git

On most Linux distros, you can find packages for all of these items easily enough using your package manager. On Mac OS X, I recommend using Homebrew to install them, though if you have another package manager installed you can use that too. If you have issues with installing any of these, a quick Google search should be sufficient to resolve the issue.

Beginning work

With all that done, we’re ready to get started. Create a folder in a suitable place on your file system and switch into it. I generally keep a dedicated folder in my home directory called Projects to use for all of my projects, and give each project a folder within it - in this case the project is called django_tutorial_blog_ng.

Now, we’ll use Git to keep track of our source code. If you prefer Mercurial, feel free to use that, but this tutorial will assume use of Git, so you’ll want to adapt the commands used accordingly. Start tracking your project with the following command from the shell, when you’re in the project directory:

Every time you come back to work on this project, you’ll need to run the previous command to make sure you’re running the version of Python installed under venv/ rather than your system Python. You can tell it’s using this because your shell prompt will be prefixed with (venv).

Why do this? Well, it means you can install whatever version of a Python module you like, without having root access, and means the Python install you’re using will only have those modules you explicitly install, rather than all of those ones available with your operating system. For instance, you could have multiple projects using different versions of Django, rather than having to update a global installation of Django and potentially break existing applications.

Now that our virtualenv is set up, we’ll install Django, as well as several other useful Python modules. Run the following command:

$ pip install django-toolbelt South

A little explanation is called for here. The package django-toolbelt includes a number of packages we’ll be using, including Django, as well as Gunicorn (a simple web server we’ll use when the time comes to deploy the app to Heroku). South is a migration tool that is commonly used with Django - basically, if you make changes to existing models, Django doesn’t natively have the capacity to apply those changes (yet - native migrations are planned at some point in the future), so South can be used to apply those changes for you without having to either manually change the database structure or dump the database and rebuild it.

Please note that one of the packages, psycopg2, may fail if you don’t have PostgreSQL installed, but don’t worry about installing it. We’ll be using SQLite for developing the application locally, and we’ll be deploying the finished product to Heroku, which does have it installed.

Once the installation is complete, run the following command to record the new modules installed:

$ pip freeze > requirements.txt

The file requirements.txt will be created, which stores the packages and versions you have installed so that they can be easily recreated. If you had issues installing psycopg2, then here’s what your requirements.txt should look like - feel free to edit it manually to look like this, as when we deploy it to Heroku, it will need to be correct to ensure that our application can be deployed successfully:

Next we’ll add a .gitignore file to ignore our virtualenv - we want to keep this out of version control because it’s something specific to that install. We have all we need to recreate it so we don’t want to store it. In addition, we also want to ignore any compiled Python files (identifiable by the .pyc suffix):

venv/
*.pyc

Let’s commit that too:

$ git add .gitignore
$ git commit -m 'Added a gitignore file'

Now, let’s generate our project’s basic skeleton:

$ django-admin.py startproject django_tutorial_blog_ng .

This application skeleton includes a basic configuration which will be sufficient for now, but you will also want to add the SQLite database file to your .gitignore:

You’ll be prompted to create a superuser - go ahead and fill in the details. Now, run the following command:

$ python manage.py runserver

This will run Django’s built-in web server on port 8000, and if you click here, you should see a page congratulating you on your first Django-powered page. Once you’re finished with it, you can stop the web server with Ctrl-C.

Your first app

Django distinguishes between the concepts of projects and apps. A project is a specific project that may consist of one or more apps, such as a web app, whereas an app is a set of functionality within a project. For instance, one website might include some flat pages, an admin interface, and a blogging engine, and these could easily be different apps. By encouraging you to separate different types of functionality into different apps, Django makes it easier for you to reuse existing content elsewhere.

We’re going to create our first app, which is the blogging engine. Run the following command to create a basic skeleton for this app:

Now, before we can use this app, we want to let South know about it so that changes to your database structure will be managed right from the start by South. Run the following command to create your initial migration:

$ python manage.py schemamigration --initial blogengine

That creates the file for your first migration,but doesn’t run it. To migrate your database structure to the latest version, run the following:

$ python manage.py migrate

This won’t actually make any changes, but it will ensure that all future changes to your models for the blogengine app are handled by South. Let’s commit our app skeleton:

So, we now have our first app set up, but it doesn’t do anything much.

Remember that I mentioned how Django differentiates between projects and apps? Well, Django actually ships with a number of useful apps, and one of those is the admin interface. I consider the Django admin to be one of the framework’s killer features because it’s easy to use and customise, and saves you a lot of grief.

In the past, the admin interface needed a little work to get it working, but in Django 1.6 it’s configured to work out of the box, so if you click here, you should see the login screen for it. You should be able to sign in using the username and password you set when you ran syncdb.

Next, we’ll set up our first model.

An introduction to MVC

MVC is a common pattern used in web development. Many web development frameworks can be loosely described as MVC, including Django, Rails, CodeIgniter, Laravel and Symfony, as well as some client-side frameworks like Backbone.js. The basic concept is that a web app is divided into three basic components:

Models - the data managed with the application

Views - the presentation of the data

Controllers - an intermediary between the models and the views

Now, Django’s interpretation of MVC is slightly different to many other frameworks. While in most frameworks the views are HTML templates for rendering the data, in Django this role is taken by the templates, and the views are functions or objects that render data from the models using a template. Effectively, you can think of Django’s views as being like controllers in other frameworks, and Django templates as being views.

In Django, you create your models as Python classes that represent your data, and you use the Django ORM to query the database. As a result, it’s rare to have to directly query your database using SQL, making it more portable between different databases.

Now, our first model is going to be of a blog post. At least initially, each post will have the following attributes:

A title

A publication date and time

Some text

Now, we could just jump straight into creating our first model, but we’re going to make a point of following the practices of test-driven development here. The basic concept of TDD is that you write a failing test before writing any code, then you write the code to pass that test afterwards. It does make things a bit slower, but it’s all too easy to neglect writing tests at all if you leave it till later.

If you take a look in the blogengine folder you’ll notice there’s a file called tests.py. Open it up and you should see the following:

from django.test import TestCase
# Create your tests here.

It’s worth taking a little time to plan out what we want to test from our post model. Each post object will have the attributes I mentioned above, and what we want to be able to do is test that we can:

Set the title

Set the publication date and time

Set the text

Save it successfully

Retrieve it successfully

So, let’s create a test for our post model. We’ll go through the relevant sections of the test bit by bit:

Here we’re importing the required functionality. TestCase is provided by Django, and is an object all of your tests should inherit from. timezone is a utility for handling dates and times correctly. Finally, Post is our model, which we have yet to implement.

Here we use the Django database API to fetch all of the Post objects, assert that there is only 1 post object, retrieve that post object, and assert that it is the same object as the post object we just saved.

If unit testing is new to you, assertions may be new to you. Essentially you’re saying to the Python interpreter, “I assert that X is true, so please raise an error if this is not true”. Here we assert that the length of the variable all_posts is 1, and that that post is the same object as the previously saved object, so that the test will fail if that is not the case.

Finally, we assert that the values of each of the post’s attributes as stored in the database match up with those in the post object we set. For the title and text fields, these are easy to validate as we can just check the values against those we set. For the pub_date field, things are a bit more complex, since this will be an object in its own right, so you need to check the day, month, year, hour, minute and second attributes separately.

Don’t worry about the error - this is exactly what we expect to see because we haven’t implemented our Post model yet. Now that we have a failing test in place, we can implement our model to make the test pass. Open up blogengine/models.py and enter the following:

Our test still fails, but if we take a look at this error message, we can see why - there is no database table for the posts (called blogengine_post). Using South, we can easily remedy that by creating a new migration to create this table:

Now, Django’s ORM is basically a layer on top of SQL that abstracts away differences between different relational databases, but the underlying queries are still being run. You can view the SQL created to generate the table by using the sqlall command. Just run python manage.py sqlall blogengine and you should see something like this:

Note the addition of the id field as the primary key. If you’re at all familiar with relational databases, you’ll know that every table must have one field, called a primary key, that is a unique reference to that row. This can be overridden, but here it’s exactly the behaviour we want.

Creating blog posts via the admin

Now, we need a way to be able to create, edit and delete blog posts. Django’s admin interface allows us to do so easily. However, before we do so, we want to create automated acceptance tests for this functionality, in order to test the ability to create posts from an end-user’s perspective. While unit tests are for testing sections of an application’s functionality from the perspective of other sections of the application, acceptance tests are testing from the user’s perspective. In other words, they test what the application needs to do to be acceptable.

First, we will test logging into the admin. Open up blogengine/tests.py and amend it as follows:

First of all, we import two new objects from django.test, LiveServerTestCase and Client. Then we create the first part of our first test for the admin, named AdminTest. Eventually, this will test that we can log successfully into the admin interface. For now, we’re just doing the following:

Creating a Client object

Fetching the /admin/ route

Asserting that the status code for this HTTP request is 200, (in other words, that the page was fetched successfully).

If you run python manage.py test, you should see that both tests pass successfully. Now we’ll extend AdminTest - we’ll verify that the response contains the string ‘Log in’, which in the Django admin interface, appears on the login page:

Here, we use the login method of the Client object to log into the admin interface, and then we fetch the /admin/ route again. We assert that we get a 200 status code, and we assert that the response contains the string ‘Log out’ - in other words, that we are logged in.

Try running python manage.py test and we’ll get an error., because the user details we’ve used to log in don’t exist. Let’s resolve that.

Now, you could put your own credentials in there, but that’s not a good idea because it’s a security risk. Instead, we’ll create a fixture for the test user that will be loaded when the tests are run. Run the following command:

$ python manage.py createsuperuser

Give the username as bobsmith, the email address as `bob@example.com, and the password aspassword`. Once that’s done, run these commands to dump the existing users to a fixture:

This test works along very similar lines. We log in, verify that ‘Log out’ is in the response, then we log out, and verify that ‘Log in’ is in the response. Run the tests again, and they should pass. Assuming they do, let’s commit our changes again:

$ git add blogengine/
$ git commit -m 'Added tests for admin auth'

This code is a little repetitive. We create the client twice, when we could do so only once. Amend the AdminTest class as follows:

The setUp() method is automatically run when the test runs, and ensures we only need to start up the client once. Run your tests to make sure they pass, then commit your changes:

$ git add blogengine/
$ git commit -m 'Refactored admin test'

Now, we’ll implement a test for creating a new post. The admin interface implements URLs for creating new instances of a model in a consistent format of /admin/app_name/model_name/add/, so the URL for adding a new post will be /admin/blogengine/post/add/.

Here we submit the new post via HTTP POST, with all the data passed through. This mirrors the form created by the Django admin interface - if you take a look at the HTML generated by the admin, you’ll see that the inputs are given names that match these. Note that the pub_date field, because it represents a datetime object, is split up into a separate date and time field. Also note the parameter follow=True - this denotes that the test client should follow any HTTP redirect.

We confirm that the POST request responded with a 200 code, denoting success. We also confirm that the response included the phrase ‘added successfully’. Finally we confirm that there is now a single Post object in the database. Don’t worry about any existing content - Django creates a dedicated test database and destroys it after the tests are done, so you can be sure that no posts are present unless you explicitly load them from a fixture.

We can now test creating a post, but we also need to ensure we can test editing and deleting them. First we’ll add a test for editing posts:

Here we create a new blog post, then verify we can edit it by resubmitting it with different values, and checking that we get the expected response, and that the data in the database has been updated. Run python manage.py test, and this should pass.

Again, this is pretty similar to what we did before. We create a new post, verify that it is the sole post in the database, and log into the admin. Then we delete the post via the admin, and confirm that the admin interface confirmed it has been deleted, and the post is gone from the database.

I think it’s now time to commit again:

$ git add blogengine/
$ git commit -m 'Post admin tests in place'

So we now know that we can create, edit and delete posts, and we have tests in place to confirm this. So our next task is to be able to display our posts.

For now, to keep things simple, we’re only going to implement the index view - in other words, all the posts in reverse chronological order. We’ll use Django’s generic views to keep things really easy.

Django’s generic views are another really handy feature. As mentioned earlier, a view is a function or class that describes how a specific route should render an object. Now, there are many tasks that recur in web development. For instance, many web pages you may have seen may be a list of objects - in this case, the index page for a blog is a list of blog posts. For that reason, Django has the ListView generic view, which makes it easy to render a list of objects.

Now, like before, we want to have a test in place. Open up blogengine/tests.py and add the following class at the end of the file:

Here we create the post, and assert that it is the sole post object. We then fetch the index page, and assert that the HTTP status code is 200 (ie. the page exists and is returned). We then verify that the response contains the post title, text and publication date.

Note that for the month, we need to do a bit of jiggery-pokery to get the month name. By default Django will return short month names (eg Jan, Feb etc), but Python stores months as numbers, so we need to format it as a short month name using %b.

If you run this, you will get an error because the index route isn’t implemented. So let’s fix that. Open up the existing django_tutorial_blog_ng/urls.py file and amend it to look like this:

A little explanation is called for. The project has its own urls.py file that handles routing throughout the project. However, because Django encourages you to make your apps reusable, we want to keep the routes in the individual apps as far as possible. So, in the project file, we include the blogengine/urls.py file.

In the app-specific urls.py, we import the Post model and the ListView generic view. We then define a route for the index page - the regular expression ^$ will match only an empty string, so that page will be the index. For this route, we then call the as_view() method of the ListView object, and set the model as Post.

Now, if you either run the tests, or run the development server and visit the index page, you’ll see that it isn’t working yet - you should see the error TemplateDoesNotExist: blogengine/post_list.html. This tells us that we need to create a template called blogengine/post_list.html, so let’s do that. First of all, add the following at the end of django_tutorial_blog_ng/settings.py:

And that’s all for this lesson! We’ve done a hell of a lot in this lesson - set up our project, created a comprehensive test suite for it, and implemented the basic functionality. Next time we’ll make it a bit prettier using Twitter Bootstrap, as well as implementing more of the basic functionality for the blog.

You can find the source code on Github. For your convenience, I’ve tagged this lesson as lesson-1, so you can just clone the repository and switch to the end of this lesson with the following commands: