Monday, March 30, 2015

Testing Django: Setting up a patch for an entire TestCase (or how to configure a mock object for an entire TestCase)

Often times you'll want to mock out some functionality in your tests. For example, if you're testing a Twitter API implementation, you don't want to actually hit the Twitter API when running your tests. You just want to "fake" a Twitter API response to make sure your code works properly. If you've used Mock objects before, you'll know that they provide the superpowers to enable this "faking" voodoo I speak of.

It's typical to mock out some functionality in your Django tests using the patch decorator:
class LoginViewTest(TestCase):
 def setUp(self):
  # set some stuff up

 @patch('django_twitter_auth.views.redirect')
 def test_view_redirects_to_auth_url(self, mock_redirect):
  # do some tests
In the above example, we're replacing the redirect function with a Mock (technically a MagicMock) object. So whenever the redirect function is called within the context of the test_view_redirects_to_auth_url test, it'll actually be calling the Mock object which we have complete control over.

If you want to mock out an object for a bunch of tests, you could apply the patch decorator to each test. You could also apply the patch decorator to the class itself:
@patch('django_twitter_auth.views.redirect')
class LoginViewTest(TestCase):
 def setUp(self):
  # set some stuff up

 def test_view_redirects_to_auth_url(self, mock_redirect):
  # do some tests

 def some_other_test(self, mock_redirect):
  # do some tests
But if each test, for example, requires the mock_redirect to be setup and configured to behave a specific way, you'll easily end up repeating a lot of code in each test.

Unfortunately you can't just pass in the mocked object to your setUp method like this:
@patch('django_twitter_auth.views.redirect')
class LoginViewTest(TestCase):
 def setUp(self, mock_redirect):
  # set the mock redirect for all the tests

 ...
But wait! There is an alternative solution that allows you to configure your Mock objects in your test class' setUp method:
class LoginViewTest(TestCase):
 def setUp(self):
  # provide the location of the object you want to mock
  patcher = patch('django_twitter_auth.views.Twython')
  # explicitly start the patch 
  self.mock_Twython = patcher.start()
  # make sure the patch is removed
  self.addCleanup(patcher.stop)

  # mock out all functionality required for view
  self.mock_Twython.some_attribute = "foo"
  self.mock_Twython.some_method.return_value = "bar"

 def test_Twython_instance_initialized_with_app_tokens(self):
  # the test has access to the Mock object!
  # let's do some fake setup here that's unique to
  # this test function
  mock_Twython_instance = self.mock_Twython.return_value
  
  # do some tests ...

  # assert some stuff

 def some_other_Twython_test(self):
  # this test also has access to the Mock object
  # watch him do some unique setup, like overriding
  # the default setup maybe?
  self.mock_Twython.some_method.return_value = "uniquebar"

  # do some tests ...

  # assert some stuff
Take away the decorator magic, and patch can be used like a normal function. It returns a patcher object. When you call the patcher object's start() method, it returns the mocked out object. The only caveat is that you need to explicitly make sure the patching is "undone". This is what the:
    self.addCleanup(patcher.stop)  
line in the setUp method in the above example does.

You can mock out multiple objects in setUp, just assign your patcher objects to different variable names like patcher_1, patcher_2, ..., etc. and then call patcher_1.start(), patcher_2.start(), ..., etc. to return the object each patcher object is mocking out.

You can read more about the start() and stop() methods in the official Mock documentation here.

Happy testing!

Sunday, January 25, 2015

Django Tip: Using the @property decorator on custom model methods

Using the @property decorator allows you to trigger custom model methods as if they were normal model attributes. For example, if you have the following custom model method:
class MyModel(object):
    def combined_name(self):
        return self.first_name + ' ' + self.last_name
Adding the @property decorator to that method allows you to access the computed value of combined_name like an attribute: MyModel().combined_name, instead of having to explicitly trigger the function like: MyModel().combined_name().

How does the @property decorator work?


There's actually a lot going on behind the scenes in Python to make this happen. First of all, the @property decorator is just a convenient way to call the property() function, which is built in to Python. This function returns a special descriptor object which allows direct access to the method's computed value. So instead of doing:
class FooBar(object):
    def foo(self):
        return "foo"
    
    x = property(foo)
You can do:
class FooBar(object):
    @property
    def foo(self):
        return "foo"
Both implementations allow you to access the computed value, the string "foo" in this case, by either doing FooBar().x or FooBar().foo. And again, this is possible because the attribute is now pointing to a descriptor object which is triggering the method foo(self) and returning the computed value.

Descriptor objects have one or more of the following methods defined: __get__, __set__, and/or __delete__. Using the property() function, you can set these methods by passing them in as arguments like so: property(fget, fset, fdel), with fget getting assigned to __get__, the fset to __set__, and fdel to __delete__. In our above examples, we're only passing in the fget positional argument. So, when you access the class attribute assigned to the descriptor object, the method we passed in is now assigned to the descriptor object's __get__ method and is triggered, with the computed value being returned.

These special methods also have access to the instance. Which is how something like the following could work:
class FooBar(object):
    def __init__(self, name='Dylan'):
        self.name = name
    
    @property
    def foo(self):
        return "My name is " + self.name + "!"
Which is why calling FooBar(name='Mark').foo would evaluate to "My name is Mark!"

Confused still? That's fine. I am too at times. This isn't the easiest of stuff to wrap your head around. Take a look at the "Further Reading" below for more helpful resources about descriptor objects and the property function.

Why use it?


I only recently stumbled upon the @property decorator. Personally, I like to use it for model methods that return computed attributes. For example, in my project www.wikivinci.com, I have an Account model. Each Account instance has a 'points' IntegerField to, well, keep track of each Account's total points. I wrote a basic model method to return the ranking of an Account instance (ie: how many other Account objects are in the database with higher 'points'). This method simply computes an attribute, so I chose to use the @property decorator.

On the other hand, I also wrote the following model method:
def award_points(self, points=0):
    self.points += points
    self.save()
It doesn't make sense to use the @property decorator here. Since this is a method that's called like a traditional function with arguments.

Django Limitations/Considerations


While using the @property decorator can make it a bit more convenient to access model methods as attributes, you can't filter querysets by them, use them for ordering, or any other operation that happens at the database level. I read somewhere that when using the the @property decorator, the computed value is cached, but I didn't find that to be true during my testing. If someone has more insight around this, and/or other limitations or best practices regarding using custom model method properties, please post them in the comments!

Further Reading


Property function
Descriptors
How does the property decorator work - Stack Overflow

Wednesday, January 7, 2015

Using S3 to Serve and Store Your Django Project's Media Files

This blog post will show you how to set up your Django project to store, and serve, your media files from an S3 bucket. It assumes you already have an S3 bucket set up, and already know how to set up your Django project to serve static files from that bucket.

When I deployed my first Django project to Heroku, I found a lot of great resources and tutorials online for setting up an S3 bucket to serve my static files. But I couldn't find anything that touched on how to set up my project to store/serve media files from S3 as well. Luckily I came across a few helpful StackOverflow answers, but I've always wanted to write a blog post to explain the steps in 1 place. This is that post!

Step 1


If you're using S3 to serve static files, you might already have boto and storages installed. If not, do the following:
pip install django-storages
pip install boto
Then, add storages and boto to your INSTALLED_APPS in your settings file.

django-storages is a collection of custom storage backends for Django. You can read more about it here. The backend we'll be using requires the boto library. You can read more about boto here. This custom backend allows your static/media files to easily be sent to your S3 bucket.

Step 2


Add the following lines to your settings file:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_STORAGE_BUCKET_NAME = 'aws_bucket_name'
AWS_ACCESS_KEY_ID = 'aws_access_key_id'
AWS_SECRET_ACCESS_KEY = 'aws_secret_access_key'
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = S3_URL
If you're already using S3 to serve static files, you might have the following lines as well:
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATIC_URL = S3_URL
The DEFAULT_FILE_STORAGE is where the magic happens. Updating this setting overwrites Django's default storage system, which implements file storage on the local filesystem.

You'll notice the STATIC_URL and MEDIA_URL are the same in this example. That's because S3 will store your static and media files in the same folder by default. If you want to store your files in different folders, like you would on your local machine, follow the next step.

Step 3


Create a s3utils.py file in your project configuration folder. Add the following lines to it:
from storages.backends.s3boto import S3BotoStorage

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')
Then, update the DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings to the following:
DEFAULT_FILE_STORAGE = 'yourproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'yourproject.s3utils.StaticRootS3BotoStorage'
Lastly, update your STATIC_URL and MEDIA_URL settings to reference the new folder. For example:
STATIC_URL = S3_URL + 'static/'
MEDIA_URL = S3_URL + 'media/'
It should be that easy. If you have any questions, or if you notice anything in this tutorial that needs to be updated, you can join the discussion on Wikivinci. Or, feel free to email me at dylanbfox[at]gmail.com.

Wednesday, November 26, 2014

Custom Django Middleware

I needed to require login for every view within a Django project. This project had multiple apps and multiple views within those apps. Sprinkling the @login_required decorator around every view was tedious, and didn't seem like a clean solution.

After doing a couple of of searches, I came across a few, older (2010/2011) middleware packages that looked like good solutions. Since the packages were older, and I have never written any custom middleware before for Django, I decided to write my own solution. I was really surprised at how easy it was to plug in some custom middleware.

Here's how I created my own middleware to require login for every view within my project.

Create a "middleware.py" for your project


I placed this in my project configuration folder. It doesn't matter where you put the file, as long as you reference the path to it properly in your settings file (more on this later).

Create your custom Middleware Component


Per Django's documentation: Each middleware component is a single Python class. So, creating your own middleware component looks like this:

class MyMiddlewareClass(Object):
  # define custom methods in here

For my purposes, I created the following class:

class LoginRequiredMiddleware(Object):
  # my custom methods in here

Build out your custom Middleware Methods


I needed to check if the user was authenticated or not, before processing the view. So, I defined the process_view method.

from django.http import HttpResponseForbidden

  class LoginRequiredMiddleware(object):

  def process_view(self, request, view_func, view_args, view_kwargs):
    if request.user.is_authenticated():
      return None
    else:
      return HttpResponseForbidden()


This is a very basic implementation of the functionality I needed. Before Django calls the view (view_func), this method simply checks if the request is authenticated or not.

If it is, it returns None, which tells Django to continue on processing the request as per usual, eventually executing view_func. If it isn't, it returns the convenient HttpResponseForbidden.

I defined the process_view method as opposed to process_request method because I wanted Django to catch 404 errors before running my custom middleware. Otherwise, my middleware would send an unauthenticated user to the login page even when they requested a URL that hasn't been defined in my URL patterns.

Obviously this is too basic. What about the STATIC_URL and MEDIA_URL? Or the Django Admin, or any other pages you want to be public?

To handle these contingencies, I built out the method a bit more to look like this:

import re

from django.http import HttpResponseRedirect
from django.conf import settings

class LoginRequiredMiddleware(object):

  PUBLIC_URL_PATTERNS = [
    r'^%s.*$' % settings.STATIC_URL,
    r'^%s.*$' % settings.MEDIA_URL,
    r'^/admin/.*$',
    r'^/(\?.*)?$',
    r'^%s.*$' % settings.LOGIN_URL,
  ]

  PUBLIC_URL_PATTERNS = [re.compile(exp) for exp in
                        PUBLIC_URL_PATTERNS]

  def process_view(self, request, view_func, view_args, view_kwargs):
    path = request.path
    if any(pattern.match(path) for pattern in 
      self.PUBLIC_URL_PATTERNS):
      return None

    elif request.user.is_authenticated():
      return None

    else:
      redirecturl = settings.LOGIN_URL + "?go=" + path
      return HttpResponseRedirect(redirecturl)


The method now checks to see if the requested URL matches any public URL defined in PUBLIC_URL_PATTERNS. It also appends a query string to the LOGIN_URL that the user is re-directed to. So that we can easily send the user to the URL they requested after he/she logs in.

This is still pretty basic. And it only accounts for a few public URL patterns. It'll need to be built out a bit to be more robust. And I might follow up with another post once I do that. I wouldn't recommended blindly copying this code for use in your own project.

Add your custom Middleware to your settings


MIDDLEWARE_CLASSES = (
    ...,
    'projectroot.middleware.LoginRequiredMiddleware',
)

Be sure to replace projectroot with the module name of your project configuration folder. My full path, for example, is "dylansproject.middleware.LoginRequiredMiddleware". Now your project is all set up to use your custom middleware!