Monday, March 30, 2015

Testing Django: Setting up a patch for an entire TestCase (or how to configure a mock object for an entire TestCase)

Often times you'll want to mock out some functionality in your tests. For example, if you're testing a Twitter API implementation, you don't want to actually hit the Twitter API when running your tests. You just want to "fake" a Twitter API response to make sure your code works properly. If you've used Mock objects before, you'll know that they provide the superpowers to enable this "faking" voodoo I speak of.

It's typical to mock out some functionality in your Django tests using the patch decorator:
class LoginViewTest(TestCase):
 def setUp(self):
  # set some stuff up

 @patch('django_twitter_auth.views.redirect')
 def test_view_redirects_to_auth_url(self, mock_redirect):
  # do some tests
In the above example, we're replacing the redirect function with a Mock (technically a MagicMock) object. So whenever the redirect function is called within the context of the test_view_redirects_to_auth_url test, it'll actually be calling the Mock object which we have complete control over.

If you want to mock out an object for a bunch of tests, you could apply the patch decorator to each test. You could also apply the patch decorator to the class itself:
@patch('django_twitter_auth.views.redirect')
class LoginViewTest(TestCase):
 def setUp(self):
  # set some stuff up

 def test_view_redirects_to_auth_url(self, mock_redirect):
  # do some tests

 def some_other_test(self, mock_redirect):
  # do some tests
But if each test, for example, requires the mock_redirect to be setup and configured to behave a specific way, you'll easily end up repeating a lot of code in each test.

Unfortunately you can't just pass in the mocked object to your setUp method like this:
@patch('django_twitter_auth.views.redirect')
class LoginViewTest(TestCase):
 def setUp(self, mock_redirect):
  # set the mock redirect for all the tests

 ...
But wait! There is an alternative solution that allows you to configure your Mock objects in your test class' setUp method:
class LoginViewTest(TestCase):
 def setUp(self):
  # provide the location of the object you want to mock
  patcher = patch('django_twitter_auth.views.Twython')
  # explicitly start the patch 
  self.mock_Twython = patcher.start()
  # make sure the patch is removed
  self.addCleanup(patcher.stop)

  # mock out all functionality required for view
  self.mock_Twython.some_attribute = "foo"
  self.mock_Twython.some_method.return_value = "bar"

 def test_Twython_instance_initialized_with_app_tokens(self):
  # the test has access to the Mock object!
  # let's do some fake setup here that's unique to
  # this test function
  mock_Twython_instance = self.mock_Twython.return_value
  
  # do some tests ...

  # assert some stuff

 def some_other_Twython_test(self):
  # this test also has access to the Mock object
  # watch him do some unique setup, like overriding
  # the default setup maybe?
  self.mock_Twython.some_method.return_value = "uniquebar"

  # do some tests ...

  # assert some stuff
Take away the decorator magic, and patch can be used like a normal function. It returns a patcher object. When you call the patcher object's start() method, it returns the mocked out object. The only caveat is that you need to explicitly make sure the patching is "undone". This is what the:
    self.addCleanup(patcher.stop)  
line in the setUp method in the above example does.

You can mock out multiple objects in setUp, just assign your patcher objects to different variable names like patcher_1, patcher_2, ..., etc. and then call patcher_1.start(), patcher_2.start(), ..., etc. to return the object each patcher object is mocking out.

You can read more about the start() and stop() methods in the official Mock documentation here.

Happy testing!

Sunday, January 25, 2015

Django Tip: Using the @property decorator on custom model methods

Using the @property decorator allows you to trigger custom model methods as if they were normal model attributes. For example, if you have the following custom model method:
class MyModel(object):
    def combined_name(self):
        return self.first_name + ' ' + self.last_name
Adding the @property decorator to that method allows you to access the computed value of combined_name like an attribute: MyModel().combined_name, instead of having to explicitly trigger the function like: MyModel().combined_name().

How does the @property decorator work?


There's actually a lot going on behind the scenes in Python to make this happen. First of all, the @property decorator is just a convenient way to call the property() function, which is built in to Python. This function returns a special descriptor object which allows direct access to the method's computed value. So instead of doing:
class FooBar(object):
    def foo(self):
        return "foo"
    
    x = property(foo)
You can do:
class FooBar(object):
    @property
    def foo(self):
        return "foo"
Both implementations allow you to access the computed value, the string "foo" in this case, by either doing FooBar().x or FooBar().foo. And again, this is possible because the attribute is now pointing to a descriptor object which is triggering the method foo(self) and returning the computed value.

Descriptor objects have one or more of the following methods defined: __get__, __set__, and/or __delete__. Using the property() function, you can set these methods by passing them in as arguments like so: property(fget, fset, fdel), with fget getting assigned to __get__, the fset to __set__, and fdel to __delete__. In our above examples, we're only passing in the fget positional argument. So, when you access the class attribute assigned to the descriptor object, the method we passed in is now assigned to the descriptor object's __get__ method and is triggered, with the computed value being returned.

These special methods also have access to the instance. Which is how something like the following could work:
class FooBar(object):
    def __init__(self, name='Dylan'):
        self.name = name
    
    @property
    def foo(self):
        return "My name is " + self.name + "!"
Which is why calling FooBar(name='Mark').foo would evaluate to "My name is Mark!"

Confused still? That's fine. I am too at times. This isn't the easiest of stuff to wrap your head around. Take a look at the "Further Reading" below for more helpful resources about descriptor objects and the property function.

Why use it?


I only recently stumbled upon the @property decorator. Personally, I like to use it for model methods that return computed attributes. For example, in my project www.wikivinci.com, I have an Account model. Each Account instance has a 'points' IntegerField to, well, keep track of each Account's total points. I wrote a basic model method to return the ranking of an Account instance (ie: how many other Account objects are in the database with higher 'points'). This method simply computes an attribute, so I chose to use the @property decorator.

On the other hand, I also wrote the following model method:
def award_points(self, points=0):
    self.points += points
    self.save()
It doesn't make sense to use the @property decorator here. Since this is a method that's called like a traditional function with arguments.

Django Limitations/Considerations


While using the @property decorator can make it a bit more convenient to access model methods as attributes, you can't filter querysets by them, use them for ordering, or any other operation that happens at the database level. I read somewhere that when using the the @property decorator, the computed value is cached, but I didn't find that to be true during my testing. If someone has more insight around this, and/or other limitations or best practices regarding using custom model method properties, please post them in the comments!

Further Reading


Property function
Descriptors
How does the property decorator work - Stack Overflow

Wednesday, January 7, 2015

Using S3 to Serve and Store Your Django Project's Media Files

This blog post will show you how to set up your Django project to store, and serve, your media files from an S3 bucket. It assumes you already have an S3 bucket set up, and already know how to set up your Django project to serve static files from that bucket.

When I deployed my first Django project to Heroku, I found a lot of great resources and tutorials online for setting up an S3 bucket to serve my static files. But I couldn't find anything that touched on how to set up my project to store/serve media files from S3 as well. Luckily I came across a few helpful StackOverflow answers, but I've always wanted to write a blog post to explain the steps in 1 place. This is that post!

Step 1


If you're using S3 to serve static files, you might already have boto and storages installed. If not, do the following:
pip install django-storages
pip install boto
Then, add storages and boto to your INSTALLED_APPS in your settings file.

django-storages is a collection of custom storage backends for Django. You can read more about it here. The backend we'll be using requires the boto library. You can read more about boto here. This custom backend allows your static/media files to easily be sent to your S3 bucket.

Step 2


Add the following lines to your settings file:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_STORAGE_BUCKET_NAME = 'aws_bucket_name'
AWS_ACCESS_KEY_ID = 'aws_access_key_id'
AWS_SECRET_ACCESS_KEY = 'aws_secret_access_key'
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = S3_URL
If you're already using S3 to serve static files, you might have the following lines as well:
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATIC_URL = S3_URL
The DEFAULT_FILE_STORAGE is where the magic happens. Updating this setting overwrites Django's default storage system, which implements file storage on the local filesystem.

You'll notice the STATIC_URL and MEDIA_URL are the same in this example. That's because S3 will store your static and media files in the same folder by default. If you want to store your files in different folders, like you would on your local machine, follow the next step.

Step 3


Create a s3utils.py file in your project configuration folder. Add the following lines to it:
from storages.backends.s3boto import S3BotoStorage

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')
Then, update the DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings to the following:
DEFAULT_FILE_STORAGE = 'yourproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'yourproject.s3utils.StaticRootS3BotoStorage'
Lastly, update your STATIC_URL and MEDIA_URL settings to reference the new folder. For example:
STATIC_URL = S3_URL + 'static/'
MEDIA_URL = S3_URL + 'media/'
It should be that easy. If you have any questions, or if you notice anything in this tutorial that needs to be updated, you can join the discussion on Wikivinci. Or, feel free to email me at dylanbfox[at]gmail.com.