Sometimes you want to know what actually happens on the datastore when running an application.
Especially when dealing with pythonic database models you use their attributes and methods without realizing at first glance that they issue a bunch of datastore queries behind the scenes.
For WSGI-based App Engine applications you can use the profiling module described in this recipe to get a clue what actually happens on the datastore.
Using the Profile Module
To enable datastore profiling import the attached profiling module in your main application module (usually called main.py) and replace your main function with the debug_main function provided by the profile module.
Example main.py:
[...]
def main():
run_wsgi_app(application)
# Uncomment the following two line to enable datastore profiling:
#import datastoreprofile
#main = datastoreprofile.debug_main(main)
[...]
Output Channels
The profiling module provides three different output channels:
'memcache' - View profiling data on a separate page.
'inline' - Adds profiling data at the bottom of a page (default).
'log' - Writes profiling data using the logging framework.
To change the output channel, set the OUTPUT_CHANNEL variable after importing the profiling module. For example:
import datastoreprofile
datastoreprofile.OUTPUT_CHANNEL = 'inline'
Using the 'inline' channel is the easiest way to view profiling data, but since this module collects a lot of data, the resulting page size could easily hit the 1MB limit.
So, using the 'memcache' channel is the recommended way to view profiling data. To enable the separate page to view collected data add
- url: /datastoreprofile/.*
script: datastoreprofile.py
to your app.yaml and open e.g. http://localhost:8080/datastoreprofile/ in your browser.
When using the 'log' channel open the log viewer, set "Minimum Severity" to "Debug" and use "^DATASTORE:" as filter to view collected data.
Known Bugs
If the application reloads when running with SDK (dev_appserver.py) some parts of the profile module get messed up. As a workaround just restart the SDK server.
Links
Source code: http://git.andialbrecht.de/gitweb/?p=gae-datastoreprofile.git
A blog post about this module: http://andialbrecht.blogspot.com/2008/12/profiling-datastore-access-on-app.html