Sign in

Google Cookbook - Google App Engine

Posted by satoshi.nakajima on Mon 23 Nov 2009 in Webapp Framework
Since I don't like large frameworks, I am writing all my App Engine applications with webapp framework. The only thing missing in this framework is a mechanism that allows me to describe URL mapping rule along with each request handler (instead of in one central place).

This is why came up with a simple dispatch mechanism, gdispatch (attached to this recipe, but also available at http://github.com/snakajima/gdispatch).

See the example below:

@gdispatch.route(lambda:('/login', LoginHandler))
class LoginHandler(webapp.RequestHandler):
def post(self):
...

@gdispatch.route(lambda:('/login', LogoutHandler))
class LogoutHandler(webapp.RequestHandler):
def get(self):
...

gdispatch.run()
Posted by paul.marrington.net on Sat 21 Nov 2009 in Mashups with App Engine
Yes, I know I can use the atom package provided for and used by gdata, but it is so hard to understand without documentation. I like my code simpler, so for fun I rolled my own. The attached code is (hopefully) well documented, so I will just discuss the highlights here. The example provides an atom feed for a traveller’s notes.
Firstly I am tired of placing the same webapp code in all documents:
application = webapp.WSGIApplication([('/feeds/traveller', TravellerFeed)], debug = True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()



So, I created a helper function so I can say:
webapp.run(__name__, '/feeds/traveller', TravellerFeed)



instead. The request handler behaves as usual. Here we get the bigtable keys for records related to groups of friends interested in the Traveller feed. The feed supplies notes from the last n days, where n defaults to 28.
class TravellerFeed(webapp.RequestHandler):
def get(self):
keys = self.request.get_all('key')
days = int(self.request.get('days', '28'))
group = Group.get(keys[0])
user = User.fromId(group.userId)
groups = [group.get(key).name for key in keys]



Now we get to the nub - the atom feed.

I use http://www.atomenabled.org/developers/syndication/

For a clear description of the basic elements needed for an atom news feed. I have a lightweight XML generator (attached) that converts python to XML. Once you have created the root element:

        feed = writer.xml('feed', xmlns = "http://www.w3.org/2005/Atom")



Then you can generate xml elements in python such that feed.a.b.c(12) would create 12. The following code produces independent line, id and title elements.

        userKey = user.key()
link = 'http://trogger.net/traveller/%s' % userKey
feed.link(link).id(userKey)
if user.fullName:
feed.title("Travel Notes for %s" % user.fullName).author.name(user.fullName)
else:
feed.title("Trave Notes")



Now we look through the database table and generate an entry element for each record.

        for record in Record.recent(user.id, 'notes', groups, days = days):
if first:
'''
Use common data from the first record to fill out the general
feed data - including the last updated date.
'''
first = False
feed.updated(str(record.updated))
trip = Trip.get(user.id, record.trip)
link = 'http://trogger.net/trip/%s' % trip.key()
entry = feed.entry
entry.id(record.key()).title(record.title).updated(str(record.updated))
entry.content(record.notes)
if user.fullName:
entry.author.name(user.fullName)
entry.link(link)



When an XML object is converted to a string, XML results:

        self.response.out.write(str(feed))



To test locally, open an appengine console and choose a key for a record that matches your code. You will need a local newsreader. I use newsfire for this task. Tell it you have a new feed, giving a URL similar to:

http://localhost:8080/feeds/traveller?key=agp0cm9nZ2VybmV0chULEgVHcm91cCIKVTE7UHJpdmF0ZQw

Posted by ikai.l@google.com on Fri 20 Nov 2009 in Java
Using the Python Upload and Download Tools with a Java application
There isn't a Java data export tool yet. However, with a little tweaking, it is possible to use the Python App Engine tools with a Java application. The steps to doing this are:

1. Install the Python SDK
2. Create a minimal version of your application in Python that defines models mapped to your Java classes
3. Create a exporter configuration in Python
4. Use the bulk download tool


Understanding application versions
Application versions are simply strings. It's possible to run several versions of the application at once. Suppose you have two applications, a Java version and a Python version.

Let's suppose your Java application's appengine-web.xml is defined as follows:

  appid
java



You would be able to access this application at http://java.latest.appid.appspot.com.

Now let's suppose you have a Python application with the following app.yaml file:

application: appid 
version: python
runtime: python
api_version: 1
handlers:
- url: /
script: main.py



You would be able to access this application at http://python.latest.appid.appspot.com.

All application versions will have access to the same datastore and memcached partitions. What's really great about this setup is that you can have multiple versions deployed, but set only one of the versions as the default to be served, but still have the other versions accessible. This can be done in the administrative console under Administration -> Versions. This technique can also be used to stage new versions of the application against live data (after ample testing, of course) before a full rollout. In the event of new bugs introduced into production, rollback is just as simple. There's more about this topic here: http://googleappengine.blogspot.com/2009/06/10-things-you-probably-didnt-know-about.html

Creating a Python exporter
The datastore is language agnostic. There's nothing that prevents data created by Java from being accessed by Python. Suppose we have a Java class defined as follows:

import com.google.appengine.api.datastore.Key;
import javax.jdo.annotations.*;
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Thing {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;
@Persistent
private String name;
@Persistent
private int number;
public Thing(String name, int number) {
this.name = name;
this.number = number;
}
public Key getKey() {
return key;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getNumber() {
return number;
}
public void setNumber(int number) {
this.number = number;
}
}



It would be possible to create a Python application that can access this data. This particular Class's doppleganger is as follows:

class Thing(db.Model):
name = db.StringProperty()
number = db.IntegerProperty()



(If you've been doing nothing but Java and this is your first exposure to Python, be warned! It's addictive and fun.)

The next step is to create an exporter. Per the Python documentation at http://code.google.com/appengine/docs/python/tools/uploadingdata.html:

from google.appengine.ext import db
from google.appengine.tools import bulkloader
class Thing(db.Model):
name = db.StringProperty()
number = db.IntegerProperty()
class ThingExporter(bulkloader.Exporter):
def __init__(self):
bulkloader.Exporter.__init__(self, 'Thing',
[('name', str, None),
('number', str, None)
])
exporters = [ThingExporter]



The exporter is a code-as-configuration file that allows you to specify how to create the output CSV file. You'll also want to update app.yaml to specify the app ID, version and to enable data export. Assuming your Python application is called "exporter":

application: appid 
version: python
runtime: python
api_version: 1
handlers:
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin



When this is done, deploy your application using appcfg.py:

appcfg.py update exporter



Make sure your default version is set to serve from the Java app! Your Python app is purely here to allow you to export data.
Exporting data
You're almost done. Now it's time to export the data. We'll need to manually specify the URL to fetch from in addition to the required exporter arguments. This URL will point to the Python version of the application:

appcfg.py --server=python.latest.appid.appspot.com download_data exporter  --filename=data.csv --kind=Thing --config_file=exporter/thing_exporter.py 



This should be all you have to do.
Posted by Lucemia on Mon 09 Nov 2009 in Django
The simplest way to add internationalization support in google app engine

#urls.py
from django.conf import settings
try:
settings.configure()
except:
pass
settings.LANGUAGE_CODE = 'zh-tw'
settings.USE_I18N = True
appdir = os.path.abspath( os.path.dirname( __file__ ) )
settings.LOCALE_PATHS = (
os.path.join( appdir, 'locale' ),
)
from django.utils.translation import *



# for each request
class Page(webapp.RequestHandler):
def getLanguage(self):
try:
language = self.request.cookies['django_language']
self.locate = language
logging.info( "Get Language as %s" % self.locate )
except:
from django.conf import settings
self.locate = settings.LANGUAGE_CODE
logging.info( "Set Language as %s" % self.locate )
translation.activate( self.locate )
def get(self):
self.getLanguage()
#...

Posted by squall on Sun 08 Nov 2009 in Mail API
This sample shows how to get mail body as plaintext.
Posted by paul.marrington.net on Sat 07 Nov 2009 in Mashups with App Engine
This example is called TripKML. It reads pre-recorded GeoPt records and generates KML for Google Maps/Earth. The basic webapp is as normal:

class TripKML(webapp.RequestHandler):
def get(self):
try:
...
except DeadlineExceededError:
self.response.clear()
self.response.set_status(500)
self.message("This operation could not be completed in time...")
application = webapp.WSGIApplication([('/service/tripKML', TripKML)], debug = True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()



First we parse the command line:

            self.traveller = User.fromId(int(self.request.get('traveller')))
trip = self.request.get('trip')



As the code creates KML, tell the browser what do do with it:

        self.response.headers['Content-Type'] = 'application/vnd.google-earth.kml+xml'



I have attached my lightweight class for generating XML. You give it the root XML tag name and attributes. Wrap everything in a folder for clarity in the tree. The last line is XML to be inserted (saved as http://…)

            rootElement = self.kml = xml('kml', xmlns = 'http://www.opengis.net/kml/2.2')
self.kml = rootElement.Folder
self.kml.name(trip)
self.kml.Style.IconStyle(id = 'noteStyle').Icon.href(iconUrl)



My style is to read the records and have factory code to call methods based on the record type. Because most records will be of one or two types, I have cached the parser method:

            parsers = dict()
query = Record.gql(
'WHERE userId=:1 AND trip=:2 ORDER BY time', self.traveller.id, trip)
for self.record in query:
try:
parser = parsers[self.record.type]
except KeyError:
method = self.record.type + "RecordProcessor";
parser = getattr(self, method, self.defaultRecordProcessor)
parsers[self.record.type] = parser
parser()



A record type called "notes" generates a note indicated by an icon on the map:

    def notesRecordProcessor(self):
self.kml.Placemark.name(self.record.title).styleUrl('noteStyle').\
description(self.record.notes).Point.coordinates(
"%f,%f" % (self.record.location.lon, self.record.location.lat))()



The “here” record processor does more work as it collects coordinates to make a line.

Back to the TripKml class, I also overlay kml generated by Picasa to display images:

            albums = Album.forTrip(self.traveller.id, trip)
for album in albums:
try:
self.kml.NetworkLink.name(trip).Link.refreshMode('onInterval').\
refreshInterval(3600).hreflang(\
album.feedUri + '(anpersand)amp;alt=kml(anpersand)amp;kind=photo')
except Exception, e:
logging.warning('Can\'t access Picasa album for trip %s: %s' % (trip.name, e.message))



The last thing we need to do is generate the KML and send it to the browser:

            self.response.out.write(str(self.kml))



I have attached TripKML.py as it is in use and xml.py for generating the XML.
Posted by robi123 on Fri 30 Oct 2009 in Python
Using the task queue, hit app every 10 seconds to keep a hot handler.

### in queue.yaml
queue:
- name: hothandler
rate: 20/m
bucket_size: 1


### in cron.yaml
cron:
- description: ensure hot handler
url: /_ah/queue/hothandler/start
schedule: every 4 hours


### in app.yaml
handlers:
- url: /_ah/queue/hothandler/.*
script: hothandler.py
login: admin


### in hothandler.py
"""
Copyright (C) 2009 twitter.com/rcb

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
"""
import random
from wsgiref.handlers import CGIHandler
from google.appengine.api.labs import taskqueue
from google.appengine.api import memcache
from google.appengine.api.capabilities import CapabilitySet
memcache_service = CapabilitySet('memcache', methods=['set','get'])
hot_handler_queue = taskqueue.Queue(name='hothandler')
HOT_HANDLER_PREFIX = '/_ah/queue/hothandler/'
def wsgi_app(env, res):
""" visit '/_ah/queue/hothandler/start' as admin to start a task """
token = env['PATH_INFO'].replace(HOT_HANDLER_PREFIX,'')
cur_token = memcache.get(HOT_HANDLER_PREFIX)
if cur_token is None:
if not memcache_service.is_enabled():
cur_token = token
if token in [cur_token, 'start']:
next_token = str(random.random())
url = '%s%s'%(HOT_HANDLER_PREFIX, next_token)
next_task = taskqueue.Task(countdown=10, url=url)
hot_handler_queue.add(next_task)
memcache.set(HOT_HANDLER_PREFIX, next_token)
res('200 OK',[('Content-Type','text/plain')])
return ['ok']
def main():
CGIHandler().run(wsgi_app)

if __name__ == '__main__':
main()




### how to launch
1) visit "/_ah/queue/hothandler/start" in your browser
2) login as an admin which inserts the first task
3) view your logs

By the way this is on the production server, the dev server does not
run the tasks automatically.

### view your logs

10-29 10:28PM 11.010 /_ah/queue/hothandler/... 200 17ms 7cpu_ms
10-29 10:28PM 01.018 /_ah/queue/hothandler/... 200 28ms 9cpu_ms
10-29 10:27PM 51.007 /_ah/queue/hothandler/... 200 16ms 5cpu_ms
10-29 10:27PM 41.011 /_ah/queue/hothandler/... 200 18ms 5cpu_ms
10-29 10:27PM 31.007 /_ah/queue/hothandler/... 200 38ms 14cpu_ms
10-29 10:27PM 21.009 /_ah/queue/hothandler/... 200 23ms 7cpu_ms
10-29 10:27PM 11.009 /_ah/queue/hothandler/... 200 20ms 7cpu_ms
10-29 10:27PM 01.009 /_ah/queue/hothandler/... 200 20ms 10cpu_ms
10-29 10:26PM 51.007 /_ah/queue/hothandler/... 200 44ms 35cpu_ms
10-29 10:26PM 41.017 /_ah/queue/hothandler/... 200 24ms 12cpu_ms

Literally like clockwork, a task hits the app once every 10 seconds and keeps it hot.

6 req/minute * 60 minutes/hour * 24 hours/day = 8640 req/day

It is a very small price to pay for good response time with a low volume app.

Memcache is used to ensure there is at most 1 active task, and cron is used
to ensure there is at least 1 active task every 4 hours, so no maintenance.

CapabilitySet is used to test if memcache is available.

Apparently, if 1 handler remains warm, all other warm handlers remain warm too, so
hothandler can exist in its own handler called hothandler.py, which is more modular.
Posted by jamespcooper on Mon 26 Oct 2009 in Java
If JSTL expressions in your JSPs aren't working, try adding this to the JSP file:

<%@ page contentType="text/html;charset=UTF-8" language="java" isELIgnored="false" %>

The key part is: isELIgnored="false"

Thanks to this blog for the insight:
http://www.digitalsanctum.com/2009/06/24/jstl-expressions-not-working-in-google-app-engine-solution/

-- James
Posted by david@twitmart.org on Fri 16 Oct 2009 in Java
This small package (less than 1000 lines of Java code) was developed to provide a small and simple package for rendering text into an image (bitmap). It was developed for Google App Engine use where AWT and BufferedImage et al do not exist. This small utility can be used to render text on App Engine within the constraints of the JRE Class White List. This is derived from the "pbmtext" utility from NETpbm Copyright (C) 1991 by Jef Poskanzer.

These are crude black-and-white bitmap fonts - no anti-aliasing. A small number of BDF fonts in fixed-sizes are bundled into the package: Courier, Lucida-Bold-Italic, Lucida-Medium, Lucida-Medium-Italic, LucidaBright-DemiBold, LucidaTypewriter, LucidaTypewriter-Bold, and a default serif font.

The following code snippet demonstrates use of litetext:

    byte[] bmp_data = doRender("Render this text into a bitmap");
// bmp_data contains a BMP file



http://code.google.com/p/litetext/
Posted by ethan.vandenberg on Thu 15 Oct 2009 in Python
def run_tasks(request):
"""
KLUDGE!!
Provides automatic task execution on dev sdk.
1. Setup this view func under the url /run_tasks.
2. If you aren't using django, modify end. (HttpResponse and redirect)
3. Run the following in a shell (you may need to change to the port number):
while true; do \
wget -qO - http://localhost:8085/run_tasks/ | wget --keep-session-cookies -i - --delete-after --wait=5; \
sleep 60; \
done
Todo:
- Add support for tasks that POST
- Add support for rate (requests per second)
"""
import os
if os.environ['SERVER_SOFTWARE'].startswith('Development'):
from datetime import datetime
from google.appengine.api import apiproxy_stub_map
stub = apiproxy_stub_map.apiproxy.GetStub('taskqueue')
tasks = []
#get all the tasks for all the queues
for queue in stub.GetQueues():
tasks.extend( stub.GetTasks(queue['name']) )
#keep only tasks that need to be executed
now = datetime.now()
tasks = filter(lambda t: datetime.strptime(t['eta'],'%Y/%m/%d %H:%M:%S')

def run_tasks(request):
"""
KLUDGE!!
Provides automatic task execution on dev sdk.
1. Setup this view func under the url /run_tasks.
2. If you aren't using django, modify end. (HttpResponse and redirect)
3. Run the following in a shell (you may need to change to the port number):
while true; do \
wget -qO - http://localhost:8085/run_tasks/ | wget --keep-session-cookies -i - --delete-after --wait=5; \
sleep 60; \
done
Todo:
- Add support for tasks that POST
- Add support for rate (requests per second)
"""
import os
if os.environ['SERVER_SOFTWARE'].startswith('Development'):
from datetime import datetime
from google.appengine.api import apiproxy_stub_map
stub = apiproxy_stub_map.apiproxy.GetStub('taskqueue')
tasks = []
#get all the tasks for all the queues
for queue in stub.GetQueues():
tasks.extend( stub.GetTasks(queue['name']) )
#keep only tasks that need to be executed
now = datetime.now()
tasks = filter(lambda t: datetime.strptime(t['eta'],'%Y/%m/%d %H:%M:%S')