Google App Engine Middleware (gaem)
Google App Engine Middleware (gaem) is a collection of utilities for hosting external sites on Google App Engine.
I created gaem because I wanted to host my blog on an unreliable server and let Google App Engine serve the content while my server was down or in maintenance mode.
The basic design is similar to that of Django's middleware. A request travels through a list of components which can either manipulate the request or provide a response. The response is filtered back through the components and served to the requester.
A basic implementation looks something like
import gaem
import gaem.middleware
import gaem.middleware.cache
import gaem.middleware.dos
import gaem.middleware.fetch
URL = 'http://www.example.org'
MIDDLEWARE_CLASSES = (
gaem.middleware.cache.Memcache(),
gaem.middleware.cache.Datastore(),
gaem.middleware.dos.DoS(),
gaem.middleware.fetch.Fetch(url=URL),
)
application = gaem.Application(MIDDLEWARE_CLASSES)
if __name__ == '__main__':
application.run()
When a request first hits this application it is handled by the Memcache component. This basic caching component checks the Google App Engine Memcache service to see if the page is in the cache. If so it simply returns the cached response, otherwise the request proceeds to the Datastore component which does pretty much the same thing except it uses Google Datastore for persistence. If both caching components fail the the request goes through a DoS protection component and is finally retrieved from the backend server by the Fetch component.
Using a setup like this gaem acts as a rudimentary reverse proxy and while most of the functionality I've implemented thus far is for this type setup, I plan on adding features not traditionally found in a reverse proxy.
Current Features
- Caching
- Backend fetch
- Gracefully handle a dead backend
- DoS protection
- API for expiring content
Possible Future Features
- Initial and periodic crawler (pre-cache and other hooks)
- RESTful API for managing content (put, delete, etc...)
- Web interface
- XSS protection
- Some ESI support
But what makes gaem really cool for developers is how easy it is to create custom components.
A basic component looks like
from gaem import middleware
class DoesNothing(middleware.Base):
def process_request(self, request):
return request
def process_response(self, response, request):
return response
A default gaem component should inherit from gaem.middleware.Base and
implement process_request and/or process_response.
The process_request function lets you either manipulate the request (ex:
strip the "cookie" header) and pass it along or use the request to create and
return a response (ex: do a fetch to the backend server). The
process_response function lets you filter the response (ex: translate body
into pig latin) and do side affects (ex: store response in memcache) before the
response is sent to the requester.
The Request/Response objects are borrowed from WebOb and can be manipulated using the interfaces defined in the WebOb documentation.
gaem is currently pretty rough and likely to change over the next couple of months, but if you're a Python hacker I'd love to hear some feedback (contact me or submit an issue).
P.S. You can see the application I created for this site in the examples directory.