I created gaem because I wanted to host my blog on an unreliable server and let Google App Engine serve the content while my server was down or in maintenance mode.

The basic design is similar to that of Django's middleware. A request travels through a list of components which can either manipulate the request or provide a response. The response is filtered back through the components and served to the requester.

A basic implementation looks something like

import gaem
import gaem.middleware
import gaem.middleware.cache
import gaem.middleware.dos
import gaem.middleware.fetch

URL = 'http://www.example.org'

MIDDLEWARE_CLASSES = (
    gaem.middleware.cache.Memcache(),
    gaem.middleware.cache.Datastore(),
    gaem.middleware.dos.DoS(),
    gaem.middleware.fetch.Fetch(url=URL),
)

application = gaem.Application(MIDDLEWARE_CLASSES)

if __name__ == '__main__':
    application.run()

When a request first hits this application it is handled by the Memcache component. This basic caching component checks the Google App Engine Memcache service to see if the page is in the cache. If so it simply returns the cached response, otherwise the request proceeds to the Datastore component which does pretty much the same thing except it uses Google Datastore for persistence. If both caching components fail the the request goes through a DoS protection component and is finally retrieved from the backend server by the Fetch component.

Using a setup like this gaem acts as a rudimentary reverse proxy and while most of the functionality I've implemented thus far is for this type setup, I plan on adding features not traditionally found in a reverse proxy.

Current Features

• Caching
• Backend fetch
• Gracefully handle a dead backend
• DoS protection
• API for expiring content

Possible Future Features

• Initial and periodic crawler (pre-cache and other hooks)
• RESTful API for managing content (put, delete, etc...)
• Web interface
• XSS protection
• Some ESI support

But what makes gaem really cool for developers is how easy it is to create custom components.

A basic component looks like

from gaem import middleware

class DoesNothing(middleware.Base):

    def process_request(self, request):
        return request

    def process_response(self, response, request):
        return response

A default gaem component should inherit from gaem.middleware.Base and implement process_request and/or process_response.

The process_request function lets you either manipulate the request (ex: strip the "cookie" header) and pass it along or use the request to create and return a response (ex: do a fetch to the backend server). The process_response function lets you filter the response (ex: translate body into pig latin) and do side affects (ex: store response in memcache) before the response is sent to the requester.

The Request/Response objects are borrowed from WebOb and can be manipulated using the interfaces defined in the WebOb documentation.

gaem is currently pretty rough and likely to change over the next couple of months, but if you're a Python hacker I'd love to hear some feedback (contact me or submit an issue).

P.S. You can see the application I created for this site in the examples directory.

1. Install requirements

   $ sudo yum install python-twisted python-stomper
   

2. Checkout stomping

   $ svn export http://silassewell.googlecode.com/svn/trunk/projects/python-stomping python-stomping
   $ cd python-stomping
   

3. Update the example.py file to use your host, port, username and password ActiveMQ/Stomp broker settings

   from stomping import Stomping, route
   
   class MyStomping(Stomping):
   
   def init(self):
       self.send('/queue/test1', 'init test1')
       self.send('/queue/test2', 'init test2')
   
   @route('/queue/test1')
   def a_test(self, message):
       print 'a_test: %s' % message['body']
   
   @route('/queue/test1')
   @route('/queue/test2')
   def b_test(self, message):
       print 'b_test: %s' % message['body']
   
   if __name__ == '__main__':
       stomp = MyStomping(host='127.0.0.1', port=61613, username='guest', password='guest')
       stomp.run()
   

4. Run example.py (Ctrl+c to quit)

   $ python example.py 
   a_test: init test1
   b_test: init test1
   b_test: init test2
   

NOTE: python-stomping is just a light wrapper around Twisted and stomper and is based on an example provided in the stomper code.

• advanced client selection (you can use + and - to add/remove hosts/groups)
• tab completion
• persistent history
• a command sub-shell
• quick help (just hit the "?" key)

funcshell took a similar approach to features as Func; it allows developers to easily extend and customize its functionality using a plugin system. It also assumes the most useful commands will be the ones you write yourself. That being said, funcshell currently comes with both the command and service module.

Here are some things you can do with funcshell.

Get a list of all clients (not very useful if you have hundreds):

funcshell> set clients *
funcshell> get clients
webapp001.example.net
webapp002.example.net
webapp003.example.net
webstatic001.example.net
webstatic002.example.net

Assuming webapp00[1-3].example.net are in the app and bpstatic00[1-2].example.net1 are in the static groups; remove some clients and restart a service:

funcshell> set clients - @static;webapp001*
funcshell> get clients
webapp002.example.net
webapp003.example.net
funcshell> service httpd restart
==> webapp003.example.net <==
True
==> webapp002.example.net <==
True

Run a command and get the results:

funcshell> set clients - webapp003*
funcshell> set clients + webstatic002*
funcshell> command run grep MemTotal /proc/meminfo | awk '{ print $2/1024 }'
==> webstatic002.example.net :: 0 <==
4099.44
==> webapp002.example.net :: 0 <==
8198.88

Get help:

funcshell> command <TAB><TAB>
exists   run      shell
funcshell> command ?
  exists Check if a command exists
  run    Run a command
  shell  Run a command shell

Run a couple of commands in a row:

funcshell> set c<TAB> @static
funcshell> command shell
> du -sh /tmp
==> webstatic002.example.net :: 0 <==
7.2M    /tmp
==> webstatic001.example.net :: 0 <==
3.4M    /tmp
> cp -r /tmp /tmp
==> webstatic002.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
==> webstatic001.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
> asdf
==> webstatic002.example.net :: 127 <==
/bin/sh: asdf: command not found
==> webstatic001.example.net :: 127 <==
/bin/sh: asdf: command not found
> <CTRL-D>
funcshell> exit

funcshell is a shell interface to Func which provides some useful features for managing a large number of machines.

Features

• Centralized configuration management
• Files live in their native locations (no symbolic linking)
• Home directory is not a Git repository
• All the power of git with a simple alias

Setup Repository

• Log into GitHub and create a repository named config
• Add your public keys to GitHub (if you haven't done so already)

• Open a terminal and switch to your home directory

  cd ~
  

• Create a configuration directory

  mkdir .config.git
  

• Add the following alias to your current session and your .bash_profile

  alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'
  echo "alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'" >> .bash_profile
  

• Add your .bash_profile to the configuration repository

  config add .bash_profile
  

• Commit the changes

  config commit -m 'Initial commit'
  

• Change the origin to GitHub

  config remote add origin git@github.com:GITHUB_USERNAME/config.git
  

• Push the changes

  config push origin master
  

If you get an error when running config pull to the effect of You asked me to pull without... run the follow:

    echo -e '[branch "master"]\n  remote = origin\n  merge = refs/heads/master' >> ~/.config.git/config

Setup Configuration Management on a Different System

1. Add your public keys to GitHub (if you haven't done so already)

2. Switch to your home directory

   cd ~
   

3. Backup your local configuration files, example:

   mv .bash_profile .bash_profile.bk
   

4. Clone your configuration repository

   git clone git@github.com:GITHUB_USERNAME/config.git config.git
   

5. Move the git metadata to ~/.config.git

   mv config.git/.git .config.git
   

6. Enable dotglob

   shopt -s dotglob
   

7. Move your configuration files to your home directory

   mv -i config.git/* .
   

8. Delete the config.git directory

   rmdir config.git
   

9. Logout and log back in

Basic Usage

• config pull - get latest configuration changes
• config add FILENAME - add a configuration file
• config commit -a - save all configuration changes
• config push - push configuration changes to GitHub
• and any other config GIT_OPTION

You can see my configuration repository at http://github.com/silas/config.

Source: Manage your $HOME with git by Robert Escriva

I created funcshell for running Func modules in a more intuitive manner.

The current implementation only supports the command module, but more will be released shortly.

Code: http://github.com/silas/funcshell

Example Usage

[root@pluto ~]# python shell.py
fs> use web*.example.org
fs> get hosts
web01.example.org
web02.example.org
fs> !du -sh /tmp
================================================================================
== web02.example.org                                                          ==
================================================================================

236K    /tmp

================================================================================
== web01.example.org                                                          ==
================================================================================

217M    /tmp

fs> !cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'
================================================================================
== web02.example.org                                                          ==
================================================================================

1027116

================================================================================
== web01.example.org                                                          ==
================================================================================

1027116

fs> exit
[root@pluto ~]#

The end result was the Common Usage Database and like Most Common Usage the website is focused on the usage of CLI commands in Linux and Unix-like operating systems.

The website uses Google for authentication so you don't need to signup if you already have a Google account. I've also setup a Google Group and the #cudb channel on Freenode.

I look forward to creating a repository of useful commands and providing an easy and open way for developers to use that repository.

Link: http://www.cudb.org/

Link: http://github.com/silas/graveyard/tree/master/python-phonefactor/

Let's say you use GoDaddy as your domain registrar and you would like to see if a domain is available using Firefox keywords, but they don't offer a GET search.

For a traditional keyword, you could construct a GET request like:

https://www.godaddy.com/gdshop/registrar/search.asp&checkAvail=1&fblur=1&tld=.com&domainToCheck=%s

But as of right now Firefox doesn't offer a way to create POST keywords. What Post a Fix does is allow you to construct a POST request using GET syntax and pass it to the Post a Get code.

So if you wanted to submit the above GET as a POST, you would construct the URL and append the special __action parameter with target URL. So the final URL would look something like: