The code is still pretty rough, but might be of interest to someone hacking on AMQP-based Twisted services.

Sample Usage

1. Create a new service and change to the project directory.

   $ txsrv create mytest
   $ cd mytest
   

2. Edit the mytest.conf file so that the spec_file option is valid.

   [connection:amqp]
   type = amqp
   host = localhost
   port = 5672
   vhost = /
   user = guest
   password = guest
   spec_file = /usr/share/amqp/amqp.0-8.xml
   
   [handler:hello]
   connection = amqp
   exchange = hello
   routing_key = default
   

3. Edit the mytest.py file so that it prints the message body twice.

   import txsrv
   
   class Service(txsrv.Service):
       @txsrv.handler('hello')
       def hello(self, message):
           print message.body * 2
   
   class ServiceMaker(txsrv.ServiceMaker):
       tapname = 'mytest'
       description = 'A mytest txsrv example.'
       service_type = Service
   

4. Start the mytest service and send a message to the hello exchange.

   $ twistd -n mytest -c mytest.conf
   2010-09-12 00:41:10-0400 [-] Log opened.
   2010-09-12 00:41:10-0400 [-] twistd 10.1.0 (/usr/bin/python 2.6.4) starting up.
   2010-09-12 00:41:10-0400 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
   2010-09-12 00:41:10-0400 [-] Starting factory <txsrv.protocol.amqp.AmqpFactory instance at 0x2cce950>
   2010-09-12 00:41:10-0400 [AmqpProtocol,client] hellohello
   

I created restarter because there isn't auto-reload functionality built into node.js (at the time of this post).

Example

[silas@blackbox keyfu.js]$ restarter node keyfu.js
Express started at http://localhost:3000/ in development mode

Usage

Usage: restarter [options] command

Options:
  -h, --help     show this help message and exit
  --path=PATH    directory to watch for file event changes
  --ignore=PATH  specifies events to ignore

import jinja2
import jinja2.ext
import markdown2

class Markdown2Extension(jinja2.ext.Extension):
    tags = set(['markdown2'])

    def __init__(self, environment):
        super(Markdown2Extension, self).__init__(environment)
        environment.extend(
            markdowner=markdown2.Markdown()
        )   

    def parse(self, parser):
        lineno = parser.stream.next().lineno
        body = parser.parse_statements(
            ['name:endmarkdown2'],
            drop_needle=True
        )
        return jinja2.nodes.CallBlock(
            self.call_method('_markdown_support'),
            [],
            [],
            body
        ).set_lineno(lineno)

    def _markdown_support(self, caller):
        return self.environment.markdowner.convert(caller()).strip()

env = jinja2.Environment(extensions=[Markdown2Extension])

text = """ 
{% markdown2 %}
Hello World
===========

 1. One
 2. {{ two }}
 3. Three
{% endmarkdown2 %}
"""

html = env.from_string(text).render(two='Two')

print html

Which would result in the following output:

<h1>Hello World</h1>

<ol>
<li>One</li>
<li>Two</li>
<li>Three</li>
</ol>

I created gaem because I wanted to host my blog on an unreliable server and let Google App Engine serve the content while my server was down or in maintenance mode.

The basic design is similar to that of Django's middleware. A request travels through a list of components which can either manipulate the request or provide a response. The response is filtered back through the components and served to the requester.

A basic implementation looks something like

import gaem
import gaem.middleware
import gaem.middleware.cache
import gaem.middleware.dos
import gaem.middleware.fetch

URL = 'http://www.example.org'

MIDDLEWARE_CLASSES = (
    gaem.middleware.cache.Memcache(),
    gaem.middleware.cache.Datastore(),
    gaem.middleware.dos.DoS(),
    gaem.middleware.fetch.Fetch(url=URL),
)

application = gaem.Application(MIDDLEWARE_CLASSES)

if __name__ == '__main__':
    application.run()

When a request first hits this application it is handled by the Memcache component. This basic caching component checks the Google App Engine Memcache service to see if the page is in the cache. If so it simply returns the cached response, otherwise the request proceeds to the Datastore component which does pretty much the same thing except it uses Google Datastore for persistence. If both caching components fail the the request goes through a DoS protection component and is finally retrieved from the backend server by the Fetch component.

Using a setup like this gaem acts as a rudimentary reverse proxy and while most of the functionality I've implemented thus far is for this type setup, I plan on adding features not traditionally found in a reverse proxy.

Current Features

• Caching
• Backend fetch
• Gracefully handle a dead backend
• DoS protection
• API for expiring content

Possible Future Features

• Initial and periodic crawler (pre-cache and other hooks)
• RESTful API for managing content (put, delete, etc...)
• Web interface
• XSS protection
• Some ESI support

But what makes gaem really cool for developers is how easy it is to create custom components.

A basic component looks like

from gaem import middleware

class DoesNothing(middleware.Base):

    def process_request(self, request):
        return request

    def process_response(self, response, request):
        return response

A default gaem component should inherit from gaem.middleware.Base and implement process_request and/or process_response.

The process_request function lets you either manipulate the request (ex: strip the "cookie" header) and pass it along or use the request to create and return a response (ex: do a fetch to the backend server). The process_response function lets you filter the response (ex: translate body into pig latin) and do side affects (ex: store response in memcache) before the response is sent to the requester.

The Request/Response objects are borrowed from WebOb and can be manipulated using the interfaces defined in the WebOb documentation.

gaem is currently pretty rough and likely to change over the next couple of months, but if you're a Python hacker I'd love to hear some feedback (contact me or submit an issue).

P.S. You can see the application I created for this site in the examples directory.

1. Install requirements

   $ sudo yum install python-twisted python-stomper
   

2. Checkout stomping

   $ svn export http://silassewell.googlecode.com/svn/trunk/projects/python-stomping python-stomping
   $ cd python-stomping
   

3. Update the example.py file to use your host, port, username and password ActiveMQ/Stomp broker settings

   from stomping import Stomping, route
   
   class MyStomping(Stomping):
   
   def init(self):
       self.send('/queue/test1', 'init test1')
       self.send('/queue/test2', 'init test2')
   
   @route('/queue/test1')
   def a_test(self, message):
       print 'a_test: %s' % message['body']
   
   @route('/queue/test1')
   @route('/queue/test2')
   def b_test(self, message):
       print 'b_test: %s' % message['body']
   
   if __name__ == '__main__':
       stomp = MyStomping(host='127.0.0.1', port=61613, username='guest', password='guest')
       stomp.run()
   

4. Run example.py (Ctrl+c to quit)

   $ python example.py 
   a_test: init test1
   b_test: init test1
   b_test: init test2
   

NOTE: python-stomping is just a light wrapper around Twisted and stomper and is based on an example provided in the stomper code.

1. Build and install the python-line_profiler package

2. Create a file called test.py with the code below

   import random, time
   
   def sleep():
       seconds = random.randint(0, 5)
       print 'Sleeping %s seconds' % seconds
       time.sleep(seconds)
   
   @profile
   def test():
       sleep()
       sleep()
       sleep()
   
   test()
   

3. Profile test.py

   [silas@silas ~]$ kernprof.py -l test.py
   Sleeping 4 seconds
   Sleeping 5 seconds
   Sleeping 2 seconds
   Wrote profile results to test.py.lprof
   

4. View the results

   [silas@silas ~]$ python -m line_profiler test.py.lprof
   Timer unit: 1e-06 s
   
   File: test.py
   Function: test at line 8
   Total time: 10.9994 s
   
   Line #      Hits         Time  Per Hit   % Time  Line Contents
   ==============================================================
        8                                           @profile
        9                                           def test():
       10         1      3999416 3999416.0     36.4      sleep()
       11         1      4999982 4999982.0     45.5      sleep()
       12         1      1999990 1999990.0     18.2      sleep()
   

NOTE: I have a package review up for line_profiler and it should be available via yum eventually.

Examples

[silas@pluto ~]$ string_slice "12345" 0 1
1
[silas@pluto ~]$ string_slice "12345" 0 3
123
[silas@pluto ~]$ string_slice "12345" 2 3
3
[silas@pluto ~]$ string_slice "12345" 2 -2
3
[silas@pluto ~]$ string_slice "12345" -3
345

Implementation

function string_slice {
    STRING="$1"
    declare -i LENGTH="${#STRING}"
    declare -i START="$2"
    declare -i END="$3"
    if [ $START -lt 0 ]; then
        START=$[ $LENGTH + $START ]
    fi
    if [ $END -le 0 ]; then
        END=$[ $LENGTH + $END ]
    fi
    START=$[ $START + 1 ]
    (echo "$STRING" | cut -c $START-$END) 2> /dev/null
}

I created funcshell for running Func modules in a more intuitive manner.

The current implementation only supports the command module, but more will be released shortly.

Code: http://github.com/silas/funcshell

Example Usage

[root@pluto ~]# python shell.py
fs> use web*.example.org
fs> get hosts
web01.example.org
web02.example.org
fs> !du -sh /tmp
================================================================================
== web02.example.org                                                          ==
================================================================================

236K    /tmp

================================================================================
== web01.example.org                                                          ==
================================================================================

217M    /tmp

fs> !cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'
================================================================================
== web02.example.org                                                          ==
================================================================================

1027116

================================================================================
== web01.example.org                                                          ==
================================================================================

1027116

fs> exit
[root@pluto ~]#

~/.bashrc

function python {
    IPYTHON="/usr/bin/ipython"
    PYTHON="/usr/bin/python"

    if [[ -n $1 ]]; then
        $PYTHON $@
    elif [[ -e $IPYTHON ]]; then
        $IPYTHON
    else
        $PYTHON
    fi
}

You can accomplish theses tasks via the command line interface

func web-*.example.net call service restart httpd

or through the Python API

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

print client.service.restart('httpd')

One of the annoying issues I'm dealing with right now is the distribution of Func modules, which for various reasons isn't as simple as packaging the modules and pushing them to all the hosts (it is, just not in my network).

My solution is Push, a Func module which lets you instantly run Python code on any or all hosts in your network.

Example usage:

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

source = """
def main():
    return 'Hello World'
"""

print client.push.code(source)

I realize this isn't the most elegant solution, but its very useful and extremely simple.

I also realize that I could use the copyfile module to push modules via Func, but I haven't found a way to restart funcd via Func (required to initialize modules), so until that happens I need a solution which doesn't require restarting Func.

Get Push Module