1. Create an SSH public/private key pair

   [user@client ~]$ ssh keygen -t rsa
   

2. Upload the public key to the server

   [user@client ~]$ scp ~/.ssh/id_rsa.pub user@server.example.org:/tmp/user.pub
   

3. SSH to the server and become root

   [user@client ~]$ ssh user@server.example.org
   user@server:~$ sudo su -
   

4. Install gitolite

   root@server:~# apt-get install gitolite
   

5. Switch to the gitolite user

   root@server:~# su - gitolite
   

6. Run the Gitolite setup script

   gitolite@server:~$ gl-setup /tmp/user.pub
   

7. From your client computer clone the gitolite-admin repository

   [user@client ~]$ git clone gitolite@server.example.org:gitolite-admin
   Cloning into gitolite-admin...
   remote: Counting objects: 6, done.
   remote: Compressing objects: 100% (4/4), done.
   remote: Total 6 (delta 0), reused 0 (delta 0)
   Receiving objects: 100% (6/6), done.
   

8. Switch to the gitolite-admin directory

   [user@client ~]$ cd gitolite-admin
   

9. Create a test repository

   [user@client gitolite-admin]$ vim conf/gitolite.conf
   [user@client gitolite-admin]$ git commit -a -m "Add test repository"
   [user@client gitolite-admin]$ git a
   [master 507045a] Add test repository
    1 files changed, 3 insertions(+), 0 deletions(-)
   [user@client gitolite-admin]$ git push
   Counting objects: 7, done.
   Delta compression using up to 4 threads.
   Compressing objects: 100% (3/3), done.
   Writing objects: 100% (4/4), 376 bytes, done.
   Total 4 (delta 1), reused 0 (delta 0)
   remote: Already on 'master'
   remote: creating test...
   remote: Initialized empty Git repository in /var/lib/gitolite/repositories/test.git/
   To gitolite@server.example.org:gitolite-admin
      87cc470..507045a  master -> master
   

10. Clone the test repository

    [user@client ~]$ git clone gitolite@server.example.org:test
    Cloning into test...
    warning: You appear to have cloned an empty repository.
    

11. Add a README file to the test repository

    [user@client test]$ echo "Test Repo" > README
    [user@client test]$ git commit -a -m "Initial commit"
    [master (root-commit) 4a49ee0] Initial commit
     1 files changed, 1 insertions(+), 0 deletions(-)
     create mode 100644 README
    

12. Push the changes to the server

    [user@client test]$ git push origin master
    Counting objects: 3, done.
    Writing objects: 100% (3/3), 218 bytes, done.
    Total 3 (delta 0), reused 0 (delta 0)
    To gitolite@server.example.org:test
     * [new branch]      master -> master
    

Check out the Gitolite wiki for more documentation.

The code is still pretty rough, but might be of interest to someone hacking on AMQP-based Twisted services.

Sample Usage

1. Create a new service and change to the project directory.

   $ txsrv create mytest
   $ cd mytest
   

2. Edit the mytest.conf file so that the spec_file option is valid.

   [connection:amqp]
   type = amqp
   host = localhost
   port = 5672
   vhost = /
   user = guest
   password = guest
   spec_file = /usr/share/amqp/amqp.0-8.xml
   
   [handler:hello]
   connection = amqp
   exchange = hello
   routing_key = default
   

3. Edit the mytest.py file so that it prints the message body twice.

   import txsrv
   
   class Service(txsrv.Service):
       @txsrv.handler('hello')
       def hello(self, message):
           print message.body * 2
   
   class ServiceMaker(txsrv.ServiceMaker):
       tapname = 'mytest'
       description = 'A mytest txsrv example.'
       service_type = Service
   

4. Start the mytest service and send a message to the hello exchange.

   $ twistd -n mytest -c mytest.conf
   2010-09-12 00:41:10-0400 [-] Log opened.
   2010-09-12 00:41:10-0400 [-] twistd 10.1.0 (/usr/bin/python 2.6.4) starting up.
   2010-09-12 00:41:10-0400 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
   2010-09-12 00:41:10-0400 [-] Starting factory <txsrv.protocol.amqp.AmqpFactory instance at 0x2cce950>
   2010-09-12 00:41:10-0400 [AmqpProtocol,client] hellohello
   

1. Download the x86_64 (or i386) RPM.

2. Install the Go package using yum.

   sudo yum localinstall --nogpgcheck go-0.0.0*.rpm
   

3. Open a new shell.

4. Create a file named hello.go and enter the following code:

   package main
   
   import "fmt"
   
   func main() {
     fmt.Printf("Hello World!")
   }
   

5. Compile, link and run (replace 6 with 8 if you're using i386).

   $ 6g hello.go
   $ 6l hello.6
   $ ./6.out
   

You can view the latest RPM source on GitHub.

Update: It looks like someone has already created a much better package.

1. Create a .screen directory in your home directory.

   mkdir "$HOME/.screen"
   

2. Add the following to your .bashrc or .bash_profile.

   sp() {
     if [[ -f "$HOME/.screen/$1" ]]; then
       screen -c "$HOME/.screen/$1"
     else
       echo "Unknown screen profile '$1'."
     fi  
   }
   

3. Create a file in the ~/.screen directory with whatever profile name you would like to use (below is a sample profile named tyrion).

   You should copy the first line into each profile as it loads your default GNU Screen settings.

   source "$HOME/.screenrc"
   
   chdir "$HOME/src/tyrion/src"
   screen -t code  0
   
   chdir "$HOME/src/tyrion"
   screen -t build  1
   
   chdir "$HOME/src/tyrion/tests"
   screen -t tests  2
   
   chdir "$HOME/src/tyrion"
   

4. Reload your shell and test your profile (replace tyrion with whatever name you picked).

   [silas@example ~]$ sp tyrion
   

Tyrion is in short a lightweight Func replacement.

Why Tyrion?

I created Tyrion because Func didn't meet some of my needs, specifically the points outlined below.

(1) Asynchronous

Tyrion was built using a protocol which is inherently asynchronous. This has lead to lower latency operations and better scalability.

That isn't to say Func couldn't be updated to be asynchronous (using the Twisted XML-RPC libraries for example), but I think current toolsets highly favor protocols which were designed from the beginning to be asynchronous.

(2) Language Agnostic

From a technical perspective Func is also language agnostic (it uses XML-RPC as a transport protocol) and although you could write a client or server in any language, from a pragmatic standpoint you're stuck writing services in Python.

On the other hand Tyrion is written in C++ and each service is a simple executable. This lends itself nicely to writing services in whatever language is most appropriate.

(3) Low memory footprint

Func is written in Python and as such suffers from a dynamic language's higher memory usage.

For example, the following Python application uses ~4.2MB memory on my Fedora 13 box.

#!/usr/bin/env python

import time

time.sleep(60)

While the complete tyrion-node application uses ~4.5MB.

For many systems administrators the difference between Func and tyrion-node's memory usage will be negligible, but if you running an infrastructure with several hundred or several thousand hosts the resource difference can be enormous.

A lower memory footprint is also advantageous to companies who heavily utilize virtualization. If you have 5 servers and you slice those up into 4 virtual machines each, you've just jumped from 5 to 20 daemons. Lets say Func uses 15MB memory, that's a 60MB to 300MB jump compared to tyrion-node's 22.5MB to 90MB.

(4) Permissions

The protocol that Tyrion uses has built-in support for users. This allows Tyrion to support user specific permissions on a service level and opens up some really cool possibility, such as exposing services to partially trusted users.

In addition because Tyrion uses a process model to run each service, a configuration setting can be defined to specify under which user and group a service is run.

(5) Cloud

This is a somewhat stupid bullet, but it does have some credibility. Func was created to rely on a hostname/certificate combindation and the assumption that we could communicate to a managed host on a defined port. This setup works great when you have full control over your network.

Unfortunately many of us use external resources (AWS, Linode, Slicehost, etc...) where some or much of the network and infrastructure configuration is out of our control.

Tyrion was built on the XMPP protocol, this means that Tyrion connects to a XMPP server and just listens for events.

With XMPP we also get authentication, an encrypted transport, some decentralization, federation, a solid selection of open source and scalable servers and clients for pretty much every language.

(6) Hot Reloads

The ability to update configuration settings and install services without a daemon restart.

Overview

This will briefly overview how Tyrion works.

The Tyrion package has two primary parts, the tyrion client and the tyrion-node.

tyrion-node is daemon which runs on each managed server. It boots with the host, connects to an XMPP server and listens for service requests.

tyrion-node Configuration

Below is the general configuration settings for tyrion-node which are located in the node.conf file:

[general]
acl_path = /etc/tyrion/acl.conf                 ; ACL configuration file
log_path = /var/log/tyrion/node.log             ; Logging path
log_level = info                                ; Logging level
service_path = /usr/share/tyrion/service        ; Service directory

[xmpp]
jid = test.node@example.org/tyrion              ; Tyrion node JID
password = test.node.pass                       ; Tyrion node password
;server = example.org                            ; Server address if different from JID domain

The node.conf file also lets you define default values for installed services.

[service:org.tyrion.service.bash]
timeout = 300                                   ; Default service timeout (seconds)
timeout_lock = false                            ; Lock default timeout setting
user = mike                                     ; Default user to run service as
user_lock = false                               ; Lock default user setting

This tells Tyrion that the org.tyrion.service.bash service should have a default timeout of 300 seconds and run as the user mike. In this case because we haven't locked the timeout or user option they can be overriden by the requester.

tyrion-node Service

If you were to look in the service_path directory you would see an executable file named org.tyrion.service.bash.

[silas@example.org]$ cat /usr/share/tyrion/service/org.tyrion.service.bash 
#!/usr/bin/env sh

/usr/bin/env bash

Obviously this service is a bit redundant and not all that interesting, but serves as a good example of how a service works.

tyrion Client

First we'll cover the basics of the tyrion client. Then we'll use that information to make a basic service request and follow that request from the client to the node and back again.

tyrion Client Configuration

First the client.conf file:

[xmpp]
jid = test.client@example.org                   ; Tyrion node JID
password = test.client.pass                     ; Tyrion node password
;server = example.org                            ; Server address if different from JID domain

[profile:test]
jid = test.node@example.org/tyrion              ; Default destination JID
service = org.tyrion.service.bash               ; Default service type
timeout = 300                                   ; Default timeout
user = usertest                                 ; Default user
group = grouptest                               ; Default group

We can see that like the node.conf file we have our XMPP options under the xmpp section. Below that we also have a default profile called test. Similiar to the service section in the node.conf file this lets us setup some basic defaults. Here we're defining a destination jid, service, timeout, user and group.

tyrion Client Options

Additionally if we ran the help option on tyrion we would see the following:

[silas@example.org]$ tyrion --help
Usage: tyrion [OPTION]...
Example: tyrion -c client.conf

Configuration options:
  -c, --config-file         the node configuration file
  -p, --profile             default service options

Service options:
  -j, --jid                 destination JID(s)
  -s, --service             service type
  -t, --timeout             max service run time
  -u, --user                run service as user
  -g, --group               run service as group

Misc options:
  --debug                   show debug information

tyrion Client Usage

So an example usage of tyrion might be:

[silas@example.org]$ echo "echo test" | tyrion -c client.conf -p test
test.node@example.org/tyrion (0): test

Here we've run tyrion with the client.conf configuration file and the profile test.

We sent echo test to the org.tyrion.service.bash service and got the exit code 0 and the result test.

tyrion-node Handler

From the other side tyrion-node sees a request from test.client@example.org to use the service org.tyrion.service.bash.

[org.tyrion.service.bash]
test.client@example.org = true

[org.tyrion.service.python]
test.client@example.org = true

[org.tyrion.service.ruby]
test.client@example.org = true

After validating that request against the above acl_path file it runs the /usr/share/tyrion/service/org.tyrion.service.bash executable and pipes echo test to stdin.

When the service request completes or reaches the defined timeout it sends the exit code, stdout and stderr back to the client.

txtyrion

The tyrion client is really just for testing and one-off hacks. It not suppose to be used in scripts or anything in production.

Because of this I created txtyrion, a Twisted-based Tyrion client library.

A simple client application might look like this:

from twisted.internet import defer, reactor

import txtyrion

class ExampleClient(txtyrion.Client):

    def success(self, response):
        print '=' * 80
        print '     ID:', response.id
        print '    JID:', response.jid
        print '   Code:', response.code
        print 'Service:', response.service
        print ' Output:', response.output
        print '  Error:', response.error

    def error(self, error):
        print '=' * 80
        print 'Error:', error

    def done(self, x):
        print '=' * 80
        reactor.stop()

    def run(self):
        jid = 'test.node@example.org/tyrion'

        command = """
        sleep 1
        echo bash says hello stdout
        echo bash says hello stderr >&2
        false
        """
        d1 = self.request(jid, 'org.tyrion.service.bash', command)
        d1.addCallbacks(self.success, self.error)

        d2 = self.request(
            jid,
            'org.tyrion.service.python',
            'print "python says hello"',
        )
        d2.addCallbacks(self.success, self.error)

        dl = defer.DeferredList([d1, d2])
        dl.addCallback(self.done)

if __name__ == '__main__':
    client = ExampleClient(
        'test.client@example.org',
        'test.client.pass',
        debug=True,
    )
    reactor.run()

txtyrion also comes with a node library for creating custom Tyrion nodes (see example).

TyrionHub

It's my goal to keep Tyrion simple; to let people utilize it for as much or as little as is appropriate for their situation.

That said I've created a separate project called TyrionHub which will provide a Twisted-based system for managing nodes and a library of Python-based services for systems automation.

The immediate goals for TyrionHub are:

• User authentication using twisted.cred
• Node authentication using twisted.cred and an integration script for ejabbard
• Node enrollment

I'm currently building TyrionHub with the following technologies:

• Server: Twisted
• Database: MongoDB (although this may change)
• Web GUI: Google Closure Tools (JavaScript/Ajax/JSON)
• Services: Python with JSON serialization (simple as possible, avoid frameworks and all unnecessary libraries)

Status

• Tyrion - working and packaged (not in any repo), although it could use some code reviews by people comfortable with C++
• TyrionHub - just getting started, lots of work to do
• txtyrion - working with various tweaks here and there

Participation

If anyone is interesting in participating in the development/testing process or would like to be contacted when TyrionHub is more feature complete, please contact me at silas@sewell.ch.

License

Everything will be BSD.

Follow-up

I'll be posting a follow-up with a step-by-step explanation on how to setup a basic working version of Tyrion in the next week or two.

The goals for txmpp and why it's a separate project from libjingle are as follows:

• All code is under a single, unified namespace
• All code is permissively licensed
• Good build and install support for POSIX operating systems
• Focus on core XMPP components (the XEP-0166 related code has been removed)

I've currently got the library building on Fedora 13, CentOS 5 and OS X. In the near future I'm going to focus on removing the Windows code and reducing the amount of code not directly tied to core XMPP functionality.

With all that said I want to keep the core as close to libjingle as possible. I think it's important that txmpp can track and integrate changes from libjingle in a relatively simple way.

If you're interested in messing around with the library I'd recommend checking out the sample code in the examples directory.

const crypto = require('crypto'),
      fs = require("fs"),
      http = require("http");

var privateKey = fs.readFileSync('privatekey.pem').toString();
var certificate = fs.readFileSync('certificate.pem').toString();

var credentials = crypto.createCredentials({key: privateKey, cert: certificate});

var handler = function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
};

var server = http.createServer();
server.setSecure(credentials);
server.addListener("request", handler);
server.listen(8000);

You can generate the privatekey.pem and certificate.pem files using the following commands:

openssl genrsa -out privatekey.pem 1024 
openssl req -new -key privatekey.pem -out certrequest.csr 
openssl x509 -req -in certrequest.csr -signkey privatekey.pem -out certificate.pem

Update: Thanks to Kord Campbell for spotting an issue and sending a fix!

I created restarter because there isn't auto-reload functionality built into node.js (at the time of this post).

Example

[silas@blackbox keyfu.js]$ restarter node keyfu.js
Express started at http://localhost:3000/ in development mode

Usage

Usage: restarter [options] command

Options:
  -h, --help     show this help message and exit
  --path=PATH    directory to watch for file event changes
  --ignore=PATH  specifies events to ignore

import jinja2
import jinja2.ext
import markdown2

class Markdown2Extension(jinja2.ext.Extension):
    tags = set(['markdown2'])

    def __init__(self, environment):
        super(Markdown2Extension, self).__init__(environment)
        environment.extend(
            markdowner=markdown2.Markdown()
        )   

    def parse(self, parser):
        lineno = parser.stream.next().lineno
        body = parser.parse_statements(
            ['name:endmarkdown2'],
            drop_needle=True
        )
        return jinja2.nodes.CallBlock(
            self.call_method('_markdown_support'),
            [],
            [],
            body
        ).set_lineno(lineno)

    def _markdown_support(self, caller):
        return self.environment.markdowner.convert(caller()).strip()

env = jinja2.Environment(extensions=[Markdown2Extension])

text = """ 
{% markdown2 %}
Hello World
===========

 1. One
 2. {{ two }}
 3. Three
{% endmarkdown2 %}
"""

html = env.from_string(text).render(two='Two')

print html

Which would result in the following output:

<h1>Hello World</h1>

<ol>
<li>One</li>
<li>Two</li>
<li>Three</li>
</ol>

I created gaem because I wanted to host my blog on an unreliable server and let Google App Engine serve the content while my server was down or in maintenance mode.

The basic design is similar to that of Django's middleware. A request travels through a list of components which can either manipulate the request or provide a response. The response is filtered back through the components and served to the requester.

A basic implementation looks something like

import gaem
import gaem.middleware
import gaem.middleware.cache
import gaem.middleware.dos
import gaem.middleware.fetch

URL = 'http://www.example.org'

MIDDLEWARE_CLASSES = (
    gaem.middleware.cache.Memcache(),
    gaem.middleware.cache.Datastore(),
    gaem.middleware.dos.DoS(),
    gaem.middleware.fetch.Fetch(url=URL),
)

application = gaem.Application(MIDDLEWARE_CLASSES)

if __name__ == '__main__':
    application.run()

When a request first hits this application it is handled by the Memcache component. This basic caching component checks the Google App Engine Memcache service to see if the page is in the cache. If so it simply returns the cached response, otherwise the request proceeds to the Datastore component which does pretty much the same thing except it uses Google Datastore for persistence. If both caching components fail the the request goes through a DoS protection component and is finally retrieved from the backend server by the Fetch component.

Using a setup like this gaem acts as a rudimentary reverse proxy and while most of the functionality I've implemented thus far is for this type setup, I plan on adding features not traditionally found in a reverse proxy.

Current Features

• Caching
• Backend fetch
• Gracefully handle a dead backend
• DoS protection
• API for expiring content

Possible Future Features

• Initial and periodic crawler (pre-cache and other hooks)
• RESTful API for managing content (put, delete, etc...)
• Web interface
• XSS protection
• Some ESI support

But what makes gaem really cool for developers is how easy it is to create custom components.

A basic component looks like

from gaem import middleware

class DoesNothing(middleware.Base):

    def process_request(self, request):
        return request

    def process_response(self, response, request):
        return response

A default gaem component should inherit from gaem.middleware.Base and implement process_request and/or process_response.

The process_request function lets you either manipulate the request (ex: strip the "cookie" header) and pass it along or use the request to create and return a response (ex: do a fetch to the backend server). The process_response function lets you filter the response (ex: translate body into pig latin) and do side affects (ex: store response in memcache) before the response is sent to the requester.

The Request/Response objects are borrowed from WebOb and can be manipulated using the interfaces defined in the WebOb documentation.

gaem is currently pretty rough and likely to change over the next couple of months, but if you're a Python hacker I'd love to hear some feedback (contact me or submit an issue).

P.S. You can see the application I created for this site in the examples directory.