Silas Sewell

python-stomping is a simple Stomp framework I wrote which makes creating stomp services in Python easy.

1. Install requirements

   $ sudo yum install python-twisted python-stomper
   

2. Checkout stomping

   $ svn export http://silassewell.googlecode.com/svn/trunk/projects/python-stomping python-stomping
   $ cd python-stomping
   

3. Update the example.py file to use your host, port, username and password ActiveMQ/Stomp broker settings

   from stomping import Stomping, route
   
   class MyStomping(Stomping):
   
   def init(self):
       self.send('/queue/test1', 'init test1')
       self.send('/queue/test2', 'init test2')
   
   @route('/queue/test1')
   def a_test(self, message):
       print 'a_test: %s' % message['body']
   
   @route('/queue/test1')
   @route('/queue/test2')
   def b_test(self, message):
       print 'b_test: %s' % message['body']
   
   if __name__ == '__main__':
       stomp = MyStomping(host='127.0.0.1', port=61613, username='guest', password='guest')
       stomp.run()
   

4. Run example.py (Ctrl+c to quit)

   $ python example.py 
   a_test: init test1
   b_test: init test1
   b_test: init test2
   

NOTE: python-stomping is just a light wrapper around Twisted and stomper and is based on an example provided in the stomper code.

The following is a quick and dirty guide to getting started with line_profiler, a Python line-by-line profiler, on Fedora.

1. Build and install the python-line_profiler package

2. Create a file called test.py with the code below

   import random, time
   
   def sleep():
       seconds = random.randint(0, 5)
       print 'Sleeping %s seconds' % seconds
       time.sleep(seconds)
   
   @profile
   def test():
       sleep()
       sleep()
       sleep()
   
   test()
   

3. Profile test.py

   [silas@silas ~]$ kernprof.py -l test.py
   Sleeping 4 seconds
   Sleeping 5 seconds
   Sleeping 2 seconds
   Wrote profile results to test.py.lprof
   

4. View the results

   [silas@silas ~]$ python -m line_profiler test.py.lprof
   Timer unit: 1e-06 s
   
   File: test.py
   Function: test at line 8
   Total time: 10.9994 s
   
   Line #      Hits         Time  Per Hit   % Time  Line Contents
   ==============================================================
        8                                           @profile
        9                                           def test():
       10         1      3999416 3999416.0     36.4      sleep()
       11         1      4999982 4999982.0     45.5      sleep()
       12         1      1999990 1999990.0     18.2      sleep()
   

NOTE: I have a package review up for line_profiler and it should be available via yum eventually.

Features

• advanced client selection (you can use + and - to add/remove hosts/groups)
• tab completion
• persistent history
• a command sub-shell
• quick help (just hit the "?" key)

funcshell took a similar approach to features as Func; it allows developers to easily extend and customize its functionality using a plugin system. It also assumes the most useful commands will be the ones you write yourself. That being said, funcshell currently comes with both the command and service module.

Here are some things you can do with funcshell.

Get a list of all clients (not very useful if you have hundreds):

funcshell> set clients *
funcshell> get clients
webapp001.example.net
webapp002.example.net
webapp003.example.net
webstatic001.example.net
webstatic002.example.net

Assuming webapp00[1-3].example.net are in the app and bpstatic00[1-2].example.net1 are in the static groups; remove some clients and restart a service:

funcshell> set clients - @static;webapp001*
funcshell> get clients
webapp002.example.net
webapp003.example.net
funcshell> service httpd restart
==> webapp003.example.net <==
True
==> webapp002.example.net <==
True

Run a command and get the results:

funcshell> set clients - webapp003*
funcshell> set clients + webstatic002*
funcshell> command run grep MemTotal /proc/meminfo | awk '{ print $2/1024 }'
==> webstatic002.example.net :: 0 <==
4099.44
==> webapp002.example.net :: 0 <==
8198.88

Get help:

funcshell> command <TAB><TAB>
exists   run      shell
funcshell> command ?
  exists Check if a command exists
  run    Run a command
  shell  Run a command shell

Run a couple of commands in a row:

funcshell> set c<TAB> @static
funcshell> command shell
> du -sh /tmp
==> webstatic002.example.net :: 0 <==
7.2M    /tmp
==> webstatic001.example.net :: 0 <==
3.4M    /tmp
> cp -r /tmp /tmp
==> webstatic002.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
==> webstatic001.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
> asdf
==> webstatic002.example.net :: 127 <==
/bin/sh: asdf: command not found
==> webstatic001.example.net :: 127 <==
/bin/sh: asdf: command not found
> <CTRL-D>
funcshell> exit

funcshell is a shell interface to Func which provides some useful features for managing a large number of machines.

The Advanced Message Queuing Protocol (AMQP) is the first open standard for Enterprise Messaging and has been adopted by Cisco, [Microsoft][microsoft], Novel, Red Hat and others.

One implementation of AMQP currently packaged by Red Hat and Fedora is Qpid. Qpid is a collection of brokers (C++, Java) and clients (C++, Java, Python, Ruby) which aims to be 100% AMQP Compliant.

Below is a step-by-step tutorial on getting started with Qpid and the Python client in Fedora.

1. Install the Qpid broker and Python client

   yum install qpidd python-qpid
   

2. Start the broker

   service qpidd start
   

3. Create a message producer (producer.py)

   from qpid.connection import Connection
   from qpid.datatypes import Message, uuid4
   from qpid.util import connect
   
   # Create connection and session
   socket = connect('localhost', 5672)
   connection = Connection(sock=socket, username='guest', password='guest')
   connection.start()
   session = connection.session(str(uuid4()))
   
   # Setup queue
   session.queue_declare(queue='message_queue')
   session.exchange_bind(exchange='amq.direct', queue='message_queue', binding_key='routing_key')
   
   # Setup routing properties
   properties = session.delivery_properties(routing_key='routing_key')
   
   # Send messages
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Message one'))
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Message two'))
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Done'))
   
   # Close session
   session.close(timeout=10)
   

4. Create a message consumer (consumer.py)

   from qpid.connection import Connection
   from qpid.datatypes import RangedSet, uuid4
   from qpid.util import connect
   
   # Create connection and session
   socket = connect('localhost', 5672)
   connection = Connection(sock=socket, username='guest', password='guest')
   connection.start()
   session = connection.session(str(uuid4()))
   
   # Define local queue
   local_queue_name = 'my_local_queue'
   
   # Create local queue
   queue = session.incoming(local_queue_name)
   
   # Route messages from message_queue to my_local_queue
   session.message_subscribe(queue='message_queue', destination=local_queue_name)
   queue.start()
   
   content = ''
   
   while content != 'Done':
       # Get message from the local queue
       message = queue.get(timeout=10)
       # Get body of the message
       content = message.body
       # Accept message (removes it from the queue)
       session.message_accept(RangedSet(message.id))
       # Print message content
       print content
   
   # Close session
   session.close(timeout=10)
   

5. First run the producer and then the consumer

   Message one
   Message two
   Done
   

Source: Qpid direct example

A simple function to slice strings in Bash similar to Python's string slicing functionality.

Examples

[silas@pluto ~]$ string_slice "12345" 0 1
1
[silas@pluto ~]$ string_slice "12345" 0 3
123
[silas@pluto ~]$ string_slice "12345" 2 3
3
[silas@pluto ~]$ string_slice "12345" 2 -2
3
[silas@pluto ~]$ string_slice "12345" -3
345

Implementation

function string_slice {
    STRING="$1"
    declare -i LENGTH="${#STRING}"
    declare -i START="$2"
    declare -i END="$3"
    if [ $START -lt 0 ]; then
        START=$[ $LENGTH + $START ]
    fi
    if [ $END -le 0 ]; then
        END=$[ $LENGTH + $END ]
    fi
    START=$[ $START + 1 ]
    (echo "$STRING" | cut -c $START-$END) 2> /dev/null
}

Func is a nifty, although not yet polished, Python-based service for running tasks on many hosts at once. It lets you do things like restart Apache instances, run yum updates or provision virtual machines.

You can accomplish theses tasks via the command line interface

func web-*.example.net call service restart httpd

or through the Python API

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

print client.service.restart('httpd')

One of the annoying issues I'm dealing with right now is the distribution of Func modules, which for various reasons isn't as simple as packaging the modules and pushing them to all the hosts (it is, just not in my network).

My solution is Push, a Func module which lets you instantly run Python code on any or all hosts in your network.

Example usage:

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

source = """
def main():
    return 'Hello World'
"""

print client.push.code(source)

I realize this isn't the most elegant solution, but its very useful and extremely simple.

I also realize that I could use the copyfile module to push modules via Func, but I haven't found a way to restart funcd via Func (required to initialize modules), so until that happens I need a solution which doesn't require restarting Func.

Get Push Module

A simple GMP Hello World example in C.

Makefile

all:
    gcc -o gmp_hello_world gmp_hello_world.c -lgmp -m64

clean:
    rm gmp_hello_world

gmp_hello_world.c

/*
 * GMP Hello World on OS X 10.5
*/
#include <gmp.h>
#include <stdio.h>

int main() {
  mpz_t add_total1, add_total2, sub_total1, sub_total2, mul_total1, mul_total2;
  mpz_t num1, num2;

  // Initialize variables
  mpz_init_set_str(num1, "345192567923875922375736284875732", 10);
  mpz_init_set_str(num2, "937929298382994742939293857584837", 10);
  mpz_init(add_total1);
  mpz_init(add_total2);
  mpz_init(sub_total1);
  mpz_init(sub_total2);
  mpz_init(mul_total1);
  mpz_init(mul_total2);

  // Do arithmetic
  mpz_add(add_total1, num1, num2);
  mpz_add_ui(add_total2, num1, 10);
  mpz_sub(sub_total1, num1, num2);
  mpz_sub_ui(sub_total2, num1, 10);
  mpz_mul(mul_total1, num1, num2);
  mpz_mul_ui(mul_total2, num1, 10);

  // Display results
  gmp_printf("Add Total 1: %Zd\n", add_total1);
  gmp_printf("Add Total 2: %Zd\n", add_total2);
  gmp_printf("Subtract Total 1: %Zd\n", sub_total1);
  gmp_printf("Subtract Total 2: %Zd\n", sub_total2);
  gmp_printf("Multiply Total 1: %Zd\n", mul_total1);
  gmp_printf("Multiply Total 2: %Zd\n", mul_total2);

  // Free space
  mpz_clear(num1);
  mpz_clear(num2);
  mpz_clear(add_total1);
  mpz_clear(add_total2);
  mpz_clear(sub_total1);
  mpz_clear(sub_total2);
  mpz_clear(mul_total1);
  mpz_clear(mul_total2);

  return 0;
}

Peafowl is a Python port of Ruby's Starling.

My attempt at wrangling a Universal Feed Parser object into JSON.

import feedparser

def json_make_normal(obj):
    if type(obj) in [str, unicode, int, float, bool, dict, set, list, tuple]:
        return obj
    try: return dict(obj)
    except: pass
    try: return list(obj)
    except: pass
    return None

def json_handle(obj):
    obj = json_make_normal(obj)
    if type(obj) in [str, unicode]:
        obj = obj.replace('\\', '\\\\')
        obj = obj.replace('"', '\\"')
        obj = obj.replace('\b', '\\\b')
        obj = obj.replace('\f', '\\\f')
        obj = obj.replace('\n', '\\\n')
        obj = obj.replace('\r', '\\\r')
        obj = obj.replace('\t', '\\\t')
        return '"%s"' % obj
    elif type(obj) in [int, float]:
        return obj
    elif type(obj) is bool:
        if obj: return 'true'
        else: return 'false'
    elif type(obj) is type(None):
        return 'null'
    elif type(obj) is dict:
        temp = ''
        for key in obj.keys():
            temp += '%s:%s, ' % (json_handle(key), json_handle(obj[key]))
        return '{%s}' % temp[:-2]
    elif type(obj) in [set, list, tuple]:
        temp = ''
        for value in obj:
            temp += '%s, ' % json_handle(value)
        return '[%s]' % temp[:-2]
    return 'null'

data = feedparser.parse('http://digg.com/rss/index.xml')

print json_handle(data)

Link: feed_parser_to_json.py

Update: According to John Paulett jsonpickle now supports Universal Feed Parser objects. Thanks for letting me know!

I've created a PHP script called Post a Get which can be copied to any PHP capable server and used to created a POST request using GET syntax.

Let's say you use GoDaddy as your domain registrar and you would like to see if a domain is available using Firefox keywords, but they don't offer a GET search.

For a traditional keyword, you could construct a GET request like:

https://www.godaddy.com/gdshop/registrar/search.asp&checkAvail=1&fblur=1&tld=.com&domainToCheck=%s

But as of right now Firefox doesn't offer a way to create POST keywords. What Post a Fix does is allow you to construct a POST request using GET syntax and pass it to the Post a Get code.

So if you wanted to submit the above GET as a POST, you would construct the URL and append the special __action parameter with target URL. So the final URL would look something like: