Silas Sewell

The following is a quick and dirty guide to getting started with line_profiler, a Python line-by-line profiler, on Fedora.

1. Build and install the python-line_profiler package

2. Create a file called test.py with the code below

   import random, time
   
   def sleep():
       seconds = random.randint(0, 5)
       print 'Sleeping %s seconds' % seconds
       time.sleep(seconds)
   
   @profile
   def test():
       sleep()
       sleep()
       sleep()
   
   test()
   

3. Profile test.py

   [silas@silas ~]$ kernprof.py -l test.py
   Sleeping 4 seconds
   Sleeping 5 seconds
   Sleeping 2 seconds
   Wrote profile results to test.py.lprof
   

4. View the results

   [silas@silas ~]$ python -m line_profiler test.py.lprof
   Timer unit: 1e-06 s
   
   File: test.py
   Function: test at line 8
   Total time: 10.9994 s
   
   Line #      Hits         Time  Per Hit   % Time  Line Contents
   ==============================================================
        8                                           @profile
        9                                           def test():
       10         1      3999416 3999416.0     36.4      sleep()
       11         1      4999982 4999982.0     45.5      sleep()
       12         1      1999990 1999990.0     18.2      sleep()
   

NOTE: I have a package review up for line_profiler and it should be available via yum eventually.

I created a simple Python script called scribe_log to tail a log file and pipe it to Scribe.

I use supervisor to start and keep the pipe running.

Relevant supervisord.conf configuration:

[program:scribe.apache.access]
command=/usr/local/sbin/scribe_log --category apache.access --file /var/log/httpd/access_log

Options

usage: scribe_log [options]

options:
  -h, --help           show this help message and exit
  --file=FILE          file to tail into Scribe
  --category=CATEGORY  Scribe category
  --host=HOST          destination Scribe host server
  --port=PORT          destination Scribe port
  --prefix=PREFIX      add to the beginning of each log line
  --postfix=POSTFIX    add to the end of each log line

"Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine. Scribe was developed at Facebook and released as open source."

I've packaged Thrift, fb303 and Scribe for CentOS 5 / RHEL 5.

SRPM

• Thrift (Source)
• fb303 (Source)
• Scribe (Source)

Scribe depends on fb303 which in turns depends on Thrift so you'll need to setup a local repository and build them in that order. To build the above packages for Fedora 9+ you'll need to tweak the Python sub-packages to include the egg files.

Both fb303 and Scribe need some tweaks upstream before they're suitable for a package review, but I'd like to get them in Fedora shortly (I've already submitted Thrift).

If you'd like to hack around with my latest Fedora specs you can grab them here.

Puppet is an open source configuration management tool written in Ruby. It allows a systems administrator to define how a system should be configured using Puppet's declarative language. Each Puppet client pulls its catalog at a regular interval and figures out how to make the catalog definitions true for the local operating system.

The Puppet Introduction uses the following diagram to show how Puppet works.

Currently Puppet is most useful when you have lots of nodes with similar setups. Unfortunately Puppet's declarative language is a bit weak when it comes to inheritance and users familiar with true object oriented systems will soon become frustrated (at least I did). I'm assuming this issue will be addressed in the future, but for now I'm going to tell you how I setup a 100+ node system while adhering to DRY principles.

A quick note, I'm assuming you already know how to use Puppet.

First lets start with the Puppetmaster configuration layout:

puppet/
 - fileserver.conf
 - manifests/
   - classes/
     - initialize.pp
   - nodes/
     - net/
       - example/
         - web01.pp   # web01.example.net
   - site.pp
   - templates.pp
 - modules/           # application specific modules
   - httpd/
     - files/         # static assets (ex: default HTML file)
     - manifests/     # application configuration logic
     - templates/     # dynamic configuration files (ex: httpd.conf)

The layout was adapted from recommendations in Pulling Strings with Puppet by James Turnbull.

When I was initially designing my Puppet setup the common practice was to define and initialize the various Puppet types and classes throughout the inheritance tree. So if you had a generic resolv.conf configuration you would include it at the top of the inheritance tree. This worked great until I needed to change one attribute of a class further down. I initially tackled this problem by hacking in if/case statements, but eventually it became unmanageable.

After a couple of rewrites I came up with the idea of wrapping the internals of each class in a conditional statement and initializing all classes at the end of each node.

The key components of my setup were:

• import all modules in the site.pp manifest
• wrap the code of each class in a conditional statement
• define and extend attributes throughout the inheritance tree (including the class conditional)
• include all classes at the end of each node

This lets you both arbitrarily redefine or extend class attributes (including the on/off state) throughout the inheritance tree.

I've created a simple example to show how this setup works. The main files are listed below (you'll want to go through each in order):

• site.pp - import modules
• templates.pp - define and extend class attributes
• web01.pp - make use of defaults
• ns01.pp - disable a class
• initialize.pp - initialize all classes
• client.pp - class implementation

Also note that I designed this setup about 8 months ago and I haven't kept as up-to-date on recent developments as I should. If there is a better way to do this please let me know.

Features

• advanced client selection (you can use + and - to add/remove hosts/groups)
• tab completion
• persistent history
• a command sub-shell
• quick help (just hit the "?" key)

funcshell took a similar approach to features as Func; it allows developers to easily extend and customize its functionality using a plugin system. It also assumes the most useful commands will be the ones you write yourself. That being said, funcshell currently comes with both the command and service module.

Here are some things you can do with funcshell.

Get a list of all clients (not very useful if you have hundreds):

funcshell> set clients *
funcshell> get clients
webapp001.example.net
webapp002.example.net
webapp003.example.net
webstatic001.example.net
webstatic002.example.net

Assuming webapp00[1-3].example.net are in the app and bpstatic00[1-2].example.net1 are in the static groups; remove some clients and restart a service:

funcshell> set clients - @static;webapp001*
funcshell> get clients
webapp002.example.net
webapp003.example.net
funcshell> service httpd restart
==> webapp003.example.net <==
True
==> webapp002.example.net <==
True

Run a command and get the results:

funcshell> set clients - webapp003*
funcshell> set clients + webstatic002*
funcshell> command run grep MemTotal /proc/meminfo | awk '{ print $2/1024 }'
==> webstatic002.example.net :: 0 <==
4099.44
==> webapp002.example.net :: 0 <==
8198.88

Get help:

funcshell> command <TAB><TAB>
exists   run      shell
funcshell> command ?
  exists Check if a command exists
  run    Run a command
  shell  Run a command shell

Run a couple of commands in a row:

funcshell> set c<TAB> @static
funcshell> command shell
> du -sh /tmp
==> webstatic002.example.net :: 0 <==
7.2M    /tmp
==> webstatic001.example.net :: 0 <==
3.4M    /tmp
> cp -r /tmp /tmp
==> webstatic002.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
==> webstatic001.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
> asdf
==> webstatic002.example.net :: 127 <==
/bin/sh: asdf: command not found
==> webstatic001.example.net :: 127 <==
/bin/sh: asdf: command not found
> <CTRL-D>
funcshell> exit

funcshell is a shell interface to Func which provides some useful features for managing a large number of machines.

The Advanced Message Queuing Protocol (AMQP) is the first open standard for Enterprise Messaging and has been adopted by Cisco, [Microsoft][microsoft], Novel, Red Hat and others.

One implementation of AMQP currently packaged by Red Hat and Fedora is Qpid. Qpid is a collection of brokers (C++, Java) and clients (C++, Java, Python, Ruby) which aims to be 100% AMQP Compliant.

Below is a step-by-step tutorial on getting started with Qpid and the Python client in Fedora.

1. Install the Qpid broker and Python client

   yum install qpidd python-qpid
   

2. Start the broker

   service qpidd start
   

3. Create a message producer (producer.py)

   from qpid.connection import Connection
   from qpid.datatypes import Message, uuid4
   from qpid.util import connect
   
   # Create connection and session
   socket = connect('localhost', 5672)
   connection = Connection(sock=socket, username='guest', password='guest')
   connection.start()
   session = connection.session(str(uuid4()))
   
   # Setup queue
   session.queue_declare(queue='message_queue')
   session.exchange_bind(exchange='amq.direct', queue='message_queue', binding_key='routing_key')
   
   # Setup routing properties
   properties = session.delivery_properties(routing_key='routing_key')
   
   # Send messages
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Message one'))
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Message two'))
   session.message_transfer(destination='amq.direct', message=Message(properties, 'Done'))
   
   # Close session
   session.close(timeout=10)
   

4. Create a message consumer (consumer.py)

   from qpid.connection import Connection
   from qpid.datatypes import RangedSet, uuid4
   from qpid.util import connect
   
   # Create connection and session
   socket = connect('localhost', 5672)
   connection = Connection(sock=socket, username='guest', password='guest')
   connection.start()
   session = connection.session(str(uuid4()))
   
   # Define local queue
   local_queue_name = 'my_local_queue'
   
   # Create local queue
   queue = session.incoming(local_queue_name)
   
   # Route messages from message_queue to my_local_queue
   session.message_subscribe(queue='message_queue', destination=local_queue_name)
   queue.start()
   
   content = ''
   
   while content != 'Done':
       # Get message from the local queue
       message = queue.get(timeout=10)
       # Get body of the message
       content = message.body
       # Accept message (removes it from the queue)
       session.message_accept(RangedSet(message.id))
       # Print message content
       print content
   
   # Close session
   session.close(timeout=10)
   

5. First run the producer and then the consumer

   Message one
   Message two
   Done
   

Source: Qpid direct example

A simple function to slice strings in Bash similar to Python's string slicing functionality.

Examples

[silas@pluto ~]$ string_slice "12345" 0 1
1
[silas@pluto ~]$ string_slice "12345" 0 3
123
[silas@pluto ~]$ string_slice "12345" 2 3
3
[silas@pluto ~]$ string_slice "12345" 2 -2
3
[silas@pluto ~]$ string_slice "12345" -3
345

Implementation

function string_slice {
    STRING="$1"
    declare -i LENGTH="${#STRING}"
    declare -i START="$2"
    declare -i END="$3"
    if [ $START -lt 0 ]; then
        START=$[ $LENGTH + $START ]
    fi
    if [ $END -le 0 ]; then
        END=$[ $LENGTH + $END ]
    fi
    START=$[ $START + 1 ]
    (echo "$STRING" | cut -c $START-$END) 2> /dev/null
}

The following describes a simple way to manage you profile configuration files using GitHub.

Features

• Centralized configuration management
• Files live in their native locations (no symbolic linking)
• Home directory is not a Git repository
• All the power of git with a simple alias

Setup Repository

• Log into GitHub and create a repository named config
• Add your public keys to GitHub (if you haven't done so already)

• Open a terminal and switch to your home directory

  cd ~
  

• Create a configuration directory

  mkdir .config.git
  

• Add the following alias to your current session and your .bash_profile

  alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'
  echo "alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'" >> .bash_profile
  

• Add your .bash_profile to the configuration repository

  config add .bash_profile
  

• Commit the changes

  config commit -m 'Initial commit'
  

• Change the origin to GitHub

  config remote add origin git@github.com:GITHUB_USERNAME/config.git
  

• Push the changes

  config push origin master
  

If you get an error when running config pull to the effect of You asked me to pull without... run the follow:

    echo -e '[branch "master"]\n  remote = origin\n  merge = refs/heads/master' >> ~/.config.git/config

Setup Configuration Management on a Different System

1. Add your public keys to GitHub (if you haven't done so already)

2. Switch to your home directory

   cd ~
   

3. Backup your local configuration files, example:

   mv .bash_profile .bash_profile.bk
   

4. Clone your configuration repository

   git clone git@github.com:GITHUB_USERNAME/config.git config.git
   

5. Move the git metadata to ~/.config.git

   mv config.git/.git .config.git
   

6. Enable dotglob

   shopt -s dotglob
   

7. Move your configuration files to your home directory

   mv -i config.git/* .
   

8. Delete the config.git directory

   rmdir config.git
   

9. Logout and log back in

Basic Usage

• config pull - get latest configuration changes
• config add FILENAME - add a configuration file
• config commit -a - save all configuration changes
• config push - push configuration changes to GitHub
• and any other config GIT_OPTION

You can see my configuration repository at http://github.com/silas/config.

Source: Manage your $HOME with git by Robert Escriva

Please note that you will lose all data on your phone when performing these steps.

• Mount the ADP MicroSD card
• Download hboot-0.95.0000.zip to the mount and rename it to update.zip
• Unmount the card and shutdown the phone (unplug the USB cable if connected)
• Hold Home+Power to boot
• When prompted press Alt+S and wait for the update to finish
• Press Home+Back to reboot and wait for second update to finish and boot into the Android recovery system
• Press Home+Back to reboot again
• Mount the MicroSD card
• Delete update.zip
• Download signed-holiday_devphone-ota-130444-debug.55489994.zip to the mount and rename it to update.zip
• Unmount the card and shutdown the phone
• Hold Home+Power to boot
• Press Alt+S to update and wait for update to finish
• Press Home+Back to reboot and wait for the second update to finish and boot into the Android system recovery utility
• Press Alt+W to wipe all data
• Press Home+Back to reboot and wait for the reboot to finish (this could take a couple of minutes)
• Re-setup the ADP (hint: press menu to setup custom APN settings)

Update: I am currently refactoring FuncShell; please use the following code: shell.py.

I created funcshell for running Func modules in a more intuitive manner.

The current implementation only supports the command module, but more will be released shortly.

Code: http://github.com/silas/funcshell

Example Usage

[root@pluto ~]# python shell.py
fs> use web*.example.org
fs> get hosts
web01.example.org
web02.example.org
fs> !du -sh /tmp
================================================================================
== web02.example.org                                                          ==
================================================================================

236K    /tmp

================================================================================
== web01.example.org                                                          ==
================================================================================

217M    /tmp

fs> !cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'
================================================================================
== web02.example.org                                                          ==
================================================================================

1027116

================================================================================
== web01.example.org                                                          ==
================================================================================

1027116

fs> exit
[root@pluto ~]#