Silas Sewell - Scribe http://www.silassewell.com/blog/tag/scribe Infrastructure Development Sat, 08 Jan 2011 00:00:00 GMT http://www.silassewell.com/ en hourly 1 Pipe Apache (or any) Logs to Scribe http://www.silassewell.com/blog/2009/05/12/pipe-apache-or-any-logs-to-scribe/ Tue, 12 May 2009 00:00:00 GMT silas http://www.silassewell.com/blog/2009/05/12/pipe-apache-or-any-logs-to-scribe/ I created a simple Python script called scribe_log to tail a log file and pipe it to Scribe.

I use supervisor to start and keep the pipe running.

Relevant supervisord.conf configuration:

[program:scribe.apache.access]
command=/usr/local/sbin/scribe_log --category apache.access --file /var/log/httpd/access_log

Options

usage: scribe_log [options]

options:
  -h, --help           show this help message and exit
  --file=FILE          file to tail into Scribe
  --category=CATEGORY  Scribe category
  --host=HOST          destination Scribe host server
  --port=PORT          destination Scribe port
  --prefix=PREFIX      add to the beginning of each log line
  --postfix=POSTFIX    add to the end of each log line
]]> Scribe – Scalable Real Time Log Aggregation for CentOS 5 / RHEL 5 http://www.silassewell.com/blog/2009/05/07/scribe-scalable-real-time-log-aggregation-for-centos-5-rhel-5/ Thu, 07 May 2009 00:00:00 GMT silas http://www.silassewell.com/blog/2009/05/07/scribe-scalable-real-time-log-aggregation-for-centos-5-rhel-5/ "Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine. Scribe was developed at Facebook and released as open source."

I've packaged Thrift, fb303 and Scribe for CentOS 5 / RHEL 5.

SRPM

Scribe depends on fb303 which in turns depends on Thrift so you'll need to setup a local repository and build them in that order. To build the above packages for Fedora 9+ you'll need to tweak the Python sub-packages to include the egg files.

Both fb303 and Scribe need some tweaks upstream before they're suitable for a package review, but I'd like to get them in Fedora shortly (I've already submitted Thrift).

If you'd like to hack around with my latest Fedora specs you can grab them here.

]]>