The default settings for Munin will result in RRD files containing 5 minute intervals of data, or 300 seconds as it is stored internally. It is possible to change this interval by changing /etc/cron.d/munin. It’s also necessary to change the update_rate in your Munin configuration, usually found in /etc/munin-node.conf. This value should match the resolution of the munin cron.

If you make this change before you start collecting data, great! If you do already have historical data that you want to keep, you need to convert the data to work with the new interval. The only documentation I was able to find on this matter on the Munin site was issue #1282 update_rate needs documentation, which in turn referenced http://www.elturista.net/2012/01/02/changing-step-time-of-a-rrd-file/ however this file is no longer available. I was able to locate it using the WayBack Machine, and in the interest of preservation I am reposting here with additional updates of my own.

rrdtool dump file.rrd > file.5.xml
./rrd_step_reduce.py file.5.xml 5 > file1.xml
rrdtool restore file1.xml file.rrd

#!/usr/bin;/python
# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import sys
from copy import deepcopy
from StringIO import StringIO

try:
    from lxml import etree
except ImportError:
    try:
        import xml.etree.cElementTree as etree
    except ImportError:
        try:
            import xml.etree.ElementTree as etree
        except ImportError:
            try:
                import cElementTree as etree
            except ImportError:
                try:
                    import elementtree.ElementTree as etree
                except ImportError:
                    raise

def main(dumpfile, factor):
        
    xmldoc = etree.parse(dumpfile)
    root = xmldoc.getroot()
    
    # change step, reducing it by a factor of "factor"
    step = root.find("step")
    assert(step!=None)
    old_step = int(step.text)
    new_step = old_step/factor
    step.text = str(new_step) 
    
    database = root.findall("rra/database")
    for d in database:
        index = 0
        count = len(d)
        while count > 0:
            for i in range(0, factor-1):
                #d.insert(index+1, NaNdoc.getroot())
                d.insert(index+1, deepcopy(d[index]))
            index = index + factor
            count = count - 1
    
    print etree.tostring(root)

if __name__ == "__main__":
    # arguments
    if len(sys.argv) != 3:
        print "rrd_step_reduce.py rrddump.xml factor"
        sys.exit(-1)
    
    # call main
    main(sys.argv[1], int(sys.argv[2])))

The post Change Interval in Munin With Existing RRD Data appeared first on Justin Silver.

The post Protected: Salesforce Continuous Integration appeared first on Justin Silver.

The version of python that is distributed with CentOS is the same of that for RHEL, which reads as not cutting edge. Typically this will be version 2.6, so even to run 2.7.6 you will need to build from source, but can still leverage yum to fetch all the dev tools you’ll need to build upgraded version of python. Here is how to upgrade python on CentOS for both 2.7 and 3.3. Tweak the version as you wish since newer versions will become available after this is posted.

Python 2.7.6

Downloads Python-2.7.6.tar.xz and builds it.

#!/usr/bin/sh

mkdir -p ~/python2.7
cd ~/python2.7
yum groupinstall -y development
yum install -y bzip2-devel openssl-devel sqlite-devel wget xz-libs zlib-dev 
wget http://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
xz -d Python-2.7.6.tar.xz
tar -xvf Python-2.7.6.tar
cd Python-2.7.6
./configure --prefix=/usr/local
make && make install

Python 3.3.3

Same same but different. Python-3.3.3.tar.xz.

#!/usr/bin/sh

mkdir -p ~/python3.3
cd ~/python3.3
yum groupinstall -y development
yum install -y bzip2-devel openssl-devel sqlite-devel wget xz-libs zlib-dev 
wget http://www.python.org/ftp/python/3.3.3/Python-3.3.3.tar.xz
xz -d Python-3.3.3.tar.xz
tar -xvf Python-3.3.3.tar
cd Python-3.3.3 
./configure
make && make install

The post Upgrade Python on CentOS appeared first on Justin Silver.

I have been using Trac for viewing my Subversion source repository and some ticketing, however I wanted to be able to integrate email directly to the ticketing system. The second part of my setup is the use of Google Apps to host our mail domain, so the easiest way to set this up was to create a new user ([email protected] / p@ssword in the example below).

My development server is running CentOS 5.8, Trac 0.12, and python2.7 (although the site-packages for CentOS are still in /var/lib/python2.4/site-packages. Because of this it was necessary to add the PYTHONPATH to the environment when calling email2trac.

The installation instructions for email2trac can be found on its homepage here, or here’s the quick version:

cd ~
wget ftp://ftp.sara.nl/pub/outgoing/email2trac.tar.gz
tar xvf email2trac.tar.gz
cd email2trac-*
./configure
make
make install

The configuration information for email2trac is located in /usr/local/etc/email2trac.conf. I made a few changes to the default configuration as you can see below, or check out the default configuration.

[DEFAULT]
project: /var/trac/project
debug: 0 
black_list: MAILER-DAEMON@
drop_spam : 1
drop_alternative_html_version: 1
email_quote: >
html2text_cmd:
ignore_trac_user_settings: 0
inline_properties: 1
reply_all : 0
spam_level: 5
strip_quotes: 1
strip_signature: 1
ticket_update: 1
ticket_update_by_subject: 1
umask: 022
verbatim_format: 1

[other_project]
project: /var/trac/other_project

On my system I also needed to do a custom build of MySQL-python to get it into the proper PYTHONPATH. With my default python set to 2.7, I ran the following:

cd ~
yum install php-imap python-devel mysql-devel
wget http://sourceforge.net/projects/mysql-python/files/latest/download
cd MySQL-python*
python setup.py install

Note that above on my system I actually only installed mysql-devel.x86_64 due to a conflict in the i386 version.

Next I used the following PHP script to import message from the Trac Google Apps user and process them into Trac one at a time. Due to the different environment I had to set the PYTHONPATH using putenv() so that email2trac would run. This file was saved as /var/trac/gmail2trac.php.

<?php
// Connect to the Google Apps / Gmail account
$mailbox = imap_open("{imap.googlemail.com:993/ssl}INBOX", "[email protected]", "p@ssword");
// Get all unread emails
$mail = imap_search($mailbox, "UNSEEN");
if ( !empty($mail) ){
        // This is required on CentOS to find the Trac egg
        putenv("PYTHONPATH=/usr/lib/python2.4/site-packages");
        // Process each email
        foreach ($mail as $id){
                // Save the email contents to a temp file
                imap_savebody($mailbox, '/tmp/gmail2trac.txt', $id);
                // Import into Trac via email2trac
                system('/usr/local/bin/email2trac -f /usr/local/etc/email2trac.conf -p project < /tmp/gmail2trac.txt');
                // Mark as read
                imap_setflag_full($mailbox, $id, "\\Seen");
        }
}
// Close connection
imap_close($mailbox);
?>

Note that if you want to reprocess any emails, all you need to do is log in as the Trac user and mark the message as unread.

Lastly to import on a schedule – I opted for a check every 5 minutes – just setup a cron job via crontab -e.

*/5 * * * * php /var/trac/gmail2trac.php

The post Google Apps + Trac Integration Using email2trac appeared first on Justin Silver.

The documentation for Trac is decent, but it isn’t super clear what to do, especially if you aren’t familiar with python. Some quick random info:

• For this example we are going to be hosting a Trac server using a vhost in Apache via mod_python
• Download Trac from http://download.edgewall.org/trac/Trac-0.12.3.tar.gz
• PYTHON_EGG_CACHE on my system is set to /tmp. It has temp files and whatnot.
• Subversion repository is install in /var/svn, but we’ll assume there is already a repository there called repository.
• Trac projects are going to be in /var/trac, go ahead and create it.
• Everywhere you see PROJECT replace it with your project name

First install the prerequisites using yum, then make sure you have everything with easy_setup

yum install python MySQL-python mod_fastcgi mod_python subversion
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py
easy_install Genshi

Now to install Trac, we must first get the source and then run the installer. Note that you will need root or sudo access to install.

cd ~
wget http://download.edgewall.org/trac/Trac-0.12.3.tar.gz
tar -xzvf Trac-0.12.3.tar.gz
cd Trac-0.12.3
sudo python ./setup.py install

Now let’s create the Trac project. It will ask you for a project name and a database connection – I’m using Mysql and already created a project schema called trac_PROJECT and user that has full access to it. My connection string looks like mysql://user:password@host/trac_PROJECT,

mkdir /var/trac
trac-admin /var/trac/PROJECT initenv

Now edit the trac.ini file for your project, located at /var/trac/PROJECT/conf/trac.ini. We want to set the properties “repository_dir = /var/svn/repository” and “repository_sync_per_request = ” – remove (default).

Next we need to create an htpasswd file, as this is what Trac uses for authentication. Enter a password for the admin user. If you want to create more users, leave out the -c since this creates a new file every time and will overwrite your first entries if it’s present. The next line gives this user TRAC_ADMIN rights to your project

sudo htpasswd -c /var/trac/.htpasswd admin 
trac-admin /var/trac/PROJECT permission add admin TRAC_ADMIN

Create the commit hook in your Subversion repository.

/var/svn/repository/hooks/post-commit

#!/bin/sh
export PYTHON_EGG_CACHE="/tmp"
/usr/bin/trac-admin /var/trac/PROJECT changeset added "$1" "$2"

Make it executable

chmod 755 /var/svn/repository/hooks/post-commit

Configure Apache to use mod_fastcgi and not mod_php to serve PHP files – this is the first step to serving up Trac pages.

First we need to create the FastCGI wrappers that we will use to execute requests. We are going to need two, one for PHP and one for Trac. Both will go in /var/www/cgi-bin and it’s important to make sure that they are executable.

/var/www/cgi-bin/php.fcgi

#!/bin/sh
export PHP_FCGI_CHILDREN=4
exec /usr/bin/php-cgi -c /etc/php.ini

The second file is named trac.fcgi and will contain the following – note that this is one place where PYTHON_EGG_CACHE is set, and that we have to include the parent path to our Trac environment. You can also serve a single project by just setting TRAC_ENV instead of TRAC_ENV_PARENT_DIR.

/var/www/cgi-bin/trac.fcgi

#!/usr/bin/env python
import os

from trac.web.main import dispatch_request
try:
    from flup.server.fcgi import WSGIServer
except ImportError:
    from trac.web._fcgi import WSGIServer

if __name__ == '__main__':
    os.environ['TRAC_ENV_PARENT_DIR'] = '/var/trac'
    os.environ['PYTHON_EGG_CACHE'] = '/tmp'
    WSGIServer(dispatch_request).run()

Make both executable:

chmod 755 /var/www/cgi-bin/*.fcgi

Now we need to tell Apache to use these FastCGI wrappers. In /etc/httpd/conf.d/ you will need rename php.conf to php.conf.off and edit fastcgi.conf.

mv /etc/httpd/conf.d/php.conf /etc/httpd/conf.d/php.conf.off

Now create the FastCGI wrapper for PHP.

/etc/httpd/conf.d/fastcgi.conf

LoadModule fastcgi_module modules/mod_fastcgi.so

FastCgiServer /var/www/cgi-bin/php.fcgi
AddHandler php-fastcgi .php

SetHandler fastcgi-script
Action php-fastcgi /cgi-bin/php.fcgi
DirectoryIndex index.html index.shtml index.cgi index.php
AddType application/x-httpd-php .php

Create a VirtualHost in Apache that will serve Trac.

<VirtualHost *:80>
        ServerName trac.DOMAIN.com
	DocumentRoot /var/trac
	<Directory /var/trac>
	        Order allow,deny
	        Allow from all
	</Directory>
	ScriptAlias / /var/www/cgi-bin/trac.fcgi/
	<LocationMatch "/[^/]+/login">
		AuthType Basic
		AuthName "Trac"
		AuthUserFile /var/trac/.htpasswd
		Require valid-user
	</LocationMatch>
</VirtualHost>

Restart Apache by running

service httpd restart

You should be able to visit http://trac.DOMAIN.com/PROJECT/login in your browser and log on using admin and the password you selected. All you have left to do is enable the plugins to tie Trac and Subversion commits together.

Choose Admin on the top right, then Plugins on the left. You will need to click the arrow to expand the list of plugins and go to the bottom. You should see CommitTicketReferenceMacro and CommitTicketUpdater – check the box next to them to enable both.

Should be good to go!

The post Trac 0.12.3 + MySQL + Apache + Subversion on CentOS 5.5 appeared first on Justin Silver.