Code V.igoro.us

Trapped in Google?

nospam@example.com (Dustin J. Mitchell) — Sat, 19 May 2012 18:33:24 -0500

Whenever I search in Aurora on my phone, I'm taken to a stripped-down version of the page with the header "this page adapted for your browser".

How do I fix this? I'd rather fix it with a Google preference, but barring that I expect that Firefox has a way for me to regain control of my online experience?

TIL about SSL certificate chains

nospam@example.com (Dustin J. Mitchell) — Wed, 09 May 2012 15:43:01 -0500

I'm laying some SSL groundwork for a project to allow puppet clients to move between puppet servers without requiring a central CA, and without requiring each client to be aware of all masters. More on that in a future post.

Based on "Multiple Certificate Authorities", I would like to have certificate chains that look like this:

      +-puppetmaster1 CA--+-puppetmaster1 server cert
      |                   |
      |                   +-client 1 server cert
root--+                   :
      |                   
      +-puppetmaster2 CA--+-puppetmaster2 server cert
                          |
                          +-client 10 server cert
                          :

Then all of the certificate validation would be done with the root CA certificate as the trusted certificate. A server certificate signed by puppetmaster2's CA cert should then validate on puppetmaster1.

Building the certificates wasn't all that difficult - see my comment on the bug for the script. However, while making sure the verification worked, I ran into some non-obvious limitations of OpenSSL that are worth writing down.

I began by running "openssl verify":

[root@relabs-puptest1 ~]# openssl verify -verbose -CAfile puptest-certs/root-ca.crt -purpose sslclient puptest-certs/relabs08.build.mtv1.mozilla.com.crt 
puptest-certs/relabs08.build.mtv1.mozilla.com.crt: CN = relabs08.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
error 20 at 0 depth lookup:unable to get local issuer certificate

the problem here is that the intermediate certificate is not available to the verification tool. Sources suggest to include it with the server cert, by concatention, with the server cert last:

cat puptest-certs/relabs-puptest1.build.mtv1.mozilla.com-ca.crt puptest-certs/relabs08.build.mtv1.mozilla.com.crt > relabs08-with-intermed.crt

However, after some struggle I learned that "openssl verify" does not recognize this format -- it will only look at the first certificate in the file (the intermediate), and if you don't look carefully you'll find that it successfully verifies the intermediate, not the server certificate! Sadly, sclient and ssever don't support it either. Apache httpd supports it with SSLCACertificatePath. This will feed the certificate chain to the client, and also allow httpd to verify client certificates without requiring the clients to support an intermediate.

The Apache config is

Listen 1443

<VirtualHost *:1443>
        ServerName relabs-puptest1.build.mtv1.mozilla.com
        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.crt
        SSLCertificateKeyFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.key
        SSLCACertificatePath /etc/httpd/ca-path

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        #SSLCARevocationFile     /etc/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient require
        SSLVerifyDepth  2

</VirtualHost>

While you're getting that set up, you're probably wondering where to get this fancy "c_rehash" utility. Don't bother. It's about as simple as:

for i in *.crt; do
        h=`openssl x509 -hash -noout -in $i`
        rm -f $h.0
        ln -s $i $h.0
done

As a side-note, the results of verification by sclient and sserver are not very obvious. Look for the overall error message near the bottom of the output. Here's the result of a client verification once I had everything put together, with some long uselessness elided:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert puptest-certs/relabs08.build.mtv1.mozilla.com.crt -key puptest-certs/relabs08.build.mtv1.mozilla.com.key -pass pass:clientpass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 5379 bytes and written 1716 bytes
---
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: E30634D9CFCC2FA327282DA813BB550C24ACDF18194E5F13C4981AA55914B5F0
    Session-ID-ctx: 
    Master-Key: 013EB09B066418694D36D74B414BBA42E52DBF0066314B60FC7A74662A60934282B6C37C5C82026F70287E60F4FF9472
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket:
    0000 - 82 5f 17 72 97 bd f3 1e-ec 24 de 69 ab 1e cd 1d   ._.r.....$.i....
    ....
    0520 - 40 05 b3 27 20 00 8d ce-93 a9 48 81 8f 0c 16 5b   @..' .....H....[

    Compression: 1 (zlib compression)
    Start Time: 1336582165
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

note the "Verify return code" at the bottom.

By way of demonstration that the server is actually checking those certs:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert bogus.crt -key bogus.key -pass pass:boguspass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
140283463366472:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1193:SSL alert number 48
140283463366472:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 3984 bytes and written 997 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 07E536F1C69A856857EA95DFD821BD6BBD499B5710642F9396D9525637EAD17C03064D5115B3D7F517EDE189E7AF40F8
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Compression: 1 (zlib compression)
    Start Time: 1336582289    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Note the handshake failures near the top, where httpd closed the connection on the client.

The next step is to make CRLs work properly, since Puppet uses them extensively.

Setting up a buildslave instance remotely on OS X Lion

nospam@example.com (Dustin J. Mitchell) — Sat, 17 Mar 2012 18:05:47 -0500

Byrce Lelbach has generously offered access to an OS X system as a metabuildbot slave. As I went about setting it up today, the process was not obvious, so I thought I'd share. This was interesting mostly because I only have SSH access to the host, so I cannot download things from the Apple Store or do any of the fancy point-and-click stuff that would make this easier.

First, I needed to get XCode installed. Note that the (much quicker to download) XCode command-line tools are not sufficient to build everything in MacPorts -- in particular, they do not support building zlib, which is required for git-core.

I got my hands on a copy of "Install XCode.app", and:

host:Downloads buildbot$ cd Install\ Xcode.app/Contents/
host:Contents buildbot$ sudo installer -package Resources/Xcode.mpkg -target /
Password:
installer: Package name is Xcode
installer: Upgrading at base path /
installer: The upgrade was successful.

Once this was done, I installed MacPorts:

host:Downloads buildbot$ hdiutil mount MacPorts-2.0.4-10.7-Lion.dmg
Checksumming Driver Descriptor Map (DDM : 0)…
     Driver Descriptor Map (DDM : 0): verified   CRC32 $A913D2D8
Checksumming Apple (Apple_partition_map : 1)…
....
     Apple (Apple_partition_map : 1): verified   CRC32 $A1DF5DC1
Checksumming disk image (Apple_HFS : 2)…
...... (...) .....
          disk image (Apple_HFS : 2): verified   CRC32 $5A3E74A0
Checksumming  (Apple_Free : 3)…
                    (Apple_Free : 3): verified   CRC32 $00000000
verified   CRC32 $D9641854
/dev/disk2              Apple_partition_scheme
/dev/disk2s1            Apple_partition_map
/dev/disk2s2            Apple_HFS                       /Volumes/MacPorts-2.0.4
host:Downloads buildbot$ pushd /Volumes/MacPorts-2.0.4/
/Volumes/MacPorts-2.0.4 ~/Downloads
host:MacPorts-2.0.4 buildbot$ sudo installer -package MacPorts-2.0.4.pkg/ -target /
Password:
installer: Package name is MacPorts-2.0.4
installer: Installing at base path /
installer: The install was successful.
host:MacPorts-2.0.4 buildbot$ popd
host:Downloads buildbot$ hdiutil unmount /Volumes/MacPorts-2.0.4

and we're off to the races.

I added /opt/local/bin to my path as suggested, and then followed the normal MacPorts setup process.

Finishing up the buildslave install required installing Git (which manages to pull in unreasonable amounts of other stuff!)

host:Contents buildbot$ sudo  /opt/local/bin/port install git-core -credential_osxkeychain-doc-pcre-python27

which is required for the source steps, then creating a virtualenv to install buildbot-slave:

host:~ buildbot$ virtualenv sandbox
New python executable in sandbox/bin/python
Installing setuptools............done.
Installing pip...............done.
host:~ buildbot$ source sandbox/bin/activate
(sandbox)host:~ buildbot$ pip install buildbot-slave
...

and then create and start a slave:

(sandbox)host:~ buildbot$ buildslave create-slave buildslave buildbot.buildbot.net:9989 HOSTNAME PASS
...
(sandbox)host:~ buildbot$ buildslave start buildslave
...

I then followed the helpful advice here to set up a plist that will start the daemon on boot.

IT and Community

nospam@example.com (Dustin J. Mitchell) — Wed, 16 Nov 2011 23:05:24 -0600

Mozilla's IT team is pivoting to a more community-focused approach. Our director of IT, mrz, has been writing extensively about it over the last few weeks.

As you can imagine, the difficult part of this is to balance security with accessibility. We'd like to be open, but we can't give the keys to the kingdom out to anyone who promises to help. The approach we're taking is to treat volunteers as we would part-time employees - post positions, interview, and then supervise to gain trust. This is a fairly common model, actually, for any organization with volunteers and a need for security. Youth programs, for example, generally do an interview and background check with new volunteers, and those volunteers will be paired with senior volunteers or staff for a while.

However, it's a bit cumbersome, both for Mozilla and for potential volunteers. We must design entire positions - ongoing tasks or roles that a volunteer can work on for an extended period of time - and then select a limited number of volunteers to fill those roles. For potential volunteers, an application and interview can mean a long time (weeks?) before they get to do anything hands-on. It also carries the risk that we'd have to turn a qualified volunteer away due to lack of suitable positions.

So what to do?

We need a more fluid way of interacting with potential contributors. Since our bug database is public, we can begin by simply tagging a few bugs that are appropriate for newcomers -- things that don't require sensitive access and are well-encapsulated so they can be completed without extensive knowledge of Mozilla's infrastructure.

Here's the list.

It's a bit short right now. There are a few things that may help:

We can get better about identifying appropriate tasks and projects and making bugs out of them.
We can identify a means of giving limited or sandboxed access to a new volunteer.
Consumers of Mozilla's IT resources can begin tagging bugs, where Mozilla can provide the resources and volunteers can do the heavy lifting - got any ideas?

Subscribe to a google group with a different address?

nospam@example.com (Dustin J. Mitchell) — Fri, 02 Sep 2011 15:04:00 -0500

Google Groups is one place where, IMHO, Google pushes its hegemony too far, making it difficult to use. I wanted to subscribe to puppet-users with my Mozilla address, but since I have a Google account, Groups assumes I want to subscribe with that address. No!

I found the fix with a bit of Googling (some irony there). It involves editing a URL:

http://groups.google.com/group/puppet-users/boxsubscribe?email=email@domain.com

where you'd substitute the name of the group you want for puppet-users and add your email at the end.

Nagios NSCA from Python

nospam@example.com (Dustin J. Mitchell) — Fri, 20 May 2011 16:55:11 -0500

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

Amanda's Transfer Mechanisms

nospam@example.com (Dustin J. Mitchell) — Sat, 22 Jan 2011 14:47:13 -0600

There's been a bit of confusion on the mailing list and IRC about how Amanda assembles transfers out of transfer elements, and how transfer mechanisms influence that.

In the final form of a transfer, any two adjacent elements must have the same mechanism. For example is, an upstream element speaking XFER_MECH_PUSH_BUFFER cannot talk to a downstream element using XFER_MECH_READ_FD (nor, more confusingly, XFER_MECH_PULL_BUFFER). So each mechanism is an isolated definition of "here's how upstream and downstream should talk". They come in pairs because generally anything upstream can do to downstream (e.g., upstream can write to downstream's fd) can occur in reverse (e.g., downstream can read from upstream's fd).

What makes this confusing is that if you specify a set of elements which can't talk directly to one another, then xfer.c will add "glue" elements between the specified elements. To make that concrete, imagine you specify a transfer as

source-holding --> filter-xor --> dest-fd

(if you like practical examples, then imagine filter-xor is a buffer-based decompression filter, and you're pulling data from holding disk, decompressing, and sending to a pipe -- something amfetchdump would do). Here are the mechanisms supported by each element:

source-holding:
 XFER_MECH_PULL_BUFFER

filter-xor
 XFER_MECH_PULL_BUFFER (input) and
 XFER_MECH_PULL_BUFFER (output)
or
 XFER_MECH_PUSH_BUFFER (input) and
 XFER_MECH_PUSH_BUFFER (output)

dest-fd
 XFER_MECH_WRITEFD (input)

In putting these together, source-holding and filter-xor can use the same mechanism (PULL_BUFFER). This leaves filter-xor using PULL_BUFFER for output, but dest-fd does not support this. So xfer.c adds a glue element that can speak PULL_BUFFER on input and WRITEFD on output. This element basically loops in a thread, calling upstream->pull_buffer and write(downstream->input_fd, buffer). So the final xfer looks like

 source-holding --(PULL_BUFFER)--> filter-xor --(PULL_BUFFER)--> glue --(WRITEFD)--> dest-fd

Hopefully that helps to explain how the glue works.

Note that one of the cool things about this arrangement is that in most cases the complexity is in the glue, not the elements. In fact, in this case the glue provides the only thread that's required to run this transfer, so the other three element implementations don't need to manage threads at all.

virtualenv for Perl

nospam@example.com (Dustin J. Mitchell) — Thu, 11 Nov 2010 10:48:43 -0600

I absolutely love virtualenv for Python development. It allows me to develop Buildbot against several versions of Python and several versions of its dependencies, without modifying my system's Python installation at all!

Now, I need to do the same thing in Perl. So I thought I'd compare the two side-by-side.

virtualenv

# install virtualenv locally
wget http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py
# set up a sandbox
python virtualenv.py sandbox
# activate it
source sandbox/bin/activate
# start installing stuff
easy_install buildbot

local::lib

There is no local::lib gentoo ebuild! I'm sure there's a good reason, but that's odd all the same!

# install local::lib locally
wget http://search.cpan.org/CPAN/authors/id/G/GE/GETTY/local-lib-1.006007.tar.gz
tar -zxf local-lib-1.006007.tar.gz
cd local-lib-1.006007
perl Makefile.PL --bootstrap # (accept lots of defaults)
make test && make install
# activate it (permanently)
echo 'eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)' >>~/.bashrc
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
# start installing stuff
perl -MCPAN -e install Config::General # (I have to accept the same defaults again??)

I think virtualenv is the clear winner here!

Firefox 4.0b7 - a few tweaks

nospam@example.com (Dustin J. Mitchell) — Wed, 10 Nov 2010 15:44:46 -0600

The new beta of Firefox 4.0 was released today. I'm not quite willing to run Minefield (nightlies), so I've been eagerly awaiting this beta to fix some nagging but not show-stopper bugs in 4.0b6. One of those involved bad interactions of App Tabs with Panorama. Now the app tabs nicely decorate the side of each tab set in the panorama view.

Another nice thing is that the Option-Space key combination, which opened panorama in 4.0b6, no longer does so. That's OK - I found that to be too easy to press anyway. It's now bound to Command-E (right there at the top of the "View" menu).

Panorama has also been re-bound to swipe-up and swipe-down, which makes me less happy. In most apps on the Mac, those swipes are equivalent to the "Home" and "End" keystrokes -- they scroll to the top or bottom of the current page. So with a little help from my new co-workers, I discovered the settings to fix that.

The full list of gesture bindings is written up here, but the two I needed to change are browser.gesture.swipe.up and .down. The scrolling commands to bind to them are cmd_scrollTop and cmd_scrollBottom.

4.0 has a number of other great UI enhancements, too. I'm excited to see 4.0 finally released!

[edit: fixed formatting]

irssi settings for status-in-nick

nospam@example.com (Dustin J. Mitchell) — Fri, 22 Oct 2010 12:06:04 -0500

I am getting started in releng at Mozilla, and IRC provides a central meeting-place for the group. As such, indicating your status to this group is an important, so others can know whether you're nearby to answer a question or take care of a problem. This is generally done by adding suffixes to the IRC nickname, e.g., "dustin|afk" or "dustin|lunch".

Before I go further, I know that this is frown on, and even results in autoignores, in some corners of IRC. If that's the case for you, read no further. Irssi's documentation is utterly incomplete. So the only way to figure out how to configure something or write a script is to find and adapt examples. So hopefully this entry will feed back into that pool.

What I want is an easy way to bind some keystrokes to commands like /NICK dustin|afk. However, I want to be very careful not to set this nick on other chatnets, particularly since I'm generally known as djmitche there, not dustin. So I began by writing a script that can double-check this:

use strict;

use vars qw ($VERSION %IRSSI);

$VERSION = 'v1.0';
%IRSSI = (
          name        => 'moznick',
          authors     => 'dustin',
          contact     => 'dustin@mozilla.com',
          url         => 'http://code.v.igoro.us/',
          license     => 'GPLv2',
          description => 'Sets my nick status for Mozilla co-workers',
         );

use Irssi;

my $last_nick = '';

sub is_mozilla {
    my ($server) = @_;
    if (!$server || $server->{'chatnet'} eq 'mozilla') {
        return 1;
    }

    Irssi::print("This isn't a mozilla channel!");
    return 0;
}

sub cmd_moznick {
    my ($data, $server, $channel) = @_;

    if (is_mozilla($server)) {
        my $new_nick = $data? "dustin|$data" : "dustin";
        return if ($new_nick eq $last_nick);

        $server->command("NICK $new_nick");
        Irssi::print("nick set to $new_nick");
        $last_nick = $new_nick;
    }
}

Irssi::command_bind("moznick", "cmd_moznick");

This just adds a /MOZNICK command that will set my nick, but only on a Mozilla chatnet. Then I added an alias and some bindings. I don't like irssi's configuration-writing stuff, as it's several times blown away my configuration. Instead, I edit the config file directly. So I have:

aliases = { 
    moznick_moz = "window goto #build; moznick $*";
};

keyboard = ( 
    { key = "meta-space"; id = "multi"; data = "command moznick_build"; },
    { key = "meta-z"; id = "multi"; data = "command moznick_build brb"; },
    { key = "meta-x"; id = "multi"; data = "command moznick_build away; command away"; }
);

The advantage of the window goto in the alias is that it will switch to a channel on the mozilla chatnet. I'd love to have a way to just switch to that chatnet without changing windows, but this isn't bad.

IPv6 and Amanda

nospam@example.com (Dustin J. Mitchell) — Fri, 16 Jul 2010 22:04:00 -0500

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case. First, a bit of background on how Amanda's networking code works, and what had to change to support IPv6. Amanda supports security mechanisms called BSD (the oldest), BSDUDP, and BSDTCP. These all authenticate (if you can call it that) using the same sorts of checks that rsh uses. The incoming connection is accepted if:

it is from a "reserved" port (less than 1024);
the address of the initiator has complementary forward and reverse DNS records in place; and
the initiator's hostname is in .amandahosts

During a backup operation, the Amanda server contacts each client host. When using the BSD authentications, this triggers amandad, which checks the above restrictions before beginning communication with the server. This initial connection is packet-based, and can be carried out over UDP (for BSD and BSDUDP) or TCP (BSDTCP). When a dump begins, several "streams" are opened to transmit the data, index, and metadata. For BSD and BSDUDP, each stream is implemented as a distinct TCP connection, where the client sends a port number to the server and the server connects to that port. BSDTCP multiplexes all streams over a single TCP connection using a basic type/length packet encapsulation.

The first challenge in adding IPv6 support was to deal properly with IPv6 addresses when querying the DNS. That meant switching to getaddrinfo and getnameinfo, as suggested by Jun-ichiro itojun Itoh. These functions bring their own compatibility problems, but Amanda uses gnulib, which provides compatibile implementations on systems where they are not available, minimizing the difficulty.

We had a lot of trouble from systems such as RHEL3 possessing IPv6 support in the compiler environment but not in the kernel. On such systems, code using constants like AF_INET6 or AI_V4MAPPED would compile without problems, but fail at runtime. We added a WORKING_IPV6 preprocessor conditional, without which all references to IPv6-related symbols were removed. At configure time, Amanda tries to create an IPv6 socket, and sets this conditional to true if it succeeds. The --without-ipv6 configure option forcibly disables IPv6 support.

The sockaddr structures and API for IPv6 are fairly difficult to use, particularly if it's not known in advance what sort of address they will contain. We added a set of macros and utility functions in sockaddr-util.c and sockaddr-util.h. Using these macros throughout Amanda removed a significant amount of code that was conditionalized on both compile-time support and runtime address family, and centralized that logic in one easily-maintained place.

On our build systems, we had to deal with different levels of support in the compile environment and the kernel. This is fine: most Amanda users install binary packages that are produced on roughly the same OS distribution and version as was used for the build, so the kernel support is generally the same. However, a third variable has tripped up lots of Amanda users: system configuration. In particular, several newer Linux distributions have shipped with localhost resolving to ::1 vi /etc/hosts, but without enough interface configuration to actually utilize a socket bound to that address. Amanda uses localhost sockets for inter-process communication, so this misconfiguration causes backup operations to fail. The solution is to either finish configuring IPv6 on the host, remove the reference to ::1 in /etc/hosts, or build Amanda with --without-ipv6.

I have not yet heard of an Amanda installation where IPv6 communication is in use. But I have heard from countless IPv4 users whose Amanda installations have failed due to bad IPv6 support. At the moment, then, I feel that adding IPv6 support to Amanda has been a net negative for the project. Although there is doubtless room for improvement, I will not entertain patches for better IPv6 support, for fear they will introduce new bugs for our exclusively IPv4 userbase.

Of course, all of this may change as dual-stack networks grow more prevalent and are replaced by pure IPv6 networks!

SSH With Snow Leopard

nospam@example.com (Dustin J. Mitchell) — Sat, 10 Jul 2010 14:27:00 -0500

I just upgraded my Macbook to Snow Leopard, and the upgrade has changed the way SSH authentication works. I have set up a system I like quite a bit, now, and thought I would share. My usage pattern is that I do most of my work via GNU screen running on my login server, euclid. So I want a simple procedure that will connect me to that screen session, with a proper SSH agent set up.

Snow Leopard automatically starts an ssh-agent process at login. This is great, but does not interoperate with SSHKeychain. Dropping SSHKeychain is OK with me - I don't use SSH tunnels, so it really only acted as a GUI for passphrase entry. It also ate CPU occasionally, which of course causes the macbook to become more painfully hot than usual.

So I have three problems to solve: 1. automatically add my key to ssh-agent; 2. automatically expire the key at appropriate times (at my paranoia level, that's at system sleep); and 3. make multiple agent instances usable from the same shell session on the server.

Adding the Key at Connection Time

Adding the key is relatively straightforward. I wrote a short script that Terminal runs when I hit ⌘-N or ⌘-T:

#! /bin/bash

# does ssh-agent not have a key?
if ! ssh-add -l; then
    ssh-add ~/.ssh/dustin || exit 1
fi

exec ssh -x -t euclid.r.igoro.us bin/startscreen

This will prompt me for the passphrase when there is not already a key active, but proceed directly to the ssh invocation if the key situation is OK. The -x option to ssh is there to turn off X11 forwarding; without this option, SSH will helpfully start the X11 app. I think this is a great feature, but I don't use X11 apps very often, so I've disabled it.

Expiring the Key Automatically

Mac OS has a nicely designed system in place to allow applications to get notified when the system is changing power states. However, I wasn't interested in writing a full OS X app for this particular project. Enter sleepwatcher. This is a beautifully simple program that just executes scripts on particular power events. I set this up to run via launchd, as directed in the README, and to run ~/.ssh/sleep on sleep. That script contains:

#! /bin/bash

# first, don't inherit a socket (sleepwatcher doesn't get the user's env)
SSH_AUTH_SOCK=

# find some sockets
echo ".ssh/sleep:" `id`
for sock in /tmp/launch-*/Listeners; do
    if [ -w $sock ]; then
        echo "Trying to remove .ssh/dustin from socket $sock"
        SSH_AUTH_SOCK=$sock /opt/local/bin/ssh-add -d ~/.ssh/dustin
    else
        echo "Skipping unwritable socket $sock"
    fi  
done

The for loop is required because a script run from sleepwatcher doesn't inherit the SSH_AUTH_SOCK variable that points to the running SSH agent. The loop simply searches for a writable SSH socket of the pattern used by the system's agent.

SSH Agent and Screen

If you naïvely set up an SSH agent, connect to a remote system, and start screen there, things will work great - until you disconnect from the screen session. When you connect to the remote system, SSH forwards the agent connection for you, and sets SSH_AUTH_SOCK on the remote system to point to this forwarded socket. Screen passes this variable along blindly, so it appears in all of the shells opened in screen windows, and things work as you'd expect. When that SSH connection is removed, and a new one established, the forwarded agent appears at a new socket. But those shells running in screen windows are still pointing to the old name, and will no longer be able to connect.

The fix is to create a socket with a well-known name that will not change from connection to connection. The following script takes care of it. WARNING: this script is vulnerable to /tmp race conditions. I am the only user on my servers, so this doesn't bother me, but fixing it should be relatively straightforward.

#! /bin/bash
# hard-link the SSH socket to one with a fixed name on the local
# machine, and set SSH_AUTH_SOCK to point to that fixed name.  Later
# invocations of this script will change the link, but the name will
# remain valid, allowing existing shells to continue to function.
setup_fixed_socket() {
  local old_socket="$SSH_AUTH_SOCK"
  local socket_dir="/tmp/$(uname -n)-$(id -u)"
  local socket_file=$socket_dir/agent

  # set up the directory and permissions
  [ -e $socket_dir ] || mkdir -p $socket_dir
  chmod 700 $socket_dir

  # remove an existing link
  [ -e $socket_file ] && rm $socket_file

  # hard-link in the new one
  ln $old_socket $socket_file

  # return the new socket
  echo $socket_file
}

# this variable will be exported to every shell opened by this
# invocation of screen -- even subsequent connections to it.  This
# variable may live for days or weeks.
export SSH_AUTH_SOCK=$(setup_fixed_socket)

# finally, fire up screen.  Try reattaching to a running
# session; otherwise start up a new one
screen -R -DD ${@} || screen

What's New in Amanda: The End of Fragmentation

nospam@example.com (Dustin J. Mitchell) — Thu, 08 Jul 2010 18:07:00 -0500

Most of my posts in this series have been about features that are available in a released version of Amanda. This time, I want to share a project I'm working on right now - one that will be available in Amanda-3.2. I'm reworking the way Amanda writes its data to tape (or any other kind of storage) to make it more efficient, more reliable, and simpler to configure.

Historically, Amanda's conservative approach to finicky tape hardware has meant that it wasted some space at the end of each tape. With the changes I'm working on, Amanda will no longer waste this space, and can also avoid some needless copying of data in most cases, with a minimum of additional risk.

Amanda's Storage Format

Before examining the new functionality, let's look at Amanda's storage format. Amanda treats all storage devices like tapes¹ - a set of sequentially numbered data files of arbitrary size, each composed of a sequence of fixed-size blocks. Each file begins with a one-block header that identifies the dump and gives information about its contents. The header is followed by blocks of raw data.

Amanda supports writing a dump across multiple tapes - spanning.² The technique is this: a dump is split into a sequence of parts, and each part is written a a single file on a volume. During recovery, Amanda reads the parts in sequence, and concatenates their data to reproduce the original dumpfile. Usually all parts are the same size, and this size is generally 1-5% of the tape capacity.

Better Safe than Sorry - At a Cost

Amanda was originally designed around tape drives - in fact, if you look at the history of Linux kernel support for tape drives, it is closely intertwined with Amanda development. Tape drives are finicky beasts, and in many cases cannot distinguish the end of the tape (called EOM) from any other fatal error. Worse, tape drives employ large caches to ensure they can write continuously, and when an error occurs all of the data in that cache is lost, and there is no way for Amanda to determine how much actually made it onto the tape. Beginning a new on-tape file (writing a filemark) flushes the cache and signals any errors immediately.

Since time immemorial, then, Amanda has treated any error from the tape drive as EOM, and assumed that all data written since the last filemark is potentially corrupt. That means that the part in progress when the error occurred is logged as PARTIAL, and Amanda will start at the beginning of that part on the next tape.

The PARTIAL part is recorded in the catalog, but will not be used for recovery, so it is effectively wasted space. A little arithmetic will tell you that, on average, each tape will waste half of the part size. This is at least excusable with real tape drives; with vtapes (disk), this wasted space is completely unnecessary. Worse, vtapes are most flexible when they are kept small and dumps are spanned over many vtapes per night; but the wasted space increases linearly with the number of vtapes used.

In order to rewind a part and write it again on a new tape, Amanda also needs to keep its data somewhere, called the part cache. When the dump is on the holding disk, the holding disk acts as a part cache. Otherwise, Amanda can cache parts in memory or on disk. Caching in memory is faster, but requires a lot of RAM for a reasonable part size. Caching on disk allows larger parts, but is considerably slower.

Logical EOM

More recent tape drives (those made in the last decade or so) have a feature called "early warning". With this feature enabled, the drive alerts the host system when it is "near" EOM, and flushes the cache to tape. Exactly what "near" means is not specified in the SCSI standard, but in general there's room to flush the cache and write a filemark, at least. This is sometimes called a logical EOM - LEOM.

Amanda can take advantage of this functionality to cleanly finish a part before running headlong into a physical EOM. This eliminates the wasted space for a PARTIAL part, and also eliminates the need to cache parts, since rewinding is not required. In one small change (well, OK, it's about 4,300 lines), Amanda gets faster and uses storage space more efficiently. What's not to love?

Better yet, all of the non-tape devices (vtapes, S3 devices, DVD-ROMs, etc.) can easily emulate LEOM, so backups to these devices will automatically benefit from this improvement.

In the Code

Three important patches toward this functionality were just committed. What remains is to set up real LEOM support for the VFS device (vtapes) and for the tape device.

The VFS device can, of course, trivially emulate LEOM when it is enforcing the MAX_VOLUME_USAGE property - the vtape length. However, predicting when a filesystem will run out of space is much more difficult. We are still discussing options, and I would love to hear suggestions here or on the mailing list.

As for the tape device, it will assume that LEOM is not supported unless the user configures it explicitly (with the "LEOM" device property) or we can determine support for LEOM from the operating system at runtime. Unfortunately, this is one of those areas so technical that only a half-dozen people know how it works, so it may take me some time to track down this information for non-Linux operating systems. Again, advice and assistance is welcome!

¹This is an ages-old design decision, but one that artificially constrains Amanda's flexibility, especially with vtapes.
²In fact, Amanda has supported spanning for something like 7 years now. Yet I occasionally see users in #amanda complaining about this serious limitation and wondering when we're going to do anything about it. Will 2003 be soon enough?

What's New in Amanda: Hackability

nospam@example.com (Dustin J. Mitchell) — Thu, 01 Jul 2010 22:29:00 -0500

It's been a while since I've posted about recent development in Amanda, but it's not for lack of interesting topics!

Today I want to talk a little bit about Amanda's development. Historically, Amanda has always had a small, core group of developers who do the lion's share of the development work. There are probably lots of reasons for this, not least of which is that a backup application isn't the sexiest project on which to spend your spare time. But I think there's a deeper reason, and it has to do with hackability. Amanda was originally written in C, which means any changes require the full set of developer tools. For example, just to fix a typo in an error message, you would need to find the source, configure and compile it, find and fix the error message, and recompile to test. Fixing something more substantial in the highly interdependent Amanda codebase also requires a deep understanding of many parts of Amanda - from the obscure configuration interface to the oddly interlinked disklist structure. This level of programming skill is not common among Amanda's user base (systems administrators), and I can count the people who understand the disklist structure on one hand.

The result has been a paltry flow of patches from anyone but the core hackers. Furthermore, no entry path has been available by which newcomers could work their way up to being core developers. While I don't want to disparage the work of any of the great programmers who have written Amanda over the years, it's a shame that there have been so few at any time, and I worry about what would happen if the number were to reach zero. So what to do?

$hackability++

We've done a few things to try to make Amanda more hackable. Probably the biggest change is to rewrite parts of Amanda in Perl. I'm asked "why" quite often, and while we had a lot of reasons, two relate directly to hackability. First, more sysadmins know Perl than C, because Perl is quite often used to build the "glue" that links systems together. Interestingly, based on many conversations, it seems that Python may also have been a good choice, as I suspected when I first proposed the rewrite. But it's too late now!

Second, and more importantly, Perl code can be hacked in place. If amvault isn't acting the way you want it to, just open up /usr/sbin/amvault and tweak away. No need to download the source, no need to compile, no segmentation faults, just hacking. When you're done, run a quick diff and send the results to amanda-hackers.

Even users who do not know Perl can take advantage of this in-situ hackability. Within Amanda's C code, if I want a user to try a patch, that user must figure out how to download Amanda's source, apply the patch, configure, compile, and install. None of those steps are trivial. With Perl code, I can often provide a patch that is simple enough to be applied directly to the installed executables by hand, or with a simple application of patch. Everyone stays focused on the bug under investigation, and the user's backups are up and running that much more quickly.

New APIs

As I mentioned before, historically Amanda's code has been highly interdependent. Details of the implementation of the holding disk were spread over most of the files in the server implementation. The dumplevel -987 has a special meaning that is documented nowhere, but referenced in several source files. All of this makes new development difficult, because it's impossible to "slice off" and study a portion of Amanda in isolation.

The solution here is to create abstract interfaces, where new functionality can be "plugged in" and Amanda can use it without changes. The Amanda developers have abused the term "API" for these interfaces, and we now have quite a few:

Application API - an abstraction of backup clients, e.g., ampgsql for Postgres;
Device API - an abstraction of backend storage devices, such as tape, disk, cloud, or DVD-RW;
Changer API - an abstraction of tape changers and other mechanisms for selecting from a set of volumes; and
Script API - a means of invoking scripts before or after certain events during a backup.

This strategy has already paid off: we have seen several new scripts and applications contributed, and the DVD-RW device arrived out of the blue as a contribution from someone who found it useful.

Other Changes

In the interest of greater accessibility to new hackers, we have also put Amanda on github and created a set of good "beginner" projects. Zmanda has even offered to pay people to hack on Amanda, as a way of easing the cost of entry. I also try to point out interesting projects on the Amanda mailing list, particularly projects that Jean-Louis and I probably will not find time to work on.

The idea here is to encourage new hackers to pick up a well-scoped project to become familiar with Amanda. The hackers can then move on to more sophisticated projects that meet their particular backup needs or address a particular interest.

Will You Join Me?

So Amanda is ready for you. When can you start?

IPv6 Configuration

nospam@example.com (Dustin J. Mitchell) — Sun, 27 Jun 2010 16:27:12 -0500

I've been meaning to get IPv6 set up on my local network for some time. My only practical reason is that Amanda supports IPv6 and I should test that support. It was also a good chance to re-immerse myself in network configuration, and Hurricane Electric has a neat certification process to add some motivation. I began by getting local IPv6 connectivity set up over the HE tunnel, using my Gentoo systems. This was fairly straightforward, as the Gentoo net scripts natively support IPv6. The firewall system I use (Shorewall) does not support IPv6 directly. Instead, there's a parallel shorewall6 package to install. Aside from the annoyance of setting up two separate firewalls, this did not cause appreciable difficulties. With all of this in place, I was at the "Explorer" level.

The next task was to set up a working IPv6 desktop. My home network uses 802.1q VLAN tagging to layer both an external, publicly routable IPv4 network (99.89.149.16/29 on VLAN 20) and an internal, NAT'd IPv4 network (172.16.1/24 on VLAN 10). I wanted to make VLAN 10 a dual-stack network, rather than invent a new VLAN for my IPv6 network. Initially, I didn't realize that HE uses, in my case, 2001:470:1f10:826::0/64 just for the tunnel (yes, two addresses out of 2⁶⁴ used -- maybe we'll need IPv8 sooner than we think!). I assumed that the /64 I was allocated was to be used for all of my nodes, and tried to subnet it locally, using 2001:470:1f10:826::0/112 for the tunnel and 2001:470:1f10::1/112 for the internal network. This worked with manual configuration, but radvd seemed to always want to advertise a /64. A little reading about the RA protocol showed this to be correct: RA provides the high 64 bits (the network portion), and the clients provide the low 64 bits using EUI-64. I was stymied until I looked at the tunnel details again and noticed that the "Routed IPv6 Prefixes" section listed a different prefix (2001:470:1f11:826/64).

With this in place, the subnet and firewall setup was a breeze. Using a manual configuration on my MacBook, I was able to communicate via IPv6. However, the stateless autoconfiguration didn't work. I briefly tried DHCPv6, but Macs do not support it. The RA client correctly combines the network and EUI-64 components to create a full address, and it correctly copies the link-local address of the router, but it does not set up a default route using that router, making the whole thing fairly useless. A trip to #ipv6 confirmed that Macs are, indeed, broken this way, so I stopped worrying about it.

The remainder of the certification process involved getting Apache, Postfix, and Bind speaking IPv6, none of which was very difficult. I did discover that BIND's $ORIGIN didn't work correctly. A zonefile with

$ORIGIN 6.2.8.0.1.1.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR knuth.r.igoro.us.

didn't work, but spelling out the entire reversed address did. I'm sure this was due to a typo, but several checks didn't reveal anything.

However, I'm now stuck at the Guru level until GoDaddy starts supporting IPv6 glue for the .us TLD. I feel cheated, somehow!