Code V.igoro.us

IT and Community

nospam@example.com (Dustin J. Mitchell) — Wed, 16 Nov 2011 23:05:24 -0600

Mozilla's IT team is pivoting to a more community-focused approach. Our director of IT, mrz, has been writing extensively about it over the last few weeks.

As you can imagine, the difficult part of this is to balance security with accessibility. We'd like to be open, but we can't give the keys to the kingdom out to anyone who promises to help. The approach we're taking is to treat volunteers as we would part-time employees - post positions, interview, and then supervise to gain trust. This is a fairly common model, actually, for any organization with volunteers and a need for security. Youth programs, for example, generally do an interview and background check with new volunteers, and those volunteers will be paired with senior volunteers or staff for a while.

However, it's a bit cumbersome, both for Mozilla and for potential volunteers. We must design entire positions - ongoing tasks or roles that a volunteer can work on for an extended period of time - and then select a limited number of volunteers to fill those roles. For potential volunteers, an application and interview can mean a long time (weeks?) before they get to do anything hands-on. It also carries the risk that we'd have to turn a qualified volunteer away due to lack of suitable positions.

So what to do?

We need a more fluid way of interacting with potential contributors. Since our bug database is public, we can begin by simply tagging a few bugs that are appropriate for newcomers -- things that don't require sensitive access and are well-encapsulated so they can be completed without extensive knowledge of Mozilla's infrastructure.

Here's the list.

It's a bit short right now. There are a few things that may help:

We can get better about identifying appropriate tasks and projects and making bugs out of them.
We can identify a means of giving limited or sandboxed access to a new volunteer.
Consumers of Mozilla's IT resources can begin tagging bugs, where Mozilla can provide the resources and volunteers can do the heavy lifting - got any ideas?

Subscribe to a google group with a different address?

nospam@example.com (Dustin J. Mitchell) — Fri, 02 Sep 2011 15:04:00 -0500

Google Groups is one place where, IMHO, Google pushes its hegemony too far, making it difficult to use. I wanted to subscribe to puppet-users with my Mozilla address, but since I have a Google account, Groups assumes I want to subscribe with that address. No!

I found the fix with a bit of Googling (some irony there). It involves editing a URL:

http://groups.google.com/group/puppet-users/boxsubscribe?email=email@domain.com

where you'd substitute the name of the group you want for puppet-users and add your email at the end.

Nagios NSCA from Python

nospam@example.com (Dustin J. Mitchell) — Fri, 20 May 2011 16:55:11 -0500

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

Amanda's Transfer Mechanisms

nospam@example.com (Dustin J. Mitchell) — Sat, 22 Jan 2011 14:47:13 -0600

There's been a bit of confusion on the mailing list and IRC about how Amanda assembles transfers out of transfer elements, and how transfer mechanisms influence that.

In the final form of a transfer, any two adjacent elements must have the same mechanism. For example is, an upstream element speaking XFER_MECH_PUSH_BUFFER cannot talk to a downstream element using XFER_MECH_READ_FD (nor, more confusingly, XFER_MECH_PULL_BUFFER). So each mechanism is an isolated definition of "here's how upstream and downstream should talk". They come in pairs because generally anything upstream can do to downstream (e.g., upstream can write to downstream's fd) can occur in reverse (e.g., downstream can read from upstream's fd).

What makes this confusing is that if you specify a set of elements which can't talk directly to one another, then xfer.c will add "glue" elements between the specified elements. To make that concrete, imagine you specify a transfer as

source-holding --> filter-xor --> dest-fd

(if you like practical examples, then imagine filter-xor is a buffer-based decompression filter, and you're pulling data from holding disk, decompressing, and sending to a pipe -- something amfetchdump would do). Here are the mechanisms supported by each element:

source-holding:
 XFER_MECH_PULL_BUFFER

filter-xor
 XFER_MECH_PULL_BUFFER (input) and
 XFER_MECH_PULL_BUFFER (output)
or
 XFER_MECH_PUSH_BUFFER (input) and
 XFER_MECH_PUSH_BUFFER (output)

dest-fd
 XFER_MECH_WRITEFD (input)

In putting these together, source-holding and filter-xor can use the same mechanism (PULL_BUFFER). This leaves filter-xor using PULL_BUFFER for output, but dest-fd does not support this. So xfer.c adds a glue element that can speak PULL_BUFFER on input and WRITEFD on output. This element basically loops in a thread, calling upstream->pull_buffer and write(downstream->input_fd, buffer). So the final xfer looks like

 source-holding --(PULL_BUFFER)--> filter-xor --(PULL_BUFFER)--> glue --(WRITEFD)--> dest-fd

Hopefully that helps to explain how the glue works.

Note that one of the cool things about this arrangement is that in most cases the complexity is in the glue, not the elements. In fact, in this case the glue provides the only thread that's required to run this transfer, so the other three element implementations don't need to manage threads at all.

virtualenv for Perl

nospam@example.com (Dustin J. Mitchell) — Thu, 11 Nov 2010 10:48:43 -0600

I absolutely love virtualenv for Python development. It allows me to develop Buildbot against several versions of Python and several versions of its dependencies, without modifying my system's Python installation at all!

Now, I need to do the same thing in Perl. So I thought I'd compare the two side-by-side.

virtualenv

# install virtualenv locally
wget http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py
# set up a sandbox
python virtualenv.py sandbox
# activate it
source sandbox/bin/activate
# start installing stuff
easy_install buildbot

local::lib

There is no local::lib gentoo ebuild! I'm sure there's a good reason, but that's odd all the same!

# install local::lib locally
wget http://search.cpan.org/CPAN/authors/id/G/GE/GETTY/local-lib-1.006007.tar.gz
tar -zxf local-lib-1.006007.tar.gz
cd local-lib-1.006007
perl Makefile.PL --bootstrap # (accept lots of defaults)
make test && make install
# activate it (permanently)
echo 'eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)' >>~/.bashrc
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
# start installing stuff
perl -MCPAN -e install Config::General # (I have to accept the same defaults again??)

I think virtualenv is the clear winner here!

Firefox 4.0b7 - a few tweaks

nospam@example.com (Dustin J. Mitchell) — Wed, 10 Nov 2010 15:44:46 -0600

The new beta of Firefox 4.0 was released today. I'm not quite willing to run Minefield (nightlies), so I've been eagerly awaiting this beta to fix some nagging but not show-stopper bugs in 4.0b6. One of those involved bad interactions of App Tabs with Panorama. Now the app tabs nicely decorate the side of each tab set in the panorama view.

Another nice thing is that the Option-Space key combination, which opened panorama in 4.0b6, no longer does so. That's OK - I found that to be too easy to press anyway. It's now bound to Command-E (right there at the top of the "View" menu).

Panorama has also been re-bound to swipe-up and swipe-down, which makes me less happy. In most apps on the Mac, those swipes are equivalent to the "Home" and "End" keystrokes -- they scroll to the top or bottom of the current page. So with a little help from my new co-workers, I discovered the settings to fix that.

The full list of gesture bindings is written up here, but the two I needed to change are browser.gesture.swipe.up and .down. The scrolling commands to bind to them are cmd_scrollTop and cmd_scrollBottom.

4.0 has a number of other great UI enhancements, too. I'm excited to see 4.0 finally released!

[edit: fixed formatting]

irssi settings for status-in-nick

nospam@example.com (Dustin J. Mitchell) — Fri, 22 Oct 2010 12:06:04 -0500

I am getting started in releng at Mozilla, and IRC provides a central meeting-place for the group. As such, indicating your status to this group is an important, so others can know whether you're nearby to answer a question or take care of a problem. This is generally done by adding suffixes to the IRC nickname, e.g., "dustin|afk" or "dustin|lunch".

Before I go further, I know that this is frown on, and even results in autoignores, in some corners of IRC. If that's the case for you, read no further. Irssi's documentation is utterly incomplete. So the only way to figure out how to configure something or write a script is to find and adapt examples. So hopefully this entry will feed back into that pool.

What I want is an easy way to bind some keystrokes to commands like /NICK dustin|afk. However, I want to be very careful not to set this nick on other chatnets, particularly since I'm generally known as djmitche there, not dustin. So I began by writing a script that can double-check this:

use strict;

use vars qw ($VERSION %IRSSI);

$VERSION = 'v1.0';
%IRSSI = (
          name        => 'moznick',
          authors     => 'dustin',
          contact     => 'dustin@mozilla.com',
          url         => 'http://code.v.igoro.us/',
          license     => 'GPLv2',
          description => 'Sets my nick status for Mozilla co-workers',
         );

use Irssi;

my $last_nick = '';

sub is_mozilla {
    my ($server) = @_;
    if (!$server || $server->{'chatnet'} eq 'mozilla') {
        return 1;
    }

    Irssi::print("This isn't a mozilla channel!");
    return 0;
}

sub cmd_moznick {
    my ($data, $server, $channel) = @_;

    if (is_mozilla($server)) {
        my $new_nick = $data? "dustin|$data" : "dustin";
        return if ($new_nick eq $last_nick);

        $server->command("NICK $new_nick");
        Irssi::print("nick set to $new_nick");
        $last_nick = $new_nick;
    }
}

Irssi::command_bind("moznick", "cmd_moznick");

This just adds a /MOZNICK command that will set my nick, but only on a Mozilla chatnet. Then I added an alias and some bindings. I don't like irssi's configuration-writing stuff, as it's several times blown away my configuration. Instead, I edit the config file directly. So I have:

aliases = { 
    moznick_moz = "window goto #build; moznick $*";
};

keyboard = ( 
    { key = "meta-space"; id = "multi"; data = "command moznick_build"; },
    { key = "meta-z"; id = "multi"; data = "command moznick_build brb"; },
    { key = "meta-x"; id = "multi"; data = "command moznick_build away; command away"; }
);

The advantage of the window goto in the alias is that it will switch to a channel on the mozilla chatnet. I'd love to have a way to just switch to that chatnet without changing windows, but this isn't bad.

IPv6 and Amanda

nospam@example.com (Dustin J. Mitchell) — Fri, 16 Jul 2010 22:04:00 -0500

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case. First, a bit of background on how Amanda's networking code works, and what had to change to support IPv6. Amanda supports security mechanisms called BSD (the oldest), BSDUDP, and BSDTCP. These all authenticate (if you can call it that) using the same sorts of checks that rsh uses. The incoming connection is accepted if:

it is from a "reserved" port (less than 1024);
the address of the initiator has complementary forward and reverse DNS records in place; and
the initiator's hostname is in .amandahosts

During a backup operation, the Amanda server contacts each client host. When using the BSD authentications, this triggers amandad, which checks the above restrictions before beginning communication with the server. This initial connection is packet-based, and can be carried out over UDP (for BSD and BSDUDP) or TCP (BSDTCP). When a dump begins, several "streams" are opened to transmit the data, index, and metadata. For BSD and BSDUDP, each stream is implemented as a distinct TCP connection, where the client sends a port number to the server and the server connects to that port. BSDTCP multiplexes all streams over a single TCP connection using a basic type/length packet encapsulation.

The first challenge in adding IPv6 support was to deal properly with IPv6 addresses when querying the DNS. That meant switching to getaddrinfo and getnameinfo, as suggested by Jun-ichiro itojun Itoh. These functions bring their own compatibility problems, but Amanda uses gnulib, which provides compatibile implementations on systems where they are not available, minimizing the difficulty.

We had a lot of trouble from systems such as RHEL3 possessing IPv6 support in the compiler environment but not in the kernel. On such systems, code using constants like AF_INET6 or AI_V4MAPPED would compile without problems, but fail at runtime. We added a WORKING_IPV6 preprocessor conditional, without which all references to IPv6-related symbols were removed. At configure time, Amanda tries to create an IPv6 socket, and sets this conditional to true if it succeeds. The --without-ipv6 configure option forcibly disables IPv6 support.

The sockaddr structures and API for IPv6 are fairly difficult to use, particularly if it's not known in advance what sort of address they will contain. We added a set of macros and utility functions in sockaddr-util.c and sockaddr-util.h. Using these macros throughout Amanda removed a significant amount of code that was conditionalized on both compile-time support and runtime address family, and centralized that logic in one easily-maintained place.

On our build systems, we had to deal with different levels of support in the compile environment and the kernel. This is fine: most Amanda users install binary packages that are produced on roughly the same OS distribution and version as was used for the build, so the kernel support is generally the same. However, a third variable has tripped up lots of Amanda users: system configuration. In particular, several newer Linux distributions have shipped with localhost resolving to ::1 vi /etc/hosts, but without enough interface configuration to actually utilize a socket bound to that address. Amanda uses localhost sockets for inter-process communication, so this misconfiguration causes backup operations to fail. The solution is to either finish configuring IPv6 on the host, remove the reference to ::1 in /etc/hosts, or build Amanda with --without-ipv6.

I have not yet heard of an Amanda installation where IPv6 communication is in use. But I have heard from countless IPv4 users whose Amanda installations have failed due to bad IPv6 support. At the moment, then, I feel that adding IPv6 support to Amanda has been a net negative for the project. Although there is doubtless room for improvement, I will not entertain patches for better IPv6 support, for fear they will introduce new bugs for our exclusively IPv4 userbase.

Of course, all of this may change as dual-stack networks grow more prevalent and are replaced by pure IPv6 networks!

SSH With Snow Leopard

nospam@example.com (Dustin J. Mitchell) — Sat, 10 Jul 2010 14:27:00 -0500

I just upgraded my Macbook to Snow Leopard, and the upgrade has changed the way SSH authentication works. I have set up a system I like quite a bit, now, and thought I would share. My usage pattern is that I do most of my work via GNU screen running on my login server, euclid. So I want a simple procedure that will connect me to that screen session, with a proper SSH agent set up.

Snow Leopard automatically starts an ssh-agent process at login. This is great, but does not interoperate with SSHKeychain. Dropping SSHKeychain is OK with me - I don't use SSH tunnels, so it really only acted as a GUI for passphrase entry. It also ate CPU occasionally, which of course causes the macbook to become more painfully hot than usual.

So I have three problems to solve: 1. automatically add my key to ssh-agent; 2. automatically expire the key at appropriate times (at my paranoia level, that's at system sleep); and 3. make multiple agent instances usable from the same shell session on the server.

Adding the Key at Connection Time

Adding the key is relatively straightforward. I wrote a short script that Terminal runs when I hit ⌘-N or ⌘-T:

#! /bin/bash

# does ssh-agent not have a key?
if ! ssh-add -l; then
    ssh-add ~/.ssh/dustin || exit 1
fi

exec ssh -x -t euclid.r.igoro.us bin/startscreen

This will prompt me for the passphrase when there is not already a key active, but proceed directly to the ssh invocation if the key situation is OK. The -x option to ssh is there to turn off X11 forwarding; without this option, SSH will helpfully start the X11 app. I think this is a great feature, but I don't use X11 apps very often, so I've disabled it.

Expiring the Key Automatically

Mac OS has a nicely designed system in place to allow applications to get notified when the system is changing power states. However, I wasn't interested in writing a full OS X app for this particular project. Enter sleepwatcher. This is a beautifully simple program that just executes scripts on particular power events. I set this up to run via launchd, as directed in the README, and to run ~/.ssh/sleep on sleep. That script contains:

#! /bin/bash

# first, don't inherit a socket (sleepwatcher doesn't get the user's env)
SSH_AUTH_SOCK=

# find some sockets
echo ".ssh/sleep:" `id`
for sock in /tmp/launch-*/Listeners; do
    if [ -w $sock ]; then
        echo "Trying to remove .ssh/dustin from socket $sock"
        SSH_AUTH_SOCK=$sock /opt/local/bin/ssh-add -d ~/.ssh/dustin
    else
        echo "Skipping unwritable socket $sock"
    fi  
done

The for loop is required because a script run from sleepwatcher doesn't inherit the SSH_AUTH_SOCK variable that points to the running SSH agent. The loop simply searches for a writable SSH socket of the pattern used by the system's agent.

SSH Agent and Screen

If you naïvely set up an SSH agent, connect to a remote system, and start screen there, things will work great - until you disconnect from the screen session. When you connect to the remote system, SSH forwards the agent connection for you, and sets SSH_AUTH_SOCK on the remote system to point to this forwarded socket. Screen passes this variable along blindly, so it appears in all of the shells opened in screen windows, and things work as you'd expect. When that SSH connection is removed, and a new one established, the forwarded agent appears at a new socket. But those shells running in screen windows are still pointing to the old name, and will no longer be able to connect.

The fix is to create a socket with a well-known name that will not change from connection to connection. The following script takes care of it. WARNING: this script is vulnerable to /tmp race conditions. I am the only user on my servers, so this doesn't bother me, but fixing it should be relatively straightforward.

#! /bin/bash
# hard-link the SSH socket to one with a fixed name on the local
# machine, and set SSH_AUTH_SOCK to point to that fixed name.  Later
# invocations of this script will change the link, but the name will
# remain valid, allowing existing shells to continue to function.
setup_fixed_socket() {
  local old_socket="$SSH_AUTH_SOCK"
  local socket_dir="/tmp/$(uname -n)-$(id -u)"
  local socket_file=$socket_dir/agent

  # set up the directory and permissions
  [ -e $socket_dir ] || mkdir -p $socket_dir
  chmod 700 $socket_dir

  # remove an existing link
  [ -e $socket_file ] && rm $socket_file

  # hard-link in the new one
  ln $old_socket $socket_file

  # return the new socket
  echo $socket_file
}

# this variable will be exported to every shell opened by this
# invocation of screen -- even subsequent connections to it.  This
# variable may live for days or weeks.
export SSH_AUTH_SOCK=$(setup_fixed_socket)

# finally, fire up screen.  Try reattaching to a running
# session; otherwise start up a new one
screen -R -DD ${@} || screen

What's New in Amanda: The End of Fragmentation

nospam@example.com (Dustin J. Mitchell) — Thu, 08 Jul 2010 18:07:00 -0500

Most of my posts in this series have been about features that are available in a released version of Amanda. This time, I want to share a project I'm working on right now - one that will be available in Amanda-3.2. I'm reworking the way Amanda writes its data to tape (or any other kind of storage) to make it more efficient, more reliable, and simpler to configure.

Historically, Amanda's conservative approach to finicky tape hardware has meant that it wasted some space at the end of each tape. With the changes I'm working on, Amanda will no longer waste this space, and can also avoid some needless copying of data in most cases, with a minimum of additional risk.

Amanda's Storage Format

Before examining the new functionality, let's look at Amanda's storage format. Amanda treats all storage devices like tapes¹ - a set of sequentially numbered data files of arbitrary size, each composed of a sequence of fixed-size blocks. Each file begins with a one-block header that identifies the dump and gives information about its contents. The header is followed by blocks of raw data.

Amanda supports writing a dump across multiple tapes - spanning.² The technique is this: a dump is split into a sequence of parts, and each part is written a a single file on a volume. During recovery, Amanda reads the parts in sequence, and concatenates their data to reproduce the original dumpfile. Usually all parts are the same size, and this size is generally 1-5% of the tape capacity.

Better Safe than Sorry - At a Cost

Amanda was originally designed around tape drives - in fact, if you look at the history of Linux kernel support for tape drives, it is closely intertwined with Amanda development. Tape drives are finicky beasts, and in many cases cannot distinguish the end of the tape (called EOM) from any other fatal error. Worse, tape drives employ large caches to ensure they can write continuously, and when an error occurs all of the data in that cache is lost, and there is no way for Amanda to determine how much actually made it onto the tape. Beginning a new on-tape file (writing a filemark) flushes the cache and signals any errors immediately.

Since time immemorial, then, Amanda has treated any error from the tape drive as EOM, and assumed that all data written since the last filemark is potentially corrupt. That means that the part in progress when the error occurred is logged as PARTIAL, and Amanda will start at the beginning of that part on the next tape.

The PARTIAL part is recorded in the catalog, but will not be used for recovery, so it is effectively wasted space. A little arithmetic will tell you that, on average, each tape will waste half of the part size. This is at least excusable with real tape drives; with vtapes (disk), this wasted space is completely unnecessary. Worse, vtapes are most flexible when they are kept small and dumps are spanned over many vtapes per night; but the wasted space increases linearly with the number of vtapes used.

In order to rewind a part and write it again on a new tape, Amanda also needs to keep its data somewhere, called the part cache. When the dump is on the holding disk, the holding disk acts as a part cache. Otherwise, Amanda can cache parts in memory or on disk. Caching in memory is faster, but requires a lot of RAM for a reasonable part size. Caching on disk allows larger parts, but is considerably slower.

Logical EOM

More recent tape drives (those made in the last decade or so) have a feature called "early warning". With this feature enabled, the drive alerts the host system when it is "near" EOM, and flushes the cache to tape. Exactly what "near" means is not specified in the SCSI standard, but in general there's room to flush the cache and write a filemark, at least. This is sometimes called a logical EOM - LEOM.

Amanda can take advantage of this functionality to cleanly finish a part before running headlong into a physical EOM. This eliminates the wasted space for a PARTIAL part, and also eliminates the need to cache parts, since rewinding is not required. In one small change (well, OK, it's about 4,300 lines), Amanda gets faster and uses storage space more efficiently. What's not to love?

Better yet, all of the non-tape devices (vtapes, S3 devices, DVD-ROMs, etc.) can easily emulate LEOM, so backups to these devices will automatically benefit from this improvement.

In the Code

Three important patches toward this functionality were just committed. What remains is to set up real LEOM support for the VFS device (vtapes) and for the tape device.

The VFS device can, of course, trivially emulate LEOM when it is enforcing the MAX_VOLUME_USAGE property - the vtape length. However, predicting when a filesystem will run out of space is much more difficult. We are still discussing options, and I would love to hear suggestions here or on the mailing list.

As for the tape device, it will assume that LEOM is not supported unless the user configures it explicitly (with the "LEOM" device property) or we can determine support for LEOM from the operating system at runtime. Unfortunately, this is one of those areas so technical that only a half-dozen people know how it works, so it may take me some time to track down this information for non-Linux operating systems. Again, advice and assistance is welcome!

¹This is an ages-old design decision, but one that artificially constrains Amanda's flexibility, especially with vtapes.
²In fact, Amanda has supported spanning for something like 7 years now. Yet I occasionally see users in #amanda complaining about this serious limitation and wondering when we're going to do anything about it. Will 2003 be soon enough?

What's New in Amanda: Hackability

nospam@example.com (Dustin J. Mitchell) — Thu, 01 Jul 2010 22:29:00 -0500

It's been a while since I've posted about recent development in Amanda, but it's not for lack of interesting topics!

Today I want to talk a little bit about Amanda's development. Historically, Amanda has always had a small, core group of developers who do the lion's share of the development work. There are probably lots of reasons for this, not least of which is that a backup application isn't the sexiest project on which to spend your spare time. But I think there's a deeper reason, and it has to do with hackability. Amanda was originally written in C, which means any changes require the full set of developer tools. For example, just to fix a typo in an error message, you would need to find the source, configure and compile it, find and fix the error message, and recompile to test. Fixing something more substantial in the highly interdependent Amanda codebase also requires a deep understanding of many parts of Amanda - from the obscure configuration interface to the oddly interlinked disklist structure. This level of programming skill is not common among Amanda's user base (systems administrators), and I can count the people who understand the disklist structure on one hand.

The result has been a paltry flow of patches from anyone but the core hackers. Furthermore, no entry path has been available by which newcomers could work their way up to being core developers. While I don't want to disparage the work of any of the great programmers who have written Amanda over the years, it's a shame that there have been so few at any time, and I worry about what would happen if the number were to reach zero. So what to do?

$hackability++

We've done a few things to try to make Amanda more hackable. Probably the biggest change is to rewrite parts of Amanda in Perl. I'm asked "why" quite often, and while we had a lot of reasons, two relate directly to hackability. First, more sysadmins know Perl than C, because Perl is quite often used to build the "glue" that links systems together. Interestingly, based on many conversations, it seems that Python may also have been a good choice, as I suspected when I first proposed the rewrite. But it's too late now!

Second, and more importantly, Perl code can be hacked in place. If amvault isn't acting the way you want it to, just open up /usr/sbin/amvault and tweak away. No need to download the source, no need to compile, no segmentation faults, just hacking. When you're done, run a quick diff and send the results to amanda-hackers.

Even users who do not know Perl can take advantage of this in-situ hackability. Within Amanda's C code, if I want a user to try a patch, that user must figure out how to download Amanda's source, apply the patch, configure, compile, and install. None of those steps are trivial. With Perl code, I can often provide a patch that is simple enough to be applied directly to the installed executables by hand, or with a simple application of patch. Everyone stays focused on the bug under investigation, and the user's backups are up and running that much more quickly.

New APIs

As I mentioned before, historically Amanda's code has been highly interdependent. Details of the implementation of the holding disk were spread over most of the files in the server implementation. The dumplevel -987 has a special meaning that is documented nowhere, but referenced in several source files. All of this makes new development difficult, because it's impossible to "slice off" and study a portion of Amanda in isolation.

The solution here is to create abstract interfaces, where new functionality can be "plugged in" and Amanda can use it without changes. The Amanda developers have abused the term "API" for these interfaces, and we now have quite a few:

Application API - an abstraction of backup clients, e.g., ampgsql for Postgres;
Device API - an abstraction of backend storage devices, such as tape, disk, cloud, or DVD-RW;
Changer API - an abstraction of tape changers and other mechanisms for selecting from a set of volumes; and
Script API - a means of invoking scripts before or after certain events during a backup.

This strategy has already paid off: we have seen several new scripts and applications contributed, and the DVD-RW device arrived out of the blue as a contribution from someone who found it useful.

Other Changes

In the interest of greater accessibility to new hackers, we have also put Amanda on github and created a set of good "beginner" projects. Zmanda has even offered to pay people to hack on Amanda, as a way of easing the cost of entry. I also try to point out interesting projects on the Amanda mailing list, particularly projects that Jean-Louis and I probably will not find time to work on.

The idea here is to encourage new hackers to pick up a well-scoped project to become familiar with Amanda. The hackers can then move on to more sophisticated projects that meet their particular backup needs or address a particular interest.

Will You Join Me?

So Amanda is ready for you. When can you start?

IPv6 Configuration

nospam@example.com (Dustin J. Mitchell) — Sun, 27 Jun 2010 16:27:12 -0500

I've been meaning to get IPv6 set up on my local network for some time. My only practical reason is that Amanda supports IPv6 and I should test that support. It was also a good chance to re-immerse myself in network configuration, and Hurricane Electric has a neat certification process to add some motivation. I began by getting local IPv6 connectivity set up over the HE tunnel, using my Gentoo systems. This was fairly straightforward, as the Gentoo net scripts natively support IPv6. The firewall system I use (Shorewall) does not support IPv6 directly. Instead, there's a parallel shorewall6 package to install. Aside from the annoyance of setting up two separate firewalls, this did not cause appreciable difficulties. With all of this in place, I was at the "Explorer" level.

The next task was to set up a working IPv6 desktop. My home network uses 802.1q VLAN tagging to layer both an external, publicly routable IPv4 network (99.89.149.16/29 on VLAN 20) and an internal, NAT'd IPv4 network (172.16.1/24 on VLAN 10). I wanted to make VLAN 10 a dual-stack network, rather than invent a new VLAN for my IPv6 network. Initially, I didn't realize that HE uses, in my case, 2001:470:1f10:826::0/64 just for the tunnel (yes, two addresses out of 2⁶⁴ used -- maybe we'll need IPv8 sooner than we think!). I assumed that the /64 I was allocated was to be used for all of my nodes, and tried to subnet it locally, using 2001:470:1f10:826::0/112 for the tunnel and 2001:470:1f10::1/112 for the internal network. This worked with manual configuration, but radvd seemed to always want to advertise a /64. A little reading about the RA protocol showed this to be correct: RA provides the high 64 bits (the network portion), and the clients provide the low 64 bits using EUI-64. I was stymied until I looked at the tunnel details again and noticed that the "Routed IPv6 Prefixes" section listed a different prefix (2001:470:1f11:826/64).

With this in place, the subnet and firewall setup was a breeze. Using a manual configuration on my MacBook, I was able to communicate via IPv6. However, the stateless autoconfiguration didn't work. I briefly tried DHCPv6, but Macs do not support it. The RA client correctly combines the network and EUI-64 components to create a full address, and it correctly copies the link-local address of the router, but it does not set up a default route using that router, making the whole thing fairly useless. A trip to #ipv6 confirmed that Macs are, indeed, broken this way, so I stopped worrying about it.

The remainder of the certification process involved getting Apache, Postfix, and Bind speaking IPv6, none of which was very difficult. I did discover that BIND's $ORIGIN didn't work correctly. A zonefile with

$ORIGIN 6.2.8.0.1.1.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR knuth.r.igoro.us.

didn't work, but spelling out the entire reversed address did. I'm sure this was due to a typo, but several checks didn't reveal anything.

However, I'm now stuck at the Guru level until GoDaddy starts supporting IPv6 glue for the .us TLD. I feel cheated, somehow!

LCD Display and TMP102 sensor

nospam@example.com (Dustin J. Mitchell) — Sat, 17 Apr 2010 21:22:01 -0500

It is incredibly easy to throw things together with an Arduino. It's common to see criticism of the device when it's used in in projects that don't require even a fraction of its power, and that might be justified. As a flexible platform for test-driving complex modules, though, the Arduino hits the mark perfectly on flexibility and usability.

I don't have a particular project in mind for my Arduino, but since I don't have a multimeter or an oscilloscope, I want to use it as a test harness for various basic components as I experiment with them. To this end, I bought a cute 16x2 character amber LCD display. With thoughts of building a fermentation-temperature monitor and learning about the I²C bus, I also bought a relatively cheap TMP102 and breakout board. With a little bit of reading, I was able to stitch them together quickly and easily.

LCD Display

The display I purchased is command-driven via a 4-bit parallel port. It's a ST7066, compatible with HD44780. There are two datasheets available -- one for the ST7066, and one for the assembled board. The latter adequately described the pinout through unattributed copying from the former, but was otherwise useless. "Qiu", "Chen", and "Ye", all three of whom helpfully signed the cover page, should be ashamed. Anyway, here's a brief description of the pins and some more details I culled from the ST7066 datasheet:

V_ss (ground) and V_dd (positive supply): these are rated up to 7V, so running from the Arduino's 5V supply is great.
V₀ sets the LCD contrast. Short on jumpers, I initially assumed I could leave this unconnected to get "reasonable" contrast. Not so! I tried setting up a few voltage dividers to get the right contrast, without any luck. Plan to add a 10k pot between ground and the positive supply, and tie the wiper to this pin.
RS (register select) selects whether an operation is to configuration registers (0) or RAM (1)
R/W (read/write) selects whether an operation reads or writes to the device
DB0-DB7 is the data bus
E (enable), strobed to move a byte across the data bus
LED+/LED- supply power to the LED backlight, and can be wired directly to the 5V supply.

The Arduino has a lot of digital pins, but all the same it's handy to save a few pins. The display supports reading character and font data, but that's not much use: the Arduino can remember whatever it needs to. So we won't need the R/W pin, and can tie it to ground (read) instead.

The display can operate in two data lengths (controlled by the DL configuration bit). In the default mode, only 4 bits (DB4-DB7) on the parallel bus are used. The display starts in 8-bit mode, but fortunately the DL bit comes in at pin DB4, so it's relatively easy to reset the data length with only 4 pins connected. Note that there's an odd sequencing required here, where you need to set 8-bit mode three times before setting 4-bit mode. Don't worry, though: the Arduino LiquidCrystal library takes care of that for you.

So aside from pins tied to V+ or GND, we only need 6 digital I/O pins. It's best to avoid the pins that have other functions on the Arduino: 0 and 1 are RS-232 I/O, and 13 is the onboard LED. I used pins 2-5 for the data bus, pin 12 for E, and pin 13 for RS, since that was at the top of the LiquidCrystal example. V_ss, LED-, and R/W go to GND; V_dd and LED+ go to the 5V supply; and V₀ is wired as described above.

From there on out, it's a straightforward application of the LiquidCrystal library:

#include <LiquidCrystal.h>

LiquidCrystal lcd(12, 11, 5, 4, 3, 2); 

void setup() {    
    lcd.begin(16, 2); 
}

void loop() {
    lcd.setCursor(0, 0);
    lcd.print("Hello, World");
}

Note that the display itself is capable of a number of interesting things not supported by the library. It has an 80-byte character memory, and can scroll that through the display without rewriting the entire screen every time. It can also accept 8 custom 5x11 glyphs, making rudimentary animation possible.

TMP102

The TMP102 is ridiculously tiny - smaller than the SMD resistors that Sparkfun has placed around it on the breakout board. It senses temperature internally, which means it's basically monitoring the temperature of its leads.

The device speaks I²C (also known as the two-wire interface or SMBus), which makes it pretty easy to connect to the Arduino. The MCU speaks I²C via hardware, so the corresponding Arduino library doesn't even need to bit-bang the data. On the Duemilanove, it uses analog-in pin 4 for SDA and analog-in pin 5 for SCL. You'll also need to hook up V+ to the 3.3V pin on the Arduino (the device is only specified for 3.6V) and GND to the TMP102's GND. Finally, the TMP102's ADD0 pin can cleverly select one of four addresses for the device by tying it to one of these four pins. I tied it to GND, giving address 0b1001000. ALARM is an output pin, so you can leave it unconnected.

The TMP102 is a very nice low-power device that's designed to handle several common tasks without any interaction from the MCU - it can trigger an interrupt when the temperature strays from a configured boundaries, poll temperature on a relatively slow schedule, or even poll on command. I didn't use any of that, relying on the device to simply sample the temperature on its own schedule.

The TMP102 has four registers, but we'll only use one -- temperature (0b01). Like many devices, this one multiplexes addresses and data over the same bus. To read a register, you first select the register by writing it to the device, then read the 16-bit value, again with the MSB first. Once a register is selected, it can be read multiple times.

The device's temperature register contains a two's-complement 12-bit value, left-aligned, where 0x7FF is 128°C; equivalently, a change of one unit in the 12-bit value is equivalent to 0.0625°C.

Putting it Together

The obvious combination of these tools is to display the ambient temperature on the LCD screen. Here's the program to do so:

#include <Wire.h>
#include <LiquidCrystal.h>

LiquidCrystal lcd(12, 11, 5, 4, 3, 2); 

#define TEMP_REG 0b00000000
int tmp102 = 0b1001000;  // with ADD0 tied to ground

void setPR(int reg) {
  Wire.beginTransmission(tmp102);
  Wire.send(reg);
  Wire.endTransmission();
}

int getReg() {
  unsigned char lo, hi;

  Wire.requestFrom(tmp102, 2);
  hi = Wire.receive();
  lo = Wire.receive();
  return (hi << 8) + lo;
}

void setup()   {    
  lcd.begin(16, 2);
  Wire.begin();
  setPR(TEMP_REG);
}

void show_temp() {
  int temp_reg = getReg();
  lcd.setCursor(0, 0);
  show_bin(temp_reg);

  temp_reg >>= 4;

  float temp_C = temp_reg * 0.0625;
  float temp_F = (temp_C * 9 / 5) + 32;

  lcd.setCursor(0, 0);
  lcd.print(temp_C);
  lcd.print("\xdf""C  ");
  lcd.setCursor(0, 1);
  lcd.print(temp_F);
  lcd.print("\xdf""F  ");
}

void loop() {
  show_temp();
}

The setup function sets up the LCD, then uses setPR to point the TMP102 at the temperature register. Subsequent reads will then return the 12-bit encoded temperature. The loop function reads the temperature register, decodes it, and displays the result in both Celsius and Fahrenheit.

Note that this is horrendously inefficient: the TMP102 only measures temperature every 26ms or so, during which loop will probably run a half-dozen times, feeding the same time strings to the LCD each run. It would be much better to put the TMP102 in one-shot mode, and measure the temperature at a much lower frequency - say once a second - with a correspondingly low update frequency for the display. I'll leave that as an exercise for the reader.

Modern Multiprocessing

nospam@example.com (Dustin J. Mitchell) — Mon, 12 Apr 2010 15:24:00 -0500

I've been thinking a lot lately about the way we accomplish multiprocessing. We've seen a significant change in the operation of Moore's law for CPU speeds: today's CPUs are about the same speed as those of a few years ago, but they have more cores, and more virtual processors on those cores. This is great for heavily-loaded servers, which have plenty of distinct tasks to place on those cores and VCPUs, but not so useful for users working with single-threaded applications.

Why are most applications still single-threaded? There are lots of good reasons. Threaded code is harder to write, and not just because it requires careful analysis and use of synchronization primitives: many common tasks are difficult to meaningfully parallelize without careful control over inter-thread communication, and in a portable application you don't have that kind of control. Threaded code generally performs badly on single-CPU systems, which are still common. Some popular languages still make threading difficult, at least in a portable fashion. And threads are still relatively heavyweight entities in most operating systems: you don't spawn ten threads to mergesort a 100-item array.

Some of these problems will go away with a little more time, but some will get worse. NUMA architectures can make sharing data between threads slow. Hyperthreading and its interaction with processor caches adds yet another level of unpredictability.

We know how to build massively parallel systems that run massively parallel algorithms. What is still unknown is how to build portable, simple software that can run efficiently across a vareity of architectures. This is a problem of practice, not theory, and there's lots of interesting work going on in this area.

Of course, there are languages designed explicitly to support communication, such as Limbo or Erlang, Haskell, and Clojure. For the most part, these languages are structured as communicating sequential processes, which is to say that they represent multiprocessing as a set of sequential threads that pass information to one another. Problems of thread safety are subsumed by the languages, but mapping the parallelism to available resources is generally left to the programmer or administrator.

One interesting project is Apple's Grand Central Dispatch. It defines a simple but highly expressive closure syntax (a block) and a mechanism to dynamically schedule execution of such closures (queues). Critically, the GCD library takes care of scaling the parallelism of the queue processing appropriately to the underlying hardware. On a single-threaded CPU, this amounts to cooperative multitasking, but on parallel hardware the operating system can dynamically allocate virtual CPUs to applications needing more parallelism.

This topic seems to come up often in my various pursuits, so I will return to it again.

Want to work on Amanda?

nospam@example.com (Dustin J. Mitchell) — Mon, 12 Apr 2010 14:04:34 -0500

I've not made any secret of the fact that I want more people hacking on Amanda. This is both for selfish reasons -- many hands make light work -- and for altruistic reasons -- a broader community of developers can provide better governance for the project and long-term continuity. With a few noticable exceptions, I haven't had a lot of satisfaction.

I think part of the reason is that Amanda has a steep learning curve, even within the new Perl code. The time to climb that curve is a big investment, and folks with only a small itch to scratch can't afford it.

In an effort to sweeten the pot, we (Zmanda) are offering to pay for flexible work on Amanda. Part-time or full-time, on your own schedule. Your choice of projects. Support and gratitude from the other hackers. And the option to become a full Zmanda employee if that's your bent.

Here are some possible projects, to pique your interest:

MySQL application (to round out the set with ampgsql)
Cyrus Imapd application (gnutar doesn't deal well with the application's tiny files and hard links)
OpenSSL for network transport, using certificates and keys for authentication
Database-backed backup catalog
Amvault upgrade
Handle Logical EOM (LEOM) on all devices that support it, drastically reducing the number of parts Amanda writes
Support for more cloud backends than just S3
Parallel writes to multiple devices

If you're interested, contact me (dustin@zmanda.com) and we'll work something out!