Code V.igoro.us - Code

Nagios NSCA from Python

nospam@example.com (Dustin J. Mitchell) — Fri, 20 May 2011 16:55:11 -0500

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

Amanda's Transfer Mechanisms

nospam@example.com (Dustin J. Mitchell) — Sat, 22 Jan 2011 14:47:13 -0600

There's been a bit of confusion on the mailing list and IRC about how Amanda assembles transfers out of transfer elements, and how transfer mechanisms influence that.

In the final form of a transfer, any two adjacent elements must have the same mechanism. For example is, an upstream element speaking XFER_MECH_PUSH_BUFFER cannot talk to a downstream element using XFER_MECH_READ_FD (nor, more confusingly, XFER_MECH_PULL_BUFFER). So each mechanism is an isolated definition of "here's how upstream and downstream should talk". They come in pairs because generally anything upstream can do to downstream (e.g., upstream can write to downstream's fd) can occur in reverse (e.g., downstream can read from upstream's fd).

What makes this confusing is that if you specify a set of elements which can't talk directly to one another, then xfer.c will add "glue" elements between the specified elements. To make that concrete, imagine you specify a transfer as

source-holding --> filter-xor --> dest-fd

(if you like practical examples, then imagine filter-xor is a buffer-based decompression filter, and you're pulling data from holding disk, decompressing, and sending to a pipe -- something amfetchdump would do). Here are the mechanisms supported by each element:

source-holding:
 XFER_MECH_PULL_BUFFER

filter-xor
 XFER_MECH_PULL_BUFFER (input) and
 XFER_MECH_PULL_BUFFER (output)
or
 XFER_MECH_PUSH_BUFFER (input) and
 XFER_MECH_PUSH_BUFFER (output)

dest-fd
 XFER_MECH_WRITEFD (input)

In putting these together, source-holding and filter-xor can use the same mechanism (PULL_BUFFER). This leaves filter-xor using PULL_BUFFER for output, but dest-fd does not support this. So xfer.c adds a glue element that can speak PULL_BUFFER on input and WRITEFD on output. This element basically loops in a thread, calling upstream->pull_buffer and write(downstream->input_fd, buffer). So the final xfer looks like

 source-holding --(PULL_BUFFER)--> filter-xor --(PULL_BUFFER)--> glue --(WRITEFD)--> dest-fd

Hopefully that helps to explain how the glue works.

Note that one of the cool things about this arrangement is that in most cases the complexity is in the glue, not the elements. In fact, in this case the glue provides the only thread that's required to run this transfer, so the other three element implementations don't need to manage threads at all.

virtualenv for Perl

nospam@example.com (Dustin J. Mitchell) — Thu, 11 Nov 2010 10:48:43 -0600

I absolutely love virtualenv for Python development. It allows me to develop Buildbot against several versions of Python and several versions of its dependencies, without modifying my system's Python installation at all!

Now, I need to do the same thing in Perl. So I thought I'd compare the two side-by-side.

virtualenv

# install virtualenv locally
wget http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py
# set up a sandbox
python virtualenv.py sandbox
# activate it
source sandbox/bin/activate
# start installing stuff
easy_install buildbot

local::lib

There is no local::lib gentoo ebuild! I'm sure there's a good reason, but that's odd all the same!

# install local::lib locally
wget http://search.cpan.org/CPAN/authors/id/G/GE/GETTY/local-lib-1.006007.tar.gz
tar -zxf local-lib-1.006007.tar.gz
cd local-lib-1.006007
perl Makefile.PL --bootstrap # (accept lots of defaults)
make test && make install
# activate it (permanently)
echo 'eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)' >>~/.bashrc
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
# start installing stuff
perl -MCPAN -e install Config::General # (I have to accept the same defaults again??)

I think virtualenv is the clear winner here!

IPv6 and Amanda

nospam@example.com (Dustin J. Mitchell) — Fri, 16 Jul 2010 22:04:00 -0500

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case. First, a bit of background on how Amanda's networking code works, and what had to change to support IPv6. Amanda supports security mechanisms called BSD (the oldest), BSDUDP, and BSDTCP. These all authenticate (if you can call it that) using the same sorts of checks that rsh uses. The incoming connection is accepted if:

it is from a "reserved" port (less than 1024);
the address of the initiator has complementary forward and reverse DNS records in place; and
the initiator's hostname is in .amandahosts

During a backup operation, the Amanda server contacts each client host. When using the BSD authentications, this triggers amandad, which checks the above restrictions before beginning communication with the server. This initial connection is packet-based, and can be carried out over UDP (for BSD and BSDUDP) or TCP (BSDTCP). When a dump begins, several "streams" are opened to transmit the data, index, and metadata. For BSD and BSDUDP, each stream is implemented as a distinct TCP connection, where the client sends a port number to the server and the server connects to that port. BSDTCP multiplexes all streams over a single TCP connection using a basic type/length packet encapsulation.

The first challenge in adding IPv6 support was to deal properly with IPv6 addresses when querying the DNS. That meant switching to getaddrinfo and getnameinfo, as suggested by Jun-ichiro itojun Itoh. These functions bring their own compatibility problems, but Amanda uses gnulib, which provides compatibile implementations on systems where they are not available, minimizing the difficulty.

We had a lot of trouble from systems such as RHEL3 possessing IPv6 support in the compiler environment but not in the kernel. On such systems, code using constants like AF_INET6 or AI_V4MAPPED would compile without problems, but fail at runtime. We added a WORKING_IPV6 preprocessor conditional, without which all references to IPv6-related symbols were removed. At configure time, Amanda tries to create an IPv6 socket, and sets this conditional to true if it succeeds. The --without-ipv6 configure option forcibly disables IPv6 support.

The sockaddr structures and API for IPv6 are fairly difficult to use, particularly if it's not known in advance what sort of address they will contain. We added a set of macros and utility functions in sockaddr-util.c and sockaddr-util.h. Using these macros throughout Amanda removed a significant amount of code that was conditionalized on both compile-time support and runtime address family, and centralized that logic in one easily-maintained place.

On our build systems, we had to deal with different levels of support in the compile environment and the kernel. This is fine: most Amanda users install binary packages that are produced on roughly the same OS distribution and version as was used for the build, so the kernel support is generally the same. However, a third variable has tripped up lots of Amanda users: system configuration. In particular, several newer Linux distributions have shipped with localhost resolving to ::1 vi /etc/hosts, but without enough interface configuration to actually utilize a socket bound to that address. Amanda uses localhost sockets for inter-process communication, so this misconfiguration causes backup operations to fail. The solution is to either finish configuring IPv6 on the host, remove the reference to ::1 in /etc/hosts, or build Amanda with --without-ipv6.

I have not yet heard of an Amanda installation where IPv6 communication is in use. But I have heard from countless IPv4 users whose Amanda installations have failed due to bad IPv6 support. At the moment, then, I feel that adding IPv6 support to Amanda has been a net negative for the project. Although there is doubtless room for improvement, I will not entertain patches for better IPv6 support, for fear they will introduce new bugs for our exclusively IPv4 userbase.

Of course, all of this may change as dual-stack networks grow more prevalent and are replaced by pure IPv6 networks!

What's New in Amanda: The End of Fragmentation

nospam@example.com (Dustin J. Mitchell) — Thu, 08 Jul 2010 18:07:00 -0500

Most of my posts in this series have been about features that are available in a released version of Amanda. This time, I want to share a project I'm working on right now - one that will be available in Amanda-3.2. I'm reworking the way Amanda writes its data to tape (or any other kind of storage) to make it more efficient, more reliable, and simpler to configure.

Historically, Amanda's conservative approach to finicky tape hardware has meant that it wasted some space at the end of each tape. With the changes I'm working on, Amanda will no longer waste this space, and can also avoid some needless copying of data in most cases, with a minimum of additional risk.

Amanda's Storage Format

Before examining the new functionality, let's look at Amanda's storage format. Amanda treats all storage devices like tapes¹ - a set of sequentially numbered data files of arbitrary size, each composed of a sequence of fixed-size blocks. Each file begins with a one-block header that identifies the dump and gives information about its contents. The header is followed by blocks of raw data.

Amanda supports writing a dump across multiple tapes - spanning.² The technique is this: a dump is split into a sequence of parts, and each part is written a a single file on a volume. During recovery, Amanda reads the parts in sequence, and concatenates their data to reproduce the original dumpfile. Usually all parts are the same size, and this size is generally 1-5% of the tape capacity.

Better Safe than Sorry - At a Cost

Amanda was originally designed around tape drives - in fact, if you look at the history of Linux kernel support for tape drives, it is closely intertwined with Amanda development. Tape drives are finicky beasts, and in many cases cannot distinguish the end of the tape (called EOM) from any other fatal error. Worse, tape drives employ large caches to ensure they can write continuously, and when an error occurs all of the data in that cache is lost, and there is no way for Amanda to determine how much actually made it onto the tape. Beginning a new on-tape file (writing a filemark) flushes the cache and signals any errors immediately.

Since time immemorial, then, Amanda has treated any error from the tape drive as EOM, and assumed that all data written since the last filemark is potentially corrupt. That means that the part in progress when the error occurred is logged as PARTIAL, and Amanda will start at the beginning of that part on the next tape.

The PARTIAL part is recorded in the catalog, but will not be used for recovery, so it is effectively wasted space. A little arithmetic will tell you that, on average, each tape will waste half of the part size. This is at least excusable with real tape drives; with vtapes (disk), this wasted space is completely unnecessary. Worse, vtapes are most flexible when they are kept small and dumps are spanned over many vtapes per night; but the wasted space increases linearly with the number of vtapes used.

In order to rewind a part and write it again on a new tape, Amanda also needs to keep its data somewhere, called the part cache. When the dump is on the holding disk, the holding disk acts as a part cache. Otherwise, Amanda can cache parts in memory or on disk. Caching in memory is faster, but requires a lot of RAM for a reasonable part size. Caching on disk allows larger parts, but is considerably slower.

Logical EOM

More recent tape drives (those made in the last decade or so) have a feature called "early warning". With this feature enabled, the drive alerts the host system when it is "near" EOM, and flushes the cache to tape. Exactly what "near" means is not specified in the SCSI standard, but in general there's room to flush the cache and write a filemark, at least. This is sometimes called a logical EOM - LEOM.

Amanda can take advantage of this functionality to cleanly finish a part before running headlong into a physical EOM. This eliminates the wasted space for a PARTIAL part, and also eliminates the need to cache parts, since rewinding is not required. In one small change (well, OK, it's about 4,300 lines), Amanda gets faster and uses storage space more efficiently. What's not to love?

Better yet, all of the non-tape devices (vtapes, S3 devices, DVD-ROMs, etc.) can easily emulate LEOM, so backups to these devices will automatically benefit from this improvement.

In the Code

Three important patches toward this functionality were just committed. What remains is to set up real LEOM support for the VFS device (vtapes) and for the tape device.

The VFS device can, of course, trivially emulate LEOM when it is enforcing the MAX_VOLUME_USAGE property - the vtape length. However, predicting when a filesystem will run out of space is much more difficult. We are still discussing options, and I would love to hear suggestions here or on the mailing list.

As for the tape device, it will assume that LEOM is not supported unless the user configures it explicitly (with the "LEOM" device property) or we can determine support for LEOM from the operating system at runtime. Unfortunately, this is one of those areas so technical that only a half-dozen people know how it works, so it may take me some time to track down this information for non-Linux operating systems. Again, advice and assistance is welcome!

¹This is an ages-old design decision, but one that artificially constrains Amanda's flexibility, especially with vtapes.
²In fact, Amanda has supported spanning for something like 7 years now. Yet I occasionally see users in #amanda complaining about this serious limitation and wondering when we're going to do anything about it. Will 2003 be soon enough?

What's New in Amanda: Hackability

nospam@example.com (Dustin J. Mitchell) — Thu, 01 Jul 2010 22:29:00 -0500

It's been a while since I've posted about recent development in Amanda, but it's not for lack of interesting topics!

Today I want to talk a little bit about Amanda's development. Historically, Amanda has always had a small, core group of developers who do the lion's share of the development work. There are probably lots of reasons for this, not least of which is that a backup application isn't the sexiest project on which to spend your spare time. But I think there's a deeper reason, and it has to do with hackability. Amanda was originally written in C, which means any changes require the full set of developer tools. For example, just to fix a typo in an error message, you would need to find the source, configure and compile it, find and fix the error message, and recompile to test. Fixing something more substantial in the highly interdependent Amanda codebase also requires a deep understanding of many parts of Amanda - from the obscure configuration interface to the oddly interlinked disklist structure. This level of programming skill is not common among Amanda's user base (systems administrators), and I can count the people who understand the disklist structure on one hand.

The result has been a paltry flow of patches from anyone but the core hackers. Furthermore, no entry path has been available by which newcomers could work their way up to being core developers. While I don't want to disparage the work of any of the great programmers who have written Amanda over the years, it's a shame that there have been so few at any time, and I worry about what would happen if the number were to reach zero. So what to do?

$hackability++

We've done a few things to try to make Amanda more hackable. Probably the biggest change is to rewrite parts of Amanda in Perl. I'm asked "why" quite often, and while we had a lot of reasons, two relate directly to hackability. First, more sysadmins know Perl than C, because Perl is quite often used to build the "glue" that links systems together. Interestingly, based on many conversations, it seems that Python may also have been a good choice, as I suspected when I first proposed the rewrite. But it's too late now!

Second, and more importantly, Perl code can be hacked in place. If amvault isn't acting the way you want it to, just open up /usr/sbin/amvault and tweak away. No need to download the source, no need to compile, no segmentation faults, just hacking. When you're done, run a quick diff and send the results to amanda-hackers.

Even users who do not know Perl can take advantage of this in-situ hackability. Within Amanda's C code, if I want a user to try a patch, that user must figure out how to download Amanda's source, apply the patch, configure, compile, and install. None of those steps are trivial. With Perl code, I can often provide a patch that is simple enough to be applied directly to the installed executables by hand, or with a simple application of patch. Everyone stays focused on the bug under investigation, and the user's backups are up and running that much more quickly.

New APIs

As I mentioned before, historically Amanda's code has been highly interdependent. Details of the implementation of the holding disk were spread over most of the files in the server implementation. The dumplevel -987 has a special meaning that is documented nowhere, but referenced in several source files. All of this makes new development difficult, because it's impossible to "slice off" and study a portion of Amanda in isolation.

The solution here is to create abstract interfaces, where new functionality can be "plugged in" and Amanda can use it without changes. The Amanda developers have abused the term "API" for these interfaces, and we now have quite a few:

Application API - an abstraction of backup clients, e.g., ampgsql for Postgres;
Device API - an abstraction of backend storage devices, such as tape, disk, cloud, or DVD-RW;
Changer API - an abstraction of tape changers and other mechanisms for selecting from a set of volumes; and
Script API - a means of invoking scripts before or after certain events during a backup.

This strategy has already paid off: we have seen several new scripts and applications contributed, and the DVD-RW device arrived out of the blue as a contribution from someone who found it useful.

Other Changes

In the interest of greater accessibility to new hackers, we have also put Amanda on github and created a set of good "beginner" projects. Zmanda has even offered to pay people to hack on Amanda, as a way of easing the cost of entry. I also try to point out interesting projects on the Amanda mailing list, particularly projects that Jean-Louis and I probably will not find time to work on.

The idea here is to encourage new hackers to pick up a well-scoped project to become familiar with Amanda. The hackers can then move on to more sophisticated projects that meet their particular backup needs or address a particular interest.

Will You Join Me?

So Amanda is ready for you. When can you start?

Modern Multiprocessing

nospam@example.com (Dustin J. Mitchell) — Mon, 12 Apr 2010 15:24:00 -0500

I've been thinking a lot lately about the way we accomplish multiprocessing. We've seen a significant change in the operation of Moore's law for CPU speeds: today's CPUs are about the same speed as those of a few years ago, but they have more cores, and more virtual processors on those cores. This is great for heavily-loaded servers, which have plenty of distinct tasks to place on those cores and VCPUs, but not so useful for users working with single-threaded applications.

Why are most applications still single-threaded? There are lots of good reasons. Threaded code is harder to write, and not just because it requires careful analysis and use of synchronization primitives: many common tasks are difficult to meaningfully parallelize without careful control over inter-thread communication, and in a portable application you don't have that kind of control. Threaded code generally performs badly on single-CPU systems, which are still common. Some popular languages still make threading difficult, at least in a portable fashion. And threads are still relatively heavyweight entities in most operating systems: you don't spawn ten threads to mergesort a 100-item array.

Some of these problems will go away with a little more time, but some will get worse. NUMA architectures can make sharing data between threads slow. Hyperthreading and its interaction with processor caches adds yet another level of unpredictability.

We know how to build massively parallel systems that run massively parallel algorithms. What is still unknown is how to build portable, simple software that can run efficiently across a vareity of architectures. This is a problem of practice, not theory, and there's lots of interesting work going on in this area.

Of course, there are languages designed explicitly to support communication, such as Limbo or Erlang, Haskell, and Clojure. For the most part, these languages are structured as communicating sequential processes, which is to say that they represent multiprocessing as a set of sequential threads that pass information to one another. Problems of thread safety are subsumed by the languages, but mapping the parallelism to available resources is generally left to the programmer or administrator.

One interesting project is Apple's Grand Central Dispatch. It defines a simple but highly expressive closure syntax (a block) and a mechanism to dynamically schedule execution of such closures (queues). Critically, the GCD library takes care of scaling the parallelism of the queue processing appropriately to the underlying hardware. On a single-threaded CPU, this amounts to cooperative multitasking, but on parallel hardware the operating system can dynamically allocate virtual CPUs to applications needing more parallelism.

This topic seems to come up often in my various pursuits, so I will return to it again.

Want to work on Amanda?

nospam@example.com (Dustin J. Mitchell) — Mon, 12 Apr 2010 14:04:34 -0500

I've not made any secret of the fact that I want more people hacking on Amanda. This is both for selfish reasons -- many hands make light work -- and for altruistic reasons -- a broader community of developers can provide better governance for the project and long-term continuity. With a few noticable exceptions, I haven't had a lot of satisfaction.

I think part of the reason is that Amanda has a steep learning curve, even within the new Perl code. The time to climb that curve is a big investment, and folks with only a small itch to scratch can't afford it.

In an effort to sweeten the pot, we (Zmanda) are offering to pay for flexible work on Amanda. Part-time or full-time, on your own schedule. Your choice of projects. Support and gratitude from the other hackers. And the option to become a full Zmanda employee if that's your bent.

Here are some possible projects, to pique your interest:

MySQL application (to round out the set with ampgsql)
Cyrus Imapd application (gnutar doesn't deal well with the application's tiny files and hard links)
OpenSSL for network transport, using certificates and keys for authentication
Database-backed backup catalog
Amvault upgrade
Handle Logical EOM (LEOM) on all devices that support it, drastically reducing the number of parts Amanda writes
Support for more cloud backends than just S3
Parallel writes to multiple devices

If you're interested, contact me (dustin@zmanda.com) and we'll work something out!

What's New in Amanda: Postgres Backups

nospam@example.com (Dustin J. Mitchell) — Thu, 25 Mar 2010 18:53:00 -0500

In the second installment a series of posts about recent work on Amanda.

The Application API allows Amanda to back up "structured" data -- data that cannot be handled well by 'dump' or 'tar'. Most databases fall into this category, and with the 3.1 release, Amanda ships with ampgsql, which supports backing up Postgres databases using the software's point-in-time recovery mechanism.

The how-to for this application is on the Amanda wiki.

Operation

Postgres, like most "advanced" databases, uses a logging system to ensure consistency even in the face of (some) hardware failures. In essence, it writes every change that it makes to the database to the logfile before changing the database itself. This is similar to the operation of logging filesystems. The idea is that, in the face of a failure, you just replay the log to re-apply any potentially corrupted changes.

Postgres calls its log files WAL (write-ahead log) files. By default, they are 16MB. Postgres runs a shell command to "archive" each logfile when it is full.

So there are two things to back up: the data itself, which can be quite large, and the logfiles. A full backup works like this:

Execute PG_START_BACKUP(ident) with some unique identifier.
Dump the data directory, excluding the active WAL logs. Note that the database is still in operation at this point, so the dumped data, taken alone, will be inconsistent.
Execute PG_STOP_BACKUP(). This archives a text file with the suffix .backup that indicates which WAL files are needed to make the dumped data consistent again.
Dump the required WAL files

An incremental backup, on the other hand, only requires backing up the already-archived WAL files.

A restore is still a manual operation -- a DBA would usually want to perform a restore very carefully. The process is described on the wiki page linked above, but boils down to restoring the data directory and the necessary WAL files, then providing postgres with a shell command to "pull" the WAL files it wants. When postgres next starts up, it will automatically enter recovery mode and replay the WAL files as necessary.

Quiet Databases

On older Postgres versions, making a full backup of a quiet database is actually impossible. After PGSTOPBACKUP() is invoked, the final WAL file required to reconstruct a consistent database is still "in progress" and thus not archived yet. Since the database is quiet, postgres does not get any closer to archiving that WAL file, and the database hangs (or, in the case of ampgsql, times out).

Newer versions of Postgres do the obvious thing: PGSTOPBACKUP() "forces" an early arciving of the current WAL file.

The best solution for older versions is to make sure transactions are being committed to the database all the time. If the database is truly silent during the dump (perhaps it is only accessed during working hours), then this may mean writing garbage rows to a throwaway table:

CREATE TABLE push_wal AS SELECT * FROM GENERATE_SERIES(1, 500000);
DROP TABLE push_wal;

Note that using CREATE TEMPORARY TABLE will not work, as temporary tables are not written to the WAL file.

As a brief encounter in #postgres taught me, another option is to upgrade to a more modern version of Postgres!

Log Incremental Backups

DBAs and backup admins generally want to avoid making frequent full backups, since they're so large. The usual pattern is to make a full backup and then dump the archived log files on a nightly basis for a week or two. As the log files are dumped, they can be deleted from the database server, saving considerable space.

In Amanda terms, each of these dumps is an "incremental", and is based on the previous night's backup. That means that the dump after the full is level 1, the next is level 2, and so on. Amanda currently supports 99 levels, but this limit is fairly arbitrary and can be increased as necessary.

The problem in ampgsql, as implemented, is that it allows Amanda to schedule incremental levels however it likes. Amanda considers a level-n backup to be everything that has changed since the last level-n-1 backup. This works great for GNU tar, but not so well for Postgres. Consider the following schedule:

Monday	level 0
Tuesday	level 1
Wednesday	level 2
Thursday	level 1

The problem is that the dump on Thursday, as a level 1, needs to capture all changes since the previous level 0, on Monday. That means that it must contain all WAL files archived since Monday, so those WAL files must remain on the database server until Thursday.

The fix to this is to only perform level 0 or level-n+1 dumps, where n is the level of the last dump performed. In the example above, this means either a level 0 or level 3 dump on Thursday. A level 0 is a full backup and requires no history. A level 3 would only contain WAL files archived since the level 2 dump on Wednesday, so any WAL files before that could be deleted from the database server.

[EDIT: replaced "corrupt" with the more accurate "inconsistent"; clarified final example]

Happy Ada Lovelace Day

nospam@example.com (Dustin J. Mitchell) — Thu, 25 Mar 2010 13:35:38 -0500

Yesterday, March 24, was Ada Lovelace Day. I was at Pumping Station: One, and decided to spend an hour or so writing something to honor the first computer programmer. I was feeling singularly uninspired, and googling for "Ada Lovelace" didn't turn up anything interesting. But it did give me an idea: write a program that googles for you!

I haven't written much JavaScript lately, but I've heard a lot about the work Google's done to provide easy JavaScript libraries and APIs. I thought it'd be interesting to try out some of these APIs. It was! I hacked up a quick HTML page, using a Searcher object, to search for Lovelace images. Here's the code:

The only downside I've found is that even if you request a "large" result set, you only get 8 results. I don't know of a way to get subsequent results. So this really only cycles through a few images. Still, it only took me an hour, and most of that was remembering how to use the DOM.

Solving an Encoding Mystery

nospam@example.com (Dustin J. Mitchell) — Mon, 15 Mar 2010 10:52:00 -0500

I don't write about it here, but I've been getting into brewing beer. I downloaded an app for my iPhone, iBrewMaster, which helps me store recipes and track batches of homebrew through the brewing, fermeting, and serving stages.

I recently decided to make a clone of Dogfish Head's Raison D'être. This beer is fantastic, but that's beside the point. I added the recipe to the app, and clicked save. In the menu, however, I saw "Raison D'√™tre". Not pretty. The app has a feature where you create a "batch" from a particular recipe. I did so, and the name of the batch appeared as "Raison D'‚àö‚Ñ¢tre". Even worse! I emailed the app developer, who replied almost immediately regarding how he was handling encodings. I won't go into detail, except to say that he is careful to always encode strings before inserting them into the internal SQLite database.

I wanted to give him more information than a simple "well, it doesn't work and you should fix it!" So I set about trying to replicate this particular sequence of characters. I know that Macs (and the iPhone is a Mac) have good support for encodings, so I assume the UI is not at fault. I know that the strings go into SQLite in UTF-8, and that SQLite just treats them as bytestrings, so a later SELECT will return the same UTF-8 bytestring as was specified in the INSERT. So the error must occurr somewhere between the SELECT and displaying the string on-screen.

Character Encodings

A word about encodings, with a bit of revisionist history. In the beginning, there was the Unicode Character Set. Every funny squiggle that the monks knew how to make on paper had a number - its Unicode codepoint. There are a lot of Unicode characters - 95156 at last count.

Then computers were invented, and we needed to represent, or encode, these squiggles in only 7 bits each. Someone (who probably only spoke English) decided "well, you can only use the first 127 characters," and thus ASCII was born. If you want to write an "ñ" in ASCII, you'll have to make do with "n". Don't even ask about "葉". We soon got a bit less stingy (get it?), and with 8 bits available, everyone rushed to put their favorite characters at code points 128-255 - regardless of what Unicode put there. For example, in latin-1, "ñ" is at code point 240. On a Mac, it's 150. Trying to decode a byte in the range 128-255 was a challenge, because the encoding was usually unknown. Those were dark days, as chaos reigned.

Finally, some enlightened souls (Rob Pike and Ken Thompson) scratched out a new encoding on a placemat. In this encoding, called UTF-8, all of the 7-bit ASCII characters still fit in one byte, and look exactly the same. But all of the other characters take more than one byte, with some cleverness applied to make the encoding both compact and easy to decode reliably. As a side note, the Unicode characters 128-255 match the latin-1 character set.

Now, if you have a sequence of unicode characters, say "青蛙吃我的餃子" that you want to store digitally, then you need to encode them, preferably in UTF-8, with the result being a bytestring. When you want the unicode characters back (perhaps to do some hyphenation), you perform the reverse operation, and decode from UTF-8 to Unicode Characters.

Back to the Chase

A common mistake with multi-byte encodings is to assume that each byte is a distinct character, perhaps with a for loop indexing an array of bytes. Since one character ("ê") turned into two ("√™"), this was a reasonable guess. A little Python (in my Unicode-enabled terminal) shows me the encoded form of "ê":

>>> print `u"ê"` 
u'\xc3\xaa'

Treating those as Unicode characters instead of bytes is simple:

>>> print u"Raison D'\u00c3\u00aatre"
Raison D'Ãªtre

No dice. Another common mistake is to encode with one encoding, and decode with another. This is especially common when the programming environment "automatically" performs encodings or decodings. For example, Python has an annoying habit of decoding to ASCII, which produces the infamous UnicodeEncodeError.

So let's try this out, guessing at the encoding that's used on the way out.

>>> orig = u"Raison D'être"
>>> print orig.encode('utf-8').decode('latin-1')
Raison D'Ãªtre

The result is the same as the single-byte treatment above. Why? Recall that the latin-1 encoding is identical to Unicode in the range 128-255, so treating a byte as a Unicode character is the same as treating it as a latin-1 character.

At this point, I perused the list of encodings Python supports, and "mac-roman" jumped out as a potential culprit.

>>> print orig.encode('utf-8').decode('mac-roman')
Raison D'√™tre

A match! What about the longer string of nonsense in the batch name?

>>> once = orig.encode('utf-8').decode('mac-roman')
>>> print once.encode('utf-8').decode('mac-roman')
Raison D'‚àö‚Ñ¢tre

Another match.

I don't know much about iPhone internals, but I assume that the string library treats a bytestring without any attached encoding as being in the Mac-Roman character set. When the value was selected out of the recipes table, this decoding was done implicitly, followed by an explicit UTF-8 encoding when inserting into the batches table, and another implicit Mac-Roman decoding when selecting the batch for display.

What's New in Amanda: Transfer Architecture

nospam@example.com (Dustin J. Mitchell) — Fri, 12 Mar 2010 14:06:47 -0600

Amanda's primary mission in life is to move large quantities of data around. Historically, this has been done through a patchwork of methods, each written separately and with its own quirks. POSIX pipes, TCP sockets, shared memory, on-disk cache files -- Amanda's done it all. But these multiple implementations were error-prone, difficult to maintain, and often not the most efficient approach.

In an effort to remedy this, we introduced the transfer architecture, abbreviated XFA. This was technically included in Amanda-2.6.1, but was only used by amvault. In the upcoming Amanda-3.1 release, however, the XFA is central to all recovery operations, and is used internally by the taper (the portion of the backup system that writes to devices).

This post highlights some of the features of the transfer architecture, and some of the improvements we'd like to make.

Transfers and Elements

A transfer is pretty simple: it moves data from one place to another. It is built from a list of transfer elements, the first being the data source and the last the destination. Any elements in the middle are filters, and could apply compression or encryption, for example. Elements are automatically connected to one another using the most efficient means available.

Transfers operate asynchronously, meaning that ordinary execution of Amanda continues in parallel to the movement of data. When the transfer needs Amanda's attention, it sends a message, and Amanda reacts accordingly.

The beauty of this architecture is in the variety of elements that can be connected.

Sources
- File or socket
- DirectTCP Connection
- Holding Disk
- Spanned dumpfile
- Random or repeated patterns (for testing)
Filters
- In-process compression or encryption (e.g., libgz)
- External utilities (e.g., gzip or amgpgcrypt)
Destinations
- File or socket
- DirectTCP Connection
- Spanned Dumpfile

The full list is given in the POD.

Benefits

The advantage of the transfer architecture is that it massively simplifies the process of transferring data. Nowhere is this more obvious than in the splitting and re-joining of dumpfiles.

The Amanda::Xfer::Recovery::Source element, which reads from spanned dumpfiles, cooperates with the Clerk, via transfer messages, to load the proper volumes and seek to the proper files to recover and entire dumpfile, even if it is distributed over more than one source volume. Similarly, Amanda::Xfer::Taper::Dest works with a Scribe to load volumes and update the catalog while spanning dumpfiles.

So the Amanda taper is a simple wrapper around a transfer from holding disk (FILE-WRITE) or socket (PORT-WRITE) to a spanned dumpfile, using a scribe. Similarly, all of the recovery tools use a clerk and a transfer to read dumpfiles off the appropriate volumes. And amvault combines the two to simultaneously read from one volume and write to another.

Future

At this point, the transfer architecture is a reliable abstraction, but it is not yet terribly efficient. The advantage of the abstraction, though, is that as it is made more efficient, all of the components of Amanda that make use of it will immediately become faster, with no changes.

There are plenty of places in Amanda where the transfer architecture will be useful, and certainly plenty of work to do to make it faster. If you're interested in helping out, please let me know!

What's New in Amanda: Automated Tests

nospam@example.com (Dustin J. Mitchell) — Fri, 12 Mar 2010 12:03:52 -0600

This is the first in what will be a series of posts about recent work on Amanda. Amanda has a reputation as old and crusty -- not so! Hopefully this series will help to illustrate some of the new features we've completed, and what's coming up. I'll be cross-posting these on the Zmanda Team Blog too.

Among open-source applications, Amanda is known for being stable and highly reliable. To ensure that Amanda lives up to this reputation, we've constructed an automated testing framework (using Buildbot) that runs on every commit. I'll give some of the technical details after the jump, but I think the numbers speak for themselves. The latest release of Amanda (which will soon be 3.1.0) has 2936 tests!

These tests range from highly-focused unit tests, for example to ensure that all of Amanda's spellings of "true" are parsed correctly, all the way up to full integration: runs of amdump and the recovery applications. The tests are implemented with Perl's Test::More and Test::Harness. The result for the current trunk looks like this:

=setupcache.....................ok
Amanda_Archive..................ok
Amanda_Changer..................ok
Amanda_Changer_compat...........ok
Amanda_Changer_disk.............ok
Amanda_Changer_multi............ok
Amanda_Changer_ndmp.............ok
Amanda_Changer_null.............ok
Amanda_Changer_rait.............ok
Amanda_Changer_robot............ok
Amanda_Changer_single...........ok
Amanda_ClientService............ok
Amanda_Cmdline..................ok
Amanda_Config...................ok
Amanda_Curinfo..................ok
Amanda_DB_Catalog...............ok
Amanda_Debug....................ok
Amanda_Device...................ok
        211/428 skipped: various reasons
Amanda_Disklist.................ok
Amanda_Feature..................ok
Amanda_Header...................ok
Amanda_Holding..................ok
Amanda_IPC_Binary...............ok
Amanda_IPC_LineProtocol.........ok
Amanda_Logfile..................ok
Amanda_MainLoop.................ok
Amanda_NDMP.....................ok
Amanda_Process..................ok
Amanda_Recovery_Clerk...........ok
Amanda_Recovery_Planner.........ok
Amanda_Recovery_Scan............ok
Amanda_Report...................ok
Amanda_Tapelist.................ok
Amanda_Taper_Scan...............ok
Amanda_Taper_Scan_traditional...ok
Amanda_Taper_Scribe.............ok
Amanda_Util.....................ok
Amanda_Xfer.....................ok
amadmin.........................ok
amarchiver......................ok
amcheck.........................ok
amcheck-device..................ok
amcheckdump.....................ok
amdevcheck......................ok
amdump..........................ok
amfetchdump.....................ok
amgetconf.......................ok
amgtar..........................ok
amidxtaped......................ok
amlabel.........................ok
ampgsql.........................ok
        40/40 skipped: various reasons
amraw...........................ok
amreport........................ok
amrestore.......................ok
amrmtape........................ok
amservice.......................ok
amstatus........................ok
amtape..........................ok
amtapetype......................ok
bigint..........................ok
mock_mtx........................ok
noop............................ok
pp-scripts......................ok
taper...........................ok
All tests successful, 251 subtests skipped.
Files=64, Tests=2936, 429 wallclock secs (155.44 cusr + 31.48 csys = 186.92 CPU)

The skips are due to tests that require external resources - tape drives, database servers, etc. The first part of the list contains tests for almost all perl packages in the Amanda namespace. These are generally unit tests of the new Perl code, although some tests integrate several units due to limitations of the interfaces. The second half of the list is tests of Amanda command-line tools. These are integration tests, and ensure that all of the documented command-line options are present and working, and that the tool's behavior is correct. The integration tests are necessarily incomplete, as it's simply not possible to test every permutation of this highly flexible package.

The =setupcache test at the top is interesting: because most of the Amanda applications need some dumps to work against, we "cache" a few completed amdump runs using tar, and re-load them as needed during the subsequent tests. This speeds things up quite a bit, and also removes some variability from the tests (there are a lot of ways an amdump can go wrong!).

The entire test suite is run at least 54 times for every commit by Buildbot. We test on 42 different architectures - about a dozen linux distros, in both 32- and 64-bit varieties, plus Solaris 8 and 10, and Darwin-8.10.1 on both x86 and PowerPC. The remaining tests are for special configurations -- server-only, client-only, special runs on a system with several tape drives, and so on.

Testing Legacy Code

nospam@example.com (Dustin J. Mitchell) — Wed, 17 Feb 2010 12:26:15 -0600

I just read Roy Osherove's The Art of Unit Testing with Examples in .NET, on the advice of a slashdot review. I was not terribly impressed with the book, but reading it did help me to solidify my thinking about testing and test-driven development, and put words to concepts I had come to on my own.

Rather late in the book, Osherove describes three properties of good tests.

Trustworthiness - Do developers believe that passing tests mean things are working? Do developers believe that failing tests indicate a real bug?
Maintainability - Do developers think that tests are easy to add and maintain, or are they likely to avoid writing tests when rushed?
Readability - Do developers often consult the unit tests to see how the system under test is supposed to work?

What most struck me was that these properties were related to developers' perceptions of the tests, not the tests themselves. Tests are as much a social artifact of a project as a technical tool.

Buildbot's Tests

Around the time I was reading this, one of the more prolific Buildbot contributors commented, "I try not to change the tests - they scare me." Buildbot's tests were badly isolated, slow, and failed intermittently. As maintainer, I had grown accustomed to saying "oh, that test fails sometimes, don't worry about it" - a trustworithiness failure. Because of the terrible isolation, changing just about anything in Buildbot would cause dozens of tests to fail, requiring repetitive editing to fix - not maintainable. And the tests consisted of long sequences of operations and assertions, written in the Twisted style, which is already not readable. As a result, even I don't know what most of the tests are actually testing. This was a bad situation for any application, but particularly embarassing for a popular testing tool!

So I blew the tests away. Well, not really - I moved them to buildbot/broken_test/ in hopes they can be useful in writing new tests, and so that the braver souls among us can still run them. Now our metabuildbot is green, and I can legitimately ask for unit tests for new code.

There are costs associated with this move, too. A lot of people have worked very hard to write tests that have now been categorically labeled "broken," to whom all I can say is "I'm sorry". With far fewer tests and thus far worse coverage, it's also difficult to have confidence that Buildbot really works. The short-term workaround is to make a few beta releases and rely on real-world testing to suss out any problem.

So this is only the first step. We - I - still need to write real tests for the vast majority of the Buildbot code. That's particularly complicated because Buildbot's units are badly isolated, and interfaces are ill-defined. I will need to do a good bit of refactoring to bring it into compliance.

Revising the allowForce option

nospam@example.com (Dustin J. Mitchell) — Sat, 13 Feb 2010 15:08:29 -0600

Buildbot's WebStatus display has, for a long time, had an allowForce option which controls what kind of mayhem can be wrought via the web interface. Historically, this has been a boolean option: either web users can do everything (force builds and shut down slaves) or nothing. Bug 701 asks that we change that to give more granular access control.

Buildbot has an interesting way of separating the status display from the control functionality. It has two parallel interface hierarchies, IStatus and IControl, implementing the necessary methods. The IStatus hierarchy is illustrated with the orange bubbles here:

The IControl hierarchy is similar, although it only goes down to the Build level right now.

When allowForce is true, the WebStatus object adapts the buildmaster to the IControl interface and adds a link to the result in its control attribute. Forcing a build or shutting down a slave then uses this object to navigate to the appropriate control object and calls a method from the corresponding interface. If the control attribute is None, no access is allowed.

This scheme has the advantage that it is difficult to accidentally expose functionality, since when allowForce is false, the control methods are inaccessible. However, it has the disadvantage of not allowing any more granular level of access control.

I just reworked the web status to have a more flexible authorization mechanism, and while I wasn't able to remove the IControl hierarchy entirely, I was able to marginalize it to only those code blocks that need to perform controlled actions, instead of passing control objects all over the place.