Code V.igoro.us - amanda

Amanda's Transfer Mechanisms

nospam@example.com (Dustin J. Mitchell) — Sat, 22 Jan 2011 14:47:13 -0600

There's been a bit of confusion on the mailing list and IRC about how Amanda assembles transfers out of transfer elements, and how transfer mechanisms influence that.

In the final form of a transfer, any two adjacent elements must have the same mechanism. For example is, an upstream element speaking XFER_MECH_PUSH_BUFFER cannot talk to a downstream element using XFER_MECH_READ_FD (nor, more confusingly, XFER_MECH_PULL_BUFFER). So each mechanism is an isolated definition of "here's how upstream and downstream should talk". They come in pairs because generally anything upstream can do to downstream (e.g., upstream can write to downstream's fd) can occur in reverse (e.g., downstream can read from upstream's fd).

What makes this confusing is that if you specify a set of elements which can't talk directly to one another, then xfer.c will add "glue" elements between the specified elements. To make that concrete, imagine you specify a transfer as

source-holding --> filter-xor --> dest-fd

(if you like practical examples, then imagine filter-xor is a buffer-based decompression filter, and you're pulling data from holding disk, decompressing, and sending to a pipe -- something amfetchdump would do). Here are the mechanisms supported by each element:

source-holding:
 XFER_MECH_PULL_BUFFER

filter-xor
 XFER_MECH_PULL_BUFFER (input) and
 XFER_MECH_PULL_BUFFER (output)
or
 XFER_MECH_PUSH_BUFFER (input) and
 XFER_MECH_PUSH_BUFFER (output)

dest-fd
 XFER_MECH_WRITEFD (input)

In putting these together, source-holding and filter-xor can use the same mechanism (PULL_BUFFER). This leaves filter-xor using PULL_BUFFER for output, but dest-fd does not support this. So xfer.c adds a glue element that can speak PULL_BUFFER on input and WRITEFD on output. This element basically loops in a thread, calling upstream->pull_buffer and write(downstream->input_fd, buffer). So the final xfer looks like

 source-holding --(PULL_BUFFER)--> filter-xor --(PULL_BUFFER)--> glue --(WRITEFD)--> dest-fd

Hopefully that helps to explain how the glue works.

Note that one of the cool things about this arrangement is that in most cases the complexity is in the glue, not the elements. In fact, in this case the glue provides the only thread that's required to run this transfer, so the other three element implementations don't need to manage threads at all.

IPv6 and Amanda

nospam@example.com (Dustin J. Mitchell) — Fri, 16 Jul 2010 22:04:00 -0500

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case. First, a bit of background on how Amanda's networking code works, and what had to change to support IPv6. Amanda supports security mechanisms called BSD (the oldest), BSDUDP, and BSDTCP. These all authenticate (if you can call it that) using the same sorts of checks that rsh uses. The incoming connection is accepted if:

it is from a "reserved" port (less than 1024);
the address of the initiator has complementary forward and reverse DNS records in place; and
the initiator's hostname is in .amandahosts

During a backup operation, the Amanda server contacts each client host. When using the BSD authentications, this triggers amandad, which checks the above restrictions before beginning communication with the server. This initial connection is packet-based, and can be carried out over UDP (for BSD and BSDUDP) or TCP (BSDTCP). When a dump begins, several "streams" are opened to transmit the data, index, and metadata. For BSD and BSDUDP, each stream is implemented as a distinct TCP connection, where the client sends a port number to the server and the server connects to that port. BSDTCP multiplexes all streams over a single TCP connection using a basic type/length packet encapsulation.

The first challenge in adding IPv6 support was to deal properly with IPv6 addresses when querying the DNS. That meant switching to getaddrinfo and getnameinfo, as suggested by Jun-ichiro itojun Itoh. These functions bring their own compatibility problems, but Amanda uses gnulib, which provides compatibile implementations on systems where they are not available, minimizing the difficulty.

We had a lot of trouble from systems such as RHEL3 possessing IPv6 support in the compiler environment but not in the kernel. On such systems, code using constants like AF_INET6 or AI_V4MAPPED would compile without problems, but fail at runtime. We added a WORKING_IPV6 preprocessor conditional, without which all references to IPv6-related symbols were removed. At configure time, Amanda tries to create an IPv6 socket, and sets this conditional to true if it succeeds. The --without-ipv6 configure option forcibly disables IPv6 support.

The sockaddr structures and API for IPv6 are fairly difficult to use, particularly if it's not known in advance what sort of address they will contain. We added a set of macros and utility functions in sockaddr-util.c and sockaddr-util.h. Using these macros throughout Amanda removed a significant amount of code that was conditionalized on both compile-time support and runtime address family, and centralized that logic in one easily-maintained place.

On our build systems, we had to deal with different levels of support in the compile environment and the kernel. This is fine: most Amanda users install binary packages that are produced on roughly the same OS distribution and version as was used for the build, so the kernel support is generally the same. However, a third variable has tripped up lots of Amanda users: system configuration. In particular, several newer Linux distributions have shipped with localhost resolving to ::1 vi /etc/hosts, but without enough interface configuration to actually utilize a socket bound to that address. Amanda uses localhost sockets for inter-process communication, so this misconfiguration causes backup operations to fail. The solution is to either finish configuring IPv6 on the host, remove the reference to ::1 in /etc/hosts, or build Amanda with --without-ipv6.

I have not yet heard of an Amanda installation where IPv6 communication is in use. But I have heard from countless IPv4 users whose Amanda installations have failed due to bad IPv6 support. At the moment, then, I feel that adding IPv6 support to Amanda has been a net negative for the project. Although there is doubtless room for improvement, I will not entertain patches for better IPv6 support, for fear they will introduce new bugs for our exclusively IPv4 userbase.

Of course, all of this may change as dual-stack networks grow more prevalent and are replaced by pure IPv6 networks!

What's New in Amanda: The End of Fragmentation

nospam@example.com (Dustin J. Mitchell) — Thu, 08 Jul 2010 18:07:00 -0500

Most of my posts in this series have been about features that are available in a released version of Amanda. This time, I want to share a project I'm working on right now - one that will be available in Amanda-3.2. I'm reworking the way Amanda writes its data to tape (or any other kind of storage) to make it more efficient, more reliable, and simpler to configure.

Historically, Amanda's conservative approach to finicky tape hardware has meant that it wasted some space at the end of each tape. With the changes I'm working on, Amanda will no longer waste this space, and can also avoid some needless copying of data in most cases, with a minimum of additional risk.

Amanda's Storage Format

Before examining the new functionality, let's look at Amanda's storage format. Amanda treats all storage devices like tapes¹ - a set of sequentially numbered data files of arbitrary size, each composed of a sequence of fixed-size blocks. Each file begins with a one-block header that identifies the dump and gives information about its contents. The header is followed by blocks of raw data.

Amanda supports writing a dump across multiple tapes - spanning.² The technique is this: a dump is split into a sequence of parts, and each part is written a a single file on a volume. During recovery, Amanda reads the parts in sequence, and concatenates their data to reproduce the original dumpfile. Usually all parts are the same size, and this size is generally 1-5% of the tape capacity.

Better Safe than Sorry - At a Cost

Amanda was originally designed around tape drives - in fact, if you look at the history of Linux kernel support for tape drives, it is closely intertwined with Amanda development. Tape drives are finicky beasts, and in many cases cannot distinguish the end of the tape (called EOM) from any other fatal error. Worse, tape drives employ large caches to ensure they can write continuously, and when an error occurs all of the data in that cache is lost, and there is no way for Amanda to determine how much actually made it onto the tape. Beginning a new on-tape file (writing a filemark) flushes the cache and signals any errors immediately.

Since time immemorial, then, Amanda has treated any error from the tape drive as EOM, and assumed that all data written since the last filemark is potentially corrupt. That means that the part in progress when the error occurred is logged as PARTIAL, and Amanda will start at the beginning of that part on the next tape.

The PARTIAL part is recorded in the catalog, but will not be used for recovery, so it is effectively wasted space. A little arithmetic will tell you that, on average, each tape will waste half of the part size. This is at least excusable with real tape drives; with vtapes (disk), this wasted space is completely unnecessary. Worse, vtapes are most flexible when they are kept small and dumps are spanned over many vtapes per night; but the wasted space increases linearly with the number of vtapes used.

In order to rewind a part and write it again on a new tape, Amanda also needs to keep its data somewhere, called the part cache. When the dump is on the holding disk, the holding disk acts as a part cache. Otherwise, Amanda can cache parts in memory or on disk. Caching in memory is faster, but requires a lot of RAM for a reasonable part size. Caching on disk allows larger parts, but is considerably slower.

Logical EOM

More recent tape drives (those made in the last decade or so) have a feature called "early warning". With this feature enabled, the drive alerts the host system when it is "near" EOM, and flushes the cache to tape. Exactly what "near" means is not specified in the SCSI standard, but in general there's room to flush the cache and write a filemark, at least. This is sometimes called a logical EOM - LEOM.

Amanda can take advantage of this functionality to cleanly finish a part before running headlong into a physical EOM. This eliminates the wasted space for a PARTIAL part, and also eliminates the need to cache parts, since rewinding is not required. In one small change (well, OK, it's about 4,300 lines), Amanda gets faster and uses storage space more efficiently. What's not to love?

Better yet, all of the non-tape devices (vtapes, S3 devices, DVD-ROMs, etc.) can easily emulate LEOM, so backups to these devices will automatically benefit from this improvement.

In the Code

Three important patches toward this functionality were just committed. What remains is to set up real LEOM support for the VFS device (vtapes) and for the tape device.

The VFS device can, of course, trivially emulate LEOM when it is enforcing the MAX_VOLUME_USAGE property - the vtape length. However, predicting when a filesystem will run out of space is much more difficult. We are still discussing options, and I would love to hear suggestions here or on the mailing list.

As for the tape device, it will assume that LEOM is not supported unless the user configures it explicitly (with the "LEOM" device property) or we can determine support for LEOM from the operating system at runtime. Unfortunately, this is one of those areas so technical that only a half-dozen people know how it works, so it may take me some time to track down this information for non-Linux operating systems. Again, advice and assistance is welcome!

¹This is an ages-old design decision, but one that artificially constrains Amanda's flexibility, especially with vtapes.
²In fact, Amanda has supported spanning for something like 7 years now. Yet I occasionally see users in #amanda complaining about this serious limitation and wondering when we're going to do anything about it. Will 2003 be soon enough?

What's New in Amanda: Hackability

nospam@example.com (Dustin J. Mitchell) — Thu, 01 Jul 2010 22:29:00 -0500

It's been a while since I've posted about recent development in Amanda, but it's not for lack of interesting topics!

Today I want to talk a little bit about Amanda's development. Historically, Amanda has always had a small, core group of developers who do the lion's share of the development work. There are probably lots of reasons for this, not least of which is that a backup application isn't the sexiest project on which to spend your spare time. But I think there's a deeper reason, and it has to do with hackability. Amanda was originally written in C, which means any changes require the full set of developer tools. For example, just to fix a typo in an error message, you would need to find the source, configure and compile it, find and fix the error message, and recompile to test. Fixing something more substantial in the highly interdependent Amanda codebase also requires a deep understanding of many parts of Amanda - from the obscure configuration interface to the oddly interlinked disklist structure. This level of programming skill is not common among Amanda's user base (systems administrators), and I can count the people who understand the disklist structure on one hand.

The result has been a paltry flow of patches from anyone but the core hackers. Furthermore, no entry path has been available by which newcomers could work their way up to being core developers. While I don't want to disparage the work of any of the great programmers who have written Amanda over the years, it's a shame that there have been so few at any time, and I worry about what would happen if the number were to reach zero. So what to do?

$hackability++

We've done a few things to try to make Amanda more hackable. Probably the biggest change is to rewrite parts of Amanda in Perl. I'm asked "why" quite often, and while we had a lot of reasons, two relate directly to hackability. First, more sysadmins know Perl than C, because Perl is quite often used to build the "glue" that links systems together. Interestingly, based on many conversations, it seems that Python may also have been a good choice, as I suspected when I first proposed the rewrite. But it's too late now!

Second, and more importantly, Perl code can be hacked in place. If amvault isn't acting the way you want it to, just open up /usr/sbin/amvault and tweak away. No need to download the source, no need to compile, no segmentation faults, just hacking. When you're done, run a quick diff and send the results to amanda-hackers.

Even users who do not know Perl can take advantage of this in-situ hackability. Within Amanda's C code, if I want a user to try a patch, that user must figure out how to download Amanda's source, apply the patch, configure, compile, and install. None of those steps are trivial. With Perl code, I can often provide a patch that is simple enough to be applied directly to the installed executables by hand, or with a simple application of patch. Everyone stays focused on the bug under investigation, and the user's backups are up and running that much more quickly.

New APIs

As I mentioned before, historically Amanda's code has been highly interdependent. Details of the implementation of the holding disk were spread over most of the files in the server implementation. The dumplevel -987 has a special meaning that is documented nowhere, but referenced in several source files. All of this makes new development difficult, because it's impossible to "slice off" and study a portion of Amanda in isolation.

The solution here is to create abstract interfaces, where new functionality can be "plugged in" and Amanda can use it without changes. The Amanda developers have abused the term "API" for these interfaces, and we now have quite a few:

Application API - an abstraction of backup clients, e.g., ampgsql for Postgres;
Device API - an abstraction of backend storage devices, such as tape, disk, cloud, or DVD-RW;
Changer API - an abstraction of tape changers and other mechanisms for selecting from a set of volumes; and
Script API - a means of invoking scripts before or after certain events during a backup.

This strategy has already paid off: we have seen several new scripts and applications contributed, and the DVD-RW device arrived out of the blue as a contribution from someone who found it useful.

Other Changes

In the interest of greater accessibility to new hackers, we have also put Amanda on github and created a set of good "beginner" projects. Zmanda has even offered to pay people to hack on Amanda, as a way of easing the cost of entry. I also try to point out interesting projects on the Amanda mailing list, particularly projects that Jean-Louis and I probably will not find time to work on.

The idea here is to encourage new hackers to pick up a well-scoped project to become familiar with Amanda. The hackers can then move on to more sophisticated projects that meet their particular backup needs or address a particular interest.

Will You Join Me?

So Amanda is ready for you. When can you start?

Want to work on Amanda?

nospam@example.com (Dustin J. Mitchell) — Mon, 12 Apr 2010 14:04:34 -0500

I've not made any secret of the fact that I want more people hacking on Amanda. This is both for selfish reasons -- many hands make light work -- and for altruistic reasons -- a broader community of developers can provide better governance for the project and long-term continuity. With a few noticable exceptions, I haven't had a lot of satisfaction.

I think part of the reason is that Amanda has a steep learning curve, even within the new Perl code. The time to climb that curve is a big investment, and folks with only a small itch to scratch can't afford it.

In an effort to sweeten the pot, we (Zmanda) are offering to pay for flexible work on Amanda. Part-time or full-time, on your own schedule. Your choice of projects. Support and gratitude from the other hackers. And the option to become a full Zmanda employee if that's your bent.

Here are some possible projects, to pique your interest:

MySQL application (to round out the set with ampgsql)
Cyrus Imapd application (gnutar doesn't deal well with the application's tiny files and hard links)
OpenSSL for network transport, using certificates and keys for authentication
Database-backed backup catalog
Amvault upgrade
Handle Logical EOM (LEOM) on all devices that support it, drastically reducing the number of parts Amanda writes
Support for more cloud backends than just S3
Parallel writes to multiple devices

If you're interested, contact me (dustin@zmanda.com) and we'll work something out!

What's New in Amanda: Postgres Backups

nospam@example.com (Dustin J. Mitchell) — Thu, 25 Mar 2010 18:53:00 -0500

In the second installment a series of posts about recent work on Amanda.

The Application API allows Amanda to back up "structured" data -- data that cannot be handled well by 'dump' or 'tar'. Most databases fall into this category, and with the 3.1 release, Amanda ships with ampgsql, which supports backing up Postgres databases using the software's point-in-time recovery mechanism.

The how-to for this application is on the Amanda wiki.

Operation

Postgres, like most "advanced" databases, uses a logging system to ensure consistency even in the face of (some) hardware failures. In essence, it writes every change that it makes to the database to the logfile before changing the database itself. This is similar to the operation of logging filesystems. The idea is that, in the face of a failure, you just replay the log to re-apply any potentially corrupted changes.

Postgres calls its log files WAL (write-ahead log) files. By default, they are 16MB. Postgres runs a shell command to "archive" each logfile when it is full.

So there are two things to back up: the data itself, which can be quite large, and the logfiles. A full backup works like this:

Execute PG_START_BACKUP(ident) with some unique identifier.
Dump the data directory, excluding the active WAL logs. Note that the database is still in operation at this point, so the dumped data, taken alone, will be inconsistent.
Execute PG_STOP_BACKUP(). This archives a text file with the suffix .backup that indicates which WAL files are needed to make the dumped data consistent again.
Dump the required WAL files

An incremental backup, on the other hand, only requires backing up the already-archived WAL files.

A restore is still a manual operation -- a DBA would usually want to perform a restore very carefully. The process is described on the wiki page linked above, but boils down to restoring the data directory and the necessary WAL files, then providing postgres with a shell command to "pull" the WAL files it wants. When postgres next starts up, it will automatically enter recovery mode and replay the WAL files as necessary.

Quiet Databases

On older Postgres versions, making a full backup of a quiet database is actually impossible. After PGSTOPBACKUP() is invoked, the final WAL file required to reconstruct a consistent database is still "in progress" and thus not archived yet. Since the database is quiet, postgres does not get any closer to archiving that WAL file, and the database hangs (or, in the case of ampgsql, times out).

Newer versions of Postgres do the obvious thing: PGSTOPBACKUP() "forces" an early arciving of the current WAL file.

The best solution for older versions is to make sure transactions are being committed to the database all the time. If the database is truly silent during the dump (perhaps it is only accessed during working hours), then this may mean writing garbage rows to a throwaway table:

CREATE TABLE push_wal AS SELECT * FROM GENERATE_SERIES(1, 500000);
DROP TABLE push_wal;

Note that using CREATE TEMPORARY TABLE will not work, as temporary tables are not written to the WAL file.

As a brief encounter in #postgres taught me, another option is to upgrade to a more modern version of Postgres!

Log Incremental Backups

DBAs and backup admins generally want to avoid making frequent full backups, since they're so large. The usual pattern is to make a full backup and then dump the archived log files on a nightly basis for a week or two. As the log files are dumped, they can be deleted from the database server, saving considerable space.

In Amanda terms, each of these dumps is an "incremental", and is based on the previous night's backup. That means that the dump after the full is level 1, the next is level 2, and so on. Amanda currently supports 99 levels, but this limit is fairly arbitrary and can be increased as necessary.

The problem in ampgsql, as implemented, is that it allows Amanda to schedule incremental levels however it likes. Amanda considers a level-n backup to be everything that has changed since the last level-n-1 backup. This works great for GNU tar, but not so well for Postgres. Consider the following schedule:

Monday	level 0
Tuesday	level 1
Wednesday	level 2
Thursday	level 1

The problem is that the dump on Thursday, as a level 1, needs to capture all changes since the previous level 0, on Monday. That means that it must contain all WAL files archived since Monday, so those WAL files must remain on the database server until Thursday.

The fix to this is to only perform level 0 or level-n+1 dumps, where n is the level of the last dump performed. In the example above, this means either a level 0 or level 3 dump on Thursday. A level 0 is a full backup and requires no history. A level 3 would only contain WAL files archived since the level 2 dump on Wednesday, so any WAL files before that could be deleted from the database server.

[EDIT: replaced "corrupt" with the more accurate "inconsistent"; clarified final example]

What's New in Amanda: Transfer Architecture

nospam@example.com (Dustin J. Mitchell) — Fri, 12 Mar 2010 14:06:47 -0600

Amanda's primary mission in life is to move large quantities of data around. Historically, this has been done through a patchwork of methods, each written separately and with its own quirks. POSIX pipes, TCP sockets, shared memory, on-disk cache files -- Amanda's done it all. But these multiple implementations were error-prone, difficult to maintain, and often not the most efficient approach.

In an effort to remedy this, we introduced the transfer architecture, abbreviated XFA. This was technically included in Amanda-2.6.1, but was only used by amvault. In the upcoming Amanda-3.1 release, however, the XFA is central to all recovery operations, and is used internally by the taper (the portion of the backup system that writes to devices).

This post highlights some of the features of the transfer architecture, and some of the improvements we'd like to make.

Transfers and Elements

A transfer is pretty simple: it moves data from one place to another. It is built from a list of transfer elements, the first being the data source and the last the destination. Any elements in the middle are filters, and could apply compression or encryption, for example. Elements are automatically connected to one another using the most efficient means available.

Transfers operate asynchronously, meaning that ordinary execution of Amanda continues in parallel to the movement of data. When the transfer needs Amanda's attention, it sends a message, and Amanda reacts accordingly.

The beauty of this architecture is in the variety of elements that can be connected.

Sources
- File or socket
- DirectTCP Connection
- Holding Disk
- Spanned dumpfile
- Random or repeated patterns (for testing)
Filters
- In-process compression or encryption (e.g., libgz)
- External utilities (e.g., gzip or amgpgcrypt)
Destinations
- File or socket
- DirectTCP Connection
- Spanned Dumpfile

The full list is given in the POD.

Benefits

The advantage of the transfer architecture is that it massively simplifies the process of transferring data. Nowhere is this more obvious than in the splitting and re-joining of dumpfiles.

The Amanda::Xfer::Recovery::Source element, which reads from spanned dumpfiles, cooperates with the Clerk, via transfer messages, to load the proper volumes and seek to the proper files to recover and entire dumpfile, even if it is distributed over more than one source volume. Similarly, Amanda::Xfer::Taper::Dest works with a Scribe to load volumes and update the catalog while spanning dumpfiles.

So the Amanda taper is a simple wrapper around a transfer from holding disk (FILE-WRITE) or socket (PORT-WRITE) to a spanned dumpfile, using a scribe. Similarly, all of the recovery tools use a clerk and a transfer to read dumpfiles off the appropriate volumes. And amvault combines the two to simultaneously read from one volume and write to another.

Future

At this point, the transfer architecture is a reliable abstraction, but it is not yet terribly efficient. The advantage of the abstraction, though, is that as it is made more efficient, all of the components of Amanda that make use of it will immediately become faster, with no changes.

There are plenty of places in Amanda where the transfer architecture will be useful, and certainly plenty of work to do to make it faster. If you're interested in helping out, please let me know!

What's New in Amanda: Automated Tests

nospam@example.com (Dustin J. Mitchell) — Fri, 12 Mar 2010 12:03:52 -0600

This is the first in what will be a series of posts about recent work on Amanda. Amanda has a reputation as old and crusty -- not so! Hopefully this series will help to illustrate some of the new features we've completed, and what's coming up. I'll be cross-posting these on the Zmanda Team Blog too.

Among open-source applications, Amanda is known for being stable and highly reliable. To ensure that Amanda lives up to this reputation, we've constructed an automated testing framework (using Buildbot) that runs on every commit. I'll give some of the technical details after the jump, but I think the numbers speak for themselves. The latest release of Amanda (which will soon be 3.1.0) has 2936 tests!

These tests range from highly-focused unit tests, for example to ensure that all of Amanda's spellings of "true" are parsed correctly, all the way up to full integration: runs of amdump and the recovery applications. The tests are implemented with Perl's Test::More and Test::Harness. The result for the current trunk looks like this:

=setupcache.....................ok
Amanda_Archive..................ok
Amanda_Changer..................ok
Amanda_Changer_compat...........ok
Amanda_Changer_disk.............ok
Amanda_Changer_multi............ok
Amanda_Changer_ndmp.............ok
Amanda_Changer_null.............ok
Amanda_Changer_rait.............ok
Amanda_Changer_robot............ok
Amanda_Changer_single...........ok
Amanda_ClientService............ok
Amanda_Cmdline..................ok
Amanda_Config...................ok
Amanda_Curinfo..................ok
Amanda_DB_Catalog...............ok
Amanda_Debug....................ok
Amanda_Device...................ok
        211/428 skipped: various reasons
Amanda_Disklist.................ok
Amanda_Feature..................ok
Amanda_Header...................ok
Amanda_Holding..................ok
Amanda_IPC_Binary...............ok
Amanda_IPC_LineProtocol.........ok
Amanda_Logfile..................ok
Amanda_MainLoop.................ok
Amanda_NDMP.....................ok
Amanda_Process..................ok
Amanda_Recovery_Clerk...........ok
Amanda_Recovery_Planner.........ok
Amanda_Recovery_Scan............ok
Amanda_Report...................ok
Amanda_Tapelist.................ok
Amanda_Taper_Scan...............ok
Amanda_Taper_Scan_traditional...ok
Amanda_Taper_Scribe.............ok
Amanda_Util.....................ok
Amanda_Xfer.....................ok
amadmin.........................ok
amarchiver......................ok
amcheck.........................ok
amcheck-device..................ok
amcheckdump.....................ok
amdevcheck......................ok
amdump..........................ok
amfetchdump.....................ok
amgetconf.......................ok
amgtar..........................ok
amidxtaped......................ok
amlabel.........................ok
ampgsql.........................ok
        40/40 skipped: various reasons
amraw...........................ok
amreport........................ok
amrestore.......................ok
amrmtape........................ok
amservice.......................ok
amstatus........................ok
amtape..........................ok
amtapetype......................ok
bigint..........................ok
mock_mtx........................ok
noop............................ok
pp-scripts......................ok
taper...........................ok
All tests successful, 251 subtests skipped.
Files=64, Tests=2936, 429 wallclock secs (155.44 cusr + 31.48 csys = 186.92 CPU)

The skips are due to tests that require external resources - tape drives, database servers, etc. The first part of the list contains tests for almost all perl packages in the Amanda namespace. These are generally unit tests of the new Perl code, although some tests integrate several units due to limitations of the interfaces. The second half of the list is tests of Amanda command-line tools. These are integration tests, and ensure that all of the documented command-line options are present and working, and that the tool's behavior is correct. The integration tests are necessarily incomplete, as it's simply not possible to test every permutation of this highly flexible package.

The =setupcache test at the top is interesting: because most of the Amanda applications need some dumps to work against, we "cache" a few completed amdump runs using tar, and re-load them as needed during the subsequent tests. This speeds things up quite a bit, and also removes some variability from the tests (there are a lot of ways an amdump can go wrong!).

The entire test suite is run at least 54 times for every commit by Buildbot. We test on 42 different architectures - about a dozen linux distros, in both 32- and 64-bit varieties, plus Solaris 8 and 10, and Darwin-8.10.1 on both x86 and PowerPC. The remaining tests are for special configurations -- server-only, client-only, special runs on a system with several tape drives, and so on.

Testing Legacy Code

nospam@example.com (Dustin J. Mitchell) — Wed, 17 Feb 2010 12:26:15 -0600

I just read Roy Osherove's The Art of Unit Testing with Examples in .NET, on the advice of a slashdot review. I was not terribly impressed with the book, but reading it did help me to solidify my thinking about testing and test-driven development, and put words to concepts I had come to on my own.

Rather late in the book, Osherove describes three properties of good tests.

Trustworthiness - Do developers believe that passing tests mean things are working? Do developers believe that failing tests indicate a real bug?
Maintainability - Do developers think that tests are easy to add and maintain, or are they likely to avoid writing tests when rushed?
Readability - Do developers often consult the unit tests to see how the system under test is supposed to work?

What most struck me was that these properties were related to developers' perceptions of the tests, not the tests themselves. Tests are as much a social artifact of a project as a technical tool.

Buildbot's Tests

Around the time I was reading this, one of the more prolific Buildbot contributors commented, "I try not to change the tests - they scare me." Buildbot's tests were badly isolated, slow, and failed intermittently. As maintainer, I had grown accustomed to saying "oh, that test fails sometimes, don't worry about it" - a trustworithiness failure. Because of the terrible isolation, changing just about anything in Buildbot would cause dozens of tests to fail, requiring repetitive editing to fix - not maintainable. And the tests consisted of long sequences of operations and assertions, written in the Twisted style, which is already not readable. As a result, even I don't know what most of the tests are actually testing. This was a bad situation for any application, but particularly embarassing for a popular testing tool!

So I blew the tests away. Well, not really - I moved them to buildbot/broken_test/ in hopes they can be useful in writing new tests, and so that the braver souls among us can still run them. Now our metabuildbot is green, and I can legitimately ask for unit tests for new code.

There are costs associated with this move, too. A lot of people have worked very hard to write tests that have now been categorically labeled "broken," to whom all I can say is "I'm sorry". With far fewer tests and thus far worse coverage, it's also difficult to have confidence that Buildbot really works. The short-term workaround is to make a few beta releases and rely on real-world testing to suss out any problem.

So this is only the first step. We - I - still need to write real tests for the vast majority of the Buildbot code. That's particularly complicated because Buildbot's units are badly isolated, and interfaces are ill-defined. I will need to do a good bit of refactoring to bring it into compliance.

Object identity in Perl

nospam@example.com (Dustin J. Mitchell) — Fri, 22 Jan 2010 11:02:00 -0600

I ran across a surprising weakness in Perl, regarding object identity. I was writing a function to handle XMsgs, which are messages used by the Amanda transfer architecture to indicate the progress of a transfer. Messages sent from any transfer element are delivered to the same handler, and in this case I needed to know which element had sent the message. Fortunately, XMsg objects have a elt attribute pointing to the sending element.

Now, in Python, I would write,

  def handle_xmsg(xmsg):
    if xmsg.elt is interesting_elt:
      # handle message from interesting_elt
    else:
      # handle message from other elements

where the is operator tests for object identity. So how do I do this in perl? My first thought was to use eq, since the string interpolation of a hashref seems to have an object identifier in it: HASH(0x711210). I soon learned, in #perl, that == would work better, as the hashref's address is used as its integer value, and this conversion is much faster than stringification.

However, the element objects in this particular case are SWIG objects, and furthermore they use overload to implement a user-readable stringification. So neither technique worked.

What I did end up doing was this:

I overrode the == operator, and wrote a short comparison function in C to compare the un-SWIGged, un-magicked, un-blessed object pointers. Whew!