Code V.igoro.us

Amanda's primary mission in life is to move large quantities of data around. Historically, this has been done through a patchwork of methods, each written separately and with its own quirks. POSIX pipes, TCP sockets, shared memory, on-disk cache files -- Amanda's done it all. But these multiple implementations were error-prone, difficult to maintain, and often not the most efficient approach.

In an effort to remedy this, we introduced the transfer architecture, abbreviated XFA. This was technically included in Amanda-2.6.1, but was only used by amvault. In the upcoming Amanda-3.1 release, however, the XFA is central to all recovery operations, and is used internally by the taper (the portion of the backup system that writes to devices).

This post highlights some of the features of the transfer architecture, and some of the improvements we'd like to make.

This is the first in what will be a series of posts about recent work on Amanda. Amanda has a reputation as old and crusty -- not so! Hopefully this series will help to illustrate some of the new features we've completed, and what's coming up. I'll be cross-posting these on the Zmanda Team Blog too.

Among open-source applications, Amanda is known for being stable and highly reliable. To ensure that Amanda lives up to this reputation, we've constructed an automated testing framework (using Buildbot) that runs on every commit. I'll give some of the technical details after the jump, but I think the numbers speak for themselves. The latest release of Amanda (which will soon be 3.1.0) has 2936 tests!

These tests range from highly-focused unit tests, for example to ensure that all of Amanda's spellings of "true" are parsed correctly, all the way up to full integration: runs of amdump and the recovery applications.

I just read Roy Osherove's The Art of Unit Testing with Examples in .NET, on the advice of a slashdot review. I was not terribly impressed with the book, but reading it did help me to solidify my thinking about testing and test-driven development, and put words to concepts I had come to on my own.

Rather late in the book, Osherove describes three properties of good tests.

• Trustworthiness - Do developers believe that passing tests mean things are working? Do developers believe that failing tests indicate a real bug?
• Maintainability - Do developers think that tests are easy to add and maintain, or are they likely to avoid writing tests when rushed?
• Readability - Do developers often consult the unit tests to see how the system under test is supposed to work?

What most struck me was that these properties were related to developers' perceptions of the tests, not the tests themselves. Tests are as much a social artifact of a project as a technical tool.

Buildbot's Tests

Around the time I was reading this, one of the more prolific Buildbot contributors commented, "I try not to change the tests - they scare me." Buildbot's tests were badly isolated, slow, and failed intermittently. As maintainer, I had grown accustomed to saying "oh, that test fails sometimes, don't worry about it" - a trustworithiness failure. Because of the terrible isolation, changing just about anything in Buildbot would cause dozens of tests to fail, requiring repetitive editing to fix - not maintainable. And the tests consisted of long sequences of operations and assertions, written in the Twisted style, which is already not readable. As a result, even I don't know what most of the tests are actually testing. This was a bad situation for any application, but particularly embarassing for a popular testing tool!

So I blew the tests away. Well, not really - I moved them to buildbot/broken_test/ in hopes they can be useful in writing new tests, and so that the braver souls among us can still run them. Now our metabuildbot is green, and I can legitimately ask for unit tests for new code.

There are costs associated with this move, too. A lot of people have worked very hard to write tests that have now been categorically labeled "broken," to whom all I can say is "I'm sorry". With far fewer tests and thus far worse coverage, it's also difficult to have confidence that Buildbot really works. The short-term workaround is to make a few beta releases and rely on real-world testing to suss out any problem.

So this is only the first step. We - I - still need to write real tests for the vast majority of the Buildbot code. That's particularly complicated because Buildbot's units are badly isolated, and interfaces are ill-defined. I will need to do a good bit of refactoring to bring it into compliance.

Buildbot's WebStatus display has, for a long time, had an allowForce option which controls what kind of mayhem can be wrought via the web interface. Historically, this has been a boolean option: either web users can do everything (force builds and shut down slaves) or nothing. Bug 701 asks that we change that to give more granular access control.

Buildbot has an interesting way of separating the status display from the control functionality. It has two parallel interface hierarchies, IStatus and IControl, implementing the necessary methods. The IStatus hierarchy is illustrated with the orange bubbles here:

The IControl hierarchy is similar, although it only goes down to the Build level right now.

When allowForce is true, the WebStatus object adapts the buildmaster to the IControl interface and adds a link to the result in its control attribute. Forcing a build or shutting down a slave then uses this object to navigate to the appropriate control object and calls a method from the corresponding interface. If the control attribute is None, no access is allowed.

This scheme has the advantage that it is difficult to accidentally expose functionality, since when allowForce is false, the control methods are inaccessible. However, it has the disadvantage of not allowing any more granular level of access control.

I just reworked the web status to have a more flexible authorization mechanism, and while I wasn't able to remove the IControl hierarchy entirely, I was able to marginalize it to only those code blocks that need to perform controlled actions, instead of passing control objects all over the place.

Charlieplexing is a technique to allow n tristate digital output pins to control n(n-1) LEDs, by exploiting the high-impedence state of the output pins and the reverse-bias tolerance of LEDs.

I put together a circuit to test this out, similar to the 6-LED diagram in the Wikipedia entry. I added three 220Ω resistors, though. This ASCII art sums it up nicely:

      +--->|---+          +--->|---+          +--->|---+    
      |  LED1  |          |  LED3  |          |  LED5  |
 <----|        |-/\/\-+---|        |-/\/\-+---|        |-/\/\-+-->
      |  LED2  |      |   |  LED4  |      |   |  LED6  |      |   
      +---|<---+      R   b---|<---+      G   +---|<---+      B   

where R, G, and B are the three connections to the Arduino, and the arrows on the left and right side are joined by a jumper. I put together some simple Arduino-level code to drive this layout in a simple counter sequence, and fired it up.

One thing I will say about the Arduino: tinkering pays off. After the usual forgotten-semicolon cleanup, LEDs started blinking immediately. This is good, because I don't have much of an Arduino-debugging toolbox at this point!

Anyway, there's a problem. When an LED is fully illuminated, two other LEDs glow at about 50% brightness. For example, when LED1 is on, LED4 and LED6 glow too. It's not hard to see why: to turn LED1 on, the arduino drives R low and B high. The path through LED1 is intended, but LED4 and LED6 are also forward-biased in series, albeit with more resistance inline and the voltage-drop from two LEDs. Can this be prevented?

Let's work out the math here, using the resistance r as the only parameter I can change. The LEDs I'm using (5mm Green LEDs from CHINA YOUNG SUN) don't have a lot of performance data, but do list a forward voltage drop of 1.8-2.2V, so let's use 2V. My Arduino is running from the USB power at 5V. The resistor to the left of R defines the current through that leg for the remaining 5-2 = 3V, so I₁ is 3/r. On the leg containing LED4 and LED6 a total of 1V remains across two resistors, so I₄ and I₆ are both 1/2r.

The LED datasheet lists a maximum average current (20ma) and peak current (30ma), but doesn't list a minimum current. Guessing 1ma, I need 1/2r < 0.001, so r > 500. At that point, though, I₁ is just 6ma - not nearly bright enough to impress, particularly with a 16% duty cycle when multiplexing all 6 LEDs. All the same, I figured I'd try it. The closest resistor I had was 470Ω, so I gave that a shot. The result was bad on all fronts: the expected LED is dim, and the unwanted LEDs are still visible.

I suspect that, to get this to work, I'd need to reduce the supply voltage to something less than twice the LEDs' forward voltage drop. Then two LEDs in series would carry no current. I considered doing this with a simple voltage divider on each microcontroller pin, but this of course ruins the tri-state behavior that's critical to charlieplexing. I could build that by using two transistors for each pin, but by that point the benefits of charlieplexing in terms of component count are long lost.

So, in short, for good charlieplexing, make sure that your source voltage is less than twice the V_f of your LEDs.

[EDIT: it turns out, after some futzing that this also "works fine" if you just leave out the ballast resistors. Presumably this relies on a limited duty cycle and a "high enough" internal resistance to not melt the LEDs]

I have an early Mac Mini that's not good for much of anything anymore. It's not too fast, and it's a 32-bit PPC chip, so most proprietary-blob software is off-limits, meaning it doesn't make a good media PC. I put Gentoo on it, and decided to use it to interface with my new Arduino Duemilanove. I'm more interested in the Arduino as a processor I can play with, rather than a gateway to flashing LEDs, so I don't want the full IDE - just a compiler and assembler, and a USB-based programmer.

This post documents the process I followed to set all of this up.

I ran across a surprising weakness in Perl, regarding object identity. I was writing a function to handle XMsgs, which are messages used by the Amanda transfer architecture to indicate the progress of a transfer. Messages sent from any transfer element are delivered to the same handler, and in this case I needed to know which element had sent the message. Fortunately, XMsg objects have a elt attribute pointing to the sending element.

Svnmerge has always been the kid brother of Subversion. It's in the project's contrib directory, and is thus unversioned. Since it's a single Python script, users just download it directly from the repository. In the last year, the script has only seen a few non-trivial commits, and the mailing list has been almost silent.

There are some simple reasons for decline. First, Subversion itself, in version 1.5, has adopted some of the functionality of Svnmerge. In particular, svn now uses properties (svn:mergeinfo) to track the revisions that have and have not been merged into a particular branch. There are some limitations, of course. Most obviously, a branch can only be "reintegrated" into trunk once, which is not the workflow of many Svnmerge users. Also, to my knowledge, Subversion cannot merge between repositories, while Svnmerge can.

Second, Git, Mercurial, and other DVCS's now provide strong support for merge-heavy workflows. Both tools also have excellent "gateways" to Subversion. For example, Amanda's Github repository tracks the Subversion repository using some simple shell scripts.

Between these two forces, I think that much of the audience for Svnmerge has disappeared. Those left, sadly, will see even less support.

This contest recently came to my attention. I think I'll give it a shot. The challenge isn't too clear -- exactly what constitutes a "superseded" order is left pretty vague -- but there are lots of opportunities for underhandedness in the task.

I love the fact that GCC has basically eliminated the need to run lint, by adding every imaginable warning. But there are two problems.

First, generated code (at this point I'm being bitten by code generated by rpcgen for NDMJOB) seldom runs without warnings. Usually these are the "unused XXX" warnings, but some generated code is really and truly broken - K&R style definitions, signed/unsigned confusions, the works.

Second, warnings come and go between GCC versions, so just because it runs fine on my system does not mean it will run fine on all of the GCC's Amanda gets built on (never mind any other compilers!). Here's where I find myself trying to figure out how to disable a particular bogus warning for some ancient version of gcc, which is naturally not in the manuals now available on the web. And here's where my favorite GCC trick is handy: -fdiagnostics-show-option will cause gcc to display the -W option that controls that particular warning. From there, you can google the warning, or just disable it if you've determined that it's spurious and not worth the code changes required to make it go away.

I thought I'd throw that up here so I don't forget about it again.

I stumbled upon a summary of the Drefyus model of skill acquisition by Andrew Hunt today, copied out of the powerpoint here. There's a bullet point in there that I hadn't seen before:

1. Level 1: Beginner
   • Little or no previous experience
   • Doesn't want to learn: wants to accomplish a goal
   • No discretionary judgement
   • Rigid adherence to rules

I've been toying with the idea of a beginner's guide to Amanda for some time. My vision was of a shallow but broad description of how Amanda fits together: clients and servers, applications, holding disks, changers, devices, and so on. Scaffolding, in the constructivist sense. The intent was to head off some of the more ignorant questions about Amanda, and give beginners a base on which to build their own Amanda configurations.

But this won't work! A beginner does not want to learn, and anything that tries to "teach" a beginner is just so much noise to him or her. In fact, most beginner guides are nothing of the sort -- they are probably only useful to those already at level 2.

This is all the more the case with Amanda: like plumbing, Amanda users just want backups to happen so they can concentrate on their core business, so even smart folks will happily remain at level 1 if they can make it work. So what can we do to support these users? And how can we encourage people to move to the next level?

(from a comment on Zen and the Art of Nonprofit Technology)

I've realized recently that open-source projects have developed a class structure. First, consider the broad middle-class of little-known but functional projects that have a number of users and several developers (Amanda, Buildbot, Cfengine, Sphinx, Curl, the autoconf archive, maybe GMT). Above these projects are the superprojects that nearly everyone uses and that have hundreds of contributors (Firefox, Linux, Python, Perl, to name a few). These are all islands in a squalid sea of abandonware and single-developer projects -- we've all got one or two, right?

What are the keys to mobility between these classes? What factors might keep an otherwise-deserving project in the lower or middle classes?

Now that I'm using git all the time, it's helpful to have bash show me what branch I'm on, and whether the working directory is dirty. I found a gist by Henrik Nyh that does just this. I souped it up a little bit to be more efficient and handle a few extra situations. Here's the result: [EDIT: I have updated this gist since this post was made; the current version is based on git-completion.sh in the git distribution]

Please use it if you like, and fork the gist if you make any improvements!

Several Amanda developers use git internally, but the Amanda source code is in Subversion at SourceForge. Git-svn manages bidirectional mirroring for us, and works flawlessly.

Recently, however, we introduced a problem through user error: due to a typo in our git-authors file, a bunch of revisions were mirrored with incorrect author information. This was more than a cosmetic problem because it caused the SHA1 hashes to differ between developers who fixed the typo at different times (because the author information is included in the data that feeds the hash algorithm). This would leave us with divergent commits forever.

The challenge, then, is to make git-svn think that it never fetched that revision. HEAD was at r1413, but r1391 had the bad author. We branched for release right after the bum commit, so there are two branches to deal with. Here's what I did:

First, roll back the remote branches (e945b67d78c239b42cb882e5c28e24354d0c05f0 is r1390)

$ git update-ref -m "roll back git-svn" ext/trunk e945b67d78c239b42cb882e5c28e24354d0c05f0
$ git update-ref -m "roll back git-svn" -d ext/amanda-261 cad126843a7649ca3e05088dd46ee41d7f17e7e2

Next, edit both maxRev values in git-svn's metadata (.git/svn/.metadata):

[svn-remote "ext"]
    reposRoot = https://amanda.svn.sourceforge.net/svnroot/amanda
    uuid = a8d146d6-cc15-0410-8900-af154a0219e0
    branches-maxRev = 1413
    tags-maxRev = 1413

Finally, delete the .reflog

$ rm .git/svn/ext/trunk/.rev_map*
$ rm .git/svn/ext/amanda-261/.rev_map*

Then just re-run the fetch:

$ git svn fetch ext

I'm watching the DNC's Rules & Bylaws Committee meeting, discussing the fate of the Michigan and Florida delegations. This committee consists of the elite among the Democratic superdelegates. These are all powerful, intelligent, important people (including the wonderful Donna Brazile).

What has me upset is that I've now heard two women explain that they don't have the "mathematical genius" to work out the proper representation for these states; no men have made such a declaration. First of all, that's bull: anyone can work out what 33% of 128 is. The difficult part is in the politics, and everyone on this committee is an expert on politics.

More importantly, though: why is it OK for these women to brag about their mathematical ignorance? What message does this send to PoliSci students struggling through calculus? One of the women to mention this was Alice Travis Germond, who was a VP of NARAL -- hardly an advocate of keeping women barefoot, pregnant, and in the kitchen. What is she thinking?

[UPDATE] The other woman to brag about her innumeracy was Tina Flournoy -- Gore's finance director! Finance!