Code V.igoro.us

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case.

Continue reading "IPv6 and Amanda"

I just upgraded my Macbook to Snow Leopard, and the upgrade has changed the way SSH authentication works. I have set up a system I like quite a bit, now, and thought I would share.

Continue reading "SSH With Snow Leopard"

Most of my posts in this series have been about features that are available in a released version of Amanda. This time, I want to share a project I'm working on right now - one that will be available in Amanda-3.2. I'm reworking the way Amanda writes its data to tape (or any other kind of storage) to make it more efficient, more reliable, and simpler to configure.

Historically, Amanda's conservative approach to finicky tape hardware has meant that it wasted some space at the end of each tape. With the changes I'm working on, Amanda will no longer waste this space, and can also avoid some needless copying of data in most cases, with a minimum of additional risk.

Continue reading "What's New in Amanda: The End of Fragmentation"

It's been a while since I've posted about recent development in Amanda, but it's not for lack of interesting topics!

Today I want to talk a little bit about Amanda's development. Historically, Amanda has always had a small, core group of developers who do the lion's share of the development work. There are probably lots of reasons for this, not least of which is that a backup application isn't the sexiest project on which to spend your spare time. But I think there's a deeper reason, and it has to do with hackability.

Continue reading "What's New in Amanda: Hackability"

I've been meaning to get IPv6 set up on my local network for some time. My only practical reason is that Amanda supports IPv6 and I should test that support. It was also a good chance to re-immerse myself in network configuration, and Hurricane Electric has a neat certification process to add some motivation.

Continue reading "IPv6 Configuration"

It is incredibly easy to throw things together with an Arduino. It's common to see criticism of the device when it's used in in projects that don't require even a fraction of its power, and that might be justified. As a flexible platform for test-driving complex modules, though, the Arduino hits the mark perfectly on flexibility and usability.

I don't have a particular project in mind for my Arduino, but since I don't have a multimeter or an oscilloscope, I want to use it as a test harness for various basic components as I experiment with them. To this end, I bought a cute 16x2 character amber LCD display. With thoughts of building a fermentation-temperature monitor and learning about the I²C bus, I also bought a relatively cheap TMP102 and breakout board. With a little bit of reading, I was able to stitch them together quickly and easily.

Continue reading "LCD Display and TMP102 sensor"

I've been thinking a lot lately about the way we accomplish multiprocessing. We've seen a significant change in the operation of Moore's law for CPU speeds: today's CPUs are about the same speed as those of a few years ago, but they have more cores, and more virtual processors on those cores. This is great for heavily-loaded servers, which have plenty of distinct tasks to place on those cores and VCPUs, but not so useful for users working with single-threaded applications.

Why are most applications still single-threaded? There are lots of good reasons. Threaded code is harder to write, and not just because it requires careful analysis and use of synchronization primitives: many common tasks are difficult to meaningfully parallelize without careful control over inter-thread communication, and in a portable application you don't have that kind of control. Threaded code generally performs badly on single-CPU systems, which are still common. Some popular languages still make threading difficult, at least in a portable fashion. And threads are still relatively heavyweight entities in most operating systems: you don't spawn ten threads to mergesort a 100-item array.

Some of these problems will go away with a little more time, but some will get worse. NUMA architectures can make sharing data between threads slow. Hyperthreading and its interaction with processor caches adds yet another level of unpredictability.

We know how to build massively parallel systems that run massively parallel algorithms. What is still unknown is how to build portable, simple software that can run efficiently across a vareity of architectures. This is a problem of practice, not theory, and there's lots of interesting work going on in this area.

Of course, there are languages designed explicitly to support communication, such as Limbo or Erlang, Haskell, and Clojure. For the most part, these languages are structured as communicating sequential processes, which is to say that they represent multiprocessing as a set of sequential threads that pass information to one another. Problems of thread safety are subsumed by the languages, but mapping the parallelism to available resources is generally left to the programmer or administrator.

One interesting project is Apple's Grand Central Dispatch. It defines a simple but highly expressive closure syntax (a block) and a mechanism to dynamically schedule execution of such closures (queues). Critically, the GCD library takes care of scaling the parallelism of the queue processing appropriately to the underlying hardware. On a single-threaded CPU, this amounts to cooperative multitasking, but on parallel hardware the operating system can dynamically allocate virtual CPUs to applications needing more parallelism.

This topic seems to come up often in my various pursuits, so I will return to it again.

I've not made any secret of the fact that I want more people hacking on Amanda. This is both for selfish reasons -- many hands make light work -- and for altruistic reasons -- a broader community of developers can provide better governance for the project and long-term continuity. With a few noticable exceptions, I haven't had a lot of satisfaction.

I think part of the reason is that Amanda has a steep learning curve, even within the new Perl code. The time to climb that curve is a big investment, and folks with only a small itch to scratch can't afford it.

In an effort to sweeten the pot, we (Zmanda) are offering to pay for flexible work on Amanda. Part-time or full-time, on your own schedule. Your choice of projects. Support and gratitude from the other hackers. And the option to become a full Zmanda employee if that's your bent.

Here are some possible projects, to pique your interest:

• MySQL application (to round out the set with ampgsql)
• Cyrus Imapd application (gnutar doesn't deal well with the application's tiny files and hard links)
• OpenSSL for network transport, using certificates and keys for authentication
• Database-backed backup catalog
• Amvault upgrade
• Handle Logical EOM (LEOM) on all devices that support it, drastically reducing the number of parts Amanda writes
• Support for more cloud backends than just S3
• Parallel writes to multiple devices

If you're interested, contact me (dustin@zmanda.com) and we'll work something out!

In the second installment a series of posts about recent work on Amanda.

The Application API allows Amanda to back up "structured" data -- data that cannot be handled well by 'dump' or 'tar'. Most databases fall into this category, and with the 3.1 release, Amanda ships with ampgsql, which supports backing up Postgres databases using the software's point-in-time recovery mechanism.

The how-to for this application is on the Amanda wiki.

Continue reading "What's New in Amanda: Postgres Backups"

Yesterday, March 24, was Ada Lovelace Day. I was at Pumping Station: One, and decided to spend an hour or so writing something to honor the first computer programmer. I was feeling singularly uninspired, and googling for "Ada Lovelace" didn't turn up anything interesting. But it did give me an idea: write a program that googles for you!

Continue reading "Happy Ada Lovelace Day"

I'll be giving a presentation about Twisted Python programming tomorrow at Pumping Station: One in Chicago. If you're in the area, you should come by!

• 8pm (doors at 7) at 3354 N Elston
• Advance copy of the Presentation
• Video (available after the talk is complete)

I don't write about it here, but I've been getting into brewing beer. I downloaded an app for my iPhone, iBrewMaster, which helps me store recipes and track batches of homebrew through the brewing, fermeting, and serving stages.

I recently decided to make a clone of Dogfish Head's Raison D'être. This beer is fantastic, but that's beside the point. I added the recipe to the app, and clicked save. In the menu, however, I saw "Raison D'√™tre". Not pretty. The app has a feature where you create a "batch" from a particular recipe. I did so, and the name of the batch appeared as "Raison D'‚àö‚Ñ¢tre". Even worse!

Continue reading "Solving an Encoding Mystery"

Amanda's primary mission in life is to move large quantities of data around. Historically, this has been done through a patchwork of methods, each written separately and with its own quirks. POSIX pipes, TCP sockets, shared memory, on-disk cache files -- Amanda's done it all. But these multiple implementations were error-prone, difficult to maintain, and often not the most efficient approach.

In an effort to remedy this, we introduced the transfer architecture, abbreviated XFA. This was technically included in Amanda-2.6.1, but was only used by amvault. In the upcoming Amanda-3.1 release, however, the XFA is central to all recovery operations, and is used internally by the taper (the portion of the backup system that writes to devices).

This post highlights some of the features of the transfer architecture, and some of the improvements we'd like to make.

Continue reading "What's New in Amanda: Transfer Architecture"

This is the first in what will be a series of posts about recent work on Amanda. Amanda has a reputation as old and crusty -- not so! Hopefully this series will help to illustrate some of the new features we've completed, and what's coming up. I'll be cross-posting these on the Zmanda Team Blog too.

Among open-source applications, Amanda is known for being stable and highly reliable. To ensure that Amanda lives up to this reputation, we've constructed an automated testing framework (using Buildbot) that runs on every commit. I'll give some of the technical details after the jump, but I think the numbers speak for themselves. The latest release of Amanda (which will soon be 3.1.0) has 2936 tests!

These tests range from highly-focused unit tests, for example to ensure that all of Amanda's spellings of "true" are parsed correctly, all the way up to full integration: runs of amdump and the recovery applications.

Continue reading "What's New in Amanda: Automated Tests"

I just read Roy Osherove's The Art of Unit Testing with Examples in .NET, on the advice of a slashdot review. I was not terribly impressed with the book, but reading it did help me to solidify my thinking about testing and test-driven development, and put words to concepts I had come to on my own.

Rather late in the book, Osherove describes three properties of good tests.

• Trustworthiness - Do developers believe that passing tests mean things are working? Do developers believe that failing tests indicate a real bug?
• Maintainability - Do developers think that tests are easy to add and maintain, or are they likely to avoid writing tests when rushed?
• Readability - Do developers often consult the unit tests to see how the system under test is supposed to work?

What most struck me was that these properties were related to developers' perceptions of the tests, not the tests themselves. Tests are as much a social artifact of a project as a technical tool.

Buildbot's Tests

Around the time I was reading this, one of the more prolific Buildbot contributors commented, "I try not to change the tests - they scare me." Buildbot's tests were badly isolated, slow, and failed intermittently. As maintainer, I had grown accustomed to saying "oh, that test fails sometimes, don't worry about it" - a trustworithiness failure. Because of the terrible isolation, changing just about anything in Buildbot would cause dozens of tests to fail, requiring repetitive editing to fix - not maintainable. And the tests consisted of long sequences of operations and assertions, written in the Twisted style, which is already not readable. As a result, even I don't know what most of the tests are actually testing. This was a bad situation for any application, but particularly embarassing for a popular testing tool!

So I blew the tests away. Well, not really - I moved them to buildbot/broken_test/ in hopes they can be useful in writing new tests, and so that the braver souls among us can still run them. Now our metabuildbot is green, and I can legitimately ask for unit tests for new code.

There are costs associated with this move, too. A lot of people have worked very hard to write tests that have now been categorically labeled "broken," to whom all I can say is "I'm sorry". With far fewer tests and thus far worse coverage, it's also difficult to have confidence that Buildbot really works. The short-term workaround is to make a few beta releases and rely on real-world testing to suss out any problem.

So this is only the first step. We - I - still need to write real tests for the vast majority of the Buildbot code. That's particularly complicated because Buildbot's units are badly isolated, and interfaces are ill-defined. I will need to do a good bit of refactoring to bring it into compliance.