Code V.igoro.us

Oops, I partitioned my drive..

nospam@example.com (Dustin J. Mitchell) — Wed, 29 May 2013 18:48:26 +0000

I did something colossally stupid yesterday. I was at the local hackerspace, hoping to cut some acrylic, and the wifi wasn't working. I was in a hurry and frustrated, so I pulled out a USB stick and tried to erase it. Suffice to say, the USB stick wasn't at /dev/sda. I wiped out the GPT on my laptop. Its disk is encrypted, so the buffer store kept things working for a while, then suddenly I had a blinking root prompt and .. nothing.

After the obligatory cold-sweat had passed, I quietly packed up and walked out. Here's the story of how I recovered from this, with the help of Jake Watkins (:dividehex). The system didn't have any irrecoverable data on it. All of my work is in version-control, and I'm religious about pushing to github. Even my home directory's dotfiles are on github (in a private repo). But I spent several weeks setting this laptop up (Linux on the desktop is only usable from a very long-term perspective!), and unfortunately one of the few things that wasn't in version control was my notes on how I'd done so. Rebuilding this from scratch would take a few weeks and make a very grumpy Dustin.

To make things interesting, this ThinkPad X1 Carbon has no Ethernet port. It's wifi only. There are dongles available from Lenovo, but I don't have one.

Backup

OK, so let's stop the bleeding.

I booted from the Fedora Live USB stick I'd used to install the system initially. Once it was up, I schlepped the entire drive over to my desktop:

ssh root@ramanujan dd if=/dev/sda > laptop-disk.img

That took about 8 hours, but that gave me time to do some other research, conduct a Baltic Porter tasting (multitasking!), and sleep.

Find the Data

So the situation is that I have a partition table with a single 256G FAT32 partition in it. Somewhere on the disk, my data is probably still intact. But where?

Jake suggested looking for the GPT backup table, which is at the end of the disk. The gdisk advanced options allow you to examine this table, but it too was empty.

Jake's other suggestion was to look for the signature of the LUKS crypto container.

dd if=/dev/sda | od -c | grep 'L   U   K   S'

This eventually turned up the beginning of the partition, 05364000000 (decimal 735051776) bytes into the disk. Dividing that by 512, the size of a sector, that's sector 1435648.

In gdisk, I created a partition beginning at that location and running to the end of the disk.

# cryptsetup luksDump /dev/sda1
LUKS header information for /dev/sda1

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4096
MK bits:        512
MK digest:      e2 f9 fa 7d 44 dc 82 f5 3c 39 1d e2 ac 6e e3 d2 d0 f5 2b 1b 
MK salt:        05 8b a0 ee 7a bf 37 73 7b 15 c1 7a 41 0b 46 19 
                56 68 e1 36 f7 7c b3 d0 74 bf 21 98 a1 e6 75 7b 
MK iterations:  13750
UUID:           4057691b-b580-4d05-afd3-9e2efe8d65d3
...

so it looks like I've found the data.

# cryptsetup luksOpen /dev/sda1 secret
  
# mount /dev/mapper/secret /mnt

This didn't work. It's not an ext4 volume - it's an LVM physical volume. Udev has already run the pvscan, so I just need to activate the volume group:

# vgscan -a y fedora
# mount /dev/fedora/root /mnt
# mount /dev/fedora/home /mnt/home

At this point, it's time for another backup. I used tar to dump the contents of both partitions, storing the tarball on my desktop machine. This took about an hour. Now I feel a lot better -- I have my notes from customizing the system, and anything else that's not backed up that I might have forgotten. But I haven't actually lost any data - can I recover this completely?

Rebuild It

I reboot the Live USB stick, just to reset all of the partition maps, luks opens, etc. that I had done. I realize that /dev/sda1 is probably not the right partition number for the full filesystem, so I move that to /dev/sda3 (by deleting and re-creating with the same first/last sectors). Another reboot.

When it starts up again, I fire up the installer and tried to install around the existing data. I figure I can do a minimal installation in some of the free space, particularly since Anaconda reassured me that it only needed 3.1G.

Long story short, this was a lie. Anaconda is a horrible piece of software, with bugs galore. For example, if you set a partition's size, it will randomly make up another size and use that instead. Ugh.

It turns out, Anaconda needs about 15G to do the install. So, I need to shrink /dev/sda3 by at least that much. I resize logical volumes all the time, but this is the reverse -- I need to shrink a physical volume. That turns out to be a bit tricky. First, after opening the LUKS volume, I run pvresize:

pvresize --setphysicalvolumesize 210g /dev/mapper/secret

This fails, telling me that there are extents beyond the new last extent. Subtract a few from that last extent and call that $boundary. Run pvdisplay /dev/mapper/secret to get the total physical extents, and call that $total. Then, use pvmove to shift those around:

pvmove --alloc anywhere /dev/mapper/secret:$boundary-$total /dev/mapper/secret:0-$boundary

and I re-run the pvresize command.

Now, I resize the LUKS container that PV is housed in. This requires a little arithmetic. LVM uses "GiB", which are the normal power-of-two things, not the stupid disk-manufacturer power-of-ten things. So 210g is 225485783040 bytes, which (divided by 512) is 440401920 sectors.

cryptsetup resize --size 440401920 /dev/mapper/secret

OK, I shrank the PV, I shrank the enclosing LUKS container, and now I need to shrink the partition. This means deleting and re-adding the partition in gdisk. The starting sector stays the same, while the last sector is calculated as $starting_sector + 440401920 - 1. The -1 is important there - I'm specifying the lastsector, not the first sector of the next partition. That'd be a sad way to lose 512 bytes of data when I least expect it. The filesystem type doesn't seem to matter - I used the default, but I noticed that Anaconda uses 0700.

Reboot and run the install again. Let Anaconda do its thing, because it's not going to listen to me if I try to customize it. Anaconda happened to choose a different VG name (fedora_ramanujan) for the new volume group. If it hadn't, I might have had to rename things. Reboot, and I have a fresh system.

If I was smart, I would have run 'yum upgrade' now, but I didn't. More on that in a second.

The idea is that I now have UEFI set up, and all of the proper boot partitions, but it's running from the wrong encrypted partition. Changing that is as simple as editing /boot/efi/EFI/fedora/grub.cfg, replacing the new VG name with the old VG name, and the new LUKS UUID with the old LUKS UUID (from 'cryptsetup dumpLuks'). Reboot, enter my passphrase, and .. "Welcome to emergency mode!". A bit of poking around shows that all of my partitions are mounted properly, but the journal shows failures mounting /boot/efi, because of a missing vfat module.

Kernel Woes

The kernel and initrd are, of course, on /boot, and are the version installed from the Live USB stick. The kernel modules are on /, and are the updated versions from several months later. They don't match up.

The quick fix is to mount the new LUKS container and copy /lib/modules over /lib/modules on the old container. This gets me a copy of all of the modules that Anaconda had installed, and lets me boot all the way into a working system .. almost. The wifi isn't working.

This is probably because I'm running an old kernel version with an otherwise-updated system. Upgrading to the latest kernel is the fix. That's tricky with no wifi, so I look up the packages on my desktop, throw them on a USB stick (being very careful not to re-partition my desktop's HDD in the process!), and run

yum reinstall /mnt/usb/kernel-*

on the laptop once the USB stick is mounted there. Reboot, and all is back to normal.

That was not the most fun I've ever had in 20 hours. But it was an adventure.

910 Days at Mozilla

nospam@example.com (Dustin J. Mitchell) — Mon, 15 Apr 2013 18:26:34 +0000

As of today, I've been at Mozilla for 910 days. That's not a magic number, but this seemed like a good day to reflect on my time here.

I've had a chance to do a bunch of exciting things here:

Drink from the Mozilla Firehose
Manage build slaves in the release engineering environment
Build out a configuration management system with Puppet
Design systems to build out new hardware platforms and operating systems
Organize a move of systems and servers out of one datacenter and into another.
Build a web cluster
Build and maintain MySQL database clusters as an apprentice DBA
Learn Ruby and hack on Puppet
Build a dynamic hardware provisioning system (Mozpool)

You'll never be bored at Mozilla! There's never a shortage of work to do, with new projects coming all the time. The organization is structured so that it's easy to take on tasks that need doing, whether they're within your skill base or not. There's lots of room to learn, and everyone's happy to teach.

I work with an incredible group of people. Just within IT, we have a huge range of skills and capabilities for a relatively small team. People who know how to really solve problems, not the half-baked temporary solutions that you find elsewhere. As but one example, the datacenter operations team is building out and operating several world-class datacenters at the same time, and still managing to turn around our remote-hands requests in matters of hours. Our infrastructure team is full of people with deep experience in all aspects of system administration who are always willing to help solve a tricky problem. And on my own team, my co-workers all manage to work miracles far beyond the resources available.

Before Mozilla, I was at Zmanda, working on Amanda -- you know, the open-source backup application you remember from your early days? It's still around! Anyway, I took that job in part because it meant I could be paid to work on open-source software. I took full advantage of that opportunity, but the company was fundamentally a company - organized around sales, support, and the bottom line. My open-source concerns always played second fiddle, if that. Mozilla's different: the Mozilla Manifesto is what we do, and that's understood nowhere better than at the top of the organization. It can be a struggle sometimes, to see how the work I do supports the people who support the people who build the products that further the mission, but the connection is there and it's important. That keeps me going.

Here's to another 1000 successful days at Mozilla!

Locking SSH keys on sleep on Linux

nospam@example.com (Dustin J. Mitchell) — Fri, 29 Mar 2013 16:42:45 +0000

I got a new laptop, a ThinkPad X1 Carbon, and I'm running Linux on it. So you're in for a series of posts describing the complex process I had to follow to accomplish simple things. Spoiler alert: 2013 is not the year of Linux on the desktop. It's not looking good for 2014 either.

I'm running Fedora 18. I tried Ubuntu 12.10, but Unity couldn't hold itself together long enough to actually do anything, so I started over with Fedora.

SSH Agent

Gnome runs a nice keychain app that acts like (but is not) OpenSSH's ssh-agent. The one obvious place it differs is that ssh-add -l will list keys even if they are "locked" (passphrase not supplied).

As long as you point the SSHAUTHSOCK variable to the right place, the agent works just fine for unlocking keys - it finds any private/public pairs in ~/.ssh, and prompts to unlock them once you issue an SSH command that needs a key. The problem is, it never re-locks the keys.

Locking

Personally, I use SSH constantly while my laptop is awake, so I don't want an arbitrary timeout. Instead, I'm careful to put it to sleep when I'm away from the keyboard. So I want a way to lock the key on sleep.

It turns out that pm-utils will run scripts in /etc/pm/sleep.d on sleep and wake. It runs them as root, unfortunately. I added the following in 01dustin-ssh-agent.sh:

#!/bin/sh

# drop keys from dustin's SSH agent

. "${PM_FUNCTIONS}"

lock()
{
        su - dustin /home/dustin/bin/ssh-lock
}

case "$1" in
        hibernate|suspend) lock ;;
        *) exit $NA ;;
esac

and then added the following in ~/bin/ssh-lock:

#!/bin/sh

# drop keys from the SSH agent, using the same trick as bin/startscreen to find
# that agent

base="/tmp"
[ -d /run/user ] && base="/run/user/$(id -u)"
socket_dir="$base/$(uname -n)-$(id -u)"
SSH_AUTH_SOCK=$socket_dir/agent ssh-add -D

See my post on tunneling ssh-agent into a screen session for the reference to bin/startscreen. I'm not sure how best to accomplish this without such a trick. I'll work on that and post again.

Documentation for MDT's CustomSettings.ini

nospam@example.com (Dustin J. Mitchell) — Thu, 25 Oct 2012 15:59:39 +0000

If you're looking for info on CustomSettings.ini, you're most likely to find questions answered with "try this script". You type it in, and if it works, great; if not, keep looking. It's well-nigh impossible to find actual documentation, and the programming-by-INI-sections design is not exactly intuitive.

It turns out there's some in the help docs for the MDT, but those are a .CHM file and Microsoft apparently doesn't post those online.

However, some helpful (Russian?) souls have done so. Behold: Microsoft® Deployment Toolkit 2012 Toolkit Reference.

Google Dependency..

nospam@example.com (Dustin J. Mitchell) — Mon, 17 Sep 2012 02:00:14 +0000

I have a Google-branded Samsung Nexus, runny Jelly Bean. It's a decent phone, except that its GPS requires a small nuclear reactor to power it.

Today S and I drove back from a weekend trip to New Haven, CT. Google let us down pretty badly.

First, on the way out, I had used the navigation feature (which requires GPS, not just Google Location Service) for most of the day. I have a power adapter in the car, and the phone was plugged in the whole time. Still, after about 6 hours, the phone's battery was at about 10% -- having started at a full charge when we left. I forgot to plug it in where we spent the night, so it was dead in the morning. No problem -- I only need to navigate home, so I'll just plug it in!

And here we reach the first problem. This phone takes 4-6 minutes to start up. Which means either I key in my destination while already on the highway, or sit in the car for 4-6 minutes while my phone starts up. Bear in mind that during most of that startup time, it has a blank screen with no backlight, and sometimes startup crashes, so it's a bit of a psychological game to resist popping out the battery early.

Fine, so it starts up, I start the navigation application, and hit "Go Home". It plots a route, I take off, get on the highway, and the phone dies. At this point I do the math -- if my phone went from 100% to 10% in 6 hours while navigating, then it requires the combined power of both the battery and the adapter to run the navigation app. So I dutifully hand the phone to S to start back up and hit the "Go Home" button again. I immediately turn off the backlight and hope for the best. The navigation voice tells me to head South for 40 miles, and then I hear nothing.

This should have been a red flag, but I was too busy composing this blog entry in my head to do a little more arithmetic: Albany is not South of New Haven. A half-hour later, S checked her iPhone and pointed out we were heading the wrong direction.

I pulled over and checked my phone. Surprisingly, it was still on! However, it was navigating to Waters View, NY, which is not where we live. We live at Waters View Circle, Cohoes, NY. Google knows this. It's set in the navigation app. It turns out that Google takes the string you type in for your home address and hits the API equivalent of the "I'm feeling lucky" button, and you're off to the races. Perhaps not the races you were looking for. This is the same fundamental flaw as plagues Google Now. If you're in, say, Springfield, MA, but for whatever reason the location-aware "I'm feeling lucky" feels Springfield, IL is the more relevant search result, you get Springfield, IL's weather. And it helpfully just says "Springfield" on the card, so you don't know anything's wrong until it claims there's torrential rains on a clear day.

At this point we killed my phone and navigated the old-fashioned way - getting directions on the iPhone and following the relevant signs. It turns out we had a great drive along the Taconic State Parkway, which is a far sight more interesting than I-90, so that was OK. And we only lost an hour of drive-time.

Aside from the battery-life issues, which are not surprising from Samsung, but are surprising from a phone with "Google" on the back, there's an important point here: Google's approach to problems is to throw gargantuan amounts of data and CPU at them, and hope the answer's right. That's fine for search, but when it comes down to building a reliable personal device, people need something a little more deterministic. Google is increasingly heading toward personalized computing -- Google Glass being the ultimate expression -- and I think the company has a lot to learn before any of that will be more than an amusingly daft automaton.

Building a partitioned log table

nospam@example.com (Dustin J. Mitchell) — Thu, 13 Sep 2012 14:46:25 +0000

For a project at Mozilla that involves re-imaging hundreds of mobile devices, we want to gather logs in a database for failure analysis. Mobile devices fail all the time -- not sure if you knew that.

We'll probably end up with 1,000-10,000 log entries per day. We'd like to expire them on a relatively aggressive schedule -- no need for historical analysis at this level. So that means not only a lot of inserts, but a lot of deletes.

We're using MySQL as the database backend, and MySQL doesn't do well with deletes - it just marks the row as deleted, but doesn't reclaim the space, and in fact doesn't even remove the row from consideration in queries. So if you blindly insert and delete in a table, MySQL will eat disk space and get progressively slower.

One fix to this is to optimize the table periodically. However, this requires a full lock of the table for the duration of the optimize, which can be quite a while. We dont' want to cause a backup of production tasks while this is going on.

The other option is to partition the table. A partitioned table is basically a set of tables (partitions) with the same columns, organized to look like a single table. There's a partitioning function that determines in which partition a particular row belongs. There are a few advantages. Each partition is a fraction of the size of the whole table, so inserts are quicker (once the appropriate table is determined). The query engine can use "partition pruning" to ignore partitions that could not hold rows relevant to the query. Finally, dropping an entire partition at once is a very simple operation, and doesn't leave any garbage that needs to be optimized away.

For logs, we want to partition by time, in this case with one partition per day. Most of the "get the logs" queries will use a limited time range, invoking query pruning and allowing a quick response.

The tricky part is, the DB server does not automatically create and destroy partitions. We need to do that. It's pretty straightforward with stored procedures, though. Here's the resulting SQL to create the logs table:

DROP TABLE IF EXISTS logs;
CREATE TABLE logs (
    -- foreign key for the board
    board_id integer not null,
    ts timestamp not null,
    -- short string giving the origin of the message (syslog, api, etc.)
    source varchar(32) not null,
    -- the message itself
    message text not null,
    -- indices
    index board_id_idx (board_id),
    index ts_idx (ts)
);

--
-- automated log partition handling
--

DELIMITER $$

-- Procedure to initialize partitioning on the logs table
DROP PROCEDURE IF EXISTS init_log_partitions $$
CREATE PROCEDURE init_log_partitions(days_past INT, days_future INT)
BEGIN
    DECLARE newpart integer;
    SELECT UNIX_TIMESTAMP(NOW()) INTO newpart;
    SELECT newpart - (newpart % 86400) INTO newpart; -- round down to the previous whole day

    -- add partitions, with a single partition for the beginning of the current day, then
    -- let update_log_partitions take it from there
    SET @sql := CONCAT('ALTER TABLE logs PARTITION BY RANGE (UNIX_TIMESTAMP(ts)) ('
                        , 'PARTITION p'
                        , CAST(newpart as char(16))
                        , ' VALUES LESS THAN ('
                        , CAST(newpart as char(16))
                        , '));');
    PREPARE stmt FROM @sql;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;

    -- do an initial update to get things synchronized
    call update_log_partitions(days_past, days_future);
END $$

-- Procedure to delete old partitions and create new ones around the current date
DROP PROCEDURE IF EXISTS update_log_partitions $$
CREATE PROCEDURE update_log_partitions(days_past INT, days_future INT)
BEGIN
    DECLARE part integer;
    DECLARE newpart integer;
    DECLARE earliest integer;
    DECLARE latest integer;

    -- add new partitions; keep adding a partition for a new day until we reach latest
    SELECT UNIX_TIMESTAMP(NOW()) + 86400 * (days_future+1) INTO latest;
    createloop: LOOP
        -- Get the newest partition (PARTITION_DESCRIPTION is the number from VALUES LESS THAN)
        -- partitions are named similarly, with a 'p' prefix
        SELECT MAX(PARTITION_DESCRIPTION) INTO part
            FROM INFORMATION_SCHEMA.PARTITIONS
            WHERE TABLE_NAME='logs'
            AND TABLE_SCHEMA='imagingservice';
        IF part < latest THEN -- note part cannot be NULL, as there must be at least one partition
            SELECT part + 86400 INTO newpart;
            SET @sql := CONCAT('ALTER TABLE logs ADD PARTITION ( PARTITION p'
                                , CAST(newpart as char(16))
                                , ' VALUES LESS THAN ('
                                , CAST(newpart as char(16))
                                , '));');
            PREPARE stmt FROM @sql;
            EXECUTE stmt;
            DEALLOCATE PREPARE stmt;
        ELSE
            LEAVE createloop;
        END IF;
    END LOOP;

    -- now, deal with pruning old partitions; select the minimum partition
    -- and delete it if it's too old
    SELECT UNIX_TIMESTAMP(NOW()) - 86400 * (days_past+1) INTO earliest;
    purgeloop: LOOP
        -- Get the oldest partition
        SELECT MIN(PARTITION_DESCRIPTION) INTO part
            FROM INFORMATION_SCHEMA.PARTITIONS
            WHERE TABLE_NAME='logs'
            AND TABLE_SCHEMA='imagingservice';
        IF part < earliest THEN
            SET @sql := CONCAT('ALTER TABLE logs DROP PARTITION p'
                                , CAST(part as char(16))
                                , ';');
            PREPARE stmt FROM @sql;
            EXECUTE stmt;
            DEALLOCATE PREPARE stmt;
        ELSE
            LEAVE purgeloop;
        END IF;
    END LOOP;
END $$

DELIMITER ;

-- initialize the partitioning
CALL init_log_partitions(14, 1);

-- and then update every day (this can't be set up in init_log_partitions)
DROP EVENT IF EXISTS update_log_partitions;
CREATE EVENT update_log_partitions  ON SCHEDULE EVERY 1 day
DO CALL update_log_partitions(14, 1);

A few notes here. First, the table is created without any partitions. This is because I don't know a priori which partitions it should have, and it's easier to get code to figure that out than do it myself. That's what the initlogpartitions function does. The updatelogpartitions function looks at the current time and makes sure there are enough partitions for the future, and drops partitions too far in the past. Finally, a MySQL event is set up to update the partitions daily.

You'll need to enable the event scheduler globally to get this to run:

set global event_scheduler=on;

Mobile data while travelling in Great Britain

nospam@example.com (Dustin J. Mitchell) — Tue, 12 Jun 2012 23:29:13 +0000

Three wireless was recommended for my trip here, but they were out of SIMs (which seems a common malady), so I dropped by the T-Mobile store. They set me up with a £5 SIM with unlimited data for 30 days, which seemed great. But read on.

As I walked out the door of the shop, I went to look up bars in Edinburgh, and was presented with a "prove you're 18 with a credit card" form, courtesy T-Mobile. Of course, I don't have a UK credit card, so I turned around and walked back into the shop. They fixed this up without any additional trouble -- but there would have been more trouble had I not immediately looked for something not "child-safe" (Which brings up the recurring question: what on earth have legislators been smoking to think that children should not be able to visit breweries' and bars' websites? Do they think that the "series of tubes" can, in the hands of n'er-do-wells, carry grain alcohol straight to the mouths of babes?).

I subsequently noticed that they block IRC - not such a big deal, since they do not block SSH, which is how I usually use IRC.

And now that I'm on my laptop and not my mobile, I notice that they are also injecting their JavaScript into every (non-SSL) page I visit, helpfully shrinking images and adding keyboard shortcuts to un-shrink them. And probably capturing passwords, too. Lovely. I'm sure they're doing so to "protect" me.

For the moment, I'm really only using the data to figure out where to go next, and what to drink when I get there, and only for a few days. Even so, this intrusion into my privacy galls me.

Steer clear of T-Mobile, if you value your online life.

Wireless Data in Belgium

nospam@example.com (Dustin J. Mitchell) — Thu, 07 Jun 2012 12:56:26 +0000

In hopes this will be useful to others:

I have an unlocked AT&T phone (Samsung Galaxy Nexus), and needed a data plan in Belgium (Brussels, specifically). I really don't intend to text or talk, but data is important. Most of the news kiosks are happy to sell you a SIM, but with no data portion.

A fellow Mozillian, Ben Kero, recommended BASE wireless. There was a spot in the airport, but they were out of SIMs. I borrowed some wifi, looked up another location on base.be, and went there.

They have a 1-GB plan for 15€, called "surf & mail 15". What you'll get is a 15€ SIM card with 15€ credit on it, and a brochure giving instructions to send a text message to activate the 15€ plan using that credit. When I did so, I immediately received a text indicating I did not have sufficient credit, but the data seems to work.

Which is good, because the wifi in the apartment we're staying at is pretty poor.

Trapped in Google?

nospam@example.com (Dustin J. Mitchell) — Sat, 19 May 2012 23:33:24 +0000

Whenever I search in Aurora on my phone, I'm taken to a stripped-down version of the page with the header "this page adapted for your browser".

How do I fix this? I'd rather fix it with a Google preference, but barring that I expect that Firefox has a way for me to regain control of my online experience?

TIL about SSL certificate chains

nospam@example.com (Dustin J. Mitchell) — Wed, 09 May 2012 20:43:01 +0000

I'm laying some SSL groundwork for a project to allow puppet clients to move between puppet servers without requiring a central CA, and without requiring each client to be aware of all masters. More on that in a future post.

Based on "Multiple Certificate Authorities", I would like to have certificate chains that look like this:

      +-puppetmaster1 CA--+-puppetmaster1 server cert
      |                   |
      |                   +-client 1 server cert
root--+                   :
      |                   
      +-puppetmaster2 CA--+-puppetmaster2 server cert
                          |
                          +-client 10 server cert
                          :

Then all of the certificate validation would be done with the root CA certificate as the trusted certificate. A server certificate signed by puppetmaster2's CA cert should then validate on puppetmaster1.

Building the certificates wasn't all that difficult - see my comment on the bug for the script. However, while making sure the verification worked, I ran into some non-obvious limitations of OpenSSL that are worth writing down.

I began by running "openssl verify":

[root@relabs-puptest1 ~]# openssl verify -verbose -CAfile puptest-certs/root-ca.crt -purpose sslclient puptest-certs/relabs08.build.mtv1.mozilla.com.crt 
puptest-certs/relabs08.build.mtv1.mozilla.com.crt: CN = relabs08.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
error 20 at 0 depth lookup:unable to get local issuer certificate

the problem here is that the intermediate certificate is not available to the verification tool. Sources suggest to include it with the server cert, by concatention, with the server cert last:

cat puptest-certs/relabs-puptest1.build.mtv1.mozilla.com-ca.crt puptest-certs/relabs08.build.mtv1.mozilla.com.crt > relabs08-with-intermed.crt

However, after some struggle I learned that "openssl verify" does not recognize this format -- it will only look at the first certificate in the file (the intermediate), and if you don't look carefully you'll find that it successfully verifies the intermediate, not the server certificate! Sadly, sclient and ssever don't support it either. Apache httpd supports it with SSLCACertificatePath. This will feed the certificate chain to the client, and also allow httpd to verify client certificates without requiring the clients to support an intermediate.

The Apache config is

Listen 1443

<VirtualHost *:1443>
        ServerName relabs-puptest1.build.mtv1.mozilla.com
        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.crt
        SSLCertificateKeyFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.key
        SSLCACertificatePath /etc/httpd/ca-path

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        #SSLCARevocationFile     /etc/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient require
        SSLVerifyDepth  2

</VirtualHost>

While you're getting that set up, you're probably wondering where to get this fancy "c_rehash" utility. Don't bother. It's about as simple as:

for i in *.crt; do
        h=`openssl x509 -hash -noout -in $i`
        rm -f $h.0
        ln -s $i $h.0
done

As a side-note, the results of verification by sclient and sserver are not very obvious. Look for the overall error message near the bottom of the output. Here's the result of a client verification once I had everything put together, with some long uselessness elided:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert puptest-certs/relabs08.build.mtv1.mozilla.com.crt -key puptest-certs/relabs08.build.mtv1.mozilla.com.key -pass pass:clientpass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 5379 bytes and written 1716 bytes
---
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: E30634D9CFCC2FA327282DA813BB550C24ACDF18194E5F13C4981AA55914B5F0
    Session-ID-ctx: 
    Master-Key: 013EB09B066418694D36D74B414BBA42E52DBF0066314B60FC7A74662A60934282B6C37C5C82026F70287E60F4FF9472
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket:
    0000 - 82 5f 17 72 97 bd f3 1e-ec 24 de 69 ab 1e cd 1d   ._.r.....$.i....
    ....
    0520 - 40 05 b3 27 20 00 8d ce-93 a9 48 81 8f 0c 16 5b   @..' .....H....[

    Compression: 1 (zlib compression)
    Start Time: 1336582165
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

note the "Verify return code" at the bottom.

By way of demonstration that the server is actually checking those certs:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert bogus.crt -key bogus.key -pass pass:boguspass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
140283463366472:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1193:SSL alert number 48
140283463366472:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 3984 bytes and written 997 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 07E536F1C69A856857EA95DFD821BD6BBD499B5710642F9396D9525637EAD17C03064D5115B3D7F517EDE189E7AF40F8
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Compression: 1 (zlib compression)
    Start Time: 1336582289    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Note the handshake failures near the top, where httpd closed the connection on the client.

The next step is to make CRLs work properly, since Puppet uses them extensively.

Setting up a buildslave instance remotely on OS X Lion

nospam@example.com (Dustin J. Mitchell) — Sat, 17 Mar 2012 23:05:47 +0000

Byrce Lelbach has generously offered access to an OS X system as a metabuildbot slave. As I went about setting it up today, the process was not obvious, so I thought I'd share. This was interesting mostly because I only have SSH access to the host, so I cannot download things from the Apple Store or do any of the fancy point-and-click stuff that would make this easier.

First, I needed to get XCode installed. Note that the (much quicker to download) XCode command-line tools are not sufficient to build everything in MacPorts -- in particular, they do not support building zlib, which is required for git-core.

I got my hands on a copy of "Install XCode.app", and:

host:Downloads buildbot$ cd Install\ Xcode.app/Contents/
host:Contents buildbot$ sudo installer -package Resources/Xcode.mpkg -target /
Password:
installer: Package name is Xcode
installer: Upgrading at base path /
installer: The upgrade was successful.

Once this was done, I installed MacPorts:

host:Downloads buildbot$ hdiutil mount MacPorts-2.0.4-10.7-Lion.dmg
Checksumming Driver Descriptor Map (DDM : 0)…
     Driver Descriptor Map (DDM : 0): verified   CRC32 $A913D2D8
Checksumming Apple (Apple_partition_map : 1)…
....
     Apple (Apple_partition_map : 1): verified   CRC32 $A1DF5DC1
Checksumming disk image (Apple_HFS : 2)…
...... (...) .....
          disk image (Apple_HFS : 2): verified   CRC32 $5A3E74A0
Checksumming  (Apple_Free : 3)…
                    (Apple_Free : 3): verified   CRC32 $00000000
verified   CRC32 $D9641854
/dev/disk2              Apple_partition_scheme
/dev/disk2s1            Apple_partition_map
/dev/disk2s2            Apple_HFS                       /Volumes/MacPorts-2.0.4
host:Downloads buildbot$ pushd /Volumes/MacPorts-2.0.4/
/Volumes/MacPorts-2.0.4 ~/Downloads
host:MacPorts-2.0.4 buildbot$ sudo installer -package MacPorts-2.0.4.pkg/ -target /
Password:
installer: Package name is MacPorts-2.0.4
installer: Installing at base path /
installer: The install was successful.
host:MacPorts-2.0.4 buildbot$ popd
host:Downloads buildbot$ hdiutil unmount /Volumes/MacPorts-2.0.4

and we're off to the races.

I added /opt/local/bin to my path as suggested, and then followed the normal MacPorts setup process.

Finishing up the buildslave install required installing Git (which manages to pull in unreasonable amounts of other stuff!)

host:Contents buildbot$ sudo  /opt/local/bin/port install git-core -credential_osxkeychain-doc-pcre-python27

which is required for the source steps, then creating a virtualenv to install buildbot-slave:

host:~ buildbot$ virtualenv sandbox
New python executable in sandbox/bin/python
Installing setuptools............done.
Installing pip...............done.
host:~ buildbot$ source sandbox/bin/activate
(sandbox)host:~ buildbot$ pip install buildbot-slave
...

and then create and start a slave:

(sandbox)host:~ buildbot$ buildslave create-slave buildslave buildbot.buildbot.net:9989 HOSTNAME PASS
...
(sandbox)host:~ buildbot$ buildslave start buildslave
...

I then followed the helpful advice here to set up a plist that will start the daemon on boot.

IT and Community

nospam@example.com (Dustin J. Mitchell) — Thu, 17 Nov 2011 05:05:24 +0000

Mozilla's IT team is pivoting to a more community-focused approach. Our director of IT, mrz, has been writing extensively about it over the last few weeks.

As you can imagine, the difficult part of this is to balance security with accessibility. We'd like to be open, but we can't give the keys to the kingdom out to anyone who promises to help. The approach we're taking is to treat volunteers as we would part-time employees - post positions, interview, and then supervise to gain trust. This is a fairly common model, actually, for any organization with volunteers and a need for security. Youth programs, for example, generally do an interview and background check with new volunteers, and those volunteers will be paired with senior volunteers or staff for a while.

However, it's a bit cumbersome, both for Mozilla and for potential volunteers. We must design entire positions - ongoing tasks or roles that a volunteer can work on for an extended period of time - and then select a limited number of volunteers to fill those roles. For potential volunteers, an application and interview can mean a long time (weeks?) before they get to do anything hands-on. It also carries the risk that we'd have to turn a qualified volunteer away due to lack of suitable positions.

So what to do?

We need a more fluid way of interacting with potential contributors. Since our bug database is public, we can begin by simply tagging a few bugs that are appropriate for newcomers -- things that don't require sensitive access and are well-encapsulated so they can be completed without extensive knowledge of Mozilla's infrastructure.

Here's the list.

It's a bit short right now. There are a few things that may help:

We can get better about identifying appropriate tasks and projects and making bugs out of them.
We can identify a means of giving limited or sandboxed access to a new volunteer.
Consumers of Mozilla's IT resources can begin tagging bugs, where Mozilla can provide the resources and volunteers can do the heavy lifting - got any ideas?

Subscribe to a google group with a different address?

nospam@example.com (Dustin J. Mitchell) — Fri, 02 Sep 2011 20:04:00 +0000

Google Groups is one place where, IMHO, Google pushes its hegemony too far, making it difficult to use. I wanted to subscribe to puppet-users with my Mozilla address, but since I have a Google account, Groups assumes I want to subscribe with that address. No!

I found the fix with a bit of Googling (some irony there). It involves editing a URL:

http://groups.google.com/group/puppet-users/boxsubscribe?email=email@domain.com

where you'd substitute the name of the group you want for puppet-users and add your email at the end.

Nagios NSCA from Python

nospam@example.com (Dustin J. Mitchell) — Fri, 20 May 2011 21:55:11 +0000

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

Amanda's Transfer Mechanisms

nospam@example.com (Dustin J. Mitchell) — Sat, 22 Jan 2011 20:47:13 +0000

There's been a bit of confusion on the mailing list and IRC about how Amanda assembles transfers out of transfer elements, and how transfer mechanisms influence that.

In the final form of a transfer, any two adjacent elements must have the same mechanism. For example is, an upstream element speaking XFER_MECH_PUSH_BUFFER cannot talk to a downstream element using XFER_MECH_READ_FD (nor, more confusingly, XFER_MECH_PULL_BUFFER). So each mechanism is an isolated definition of "here's how upstream and downstream should talk". They come in pairs because generally anything upstream can do to downstream (e.g., upstream can write to downstream's fd) can occur in reverse (e.g., downstream can read from upstream's fd).

What makes this confusing is that if you specify a set of elements which can't talk directly to one another, then xfer.c will add "glue" elements between the specified elements. To make that concrete, imagine you specify a transfer as

source-holding --> filter-xor --> dest-fd

(if you like practical examples, then imagine filter-xor is a buffer-based decompression filter, and you're pulling data from holding disk, decompressing, and sending to a pipe -- something amfetchdump would do). Here are the mechanisms supported by each element:

source-holding:
 XFER_MECH_PULL_BUFFER

filter-xor
 XFER_MECH_PULL_BUFFER (input) and
 XFER_MECH_PULL_BUFFER (output)
or
 XFER_MECH_PUSH_BUFFER (input) and
 XFER_MECH_PUSH_BUFFER (output)

dest-fd
 XFER_MECH_WRITEFD (input)

In putting these together, source-holding and filter-xor can use the same mechanism (PULL_BUFFER). This leaves filter-xor using PULL_BUFFER for output, but dest-fd does not support this. So xfer.c adds a glue element that can speak PULL_BUFFER on input and WRITEFD on output. This element basically loops in a thread, calling upstream->pull_buffer and write(downstream->input_fd, buffer). So the final xfer looks like

 source-holding --(PULL_BUFFER)--> filter-xor --(PULL_BUFFER)--> glue --(WRITEFD)--> dest-fd

Hopefully that helps to explain how the glue works.

Note that one of the cool things about this arrangement is that in most cases the complexity is in the glue, not the elements. In fact, in this case the glue provides the only thread that's required to run this transfer, so the other three element implementations don't need to manage threads at all.