Sysadmin

I did something colossally stupid yesterday. I was at the local hackerspace, hoping to cut some acrylic, and the wifi wasn't working. I was in a hurry and frustrated, so I pulled out a USB stick and tried to erase it. Suffice to say, the USB stick wasn't at /dev/sda. I wiped out the GPT on my laptop. Its disk is encrypted, so the buffer store kept things working for a while, then suddenly I had a blinking root prompt and .. nothing.

After the obligatory cold-sweat had passed, I quietly packed up and walked out. Here's the story of how I recovered from this, with the help of Jake Watkins (:dividehex).

Continue reading "Oops, I partitioned my drive.."

I got a new laptop, a ThinkPad X1 Carbon, and I'm running Linux on it. So you're in for a series of posts describing the complex process I had to follow to accomplish simple things. Spoiler alert: 2013 is not the year of Linux on the desktop. It's not looking good for 2014 either.

I'm running Fedora 18. I tried Ubuntu 12.10, but Unity couldn't hold itself together long enough to actually do anything, so I started over with Fedora.

SSH Agent

Gnome runs a nice keychain app that acts like (but is not) OpenSSH's ssh-agent. The one obvious place it differs is that ssh-add -l will list keys even if they are "locked" (passphrase not supplied).

As long as you point the SSHAUTHSOCK variable to the right place, the agent works just fine for unlocking keys - it finds any private/public pairs in ~/.ssh, and prompts to unlock them once you issue an SSH command that needs a key. The problem is, it never re-locks the keys.

Locking

Personally, I use SSH constantly while my laptop is awake, so I don't want an arbitrary timeout. Instead, I'm careful to put it to sleep when I'm away from the keyboard. So I want a way to lock the key on sleep.

It turns out that pm-utils will run scripts in /etc/pm/sleep.d on sleep and wake. It runs them as root, unfortunately. I added the following in 01dustin-ssh-agent.sh:

#!/bin/sh

# drop keys from dustin's SSH agent

. "${PM_FUNCTIONS}"

lock()
{
        su - dustin /home/dustin/bin/ssh-lock
}

case "$1" in
        hibernate|suspend) lock ;;
        *) exit $NA ;;
esac

and then added the following in ~/bin/ssh-lock:

#!/bin/sh

# drop keys from the SSH agent, using the same trick as bin/startscreen to find
# that agent

base="/tmp"
[ -d /run/user ] && base="/run/user/$(id -u)"
socket_dir="$base/$(uname -n)-$(id -u)"
SSH_AUTH_SOCK=$socket_dir/agent ssh-add -D

See my post on tunneling ssh-agent into a screen session for the reference to bin/startscreen. I'm not sure how best to accomplish this without such a trick. I'll work on that and post again.

If you're looking for info on CustomSettings.ini, you're most likely to find questions answered with "try this script". You type it in, and if it works, great; if not, keep looking. It's well-nigh impossible to find actual documentation, and the programming-by-INI-sections design is not exactly intuitive.

It turns out there's some in the help docs for the MDT, but those are a .CHM file and Microsoft apparently doesn't post those online.

However, some helpful (Russian?) souls have done so. Behold: Microsoft® Deployment Toolkit 2012 Toolkit Reference.

For a project at Mozilla that involves re-imaging hundreds of mobile devices, we want to gather logs in a database for failure analysis. Mobile devices fail all the time -- not sure if you knew that.

We'll probably end up with 1,000-10,000 log entries per day. We'd like to expire them on a relatively aggressive schedule -- no need for historical analysis at this level. So that means not only a lot of inserts, but a lot of deletes.

We're using MySQL as the database backend, and MySQL doesn't do well with deletes - it just marks the row as deleted, but doesn't reclaim the space, and in fact doesn't even remove the row from consideration in queries. So if you blindly insert and delete in a table, MySQL will eat disk space and get progressively slower.

One fix to this is to optimize the table periodically. However, this requires a full lock of the table for the duration of the optimize, which can be quite a while. We dont' want to cause a backup of production tasks while this is going on.

The other option is to partition the table. A partitioned table is basically a set of tables (partitions) with the same columns, organized to look like a single table. There's a partitioning function that determines in which partition a particular row belongs. There are a few advantages. Each partition is a fraction of the size of the whole table, so inserts are quicker (once the appropriate table is determined). The query engine can use "partition pruning" to ignore partitions that could not hold rows relevant to the query. Finally, dropping an entire partition at once is a very simple operation, and doesn't leave any garbage that needs to be optimized away.

For logs, we want to partition by time, in this case with one partition per day. Most of the "get the logs" queries will use a limited time range, invoking query pruning and allowing a quick response.

The tricky part is, the DB server does not automatically create and destroy partitions. We need to do that. It's pretty straightforward with stored procedures, though. Here's the resulting SQL to create the logs table:

DROP TABLE IF EXISTS logs;
CREATE TABLE logs (
    -- foreign key for the board
    board_id integer not null,
    ts timestamp not null,
    -- short string giving the origin of the message (syslog, api, etc.)
    source varchar(32) not null,
    -- the message itself
    message text not null,
    -- indices
    index board_id_idx (board_id),
    index ts_idx (ts)
);

--
-- automated log partition handling
--

DELIMITER $$

-- Procedure to initialize partitioning on the logs table
DROP PROCEDURE IF EXISTS init_log_partitions $$
CREATE PROCEDURE init_log_partitions(days_past INT, days_future INT)
BEGIN
    DECLARE newpart integer;
    SELECT UNIX_TIMESTAMP(NOW()) INTO newpart;
    SELECT newpart - (newpart % 86400) INTO newpart; -- round down to the previous whole day

    -- add partitions, with a single partition for the beginning of the current day, then
    -- let update_log_partitions take it from there
    SET @sql := CONCAT('ALTER TABLE logs PARTITION BY RANGE (UNIX_TIMESTAMP(ts)) ('
                        , 'PARTITION p'
                        , CAST(newpart as char(16))
                        , ' VALUES LESS THAN ('
                        , CAST(newpart as char(16))
                        , '));');
    PREPARE stmt FROM @sql;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;

    -- do an initial update to get things synchronized
    call update_log_partitions(days_past, days_future);
END $$

-- Procedure to delete old partitions and create new ones around the current date
DROP PROCEDURE IF EXISTS update_log_partitions $$
CREATE PROCEDURE update_log_partitions(days_past INT, days_future INT)
BEGIN
    DECLARE part integer;
    DECLARE newpart integer;
    DECLARE earliest integer;
    DECLARE latest integer;

    -- add new partitions; keep adding a partition for a new day until we reach latest
    SELECT UNIX_TIMESTAMP(NOW()) + 86400 * (days_future+1) INTO latest;
    createloop: LOOP
        -- Get the newest partition (PARTITION_DESCRIPTION is the number from VALUES LESS THAN)
        -- partitions are named similarly, with a 'p' prefix
        SELECT MAX(PARTITION_DESCRIPTION) INTO part
            FROM INFORMATION_SCHEMA.PARTITIONS
            WHERE TABLE_NAME='logs'
            AND TABLE_SCHEMA='imagingservice';
        IF part < latest THEN -- note part cannot be NULL, as there must be at least one partition
            SELECT part + 86400 INTO newpart;
            SET @sql := CONCAT('ALTER TABLE logs ADD PARTITION ( PARTITION p'
                                , CAST(newpart as char(16))
                                , ' VALUES LESS THAN ('
                                , CAST(newpart as char(16))
                                , '));');
            PREPARE stmt FROM @sql;
            EXECUTE stmt;
            DEALLOCATE PREPARE stmt;
        ELSE
            LEAVE createloop;
        END IF;
    END LOOP;

    -- now, deal with pruning old partitions; select the minimum partition
    -- and delete it if it's too old
    SELECT UNIX_TIMESTAMP(NOW()) - 86400 * (days_past+1) INTO earliest;
    purgeloop: LOOP
        -- Get the oldest partition
        SELECT MIN(PARTITION_DESCRIPTION) INTO part
            FROM INFORMATION_SCHEMA.PARTITIONS
            WHERE TABLE_NAME='logs'
            AND TABLE_SCHEMA='imagingservice';
        IF part < earliest THEN
            SET @sql := CONCAT('ALTER TABLE logs DROP PARTITION p'
                                , CAST(part as char(16))
                                , ';');
            PREPARE stmt FROM @sql;
            EXECUTE stmt;
            DEALLOCATE PREPARE stmt;
        ELSE
            LEAVE purgeloop;
        END IF;
    END LOOP;
END $$

DELIMITER ;

-- initialize the partitioning
CALL init_log_partitions(14, 1);

-- and then update every day (this can't be set up in init_log_partitions)
DROP EVENT IF EXISTS update_log_partitions;
CREATE EVENT update_log_partitions  ON SCHEDULE EVERY 1 day
DO CALL update_log_partitions(14, 1);

A few notes here. First, the table is created without any partitions. This is because I don't know a priori which partitions it should have, and it's easier to get code to figure that out than do it myself. That's what the initlogpartitions function does. The updatelogpartitions function looks at the current time and makes sure there are enough partitions for the future, and drops partitions too far in the past. Finally, a MySQL event is set up to update the partitions daily.

You'll need to enable the event scheduler globally to get this to run:

set global event_scheduler=on;

I'm laying some SSL groundwork for a project to allow puppet clients to move between puppet servers without requiring a central CA, and without requiring each client to be aware of all masters. More on that in a future post.

Based on "Multiple Certificate Authorities", I would like to have certificate chains that look like this:

      +-puppetmaster1 CA--+-puppetmaster1 server cert
      |                   |
      |                   +-client 1 server cert
root--+                   :
      |                   
      +-puppetmaster2 CA--+-puppetmaster2 server cert
                          |
                          +-client 10 server cert
                          :

Then all of the certificate validation would be done with the root CA certificate as the trusted certificate. A server certificate signed by puppetmaster2's CA cert should then validate on puppetmaster1.

Building the certificates wasn't all that difficult - see my comment on the bug for the script. However, while making sure the verification worked, I ran into some non-obvious limitations of OpenSSL that are worth writing down.

I began by running "openssl verify":

[root@relabs-puptest1 ~]# openssl verify -verbose -CAfile puptest-certs/root-ca.crt -purpose sslclient puptest-certs/relabs08.build.mtv1.mozilla.com.crt 
puptest-certs/relabs08.build.mtv1.mozilla.com.crt: CN = relabs08.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
error 20 at 0 depth lookup:unable to get local issuer certificate

the problem here is that the intermediate certificate is not available to the verification tool. Sources suggest to include it with the server cert, by concatention, with the server cert last:

cat puptest-certs/relabs-puptest1.build.mtv1.mozilla.com-ca.crt puptest-certs/relabs08.build.mtv1.mozilla.com.crt > relabs08-with-intermed.crt

However, after some struggle I learned that "openssl verify" does not recognize this format -- it will only look at the first certificate in the file (the intermediate), and if you don't look carefully you'll find that it successfully verifies the intermediate, not the server certificate! Sadly, sclient and ssever don't support it either. Apache httpd supports it with SSLCACertificatePath. This will feed the certificate chain to the client, and also allow httpd to verify client certificates without requiring the clients to support an intermediate.

The Apache config is

Listen 1443

<VirtualHost *:1443>
        ServerName relabs-puptest1.build.mtv1.mozilla.com
        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.crt
        SSLCertificateKeyFile /etc/httpd/relabs-puptest1.build.mtv1.mozilla.com.key
        SSLCACertificatePath /etc/httpd/ca-path

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        #SSLCARevocationFile     /etc/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient require
        SSLVerifyDepth  2

</VirtualHost>

While you're getting that set up, you're probably wondering where to get this fancy "c_rehash" utility. Don't bother. It's about as simple as:

for i in *.crt; do
        h=`openssl x509 -hash -noout -in $i`
        rm -f $h.0
        ln -s $i $h.0
done

As a side-note, the results of verification by sclient and sserver are not very obvious. Look for the overall error message near the bottom of the output. Here's the result of a client verification once I had everything put together, with some long uselessness elided:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert puptest-certs/relabs08.build.mtv1.mozilla.com.crt -key puptest-certs/relabs08.build.mtv1.mozilla.com.key -pass pass:clientpass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 5379 bytes and written 1716 bytes
---
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: E30634D9CFCC2FA327282DA813BB550C24ACDF18194E5F13C4981AA55914B5F0
    Session-ID-ctx: 
    Master-Key: 013EB09B066418694D36D74B414BBA42E52DBF0066314B60FC7A74662A60934282B6C37C5C82026F70287E60F4FF9472
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket:
    0000 - 82 5f 17 72 97 bd f3 1e-ec 24 de 69 ab 1e cd 1d   ._.r.....$.i....
    ....
    0520 - 40 05 b3 27 20 00 8d ce-93 a9 48 81 8f 0c 16 5b   @..' .....H....[

    Compression: 1 (zlib compression)
    Start Time: 1336582165
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

note the "Verify return code" at the bottom.

By way of demonstration that the server is actually checking those certs:

[root@relabs-puptest1 ~]# openssl s_client -verify 2 -CAfile puptest-certs/root-ca.crt -cert bogus.crt -key bogus.key -pass pass:boguspass -connect localhost:1443
verify depth is 2
CONNECTED(00000003)
depth=2 CN = PuppetAgain Root CA, emailAddress = release@mozilla.com, OU = Release Engineering, O = "Mozilla, Inc."
verify return:1
depth=1 CN = CA on relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
depth=0 CN = relabs-puptest1.build.mtv1.mozilla.com, emailAddress = release@mozilla.com, O = "Mozilla, Inc.", OU = Release Engineering
verify return:1
140283463366472:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1193:SSL alert number 48
140283463366472:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
Certificate chain
 0 s:/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
 1 s:/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
 2 s:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
   i:/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEeTCCA2GgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBkTE1MDMGA1UEAxMsQ0Eg
...
H90rZMVxsVyPHjjfXkeeFcSWyUnV/z3G9osrI9I9SaQ1o9bDc7ZheyHbWbhn
-----END CERTIFICATE-----
subject=/CN=relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
issuer=/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
Acceptable client certificate CA names
/CN=PuppetAgain Root CA/emailAddress=release@mozilla.com/OU=Release Engineering/O=Mozilla, Inc.
/CN=CA on relabs-puptest1.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
/CN=CA on relabs-puptest2.build.mtv1.mozilla.com/emailAddress=release@mozilla.com/O=Mozilla, Inc./OU=Release Engineering
---
SSL handshake has read 3984 bytes and written 997 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 07E536F1C69A856857EA95DFD821BD6BBD499B5710642F9396D9525637EAD17C03064D5115B3D7F517EDE189E7AF40F8
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Compression: 1 (zlib compression)
    Start Time: 1336582289    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Note the handshake failures near the top, where httpd closed the connection on the client.

The next step is to make CRLs work properly, since Puppet uses them extensively.

Byrce Lelbach has generously offered access to an OS X system as a metabuildbot slave. As I went about setting it up today, the process was not obvious, so I thought I'd share. This was interesting mostly because I only have SSH access to the host, so I cannot download things from the Apple Store or do any of the fancy point-and-click stuff that would make this easier.

First, I needed to get XCode installed. Note that the (much quicker to download) XCode command-line tools are not sufficient to build everything in MacPorts -- in particular, they do not support building zlib, which is required for git-core.

I got my hands on a copy of "Install XCode.app", and:

host:Downloads buildbot$ cd Install\ Xcode.app/Contents/
host:Contents buildbot$ sudo installer -package Resources/Xcode.mpkg -target /
Password:
installer: Package name is Xcode
installer: Upgrading at base path /
installer: The upgrade was successful.

Once this was done, I installed MacPorts:

host:Downloads buildbot$ hdiutil mount MacPorts-2.0.4-10.7-Lion.dmg
Checksumming Driver Descriptor Map (DDM : 0)…
     Driver Descriptor Map (DDM : 0): verified   CRC32 $A913D2D8
Checksumming Apple (Apple_partition_map : 1)…
....
     Apple (Apple_partition_map : 1): verified   CRC32 $A1DF5DC1
Checksumming disk image (Apple_HFS : 2)…
...... (...) .....
          disk image (Apple_HFS : 2): verified   CRC32 $5A3E74A0
Checksumming  (Apple_Free : 3)…
                    (Apple_Free : 3): verified   CRC32 $00000000
verified   CRC32 $D9641854
/dev/disk2              Apple_partition_scheme
/dev/disk2s1            Apple_partition_map
/dev/disk2s2            Apple_HFS                       /Volumes/MacPorts-2.0.4
host:Downloads buildbot$ pushd /Volumes/MacPorts-2.0.4/
/Volumes/MacPorts-2.0.4 ~/Downloads
host:MacPorts-2.0.4 buildbot$ sudo installer -package MacPorts-2.0.4.pkg/ -target /
Password:
installer: Package name is MacPorts-2.0.4
installer: Installing at base path /
installer: The install was successful.
host:MacPorts-2.0.4 buildbot$ popd
host:Downloads buildbot$ hdiutil unmount /Volumes/MacPorts-2.0.4

and we're off to the races.

I added /opt/local/bin to my path as suggested, and then followed the normal MacPorts setup process.

Finishing up the buildslave install required installing Git (which manages to pull in unreasonable amounts of other stuff!)

host:Contents buildbot$ sudo  /opt/local/bin/port install git-core -credential_osxkeychain-doc-pcre-python27

which is required for the source steps, then creating a virtualenv to install buildbot-slave:

host:~ buildbot$ virtualenv sandbox
New python executable in sandbox/bin/python
Installing setuptools............done.
Installing pip...............done.
host:~ buildbot$ source sandbox/bin/activate
(sandbox)host:~ buildbot$ pip install buildbot-slave
...

and then create and start a slave:

(sandbox)host:~ buildbot$ buildslave create-slave buildslave buildbot.buildbot.net:9989 HOSTNAME PASS
...
(sandbox)host:~ buildbot$ buildslave start buildslave
...

I then followed the helpful advice here to set up a plist that will start the daemon on boot.

Mozilla's IT team is pivoting to a more community-focused approach. Our director of IT, mrz, has been writing extensively about it over the last few weeks.

As you can imagine, the difficult part of this is to balance security with accessibility. We'd like to be open, but we can't give the keys to the kingdom out to anyone who promises to help. The approach we're taking is to treat volunteers as we would part-time employees - post positions, interview, and then supervise to gain trust. This is a fairly common model, actually, for any organization with volunteers and a need for security. Youth programs, for example, generally do an interview and background check with new volunteers, and those volunteers will be paired with senior volunteers or staff for a while.

However, it's a bit cumbersome, both for Mozilla and for potential volunteers. We must design entire positions - ongoing tasks or roles that a volunteer can work on for an extended period of time - and then select a limited number of volunteers to fill those roles. For potential volunteers, an application and interview can mean a long time (weeks?) before they get to do anything hands-on. It also carries the risk that we'd have to turn a qualified volunteer away due to lack of suitable positions.

So what to do?

We need a more fluid way of interacting with potential contributors. Since our bug database is public, we can begin by simply tagging a few bugs that are appropriate for newcomers -- things that don't require sensitive access and are well-encapsulated so they can be completed without extensive knowledge of Mozilla's infrastructure.

Here's the list.

It's a bit short right now. There are a few things that may help:

• We can get better about identifying appropriate tasks and projects and making bugs out of them.
• We can identify a means of giving limited or sandboxed access to a new volunteer.
• Consumers of Mozilla's IT resources can begin tagging bugs, where Mozilla can provide the resources and volunteers can do the heavy lifting - got any ideas?

Google Groups is one place where, IMHO, Google pushes its hegemony too far, making it difficult to use. I wanted to subscribe to puppet-users with my Mozilla address, but since I have a Google account, Groups assumes I want to subscribe with that address. No!

I found the fix with a bit of Googling (some irony there). It involves editing a URL:

http://groups.google.com/group/puppet-users/boxsubscribe?email=email@domain.com

where you'd substitute the name of the group you want for puppet-users and add your email at the end.

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case.

Continue reading "IPv6 and Amanda"

I just upgraded my Macbook to Snow Leopard, and the upgrade has changed the way SSH authentication works. I have set up a system I like quite a bit, now, and thought I would share.

Continue reading "SSH With Snow Leopard"

I've been meaning to get IPv6 set up on my local network for some time. My only practical reason is that Amanda supports IPv6 and I should test that support. It was also a good chance to re-immerse myself in network configuration, and Hurricane Electric has a neat certification process to add some motivation.

Continue reading "IPv6 Configuration"

I saw an interesting post on the Subversion development list a while ago. In part:

This note is to inform you that the Shell Group will be migrating from Windows 2000 to Microsoft's new operating system known as Windows Vista with effect from Q1 2008, and to seek your assistance and support in minimising disruption to users and applications during and after the migration.

The note goes on to request some fairly specific information about the upgrade path for TortiseSVN, the Windows Subversion client. They are the sorts of questions that all IT shops would love to ask all of their vendors, with the expectation of a full and well-researched answer.

As an admin at a small K-12 school, questions of this sort were met with blank stares from vendors. At best, we could get a demo unit, but any sort of analysis of the potential fit of a product (besides the "analysis" the salesmen would do) was simply out of the question for an account of our size. On the other hand, I could usually count on honest assessments from open-source software mailing lists, even if they didn't represent full-scale implementation analyses.

The Shell Group request turns the situation around. Shell Group is a very large client and is probably accustomed to contacting peers like Dell, Aramark, or HBN-AMRO with requests like this. Yet here they are making these requests of a gaggle of developers, none of whom want to be "the main liaison for ALL matters pertaining to Vista compatibility." There were no on-list responses, so I can't say what became of the request.

There's clearly a business need here, but it's not the typical "sell support for open source software" niche. Rather, Shell Group wants a business entity with which they can have a more contractual relationship: one that can get the software certified by Microsoft, make projections as to deliverable dates, and so on. An entity that can answer support calls but does not have significant control of the development community is simply not capable of these things, but neither is a development community without a legally representative organization.

I'm interested to see if this kind of request occurs more often, and what effect it has on the landscape of adoption of OSS in big business.

Continue reading "Open-Source Support?"

Mrs. Zwicky gave a really excellent talk that balanced real research in education, in problem solving, and in systems administration. She teaches systems administration to Navy recruits for a defense contractor, in a tutoring setting. The talk addressed the common belief that problem solving skills are essentially innate and can't be taught. She discussed the problem-solving process in general, using lots of examples (well, "war stories") from systems admin. Finally, she talked about some of the techniques and skills needed to teach problem solving (or anything, really).

These techniques included scaffolding -- building the learners' conceptual understanding by presenting the right tasks, offering the right support, and convincing the learner to talk about the concepts, not just "what do I type". Also included was "spotting", which I assume comes from sports -- the idea here is to make sure that the learner doesn't suffer any horrible consequences from making mistakes.

Continue reading ""Teaching Problem Solving: You Can and You Should" (Elizabeth Zwicky)"

So what better time to inagurate this blog than now, during the keynote to the LISA (Large Installation Systems Administration) conference. It's an interesting general talk about one of the many pressing non-technical issues in this community: DRM and restrictive licensing.

Continue reading "USENIX LISA"