Code V.igoro.us - Sysadmin

IT and Community

nospam@example.com (Dustin J. Mitchell) — Wed, 16 Nov 2011 23:05:24 -0600

Mozilla's IT team is pivoting to a more community-focused approach. Our director of IT, mrz, has been writing extensively about it over the last few weeks.

As you can imagine, the difficult part of this is to balance security with accessibility. We'd like to be open, but we can't give the keys to the kingdom out to anyone who promises to help. The approach we're taking is to treat volunteers as we would part-time employees - post positions, interview, and then supervise to gain trust. This is a fairly common model, actually, for any organization with volunteers and a need for security. Youth programs, for example, generally do an interview and background check with new volunteers, and those volunteers will be paired with senior volunteers or staff for a while.

However, it's a bit cumbersome, both for Mozilla and for potential volunteers. We must design entire positions - ongoing tasks or roles that a volunteer can work on for an extended period of time - and then select a limited number of volunteers to fill those roles. For potential volunteers, an application and interview can mean a long time (weeks?) before they get to do anything hands-on. It also carries the risk that we'd have to turn a qualified volunteer away due to lack of suitable positions.

So what to do?

We need a more fluid way of interacting with potential contributors. Since our bug database is public, we can begin by simply tagging a few bugs that are appropriate for newcomers -- things that don't require sensitive access and are well-encapsulated so they can be completed without extensive knowledge of Mozilla's infrastructure.

Here's the list.

It's a bit short right now. There are a few things that may help:

We can get better about identifying appropriate tasks and projects and making bugs out of them.
We can identify a means of giving limited or sandboxed access to a new volunteer.
Consumers of Mozilla's IT resources can begin tagging bugs, where Mozilla can provide the resources and volunteers can do the heavy lifting - got any ideas?

Subscribe to a google group with a different address?

nospam@example.com (Dustin J. Mitchell) — Fri, 02 Sep 2011 15:04:00 -0500

Google Groups is one place where, IMHO, Google pushes its hegemony too far, making it difficult to use. I wanted to subscribe to puppet-users with my Mozilla address, but since I have a Google account, Groups assumes I want to subscribe with that address. No!

I found the fix with a bit of Googling (some irony there). It involves editing a URL:

http://groups.google.com/group/puppet-users/boxsubscribe?email=email@domain.com

where you'd substitute the name of the group you want for puppet-users and add your email at the end.

Nagios NSCA from Python

nospam@example.com (Dustin J. Mitchell) — Fri, 20 May 2011 16:55:11 -0500

I've been working on improving the monitoring of the build slaves at Mozilla. As part of this project, I needed to be able to submit passive check results to the Nagios servers via NSCA during system startup. I'm doing this from a Python script that needs to run on a wide array of systems using whatever random Python is available. We run some oddball stuff, so the common denominator is Python 2.4.

It turns out that there's no Python NSCA library, although there is Net::Nsca in Perl. So, I wrote one, and put it on github: https://github.com/djmitche/pynsca.

At the moment, this only knows XOR, and only does service checks. That's all I need, but hopefully it can be easily expanded to cover other purposes. The one thing I want to avoid is adding mandatory requirements -- this should work, at least in plain-text and XOR modes, on a plain-vanilla Python installation.

By the way, the startup script I'm working on is runslave.py, which includes a modified copy of pynsca and does a number of other housekeeping jobs as well. More on that in a subsequent post.

IPv6 and Amanda

nospam@example.com (Dustin J. Mitchell) — Fri, 16 Jul 2010 22:04:00 -0500

Amanda joined the IPv6 revolution in November 2006 - all of the BSD-style authentication mechanisms can support IPv6 endpoints. However, it's generally agreed that this was a mistake, and in this post I will talk about why that's the case. First, a bit of background on how Amanda's networking code works, and what had to change to support IPv6. Amanda supports security mechanisms called BSD (the oldest), BSDUDP, and BSDTCP. These all authenticate (if you can call it that) using the same sorts of checks that rsh uses. The incoming connection is accepted if:

it is from a "reserved" port (less than 1024);
the address of the initiator has complementary forward and reverse DNS records in place; and
the initiator's hostname is in .amandahosts

During a backup operation, the Amanda server contacts each client host. When using the BSD authentications, this triggers amandad, which checks the above restrictions before beginning communication with the server. This initial connection is packet-based, and can be carried out over UDP (for BSD and BSDUDP) or TCP (BSDTCP). When a dump begins, several "streams" are opened to transmit the data, index, and metadata. For BSD and BSDUDP, each stream is implemented as a distinct TCP connection, where the client sends a port number to the server and the server connects to that port. BSDTCP multiplexes all streams over a single TCP connection using a basic type/length packet encapsulation.

The first challenge in adding IPv6 support was to deal properly with IPv6 addresses when querying the DNS. That meant switching to getaddrinfo and getnameinfo, as suggested by Jun-ichiro itojun Itoh. These functions bring their own compatibility problems, but Amanda uses gnulib, which provides compatibile implementations on systems where they are not available, minimizing the difficulty.

We had a lot of trouble from systems such as RHEL3 possessing IPv6 support in the compiler environment but not in the kernel. On such systems, code using constants like AF_INET6 or AI_V4MAPPED would compile without problems, but fail at runtime. We added a WORKING_IPV6 preprocessor conditional, without which all references to IPv6-related symbols were removed. At configure time, Amanda tries to create an IPv6 socket, and sets this conditional to true if it succeeds. The --without-ipv6 configure option forcibly disables IPv6 support.

The sockaddr structures and API for IPv6 are fairly difficult to use, particularly if it's not known in advance what sort of address they will contain. We added a set of macros and utility functions in sockaddr-util.c and sockaddr-util.h. Using these macros throughout Amanda removed a significant amount of code that was conditionalized on both compile-time support and runtime address family, and centralized that logic in one easily-maintained place.

On our build systems, we had to deal with different levels of support in the compile environment and the kernel. This is fine: most Amanda users install binary packages that are produced on roughly the same OS distribution and version as was used for the build, so the kernel support is generally the same. However, a third variable has tripped up lots of Amanda users: system configuration. In particular, several newer Linux distributions have shipped with localhost resolving to ::1 vi /etc/hosts, but without enough interface configuration to actually utilize a socket bound to that address. Amanda uses localhost sockets for inter-process communication, so this misconfiguration causes backup operations to fail. The solution is to either finish configuring IPv6 on the host, remove the reference to ::1 in /etc/hosts, or build Amanda with --without-ipv6.

I have not yet heard of an Amanda installation where IPv6 communication is in use. But I have heard from countless IPv4 users whose Amanda installations have failed due to bad IPv6 support. At the moment, then, I feel that adding IPv6 support to Amanda has been a net negative for the project. Although there is doubtless room for improvement, I will not entertain patches for better IPv6 support, for fear they will introduce new bugs for our exclusively IPv4 userbase.

Of course, all of this may change as dual-stack networks grow more prevalent and are replaced by pure IPv6 networks!

SSH With Snow Leopard

nospam@example.com (Dustin J. Mitchell) — Sat, 10 Jul 2010 14:27:00 -0500

I just upgraded my Macbook to Snow Leopard, and the upgrade has changed the way SSH authentication works. I have set up a system I like quite a bit, now, and thought I would share. My usage pattern is that I do most of my work via GNU screen running on my login server, euclid. So I want a simple procedure that will connect me to that screen session, with a proper SSH agent set up.

Snow Leopard automatically starts an ssh-agent process at login. This is great, but does not interoperate with SSHKeychain. Dropping SSHKeychain is OK with me - I don't use SSH tunnels, so it really only acted as a GUI for passphrase entry. It also ate CPU occasionally, which of course causes the macbook to become more painfully hot than usual.

So I have three problems to solve: 1. automatically add my key to ssh-agent; 2. automatically expire the key at appropriate times (at my paranoia level, that's at system sleep); and 3. make multiple agent instances usable from the same shell session on the server.

Adding the Key at Connection Time

Adding the key is relatively straightforward. I wrote a short script that Terminal runs when I hit ⌘-N or ⌘-T:

#! /bin/bash

# does ssh-agent not have a key?
if ! ssh-add -l; then
    ssh-add ~/.ssh/dustin || exit 1
fi

exec ssh -x -t euclid.r.igoro.us bin/startscreen

This will prompt me for the passphrase when there is not already a key active, but proceed directly to the ssh invocation if the key situation is OK. The -x option to ssh is there to turn off X11 forwarding; without this option, SSH will helpfully start the X11 app. I think this is a great feature, but I don't use X11 apps very often, so I've disabled it.

Expiring the Key Automatically

Mac OS has a nicely designed system in place to allow applications to get notified when the system is changing power states. However, I wasn't interested in writing a full OS X app for this particular project. Enter sleepwatcher. This is a beautifully simple program that just executes scripts on particular power events. I set this up to run via launchd, as directed in the README, and to run ~/.ssh/sleep on sleep. That script contains:

#! /bin/bash

# first, don't inherit a socket (sleepwatcher doesn't get the user's env)
SSH_AUTH_SOCK=

# find some sockets
echo ".ssh/sleep:" `id`
for sock in /tmp/launch-*/Listeners; do
    if [ -w $sock ]; then
        echo "Trying to remove .ssh/dustin from socket $sock"
        SSH_AUTH_SOCK=$sock /opt/local/bin/ssh-add -d ~/.ssh/dustin
    else
        echo "Skipping unwritable socket $sock"
    fi  
done

The for loop is required because a script run from sleepwatcher doesn't inherit the SSH_AUTH_SOCK variable that points to the running SSH agent. The loop simply searches for a writable SSH socket of the pattern used by the system's agent.

SSH Agent and Screen

If you naïvely set up an SSH agent, connect to a remote system, and start screen there, things will work great - until you disconnect from the screen session. When you connect to the remote system, SSH forwards the agent connection for you, and sets SSH_AUTH_SOCK on the remote system to point to this forwarded socket. Screen passes this variable along blindly, so it appears in all of the shells opened in screen windows, and things work as you'd expect. When that SSH connection is removed, and a new one established, the forwarded agent appears at a new socket. But those shells running in screen windows are still pointing to the old name, and will no longer be able to connect.

The fix is to create a socket with a well-known name that will not change from connection to connection. The following script takes care of it. WARNING: this script is vulnerable to /tmp race conditions. I am the only user on my servers, so this doesn't bother me, but fixing it should be relatively straightforward.

#! /bin/bash
# hard-link the SSH socket to one with a fixed name on the local
# machine, and set SSH_AUTH_SOCK to point to that fixed name.  Later
# invocations of this script will change the link, but the name will
# remain valid, allowing existing shells to continue to function.
setup_fixed_socket() {
  local old_socket="$SSH_AUTH_SOCK"
  local socket_dir="/tmp/$(uname -n)-$(id -u)"
  local socket_file=$socket_dir/agent

  # set up the directory and permissions
  [ -e $socket_dir ] || mkdir -p $socket_dir
  chmod 700 $socket_dir

  # remove an existing link
  [ -e $socket_file ] && rm $socket_file

  # hard-link in the new one
  ln $old_socket $socket_file

  # return the new socket
  echo $socket_file
}

# this variable will be exported to every shell opened by this
# invocation of screen -- even subsequent connections to it.  This
# variable may live for days or weeks.
export SSH_AUTH_SOCK=$(setup_fixed_socket)

# finally, fire up screen.  Try reattaching to a running
# session; otherwise start up a new one
screen -R -DD ${@} || screen

IPv6 Configuration

nospam@example.com (Dustin J. Mitchell) — Sun, 27 Jun 2010 16:27:12 -0500

I've been meaning to get IPv6 set up on my local network for some time. My only practical reason is that Amanda supports IPv6 and I should test that support. It was also a good chance to re-immerse myself in network configuration, and Hurricane Electric has a neat certification process to add some motivation. I began by getting local IPv6 connectivity set up over the HE tunnel, using my Gentoo systems. This was fairly straightforward, as the Gentoo net scripts natively support IPv6. The firewall system I use (Shorewall) does not support IPv6 directly. Instead, there's a parallel shorewall6 package to install. Aside from the annoyance of setting up two separate firewalls, this did not cause appreciable difficulties. With all of this in place, I was at the "Explorer" level.

The next task was to set up a working IPv6 desktop. My home network uses 802.1q VLAN tagging to layer both an external, publicly routable IPv4 network (99.89.149.16/29 on VLAN 20) and an internal, NAT'd IPv4 network (172.16.1/24 on VLAN 10). I wanted to make VLAN 10 a dual-stack network, rather than invent a new VLAN for my IPv6 network. Initially, I didn't realize that HE uses, in my case, 2001:470:1f10:826::0/64 just for the tunnel (yes, two addresses out of 2⁶⁴ used -- maybe we'll need IPv8 sooner than we think!). I assumed that the /64 I was allocated was to be used for all of my nodes, and tried to subnet it locally, using 2001:470:1f10:826::0/112 for the tunnel and 2001:470:1f10::1/112 for the internal network. This worked with manual configuration, but radvd seemed to always want to advertise a /64. A little reading about the RA protocol showed this to be correct: RA provides the high 64 bits (the network portion), and the clients provide the low 64 bits using EUI-64. I was stymied until I looked at the tunnel details again and noticed that the "Routed IPv6 Prefixes" section listed a different prefix (2001:470:1f11:826/64).

With this in place, the subnet and firewall setup was a breeze. Using a manual configuration on my MacBook, I was able to communicate via IPv6. However, the stateless autoconfiguration didn't work. I briefly tried DHCPv6, but Macs do not support it. The RA client correctly combines the network and EUI-64 components to create a full address, and it correctly copies the link-local address of the router, but it does not set up a default route using that router, making the whole thing fairly useless. A trip to #ipv6 confirmed that Macs are, indeed, broken this way, so I stopped worrying about it.

The remainder of the certification process involved getting Apache, Postfix, and Bind speaking IPv6, none of which was very difficult. I did discover that BIND's $ORIGIN didn't work correctly. A zonefile with

$ORIGIN 6.2.8.0.1.1.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR knuth.r.igoro.us.

didn't work, but spelling out the entire reversed address did. I'm sure this was due to a typo, but several checks didn't reveal anything.

However, I'm now stuck at the Guru level until GoDaddy starts supporting IPv6 glue for the .us TLD. I feel cheated, somehow!

Open-Source Support?

nospam@example.com (Dustin J. Mitchell) — Sat, 16 Jun 2007 14:01:08 -0500

I saw an interesting post on the Subversion development list a while ago. In part:

This note is to inform you that the Shell Group will be migrating from Windows 2000 to Microsoft's new operating system known as Windows Vista with effect from Q1 2008, and to seek your assistance and support in minimising disruption to users and applications during and after the migration.

The note goes on to request some fairly specific information about the upgrade path for TortiseSVN, the Windows Subversion client. They are the sorts of questions that all IT shops would love to ask all of their vendors, with the expectation of a full and well-researched answer.

As an admin at a small K-12 school, questions of this sort were met with blank stares from vendors. At best, we could get a demo unit, but any sort of analysis of the potential fit of a product (besides the "analysis" the salesmen would do) was simply out of the question for an account of our size. On the other hand, I could usually count on honest assessments from open-source software mailing lists, even if they didn't represent full-scale implementation analyses.

The Shell Group request turns the situation around. Shell Group is a very large client and is probably accustomed to contacting peers like Dell, Aramark, or HBN-AMRO with requests like this. Yet here they are making these requests of a gaggle of developers, none of whom want to be "the main liaison for ALL matters pertaining to Vista compatibility." There were no on-list responses, so I can't say what became of the request.

There's clearly a business need here, but it's not the typical "sell support for open source software" niche. Rather, Shell Group wants a business entity with which they can have a more contractual relationship: one that can get the software certified by Microsoft, make projections as to deliverable dates, and so on. An entity that can answer support calls but does not have significant control of the development community is simply not capable of these things, but neither is a development community without a legally representative organization.

I'm interested to see if this kind of request occurs more often, and what effect it has on the landscape of adoption of OSS in big business.

Fact-Checking

Some basic googling for other messages like this turned up nothing. It's quite possible that this is a hoax. If so, I'm sorry for promoting it, but I think the points it brought up are interesting nonetheless.

"Teaching Problem Solving: You Can and You Should" (Elizabeth Zwicky)

nospam@example.com (Dustin J. Mitchell) — Wed, 06 Dec 2006 10:06:49 -0600

Mrs. Zwicky gave a really excellent talk that balanced real research in education, in problem solving, and in systems administration. She teaches systems administration to Navy recruits for a defense contractor, in a tutoring setting. The talk addressed the common belief that problem solving skills are essentially innate and can't be taught. She discussed the problem-solving process in general, using lots of examples (well, "war stories") from systems admin. Finally, she talked about some of the techniques and skills needed to teach problem solving (or anything, really).

These techniques included scaffolding -- building the learners' conceptual understanding by presenting the right tasks, offering the right support, and convincing the learner to talk about the concepts, not just "what do I type". Also included was "spotting", which I assume comes from sports -- the idea here is to make sure that the learner doesn't suffer any horrible consequences from making mistakes. This topic was interesting to me, as someone who bridges education, systems administration, and development. I think it's important for well-trained, intelligent people to think about and participate in education -- systems administrators included (for the record, I'm happy with the NSF's requirement that scientists do some sort of "community service" as a part of their work for a grant).

I found it relevant to me in two ways. First, the person who replaced me at my previous job is a very green admin. He's been doing basic IT legwork for a few years -- repairing computers, user support, etc. Now he's in charge of a heterogenous Linux/Windows shop with a bunch of web services, funky applications, and so on. Since he's in a production environment, questions are always "how do I do XYZ?" rather than "how does SSL work." That makes it hard to concentrate on teaching the problem-solving that underlies all of this.

I have also tried to teach varoius IT-related things to a bunch of students (programming, administration, etc.). For most, the motivation was missing, and I never figured out how to get around that. For one, though, I found that spotting was an effective way to motivate her to actually try to solve a problem, rather than just requesting and following steps. I asked her to add a printer to a Windows network, but said I wouldn't answer any questions, but would fix anything that broke while she was working on it. It took a few iterations of the assurances before she started, and it took her a while to work through the process, but she now reflects on this as the best learning experience of our time working together.

During the Q&A session for this talk, I was somewhat disappointed that all of the questions focused on Problem Solving / System Administration -- horror stories, "what kind of problem solving is this", etc. -- nobody was interested in the teaching of these skills.

USENIX LISA

nospam@example.com (Dustin J. Mitchell) — Wed, 06 Dec 2006 09:17:41 -0600

So what better time to inagurate this blog than now, during the keynote to the LISA (Large Installation Systems Administration) conference. It's an interesting general talk about one of the many pressing non-technical issues in this community: DRM and restrictive licensing.

He's boiled it down nicely: in the traditional crypto challenge, Alice talking to Bob, with Carol trying to eavesdrop, Carol has the ciphertext and the cipher, but not the key -- that's the feature that differentiates Carol from Bob, allowing Bob, but not Carol, to decrypt it. But under DRM and other legal challenges Carol and Bob are the same person. Companies are sending cyphertext to Carol/Bob, with the restriction that they can use it in one capacity (as Bob, the consumer) but not in another (as Carol, the same consumer who wants to watch the DVD on her computer). Obviously, the only technical way to make this work is to control the cipher (the algorithm). It's easy to build a cipher to do what they want. It's technically impossible to prevent others from building ciphers that don't. So they turn to the law.

This all reminds me of the "good old days" of the Apple II and crazy technical copy-protection schemes -- schemes which faded in popularity, and seem to have come back in the last decade. Doctrow's answer was, to put it simply, DMCA. DMCA gave these companies the legal muscle they needed.

This keynote highlights what I'd like to talk about on this site -- I'm not much interested in posting code samples, or kvetching about this language or that language. I'd rather talk about these much more important issues.