Code Vigorous

Dustin J. Mitchell

Four Years at Mozilla

18 Oct 2014

I started at Mozilla on October 18, 2010, four years ago today. This is the longest I’ve stayed at any (full-time) job before, which might seem a little weird but is fairly common in this industry. Still, I feel like itinerant employment is a little selfish – stay at a company only until the winds change or a better thing comes along, so there’s not much incentive to try to improve things within a company. That’s probably reasonable at the typical corporation where the mission is to make as much money as possible – the corporation feels no loyalty to me, so why should I be loyal? But Mozilla’s fundamentally different: we have an important mission that I care about, and that I have the skills to advance.

So when I started here, I comitted to stay as long as possible - and that’s still my intent (this isn’t a farewall post!). This was made a little easier by Mozilla’s ease of movement within the company: I moved from Release Engineering to IT, and then into Platform Engineering Operations (as once again a part of Release Engineering).

Even in just four years, things change.

  • When I started, my approach to systems administration was to work on individual hosts, always documenting things and tracking bugs. Now hosts are disposable (Luke Kanies describes the difference as “pets to cattle”) and configured automatically.
  • I’ve learned Puppet and contributed to the project.
  • I introduced and managed devleopment of a configuration management system for release engineering, PuppetAgain.
  • I am working on a new system to manage firewall configurations (fwunit).
  • I’ve expanded my AWS experience from just S3 to include EC2, Elastic Beanstalk, and VPCs.
  • I’ve gotten my hands on Juniper switch and firewall configuration, and how to reason about network flows.
  • I spent some time working with the database engineering group, learning a thing or two about what makes databases and DBAs tick.
  • I was promoted to a “Staff” level, which involves autonomously driving projects (like PuppetAgain or fwunit), coordinating with other groups, mentoring, and building my own credibility within and outside the organization. I still have lots of work to do on all of those things!

The world has changed during this time, too:

  • Desktop computing is squarely on the decline, and mobile is the future.
  • Widely available system- and container-related virtualization have radically changed the way we think of a system.
  • Running an email server is no longer “fun” – it’s a struggle to keep up with filtering the incoming spam, and being filtered by other email destinations.
  • The Web is no longer just HTTP: it’s apps, walled gardens (again!), social networks, and bucketloads of amorphous drivel we call “content”.

And Mozilla has changed, too:

  • Firefox OS is a real thing – we’re not just building an app on the second-place platform, we’re building the Web the way we want it to be.
  • When I was hired, it had been 9 months since the last Firefox release (3.6), and the next release (4.0) wasn’t made for another five months. Now we ship a new Firefox every six weeks, with dozens of releases in between (chemspills, Aurora and Beta, Thunderbird, B2G, and so on).
  • The foundation has branched out into journalism, science, and education – following the increasingly deep penetration of the Web into these areas.
  • We’ve introduced new tools and paradigms to the Web, including Do Not Track, federated identity (Persona), and Lightbeam.
  • When I was hired, our web presence was chiefly comprised of Firefox downloads, addons, and marketing sites. Now we have a really substantial cloud-services infrastructure supporting the Firefox OS Marketplace, Firefox Accounts, and more to come.
  • As a company, we’ve grown massively - there were fewer than 300 employees when I was hired, along with a “we’re all in this together” attitude – even if things sometimes didn’t get done due to limited resources.
  • As a result of the growth, we’ve subdivided and specialized quite a bit, both in engineering and operations. This means we have world-class subject-matter experts on many topics, but it also means that projects require time and resource commitments from dozens of people.
  • I’ve outlasted two CEOs (although again, 1-2 years is typical in the industry so this isn’t surprising).

What might be in the future?

I’m no futurist, but here goes. I expect to learn a lot more about cloud computing environments – and probably build one or two. If history is any indication, I’ll learn a few more programming languages – it looks like Lua is next, since we’ll be adopting Heka. At some point – the sooner the better! – systems that are secure by design will gain a competitive advantage and become more common. I think we’ll see some of the “wild west” nature of the Internet – from spam and fraud to cyberwarfare (ugh, I hate that term) – tamed, both by political and technical means. As for Mozilla, I think we’re already seeing a return to our roots, after an awkward attempt to fit into the suit of a mobile company. We’ll shed some of the complexity and redundancy that we’ve built up, and find ways to be more inclusive of non-employees in our core activities.