Blog

(No this is not a post about commercialization of AI)

It occurred to me today that there is an interesting parallel between AI and capitalism. They're both tools that can be used to accomplish things very effectively. And they both have "alignment" issues.

Capitalism provides incentives and a free market, and that can result in extremely efficient action because people are trying to maximize profit. However, left unchecked, this can produce all kinds of negative externalities which are not really what we want. This is the alignment issue - we need to be diligent in "aligning" capitalism so that it incentivizes and produces outcomes that are actually what we want. Primarily governments do this with the use of taxes and subsidies - make bad behaviour (produces negative externalities) more expensive, and make good behaviour (produces positive externalities) less expensive. In my view, this is the primary function of a government (at least in a capitalist society). However, I think most people would agree we haven't yet perfected this part of things, and we still suffer from all kinds of negative externalities.

AI/AGI is very similar - it can produce extremely efficient action, but suffers from an alignment problem in that what it does may not be exactly what we want it to do. Lots has already been written about the AI alignment problem.

My main point here is that we, as a society, haven't even figured out how to align capitalism reliably. Which makes me a bit worried about how we're going to handle AGI. We should probably think hard about why we haven't been able to align capitalism perfectly (e.g. maybe because not all subsets of people want the same thing?) and see if those reasons carry over to AGI as well.

[ 2 Comments... ]

An author of popular free software packages intentionally inserted infinite loops into his code to break downstream users, as a form of protest. It's not the first time this sort of thing has happened, and it won't be the last. In this particular case I would say that both primary parties are in the wrong.

To the author: if you make your software freely available, obviously people are going to use it without paying for it. Why would you expect anything else?

To the users: if you use some random piece of software in your code without verifying it and just keep blindly updating it, obviously stuff like this is going to happen. Why would you expect anything else?

To me it seems like there's no strongly defined interface here, on either side. So you have to be prepared to accept any kind of behavior from the other party. What's surprising is that this doesn't happen more frequently.

What I would like to see is some mechanism for a formal "pay what you can" style of money transfer. As an author of various open source packages, I do it mostly for personal pleasure, but I also wouldn't be opposed to getting money from those that profit from the work. This is not my primary concern, but if it were easy to set up, why not? On the flip side, as a professional software developer working for a profitable company, I would love to be able to support the (large amounts of) open source work we rely on.

Approaches like Patreon are fine for "artisanal" stuff but they don't scale. What I want to be able to do is take a budget of any amount, say $1m and distribute it programmatically across the free software projects that we use. Ideally with most of it actually going to the project as opposed to getting sucked into transaction fees or middlemen.

So here's a strawman proposal: open source projects that are accepting donations should include a DONATIONS.txt file in their source tree, similar to README.txt, that provides a machine-readable money transfer address. These files can be scraped, collated, and funded by any consumer of the project.

The hard part, of course, is the "machine-readable transfer address" part. Obviously cryptocurrency seems like a good fit for this problem. It's trivial to create a wallet and let it accumulate donations, and only go through the hassle of extracting the value in fiat if there's enough to be worthwhile. And it works at internet scale, that is to say, regardless of what geopolitical area of the world the parties are in.

For people opposed to cryptocurrencies: sure, there are good reasons to be sceptical. But I challenge you offer a viable alternative instead of shooting down this proposal merely on the grounds that it involves cryptocurrencies.

[ 1 Comment... ]

I should probably note here that November 20 was my last day as a Mozilla employee. In theory, that shouldn't really change much, given the open-source nature of Mozilla. In practice, of course, it does. I did successfully set up a non-staff account and migrate things to that, so I still retain some level of access. I intend to continue contributing; however, my contributions will likely be restricted to things that don't require paging in huge chunks of code, or require large chunks of time. In other words, mostly cleanup-type stuff, or smaller bugfixes/enhancements.

I still believe that the Mozilla mission to make the Internet healthier is important, but over the course of the past year, I've come to realize that there are other problems facing society are perhaps more important and fundamental. That, combined with more companies opening up remote positions, provided me with an opportunity that I decided to take. In January, I'll be starting work on the Cash App Platform team at Square, and hopefully will be able to help them move the needle on economic empowerment.

Working at Mozilla was in many ways a dream come true. It was truly an honour to work alongside so many world-class engineers, on so many different problems. I'm going to miss it, for sure, but I am also excited to see what the future holds.

A final note: if you follow this blog via Planet Mozilla, keep in mind that only posts I tag as "mozilla" show up there, and those posts will be much fewer in number going forward (not that they were particularly numerous before...). I have an RSS feed (yes, RSS is still a thing) if you care to follow that.

[ 0 Comments... ]

Don't panic.

I spent a good chunk of this year fiddling with taskcluster configurations in order to get various bits of continuous integration stood up for WebRender. Taskcluster configuration is very flexible and powerful, but can also be daunting at first. This guide is intended to give you a mental model of how it works, and how to add new jobs and modify existing ones. I'll try and cover things in detail where I believe the detail would be helpful, but in the interest of brevity I'll skip over things that should be mostly obvious by inspection or experimentation if you actually start digging around in the configurations. I also try and walk through examples and provide links to code as much as possible.

Based on my experience, there are two main kinds of changes most Gecko hackers might want to do:
(1) modify existing Gecko test configurations, and
(2) add new jobs that run in automation.
I'm going to explain (2) in some detail, because on top of that explaining (1) is a lot easier.

Overview of fundamentals

The taskcluster configuration lives in-tree in the taskcluster/ folder. The most interesting subfolders there are the ci/ folder, which contain job defintiions in .yml files, and the taskgraph/ folder which contain python scripts to do transforms. A quick summary of the process is that the job definitions are taken from the .yml files, run through a series of "transforms", finally producing a task definition. The task definitions may have dependencies on other task definitions; together this set is called the "task graph". That is then submitted to Taskcluster for execution. Conceptually you can think of the stuff in the .yml files as a higher-level definition, which gets "compiled" down to the final taskgraph (the "machine code") that Taskcluster actually understands and executes.

It's helpful to walk through a quick example of a transform. Consider the webrender-linux-release job definition. At the top of the kind.yml file, there are a number of transforms listed, so each of those gets applied to the job definition in turn. The first one is use_toolchains, the code for which you can find in use_toolchains.py. This transform takes the toolchains attributes of the job definition (in this example, these attributes), and figures out the corresponding toolchain jobs and artifacts (artifacts are files that are outputs of a task), The toolchain jobs are defined in taskcluster/ci/toolchains/, so we can see that the linux64-rust toolchain is here and the wrench-deps toolchain is here. The transform code discovers this, then adds dependencies for those tasks to webrender-linux-release, and populates the MOZ_TOOLCHAINS env var with the links to the artifacts. Taskcluster ensures that tasks only get run after their dependencies are done, and the MOZ_TOOLCHAINS is used by ./mach artifact toolchain to actually download and unpack those toolchains when the webrender-linux-release task runs.

Similar to this, there are lots of other transforms that live in taskcluster/taskgraph/transforms/ that do other transformations on the job definitions. Some of the job definitions in the .yml files look very different in terms of what attributes are defined, and go through many layers of transformation before they produce the final task definition. In a sense, each folder in taskcluster/ci is a domain-specific language, with the transforms eventually compiling them down to the same machine code.

One of the more important transforms is the job transform, which determines which wrapper script will be used to run your job's commands. This is usually specified in your job definition using the run.using attribute. Examples include mach or toolchain-script. These both get transformed (see transforms for mach and toolchain-script) into commands that get passed to run-task. Or you can just use run-task directly in your job, and specify the command you want to have run. In the end most jobs will boil down to run-task commands; the run-task wrapper script just does some basic abstraction over the host machine/VM and then runs the command.

Host environment and Docker

Taskcluster can manage many different underlying host systems - from Amazon Linux VMs, to dedicated physical macOS machines, and everything in between. I'm not going to into details of provisioning but there's the notion of a "worker" which is useful to know. This is the binary that runs on the host system, polls taskcluster to find new tasks scheduled to be run on that kind of host system, and executes them in whatever sandboxing is available on that host system. For example, a docker-worker instance will start a specified docker image and execute the task commands in that. A generic-worker instance will instead create a dedicated work folder and run stuff in there, cleaning up afterwards.

If you're adding a new job (e.g. some sort of code analysis or whatever) most likely you should run on Linux using docker. This means you need to specify a docker image, which you can also do using the taskcluster configuration. Again I will use the webrender-linux-release job as an example. It specifies the webrender docker image to use, which is defined with an in-tree Dockerfile. The docker image is itself built using taskcluster with the job definition here (this job definition is an example of one that looks very different from other job definitions, because the job description is literally two lines and transforms do most of the work in compiling this into a full task definition).

Circling back to generic-worker, this is the default for jobs that need to run on Windows and macOS, because Docker doesn't run either of those platforms as a target. An example is the webrender-macos-debug job, which specifies using: run-task on a t-osx-1010 worker type, which will cause it go through this transform and eventually run using the run-task wrapper under generic worker on the macOS instance. For the most part you probably won't need to care about this but it's something to be aware of if you're running jobs targetted at Windows or macOS.

Caching

As we've seen from previous sections, jobs can use docker images and toolchain artifacts that are produced by other tasks in the taskgraph. Of course, we don't want to rebuild these docker images and toolchains on every single push, as that would be quite expensive. Taskcluster provides a mechanism for caching and reusing artifacts from previous pushes. I'm not going to go into too much detail on this, but you can look at the cached_tasks transform if you're interested. I will, however, point to the %include comments in e.g. this Dockerfile and note that these include statements are special because the mark the docker image as dependent on the given file. So if the included file changes, the docker image will be rebuilt. For toolchain tasks you can specify additional dependencies on inputs using the resources attribute; this also triggers toolchain rebuilds if the dependent inputs change.

The other thing to keep in mind when adding a new job is that you want to avoid too much network traffic or redundant work. So if your job involves downloading or building stuff that usually doesn't change from one push to the next, you probably want to split up your job so that the mostly-static part is done by a toolchain or other job, and the result of that is cached and reused by your main job. This will reduce overall load and also improve the runtime of your per-push job.

Even if you don't need caching across pushes, you might want to refactor two jobs so that their shared work is extracted into a dependency, and the two dependent jobs then just do their unique postprocessing bits. This can be done by manually specifying the dependencies and pulling in artifacts from those dependencies using the fetches attribute. See here for an example. In this scenario taskcluster will again ensure the jobs run in the right order, and you can use artifacts from the dependencies, but no caching across pushes takes place.

Adding new jobs

So hopefully with the above sections you have a general idea of how taskcluster configuration works in mozilla-central CI. To add a new job, you probably want to find an existing job kind that fits what you want, and then add your job to that folder, possibly by copy/pasting an existing job with appropriate modifications. Or if you have a new type of job you want to run that's significantly different from existing ones, you can add a new kind (a new subfolder in taskcluster/ci and documented in kinds.rst). Either way, you'll want to ensure the transforms being used are appropriate and allow you to reuse the features (e.g. toolchain dependencies) that you need and that already exist.

Gecko testing

The Gecko test jobs are defined in the taskcluster/ci/test/ folder. The entry point, as always, is the kind.yml file in that folder, which lists the transforms that get applied. The tests transform is one of the largest and most complex transforms. It does a variety of things (e.g. generating fission-enabled and fission-disabled tasks for jobs), but thankfully you probably won't need to fiddle with that too much, unless you find your test suite is behaving unexpectedly. Instead, you can mostly do copy-pasting in the other .yml files to enable test suites on particular platforms or adjust options. The test-platforms.yml file allows you define "test platforms" which show up as new rows on TreeHerder and run sets of tests on a particular build. The sets of tests are defined in test-sets.yml, which in turn reference the individual test jobs defined in the various other .yml files in that folder. Enabling a test suite on a platform is generally as easy as adding the test to the test set that you care about, and maybe tweaking some of the per-platform test attributes (e.g. number of chunks, or what trees to run on) to suit your new platform. I found the .yml files to mostly self-explanatory so I won't walk through any examples here.

What I will briefly mention is how the tests are actually run. This is not strictly part of taskcluster but is good to know anyway. The test tasks generally run using mozharness, which is a set of scripts in testing/mozharness/scripts and configuration files in testing/mozharness/configs. Mozharness is responsible for setting up the firefox environment and then delegates to the actual test harness. For example, running reftests on desktop linux would run testing/mozharness/scripts/desktop_unittest.py with the testing/mozharness/configs/unittests/linux_unittest.py config (which are indicated in the job description here). The config file, among other things, tells the mozharness script where the actual test harness entrypoint is (in this case, runreftest.py) and the mozharness script will invoke that test harness after doing some setup. There's many layers here (run-task, mozharness, test harness, etc.) with the number of layers varying across platforms (mozharness scripts for Android device/emulator are totally different than desktop), and I don't have as good a grasp on all this as I would like, but hopefully this is sufficient to point you in the right direction if you need to fiddle with tests at this level.

Debugging

As always, when modifying configs you might run into unexpected problems and need to debug. There are a few tools that are useful here. One is the ./mach taskgraph command, which can run different steps of the "decision" task and generate taskgraphs. When trying to debug task generation issues my go-to technique would be to download a parameters.yml file from an existing decision task on try or m-c (you can find it in the artifacts list for the decision task on TreeHerder), and then run ./mach taskgraph target-graph -p parameters.yml. This runs the taskgraph code and emits a list of tasks that would be scheduled given the taskcluster configuration in your local tree and the parameters provided. Likewise, ./mach taskcluster-build-image and ./mach taskcluster-load-image are useful for building and testing docker images for use with jobs. You can use these to e.g. run a docker image with your local Docker installation, and see what all you might need to install on it to make it ready to run your job.

Another useful debugging tool as you start doing try pushes to test your new tasks, is the task/group inspector at tools.taskcluster.net. This is easily accessible from TreeHerder by clicking on a job and using the "Inspect task" link in the details pane (bottom left). TreeHerder provides some information, but the Taskcluster tools website provides a much richer view including the final task description that was submitted to Taskcluster, its dependencies, environment variables, and so on. While TreeHerder is useful for looking at overall push health, the Taskcluster web UI is better for debugging specific task problems, specially as you're in the process of standing up a new task.

In general the taskcluster scripts are pretty good at identifying errors and printing useful error messages. Schemas are enforced, so it's rare to run into silent failures because of typos and such.

Conclusion

There's a lot of power and flexibility afforded by Taskcluster, but with that goes a steep learning curve. Thankfully once you understand the basic shape of how Taskcluster works, most of the configuration tends to be fairly intuitive, and reading existing .yml files is a good way to understand the different features available. grep/searchfox in the taskcluster/ folder will help you out a lot. If you run into problems, there's always people willing to help - as of this writing, :tomprince is the best first point of contact for most of this, and he can redirect you if needed.

[ 0 Comments... ]

I tend to do a lot of try pushes for testing changes to Gecko and other stuff, and by using one of TreeHerder's (apparently) lesser-known features, managing these pushes to see their results is really easy. If you have trouble managing your try pushes, consider this:

Open a tab with an author filter for yourself. You can do this by clicking on your email address on any of your try pushes (see highlighted area in screenshot below). Keep this tab open, forever. By default it shows you the last 10 try pushes you did, and if you leave it open, it will auto-update to show newer try pushes that you do.

With this tab open, you can easily keep an eye on your try pushes. Once the oldest try pushes are "done" (all jobs completed, you've checked the result, and you don't care about it anymore), you can quickly and easily drop it off the bottom by clicking on the "Set as bottom of range" menu item on the oldest push that you do want to keep. (Again, see screenshot below).

 
(click to embiggen)

This effectively turns this tab into a rotating buffer of the try pushes you care about, with the oldest ones moving down and eventually getting removed via use of "Set as bottom of range" and the newer ones automatically appearing on top.

Note: clicking on the "Set as bottom of range" link will also reload the TreeHerder page, which means errors that might otherwise accumulate (due to e.g. sleeping your laptop for a time, or a new TreeHerder version getting deployed) get cleared away, so it's even self-healing!

Bonus tip: Before you clear away old try pushes that you don't care about, quickly go through them to make sure they are all marked "Complete". If they still have jobs running that you don't care about, do everybody a favor and hit the push cancellation button (the "X" icon next to "View Tests") before resetting the bottom of range, as that will ensure we don't waste machine time running jobs nobody cares about.

Extra bonus tip: Since using this technique this makes all those "Thank you for your try submission" Taskcluster emails redundant, set up an email filter to reroute those emails to the /dev/null of your choice. Less email results in a happier you!

Final bonus tip: If you need to copy a link to a specific try push (for pasting in a bug, for example), right-click on the timestamp for that try push (to the left of your email address), and copy the URL for that link. That link is for that specific push, and can be shared to get the desired results.

And there you have it, folks, a nice simple way to manage all your try pushes on a single page and not get overwhelmed.

[ 4 Comments... ]

I recently migrated a bunch of stuff (including this website) to Amazon EC2, running on a FEMP (FreeBSD, nginx, MySQL, PHP) stack. I had to fiddle with a few things to get it running smoothly, and wanted to document the steps in case anybody else is trying to do this (or I need to do it again later). This assumes you have an Amazon AWS account and some familiarity with how to use it.

Before you start

Ensure you know what region and instance type you want. I used the Canada (Central) region but it should work the same in any other region. And I used a t2.micro instance type because I have a bunch of stuff running on the instance, but presumably you could use a t2.nano type if you wanted to go even lighter. Also, I'm using Amazon Route53 to handle the DNS, but if you have DNS managed separately that's fine too.

Upload your SSH public key

In the EC2 dashboard, select "Key Pairs" under the "Network and Security" section in the left pane. Click on "Import Key pair" and provide the public half of your SSH keypair. This will get installed into the instance so that you can SSH in to the instance when it boots up.

Create the instance

Select "Instances" in the EC2 dashboard, and start the launch wizard by clicking "Launch Instance". You'll find the FreeBSD images under "Community AMIs" if you search for FreeBSD using the search. Generally you want to grab the most recent FreeBSD release you can find (note: the search results are not sorted by recency). If you want to make sure you're getting an official image, head over to the freebsd-announce mailing list, and look for the most recent release announcement email. As of this writing it is 11.2-RELEASE. (Note: be sure to use a -RELEASE version, not a -STABLE version). The email should contain AMI identifiers for all the different EC2 regions; for example the Canada AMI for 11.2-RELEASE is ami-a2f97bc6. Searching for that in the launch wizard finds it easily.

Next step is to select the instance type. Select one that's appropriate for your needs (I'm using t2.micro). The next step is to configure instance details. Unless you have specific changes you want to make here you can leave this with the default settings. Next you have to choose the root volume size. With my instances I like using a 10 GB root volume for the system and swap, and using a separate EBS volume for the "user data" (home folders and whatnot). For reference my 10G root volume is currently 54% full with the base system, extra packages, and a 2G swap file.

After that you can add tags if you want, change the security groups, and finally review everything. I just go with the defaults for the security groups, since it allows external SSH access and you can always change it later. Finally, time to launch! In the launch dialog you select the keypair from the previous step and you're off to the races.

Logging in

Once the instance is up, the EC2 console should display the public IP address. You'll need to log in with the user ec2-user at that IP address, using the keypair you selected previously. If you're paranoid about security (and you should be), you can verify the host key that SSH shows you by selecting the instance in the EC2 console, going to Actions -- Instance Settings -- Get Instance Screenshot. The screenshot should display the host keys as shown on the instance itself, and you can compare it to what SSH is showing to ensure you're not getting MITM'd.

Initial housekeeping

This part is sort of optional, but I like having a reasonable hostname and shell installed before I get to work. I'm going to use jasken.example.com as the hostname and I like using bash as my default shell. To do this, run the following commands:

su                                  # switch to root shell
sysrc hostname="jasken.example.com" # this modifies /etc/rc.conf
pkg update                          # update package manager
pkg install -y vim bash             # install useful packages
chsh -s /usr/local/bin/bash root    # change shell to bash for root and ec2-user
chsh -s /usr/local/bin/bash ec2-user

At this point I also like to reboot the machine (pretty much the only time I ever have to) because I've found that not everything picks up the hostname change if you change it via the hostname command while the instance is running. So run reboot and log back in once the instance is back up. The rest of the steps need root access so go ahead and su to root once you're back in.

IPv6 configuration

While you're rebooting, you can also set up IPv6 support. FreeBSD has everything built-in, you just need to fiddle with the VPC settings in AWS to get an IP address assigned. Note the VPC ID and Subnet ID in your instance's details, and then go to the VPC dashboard (it's a separate AWS service, not inside EC2). Find the VPC your instance is in, then go to Actions -- Edit CIDRs. Click on the "Add IPv6 CIDR" button and then "Close". Still in the VPC dashboard, select "Subnets" from the left panel and select the subnet of your instance. Here again, go to Actions -- Edit IPv6 CIDRs, and then click on "Add IPv6 CIDR". Put "00" in the box that appears to fill in the IPv6 subnet and hit ok.

Next, go to the "Route Tables" section of the VPC dashboard, and select the route table for the VPC. In the Routes tab, add a new route with destination ::/0 and the same gateway as the 0.0.0.0/0 entry. This ensures that outbound IPv6 connections will use the external network gateway.

Finally, go back to the EC2 dashboard, select your instance, and go to Actions -- Networking -- Manage IP addresses. Under IPv6 addresses, click "Assign new IP" and "Yes, Update" to auto-assign a new IPv6 address. That's it! If you SSH in to the instance you should be able to ping6 google.com successfully for example. It might take a minute or so for the connection to start working properly.

Installing packages

For the "EMP" part of the FEMP stack, we need to install nginx, mysql, and php. Also because we're not barbarians we're going to make sure our webserver is TLS-enabled with a Let's Encrypt certificate that renews automatically, for which we want certbot. So:

pkg install -y nginx mysql56-server php56 php56-mysql php56-mysqli php56-gd php56-json php56-xml php56-dom php56-openssl py36-certbot

Note that the set of PHP modules you need may vary; I'm just listing the ones that I needed, but you can always install/uninstall more later if you need to.

PHP setup

To use PHP over CGI with nginx we're going to use the php-fpm service. Instead of having the service listen over a network socket, we'll have it listen over a file socket, and make sure PHP and nginx are in agreement about the info passed back and forth. The sed commands below do just that.

cd /usr/local/etc/
sed -i "" -e "s#listen = 127.0.0.1:9000#listen = /var/run/php-fpm.sock#" php-fpm.conf
sed -i "" -e "s#;listen.owner#listen.owner#" php-fpm.conf
sed -i "" -e "s#;listen.group#listen.group#" php-fpm.conf
sed -i "" -e "s#;listen.mode#listen.mode#" php-fpm.conf
sed -e "s#;cgi.fix_pathinfo=1#cgi.fix_pathinfo=0#" php.ini-production > php.ini
sysrc php_fpm_enable="YES"
service php-fpm start

MySQL setup

This is really easy to set up. The hard part is optimizing the database for your workload, but that's outside the scope of my knowledge and of this tutorial.

sysrc mysql_enable="YES"
service mysql-server start
mysql_secure_installation   # this is interactive, you'll want to set a root password
service mysql-server restart

Swap space

MySQL can eat up a bunch of memory, and it's good to have some swap set up. Without this you might find, as I did, that weekly periodic tasks such as rebuilding the locate database can result in OOM situations and take down your database. On a t2.micro instance which has 1GB of memory, a 2GB swap file works well for me:

# Make a 2GB (2048 1-meg blocks) swap file at /usr/swap0
dd if=/dev/zero of=/usr/swap0 bs=1m count=2048
chmod 0600 /usr/swap0
# Install the swap filesystem
echo 'md99 none swap sw,file=/usr/swap0,late 0 0' >> /etc/fstab
# and enable it
swapon -aL

You can verify the swap is enabled by using the swapinfo command.

nginx setup

Because the nginx config can get complicated, specially if you're hosting multiple websites, it pays to break it up into manageable pieces. I like having an includes/ folder which contains snippets of reusable configuration (error pages, PHP stuff, SSL stuff), and a sites-enabled/ folder that has a configuration per website you're hosting. Also, we want to generate some Diffie-Hellman parameters for TLS purposes. So:

cd /usr/local/etc/nginx/
openssl dhparam -out dhparam.pem 4096
mkdir includes
cd includes/
# This creates an error.inc file with error handling snippet
cat >error.inc <<'END'
error_page 500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/local/www/nginx-dist;
}
END
# PHP snippet
cat >php.inc <<'END'
location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass   unix:/var/run/php-fpm.sock;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $request_filename;
    include        fastcgi_params;
}
END
# SSL snippet
cat >ssl.inc <<'END'
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /usr/local/etc/nginx/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
END

If you want to fiddle with the above, feel free. I'm not a security expert so I don't know what a lot of the stuff in the ssl.inc does, but based on the Qualys SSL test it seems to provide a good security/compatibility tradeoff. I mostly cobbled it together from various recommendations on the Internet.

Finally, we set up the server entry (assuming we're serving up the website "example.com") and start nginx:

cd /usr/local/etc/nginx/
cat >nginx.conf <<'END'
user  www;
worker_processes  1;

error_log  /var/log/nginx/error.log info;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;
    resolver 127.0.0.1;

    include  sites-enabled/example;
}
END
mkdir sites-enabled
cd sites-enabled/
cat >example <<'END'
server {
    listen       [::]:80;
    listen       80;
    server_name  example.com;
    root /usr/local/www/nginx;
    index  index.php index.html index.htm;

    include includes/php.inc;

    location / {
        try_files $uri $uri/ =404;
    }

    include includes/error.inc;
}
END
sysrc nginx_enable="YES"
service nginx start

Open up ports

The nginx setup above is sufficient to host an insecure server on port 80, which is what we need in order to get the certificate that we need to enable TLS. So at this point go to your DNS manager, wherever that is, and point the A and AAAA records for "example.com" (or whatever site you're hosting) to the public IP addresses for your instance. Also, go to the "Security Groups" pane in the EC2 dashboard and edit the "Inbound" tab for your instance's security group to allow HTTP traffic on TCP port 80 from source 0.0.0.0/0, ::/0, and the same for HTTPS traffic on TCP port 443.

After you've done that and the DNS changes have propagated, you should be able to go to http://example.com in your browser and get the nginx welcome page, served from your very own /usr/local/www/nginx folder.

TLS

Now it's time to get a TLS certificate for your example.com webserver. This is almost laughably easy once you have regular HTTP working:

certbot-3.6 certonly --webroot -n --agree-tos --email 'admin@example.com' -w /usr/local/www/nginx -d example.com
crontab <(echo '0 0 1,15 * * certbot-3.6 renew --post-hook "service nginx restart"')

Make sure to replace the email address and domain above as appropriate. This will use certbot's webroot plugin to get a Let's Encrypt TLS cert and install it into /usr/local/etc/letsencrypt/live/. It also installs a cron job to automatically attempt renewal of the cert twice a month. This does nothing if the cert isn't about to expire, but otherwise renews it using the same options as the initial request. The final step is updating the sites-enabled/example config to redirect all HTTP requests to HTTPS, and use the aforementioned TLS cert.

cd /usr/local/etc/nginx/sites-enabled/
cat >example <<'END'
server {
    listen       [::]:80;
    listen       80;
    server_name  example.com;
    return 301 https://$host$request_uri;
}

server {
    listen       [::]:443;
    listen       443;
    server_name  example.com;
    root   /usr/local/www/nginx;
    index  index.php index.html index.htm;

    include includes/ssl.inc;
    ssl_certificate /usr/local/etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /usr/local/etc/letsencrypt/live/example.com/privkey.pem;

    include includes/php.inc;

    location / {
        try_files $uri $uri/ =404;
    }

    include includes/error.inc;
}
END
service nginx restart

And that's all, folks!

Parting words

The above commands set things up so that they persist across reboots. That is, if you stop and restart the EC2 instance, everything should come back up enabled. The only problem is that if you stop and restart the instance, the IP address changes so you'll have to update your DNS entry.

If there's commands above you're unfamiliar with, you should use the man pages to read up on them. In general copying and pasting commands from some random website into a command prompt is a bad idea unless you know what those commands are doing.

One thing I didn't cover in this blog post is how to deal with the daily emails that FreeBSD will send to the root user. I also run a full blown mail gateway with postfix and I plan to cover that in another post.

[ 2 Comments... ]

I previously wrote about setting up multiple VLANs to segment your home network and improve the security characteristics. Since then I've added more devices to my home network, and keeping everything in separate VLANs was looking like it would be a hassle. So instead I decided to put everything into the same VLAN but augment the router's firewall rules to continue restricting traffic between "trusted" and "untrusted" devices.

The problem is that didn't work. I set up all the firewall rules but for some reason they weren't being respected. After (too much) digging I finally discovered that you have to install the kmod-ebtables package to get this to actually work. Without it, the netfilter code in the kernel doesn't filter traffic between hosts on the same VLAN and so any rules you have for that get ignored. After installing kmod-ebtables my firewall rules started working. Yay!

Along the way I also discovered that OpenWRT is basically dead now (they haven't had a release in a long time) and the LEDE project is the new fork/successor project. So if you were using OpenWRT you should probably migrate. The migration was relatively painless for me, since the images are compatible. (Edited 06-Jan-2019: LEDE has merged back into OpenWRT since I wrote this)

There's one other complication that I've run into but haven't yet resolved. After upgrading to LEDE and installing kmod-ebtables, for some reason I couldn't connect between two FreeBSD machines on my network via external IP and port forwarding. The setup is like so:

• Machine A has internal IP address 192.168.1.A
• Machine B has internal IP address 192.168.1.B
• The router's external IP address is E
• The router is set to forward port P to machine A
• The router is set to forward port Q to machine B

Now, from machine B, if connect to E:P, it doesn't work. Likewise, from machine A, connecting to E:Q doesn't work. I can connect using the internal IP address (192.168.1.A:P or 192.168.1.B:Q) just fine; it's only the via the external IP that it doesn't work. All the other machines on my network can connect to E:P and E:Q fine as well. It's only machines A and B that can't talk to each other. The thing A and B have in common is they are running FreeBSD; the other machines I tried were Linux/OS X.

Obviously the next step here is to fire up tcpdump and see what's going on. Funny thing is, when I run tcpdump on my router, the problem goes away and the machines can connect to each other. So there's that. I'm sure with more investigation I'll get to the bottom of this but for now I've shelved it under "mysteries that I can work around easily". If anybody has run into this before I'd be interested in hearing about it.

Also if anybody knows of good tools to visualize and debug iptables rules I'd be interested to try them out, because I haven't found anything good yet. I've been using the counters in the tables to try and figure out which rules the packets are hitting but since I'm debugging this "live" there's a lot of noise from random devices and the counters are not as reliable as I'd like.

[ 0 Comments... ]

The web as we know it basically runs on advertising. Which is not really great, for a variety of reasons. But charging people outright for content doesn't work that great either. How about bitcoin mining instead?

Webpages can already run arbitary computation on your computer, so instead of funding themselves through ads, they could instead include a script that does some mining client-side and submits the results back to their server. Instead of paying with dollars and cents you're effectively paying with electricity and compute cycles. Seems a lot more palatable to me. What do you think?

[ 18 Comments... ]

When typing on a laptop keyboard, I find that my posture tends to get very closed and hunched. To fix this I resurrected an old low-tech solution I had for this problem: using two keyboards. Simply plug in an external USB keyboard, and use one keyboard for each hand. It's like a split keyboard, but better, because you can position it wherever you want to get a posture that's comfortable for you.

I used to do this on a Windows machine back when I was working at RIM and it worked great. Recently I tried to do it on my Mac laptop, but ran into the problem where the modifier state from one keyboard didn't apply to the other keyboard. So holding shift on one keyboard and T on the other wouldn't produce an uppercase T. This was quite annoying, and it seems to be an OS-level thing. After some googling I found Karabiner which solves this problem. Well, really it appears to be a more general keyboard customization tool, but the default configuration also combines keys across keyboards which is exactly what I wanted. \o/

Of course, changing your posture won't magically fix everything - moving around regularly is still the best way to go, but for me personally, this helps a bit :)

[ 0 Comments... ]

At some point in the past, I learned about the difference between frameworks and libraries, and it struck me as a really important conceptual distinction that extends far beyond just software. It's really a distinction in process, and that applies everywhere.

The fundamental difference between frameworks and libraries is that when dealing with a framework, the framework provides the structure, and you have to fill in specific bits to make it apply to what you are doing. With a library, however, you are provided with a set of functionality, and you invoke the library to help you get the job done.

It may not seem like a very big distinction at first, but it has a huge impact on various properties of the final product. For example, a framework is easier to use if what you are trying to do lines up with the goal the framework is intended to accomplish. The only thing you need to do is provide (or override) specific things that you need to customize, and the framework takes care of the rest. It's like a builder building your house, and you picking which tile pattern you want for the backsplash. With libraries there's a lot more work - you have a Home Depot full of tools and supplies, but you have to figure out how to put them together to build a house yourself.

The flip side, of course, is that with libraries you get a lot more freedom and customizability than you do with frameworks. With the house analogy, a builder won't add an extra floor for your house if it doesn't fit with their pre-defined floorplans for the subdivision. If you're building it yourself, though, you can do whatever you want.

The library approach makes the final workflow a lot more adaptable when faced with new situations. Once you are in a workflow dictated by a framework, it's very hard to change the workflow because you have almost no control over it - you only have as much control as it was designed to let you have. With libraries you can drop a library here, pick up another one there, and evolve your workflow incrementally, because you can use them however you want.

In the context of building code, the *nix toolchain (a pile of command-line tools that do very specific things) is a great example of the library approach - it's very adaptable as you can swap out commands for other commands to do what you need. An IDE, on the other hand, is more of a framework. It's easier to get started because the heavy lifting is taken care of, all you have to do is "insert code here". But if you want to do some special processing of the code that the IDE doesn't allow, you're out of luck.

An interesting thing to note is that usually people start with frameworks and move towards libraries as their needs get more complex and they need to customize their workflow more. It's not often that people go the other way, because once you've already spent the effort to build a customized workflow it's hard to justify throwing the freedom away and locking yourself down. But that's what it feels like we are doing at Mozilla - sometimes on purpose, and sometimes unintentionally, without realizing we are putting on a straitjacket.

The shift from Bugzilla/Splinter to MozReview is one example of this. Going from a customizable, flexible tool (attachments with flags) to a unified review process (push to MozReview) is a shift from libraries to frameworks. It forces people to conform to the workflow which the framework assumes, and for people used to their own customized, library-assisted workflow, that's a very hard transition. Another example of a shift from libraries to frameworks is the bug triage process that was announced recently.

I think in both of these cases the end goal is desirable and worth working towards, but we should realize that it entails (by definition) making things less flexible and adaptable. In theory the only workflows that we eliminate are the "undesirable" ones, e.g. a triage process that drops bugs on the floor, or a review process that makes patch context hard to access. In practice, though, other workflows - both legitimate workflows currently being used and potential improved workflows get eliminated as well.

Of course, things aren't all as black-and-white as I might have made them seem. As always, the specific context/situation matters a lot, and it's always a tradeoff between different goals - in the end there's no one-size-fits-all and the decision is something that needs careful consideration.

[ 3 Comments... ]