Blog

Book #7 of 2019 is Past Tense by Lee Child. I've been a fan of the Jack Reacher books for a few years now. While they tend to be a bit outlandish, I do enjoy the way they're written and the Reacher character. This one was no exception. I just finished it so it's hard to give an unbiased comparison to the other books in the series but I would place it a bit above average.

It's fiction so I don't have much else to say here. If you like the Reacher books I'm sure you'll read it at some point anyway.

[ 0 Comments... ]

Book #6 of 2019 is Why We Sleep by Matthew Walker. This was a really good book, and I plan to purchase a copy for myself. Written by a researcher in the field, it's full of interesting results about the importance of sleep and in particular what REM vs NREM sleep does to your brain. Again, I'm amazed by how much research has been done in this area, and how little of it is part of "common knowledge".

As with The Circadian Code this book cites numerous studies, but it doesn't feel repetitive at all and the information presented is well divided into separate topics/themes. The author also mostly stays away from personal anecdotes, but provides a touch of humour throughout the book which kept me engaged.

I would recommend this book to anybody and everybody. It's really eye-opening and brings home the detrimental effects of cutting out sleep.

[ 1 Comment... ]

Book #5 of 2019 is In Your Best Interest: The Ultimate Guide to the Canadian Bond Market. I got interested in learning about bonds after watching some Khan Academy video on the difference between bonds and stocks. Despite it (a) being pretty much the only book I could find on the Canadian bond market and (b) supposedly being written with beginners in mind, I did not find the book particularly illuminating.

While the author seems to have knowledge and experience in the area of bonds, I don't think he did a good job translating that knowledge into a book. Much of the basic stuff was repeated over and over, while the more complex stuff was not explained properly. Also at one point the writing randomly switched (mid-paragraph) from being directed towards a retail investor (i.e. me) to being directed towards an investment advisor. That was just weird.

Anyway, I ended up skimming a big chunk of this book so I can't really claim to have "read" it. I did read the parts that I thought would be most useful, and they did not live up to my expectations. Ah well.

[ 0 Comments... ]

Book #4 of 2019 is Bad Blood, by John Carreyrou. It's the story of Theranos and how they managed to dupe so many people with their blood analysis device that didn't really work.

Although I'd seen the name Theranos come up in my newsfeed from time to time, I mostly just ignored it, and so was basically unaware of the whole scandal. The book was quite a page-turner, written almost like a work of fiction, except that it's not. The author is the Wall Street Journal investigative reporter who first exposed Theranos, so the book is fairly authoritative on the topic. A fun read, and also sorta scary that such a thing could happen.

[ 0 Comments... ]

Book #3 of 2019 is The Circadian Code, by Satchin Panda. This was a pretty interesting book written by a researcher in the field of circadian rhythms. Interesting to me at least because there's a lot more research in this field than I thought there would be, and (if one believes the book) the evidence overwhelmingly indicates that the body's circadian rhythms are hugely important in maintaining good health. If even a tenth of the claims he makes in the book are true, it's worth applying his suggestions.

The quick summary is that different organs/systems in the body have different clocks that are controlled by different things. e.g. the brain alertness/awakeness is controlled by exposure to blue light; the digestive system is controlled by when you eat, and so on. Ensuring that you allow the different clocks to work their full cycles properly leads to better health outcomes. The single biggest thing to do is ensure that all your eating stays within a 12 hour (or smaller) window in a day, so as to give your digestive and other related systems time to rest and recover. This is called time-restricted eating (TRE) and many studies have shown it helps in all sorts of ways (again, per the book). I haven't even consulted any alternative sources on this but it's worth following up on.

The book itself was a mix of science, anecdotes, and personal stories/advice. The mix was pretty good although after a while the sheer number of studies being described and cited started to make my eyes glaze over so towards the end I was sort of just skimming. The "TRE is awesome" theme got hammered in a little too much, in my opinion. Still, overall it was a pretty informative read.

[ 0 Comments... ]

Book #2 of 2019 is Mindstorms, by Seymour Papert. This one has been on my reading list for a long time. I think I first heard about it in one of Bret Victor's talks. It was written almost 40 years ago (!), and is about the intersection of computers and education. Specifically, Papert talks about his ideas on how computers can help kids learn stuff.

I found it really quite interesting, and also sad in that the vision and ideas he had seem to have been forgotten and not implemented widely. He worked with Piaget for a while (who did a lot of work in child development) and a good chunk of the book rests heavily on that collaboration, so it's not just random stuff he made up - it's based on research and investigations that they did in real-world settings.

That being said, I also found it a somewhat hard book to read because of the plethora of big words he uses, and I ended up skimming a couple of the chapters that were a little too abstract for me. But the parts where he talked about specific examples of how the LOGO programming environment was used to enhance a child's learning experience was quite inspirational. If you're interested in education and how people learn, I highly recommend giving it a read.

[ 0 Comments... ]

I'd like to blog more in 2019. So to get me started I'm going to start blogging about books I'm reading.

I just finished the Sleep Solution by W. Chris Winter. It was recommended by a number of people on one of the HN "best books of 2018" threads. It's a pretty good book covering sleep - how it works, why you might be getting not enough or poor quality sleep, and what you can do to improve it.

The book is written in a very funny/colloquial style that's easy to read, and generally is light-hearted. Ironically the best time I have for reading these days is before going to sleep, so I felt a bit guilty about sacrificing sleep time to read a book which was telling me how important sleep is. It's also the first book I'm reading specifically about sleep so it had a number of new concepts for me and helped me build a better mental model of what happens when I sleep and how to improve my own sleep.

I don't consider myself as having any real sleep issues, so it's not like I was looking for a solution in this book, but I found it useful nonetheless, and would recommend it to anybody as an easy-to-read introductory (albeit somewhat opinionated) book on the topic.

I have another sleep-related book on my reading list so it'll be interesting to compare the two.

[ 0 Comments... ]

I tend to do a lot of try pushes for testing changes to Gecko and other stuff, and by using one of TreeHerder's (apparently) lesser-known features, managing these pushes to see their results is really easy. If you have trouble managing your try pushes, consider this:

Open a tab with an author filter for yourself. You can do this by clicking on your email address on any of your try pushes (see highlighted area in screenshot below). Keep this tab open, forever. By default it shows you the last 10 try pushes you did, and if you leave it open, it will auto-update to show newer try pushes that you do.

With this tab open, you can easily keep an eye on your try pushes. Once the oldest try pushes are "done" (all jobs completed, you've checked the result, and you don't care about it anymore), you can quickly and easily drop it off the bottom by clicking on the "Set as bottom of range" menu item on the oldest push that you do want to keep. (Again, see screenshot below).

 
(click to embiggen)

This effectively turns this tab into a rotating buffer of the try pushes you care about, with the oldest ones moving down and eventually getting removed via use of "Set as bottom of range" and the newer ones automatically appearing on top.

Note: clicking on the "Set as bottom of range" link will also reload the TreeHerder page, which means errors that might otherwise accumulate (due to e.g. sleeping your laptop for a time, or a new TreeHerder version getting deployed) get cleared away, so it's even self-healing!

Bonus tip: Before you clear away old try pushes that you don't care about, quickly go through them to make sure they are all marked "Complete". If they still have jobs running that you don't care about, do everybody a favor and hit the push cancellation button (the "X" icon next to "View Tests") before resetting the bottom of range, as that will ensure we don't waste machine time running jobs nobody cares about.

Extra bonus tip: Since using this technique this makes all those "Thank you for your try submission" Taskcluster emails redundant, set up an email filter to reroute those emails to the /dev/null of your choice. Less email results in a happier you!

Final bonus tip: If you need to copy a link to a specific try push (for pasting in a bug, for example), right-click on the timestamp for that try push (to the left of your email address), and copy the URL for that link. That link is for that specific push, and can be shared to get the desired results.

And there you have it, folks, a nice simple way to manage all your try pushes on a single page and not get overwhelmed.

[ 4 Comments... ]

I recently migrated a bunch of stuff (including this website) to Amazon EC2, running on a FEMP (FreeBSD, nginx, MySQL, PHP) stack. I had to fiddle with a few things to get it running smoothly, and wanted to document the steps in case anybody else is trying to do this (or I need to do it again later). This assumes you have an Amazon AWS account and some familiarity with how to use it.

Before you start

Ensure you know what region and instance type you want. I used the Canada (Central) region but it should work the same in any other region. And I used a t2.micro instance type because I have a bunch of stuff running on the instance, but presumably you could use a t2.nano type if you wanted to go even lighter. Also, I'm using Amazon Route53 to handle the DNS, but if you have DNS managed separately that's fine too.

Upload your SSH public key

In the EC2 dashboard, select "Key Pairs" under the "Network and Security" section in the left pane. Click on "Import Key pair" and provide the public half of your SSH keypair. This will get installed into the instance so that you can SSH in to the instance when it boots up.

Create the instance

Select "Instances" in the EC2 dashboard, and start the launch wizard by clicking "Launch Instance". You'll find the FreeBSD images under "Community AMIs" if you search for FreeBSD using the search. Generally you want to grab the most recent FreeBSD release you can find (note: the search results are not sorted by recency). If you want to make sure you're getting an official image, head over to the freebsd-announce mailing list, and look for the most recent release announcement email. As of this writing it is 11.2-RELEASE. (Note: be sure to use a -RELEASE version, not a -STABLE version). The email should contain AMI identifiers for all the different EC2 regions; for example the Canada AMI for 11.2-RELEASE is ami-a2f97bc6. Searching for that in the launch wizard finds it easily.

Next step is to select the instance type. Select one that's appropriate for your needs (I'm using t2.micro). The next step is to configure instance details. Unless you have specific changes you want to make here you can leave this with the default settings. Next you have to choose the root volume size. With my instances I like using a 10 GB root volume for the system and swap, and using a separate EBS volume for the "user data" (home folders and whatnot). For reference my 10G root volume is currently 54% full with the base system, extra packages, and a 2G swap file.

After that you can add tags if you want, change the security groups, and finally review everything. I just go with the defaults for the security groups, since it allows external SSH access and you can always change it later. Finally, time to launch! In the launch dialog you select the keypair from the previous step and you're off to the races.

Logging in

Once the instance is up, the EC2 console should display the public IP address. You'll need to log in with the user ec2-user at that IP address, using the keypair you selected previously. If you're paranoid about security (and you should be), you can verify the host key that SSH shows you by selecting the instance in the EC2 console, going to Actions -- Instance Settings -- Get Instance Screenshot. The screenshot should display the host keys as shown on the instance itself, and you can compare it to what SSH is showing to ensure you're not getting MITM'd.

Initial housekeeping

This part is sort of optional, but I like having a reasonable hostname and shell installed before I get to work. I'm going to use jasken.example.com as the hostname and I like using bash as my default shell. To do this, run the following commands:

su                                  # switch to root shell
sysrc hostname="jasken.example.com" # this modifies /etc/rc.conf
pkg update                          # update package manager
pkg install -y vim bash             # install useful packages
chsh -s /usr/local/bin/bash root    # change shell to bash for root and ec2-user
chsh -s /usr/local/bin/bash ec2-user

At this point I also like to reboot the machine (pretty much the only time I ever have to) because I've found that not everything picks up the hostname change if you change it via the hostname command while the instance is running. So run reboot and log back in once the instance is back up. The rest of the steps need root access so go ahead and su to root once you're back in.

IPv6 configuration

While you're rebooting, you can also set up IPv6 support. FreeBSD has everything built-in, you just need to fiddle with the VPC settings in AWS to get an IP address assigned. Note the VPC ID and Subnet ID in your instance's details, and then go to the VPC dashboard (it's a separate AWS service, not inside EC2). Find the VPC your instance is in, then go to Actions -- Edit CIDRs. Click on the "Add IPv6 CIDR" button and then "Close". Still in the VPC dashboard, select "Subnets" from the left panel and select the subnet of your instance. Here again, go to Actions -- Edit IPv6 CIDRs, and then click on "Add IPv6 CIDR". Put "00" in the box that appears to fill in the IPv6 subnet and hit ok.

Next, go to the "Route Tables" section of the VPC dashboard, and select the route table for the VPC. In the Routes tab, add a new route with destination ::/0 and the same gateway as the 0.0.0.0/0 entry. This ensures that outbound IPv6 connections will use the external network gateway.

Finally, go back to the EC2 dashboard, select your instance, and go to Actions -- Networking -- Manage IP addresses. Under IPv6 addresses, click "Assign new IP" and "Yes, Update" to auto-assign a new IPv6 address. That's it! If you SSH in to the instance you should be able to ping6 google.com successfully for example. It might take a minute or so for the connection to start working properly.

Installing packages

For the "EMP" part of the FEMP stack, we need to install nginx, mysql, and php. Also because we're not barbarians we're going to make sure our webserver is TLS-enabled with a Let's Encrypt certificate that renews automatically, for which we want certbot. So:

pkg install -y nginx mysql56-server php56 php56-mysql php56-mysqli php56-gd php56-json php56-xml php56-dom php56-openssl py36-certbot

Note that the set of PHP modules you need may vary; I'm just listing the ones that I needed, but you can always install/uninstall more later if you need to.

PHP setup

To use PHP over CGI with nginx we're going to use the php-fpm service. Instead of having the service listen over a network socket, we'll have it listen over a file socket, and make sure PHP and nginx are in agreement about the info passed back and forth. The sed commands below do just that.

cd /usr/local/etc/
sed -i "" -e "s#listen = 127.0.0.1:9000#listen = /var/run/php-fpm.sock#" php-fpm.conf
sed -i "" -e "s#;listen.owner#listen.owner#" php-fpm.conf
sed -i "" -e "s#;listen.group#listen.group#" php-fpm.conf
sed -i "" -e "s#;listen.mode#listen.mode#" php-fpm.conf
sed -e "s#;cgi.fix_pathinfo=1#cgi.fix_pathinfo=0#" php.ini-production > php.ini
sysrc php_fpm_enable="YES"
service php-fpm start

MySQL setup

This is really easy to set up. The hard part is optimizing the database for your workload, but that's outside the scope of my knowledge and of this tutorial.

sysrc mysql_enable="YES"
service mysql-server start
mysql_secure_installation   # this is interactive, you'll want to set a root password
service mysql-server restart

Swap space

MySQL can eat up a bunch of memory, and it's good to have some swap set up. Without this you might find, as I did, that weekly periodic tasks such as rebuilding the locate database can result in OOM situations and take down your database. On a t2.micro instance which has 1GB of memory, a 2GB swap file works well for me:

# Make a 2GB (2048 1-meg blocks) swap file at /usr/swap0
dd if=/dev/zero of=/usr/swap0 bs=1m count=2048
chmod 0600 /usr/swap0
# Install the swap filesystem
echo 'md99 none swap sw,file=/usr/swap0,late 0 0' >> /etc/fstab
# and enable it
swapon -aL

You can verify the swap is enabled by using the swapinfo command.

nginx setup

Because the nginx config can get complicated, specially if you're hosting multiple websites, it pays to break it up into manageable pieces. I like having an includes/ folder which contains snippets of reusable configuration (error pages, PHP stuff, SSL stuff), and a sites-enabled/ folder that has a configuration per website you're hosting. Also, we want to generate some Diffie-Hellman parameters for TLS purposes. So:

cd /usr/local/etc/nginx/
openssl dhparam -out dhparam.pem 4096
mkdir includes
cd includes/
# This creates an error.inc file with error handling snippet
cat >error.inc <<'END'
error_page 500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/local/www/nginx-dist;
}
END
# PHP snippet
cat >php.inc <<'END'
location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass   unix:/var/run/php-fpm.sock;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $request_filename;
    include        fastcgi_params;
}
END
# SSL snippet
cat >ssl.inc <<'END'
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /usr/local/etc/nginx/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
END

If you want to fiddle with the above, feel free. I'm not a security expert so I don't know what a lot of the stuff in the ssl.inc does, but based on the Qualys SSL test it seems to provide a good security/compatibility tradeoff. I mostly cobbled it together from various recommendations on the Internet.

Finally, we set up the server entry (assuming we're serving up the website "example.com") and start nginx:

cd /usr/local/etc/nginx/
cat >nginx.conf <<'END'
user  www;
worker_processes  1;

error_log  /var/log/nginx/error.log info;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;
    resolver 127.0.0.1;

    include  sites-enabled/example;
}
END
mkdir sites-enabled
cd sites-enabled/
cat >example <<'END'
server {
    listen       [::]:80;
    listen       80;
    server_name  example.com;
    root /usr/local/www/nginx;
    index  index.php index.html index.htm;

    include includes/php.inc;

    location / {
        try_files $uri $uri/ =404;
    }

    include includes/error.inc;
}
END
sysrc nginx_enable="YES"
service nginx start

Open up ports

The nginx setup above is sufficient to host an insecure server on port 80, which is what we need in order to get the certificate that we need to enable TLS. So at this point go to your DNS manager, wherever that is, and point the A and AAAA records for "example.com" (or whatever site you're hosting) to the public IP addresses for your instance. Also, go to the "Security Groups" pane in the EC2 dashboard and edit the "Inbound" tab for your instance's security group to allow HTTP traffic on TCP port 80 from source 0.0.0.0/0, ::/0, and the same for HTTPS traffic on TCP port 443.

After you've done that and the DNS changes have propagated, you should be able to go to http://example.com in your browser and get the nginx welcome page, served from your very own /usr/local/www/nginx folder.

TLS

Now it's time to get a TLS certificate for your example.com webserver. This is almost laughably easy once you have regular HTTP working:

certbot-3.6 certonly --webroot -n --agree-tos --email 'admin@example.com' -w /usr/local/www/nginx -d example.com
crontab <(echo '0 0 1,15 * * certbot-3.6 renew --post-hook "service nginx restart"')

Make sure to replace the email address and domain above as appropriate. This will use certbot's webroot plugin to get a Let's Encrypt TLS cert and install it into /usr/local/etc/letsencrypt/live/. It also installs a cron job to automatically attempt renewal of the cert twice a month. This does nothing if the cert isn't about to expire, but otherwise renews it using the same options as the initial request. The final step is updating the sites-enabled/example config to redirect all HTTP requests to HTTPS, and use the aforementioned TLS cert.

cd /usr/local/etc/nginx/sites-enabled/
cat >example <<'END'
server {
    listen       [::]:80;
    listen       80;
    server_name  example.com;
    return 301 https://$host$request_uri;
}

server {
    listen       [::]:443;
    listen       443;
    server_name  example.com;
    root   /usr/local/www/nginx;
    index  index.php index.html index.htm;

    include includes/ssl.inc;
    ssl_certificate /usr/local/etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /usr/local/etc/letsencrypt/live/example.com/privkey.pem;

    include includes/php.inc;

    location / {
        try_files $uri $uri/ =404;
    }

    include includes/error.inc;
}
END
service nginx restart

And that's all, folks!

Parting words

The above commands set things up so that they persist across reboots. That is, if you stop and restart the EC2 instance, everything should come back up enabled. The only problem is that if you stop and restart the instance, the IP address changes so you'll have to update your DNS entry.

If there's commands above you're unfamiliar with, you should use the man pages to read up on them. In general copying and pasting commands from some random website into a command prompt is a bad idea unless you know what those commands are doing.

One thing I didn't cover in this blog post is how to deal with the daily emails that FreeBSD will send to the root user. I also run a full blown mail gateway with postfix and I plan to cover that in another post.

[ 2 Comments... ]

I previously wrote about setting up multiple VLANs to segment your home network and improve the security characteristics. Since then I've added more devices to my home network, and keeping everything in separate VLANs was looking like it would be a hassle. So instead I decided to put everything into the same VLAN but augment the router's firewall rules to continue restricting traffic between "trusted" and "untrusted" devices.

The problem is that didn't work. I set up all the firewall rules but for some reason they weren't being respected. After (too much) digging I finally discovered that you have to install the kmod-ebtables package to get this to actually work. Without it, the netfilter code in the kernel doesn't filter traffic between hosts on the same VLAN and so any rules you have for that get ignored. After installing kmod-ebtables my firewall rules started working. Yay!

Along the way I also discovered that OpenWRT is basically dead now (they haven't had a release in a long time) and the LEDE project is the new fork/successor project. So if you were using OpenWRT you should probably migrate. The migration was relatively painless for me, since the images are compatible. (Edited 06-Jan-2019: LEDE has merged back into OpenWRT since I wrote this)

There's one other complication that I've run into but haven't yet resolved. After upgrading to LEDE and installing kmod-ebtables, for some reason I couldn't connect between two FreeBSD machines on my network via external IP and port forwarding. The setup is like so:

• Machine A has internal IP address 192.168.1.A
• Machine B has internal IP address 192.168.1.B
• The router's external IP address is E
• The router is set to forward port P to machine A
• The router is set to forward port Q to machine B

Now, from machine B, if connect to E:P, it doesn't work. Likewise, from machine A, connecting to E:Q doesn't work. I can connect using the internal IP address (192.168.1.A:P or 192.168.1.B:Q) just fine; it's only the via the external IP that it doesn't work. All the other machines on my network can connect to E:P and E:Q fine as well. It's only machines A and B that can't talk to each other. The thing A and B have in common is they are running FreeBSD; the other machines I tried were Linux/OS X.

Obviously the next step here is to fire up tcpdump and see what's going on. Funny thing is, when I run tcpdump on my router, the problem goes away and the machines can connect to each other. So there's that. I'm sure with more investigation I'll get to the bottom of this but for now I've shelved it under "mysteries that I can work around easily". If anybody has run into this before I'd be interested in hearing about it.

Also if anybody knows of good tools to visualize and debug iptables rules I'd be interested to try them out, because I haven't found anything good yet. I've been using the counters in the tables to try and figure out which rules the packets are hitting but since I'm debugging this "live" there's a lot of noise from random devices and the counters are not as reliable as I'd like.

[ 0 Comments... ]