|
I recently migrated a bunch of stuff (including this website) to Amazon EC2, running on a FEMP (FreeBSD, nginx, MySQL, PHP) stack. I had to fiddle with a few things to get it running smoothly, and wanted to document the steps in case anybody else is trying to do this (or I need to do it again later). This assumes you have an Amazon AWS account and some familiarity with how to use it.
Before you start
Ensure you know what region and instance type you want. I used the Canada (Central) region but it should work the same in any other region. And I used a t2.micro instance type because I have a bunch of stuff running on the instance, but presumably you could use a t2.nano type if you wanted to go even lighter. Also, I'm using Amazon Route53 to handle the DNS, but if you have DNS managed separately that's fine too.
Upload your SSH public key
In the EC2 dashboard, select "Key Pairs" under the "Network and Security" section in the left pane. Click on "Import Key pair" and provide the public half of your SSH keypair. This will get installed into the instance so that you can SSH in to the instance when it boots up.
Create the instance
Select "Instances" in the EC2 dashboard, and start the launch wizard by clicking "Launch Instance". You'll find the FreeBSD images under "Community AMIs" if you search for FreeBSD using the search. Generally you want to grab the most recent FreeBSD release you can find (note: the search results are not sorted by recency). If you want to make sure you're getting an official image, head over to the freebsd-announce mailing list, and look for the most recent release announcement email. As of this writing it is 11.2-RELEASE. (Note: be sure to use a -RELEASE version, not a -STABLE version). The email should contain AMI identifiers for all the different EC2 regions; for example the Canada AMI for 11.2-RELEASE is ami-a2f97bc6. Searching for that in the launch wizard finds it easily.
Next step is to select the instance type. Select one that's appropriate for your needs (I'm using t2.micro). The next step is to configure instance details. Unless you have specific changes you want to make here you can leave this with the default settings. Next you have to choose the root volume size. With my instances I like using a 10 GB root volume for the system and swap, and using a separate EBS volume for the "user data" (home folders and whatnot). For reference my 10G root volume is currently 54% full with the base system, extra packages, and a 2G swap file.
After that you can add tags if you want, change the security groups, and finally review everything. I just go with the defaults for the security groups, since it allows external SSH access and you can always change it later. Finally, time to launch! In the launch dialog you select the keypair from the previous step and you're off to the races.
Logging in
Once the instance is up, the EC2 console should display the public IP address. You'll need to log in with the user ec2-user at that IP address, using the keypair you selected previously. If you're paranoid about security (and you should be), you can verify the host key that SSH shows you by selecting the instance in the EC2 console, going to Actions -- Instance Settings -- Get Instance Screenshot. The screenshot should display the host keys as shown on the instance itself, and you can compare it to what SSH is showing to ensure you're not getting MITM'd.
Initial housekeeping
This part is sort of optional, but I like having a reasonable hostname and shell installed before I get to work. I'm going to use jasken.example.com as the hostname and I like using bash as my default shell. To do this, run the following commands:
su # switch to root shell
sysrc hostname="jasken.example.com" # this modifies /etc/rc.conf
pkg update # update package manager
pkg install -y vim bash # install useful packages
chsh -s /usr/local/bin/bash root # change shell to bash for root and ec2-user
chsh -s /usr/local/bin/bash ec2-user
At this point I also like to reboot the machine (pretty much the only time I ever have to) because I've found that not everything picks up the hostname change if you change it via the hostname command while the instance is running. So run reboot and log back in once the instance is back up. The rest of the steps need root access so go ahead and su to root once you're back in.
IPv6 configuration
While you're rebooting, you can also set up IPv6 support. FreeBSD has everything built-in, you just need to fiddle with the VPC settings in AWS to get an IP address assigned. Note the VPC ID and Subnet ID in your instance's details, and then go to the VPC dashboard (it's a separate AWS service, not inside EC2). Find the VPC your instance is in, then go to Actions -- Edit CIDRs. Click on the "Add IPv6 CIDR" button and then "Close". Still in the VPC dashboard, select "Subnets" from the left panel and select the subnet of your instance. Here again, go to Actions -- Edit IPv6 CIDRs, and then click on "Add IPv6 CIDR". Put "00" in the box that appears to fill in the IPv6 subnet and hit ok.
Next, go to the "Route Tables" section of the VPC dashboard, and select the route table for the VPC. In the Routes tab, add a new route with destination ::/0 and the same gateway as the 0.0.0.0/0 entry. This ensures that outbound IPv6 connections will use the external network gateway.
Finally, go back to the EC2 dashboard, select your instance, and go to Actions -- Networking -- Manage IP addresses. Under IPv6 addresses, click "Assign new IP" and "Yes, Update" to auto-assign a new IPv6 address. That's it! If you SSH in to the instance you should be able to ping6 google.com successfully for example. It might take a minute or so for the connection to start working properly.
Installing packages
For the "EMP" part of the FEMP stack, we need to install nginx, mysql, and php. Also because we're not barbarians we're going to make sure our webserver is TLS-enabled with a Let's Encrypt certificate that renews automatically, for which we want certbot. So:
pkg install -y nginx mysql56-server php56 php56-mysql php56-mysqli php56-gd php56-json php56-xml php56-dom php56-openssl py36-certbot
Note that the set of PHP modules you need may vary; I'm just listing the ones that I needed, but you can always install/uninstall more later if you need to.
PHP setup
To use PHP over CGI with nginx we're going to use the php-fpm service. Instead of having the service listen over a network socket, we'll have it listen over a file socket, and make sure PHP and nginx are in agreement about the info passed back and forth. The sed commands below do just that.
cd /usr/local/etc/
sed -i "" -e "s#listen = 127.0.0.1:9000#listen = /var/run/php-fpm.sock#" php-fpm.conf
sed -i "" -e "s#;listen.owner#listen.owner#" php-fpm.conf
sed -i "" -e "s#;listen.group#listen.group#" php-fpm.conf
sed -i "" -e "s#;listen.mode#listen.mode#" php-fpm.conf
sed -e "s#;cgi.fix_pathinfo=1#cgi.fix_pathinfo=0#" php.ini-production > php.ini
sysrc php_fpm_enable="YES"
service php-fpm start
MySQL setup
This is really easy to set up. The hard part is optimizing the database for your workload, but that's outside the scope of my knowledge and of this tutorial.
sysrc mysql_enable="YES"
service mysql-server start
mysql_secure_installation # this is interactive, you'll want to set a root password
service mysql-server restart
Swap space
MySQL can eat up a bunch of memory, and it's good to have some swap set up. Without this you might find, as I did, that weekly periodic tasks such as rebuilding the locate database can result in OOM situations and take down your database. On a t2.micro instance which has 1GB of memory, a 2GB swap file works well for me:
# Make a 2GB (2048 1-meg blocks) swap file at /usr/swap0
dd if=/dev/zero of=/usr/swap0 bs=1m count=2048
chmod 0600 /usr/swap0
# Install the swap filesystem
echo 'md99 none swap sw,file=/usr/swap0,late 0 0' >> /etc/fstab
# and enable it
swapon -aL
You can verify the swap is enabled by using the swapinfo command.
nginx setup
Because the nginx config can get complicated, specially if you're hosting multiple websites, it pays to break it up into manageable pieces. I like having an includes/ folder which contains snippets of reusable configuration (error pages, PHP stuff, SSL stuff), and a sites-enabled/ folder that has a configuration per website you're hosting. Also, we want to generate some Diffie-Hellman parameters for TLS purposes. So:
cd /usr/local/etc/nginx/
openssl dhparam -out dhparam.pem 4096
mkdir includes
cd includes/
# This creates an error.inc file with error handling snippet
cat >error.inc <<'END'
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/local/www/nginx-dist;
}
END
# PHP snippet
cat >php.inc <<'END'
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $request_filename;
include fastcgi_params;
}
END
# SSL snippet
cat >ssl.inc <<'END'
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /usr/local/etc/nginx/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
END
If you want to fiddle with the above, feel free. I'm not a security expert so I don't know what a lot of the stuff in the ssl.inc does, but based on the Qualys SSL test it seems to provide a good security/compatibility tradeoff. I mostly cobbled it together from various recommendations on the Internet.
Finally, we set up the server entry (assuming we're serving up the website "example.com") and start nginx:
cd /usr/local/etc/nginx/
cat >nginx.conf <<'END'
user www;
worker_processes 1;
error_log /var/log/nginx/error.log info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
resolver 127.0.0.1;
include sites-enabled/example;
}
END
mkdir sites-enabled
cd sites-enabled/
cat >example <<'END'
server {
listen [::]:80;
listen 80;
server_name example.com;
root /usr/local/www/nginx;
index index.php index.html index.htm;
include includes/php.inc;
location / {
try_files $uri $uri/ =404;
}
include includes/error.inc;
}
END
sysrc nginx_enable="YES"
service nginx start
Open up ports
The nginx setup above is sufficient to host an insecure server on port 80, which is what we need in order to get the certificate that we need to enable TLS. So at this point go to your DNS manager, wherever that is, and point the A and AAAA records for "example.com" (or whatever site you're hosting) to the public IP addresses for your instance. Also, go to the "Security Groups" pane in the EC2 dashboard and edit the "Inbound" tab for your instance's security group to allow HTTP traffic on TCP port 80 from source 0.0.0.0/0, ::/0, and the same for HTTPS traffic on TCP port 443.
After you've done that and the DNS changes have propagated, you should be able to go to http://example.com in your browser and get the nginx welcome page, served from your very own /usr/local/www/nginx folder.
TLS
Now it's time to get a TLS certificate for your example.com webserver. This is almost laughably easy once you have regular HTTP working:
certbot-3.6 certonly --webroot -n --agree-tos --email 'admin@example.com' -w /usr/local/www/nginx -d example.com
crontab <(echo '0 0 1,15 * * certbot-3.6 renew --post-hook "service nginx restart"')
Make sure to replace the email address and domain above as appropriate. This will use certbot's webroot plugin to get a Let's Encrypt TLS cert and install it into /usr/local/etc/letsencrypt/live/. It also installs a cron job to automatically attempt renewal of the cert twice a month. This does nothing if the cert isn't about to expire, but otherwise renews it using the same options as the initial request. The final step is updating the sites-enabled/example config to redirect all HTTP requests to HTTPS, and use the aforementioned TLS cert.
cd /usr/local/etc/nginx/sites-enabled/
cat >example <<'END'
server {
listen [::]:80;
listen 80;
server_name example.com;
return 301 https://$host$request_uri;
}
server {
listen [::]:443;
listen 443;
server_name example.com;
root /usr/local/www/nginx;
index index.php index.html index.htm;
include includes/ssl.inc;
ssl_certificate /usr/local/etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /usr/local/etc/letsencrypt/live/example.com/privkey.pem;
include includes/php.inc;
location / {
try_files $uri $uri/ =404;
}
include includes/error.inc;
}
END
service nginx restart
And that's all, folks!
Parting words
The above commands set things up so that they persist across reboots. That is, if you stop and restart the EC2 instance, everything should come back up enabled. The only problem is that if you stop and restart the instance, the IP address changes so you'll have to update your DNS entry.
If there's commands above you're unfamiliar with, you should use the man pages to read up on them. In general copying and pasting commands from some random website into a command prompt is a bad idea unless you know what those commands are doing.
One thing I didn't cover in this blog post is how to deal with the daily emails that FreeBSD will send to the root user. I also run a full blown mail gateway with postfix and I plan to cover that in another post.
[ 2 Comments... ]
I previously wrote about setting up multiple VLANs to segment your home network and improve the security characteristics. Since then I've added more devices to my home network, and keeping everything in separate VLANs was looking like it would be a hassle. So instead I decided to put everything into the same VLAN but augment the router's firewall rules to continue restricting traffic between "trusted" and "untrusted" devices.
The problem is that didn't work. I set up all the firewall rules but for some reason they weren't being respected. After (too much) digging I finally discovered that you have to install the kmod-ebtables package to get this to actually work. Without it, the netfilter code in the kernel doesn't filter traffic between hosts on the same VLAN and so any rules you have for that get ignored. After installing kmod-ebtables my firewall rules started working. Yay!
Along the way I also discovered that OpenWRT is basically dead now (they haven't had a release in a long time) and the LEDE project is the new fork/successor project. So if you were using OpenWRT you should probably migrate. The migration was relatively painless for me, since the images are compatible. (Edited 06-Jan-2019: LEDE has merged back into OpenWRT since I wrote this)
There's one other complication that I've run into but haven't yet resolved. After upgrading to LEDE and installing kmod-ebtables, for some reason I couldn't connect between two FreeBSD machines on my network via external IP and port forwarding. The setup is like so:
- Machine A has internal IP address 192.168.1.A
- Machine B has internal IP address 192.168.1.B
- The router's external IP address is E
- The router is set to forward port P to machine A
- The router is set to forward port Q to machine B
Now, from machine B, if connect to E:P, it doesn't work. Likewise, from machine A, connecting to E:Q doesn't work. I can connect using the internal IP address (192.168.1.A:P or 192.168.1.B:Q) just fine; it's only the via the external IP that it doesn't work. All the other machines on my network can connect to E:P and E:Q fine as well. It's only machines A and B that can't talk to each other. The thing A and B have in common is they are running FreeBSD; the other machines I tried were Linux/OS X.
Obviously the next step here is to fire up tcpdump and see what's going on. Funny thing is, when I run tcpdump on my router, the problem goes away and the machines can connect to each other. So there's that. I'm sure with more investigation I'll get to the bottom of this but for now I've shelved it under "mysteries that I can work around easily". If anybody has run into this before I'd be interested in hearing about it.
Also if anybody knows of good tools to visualize and debug iptables rules I'd be interested to try them out, because I haven't found anything good yet. I've been using the counters in the tables to try and figure out which rules the packets are hitting but since I'm debugging this "live" there's a lot of noise from random devices and the counters are not as reliable as I'd like.
[ 0 Comments... ]
My mind was blown by a pair of articles today: When Factory Jobs Vanish, Men Become Less Desirable Partners, and one from 2010, The End of Men. To me these articles present a view that I had been largely unaware of before, and explains a lot of things that I've been puzzling over.
The articles point out that women are increasingly dominating most professions. Being a software engineer, one of the few industries that is the exception to this trend, I hadn't really paid much attention to this. But the article goes further: it says that as a result of this shift, men are losing the dominant role in families, are often unable to get married, leading to many women becoming single mothers.
This produces what I think of as a situation not in equilibrium. We're in the middle of a societal shift from men being the dominant gender to women being the dominant gender, but we're only partway through. Women have already become dominant in a lot of industries and roles, which is fine. What hasn't caught up yet, though, is the societal perception of "who should wear the pants". I think a lot of men still feel that society expects them to be dominant - to be the breadwinner and the head of the nuclear family.
Much like a difference in electric potential between two points produces a current, this difference between what is and what is expected produces frustration and anger. If I imagine myself in the position of one of these men - having lost my job, unable to find a woman who wants to start a family with me, while still having the societal pressure to somehow take charge - I can easily picture myself feeling like a failure, and having a lot of anger. Across an entire population of men feeling this way, it's not surprising to see violent action to make a change.
The thing is, there's two ways to bring the system back into equilibrium. One is to go backwards - to reinstate jobs in industries where men can dominate, like manufacturing. This allows men to again live up to the societal expectations. This is what Trump has been promising, and it makes sense to me (finally!) what drove these men to vote for him. It also explains the rash of hate crimes against immigrants, the anti-globalism movement, and so on - these are all things that are perceived to have contributed to the loss in manufacturing jobs and therefore to men's loss of dominance. However, I don't think going backwards is really a viable path. The reason we entered this state of disequilibrium in the first place is because we were correcting a different disequilibrium, and going backwards will not really help. Those who fight to go backwards are guaranteed to lose, but the longer they fight, the more painful it will be for everybody.
The better way out is to keep going forward. We have to complete the transition by also changing the societal expectations of men's role in the world. Instead of continuing to implicitly or explicitly typecast them as the dominant gender, we must accept (and encourage) that they can take on roles that were previously stereotypically "female" roles, for example being a stay-at-home parent. This shift is well under way, but until we have a whole generation of men raised in a culture where that's the norm, it won't be complete.
The "End of Men" article linked above also mentions the rise of "herbivores" in Japan - men "who are rejecting the hard-drinking salaryman life of their fathers and are instead gardening, organizing dessert parties, acting cartoonishly feminine, and declining to have sex." As odd as this behaviour sounds to me, I feel now that this is actually the correct direction to be heading. I'm not saying all men should behave this way. What I'm saying is that men should feel accepted by society even if they do behave this way. There should be no perceived societal pressure to behave any differently. No judgements.
That being said, the article also says things like "clearly, some percentage of boys are just temperamentally unsuited to college" which I disagree with. The underlying premise here is that the genetic or biologic make-up of males is unsuited to higher education or adapting to these new circumstances, but I strongly believe this is more a result of societal expectations and cultural influence. In other countries with different cultures, where higher education is more valued, I think the results are very different.
In the end, I think the notion of gender itself will become obsolete, but until it does, we should be more explicit in our support of people of all genders to take on any useful role in society.
[ 0 Comments... ]
The web as we know it basically runs on advertising. Which is not really great, for a variety of reasons. But charging people outright for content doesn't work that great either. How about bitcoin mining instead?
Webpages can already run arbitary computation on your computer, so instead of funding themselves through ads, they could instead include a script that does some mining client-side and submits the results back to their server. Instead of paying with dollars and cents you're effectively paying with electricity and compute cycles. Seems a lot more palatable to me. What do you think?
[ 18 Comments... ]
When typing on a laptop keyboard, I find that my posture tends to get very closed and hunched. To fix this I resurrected an old low-tech solution I had for this problem: using two keyboards. Simply plug in an external USB keyboard, and use one keyboard for each hand. It's like a split keyboard, but better, because you can position it wherever you want to get a posture that's comfortable for you.
I used to do this on a Windows machine back when I was working at RIM and it worked great. Recently I tried to do it on my Mac laptop, but ran into the problem where the modifier state from one keyboard didn't apply to the other keyboard. So holding shift on one keyboard and T on the other wouldn't produce an uppercase T. This was quite annoying, and it seems to be an OS-level thing. After some googling I found Karabiner which solves this problem. Well, really it appears to be a more general keyboard customization tool, but the default configuration also combines keys across keyboards which is exactly what I wanted. \o/
Of course, changing your posture won't magically fix everything - moving around regularly is still the best way to go, but for me personally, this helps a bit :)
[ 0 Comments... ]
At some point in the past, I learned about the difference between frameworks and libraries, and it struck me as a really important conceptual distinction that extends far beyond just software. It's really a distinction in process, and that applies everywhere.
The fundamental difference between frameworks and libraries is that when dealing with a framework, the framework provides the structure, and you have to fill in specific bits to make it apply to what you are doing. With a library, however, you are provided with a set of functionality, and you invoke the library to help you get the job done.
It may not seem like a very big distinction at first, but it has a huge impact on various properties of the final product. For example, a framework is easier to use if what you are trying to do lines up with the goal the framework is intended to accomplish. The only thing you need to do is provide (or override) specific things that you need to customize, and the framework takes care of the rest. It's like a builder building your house, and you picking which tile pattern you want for the backsplash. With libraries there's a lot more work - you have a Home Depot full of tools and supplies, but you have to figure out how to put them together to build a house yourself.
The flip side, of course, is that with libraries you get a lot more freedom and customizability than you do with frameworks. With the house analogy, a builder won't add an extra floor for your house if it doesn't fit with their pre-defined floorplans for the subdivision. If you're building it yourself, though, you can do whatever you want.
The library approach makes the final workflow a lot more adaptable when faced with new situations. Once you are in a workflow dictated by a framework, it's very hard to change the workflow because you have almost no control over it - you only have as much control as it was designed to let you have. With libraries you can drop a library here, pick up another one there, and evolve your workflow incrementally, because you can use them however you want.
In the context of building code, the *nix toolchain (a pile of command-line tools that do very specific things) is a great example of the library approach - it's very adaptable as you can swap out commands for other commands to do what you need. An IDE, on the other hand, is more of a framework. It's easier to get started because the heavy lifting is taken care of, all you have to do is "insert code here". But if you want to do some special processing of the code that the IDE doesn't allow, you're out of luck.
An interesting thing to note is that usually people start with frameworks and move towards libraries as their needs get more complex and they need to customize their workflow more. It's not often that people go the other way, because once you've already spent the effort to build a customized workflow it's hard to justify throwing the freedom away and locking yourself down. But that's what it feels like we are doing at Mozilla - sometimes on purpose, and sometimes unintentionally, without realizing we are putting on a straitjacket.
The shift from Bugzilla/Splinter to MozReview is one example of this. Going from a customizable, flexible tool (attachments with flags) to a unified review process (push to MozReview) is a shift from libraries to frameworks. It forces people to conform to the workflow which the framework assumes, and for people used to their own customized, library-assisted workflow, that's a very hard transition. Another example of a shift from libraries to frameworks is the bug triage process that was announced recently.
I think in both of these cases the end goal is desirable and worth working towards, but we should realize that it entails (by definition) making things less flexible and adaptable. In theory the only workflows that we eliminate are the "undesirable" ones, e.g. a triage process that drops bugs on the floor, or a review process that makes patch context hard to access. In practice, though, other workflows - both legitimate workflows currently being used and potential improved workflows get eliminated as well.
Of course, things aren't all as black-and-white as I might have made them seem. As always, the specific context/situation matters a lot, and it's always a tradeoff between different goals - in the end there's no one-size-fits-all and the decision is something that needs careful consideration.
[ 3 Comments... ]
In the Firefox family of products, we've had asynchronous scrolling (aka async pan/zoom, aka APZ, aka compositor-thread scrolling) in Firefox OS and Firefox for Android for a while - even though they had different implementations, with different behaviors. We are now in the process of taking the Firefox OS implementation and bringing it to all our other platforms - including desktop and Android. After much hard work by many people, including but not limited to :botond, :dvander, :mattwoodrow, :mstange, :rbarker, :roc, :snorp, and :tn, we finally have APZ enabled on the nightly channel for both desktop and Android. We're working hard on fixing outstanding bugs and getting the quality up before we let it ride the trains out to DevEdition, Beta, and the release channel.
If you want to try it on desktop, note that APZ requires e10s to be enabled, and is currently only enabled for mousewheel/trackpad scrolling. We do have plans to implement it for other input types as well, although that may not happen in the initial release.
Although getting the basic machinery working took some effort, we're now mostly done with that and are facing a different but equally challenging aspect of this change - the fallout on web content. Modern web pages have access to many different APIs via JS and CSS, and implement many interesting scroll-linked effects, often triggered by the scroll event or driven by a loop on the main thread. With APZ, these approaches don't work quite so well because inherently the user-visible scrolling is async from the main thread where JS runs, and we generally avoid blocking the compositor on main-thread JS. This can result in jank or jitter for some of these effects, even though the main page scrolling itself remains smooth. I picked a few of the simpler scroll effects to discuss in a bit more detail below - not a comprehensive list by any means, but hopefully enough to help you get a feel for some of the nuances here.
Smooth scrolling
Smooth scrolling - that is, animating the scroll to a particular scroll offset - is something that is fairly common on web pages. Many pages do this using a JS loop to animate the scroll position. Without taking advantage of APZ, this will still work, but can result in less-than-optimal smoothness and framerate, because the main thread can be busy with doing other things.
Since Firefox 36, we've had support for the scroll-behavior CSS property which allows content to achieve the same effect without the JS loop. Our implementation for scroll-behavior without APZ enabled still runs on the main thread, though, and so can still end up being janky if the main thread is busy. With APZ enabled, the scroll-behavior implementation triggers the scroll animation on the compositor thread, so it should be smooth regardless of load on the main thread. Polyfills for scroll-behavior or old-school implementations in JS will remain synchronous, so for best performance we recommend switching to the CSS property where possible. That way as APZ rolls out to release, you'll get the benefits automatically.
Here is a simple example page that has a spinloop to block the main thread for 500ms at a time. Without APZ, clicking on the buttons results in a very janky/abrupt scroll, but with APZ it should be smooth.
position:sticky
Another common paradigm seen on the web is "sticky" elements - they scroll with the page for a bit, and then turn into position:fixed elements after a point. Again, this is usually implemented with JS listening for scroll events and updating the styles on the elements based on the scroll offset. With APZ, scroll events are going to be delayed relative to what the user is seeing, since the scroll events arrive on the main thread while scrolling is happening on the compositor thread. This will result in glitches as the user scrolls.
Our recommended approach here is to use position:sticky when possible, which we have supported since Firefox 32, and which we have support for in the compositor. This CSS property allows the element to scroll normally but take on the behavior of position:fixed beyond a threshold, even with APZ enabled. This isn't supported across all browsers yet, but there are a number of polyfills available - see the resources tab on the Can I Use position:sticky page for some options.
Again, here is a simple example page that has a spinloop to block the main thread for 500ms at a time. With APZ, the JS version will be laggy but the position:sticky version should always remain in the right place.
Parallax
Parallax. Oh boy. There's a lot of different ways to do this, but almost all of them rely on listening to scroll events and updating element styles based on that. For the same reasons as described in the previous section, implementations of parallax scrolling that are based on scroll events are going to be lagging behind the user's actual scroll position. Until recently, we didn't have a solution for this problem.
However, a few days ago :mattwoodrow landed compositor support for asynchronous scroll adjustments of 3D transforms, which allows a pure CSS parallax implementation to work smoothly with APZ. Keith Clark has a good writeup on how to do this, so I'm just going to point you there. All of his demo pages should scroll smoothly in Nightly with APZ enabled.
Unfortunately, it looks like this CSS-based approach may not work well across all browsers, so please make sure to test carefully if you want to try it out. Also, if you have suggestions on other methods on implementing parallax so that it doesn't rely on a responsive main thread, please let us know. For example, :mstange created one at http://tests.themasta.com/transform-fixed-parallax.html which we should be able to support in the compositor without too much difficulty.
Other features
I know that there are other interesting scroll-linked effects that people are doing or want to do on the web, and we'd really like to support them with asynchronous scrolling. The Blink team has a bunch of different proposals for browser APIs that can help with these sorts of things, including things like CompositorWorker and scroll customization. For more information and to join the discussion on these, please see the public-houdini mailing list. We'd love to get your feedback!
(Thanks to :botond and :mstange for reading a draft of this post and providing feedback.)
[ 6 Comments... ]
At the start of 2014, I became a "manager". At least in the sense that I had a couple of people reporting to me. Like most developers-turned-managers I was unsure if management was something I wanted to do but I figured it was worth trying at least. Somebody recommended the book First, Break All The Rules to me as a good book on management, so I picked up a copy and read it.
The book is based on data from many thousands of interviews and surveys that the Gallup organization did, across all sorts of organizations. There were lots of interesting points in the book, but the main takeaway relevant here was that people who build on their strengths instead of trying to correct their weaknesses are generally happier and more successful. This leads to some obvious follow-up questions: how do you know what your strengths are? What does it mean to "build on your strengths"?
To answer the first question I got the sequel, Now, Discover Your Strengths, which includes a single-use code for the online StrengthsFinder assessment. I read the book, took the assessment, and got a list of my top 5 strengths. While interesting, the list was kind of disappointing, mostly because I didn't really know what to do with it. Perhaps the next book in the series, Go Put Your Strengths To Work, would have explained but at this point I was disillusioned and didn't bother reading it.
Fast-forward to a month ago, when I finally got to attend the first TRIBE session. I'd heard good things about it, without really knowing anything specific about what it was about. Shortly before it started though, they sent us a copy of Strengths Based Leadership, which is a book based on the same Gallup data as the aforementioned books, and includes a code to the 2.0 version of the same online StrengthsFinder assessment. I read the book and took the new assessment (3 of the 5 strengths I got matched my initial results; the variance is explained on their FAQ page) but didn't really end up with much more information than I had before.
However, the TRIBE session changed that. It was during the session that I learned the answer to my earlier question about what it means to "build on strengths". If you're familiar with the 4 stages of competence, that TRIBE session took me from "unconscious incompetence" to "conscious incompetence" with regard to using my strengths - it made me aware of when I'm using my strengths and when I'm not, and to be more purposeful about when to use them. (Two asides: (1) the TRIBE session also included other useful things, so I do recommend attending and (2) being able to give something a name is incredibly powerful, but perhaps that's worth a whole 'nother blog post).
At this point, I'm still not 100% sure if being a manager is really for me. On the one hand, the strengths I have are not really aligned with the strengths needed to be a good manager. On the other hand, the Strengths Based Leadership book does provide some useful tips on how to leverage whatever strengths you do have to help you fulfill the basic leadership functions. I'm also not really sold on the idea that your strengths are roughly constant over your lifetime. Having read about neuroplasticity I think your strengths might change over time just based on how you live and view your life. That's not really a case for or against being a manager or leader, it just means that you'd have to be ready to adapt to an evolving set of strengths.
Thankfully, at Mozilla, unlike many other companies, it is possible to "grow" without getting pushed into management. The Mozilla staff engineer level descriptions provide two tracks - one as an individual contributor and one as a manager (assuming these descriptions are still current - and since the page was last touched almost 2 years ago it might very well not be!). At many companies this is not even an option.
For now I'm going to try to level up to "conscious competence" with respect to using my strengths and see where that gets me. Probably by then the path ahead will be more clear.
[ 0 Comments... ]
TL;DR: If you have a home wi-fi network, think about setting multiple separate VLANs as a "defense in depth" technique to protect hosts from malware.
The long version: A few years ago when I last needed to get a router, I got one which came with DD-WRT out of the box (made by Buffalo). I got it because DD-WRT (and Tomato) were all the rage back then and I wanted to try it out. While I was setting it up I noticed I could set up multiple Wi-Fi SSIDs on my home network, each with different authentication parameters. So I decided to create two - one for my own use (WPA2 encrypted) and one for guests (with a hidden SSID and no encryption). That way when somebody came over and wanted to use my Wi-Fi I could just give them the (hidden) SSID name and they would be able to connect without a password.
This turned out to a pretty good idea and served me well. Since then though I've acquired many more devices that also need Wi-Fi access and in the interest of security I've made my setup a little more complex. Consider the webcam I bought a few months ago. It shipped from somewhere in China and comes with software that I totally don't trust. Not only is it not open-source, it's not upgradeable and regularly tries to talk to some Amazon EC2 server. It would be pretty bad if malware managed to infect the webcam and not only used it to spy on me, but also used as a staging area to attack other devices on my network.
(Aside: most people with home Wi-Fi networks implicitly treat the router as a firewall, in that random devices outside the network can't directly connect to devices inside the network. For the most part this is true, but of course it's not hard for a persistent attacker to do periodic port scans to see if there are any hosts inside your network listening for connections via UPnP or whatever, and use that as an entrance vector if the service has vulnerabilities.)
Anyway, back to the webcam. I ended up only allowing it connect to an isolated Wi-Fi network and used firewall rules on the router to prevent all access to or from it, except to a single server which could access a single port on it. That server basically extracted the webcam feed and exposed it to the rest of my network. Doing this isn't a perfect solution but it adds a layer of security that makes it harder for malware to penetrate.
There's a ton of other Wi-Fi devices on my network - a printer, various smartphones, a couple of Sonos devices, and so on. As the "Internet of Things" grows this list is bound to grow as well. If you care about ensuring the security of machines on your network, and not letting become part of some random hacker's botnet, knowing how to turn your router into a full-fledged firewall is a very useful tool indeed. Even if you choose not to lock things down to the extent that I do, simply monitoring connections between devices inside your network and hosts outside your network can be a huge help.
[ 1 Comment... ]
Have you ever had the experience of trying to write a document in MS Word (or Open/LibreOffice) and it keeps "correcting" your formatting to something you don't want? The last time I experienced that was about a year ago, and that was when I decided "screw it, I'll just write this in HTML instead". That was a good decision.
Pretty much anything you might want to use a word processor for, you can do in HTML - and oftentimes it's simpler. Sure, there's a bit of a learning curve if you don't know HTML, but that's true for anything. Now anytime I need to create "a document" (a letter, random notes or signs to print, etc.) I always do it in HTML rather than LibreOffice, and I'm the happier for it. I keep all my data in git repositories, and so it's a bonus that these documents are now in a plaintext format rather than a binary blob.
I realized that this is probably part of a trend - a lot of people I know nowadays to "powerpoint" presentations using web technologies such as reveal.js. I haven't seen many people comment on using web tech to do word processing, but I know I do it. The only big "office suite" thing left is the spreadsheet. It would be awesome if somebody wrote a drop-in JS spreadsheet library that you could include into a HTML page and instantly turn a table element into a spreadsheet.
I'm reminded of this old blog post by Joel Spolsky: How Trello is different. He talks about most of the people who use Excel really just use it because it provides a table format for entering things, rather than it's computational ability. HTML already provides that, but whenever I've tried doing that I find the markup/content ratio too high, so it always seemed like a pain. It would be nice to have a WSYIWYG tool that let you build a table (or spreadsheet) and import/export it as raw HTML that you can publish, print, share, etc.
As an addendum, that blog post by Joel also introduced me to the concept of unshipped code as "inventory", which is one of the reasons I really hate finding old bugs sitting around in Bugzilla with perfectly good patches that never landed!
[ 6 Comments... ]
|