Unraveling coordinate systems

Posted by: stak
Tags: mozilla
Posted on: 2013-06-24 22:28:21

For the last few weeks I've been hacking away at a jungle of coordinate systems in the graphics code, trying to make code easier to understand and to work with. This work is technically part of the effort to make subframes asynchronously scrollable in OMTC, but it has helped us to fix other bugs as well. This post a braindump of the coordinate systems I've uncovered and the mental model I have.

As of this writing, I have defined four "pixel" types in layout/base/Units.h. These are CSSPixel, LayoutDevicePixel, LayerPixel, and ScreenPixel. CSSPixel is the simplest - it represents a CSS pixel, which is what web content authors use to specify dimensions of things. Almost every Web API [1] deals with CSS pixels.

In the days of yore, 1 CSS pixel corresponded exactly to one screen pixel. That is, when you created a div that was 50 CSS pixels wide, it would show up as 50 pixels wide on your screen. If you have a desktop monitor that is 1024 pixels wide [2] and you create a div that is 1024 CSS pixels wide, it takes up your entire monitor width. Makes sense, right?

Now, let us enter the world of HiDPI display devices. On HiDPI devices, the screen pixels are so small and packed together so tightly that mapping 1 CSS pixel to 1 screen pixel doesn't make sense any more. Sure, you can fit more on the screen, but it's too tiny to be readable or useful. So we have this concept of a widget scale. The nsIWidget::GetDefaultScale() returns a scaling factor to account for HiDPI displays. For example, see the gonk implementation which makes B2G perform a scale amount based on the actual DPI of the device. This introduces a new coordinate system that layout refers to as "device pixels" and what I've called LayoutDevicePixels.

In the layout code, the widget scale affects two main things. One is the size of the CSS viewport [3]. If we have a widget scale of 2.0, then each CSS pixel is 2 screen pixels wide, which means that on a 1024x768 screen with a maximized browser window, you can only fit 512 CSS pixels across. So the CSS viewport width is adjusted to account for this. The other is the scale factor from app units [4] to screen pixels. Instead of the usual 60 app units being converted to 1 screen pixel you would normally get, setting a 2.0 widget scale results in 30 app units being converted to 1 screen pixel. This effectively makes each CSS pixel twice as wide when being rendered to the screen, which is the point of the whole exercise.

But that's not all! There is another factor that comes into play between CSS pixels and LayoutDevicePixels, and that is the "full zoom". This is when, on desktop Firefox, you perform the "Zoom In" or "Zoom Out" actions. Doing a "Zoom In" action increases the full zoom, which works almost identically to increasing the widget scale. That is, setting a zoom of 110% is pretty much equivalent to having a widget scale of 1.1. The only difference I'm aware of is that if you specify your CSS dimensions in real-world units such as inches, then they are affected by the widget scale but not by the "full zoom". Tricksy!

So to summarize what we have so far: in a world with just CSSPixels and LayoutDevicePixels, dimensions are specified in CSSPixels by content authors, and the browser maps them to LayoutDevicePixels based on the widget scale and full zoom. These LayoutDevicePixels are then displayed 1:1 on screen pixels as defined by the underlying platform.

But what about mobile? Welcome to the land of OMTC and pinch-zoom. OMTC stands for off-main-thread compositor, and is what allows you to pinch a page on Fennec and have it instantly zoom. What's happening here is the painted page is transformed in OpenGL [5], without Gecko really knowing about what's going on. Since Gecko isn't repainting anything, this is super fast, and allows us to animate pinch-zoom at 60 frames per second (or close to it).

Unfortunately, it also introduces a new coordinate system, because the LayoutDevicePixels that Gecko produced are no longer displayed 1:1 on the screen. Gecko could have painted something 10 LayoutDevicePixels wide (and so the texture uploaded to the graphics hardware would be 10 pixels wide) but then the user does a pinch-zoom and BAM! now it's taking up 20 pixels on the screen, because we told OpenGL to scale it up by 2x. So here we have our third pixel type defined: the ScreenPixel. In the preceding example the pinch-zoom produced a scale factor of 2.0 and so 10 LayoutDevicePixels would get mapped to 20 ScreenPixels.

Now say you're viewing a page in Fennec and zoom in using pinch-zoom. And then you zoom in some more. And then some more. If all we did was take the LayoutDevicePixels and tell OpenGL to render them bigger by scaling it in hardware, you would end up with a very pixellated and blurry view of the page. In order to make it look good again, we have to go back to Gecko and tell it to repaint the visible area of the page at a higher density, allowing us to remove the OpenGL scaling. For example, instead of rendering a paragraph of text into a texture and scaling that up in OpenGL to display a single word really big, we can tell Gecko to just render that one word really big, and to use up the entire texture to do it. This is done by a call to nsIPresShell::SetResolution().

Setting the resolution doesn't change our definition of CSSPixels or LayoutDevicePixels, but it does change something. To describe this change, we need to introduce a new coordinate system. This is the LayerPixel [6], and it sits between LayoutDevicePixel and ScreenPixel. That is, the resolution changes how many LayerPixels are produced for each LayoutDevicePixel, and now pinch-zooming affects the scale between LayerPixels and ScreenPixels. This is a little cyclical because as you perform a pinch-zoom, LayerPixels and ScreenPixels get farther apart in size. Then, once you finish the zoom, we tell Gecko to re-render the content at a new resolution such that LayerPixels and ScreenPixels are the same size once again, and we render the new LayerPixels at a 1:1 scale on the screen. When this happens the visible content goes from being blurry back to being sharp, which you can see in Fennec if you pay close attention when zooming in.

So to summarize, here is what we have now:
CSSPixel x widget scale x full zoom = LayoutDevicePixel
LayoutDevicePixel x resolution = LayerPixel
LayerPixel x OMTC transforms = ScreenPixel

And so ends my braindump. Hopefully what I've written above will not change going forward, and the terms I've used can become part of the Gecko developer vocabulary, so that when dealing with code in different coordinate systems it's much easier to agree on what we mean.

(Update Aug 5, 2013: I now have a part 2 that describes how CSS transforms fit into this picture.)

Notes:
[1] The only exception I can think of is window.outerWidth, which is in screen pixels. (Edit 2013-06-27: I was wrong, I think outerWidth is also in CSS pixels.)
[2] Note that the number of screen pixels displayed on a physical device may be modified by the operating system. For example, you can change your screen resolution from 1024x768 to 800x600, which changes the size and number of screen pixels. That's fine - we don't need to account for that in our code as the OS takes care of it.
[3] The CSS viewport affects how wide the page is laid out by the layout code. One way of visualizing this is that if you have a page with just plain text, the text will be wrapped so that it doesn't exceed the CSS viewport width.
[4] App units are what layout does all of its calculations in. One app unit is exactly one-sixtieth of a CSS pixel. The "60" was chosen because it has many integer factors and so allows representing common fractions losslessly.
[5] The actual graphics system in use depends on the platform. OpenGL is just an example.
[6] When Gecko paints the various elements on a page, it flattens them into "layers" that it hands off to the graphics stack. This is where the term LayerPixel comes from.

[ Add a new comment ]