Unraveling coordinate systems

All timestamps are based on your local time of:

Posted by: stak
Tags: mozilla
Posted on: 2013-06-24 22:28:21

For the last few weeks I've been hacking away at a jungle of coordinate systems in the graphics code, trying to make code easier to understand and to work with. This work is technically part of the effort to make subframes asynchronously scrollable in OMTC, but it has helped us to fix other bugs as well. This post a braindump of the coordinate systems I've uncovered and the mental model I have.

As of this writing, I have defined four "pixel" types in layout/base/Units.h. These are CSSPixel, LayoutDevicePixel, LayerPixel, and ScreenPixel. CSSPixel is the simplest - it represents a CSS pixel, which is what web content authors use to specify dimensions of things. Almost every Web API [1] deals with CSS pixels.

In the days of yore, 1 CSS pixel corresponded exactly to one screen pixel. That is, when you created a div that was 50 CSS pixels wide, it would show up as 50 pixels wide on your screen. If you have a desktop monitor that is 1024 pixels wide [2] and you create a div that is 1024 CSS pixels wide, it takes up your entire monitor width. Makes sense, right?

Now, let us enter the world of HiDPI display devices. On HiDPI devices, the screen pixels are so small and packed together so tightly that mapping 1 CSS pixel to 1 screen pixel doesn't make sense any more. Sure, you can fit more on the screen, but it's too tiny to be readable or useful. So we have this concept of a widget scale. The nsIWidget::GetDefaultScale() returns a scaling factor to account for HiDPI displays. For example, see the gonk implementation which makes B2G perform a scale amount based on the actual DPI of the device. This introduces a new coordinate system that layout refers to as "device pixels" and what I've called LayoutDevicePixels.

In the layout code, the widget scale affects two main things. One is the size of the CSS viewport [3]. If we have a widget scale of 2.0, then each CSS pixel is 2 screen pixels wide, which means that on a 1024x768 screen with a maximized browser window, you can only fit 512 CSS pixels across. So the CSS viewport width is adjusted to account for this. The other is the scale factor from app units [4] to screen pixels. Instead of the usual 60 app units being converted to 1 screen pixel you would normally get, setting a 2.0 widget scale results in 30 app units being converted to 1 screen pixel. This effectively makes each CSS pixel twice as wide when being rendered to the screen, which is the point of the whole exercise.

But that's not all! There is another factor that comes into play between CSS pixels and LayoutDevicePixels, and that is the "full zoom". This is when, on desktop Firefox, you perform the "Zoom In" or "Zoom Out" actions. Doing a "Zoom In" action increases the full zoom, which works almost identically to increasing the widget scale. That is, setting a zoom of 110% is pretty much equivalent to having a widget scale of 1.1. The only difference I'm aware of is that if you specify your CSS dimensions in real-world units such as inches, then they are affected by the widget scale but not by the "full zoom". Tricksy!

So to summarize what we have so far: in a world with just CSSPixels and LayoutDevicePixels, dimensions are specified in CSSPixels by content authors, and the browser maps them to LayoutDevicePixels based on the widget scale and full zoom. These LayoutDevicePixels are then displayed 1:1 on screen pixels as defined by the underlying platform.

But what about mobile? Welcome to the land of OMTC and pinch-zoom. OMTC stands for off-main-thread compositor, and is what allows you to pinch a page on Fennec and have it instantly zoom. What's happening here is the painted page is transformed in OpenGL [5], without Gecko really knowing about what's going on. Since Gecko isn't repainting anything, this is super fast, and allows us to animate pinch-zoom at 60 frames per second (or close to it).

Unfortunately, it also introduces a new coordinate system, because the LayoutDevicePixels that Gecko produced are no longer displayed 1:1 on the screen. Gecko could have painted something 10 LayoutDevicePixels wide (and so the texture uploaded to the graphics hardware would be 10 pixels wide) but then the user does a pinch-zoom and BAM! now it's taking up 20 pixels on the screen, because we told OpenGL to scale it up by 2x. So here we have our third pixel type defined: the ScreenPixel. In the preceding example the pinch-zoom produced a scale factor of 2.0 and so 10 LayoutDevicePixels would get mapped to 20 ScreenPixels.

Now say you're viewing a page in Fennec and zoom in using pinch-zoom. And then you zoom in some more. And then some more. If all we did was take the LayoutDevicePixels and tell OpenGL to render them bigger by scaling it in hardware, you would end up with a very pixellated and blurry view of the page. In order to make it look good again, we have to go back to Gecko and tell it to repaint the visible area of the page at a higher density, allowing us to remove the OpenGL scaling. For example, instead of rendering a paragraph of text into a texture and scaling that up in OpenGL to display a single word really big, we can tell Gecko to just render that one word really big, and to use up the entire texture to do it. This is done by a call to nsIPresShell::SetResolution().

Setting the resolution doesn't change our definition of CSSPixels or LayoutDevicePixels, but it does change something. To describe this change, we need to introduce a new coordinate system. This is the LayerPixel [6], and it sits between LayoutDevicePixel and ScreenPixel. That is, the resolution changes how many LayerPixels are produced for each LayoutDevicePixel, and now pinch-zooming affects the scale between LayerPixels and ScreenPixels. This is a little cyclical because as you perform a pinch-zoom, LayerPixels and ScreenPixels get farther apart in size. Then, once you finish the zoom, we tell Gecko to re-render the content at a new resolution such that LayerPixels and ScreenPixels are the same size once again, and we render the new LayerPixels at a 1:1 scale on the screen. When this happens the visible content goes from being blurry back to being sharp, which you can see in Fennec if you pay close attention when zooming in.

So to summarize, here is what we have now:
CSSPixel x widget scale x full zoom = LayoutDevicePixel
LayoutDevicePixel x resolution = LayerPixel
LayerPixel x OMTC transforms = ScreenPixel

And so ends my braindump. Hopefully what I've written above will not change going forward, and the terms I've used can become part of the Gecko developer vocabulary, so that when dealing with code in different coordinate systems it's much easier to agree on what we mean.

(Update Aug 5, 2013: I now have a part 2 that describes how CSS transforms fit into this picture.)

[1] The only exception I can think of is window.outerWidth, which is in screen pixels. (Edit 2013-06-27: I was wrong, I think outerWidth is also in CSS pixels.)
[2] Note that the number of screen pixels displayed on a physical device may be modified by the operating system. For example, you can change your screen resolution from 1024x768 to 800x600, which changes the size and number of screen pixels. That's fine - we don't need to account for that in our code as the OS takes care of it.
[3] The CSS viewport affects how wide the page is laid out by the layout code. One way of visualizing this is that if you have a page with just plain text, the text will be wrapped so that it doesn't exceed the CSS viewport width.
[4] App units are what layout does all of its calculations in. One app unit is exactly one-sixtieth of a CSS pixel. The "60" was chosen because it has many integer factors and so allows representing common fractions losslessly.
[5] The actual graphics system in use depends on the platform. OpenGL is just an example.
[6] When Gecko paints the various elements on a page, it flattens them into "layers" that it hands off to the graphics stack. This is where the term LayerPixel comes from.

Posted by Robert O'Callahan at 2013-06-25 00:19:44
You're the man!

Surely [1] is a bug we should fix.
[ Reply to this ]
Posted by stak at 2013-06-25 07:38:10
I actually think it makes sense for outerWidth to be in screen pixels, because it represents the size of the browser window including chrome. On a device that has pinch-zoom but also allows a non-maximized browser window, outerWidth would be the only way for content to know how many pixels on the screen the browser is taking up.
[ Reply to this ]
Posted by stak at 2013-06-27 10:53:39
Actually I think I was wrong. outerWidth and outerHeight are also in CSS pixels, but I'm not sure it's a meaningful metric when the LayoutDevicePixel is a different size from the CSSPixel. It also means that the window.outerWidth value changes as you apply "full zoom" to the page.
[ Reply to this ]
Posted by Emanuel Hoogeveen at 2013-06-25 02:53:40
"The only difference I'm aware of is that if you specify your CSS dimensions in real-world units such as inches, then they are affected by the widget scale but not by the "full zoom"."
Why is this? It feels like full zoom should work the same way as pinch zoom - that is, I don't understand why it would want to change the size of some elements but not others.
[ Reply to this ]
Posted by stak at 2013-06-25 07:49:42
Full zoom does the change the size of all elements. The main difference between full zoom and pinch-zoom is that full zoom tries to reflow the content so that it doesn't get wider than the browser window, even at the enlarged size. This makes it more useful than pinch-zoom, in my mind, because pinch-zoom puts you in a state that requires horizontal scrolling (for which we need to implement yet other workarounds). The reason for this I think is largely interaction patterns - when a user does a pinch action it is far more intuitive for the entire page to scale so that the points under the user's finger remain the same.

As for the inches behaviour, increasing zoom via full zoom and pinch-zoom both make CSS-specified inches appear larger on the screen. This makes sense because these zooms are intended as accessibility features for cases where the author and user disagree on what is too small. Widget scale, on the other hand, is more of a hardware compensation behaviour, and we assume that if authors are using units like inches then they should actually get inches regardless of DPI.
[ Reply to this ]
Posted by Emanuel Hoogeveen at 2013-06-25 08:12:40
Thanks for the response. I'm glad to hear it's not just some legacy behavior that's stuck around. I'm still a bit confused by your response, however. In particular, these statements appear to contradict each other:

"if you specify your CSS dimensions in real-world units such as inches, then they are affected by the widget scale but not by the "full zoom"."

"increasing zoom via full zoom and pinch-zoom both make CSS-specified inches appear larger on the screen."
Allowed expansions in comments/replies: [i]italic[/i], [u]underline[/u], [b]bold[/b], [code]code[/code], [sub]subscript[/sub], [sup]superscript[/sup], [url=http://some.url]linked text[/url]
Human verification: Sum of twenty-one and forty-one =
Posted by stak at 2013-06-25 09:20:19
Ah, I think my first statement you quoted was ambiguous. What I meant there is that "If you specify your CSS dimensions in real-world units such as inches, then their size in CSS pixels is affected by the widget scale but not by the full zoom". Does that help clarify?
[ Reply to this ]
Posted by Emanuel Hoogeveen at 2013-06-25 11:13:06
Ah! Their size in CSS pixels, not screen pixels. I think that's a somewhat convoluted way of saying it, but I understand now.
[ Reply to this ]
Posted by Zack at 2013-06-25 08:42:50
If you're hacking on this stuff, you might wanna take a look at bug 651018 and bug 651022. I'm not sure how much that stuff is still used in the layer world, but it's still taking up mindspace regardless.
[ Reply to this ]
Posted by stak at 2013-06-25 10:03:00
Thanks for the pointers!
[ Reply to this ]

[ Add a new comment ]

(c) Kartikaya Gupta, 2004-2024. User comments owned by their respective posters. All rights reserved.
You are accessing this website via IPv4. Consider upgrading to IPv6!