Unraveling coordinate systems, part 2

Posted by: stak
Tags: mozilla
Posted on: 2013-08-05 19:47:56

Please read Unraveling coordinate systems, part 1 for the backstory. This post describes how CSS transforms fit into the world of coordinate systems I described there.

But first, a quick status update: since I wrote that post, a fair amount of code has been converted to use the new typed classes (CSSPoint, LayoutDevicePoint, etc.). Special thanks to Ms2ger who has been incrementally converting various bits of layout code to use these where appropriate. The conversion has turned up some type mismatches and flushed out bugs on B2G, Fennec, and recently even Metro.

And now, on to the main event: CSS transforms. CSS transforms make things complicated because they are not really applied in layout code, but saved as a transform on the layer and applied at composition time. (Recall that the layer can be thought of as the rendered texture that gets uploaded to the graphics hardware). From a quick reading of the CSS transforms spec this makes sense, because the transforms are not meant to affect the layout properties of elements. However it complicates coordinate system work because a "transform: scale(2)" CSS style, for instance, doesn't change the size of the element in LayoutDevicePixels but does affect the ScreenPixel size.

If you recall from part 1, I defined LayoutDevicePixels as taking into account the device DPI and full zoom amounts; they're basically the coordinate system that the layout code outputs positioning data in. If there is an element with a 2x scale CSS transform applied, then the ScreenPixel height of the element must be twice the LayoutDevicePixel height of the element (for the moment let us ignore any OMTC transforms and resolution changes that might also be applied). However, there are no restrictions on the size of the layer, which means the LayerPixel that sits in between the LayoutDevicePixel and ScreenPixel can be anything.

So, we could, for example, have the layer be the same size as the untransformed element (1 LayoutDevicePixel == 1 LayerPixel) and then do the entire 2x scale in hardware (1 LayerPixel == 2 ScreenPixels). Or we could do something like have gecko render at higher density (1 LayoutDevicePixel == 2 LayerPixels) and then not scale it in hardware (1 LayerPixel == 1 ScreenPixel). We could even render it 4x in gecko and then scale it down in hardware by 2x, if we wanted. Rendering it small in gecko and scaling it up in hardware results in a blurry image, and rendering it large in gecko and scaling it down in hardware is a waste of memory/CPU. In theory, the scaling that results in the best visual effect without wasting resources is to have no hardware scaling and have gecko render exactly at the specified CSS transform. However, in practice, we use some heuristics to figure out the best tradeoff here because the CSS transform can change over time.

And while we're on the subject, a similar thing actually happens with the OMTC transform. If we take a regular page and pinch-zoom it to 2x, what actually happens on the layout side is that the presShell resolution is set to 2x, the CSS transform on the page is updated to be 0.5x, and the OMTC transform is again 2x. So the net effect is that the page is scaled up by 2x, with 1 LayoutDevicePixel == 2 LayerPixels == 2 ScreenPixels, but the fact that the CSS transform on the layer is modified is important from an implementation point of view.

Ok, so that's all well and good, and gives us nice-looking CSS transforms efficiently, at the expense of some complicated coordinate transforms. Now we can flesh out our summary from last time to be a little more precise:

CSSPixel x widget scale x full zoom = LayoutDevicePixel
LayoutDevicePixel x CSS-driven resolution x OMTC-driven resolution = LayerPixel
LayerPixel x CSS transforms x OMTC transforms = ScreenPixel

But that's not all. Note that CSS transforms apply to elements in the DOM rather than the entire document. This means that different layers in the layer tree can have different transforms, and these transforms accumulate when mapping a particular layer up to the screen. Madness! That means there's no longer just one set of LayerPixels but many! And with the work that I've been doing to make subframes asynchronously scrollable, many of the layers can have their own individual OMTC-driven resolutions and OMTC transforms as well! More madness! (In practice only top-level layers for PresShells will have OMTC resolutions but I wouldn't count on that always remaining the case.)

So to update the summary again, I need to introduce some layers. Assume that layer L is some random layer in the tree, it has a parent layer M, grandparent layer N, and so on up until the root layer R. Then:

CSSPixel x widget scale x full zoom = LayoutDevicePixel
LayoutDevicePixel x CSS-driven resolution (for R) x OMTC-driven resolution (for R) = LayerPixel (for R)
LayerPixel (for R) x CSS-driven resolution (for Q) x OMTC-driven resolution (for Q) = LayerPixel (for Q)
LayerPixel (for Q) x CSS-driven resolution (for P) x OMTC-driven resolution (for P) = LayerPixel (for P)
...
LayerPixel (for M) x CSS-driven resolution (for L) x OMTC-driven resolution (for L) = LayerPixel (for L)
LayerPixel (for L) x CSS transform (for L) x OMTC transform (for L) x CSS transform (for M) x OMTC transform (for M) x ... x CSS transform (for R) x OMTC transform (for R) = ScreenPixel

Simple, right? :) If I get around to it, the next post will cover how input events are mapped back from screen space to layout space. (It's not as straightforward as you might think.)

Thanks to Robert O'Callahan for reading a draft of this post and pointing out some things to fix.

[ Add a new comment ]