Charles Petzold: VirtualSurfaceImageSource and DPI Resolution

VirtualSurfaceImageSource and DPI Resolution

October 23, 2013
New York, N.Y.

Sometimes you get some software working just fine, but the visuals don't look quite right. "Maybe it's just me," you might think. "Maybe I'm just hypercritical. Maybe nobody else will notice."

But it continues to nag at you, and finally you need to figure out what the problem is, and (with any luck) how to fix it.

This happened recently when I was working on "Text Formatting and Scrolling with DirectWrite", the most recent installment of my DirectX Factor column for MSDN Magazine. In the article I discuss different approaches of using DirectWrite to display and scroll Chapter 7 of Lewis Carroll’s Alice’s Adventures in Wonderland.

The penultimate program in that article, called ScrollableAlice, displayed text that looked like this on my Microsoft Surface Pro:

But the last program in that article was the one that didn't look quite right. This one was named BounceScrollableAlice, and the enhancement over the previous version was the addition of some Windows 8 style "bounce" to the scrolling. Like I said, otherwise the program worked just fine. The text was the same size as in ScrollableAlice, and the paragraphs were formatted in exactly the same way. But... well, take a look for yourself:

In the side-by-side comparison, it's obvious something is wrong. The text really shouldn't be this fuzzy.

From the user's perspective, the difference between ScrollableAlice and BounceScrollableAlice is just a little bounce in the scrolling. But internally, the programs are very different. Adding the bounce required a different approach to rendering the text.

Mostly when rendering DirectX graphics in Windows 8 programs I've been using the SwapChainBackgroundPanel and (since the availability of Windows 8.1) the more versatile SwapChainPanel. Visual Studio has made this option easy by supplying a project template that generates all the messy overhead. The ScrollableAlice program uses the SwapChainPanel.

For the BounceScrollableAlice program I wanted to put the rendering surface inside a ScrollViewer to get the familiar Windows 8 bounce when you try to scroll beyond the top or bottom. In theory, you can indeed put a SwapChainPanel in a ScrollViewer and scroll it. It is one of the enhancements of SwapChainPanel over SwapChainBackgroundPanel that it can occupy a place in the visual tree that is not the root of the page.

However, although you can put a SwapChainPanel in a ScrollViewer you should generally avoid doing so. The big reason to do it in the first place is to scroll content that is larger than the screen, but how much larger is it? SwapChainPanel has the same size restriction as other DirectX bitmaps. I've found the GetMaximumBitmapSize method of the ID2D1DeviceContext1 object to typically return a value of 16384 pixels, which is the limit on either dimension. That's about 15 times the height of a 1080-line display, which means that if you tried to display a really long chapter or an entire book on a SwapChainPanel, you'll probably hit the limit.

Windows 8 offers another approach to integrating DirectX and controls such as ScrollViewer. The SurfaceImageSource class derives from the Windows Runtime ImageSource class so you can put a SurfaceImageSource in an Image element, which then goes into the ScrollViewer. However, SurfaceImageSource has the same bitmap size restriction as SwapChainPanel.

But there's still another option: The VirtualSurfaceImageSource class derives from SurfaceImageSource and can be as large as you want. The catch is that your program needs to redraw the VirtualSurfaceImageSource surface as off-screen parts are scrolled into view. What you draw is not cached as the areas scroll off the screen. It's a completely dynamic process.

It was obvious to me that VirtualSurfaceImageSource was precisely what I needed for the BounceScrollableAlice program, but SurfaceImageSource and VirtualSurfaceImageSource are not quite as easy to use as SwapChainPanel, mainly because there's no Visual Studio template to help with all the overhead. When putting together the BounceScrollableAlice project, I looked at some sample Windows 8.1 code from Microsoft, and copied some overhead from that.

To handle callbacks from the VirtualSurfaceImageSource object you need a class that implements the IVirtualSurfaceUpdatesCallbackNative interface. The BounceScrollableAlice program includes a class I named VirtualSurfaceImageSourceRenderer that implements this interface, and this turned out to be a convenient place to put all the overhead necessary to use the VirtualSurfaceImageSource. A project-specific class named AliceVsisRenderer class then derives from VirtualSurfaceImageSourceRenderer.

My current development setup is built around a Microsoft Surface Pro running Windows 8.1 with a real keyboard and mouse attached via a USB hub, and a larger secondary monitor set up to extend the tablet screen. Visual Studio 2013 runs on the desktop on the large monitor, and the Windows Store application that I'm currently developing usually runs on the Surface Pro's monitor, but I can also run Windows Store programs on the big screen.

The difference in rendered text between the ScrollableAlice and BounceScrollableAlice programs only existed when running the two programs on the Surface Pro screen. On the large monitor, they looked the same, like this:

As you can see, this screen shot looks quite different from the first two. On the larger monitor, the font has a smaller pixel size and the text is formatted differently. Yet both the Surface Pro screen and the external monitor have pixel resolutions of 1920 × 1080.

The difference between the two monitors is very much related to the problem I was experiencing with the fuzzy text. Both involve the assumed DPI resolution of the video display.

A little background:

It's convenient for programmers working in a graphical environment to be able to draw objects that aren't too small and aren't too large, and with any luck are "just right." The programmer wants some assurance that a button is big enough to press with a finger, and a font is readable on different display sizes. At the same time, it's most convenient for the programmer to hard-code sizes of graphical objects and fonts, and to use these same sizes when sending graphics to printers, which generally have much higher resolutions than video displays.

For this reason, the Windows Runtime basically guarantees that developers can pretty much safely assume that the graphical output device — be it a video display or a printer — has a resolution of 96 dots to the inch. You want to draw a one-quarter-inch square? Use dimensions of 24. Want to display a 12-point font? Specify a font size of 16.

Now obviously not every graphical output device has a resolution of 96 DPI. The two screens in my development setup both have pixel dimensions of 1920 × 1080, but the Surface Pro has a diagonal screen size of about 10½ inches, while the external monitor is double that: 21 inches. Do the math and you'll discover that the Surface Pro has an actual resolution of about 210 dots per inch, while the external monitor is about 105 DPI.

To allow these two screens to be used in a device-independent manner, the Windows Runtime makes an assumption. By default, it assumes that my external monitor has a resolution of 96 DPI — which is pretty close to reality — but the Surface Pro screen has a resolution 40% higher, or 134.4 DPI.

The Windows Runtime uses this DPI assumption to make adjustments to all sizes and coordinates used or encountered by an application. To a Windows Runtime program running on the Surface Pro, the screen doesn't have a size of 1920 × 1080 but instead appears to be 1371.43 × 771.429. This is what's reported in the ActualWidth and ActualHeight properties of a full-screen element, and this is what's reported by the SizeChanged event handler on a full-screen element. All graphics displayed by the program are also scaled by 40%. If a program draws a line from (100, 100) to (150, 100), the rendered size will actually encompass 70 pixels. If the program specifies a font size of 16, the actual font size is 22.4 pixels.

Shouldn't Windows be assuming that the Surface Pro screen has a resolution closer to 200 DPI rather than 134.4 DPI? Well, if you go solely by the pixels and diagonal measurement, yes. But keep in mind that the higher the assumed resolution, the smaller the screen appears to the application. If the Surface Pro screen resolution were actually assumed to be about 200 DPI, it would only display a quarter of the content of a larger monitor of the same pixel dimensions. But a higher pixel density makes graphics sharper, so they can afford to be somewhat smaller and still be visible, so 134.4 is somewhat of a compromise.

Usually programmers working with the Windows Runtime simply assume to be dealing entirely in units of pixels. But it is more correct to say that we're working in "device-independent units" (DIUs) or "device-independent pixels" (although this latter term grates on me). I first encountered this concept in the Windows Presentation Foundation, and blogged about it 8 years ago. (Eight years? OMG!)

A user can change the assumed resolution of a video display in the Display Properties applet of the Control Panel. Under Windows 8, if you extend the tablet screen with an external monitor, both screens were forced to have the same DPI resolution. In Windows 8.1, they can have different resolutions and by default, they do. If you drag a desktop app from one screen to the other, you can actually see it change size as it crosses the threshold. You can't drag Windows Store apps from one screen to another, but on either screen you can select a program from the app list (viewable with a sweep on the left side of the screen) if it's not currently active on either screen.

Mostly the DPI adjustment is invisible to Windows Runtime applications. If you're displaying vector graphics and text, everything usually works just fine if you simply ignore it. The graphics get rendered based on the actual pixels of the video display rather than the assumed resolution. If your program uses DirectX to render graphics, you can also use device-independent units for consistency with the rest of your program, and DirectX can adjust for that. (ID2D1RenderTarget has a SetDpi method for this purpose.) In the Windows Store DirectX templates in Visual Studio, the DirectXBase class has the necessary maintenance code, and DirectXPage notifies it when the device resolution changes.

Bitmaps present a more difficult problem. You probably want to render bitmaps so that bitmap pixels map directly to display pixels. On a 96 DPI screen, you can display a 300 × 300 bitmap in its pixel size, and it will occupy about 3 square inches on the screen. On a 134.4 DPI screen, you probably also want the bitmap to occupy 3 square inches so it should also be displayed in 300 square device-independent units, but the bitmap itself should be 420 pixels square for maximum fidelity. The Windows Runtime allows a program to maintain bitmaps of several different resolutions and can automatically choose between them. The AutoImageSelection program in Chapter 12 of Programming WIndows, 6th edition demonstrates this technique.

The SurfaceImageSource and VirtualSurfaceImageSource are basically bitmaps, and my first version of BounceScrollableAlice didn't make any adjustment for the DPI resolution of the screen it was running on. When running on the Surface Pro screen, the program sets the width of the VirtualSurfaceImageSource to 1371 pixels, which it perceives to be the width of the screen. All the text is rendered on this surface under the assumption that the display has a resolution of 96 DPI. But then this VirtualSurfaceImageSource is rendered at the actual pixel width of 1920, which causes the image to be expanded and made a bit fuzzy.

Modifying BounceScrollableAlice to be DPI aware was a bit of a challenge. I knew immediately I'd have to add a SetDpi method to VirtualSurfaceImageSourceRenderer, but I discovered I also needed an override to that method in the derived AliceVsisRenderer class.

The XAML file with the ScrollViewer and Image element looks like this:

After the MainPage constructor In the code-behind file creates the VirtualSurfaceImageSource and AliceVsisRenderer classes, it defines two anonymous event handlers for the SizeChanged event and the DpiChanged event of DisplayInformation:

MainPage::MainPage() : m_dpi(96)
{
    InitializeComponent();

    // Create VirtualSurfaceImageSource to display text
    VirtualSurfaceImageSource^ vsis =
            ref new VirtualSurfaceImageSource(0, 0, true);
    img->Source = vsis;

    // Create AliceVsisRenderer based on that VirtualSurfaceImageSource
    m_vsisRenderer = new AliceVsisRenderer(vsis);
    m_vsisRenderer->Initialize();

    // Load in the text
    ...

    // Set a new width for the display
    SizeChanged += ref new SizeChangedEventHandler(
            [this] (Object^, SizeChangedEventArgs^ args)
    {
        m_vsisRenderer->SetWidth(args->NewSize.Width);
        SetImageSize();
    });

    DisplayInformation^ displayInformation =
            DisplayInformation::GetForCurrentView();
    m_dpi = displayInformation->LogicalDpi;
    m_vsisRenderer->SetDpi(m_dpi);

    displayInformation->DpiChanged +=
        ref new TypedEventHandler<DisplayInformation^, Object^>(
                [this](DisplayInformation^ sender, Object^)
    {
        m_dpi = sender->LogicalDpi;
        m_vsisRenderer->SetDpi(m_dpi);
        SetImageSize();
    });
}

Notice that the width of the screen that SetWidth sends to AliceVsisRenderer is in device-independent units. Both the SetWidth and SetDpi methods of AliceVsisRenderer shown below call ResizeAndInvalidate. This method uses an instance of AliceParagraphGenerator to format the paragraphs based on the device-independent width, and obtains a device-independent height. It also calculates a pixel width and height to resize the VirtualSurfaceImageSource object, so that object has a size that's appropriate for the display:

void AliceVsisRenderer::SetWidth(float width)
{
    this->m_width = width;
    ResizeAndInvalidate();
}

void AliceVsisRenderer::SetDpi(float dpi)
{
    VirtualSurfaceImageSourceRenderer::SetDpi(dpi);
    ResizeAndInvalidate();
}

void AliceVsisRenderer::ResizeAndInvalidate()
{
    if (m_width == 0 || m_aliceParagraphGenerator.ParagraphCount() == 0)
        return;

    // Hard code 50-DIU left and right margin
    float height = m_aliceParagraphGenerator.SetWidth(m_width - 100);

    // Calculate pixel width and height
    m_pixelWidth = (int) (m_dpi * m_width / 96);
    m_pixelHeight = (int) (m_dpi * height / 96);

    // Resize the VSIS
    m_vsisNative->Resize(m_pixelWidth, m_pixelHeight);

    // Invalidate the VSIS
    RECT rect;
    rect.left = 0;
    rect.top = 0;
    rect.right = m_pixelWidth;
    rect.bottom = m_pixelHeight;
    m_vsisNative->Invalidate(rect);
}

SIZE AliceVsisRenderer::GetPixelSize()
{
    SIZE pixelSize = { m_pixelWidth, m_pixelHeight };
    return pixelSize;
}

Notice the GetPixelSize method. This is called by MainPage after setting a new device-independent width or DPI setting:

void MainPage::SetImageSize()
{
    SIZE pixelSize = m_vsisRenderer->GetPixelSize();
    img->Width = 96 * pixelSize.cx / m_dpi;
    img->Height = 96 * pixelSize.cy / m_dpi;
}

That code sets the size of the Image element to the same size as the VirtualImageSource that it's hosting, but converted to device-independent values.

The final two pieces of this enhancement are in VirtualSurfaceImageSourceRenderer. The SetDpi method is implemented here by simply passing the value to the ID2D1DeviceContext object:

void VirtualSurfaceImageSourceRenderer::SetDpi(float dpi)
{
m_dpi = dpi;
m_d2dContext->SetDpi(dpi, dpi);
}

This setting causes coordinates and sizes passed to drawing methods to be properly scaled. In this particular program, the AliceParagraphGenerator specifies a font size of 24 in the CreateTextFormat method. When SetDpi is called on the device context, and calls to DrawTextLayout occur later, this font size will be scaled based on the DPI setting. For a setting of 134.4 dots per inch, the font size is scaled to 33.6 pixels. That results in the larger text you see in the first two screen shots of this blog entry. But that text is being rendered on a VirtualSurfaceImageSource with a width of 1920 pixels, which is then displayed on the Surface Pro screen, so it doesn't seem quite so large.

But we're not done yet! My VirtualSurfaceImageSourceRenderer class implements the IVirtualSurfaceUpdatesCallbackNative interface, the primary purpose of which is to provide an UpdatesNeeded method for drawing on the VirtualSurfaceImageSource. The update rectangle is in units of pixels, so those pixels must be converted to device-indendent units before that information is used to draw on the bitmap:

void VirtualSurfaceImageSourceRenderer::Draw(RECT updateRect)
{
    POINT offset;
    ComPtr<IDXGISurface> surface;
    HRESULT hr = m_vsisNative->BeginDraw(updateRect, &surface, &offset);
    ...

    // Convert offset and updateRect to DIUs
    float dpiAdj = 96 / m_dpi;
    D2D_POINT_2F fOffset = Point2F(dpiAdj * offset.x, dpiAdj * offset.y);
    D2D_RECT_F fUpdateRect = RectF(dpiAdj * updateRect.left,
                                   dpiAdj * updateRect.top,
                                   dpiAdj * updateRect.right,
                                   dpiAdj * updateRect.bottom);
    ...
}

If I had run the BounceScrollableAlice program solely on a 96 DPI monitor, I might never have known there was a problem. Which, of course, makes me a little nervous: In the complexity of DirectX, I wonder what else I'm missing!