Linux support for graphics cards are very important for desktop and mobile users: they want to run games, composite their applications and have a nice and modern user experience.

So it’s usual that all the eyes are on this area when you want to optimize your embedded device’s user experience… but how the graphics drivers communicate with user-space applications?

Víctor Jáquez, fromIgalia, wrote a very nice introduction to this topic. If you are interest on Linux graphics stack in general, you have thispost wrote by Jasper St. Pierre.

The Direct Rendering Infrastructure (DRI) in Linux is like is shown at the above picture. Using Xorg as an X server example, we see that it is in charge of X11 drawing commands along withMesa or Gallium3D as the OpenGL software implementation for rendering three-dimensional graphics.

How they communicate with the actual HW? One possibility is using libdrm as a backend to talk with theDirect Rendering Manager at Linux kernel. DRM is divided in two in-kernel drivers: a generic drm driver and another which has specific support for the video hardware.

There is a Xorg driver running in user-space who receives the data to be passed to the HW. Using Nouveau as an example, thexf86-video-nouveau is the DDX (Device Dependent X) component inside of X stack.

It communicates with the in-kernel driver using ioctl. Some of them are generic for all the drivers and they are defined in the generic drm driver. Inside ofdrivers/gpu/drm/drm_drv.c file you have the relationship between the ioctl number, its corresponding function and its capabilities.

However, the specific driver for the video hardware can define its own ioctls in a similar way. For example, here you have the corresponding ones for the Nouveau driver (drivers/gpu/drm/nouveau/nouveau_drm.c file).

As you can see, most of these ioctls can be grouped by functionality:

• Get information from the drm generic driver: stats, version number, capabilities, etc.
• Get/set parameters: gamma values, planes, color pattern, etc.
• Buffer’s management: ask for new buffer, destroy, push buffer, etc.
• Memory management: GEM, setup MMIO, etc.
• Framebuffer management.
• In case of Nouveau: channel allocation (for context switching).
• …

Basically, the xorg user-space driver may prepare a buffer of commands to be executed on the GPU and pass it through a ioctl call. Sometimes, it just want to use the framebuffer to draw on the screen directly. Others, it usesKMS capabilities to change parameters of the screen: video mode, resolution, gamma correction, etc.

There are more things that DRM can do inside of the kernel. It can setup the color pattern used to draw pixels on the screen, it can select which encoder is going to be used and on which connector (LVDS, D-Sub, HDMI, etc), pickEDID information from the monitor, manage vblank IRQ…

At Igalia we have a long background working inLinux multimedia stack (GStreamer, WebGL, OpenCL,drivers, etc). If you need help to develop, optimize or just you want advice, please don’t hesitate tocontact us.

http://blog.samuelig.es/2013/05/14/introduction-to-linux-graphics-drivers-drm/

Brief introduction to graphics in Linux

If you want to dump images onto your screen, you can simply use the frame buffer device. It provides an abstraction for the graphics hardware and represents the frame buffer of the video hardware. This kernel device allows user application to access the graphics hardware without knowing the low-level details [1].

In GStreamer, we have two options for displaying images using the frame buffer device; or three, if we use OMAP3: fbvideosink, fbdevsink and gst-omapfb.

Nevertheless, since the appearance of the GPUs, the frame buffer device interface has not been sufficient to fulfill all their capabilities. A new kernel interface ought to emerge. And that was the Direct Rendering Manager (DRM).

What in the hell is DRM?

The DRM layer is intended to support the needs of complex graphics devices, usually containing programmable pipelines well suited to 3D graphics acceleration [2]. It deals with [3]:

1. A DMA queue for graphic buffers transfers [4].
2. It provides locks for graphics hardware, treating it as shared resource for simultaneous 3D applications [5].
3. And it provides secure hardware access, preventing clients from escalating privileges [6].

The DRM layer consists of two in-kernel drivers: a generic DRM driver, and another which has specific support for the video hardware [7]. This is possible because the DRM engine is extensible, enabling the device-specific driver to hook out those functionalities that are required by the hardware. For example, in the case of the Intel cards, the Linux kernel driver i915 supports this card and couples its capabilities to the DRM driver.

The device-specific driver, in particular, should cover two main kernel interfaces: the Kernel Mode Settings (KMS) and the Graphics Execution Manager (GEM). Both elements are also exposed to the user-space through the DRM.

With KMS, the user can ask the kernel to enable native resolution in the frame buffer, setting certain display resolution and colour depth mode. One of the benefits of doing it in kernel is that, since the kernel is in complete control of the hardware, it can switch back in the case of failure [8].

In order to allocate command buffers, cursor memory, scanout buffers, etc., the device-specific driver should support a memory manager, and GEM is the manager with more acceptance these days, because of its simplicity [9].

Beside to the graphics memory management, GEM ensures conflict-free sharing of data between applications by managing the memory synchronization. This is important because modern graphics hardware are essentially NUMA environments.

The following diagram shows the components view of the DRM layer:

What is the deal with KMS?

KMS is important because on it relies GEM and DRM to allocate frame buffers and to configure the display. And it is important to us because almost all of the ioctls called by the GStreamer element are part of the KMS subset.

Even more, there are some voices saying that KMS is the future replacement for the frame buffer device [10].

To carry out its duties, the KMS identifies five main concepts [11,12]:

Frame buffer:

The frame buffer is just a buffer, in the video memory, that has an image encoded in it as an array of pixels. As KMS configures the ring buffer in this video memory, it holds a the information of this configuration, such as width, height, color depth, bits per pixel, pixel format, and so on.

CRTC:

Stands for Cathode Ray Tube Controller. It reads the data out of the frame buffer and generates the video mode timing. The CRTC also determines what part of the frame buffer is read; e.g., when multi-head is enabled, each CRTC scans out of a different part of the video memory; in clone mode, each CRTC scans out of the same part of the memory.Hence, from the KMS perspective, the CRTC’s abstraction contains the display mode information, including, resolution, depth, polarity, porch, refresh rate, etc. Also, it has the information of the buffer region to display and when to change to the next frame buffer.

Overlay planes:

Overlays are treated a little like CRTCs, but without associated modes our encoder trees hanging off of them: they can be enabled with a specific frame buffer attached at a specific location, but they don’t have to worry about mode setting, though they do need to have an associated CRTC to actually pump their pixels out [13].

Encoder:

The encoder takes the digital bitstream from the CRTC and converts it to the appropriate format across the connector to the monitor.

Connector:

The connector provides the appropriate physical plug for the monitor to connect to, such as HDMI, DVI-D, VGA, S-Video, etc..

And what about this KMSSink?

KMSSink is a first approach towards a video sink as a DRI client. For now it only works in the panda-board with a recent kernel (I guess, 3.3 would make it).

For now it only uses the custom non-tiled buffers and use an overlay plane to display them. So, it is in the to-do, add support for more hardware.

 Bibliography

[1] http://free-electrons.com/kerneldoc/latest/fb/framebuffer.txt
[2] http://free-electrons.com/kerneldoc/latest/DocBook/drm/drmIntroduction.html
[3] https://www.kernel.org/doc/readme/drivers-gpu-drm-README.drm
[4] http://dri.sourceforge.net/doc/drm_low_level.html
[5] http://dri.sourceforge.net/doc/hardware_locking_low_level.html
[6] http://dri.sourceforge.net/doc/security_low_level.html
[7] https://en.wikipedia.org/wiki/Direct_Rendering_Manager
[8] http://www.bitwiz.org.uk/s/how-dri-and-drm-work.html
[9] https://lwn.net/Articles/283798/
[10] http://phoronix.com/forums/showthread.php?23756-KMS-as-anext-gen-Framebuffer
[11] http://elinux.org/images/7/71/Elce11_dae.pdf
[12] http://www.botchco.com/agd5f/?p=51
[13] https://lwn.net/Articles/440192/