White Paper on KGI, the Kernel Graphics Interface

Abstract

This draft is about the Kernel Graphics Interface developed by the KGI Project. It is intended to give a short overview of the concepts behind, being kind of a white paper. It'll take about 20min to read, if you like to know more about it or comment on it, feel free to email the maintainer of this document. You can download a snapshot of the current sample implementation from ftp://kgi.sourceforge.net/pub/kgi. A sample implementation for Linux is currently under development, the first part - a reimplementation of the Linux console system to meet the requirements of the KGI architecture - being finished already. This includes an implementation of most of the concepts explained here. Any file references given here refer to files included in the snapshot. An online version of the source tree is available at http://kgi.sourceforge.net/source/kgi-0.9

Introduction

The development of a Kernel Graphics Interface (KGI) is part of the GGI Projects effort to create a portable, fast and secure General Graphics Interface (GGI). While GGI itself is more a suite of application programming interfaces, most applications will not directly interface to KGI. Instead, KGI is intended to provide the neccessary kernel support (protection, virtualization and abstraction) in order to allow applications to benefit from graphics hardware with as little overhead as possible. Additionally, a certain amount of kernel level support for display hardware can ease the effort to meet the security and stability
requirements of modern multi-user multi-tasking operating systems.

Display Hardware Characteristics and KGI Desing implications

As mentioned, KGI deals primarily with display hardware. This section should give some fundamental properties of graphics hardware and their implication on KGI design. Some of these properties make it special compared to other computer subsystems, while others are just implied by the fact that it is hardware. Most of the KGI design goals and constraints can be summarized as:
Display Hardware is essential for human-machine interaction.
This very obvious property makes display hardware different from all other computer componentes and implies the following design requirements for KGI:

Display ...

Display hardware must become available very early during the boot process in order to display diagnostic messages. KGI must therefore provide means to output diagnostic messages from the kernel and do this as early as possible.
... hardware ...

The primary intention of advanced display hardware (other than simple frame buffers) is to offload computing or memory bandwith intensive operations to specialized (co-)processing hardware, thereby freeing the main CPU for other tasks. In some aspects, display hardware therefore acts as a co-processor that needs to be virtualized - just like the main (co-)processing unit(s) (e.g. FPU, ...). On the other hand, display hardware is also a system resource that needs to be protected from accidental or intentional misuse. Finally, KGI drivers - or firmware - are (as far as common features are concerned) considered part of the display hardware and must be designed to be independent of the OS and host architecture.
... is essential for ...

Display hardware must be reliable. This implies that the user or operator should be able to get access to a 'rescue shell' even if graphical user interfaces are not accessible. (E.g. the graphical user interface sever crashed.) Therefore KGI must not rely on applications to free resources properly.
... human-machine...

Display hardware is built for humans working with computers. This requires KGI neither to prescribe nor to prevent policies if/how to share display hardware, what to do in case of non-fatal errors etc.
... interaction.

Display hardware must be responsive. This implies that even for highly loaded machines delays between user input and visible feedback due to this change must be short. With 3D graphics becoming more and more standard, this requires advanced hardware programming techniques, such as graphics processor virtualization, priority scheduling, "processes models" for graphics processors etc. KGI must therefore provide means to handle several processes sharing resources where this makes sense.
These requirements are accompanied by the fact that display hardware is not standardized and probably the fastest evolving computer subsystem. KGI should therefore be open for future developments and not be specific to a particular graphics application programming interface. And last but not least, KGI is intended for UNIX-like operating systems, which feature multi-user multi-tasking environments. Thus KGI should allow for simple setup of independent multihead machines, featuring several workplaces per machine.

The KGI Architecture

The KGI architecture can be structured into the following 'subsystems':

a human-machine interaction model

an abstract display hardware model

physical I/O and environment operating system support

display hardware drivers and management

input hardware drivers and management

application interface (graphics process management)

Additionally the current KGI sample implementation for Linux has also a console subsystem and compatibility module, adopted to the changes neccessary to implement the KGI architecture.
The subsequent text will focus on the interaction and hardware models,OS environment, display driver and application interface. The input hardware driver, console and compatibility module will not be described in much detail.

The KGI Human-Machine Interaction Model

KGI is developed for multi-user multi-tasking operating systems that allow several users to run processes simultaneously, providing means to isolate (protect) processes from each other, but also to share limited resources. KGI is consequently designed to allow several users to work interactively with the same machine. It allows each user to use an arbitrary number of display and input hardware (often referred to as independent multi-head). Input and output devices are grouped to "workplaces" owned by a particular user (the one using it). In order to access resources of a particular workplace, processes must belong to the user or have explicitly been granted access to these resources.
A side note:
Actually all human-machine interaction hardware like sound, workplace local peripherals, etc. needs to be treated this way. However, KGI is currently limited to input and visual output. Actually, Kernel User Interface describes better what most of the KGI sample implementation for Linux does, but originally KGI was intended to handle display hardware only. Also, it should be pointed out clearly, that this Kernel User Interface is restricted to hardware drivers, protection and virtualization. Especially it must not be misunderstood as running a Graphical User Interface at kernel level, which is a common misunderstanding about KGI.
Additionally, KGI allows for virtual (textual and graphical) consoles that are mapped on physical display and input hardware (thereby becoming visible/active).
Analysing the flow of information gives a very natural model of human-machine interaction. Information going into the machine is "digitized" by input hardware. This digitizing is dealt with by the Kernel Input Interface file:kgi-0.9/kgi/include/kii/kii.h Information going out of the machine is converted by output (display) hardware. This is dealt with by the Kernel Graphics Interface file:kgi-0.9/kgi/include/kgi/kgi.h The actual information processing is done in processes that interface to hardware through abstract hardware representations, referred to as devices. Devices virtualize limited hardware resources.
All input device drivers (event sources - kii_input_t) register with a central mangagement code ( file:kgi-0.9/kgi/src/Linux/kgi.c) that enumerates input devices and groups these into so-called focuses kii_focus_t. All input devices associated with a focus send events to the same focused object (e.g. a virtual console) represented by a device (event-sink) kii_device_t. The focused object then processes the event, the result being commands sent to the display hardware represented by a device kgi_device_t. These get handed to the display hardware driver kgi_display_t, which in turn sends commands to the hardware, usually resulting in some feedback to the user.
For both KGI and KII care has been taken to separate hardware device driver specific issues (kii_input_t, kgi_display_t) from logical device representation (kii_device_t, kgi_device_t).

The Display Hardware Model

Despite all differences between the various graphics hardware, ranging from simple text frame buffers (Monochrome Display Adapter, LCD panels for embedded systems, etc.) up to virtual reality hardware as SGI's "Infinite Reality(TM)", certain properties of graphics hardware can be modeled and abstracted. KGI's display hardware model is intended to give a functional representation of the underlying hardware, hiding implementation details from applications.

Attributes

Display hardware controls some attributes of the visible display area either independently for small sub-areas (e.g. the color of a pixel) or the whole display area (e.g. brightness). KGI categorizes attributes into attributes that can be controlled independently per pixel, left and probably right frame (e.g. color, alpha value) and attributes that can be controlled independently per pixel, but common to all frames (e.g. z-value, stencil masks, ...) Sometimes the number for bits stored per attribute needs to be given, which is done using a bitmask indicating which attributes are present and a zero-terminated array of 8bit unsigned integers. (file:kgi-0.9/kgi/include/kgi/kgi.h gives details about the attributes defined.)
Images

Images are a rectangular set of pixels, the attributes of those being stored as an array of unsigned integers (the frame buffer). Not all of the bits stored per pixel may control visible attributes (e.g. z-values). Certain attributes (usually color or color index value) are read from the buffer in a certain order, control the properties of a given visible image unit (dot) and serialized into a dot stream. There might be some scaling and interpolation units or look up tables before the final dot stream assembly, which is why KGI distinguishes between image units (pixel) and dot stream units (dots).
Dot Stream Converters and Dot Ports

The dot streams generated by several image units (e.g. a GUI frame buffer and a video overlay buffer) are passed to some conversion device or processing unit before being converted into a sensable physical property. These devices are referred to as "dot stream converters" (e.g. a overlay processor, video DAC, etc.). KGI represents these devices by a dot port, which is a description of the transfer protocol or dot stream format. A dot stream converter is simply a number of input dot ports an one output dot port. Several dot stream converters may be connected, building a tree with dot stream converters as nodes and images as leaves. The root node is a dot port that represents the monitor.
Display Modes

Modern display hardware can be operated in a great variety of modes, e.g. color indexed mode, text mode, true color mode, true color with video overlay etc. Instead of enumerating and standardizing these modes (like1024x768 pixel with 8bit Red, 8bit Green, 8bit Blue and 8bit Alpha), KGI provides means to check if a certain requested mode is possible or not. Roughly speaking, a mode describes the hardware operation mode (number of images, attributes per image, dot stream converter properties, etc.).
Display Resources

As a result of a successful mode-check, the hardware driver(s) will return a set of resources available in this mode. While images, dot streams and dot stream converters describe the transformation process from the internal buffer representation to a visible image, display resources provide means to alter image buffers and internal buffers (like texture memory). Currently KGI specifies memory mapped I/O regions and accelerators. It is planned to extend these to support the VESA Media Interface (VMI), card-local I2O buses, proprietary buses (S3 Streams Interface), etc.
Some resources are static, e.g. video input and output buffers, and may influence the number of possible modes. Other resources are mode-dependent, e.g. may only be availiable in certain modes or operating system environments. For each resource access policy restrictions are given, describing the hardware driver's constraints about sharing this resource (exclusive maps, write/read re-ordering, allowed access widths, etc.). Some display resources (e.g. frame buffers) may be exported directly to applications, allowing for fast access and high performance. Other resources (e.g. accelerators) are not directly exported, but require some buffering and context management mechanism. This is described in more detail in the Application Interface section below.

Display Drivers and Display Management

KGI display drivers bridge between the display hardware and the application interface. For the most common hardware design, a modular driver architecture has been developed to allow for maximum re-usage of coding efforts. The most common design of display hardware can be sketched like this:

Though recent developments allow the integration of some parts into one piece of silicon, the functional units can still be identified. Each functional unit (except memory) is represented by a driver module, using a well defined internal interface. A fully operational display hardware driver is obtained by linking the proper (binary) subsystem drivers together. For example, a graphics card based on a 3Dlabs PermediaNT, with a TI TVP 3026 ramdac with integrated clock chip and a multisync monitor may reuse the ramdac, clock and monitor driver written for a S3 Vision968 based board with TI TVP 3026 and multisync monitor. A proof-of-concept implementation for this concept has been released by the GGI Project in early 1998. Valuable feedback and conclusions have been drawn from this implementation, resulting in an improved version of the internal module interface. A sample driver for the architecture described in this document is being developed, but only semi-operational yet. However, it is developed far enough to give a impression how to use the internal module interface. As KGI display drivers may run in kernel mode, the use of floating point operations is not allowed. All drivers must conform to certain requirements, such as they must not call other than the KGI environment functions. Also, KGI drivers must not rely on certain advanced features such as interrupt handling, DMA support etc. being available. However, if present they may benefit from this. This allows KGI driver to be used as userspace drivers or in-kernel drivers without modification.
The intra-display driver interface is defined in kgi-0.9/drv/display/kgi/module.h

The Physical I/O and Environment OS Support

A main goal for the display driver modules was to be independent of the underlying Operating System Environment. This requires the usual abstraction of system-dependent data types, physical I/O methods, IRQ/DMA handling and environment. The system dependent data types and the environment functions (memset, etc.) are considered straightforward and will not be discussed in detail here.

I/O regions

Physical I/O is abstracted by so-called regions, which are a physical contigous part of an address space. For each address space, the following abstractions/operations apply:

for each address space virtual, physical, I/O and bus addresses are defined. Virtual addresses are used by the driver access this region. Physical addresses are used to establish mappings of this region into other virtual address spaces. I/O addresses are the addresses that the device will respond to if applied at the address select lines. Bus addresses are used to access the device from other devices on the same bus. (E.g. a frame grabber writing directly to the frame buffer.)

before accessing a region, the I/O address of a device must be specified and the region must either be allocated (allowing auto-configuration) or claimed. If this is successful, the driver is granted exclusive use of this region. Upon driver termination or if not neccessary for operating the hardware, the allocated regions must be freed again.

The driver may only access the region using the virtual address obtained during registration and special I/O functions. Currently defined are simple input and output (read/write) operations, as well as copy and string input/output operations. Some operations may not be defined, depending on the capabilities of the I/O bus system.

Subregions may be exported to application or other drivers, e.g. a driver may claim a region of 64MB, containing 32MB frame buffer and 32MB control registers and export only 32MB to the application using
a memory mapped I/O resource. For details of the definitions, see file:kgi-0.9/kgi/include/kgi/io.h

Input drivers and management

KGI contains some input drivers modified to be aware of the multihead issues. This is for historical reasons as at the time of project start no unified input drivers were around.

The Application Interface

Applications do not interface to graphics hardware directly, unless approved by the display driver. Instead, an abstract representation of graphics hardware is defined and accessed by applications using a special device file and standard UNIX file operations (open, close, ioctl, mmap, read).

open close	are used to maintain virtualization of display hardware resources
ioctl	is used to control overall behaviour of display hardware and to send commands to the hardware (e.g. setting of look-up tables, control of video overlay scaling, etc.).
read	is used to report asynchronous events (accelerator programming error, accelerator finished, request for shared mapping from other processes etc.)
mmap	is used to gain direct access to accelerator command buffers, memory mapped I/O regions, such as video stream ports, accelerator register files, frame buffer regions etc. Applications that want to use direct access may be required to conform to certain restrictions, e.g. access only with certain widths, etc. This may be indicated by setting access permissions for the special file and using set-user-id features, or implementing some verification protocol with the session manager. KGI tries not to impose policy here, except that the driver's constraints have to be met.

Summary and State of Development

This article was intended to give an overview about the KGI architecture to handle display hardware. Currently the KGI Project is working on a improved proof-of-concept implementation (kgi-0.9) using Linux as a target OS, that will implement the enhanced concepts presented in this articles, such as graphics processor virtualization as part of a first version of a direct-rendering architecture. This will include an experimental X server (XKGI) adopted to this architecture.
The implementation of the KGI human-machine interface model is already finished and currently it beta-testing state. It already allows to setup true multi-head machines with XFree-4.0. However, most advanced features require native KGI drivers, which are still under development.