Archive:Hardware Accelerated Video Decoding Development

This article will cover potential methods of and the development around Hardware Accelerated Video Decoding.

Note! XBMC does not yet support any methods of hardware accelereated video decoding. Developers wanted!

Hardware Accelerated Video Decoding

"Hardware Accelerated Video Decoding" is the when a video-playback software-application offload portions of the video decoding process to the GPU (Graphic) hardware, it does this by executing specific code algorithms on the GPU. In theory this process should also reduce bus bandwidth requirements.

FFmpeg (and MPlayer) should probably be the reference and test platform for all hardware accelerated video decoding development. The reason for this is that XBMC used FFmpeg as the base for its DVDPlayer-playback-core core, and FFmpeg is used as the codec-suit inside XBMC MPlayer-playback-core. Also, since both FFmpeg and MPlayer are cross-platform code we can get help from non-XBMC developers.

Hardware Accelerated Video Decoding under Linux

Developers wanted!, for more information on XBMC and Linux please see the Linux port project article in this WIKI.

Video decoding processes which could be accelerated

XvMC for Linux could possible be extended in the future to support the same processes:

• Motion compensation (mo comp)
• Inverse Discrete Cosine Transform (iDCT)
• Inverse Telecine 3:2 and 2:2 pull-down correction
• Bitstream processing (CAVLC/CABAC)
• in-loop deblocking
• inverse quantization (IQ)
• Variable-Length Decoding (VLD), more commonly known as slice level acceleration
• Spatial-Temporal De-Interlacing, (plus automatic interlace/progressive source detection)

XvMC

X-Video Motion Compensation (XvMC), is an extension of the X video extension (Xv) for the X Window System. The XvMC API allows a simple way to add hardware accelerated video decoding to video-playback software-application

NVIDIA

Even though NVIDIA closed source binary device driver for Linux currently only support XvMC hardware acceleration of motion compensation (mo comp), and inverse discrete cosine transform (iDCT) for MPEG-2, the closed source binary device driver for Microsoft Windows from NVIDIA features many more additional video decoding processes which can be passed on to modern GPU (such as NVIDIA's GeForce 6-series, from 6150 and on) which on Microsoft Windows can accelerate many more video decoding processes via Microsoft's DxVA (DirectX Video Acceleration) API, (the Microsoft Windows equivalent to XvMC).

NVIDIA PureVideo Technology

NVIDIA's GeForce 6-series (from GeForce 6150 and on) features a video acceleration engine called "PureVideo", NVIDIA's GeForce 8-series (with the exception of GeForce 8800) features an updated version of the PureVideo technology which NVIDIA calls "PureVideo HD". NVIDIA PureVideo Technology is a true discrete programmable processing core inside the NVIDIA GPU. The NVIDIA PureVideo technology is a combination of a hardware video processor and video decode software, meaning it only offloads parts of the video decoding to the GPU (but since those are the 'heavy' and processor intensive parts it results in a huge diffrence on CPU usage when using PureVideo vs. not using PureVideo).

NVIDIA closed source binary device driver for Microsoft Windows has since ForceWare version 85 featured PureVideo Technology support for accelerated hardware video decoding of MPEG-2, MPEG-4 AVC (H.264), VC-1, and WMV9, (plus it also Spatial-Temporal De-Interlacing), via DXVA (which is Microsoft's equivalent of the XvMC API).

Intel (GMA)

Intel's open source device drivers for Linux supports motion compensation (mo comp), and inverse discrete cosine transform (iDCT), and de-interlacing for MPEG-2.

Intel Clear Video Technology

Intel® Clear Video Technology is a combination of video processing hardware and software technologies for a wide range of digital displays. This technology is available on all Intel® G965 Express Chipset-based hardware platforms. According to Intel, Clear Video Technology enables; Enhanced high-definition video playback, Sharper images, Precise color control, and Advanced display capability.

Intel Clear Video Technology Features and benefits;

• MPEG-2 decode iDCT + motion compensation. Up to 2 stream support (1 HD and 1 SD)
• De-interlacing Advanced pixel adaptive (SD/HD-1080i)
• Color control ProcAmp: brightness, hue, saturation, contrast.
• Video scaling 4x4 scaling
• Digital Display Support (through SDVO) Digital Video Interface (DVI), High-Definition Multimedia Interface (HDMI)
• Display support RGB (QXGA), HDMI, UDI, DVI, HDTV (1080i/p, 720p), Composite, Component, S-Video (via Intel Serial Digital Video Out), TV-out, CRT
• Aspect ratio 16:9, 4:3, letterbox
• Maximum resolution support 2048 x 1536 at 75 Hz, RGB (QXGA)

Alternative methods of hardware accelerated video decoding

Video decoding processes could possible also be accelerated under Linux/UNIX (and Microsoft Windows) by using other methods than the previously mentioned PureVideo Technology from NVIDIA (alternative methods could also be used in combination with PureVideo to run video decoding processes that PureVideo do not support). Programming shaders (Pixel Shader or Vertex Shader), with one shader for each video decoding process that one would wish to accelerate is one such method. GPGPU (General-Purpose Computing on Graphics Processing Units) is another possible method. All these alternative methods requires Shader Model 3.0 support by the GPU, (which is one of the reasons why we made Shader Model 3.0 a minimum end-user requirement for the XBMC Linux port).

GLSL (OpenGL Shading Language)

OpenGL Shading Language (GLSL, a.k.a. GLslang) is a high-level shader programming-language (based on the C programming-language) which offers such possibilities, (GLSL was originally introduced as an extension to OpenGL 1.5 but the OpenGL ARB only formally included GLSL into the OpenGL 2.0 core). http://en.wikipedia.org/wiki/OpenGL_Shading_Language

Cg (C for Graphics)

Cg (or "C for Graphics") is a another high-level shading programming-language, created by NVIDIA for programming vertex and pixel shaders it is compatible with other GPU hardware manufactures as well. Like GLSL, Cg is also based on the C programming language, and although they share the same syntax, some features of C were modified and new data types were added to make Cg more suitable for programming graphics processing units. The Cg programming-language seems to have survived the introduction of the newer shading languages very well, mainly (???) of its established momentum in the digital content creation area, although the language is seldom used in final products.

GPGPU (General-Purpose Computing on Graphics Processing Units)

General-Purpose Computing on Graphics Processing Units (GPGPU, also referred to as GPGP and to a lesser extent GP²) is a recent trend in computer science that uses the Graphics Processing Unit to perform the computations rather than the CPU. The addition of programmable stages and higher precision arithmetic to the GPU rendering pipeline have allowed software developers to use the GPU for non graphics related applications. Because of the extremely parallel nature of the graphics pipeline the GPU is especially useful for programs that can be cast as stream processing and real-time computing problems. Simplest way to enable GPGPU support is by using a library such as Lib Sh (GPGPU library for C++), or BrookGPU (the latter, BrookGPU is probably the better of the two for XBMC video decoding purposes).

CUDA (Compute Unified Device Architecture)

CUDA (Compute Unified Device Architecture) is an GPGPU technology and API that NVIDIA introduced in the GeForce 8-series (G8X based) GPUs. CUDA allows a programmer to use the C programming-language to code algorithms for execution on the GPU. A video decoding process could be a such code algorithms that is executed on the GPU via CUDA.

Possible development tools and resources

Tools and resources that could possible help in the development. Note that GLSL shaders will need to be created and tested in a development tool prior to the injection in the video-playback software-application that will use them, (to do so, GLSL developer tools exists, see "Development Tools" below).

Development Tools

• Lumina - GLSL development tool (IDE). It is platform independent and the interface uses the Qt (toolkit).
• NVIDIA ShaderPerf1.8 and ShaderPerf 2.0 Alpha - handy utility that reports detailed shader performance metrics for a wide range of inputs. It is available both as a command line utility and with a user interface in FX Composer. Please note that ShaderPerf 2.0 Alpha only supports DirectX shaders written in HLSL or assembly, (so either use version 1.8 or use HLSL2GLSL which can be used to convert a HLSL shader into a GLSL shader).
• FX Composer - provides an IDE interface to create, compile and debug GLSL (as well as DirectX) shaders.
• RenderMonkey - provides an IDE interface to create, compile and debug GLSL (as well as DirectX) shaders.
• Blender - This popular opensource 3D modeling and animation package can now use GLSL materials, thus allowing any shader developer to use it as a developement tool.
• OpenSceneGraph - open source multiplatform graphics and shader IDE (also see GLSL Shading with OSG — 1.20MB zipped PDF)
• HLSL2GLSL - library and tool that converts HLSL (High Level Shader Language) shaders to GLSL (OpenGL Shading Language)
• DirectX OpenGL Wrapper - emulates API calls thru OpenGL commands and other platform specific commands in order to run DirectX 8 application running on other platform than Windows.

Open Source Device Drivers

• intellinuxgraphics.org open source Linux Graphics Device Drivers from Intel (with XvMC for MPEG-2 acceleration support)
• openChrome Project - open source device drivers for VIA (has updated XvMC with MPEG-2/MPEG-4 acceleration support)
• Nouveau - open source device driver for NVIDIA-based graphic controllers (does not yet feature any XvMC support)

Source Code and Libraries

• BrookGPU - GPGPU library in ANSI C for general purpose computations on GPU (OpenGL and DirectX compatible)
• Lib Sh - GPGPU library in metaprogramming language and C++ for general purpose computations on GPU
• Discrete Wavelet Transform (DWT) of JPEG 2000 (JasPer) on GPU written in Cg shader
• OpenCV (Open Computer Vision Library) - a collection of algorithms and sample code for various computer vision problems. The library is compatible with Intel Image Processing Library (IPL) and utilizes Intel Integrated Performance Primitives for better performance. Features a bi-linear interpolation and color space conversion functions in IPL, (I also read that motion estimation with block matching and Hough transform is on the roadmap so you might want to check out their CVS).
• Anti-Grain Geometry - A High Quality Rendering Engine (High Fidelity 2D Graphics Renderer) for C++ (GPL licensced)
• SDL_buffer - a SDL extension library that is useful when you have to resize an image multiple times.
• SDL_Resize - basic image resizing library, high quality output suited for prerendering images.
• SDL_Config - Library designed for reading and writing configuration (.ini) files in an easy, cross-platform way.
• SDL_bgrab - SDL convertion of libbgrab (a framegrabber lib from the same author).
• NVIDIA Shader Library (color space conversions, blurring, interpolation, anti-aliasing, etc.)

Other developer resources

Online Documentation and Tutorials

• GLSL (OpenGL Shader Language) Tutorial @ Lighthouse 3D
• OpenGL specification and OpenGL Shading Language reference documents (3DLabs)

Books (hard-copy)

• GPU Gems 2 (published by NVIDIA) for and by developers
  • there is also GPU Gems 1 but it does not cover GPCPU

External links

XvMC

• Wikipedia.org article on XvMC (X-Video_Motion_Compensation)
• MythTV WIKI article on XvMC under Linux
• bit-tech.net article on NVIDIA PureVideo Technology

GLSL

• Wikipedia.org article on GLSL (OpenGL Shading Language)
• shadertech.com - Shader development news, forums, tools, code, and links.