Mobile Application Development: VES Library Insights

According to a report published by the International Telecommunication Union, there were six billion mobile subscriptions at the end of 2011 (see Figure 1). Also, in 2011, sales of smartphone surged to 472 million units. A smartphone is a mobile phone built on a mobile computing platform, with more advanced computing ability and connectivity than a feature phone. The new generation of smartphones offer various multimedia capabilities including interactive games and applications. This is become possible because of the recent improvements in mobile hardware and software. These significant improvements also enabled scientific community to use mobile devices for visualization purposes. For example, Zhou et al [3] has investigated how to achieve volume visualization on mobile devices; Tesch et al [4] developed a system that uses mobile devices for interacting and exchanging data between heterogeneous scientific visualization systems; and Roudaut et al [5] presented their work on interaction techniques for mobile devices. Though technically impressive, none of these offer a general, extensible, and open-source framework for enabling developers to build high performance visualization applications on mobile devices.

The VES framework [11] has been developed to address these limitations. It puts powerful tools at the fingertips of developers to enable the building of high performance visualization applications on mobile devices. VES is an open-source framework for accelerating mobile visualizations on current generation mobile hardware. VES consists of two core libraries: VES and Kiwi. The VES library provides rendering capabilities and infrastructure for scene management by utilizing OpenGL ES 2.0 [8] whereas the Kiwi library provides an application framework and is built on top of VES and VTK [6]. In this article, we will discuss the architecture and various components of the VES library along with code snippets to explain the API offered by the VES library.

Figure 3: Architecture of VES

Architecture & ImplementationAs previously mentioned, the VES library provides scene management and rendering functionalities to the applications developed using the VES framework. It uses OpenGL ES 2.0 as the underlying rendering API, which is a subset of OpenGL for desktop and replaces the fixed function transformations and fragment pipeline of OpenGL 1.x. OpenGL does not provide a high-level, object-oriented API for the application development, which the VES library provides to reduce time and effort to build visualization applications.

Figure 4: Components of VES Library

As shown in Figure 4, the VES library is composed of multiple components, each of which offers a unique feature within the library. The next few sections will cover some of these components and the underlying technology in detail.

Scene GraphThe VES library uses scene graph data structures to manage scenes efficiently. A scene graph is a data structure that provides spatial and logical relationships between various entities of a scene. A scene graph can be implemented in many ways and some of the open source implementations of a scene graph are inspired by the design of OpenGL Performer, one of the well-known scene graph libraries from SGI [7]. The VES library is built using the same core principles and additionally provides a consistent, easy-to-use API to allow applications to take advantage of programmable pipeline functionality of OpenGL ES 2.0. Figure 5 shows the inheritance hierarchy in the VES scene graph.

Figure 5: Inheritance Hierarchy in VES Scene Graph

Rendering PipelineRendering in the VES library is a three-pass algorithm as shown in Figure 6. VES renders the scene by traversing it in a depth-first manner. At the API level, the VES library separates the geometry data from the appearance of the geometry. The appearance of the geometry is captured by the notion of a material in the VES library.

Figure 6: Rendering Model in VES

This separation facilitates reuse of geometry and/or material if required by an application. Since state change in OpenGL is an expensive operation, VES library constructs and renders a scene tree in which geometry data is grouped by the material. Also, the library provides a notion of bins, where a material and associated geometries with a higher bin number are rendered after the material and associated geometries with a lower bin number.

In OpenGL ES 2.0, it is necessary to provide a vertex and a fragment shader in order to render geometry primitives. Vertex shaders can be used for traditional, vertex-based operations such as transforming the position with a matrix, computing the lighting equation to generate a per-vertex color, and generating or transforming the texture coordinates. The fragment shader is a general-purpose method for interacting with fragments. A simple in-source shader to render geometry in the VES library is shown in Listing 1.

The way vertex attributes and uniforms are passed to the pipeline are somewhat different; but due to the fact that they are both inputs to the shader program, it is important to provide a consistent API to bind uniforms and vertex attributes to a shader. In Listing 2, we have shown how to pass uniforms and vertex attributes to the rendering pipeline.

In the previous example, we used predefined vertex attributes keys such as Position, Normal, and Color. These keys are provided for convenience (internally defined as enums) and applications are allowed to add a new key if desired. This design makes it convenient and flexible to bind multiple vertex attributes to the shader, where each attribute is identified by its key. The mechanism for linking vertex attributes to the geometry data using a key is explained in the following section on geometry data.

Geometry DataThe VES library provides a very flexible data structure for defining geometry for the purpose of rendering. Some of the highlights of the VES library geometry data are:

Support for interleaved or separated data arrays

Any number of coordinate systems for the point data

Support for different basic types for the point data

Separation of point data from the cell data

Extensible data structure

VES geometry data structure is inspired by the collada file format specifications [9] where geometry is composed of one-or-more sources. Source in the VES library defines the per vertex data to be used for rendering or computation purposes. For example, in Listing 3, we have shown a code snippet to define a source data, which stores the vertex positions of a geometry.

It is to be noted that in the code above,vesGenericSourceData uses vesVertexData3f as the template argument. The VES library provides several such source data types for convenience. Developers can create their own source data type that can use either predefined vesVertexData* types or some other custom vertex data structure. Also in the previous section we mentioned that the vertex attribute binding to the shader uses attribute keys. In order to notify VES library to link a vertex attribute to the vertex source data it is required that they both share the same key value; for example, Position as shown in the example code.

VES library source data structure enables applications to define and use interleaved or separate arrays for each vertex attributes. Interleaved vertex arrays are useful to achieve performance in most situations. Information on how to construct cells from the source data is defined by the vesPrimitive. Just like source data, VES library geometry data can contain one or more instances of vesPrimitive. Listing 4 shows a code snippet to define triangle primitives.

VES library enables developers to use unsigned short or unsigned int type for the indices based on the requirement and whether or not the device and the driver supports unsigned int as the index type.

Appearance and State ManagementIn the VES library, appearance and rendering state of the geometry is defined by the material used by the node of the scene graph. A material is composed of one or more material attributes. For example, shader program and textures are attributes of a material. Similar to a shader program (see Listing 2), a texture can be added to a material. VES provides support for 1D and 2D textures. Below is a list of data formats and types supported by VES library texture: Internal and pixel format:Alpha, Luminance, LuminanceAlpha, RGB, and RGBA.Pixel data type:PixelDataTypeNone, UnsignedByte, UnsignedShort565,UnsignedShort4444, UnsignedShort5551.

Control over depth buffer and blending is provided by vesDepth and vesBlend material attributes. The VES library API makes it easy to add a new material attribute in case the functionally is not provided by the existing attributes.OpenGL ES ExtensionsThe VES library provides basic support to query OpenGL ES extensions available on a hardware. For example, using this feature, an application can query for the GL_OES_element_index_uint extension and if available, can use an unsigned int type for the index to support large geometry data. However, current implementations do not support applications acquiring function pointer for the extensions.FrameBuffer Object (FBO)Framebuffer Objects are a mechanism for rendering to images other than the OpenGL default Framebuffer. OpenGL introduced FBO to make rendering to texture objects much more efficient and easier to perform when compared with pbuffer alternatives. The VES library enables applications to use the render to texture feature by creating an instance of vesRenderToTexture, setting appropriate parameters for the render target and setting the render target on the camera. Currently, VES supports attachments of type GL_COLOR_ATTACHMENT0 and GL_DEPTH_ATTACHMENT.Interfacing with VTKOne of the strong motivations behind VES development is to enable developers to take advantage of state-of-the-art VTK algorithms on mobile devices. In order to provide an interface similar to VTK, the VES library offers vesActor and vesMapper. It is important to mention that the VES library does not depend on VTK as it is expected that an application that requires VTK support will be developed using the application level API provided by Kiwi.

ResultsIn Figure 7 we show the performance of VES and Kiwi against different size models. We measured the rendering performance of the VES library using KiwiViewer [REF 10] on a third generation Apple iPad. We measured the interactive frames per second (FPS) by averaging the number of times rendering is triggered by the iPad in a 20 second interval. In order to get a consistent performance measure, we didn’t include FPS value reported by the application at the start and exit of the application.

Figure 8: Performance Graph of VES Rendering

Conclusions & Future WorkThe VES library, along with Kiwi, have been very successful in delivering a high performance application on iOS and Android platforms. Since the time we open sourced the VES and Kiwi libraries, we have received many contributions and much feedback from the community. We will continue to improve the current features of the VES library, and look forward to adding support for new features such as volume rendering and vector text rendering.

Figure 9: Rendering on Qt based MeeGO Platform

Special thanks to Riccardo Vianello for porting VES to the Qt-based MeeGO platform and providing screen captures for this article.

Aashish Chaudhary is an R&D Engineer on the Scientific Computing team at Kitware. Prior to joining Kitware, he developed a graphics engine and open-source tools for information and geo-visualization. Some of his interests are software engineering, rendering, and visualization