Video support on android – Python bindings for ffmpeg have been done and are available on github. Theses are focused to be usable on android within Kivy project, but any help to clean and make the code more crossplatform is welcome !

If you still hesitating about using Kivy… just look at the frontpage for the current awesome projects made by the community. And they are lot of others projects going to be released during the next month ! We are also glad to see some big companies to use Kivy as the base for building their products.
If you have also cool project going on, don’t hesitate to share 🙂

To finish, here is a video from Cyril that playing on Icarus Touch, an instrument that use Kivy for his UI. Awesome work !

We all care about cash, time, life, love, and if you’re doing computer graphics, you might care about the memory consumption of your graphics card. Why ? For the same reason when you running out of cash 🙂

I’ll explain why does it matter to compress texture, and compare available possibilities. My personnal goal is to be able to load a lot of FULL HD pictures on tablet, for a museum project. The analysis is focused on DXT1 compression and size. I’m looking forward to ETC1 and will update that blog post with the result in the future.

What are we dealing with

If you are doing an application that display lot of hd pictures, that’s matter. We’ll start from this simple math statement: a full HD picture is 1980×1020 with 4 channels (RGBA). Whatever if your pictures is in PNG, or JPEG, your graphics card is not able to understand it, and will store it in its memory decompressed. So this image will eat:

In theory. Because it might be more if your graphics card doesn’t support NPOT texture. If not, usually, the texture will be resized to the closest POT texture available, mean for us: 2048 x 2048. Then the size for POT will be:

We are seeing that ETC1 is standard compression for OpenGL ES 2, unfortunately, it will not work on desktop.
PVRTC is specific to PowerVR device: it’s a standard on Ipad/Iphone.

Using DXT1

If you use DXT1, you need a POT image. DXT1 doesn’t work on NPOT.

To convert any image to DXT1 without any tool, you must know that your graphics card is capable of doing it, using specials opengl functions. But i wanted to precalculate them.Nvidia texture tools contains tools for converting them, but you need an Nvidia card. For all others, you might want to look at Libsquish. It’s designed to compress in software a raw image to DXTn.
The result will be not a DXT1 “file”, because DXT1 is the compression format. The result will be stored in a DDS file, that we’ll see later about it.

For all the guys that was interested about Kivy lightning talk at EuroPython 2011, due to some incomprehension about what is Kivy, here is a lightning explanation to make it clear.

Kivy is a Python framework designed for creating of Natural Users Interfaces. The framework containing abstraction for loading image, video, audio. It have a complete new approach about input events, and widgets. For example, you can use lot of widgets at the _same_ time, something not really possible in classical framework (qt, gtk…): try to touch on a button while selecting something in a list. This is not only about multitouch for one user, but also for multi users. Kivy graphics engine is in OpenGL ES2, and all the widgets are using it.

If you write an application in top of Kivy, you can deploy it on Linux, MacOSX, Windows and Android. Without changing anything in your code. Because it’s in Python.

The presentation tool i’ve used is PreseMT. It have been made by Christopher and Me. And it’s an application built using Kivy. A version of this tool is already published on Android Market.
PreseMT have been written in one week, and use lot of Kivy features. But it’s still not finished, and we are missing lot of features, like the ability to export the presentation in a “good” format. We have plan to make an export in HTML5, that will support animation too.

Apitrace is a tool for recording all the gl commands in a trace file. The trace file can be replay in later time, and they got a nice gui for checking all the gl call every frame, with introspection. They have a glretrace software that replay a trace file. We can use it to get the output of everyframe and push it in a gstreamer pipeline to make a video.

Why not using gtkRecordMyDesktop or other screen capture ? Sometime, the overhead of capturing and encoding video on live take too much CPU. And the application start to slow down. I didn’t see any slowdown using apitrace, and the trace file is very small compared to video output or raw video output.

So first, compile apitrace with stdout support:

1

2

3

4

5

6

7

$git clonegit://github.com/tito/apitrace.git

$cdapitrace

$git checkout snapshot-stdout

$mkdirbuild

$cdbuild

$cmake..

$make

Take any opengl application, and make a trace file. The trace file will have the name of the binary. In my case, python is an alias to python2.7: the trace file will be python2.7.trace.

From 3 weeks now, i’m working on packaging kivy application, to create a installer/bundle/deb of a Kivy application. The reason is simple: as soon as you are doing an application, the user should not care about installing Kivy itself. In the same time, i’ve work on other projects that require to have their own configuration. From a long time, we always wanted to have some in-app settings for configure Kivy. Even Android have a “settings” button, we wanted to use it. 🙂

This is now possible.

Yes, it look like the honeycomb settings panel. Kind of. Well.

The configuration is automatically handled by the App class, and you can put your own token on it. The settings UI (that you’re seeing on the screen) is created from a JSON definitions. You can press F1 or the settings key on android to bring the settings panel, hook the on_config_change to know when a configuration token is changed from the settings ui, etc.

If you already done OpenGL development, you should be aware of POT (Power of two) texture. Because of very old conventions, the texture size must be a power of two size. Not necessarily the same for width and height though : 256×256 is valid as 128×512.

The usual thing to do when you want to load an NPOT texture (like 23×61) is to:

take his closed POT size: 32×64

depending of the book you’re reading: blit/strech the 23×61 to the 32×64 texture

OR blit without stretch, and adjust texture coordinate (this is what kivy does right now.)

The downside part of this approach is that you’re lost a part of memory. Bad.

While ago, i remember to found the Rectangle texture support from NVidia. Aaah, finally, is it what we was waiting from a long time ? Erm, no. Their implementation have lot of downsides:

The usage of a specific texture target: GL_TEXTURE_RECTANGLE_NV

No mipmap support

The texture coordinates are not normalized from 0-1… but from 0-width/height of the image

Some wrap mode are not supported (GL_REPEAT for eg.)

But today… i discover that most graphics card are supporting rectangle texture. If the extension GL_ARB_texture_non_power_of_two (OES_texture_npot for OpenGL ES platform), you can finally ensure that loading NPOT texture will… just work as expected :

You can still use GL_TEXTURE_2D

Mipmapping are supported

Texture coordinates are from 0-1

All wrap mode are supported

A little note here, in OpenGL ES 2, they have a native support for NPOT texture, but with somes limitations related to mipmapping.

If you want to just load NPOT texture safely without using rectangle texture, just check the availability of theses extensions :

I’m currently on a project that involve disabled peoples, audio and kinect. Theses boys and girls are doing lot of loud sounds, so the idea is to use their sound as a trigger. We can use gstreamer to make that work quite easily, cause it have everything we need: a audio source, and level calculator.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

importpygst

pygst.require('0.10')

importgst,gobject

gobject.threads_init()

pipeline=gst.parse_launch(

'pulsesrc ! audioconvert ! '

'audio/x-raw-int,channels=2,rate=44100,endianness=1234,'

'width=32,depth=32,signed=(bool)True !'

'level name=level interval=10000000 !'

'fakesink')

level=pipeline.get_by_name('level')

bus=pipeline.get_bus()

bus.add_signal_watch()

defshow_peak(bus,message):

# filter only on level messages

ifmessage.src isnotlevel:

return

ifnotmessage.structure.has_key('peak'):

return

# read peak

print'peak',message.structure['peak'][0]

# connect the callback

bus.connect('message',show_peak)

# start the pipeline

pipeline.set_state(gst.STATE_PLAYING)

ctx=gobject.gobject.main_context_default()

whilectx:

ctx.iteration()

The output could be something like this :

1

2

3

4

5

6

7

8

9

10

11

12

peak-35.2370719856

peak-35.0252114393

peak-10.8591229158

peak-4.6007387433

peak-4.85102463679

peak-6.45292575653

peak-6.83102903668

peak-7.39486319074

peak-13.9852340247

peak-17.423901433

peak-35.0852178272

peak-35.8725208237

Next, we can use that information to record their sound, and use it on some scenario. So, instead of use the fakesink, we can use appsink. This module allow you to read the buffer pushed by the previous module. So we can put theses buffers into a list, and use them when needed 🙂

The state machine will handle the 3 phases :

Wait for a peak > -30db

Recording the sound, stop when the peak is < -32db

Replay the last sound

Note: The -30 / -32 are taken from my tests. If you have more noise, you need to adjust theses triggers.

Maybe during the next release, or a little bit after, i’ll release a software to create an Android package of a Kivy application. The code is already on launchpad, but it’s still a work in progress. As soon as i have finished, i’ll publish it on kivy-dev mailing list. If you didn’t subscribe yet, do it now ! 🙂