dbus-cpp is the libary that we are currently using to perform dbus calls from cpp within ubuntu touch. For those that are used to dbus-cpp know that the dbus interface can be defined via structures that are later checked at compile time. The following is an example of those definitions:

This is quite a lot of work and quite repetitive. We did notice that and used in some of our projects a set of macros to make us type less. We now will provide the following macros for other developers to use as part of dbus-cpp

In the ubuntu download manager we are using the new connection style syntax so that if there are errors in the signal connections we will be notified at compile time. However, in recent versions of udm we have noticed that the udm tests that ensure that the qt signals are emitted correctly have started failing randomly in the build servers.

As it can be seen in the following build logs the compilation does finish with no errors but the tests raise errors at runtime (an assert was added for each of the connect calls in the project):

I am not the only one that have encoutered this bug within canonical (check out this bug). Apprently -Bsymbolic breaks PMF (Pointer to Member Function) comparison under ARM as it was reported in linaro. As it is explained in the Linaro mailing list a workaround to this (since the correct way would be to fix the linker) is to build with PIE support. The Qt guys have decided to drop -Bsymbolic* on anything but x86 and x86-64. I hope all this info help others that might find the same problem.

This is a small tip for those thos want to use QTest and Google Mock. To ensure that the expectations are check at the end of the execution of the test function and that errors are correctly reported you have to check the expectaions manually and pass the results to a QVERIFY macro. The following examples should be good to get you started:

There we are passing the result of Mock::VerifyAndClearExpectations, where VerifyAndClearExpectations verifies and removes the expectations on a mocked object and returls a bool if it was successful. This way if the expectations are not met the QTest will fail.

For those interested in the error output it would be somthing of the following stype:

home/mandel/Canonical/udm/upload-interface/ubuntu-download-manager-tests/test_base_download.cpp:135: Failure
Mock function called more times than expected - returning directly.
Function call: cancelDownload()
Expected: to be never called
Actual: called once - over-saturated and active
FAIL! : TestBaseDownload::testCancelNotQueued() 'Mock::VerifyAndClearExpectations(down.data())' returned FALSE. ()
Loc: [/home/mandel/Canonical/udm/upload-interface/ubuntu-download-manager-tests/test_base_download.cpp(139)]

Well, you are not the only one and I might be able to point you to the correct direction, your code is probably returnning a QDBusError that you created using the QDBusError::Other enum. The problem here is that the enum value your are using only indicates that the error name is not known and therefore cannot be match to a value in the QDBusError enum. When you use that enumerator the message created does have an incorrect name as follows:

Ok, imaging that you are working with Qt 5 and using the new way to connect signals, lets for example say we are working with QNetworkReply and we want to have a slot for the QNetworkReply::error signals that takes a QNetworkReply::NetworkError, the way to do it is the following:

The static_cast is helping the compiler know what method (the signals or the actual method) you are talking about. I know, it is not nice at all but works at compile time better than getting a qWarning at runtime.

The problem is that without the help the compiler does not know what method error you are talking about :-/

I the last few months I have been working on the Ubuntu Download Manager, one of The Big Rocks of August. The u-d-m provide a dbus service that allows applications to request downloads to be performed, to such download requests it adds some nice features that a user on a mobile phone, and probably a desktop, is interested to have. Some of those features are:

Apparmor isolation per download. That means that only you application can interact with its downloads.

Pause/Resume downloads

Autodetect network connection.

WIFI only downloads.

Hash check support.

Allow downloads to be performed while an application has been paused or killed.

Group downloads, where a bunch of files are provided and the different downloads are performed as a single atomic operation.

A download might seem a simple action to perform, right? Well, as soon as you start supporting all the above a single download operation becomes a fairly complicated matter. The following is a state machine that identifies the states of a download that would support such features:

As you can see, it is a complicated matter and all these has to be tested and check by the QA team. By providing u-d-m (and later a client library to use approach in C and in the Ubuntu SDK, I’m terribly sorry but I did not have the time to finish it on time for the release) we are helping developers to perform simple downloads with robust code and do not worry about all the corner cases. Performing a download is as simple as requesting it and listen to the different signals. This kind of service is also provided by FirefoxOs, WEbOs and Tizan (but no in IOS or SailFish) but I believe we are doing a better job at exposing a richer API. Of course all this is open source and at least our friend at Jolla (and I really mean friends, I think they are doing an awesome work!!! and competition + collaboration is great).

In the following days I’ll be posting on hot to use the API via C, C++ and DBus.

I have not updating this page lately for a very simple reason: TO MUCH TO CODE.

After a crazy amount of work to push the Ubuntu Download Manager to be the centralized daemon used to perform downloads in Ubuntu Touch I have some more time to write. In the following weeks I’m going to be focusing on explaining some issues and patterns I have found using Qt while developing u-d-m (short of ubuntu download manager from now on).

PS: Is quite embarrasing that my last post was the ‘She got me dancing’ videoclip… you should take a look nevertheless

I have recently been doing some work with Qt and DBus and I got stuck a little on how to correctly send a {sv} over DBus. Either my google-fu is terrible or there are not many examples on how to do this, therefore here is a small static method that will return a {sv} that can be send via DBus without getting a wrong parameters error:

Following the small work I’m doing to get better at golang here is another piece of work I have done to add a golang binding to one of the C libraries I’m depending on. The following allows to use the libaccounts-glib library from go:

I have been playing a little with golang lately and I have decided to do a small desktop application to try an practice more. Because I wanted to integrate with the gnome-online-accounts API I had to do a small go interface for the API. I leave it here in the wild to see if people can spot errors with it or find it useful:

So yet again I have been confronted with broken tests in Ubuntu One. As I have already mentioned before I have spent a significant amount of time ensuring that the tests of Ubuntu One (which use twisted a lot) are deterministic and we do not leave a dirty reactor in the way. In order to do that a few week a go I wrote the following code that will help the rest of the team write such tests:

importosimportshutilimporttempfilefrom twisted.internetimport defer, endpoints, protocol
from twisted.spreadimport pb
from ubuntuone.devtools.testcasesimport BaseTestCase
# no init method + twisted common warnings# pylint: disable=W0232, C0103, E1101def server_protocol_factory(cls):
"""Factory to create tidy protocols."""if cls isNone:
cls = protocol.Protocolclass ServerTidyProtocol(cls):
"""A tidy protocol."""def connectionLost(self, *args):
"""Lost the connection."""
cls.connectionLost(self, *args)# lets tell everyone# pylint: disable=W0212if(self.factory._disconnecting
andself.factory.testserver_on_connection_lostisnotNoneandnotself.factory.testserver_on_connection_lost.called):
self.factory.testserver_on_connection_lost.callback(self)# pylint: enable=W0212return ServerTidyProtocol
def client_protocol_factory(cls):
"""Factory to create tidy protocols."""if cls isNone:
cls = protocol.Protocolclass ClientTidyProtocol(cls):
"""A tidy protocol."""def connectionLost(self, *a):
"""Connection list."""# pylint: disable=W0212if(self.factory._disconnecting
andself.factory.testserver_on_connection_lostisnotNoneandnotself.factory.testserver_on_connection_lost.called):
self.factory.testserver_on_connection_lost.callback(self)# pylint: enable=W0212
cls.connectionLost(self, *a)return ClientTidyProtocol
class TidySocketServer(object):
"""Ensure that twisted servers are correctly managed in tests.
Closing a twisted server is a complicated matter. In order to do so you
have to ensure that three different deferreds are fired:
1. The server must stop listening.
2. The client connection must disconnect.
3. The server connection must disconnect.
This class allows to create a server and a client that will ensure that
the reactor is left clean by following the pattern described at
http://mumak.net/stuff/twisted-disconnect.html
"""def__init__(self):
"""Create a new instance."""self.listener=Noneself.server_factory=Noneself.connector=Noneself.client_factory=Nonedef get_server_endpoint(self):
"""Return the server endpoint description."""raiseNotImplementedError('To be implemented by child classes.')def get_client_endpoint(self):
"""Return the client endpoint description."""raiseNotImplementedError('To be implemented by child classes.')@defer.inlineCallbacksdef listen_server(self, server_class, *args, **kwargs):
"""Start a server in a random port."""from twisted.internetimport reactor
self.server_factory= server_class(*args, **kwargs)self.server_factory._disconnecting =Falseself.server_factory.testserver_on_connection_lost= defer.Deferred()self.server_factory.protocol= server_protocol_factory(self.server_factory.protocol)
endpoint = endpoints.serverFromString(reactor,self.get_server_endpoint())self.listener=yield endpoint.listen(self.server_factory)
defer.returnValue(self.server_factory)@defer.inlineCallbacksdef connect_client(self, client_class, *args, **kwargs):
"""Conect a client to a given server."""from twisted.internetimport reactor
ifself.server_factoryisNone:
raiseValueError('Server Factory was not provided.')ifself.listenerisNone:
raiseValueError('%s has not started listening.',self.server_factory)self.client_factory= client_class(*args, **kwargs)self.client_factory._disconnecting =Falseself.client_factory.protocol= client_protocol_factory(self.client_factory.protocol)self.client_factory.testserver_on_connection_lost= defer.Deferred()
endpoint = endpoints.clientFromString(reactor,self.get_client_endpoint())self.connector=yield endpoint.connect(self.client_factory)
defer.returnValue(self.client_factory)def clean_up(self):
"""Action to be performed for clean up."""ifself.server_factoryisNoneorself.listenerisNone:
# nothing to cleanreturn defer.succeed(None)ifself.listenerandself.connector:
# clean client and serverself.server_factory._disconnecting =Trueself.client_factory._disconnecting =Trueself.connector.transport.loseConnection()
d = defer.maybeDeferred(self.listener.stopListening)return defer.gatherResults([d,self.client_factory.testserver_on_connection_lost,self.server_factory.testserver_on_connection_lost])ifself.listener:
# just clean the server since there is no clientself.server_factory._disconnecting =Truereturn defer.maybeDeferred(self.listener.stopListening)class TidyTCPServer(TidySocketServer):
"""A tidy tcp domain sockets server."""
client_endpoint_pattern ='tcp:host=127.0.0.1:port=%s'
server_endpoint_pattern ='tcp:0:interface=127.0.0.1'def get_server_endpoint(self):
"""Return the server endpoint description."""returnself.server_endpoint_patterndef get_client_endpoint(self):
"""Return the client endpoint description."""ifself.server_factoryisNone:
raiseValueError('Server Factory was not provided.')ifself.listenerisNone:
raiseValueError('%s has not started listening.',self.server_factory)returnself.client_endpoint_pattern % self.listener.getHost().portclass TidyUnixServer(TidySocketServer):
"""A tidy unix domain sockets server."""
client_endpoint_pattern ='unix:path=%s'
server_endpoint_pattern ='unix:%s'def__init__(self):
"""Create a new instance."""super(TidyUnixServer,self).__init__()self.temp_dir=tempfile.mkdtemp()self.path=os.path.join(self.temp_dir,'tidy_unix_server')def get_server_endpoint(self):
"""Return the server endpoint description."""returnself.server_endpoint_pattern % self.pathdef get_client_endpoint(self):
"""Return the client endpoint description."""returnself.client_endpoint_pattern % self.pathdef clean_up(self):
"""Action to be performed for clean up."""
result =super(TidyUnixServer,self).clean_up()# remove the dir once we are disconnected
result.addCallback(lambda _: shutil.rmtree(self.temp_dir))return result
class ServerTestCase(BaseTestCase):
"""Base test case for tidy servers."""@defer.inlineCallbacksdef setUp(self):
"""Set the diff tests."""yieldsuper(ServerTestCase,self).setUp()try:
self.server_runner=self.get_server()exceptNotImplementedError:
self.server_runner=Noneself.server_factory=Noneself.client_factory=Noneself.server_disconnected=Noneself.client_connected=Noneself.client_disconnected=Noneself.listener=Noneself.connector=Noneself.addCleanup(self.tear_down_server_client)def get_server(self):
"""Return the server to be used to run the tests."""raiseNotImplementedError('To be implemented by child classes.')@defer.inlineCallbacksdef listen_server(self, server_class, *args, **kwargs):
"""Listen a server.
The method takes the server class and the arguments that should be
passed to the server constructor.
"""self.server_factory=yieldself.server_runner.listen_server(
server_class, *args, **kwargs)self.server_disconnected=self.server_factory.testserver_on_connection_lostself.listener=self.server_runner.listener@defer.inlineCallbacksdef connect_client(self, client_class, *args, **kwargs):
"""Connect the client.
The method takes the client factory class and the arguments that
should be passed to the client constructor.
"""self.client_factory=yieldself.server_runner.connect_client(
client_class, *args, **kwargs)self.client_disconnected=self.client_factory.testserver_on_connection_lostself.connector=self.server_runner.connectordef tear_down_server_client(self):
"""Clean the server and client."""ifself.server_runner:
returnself.server_runner.clean_up()class TCPServerTestCase(ServerTestCase):
"""Test that uses a single twisted server."""def get_server(self):
"""Return the server to be used to run the tests."""return TidyTCPServer()class UnixServerTestCase(ServerTestCase):
"""Test that uses a single twisted server."""def get_server(self):
"""Return the server to be used to run the tests."""return TidyUnixServer()class PbServerTestCase(ServerTestCase):
"""Test a pb server."""def get_server(self):
"""Return the server to be used to run the tests."""raiseNotImplementedError('To be implemented by child classes.')@defer.inlineCallbacksdef listen_server(self, *args, **kwargs):
"""Listen a pb server."""yieldsuper(PbServerTestCase,self).listen_server(pb.PBServerFactory,
*args, **kwargs)@defer.inlineCallbacksdef connect_client(self, *args, **kwargs):
"""Connect a pb client."""yieldsuper(PbServerTestCase,self).connect_client(pb.PBClientFactory,
*args, **kwargs)class TCPPbServerTestCase(PbServerTestCase):
"""Test a pb server over TCP."""def get_server(self):
"""Return the server to be used to run the tests."""return TidyTCPServer()class UnixPbServerTestCase(PbServerTestCase):
"""Test a pb server over Unix domain sockets."""def get_server(self):
"""Return the server to be used to run the tests."""return TidyUnixServer()

The idea of the code is that developers do not need to worry about how to stop listening ports in their tests and just write tests like the following:

As you can see those tests do not give a rats ass about ensuring that the clients lose connection or we stop listening ports… Or so I though because the following code made such approach break in Mac OS X (although I suspect it was broken on Linux and Windows but we never experienced it):

While in all the other platforms the tests passed with no problems on Mac OS X the tests would block in the clean_up method from the server because the deferred that was called in the connectionLost from the ServerTidyProtocol was never fired… Interesting.. After digging in the code I realized that the main issue with the approach of the clean_up code was wrong. The problem relies on the way in which the NullProtocol works. As you can see in the code the protocol loses its connections as soon as it made. This results in to possible things:

The server does know that we have a client connected and calls buildProtocol.

The connection is lost so fast that the buildProtocol on the ServerFactory does not get call.

When running the tests on Windows and Linux we were always facing the first scenario, buildProtocol was called which meant that connectionLost in the server protocol would be called. On the other hand, on Mac OS X, 1 out of 10 runs of the tests would block in the clean up because we would be in the second scenario, that is, no protocol would be build in the ServerFactory which results in the connectionLost never being called because it was no needed. The work around this issue is quite simple once you understand what is going on. The ServerFactory has to be modified to set the deferred when buildProtocol is called and not before ensuring that when we cleanup we check if the deferred is None and if it is not we wait for it to be fired. The fixed version of the helper code is the following:

The last few days I have taken a considerable time to remove all the errors from the Ubuntu One tests on Windows. The tests were leaving tcp connections in the twisted reactor that on some Windows systems will result on a DirtyReactorError which is a PITA. Due to a refactor the pattern I described here was remove (ouch!) and I had to re-add it. In this case, instead of doing the pattern everywhere is needed I created some helper classes that will ensure that the clients and servers are correctly cleaned up after you use them.

When using this webserver you have to be careful because we do not longer pay attention to the client protocols, if you do not use twisted to access it you have no problems (libsoup, qtnetwork etc..) but if, for example, you use the HTTPClientFactory you have to do something similar to this in your TestCase setUp:

Due to a small project I’m developing at the moment I had to access the Mac OS X keyring over python. There is some code of seen around but was not very ‘clean’. The following is my attempt to access the Mac OS X Keyring via python using ctypes, I hope it helps someone out there:

I have not added the code of the decorator because they are just noise, the only thing they do is to check that the keyring was indeed opened (self.keyring != None) and that the parameters with the given index are not None (I’m lazy and I prefer to use decorators for this mundane tasks that are done everywhere.

On Windows Ubuntu One uses the twisted reactor to run the Qt UI. The main reason for this is that the IPC protocol that is used was written in twisted. This has been giving us a number of head aches like the one we experienced today.

The following code simply shows a dialog that will as the user for his proxy credentials and will store them in the key-ring of which ever platform is being used:

And to give even more details, the following is what is used to spawn a thread to store the credentials on windows:

def set_credentials(self, app_name, cred):
"""Set the credentials of the Ubuntu SSO item."""# the windows keyring can only store a pair username-password# so we store the data using ubuntu_sso as the user name. Then# the cred will be stored as the string representation of the dict.return deferToThread(self.keyring.set_password, app_name, USERNAME,
dumps(cred))

A priori there is nothing wrong with the code, or is it? Doing an IRL test you will see that the credentials are never stored, what’s even more none of the deferreds are called. But why? In theory the qt reactor should be taking care of everything which includes the deferreds, the deferToThread and the execution of the ui.. well, it is not. When we look a little closer we can see that we use the exec_ method from the QDialog and this is the root of the bug. Lets put an example, the following is possible in Qt:

As you can see we are launching the dialog but we did not execute the application, but why is that? The main reason is found in the implementation of exec in the QDialog class (this time in C++, ouch!):

As you can see the implementation uses a QEventLoop, if you read the documentation you will noticed that using exec of the event loops and the default flag all the events will me processed by this event loop, which is not the main event loop, but a child one (small brain fuck here). This means that, due to the implementation of the qtreactor, when using a dialog exec_ (or better say a QEventLoop.exec method) the reactor main loop is not processing the events which means that all your deferreds, deferToThread etc.. will not be processed

Nevertheless there is a way to work around this bug in the qtreactor implementation (because is not the qt fault and it is well documented) which is doing the following workaround:

I have been writing some integration tests lately between Ubuntu One and proxies which use SSL certificates. The idea behind this tests was to be able to test that we deal correctly with those certificates that are not correct (notify the user, remember exceptions, etc..) For that I wrote this small function that I used to generate the certificates.

Recently in Ubuntu One we have been working or using PyQt for the UI of our application so that we could keep a smaller code base. While doing the move I noticed that we needed to have a widget similar to GtkArrow and to my surprise there is not one.

The following is a small implementation fo such a widget that might help other people that are in need of it. Even maybe someone that cares enough will write it in C++ and will propose it to Qt, sorry but I don’t have the time.

I have been playing around with jquery a little lately and I found that it is not very well documented how to add an input field like a checkbox inside the header of an accordion so that you can click on the header and the accordeon works as expected and the checkbox can be clicked. I’m not an expert in this things but I found the way to work around the issue:

Yesterday I was working on a small UI that uses a WebKit.WebView to show certain information to the user. My idea yesterday was to try and show a native context menu on the view that does not have the default menu items that the WebView has, the only way I’ve found to this is a as follows:

from gi.repositoryimport GObject, Gdk, Gtk, WebKit
from utils import get_data_file
class CustomView(Gtk.EventBox):
"""Main window used to show the ui and generate signals."""def__init__(self):
"""Create a new instance."""super(CustomView, self).__init__()self.set_events(Gdk.EventMask.BUTTON_PRESS_MASK)self._create_context_menu()self.scroll = Gtk.ScrolledWindow()self.scroll.show()self.view = WebKit.WebView()self.view.show()self.scroll.add(self.view)# load the html used for the main window, from that point# on we will let the html and js do the work and raise# signals via the titleself.view.open(get_data_file(MAIN_WINDOW_HMTL))# lets no use the webkit context menu
settings = self.view.get_settings()
settings.set_property('enable-default-context-menu',
False)self.view.set_settings(settings)self.add(self.scroll)# connect the on button pressed event of the event box# so that we can add a context menuself.connect('button_press_event', self._on_button_press_event)def _create_context_menu(self):
"""Create the context menu."""self.menu = Gtk.Menu()
delete_menu = Gtk.MenuItem("Delete Task")self.menu.append(delete_menu)def _on_button_press_event(self, widget, event):
"""Deal with the button press event."""if event.button == 3:
self.menu.popup(None, None, None, None, event.button, event.time)self.menu.show_all()

Does any one out there know a better way to do this. I know that there is a “populate-popup” signal I can listen to but I cannot get my head around on how to do exactly what I want, which is remove all defaul menu items and add my own. And, does it make sense to have to do that for every popup signal?