From the MySQL Telecom team

Wednesday Oct 29, 2008

Q: So what are some applications or prototypes you are actually working on? Which do you see as the most interesting ones? Can I do something useful with this today already?

In general, what I find most interesting are the use-cases that utilize the aspects that make a web-server on a mobile personal device unique. Use-cases that take advantage of the fact that the context - location, surrounding devices and people, etc. - constantly changes, and the fact that the web-site "administrator" is always there.

And I get all worked up when I think on the implications - even if I obviously don't know what they all might be - if all mobile phones were equipped with a globally accessible web-server (I ignore all technical challenges). For instance, we already have an implementation of "linking by proximity". That is, having reached one mobile web server, you can reach other mobile web servers in the proximity of the first. The servers are related because they happen to be in the same place. A completely new way for linking sites, which I think is very cool, but it does not work before the likelyhood that two mobile web servers accidentally are in the proximity of each other is significantly greater than zero:)

I am also really excited about the idea of building social networks - similar to Facebook, for instance - without any kind of centralized server. The devices just talk to each other directly and events and notifications are routed following the social network. That is, they reach people important to you, before they reach people you saw last time in school 25 years ago. But it's a long way before anything like that can be realized.

Q: You make a strong case, I have to admit!

And yes, you can already do useful things with this:) Just install the version available from Nokia Beta Labs and you'll get some ready-to-use web-applications.

Q: Thinking "outside the box", there are of course some use cases where you discard the traditional notion of a phone, where this could be interesting. For instance, with this software you could just leave your phone at some place, your summer cottage in Nagu for instance, go home and point your browser to that phone, and you have an instant surveillance camera! Is there any research in this direction, or are you really working mostly on the assumption that people want to have a web server in their pocket?

If you by "web server" think of, say, Amazon, then no, that's not what you want to have in your pocket. In fact, in the beginning we actually stopped talking for a while about "web server on your mobile" because far too many made that Amazon association, and talked about "HTTP access to your mobile phone from the Internet" instead. Same thing, but the latter does not give the same associations as the former:)

A good litmus test for the feasibility of running a particular web application or service on your mobile is "so why don't you upload the data to a regular server and share it from there"? If you don't have a good answer to that question, then perhaps you should not try hosting it on your mobile.

Your instant surveillance camera case above is one that would be hard to replicate using a regular server. With the mobile web server it's trivial to do that; it's just a few lines of Python or PHP code. And it actually illustrates one of the interesting aspects of having a website on a mobile device - the web server moves, and hence the content it returns may depend on the location and other context of the device. But this is a little contrived usecase - how often do you really need a surveillance camera and when you do, would you want to leave your mobile behind:) Incidentally, this exists already, so all you need to get this functionality is to install the system on your mobile.

Q: So you proved that a) it has already been done and b) still it's maybe not useful... perhaps I need to come up with a better idea :-)

A more interesting alternative is to involve the phone owner in the scenario. You know your spouse is in Paris and you wonder what she is up to, so you browse to her mobile site, click on a link that causes the phone to beep and display a message "Henrik would like to know what you are up to!" at which point she takes a picture. The page you get contains a map with her location, the image she just took and perhaps clickable icons representing other mobile web sites in her proximity.

There are a couple of interesting points in this scenario; part of the content was generated interactively, it was generated on demand, and the content did not exist before it was asked for and will not continue to exist after it has been returned. Further, it illustrates the personal touch of it all. The whole scenario doesn't really make sense unless the one browsing knows the owner of the mobile web server.

This last point is quite relevant really, because it means that scenarios that on first thought might not make sense, actually may do. For instance, one obvious web application is a dynamic gallery of the images you have on the phone. A counter argument for that idea is: "why don't you simply upload the images to some image-sharing site. The mobile will never be able to handle the load". Well, that's very true, if the intention is to share the images with the world. However, what if the only one consuming those images is your spouse for whom you take pictures when you are on your businesstrip? In this case the load is not a problem and you do not need to create an account on any 3rd-party site, thus leaving the images firmly in your control.

And you should not make the mistake of assuming that having a web-server on a mobile necessarily means that it must expose a "regular" web-site. For instance, it would be trivial to provide SMS and MMS kind of functionality using a mobile web server, in which case the functionality would also become a seamless part of the web.

Further, the situation is not either/or as you could easily utilize the mobile web server for providing real-time data that is mashed up with other data on a regular site. An example of this is the Facebook integration we have made. Using data fetched from the mobile, we can display in Facebook a map-view that shows your current location. And from within Facebook you can type a message that is delivered directly to the inbox of your mobile. That is, you can be present on the social network without actually being logged on.

And now I've only covered the what happens when you put a web-server on a personal mobile device. The way you use a mobile phone also changes once there is http access to it. For instance, it is trivial to create a web-interface to all core applications of the phone, so that whenever you are next to a PC - any PC, not just your own - you can use the big screen and proper keyboard of the PC for reading and sending SMSs, managing your contacts, interacting with the calendar, etc.

Q: You know I'm a big fan of using real keyboards! In fact this was my first thought on how to use this.

And with the added bonus of being able to do that from the office when you have forgotten your phone at home. Not for making phone calls obviously, but you will be aware if someone tries to call you and be able to send him or her an SMS telling on which number you can be reached.

Q: Amazing, hadn't thought of it that way. Since I work from a home office I guess it is unlikely to happen to me, but in my previous career I did feel really lost when I had forgotten my phone at home.

Q: Is there some other question I should have asked but didn't?

Q: Where can I download this thing?

The consumer oriented version - MWS - is available at http://mymobilesite.net. The entrypoint for our experimental PAMP stack that then provides you with PHP and MySQL is http://wiki.opensource.nokia.com/projects/PAMP. PAMP can be used stand-alone or be installed on top of MWS.

Q: It was also a surprise to hear that suddenly the whole Symbian platform is going to be Open Source. What do you think this tells us about where the world is going and where the telecom world is going? Or even just where Nokia is going?

I can't speak for Nokia, but in my opinion it tells that there in general are fewer and fewer reasons for keeping code closed, if there ever were any good ones.

Open sourcing something is surely no guarantee that a vibrant development community will appear, but not doing it is a sure guarantee that it will not happen.

One thing that open sourcing always brings is ease of communication. It is so much easier to communicate and share ideas etc. if you don't have to think about NDAs and the like, but can in all fora speak openly about issues related to the code.

The open sourcing of Symbian touches my work rather closely actually. Since I have been working with open source software, I have paid great care to only use public Symbian/S60 information. For instance, I only use the public SDKs and never internal ones where I would have had access to all the source. This in order reduce the risk that Nokia or Symbian IPR would "leak" via me into the open source. Obviously this has caused problems at times when something have not worked and a peek at the source immediately would have revealed the problem. Now that issue is about to disappear.

Q: I hear you there! When I was working on S60 projects myself, it was
the first time I really understood the value of having source code available. It is not that you'd want to touch the source code - mostly you don't, but you often do want to read it! (Especially with the S60 documentation we had few years back :-)

Exactly, without the source it might takes days or weeks to figure out the cause of a problem and the workaround for it, with the source at hand it might take 15 minutes.

Q: Final question: As most employees of Nokia Research Center I see you have patents or patent applications pending. What would be your opinion on software patents? And is there a way to answer this question that will satisfy both yourself, your employer and the MySQL community?

First, I must stress that I am not a lawyer, that I really don't know too much about software patents and patents in general, and that I here especially speak for myself.

In my opinion, a large number of software patents are bogus, in the sense that they describe a solution that any engineer, when given the same problem, would come up with. In other words, the novelty and the non-obviousness that should be there, is not there.

Furthermore, often software patents seem to be the patenting of an idea. That is, there is not really any significant investment behind it. If you invest a great deal into the research of something and come up with a novel result, then I think it is right that you can protect that result so that somebody else simply can't go ahead and use the result for reaping the harvest of your investment. But if it's just a bright idea that anyone could have had, then the protection in terms of a patent should not be there. In any case, given the pace of the software industry, at least the lifetime of a software patents should be significantly shortened.

Generally speaking I don't think software patents work and that it would be in the interest of everyone if they simply were abandoned. However, from a practical perspective software patents are a fact of life and there are patent sharks and others out there that are more than willing to go after companies who don't play the patent game. So until the global rules are changed, unilateral disarmament would not be without significant risk.

Q: Johan, thanks for a very interesting chat on an interesting technical topic!

Monday Oct 27, 2008

Q: But, we digress... so let me instead ask you the question everyone asks me when they hear about Apache and MySQL on a mobile phone: Why on earth would anyone want to do THAT?

Because we can:)

No seriously, there are good reasons. If we assume that it makes sense to run a web server on your mobile (see further down for reasons for that) and the web-server you use is Apache, then it's quite obvious that you also want to provide both PHP and MySQL. After all, some 40% of all web-sites in the world are powered by (L)AMP, so if you provide the same environment on the mobile, you have hundreds of thousands of developers who are familiar with the stack.

But, in my mind, there are also compelling reasons to have a proper database on the mobile. Currently, the way applications store their data varies a lot; many even use crude binary files. This means that there in many cases is no way to manipulate the application data, without detailed knowledge about the structure of the data or with the cooperation of the application.

Think how different it would be if all applications stored their data in a DBMS. Making backups would be trivial, as you could just interact directly with the DBMS, without having to know anything about the applications storing their data there. And the only thing an application would have to do in order to be included in the backup, would be to store its data in the DBMS.

Q: Clearly you have no idea how weird backup setups people use on MySQL too! Well, I guess for an environment like a mobile phone it would be relatively trivial, yes. Being able to see and modify your data with SQL is a cool idea, I have to admit!

And since my hammer is a web server on a mobile phone, the use of a DBMS would also make it trivial to expose the core data of the phone via a REST API that could be generated on the fly. Again, the only thing an application would have to do in order to publish its data would be to store its data in the DBMS.

Q: Hmmm... This reminds me, I should ask about security. I mean, how do people react to the great news that their photos, SMS and calendar could be automatically available on the Internet?

That's really a good question. The simple answer is that all security mechanisms available on regular websites are just as applicable on mobile ones. If you are certain that others cannot access, for instance, your Hotmail or Amazon account, then you should also be certain that you can prevent others from accessing your SMSs on your mobile website.

However, in practice it is more complex than that. Firstly, it seems that quite a few have a very personal relationship with their mobile phone. The mere thought that someone might be poking around in it, is scary. So, the security mechanisms must be such that not only is the data safe, but people must also be really certain of it. Secondly, while regular websites are managed by professionals or semi-professionals, mobile websites will typically not be. That is, the security arrangement must be such that a non-professional can understand it and thus be able to make informed decisions.

One thing I am certain of, is that a straightforward mobile website specific account/password arrangement will not do. The situation becomes totally unmanageable if you need a specific account for each mobile website you intend to access and where you should have a distinct password on each. That would also turn the phone owner into a regular administrator that must manage accounts and "answer support calls" when someone has forgotten his password. Some single sign-on mechanism is needed, where the bulk of the account management is moved away from the mobile website itself. The phone owner should at most have to decide, for instance, which ones of his contacts are allowed to see his gallery.

I also think that it is important that the phone owner can put the same amount on trust into a "message", regardless of how it has reached his phone. For instance, if you get an SMS from a certain number you implicitly assume it really is sent by the owner of that number. Now, if, for instance, an instant message, delivered over HTTP, is displayed on the screen of your phone and you are told it is from your contact John, you should be able to trust that information as much as if you would have gotten that message as an SMS from John.

Q: Going back to the specialities of Symbian and embedded programming, isn't it contrary to the whole idea of saving battery and sleeping as much as possible, if you suddenly start to run servers on your phone?

Well, sending and receiving SMSs consumes energy as well, as does making a phone call. So it's a question of utility - is the functionality you get, worth the energy it consumes?

The primary problem is that with most other energy consuming activities on the mobile, the user is in control. If the battery is low, you can choose not to surf the web in order to be able to make a phone call later. If you have a server, especially one that is accessible from the Internet, on the mobile, in principle you are no longer in control. There can be activity of which you are not even aware.

And that's a significant difference between stationary and mobile web servers. If you have a regular website, as long as you are not slashdotted, you don't really care who visits it. In the mobile case, the situation is different as all access costs, if not in monetary terms then in terms of energy consumption. With the advent of flat-fee operator agreements the monetary issue is going away, but the energy issue is not. And even if you have access control on your site preventing access by anyone but the ones explicitly allowed, the mere delivery of the rejected request consumes energy.

Now, this is only a problem if direct access to the mobile is possible. In most operator networks it's not and in order be able to reach the mobile from the Internet, the access must go through our transparent gateway. And that, of course, can and indeed does perform access control on behalf of the mobile. That is, an unauthorized browser cannot cause any cost to you.

And that's actually an interesting aspect of the whole thing. You want gateway based access control, not necessarily in order to protect the data, but in order to control who can cause cost to you. That is, if you are at home, connected over WLAN and with the phone connected to a charger, the need for that kind of access control disappears. So, not only may the content be context dependent, as pointer out further down, but the access control may have to be as well.

Q: So maybe Drizzle could be interesting to you, if it becomes a leaner version of MySQL? I remember Symbian is all-Unicode too, so they would fit well together.

I really don't know much about Drizzle, so I can't say anything specific. In principle the leanness is quite attractive, but then again, the mobile phones of today really have quite a lot of memory and other resources. The high-end Nokia phones today have 128MB of RAM and if you compare the mobiles from a few years ago to the mobiles of today and use that for extrapolating where the mobiles will be a few years from now, it is not unrealistic to assume that they will have gigabytes of RAM and hundreds of gigabytes of storage. I suppose that's a lot more than what the servers had that were used for running many MySQL servers just a few years ago:)

Q: I know this but it is still hard to grasp... So if we plug in the phone to a charger, so we have electricity and can disregard thinking about battery all the time, you're saying my Nokia Communicator is more powerful than the servers I was administering in the 90's?

Yes:)

We all know that comparing raw CPU speed does not tell you that much, but I'll throw in some numbers nonetheless. For instance, the N95, which no longer is top of the line, has an ARM11 that, according to ARM's website, delivers up to 740 Dhrystone. I googled around and according to one site, the 450 Mhz Intel Pentium II, which was introduced in the fall of 1998, delivered 813 Dhrystone. So that's roughly where we are in terms of CPU power.

And as for memory... In the early nineties I worked on a HP/UX workstation that had 256MBs of memory. In those days, that was considered an almost obscene amount of memory. All recent Nokia smartphones have 128MBs of RAM.

Sunday Oct 26, 2008

By the end of 2007, to the surprise of many of us, a guy at Nokia Research Center announced that they had ported and were about to publish the full LAMP stack running on the Symbian/S60 platform of Nokia mobile phones. They dubbed this the Personal AMP stack: PAMP, and you can run most of the popular PHP apps like Wordpress, Drupal, phpMyAdmin... out of the box on a Nokia phone now.

Today we had the opportunity to have a chat with Johan Wikman, the man leading the efforts of porting the AMP stack to Symbian. Johan works as Principal Research Engineer at Nokia Research Center and as such has also previously participated in porting other interesting things to Nokia phones, such as the Linux kernel, eventually leading to what maemo is today.

There is an interesting "it's a small world" aspect in that Johan used to study at Helsinki University of Technology about the same time as Monty, Mårten and Kaj were there.

Q: Hi Johan. I was thinking, for our US readers it might be necessary to start with the basic questions: What is Nokia? What is Symbian? And what is S60?

Hi Henrik.

Well, Nokia is the world's biggest mobile phone manufacturer with a global market share of just below 40%. In the US, however, the market share is significantly smaller. Nokia has its roots in Finland and a great deal of research & development is done over here and the headquarter is here as well, but otherwise Nokia really is a global company with activities all over the globe.

Nokia manufactures phones in all categories, from low-end phones sold in developing countries to high-end smartphone containing all the goodies you can think of, and then some. And these high-end phones run the Symbian operating system. So, just like you can talk about a Windows phone or a Linux phone, you can talk about a Symbian phone. S60 is basically a UI and look&feel layer on top of Symbian, but at times it is a bit difficult to say where Symbian ends and S60 begins.

Q: Also, maybe it is worth talking about development on Symbian a little more. For those who have never done it, could you explain how it differs from, say, creating a Windows app or web development?

Symbian as a programming environment is basically quite different from pretty much everything else. It has a C++ API that takes time getting used to and although it has processes and threads like most other OSs, you usually handle concurrent processing within an application using so called active objects and an active scheduler. Basically its cooperative multitasking between different activities of the application, where you must ensure that you frequently enough give the control back to the scheduler, so that it can let other active objects run.

It may sound primitive, but it actually works quite well, since the operating system itself is well suited for this model, and in practice you don't really have to worry about keeping the CPU for too long. Most OS functionality can be invoked asynchronously, so that you instead of making a call and then block while waiting for it to finish, just initiate the call and are then later called back when operation has finished. While the asynchronous call is executing other active objects can run. But obviously this model has an effect on how you structure your program.

However, from high enough vantage point, creating an application in Symbian is similar to creating an application in other environments. You just have to learn and get used to the required API frameworks. Admittedly, the initial threshold in native Symbian development is higher than in other environments. But nowadays you can also do application development using Python and that really makes things easy.

Q: And I haven't tried it myself, but I've read that you can now program on Symbian pretty much like a Linux system? That is, any POSIX-like system, to be precise.

Yes, recently, with the introduction of the Open-C libraries, the situation has changed significantly. Open-C brings a very large set of functions, typically available in a Linux/Unix environment, to Symbian. So now it is possible to program for Symbian the way you program for Linux. That applies mostly to non-UI stuff, but usually, regardless of the environment, you anyway have to learn the particular UI framework you intend to use. But just a short time ago it was announced that the Qt framework will be made available for S60. That is, provided you are familiar with Qt, you will be able to create software for Nokia smartphones without learning the native APIs.

Although Open-C makes the porting of existing open source software to Symbian much easier than it used to be, there are still a couple of things still that can cause headache. First of all, the security model of Symbian is completely different from that of Linux. In Linux everything revolves around the concept of a user (not necessarily a real one); files and other resources are are owned by a user and processes are executed using the credentials of a particular user. In Symbian there is no concept of a user and prior Symbian version 9 there was basically no security. At runtime processes were protected from each other, but any process could access any file anywhere and also use any system resource, for instance, initiate a phone call. You could say that all processes run as root.

In Symbian 9, platform security was introduced and now the situation is quite different. At build-time binaries are assigned capabilities and those capabilities decide what the process can do at runtime. What capabilities you can assign a binary, depends upon what kind of certificate you are going to sign the installation package with. If you manage with a limited set of capabilities, then a self-signed certificate - one that you create yourself - is sufficient. However, if you need a wider set of capabilities, then a proper certificate is needed. Anyway, trust is effectively given to applications, not to the person who runs them.

But because of the signing process there is traceability, so if some malicious software turns up, you know who is responsible for it. Except for the self-signed applications, of course, but they can't do that much harm anyway and there are lots of warnings when you install them. Further, every application now also has a private directory that no other application, except one with a very hefty set of capabilities, can access.

A more practical problem is that on Symbian all symbols of a DLL (shared library) are exported by ordinal and not by name. So, loading a function explicitly by name from a DLL simply is not possible. So, if your open source software needs that functionality, then you need to work around the limitation. A closely related problem is that it is not possible to export data, but only functions, from a DLL. Since a lot of open source software export data from DLLs, that's something that also must be worked around, and it may not always be trivial to do that without having to touch the code too much.

Q: Which brings us nicely into what we really wanted to know... Was it a lot of work to port MySQL? What about Apache and PHP? Did you use the Open_C library for this, or maybe that's the whole point of the excercise?

Apache was the first component that was ported. And that was originally made when Open-C was not available. At that point there were only a rather limited and buggy C-library, that originally (to the best of my knowledge) was created in order to port the Java VM to Symbian. So, a fair amount of modifications were necessary and, for instance, the dynamic loading of Apache modules was not supported, but everything had to be built into the main executable.

Then came the already mentioned Symbian 9 and Open-C. At that point I decided to throw the existing port away, take the latest Apache httpd version into use and redo the port. Obviously that was much simpler this time, not only could a number of workarounds be thrown out but a lot of things just worked. However, when Apache loads a dynamic module, it looks up a data structure by name from the DLL, and that, as I earlier mentioned, is not possible on Symbian. But I could work around that in such a way that now only a single line must be added to a module in order to make it dynamically loadable.

The next component that was ported was PHP. Initially it was made as an experiment by a Hannu Kankaanpää, a smart summertrainee we had. Later when we really started working on our PAMP stack, the PHP port was updated by Petteri Muilu, Markus Heikelä and Markus Grönholm from Futurice (http://www.futurice.com). From a Symbian perspective PHP is quite problematic, because there are loads of global variables that not only are used by other DLLs, but stored in static structures in other DLLs. Namely, the usual way of working around the data export limitation is to export a function that returns the address of the variable and then define a macro that calls that function and dereferences it. That way client code need not be modified. However, when the addresses of those variables are stored in static structures it no longer works, but somehow the structures must explicitly be initialized at runtime.

Primarily of this reason, PHP has been built as a monolith where all extensions are linked in. However, I'm currenly working on redoing this so that the dynamic loading of PHP extensions will be possible. That way you don't have to link in every PHP extension just in case.

Q: And how about MySQL then?

MySQL was also initially ported by Petteri from Futurice. For sake of simplicity everything is built as static libraries, which currently makes PHP very large, since a lot of MySQL code is linked into PHP. I'm currently also working on changing this so that the client library of MySQL is exposed as a DLL. But it was quite straightforward to port MySQL; if memory serves me, the first version run after just a few weeks, which tells about the portability of MySQL and also about the "Linuxness" of Open-C.

Getting the first version of an open source software to run on top of Open-C would be even faster if it only were possible to run the configure script that open source software typically use for configuring themselves, according to what the current environment provides. Unfortunately, that is not possible at the moment, so what configure usually does, must be done manually on Symbian/Open-C.

But the devil is in the details. Even if you can make the software run quickly, there will be rough edges that it may take a lot more time to fix. Especially if the edges are related to some strange problem that you first must be able to pinpoint.

Q: Did you get - or even need - any help from MySQL employees or community members, or our online resources?

There is some information about porting MySQL, but not much when the target is a completely new environment. So the port was made without any external help. After we had released the PAMP stack there was a report that it is not possible to create indeces after a table has been created, but only as part of the table creation. So I posted a question in the MySQL forum, but before I got a reply (that said that the MySQL developers hang out on IRC), I had already figured out what the problem
was:)

The problem was caused by one fundamental difference between Unix and Symbian filesystems. On Unix you can open a file and then remove the file, but still continue reading from and writing to the file. Only when you close the file it is lost. This works, because in Unix a file and its name are actually quite distinct. If that sounds strange, just think about Unix hard links that allow you to have the very same file in two or more places under different names. You can "remove" the file in one place but it continues to exist in all others. So "remove" just removes the directory entry, and only when there are no entries left, the file itself is lost.

Anyway, on Symbian you simply do not have this behaviour. If a file has been opened, it cannot be removed, not by another process and not by you. And because of this, the index creation fails. And this is quite hard to fix without modifying the code extensively, which you don't want to do. So for the time being you have to create the index at table creation time or live without it.

Q: I believe Windows has this same behavior! At least I remember a lot of times I tried to delete a file or folder and couldn't do it, and it is more or less impossible to trace down which application might be holding some files open. And Symbian also has C:, D: and E: drives :-)

Yes, Windows is somewhere in between, but still closer to Unix. You can delete a file that someone else has opened, but the directory entry will not disappear until all handles to that file has been closed.