Looking for engin.io replacement -joining our effort initiative

Following the announcement of engin.io ramping down, I need to look for another Backend As A Service solution replacement...
In addition to the 'standard' features (authentification, authorisation, social login, push notification, no-sql data store, database, logging...), I am looking after a "Qt friendly" solution that would be simple to integrate, robust and mature...in short, the perfect solution 😉.
Does any of you would have some feedbacks, recommandations to share ? I think this would be extremely helpful to share knowledge here...
I am planning to start evaluating from tomorrow Amazon Web Services and Google Cloud but these frameworks are so huge I am even not sure about where to start...any advice will be appreciated.
I would use this topic later on to share this experience with others having similar needs.

I am not sure if this is the right place for this topic...anyhow here are some feedback regarding my today progress :

Amazon : First, I have started to read the AWS documentation, but I found it very complex to dive in and determine what are the minimal sets of AWS module to implement ( do you have any recommandation about where to start ?). So I decided to put aside this option for now. Hope I will find something simpler to set up...

Google : then I switched to Google cloud and found a C++ google client SDK(great!) which I hopefull was able to build. Still after one hour of reading various documentation, I am still lost in the hundred of google API to figure out which one would best fit Qt for cross platform mobile application...so I decided once again to leave this option for a while...

Firebase (still google), I have started to evaluate Firebase which seems promising. I have check the REST API and it seems to be quite OK for NoSQL datastorage but :

Firebase use PATCH http method which is not (yet) supported by Qt XMLHttpRequest. It might be possible to use x-http-method-override uri to overcome this limitation but so far I didn't succeed...

I haven't been able neither to use .sv timestamp

Firebase support asynchronous notificaiton through what they call "rest streaming" but I don't understand yet how to set it up

Engin.io had a nice feature for creating server side thumbnails variant of uploaded pictures - I haven't see something similar with Firebase.

it seems FB supports social login, but this is not straight forward using the REST api (once again I get lost on the required steps to achieve this...). This seems pretty simple with the SDK (javascript, android or iOS), too bad it doesn't propose a C++ sdk.

That's it for today, I will continue tomorrow, but I will be glad if meanwhile someone could share experience on this topic.

I go on the evaluation of Firebase which seems quite matching my needs so far...but the REST API seems to be limited for now. I am considering starting a Qt wrapper around Firebase SDK. My plan would be to use the native Firebase SDKs depending of the OS and fall back using the REST api when the OS is not Android or iOS. In addition, I would like the wrapper to integrate a QabstractItemModel base that could be used from QML. Maybe this component could support others Backend...What do you think ?

I found this project an github which is a thin wrapper on Firebase REST API.

I don't use cloud storage myself so I don't have much to contribute here, but I wanted to say thanks for starting this initiative. I hope others who are interested will join in, so that everyone benefits.

I'm also a Qt CS user (Enginio + MWS), and evaluating other backends.
Firebase has some nice features, but the query API is very basic (no OR, can't chain AND,...). Another problem is that if you want fast data access, you have to use their SDK, the REST API works but is a lot slower. It's also a complete lock-in to the service.
Baasbox looks like a small(ish) project, as do most others (deployd, remotestorage.io, ...).

The best contender I've found so far is IBM Cloudant:

full featured (lots of features)

scalable

can be self-hosted

language-agnostic: REST API is not second-class

mostly OSS (CouchDB-based), less lock-in that you could imagine from IBM

Follow-up on my Cloudant evaluation: I ran into problems trying to figure out how to encrypt user passwords.
I'll have a conversation on that subject with IBMers tomorrow.
I hope to also discuss the broader subject of whether Cloudant is a good fit for 2-tiers apps (no server code, or almost no server code).

If you guys have questions, tell me, I'll try to insert them in the conversation.

I am almost ready to upload my code.
I just miss a class... that i used lately at my job and i should remove it from my personal code now...

It is a swissknife class that it is almost no needed, but does a basic task.

For the login.

My code uses google login in android, so security is taken by google and baasbox. It should be possible to do something similar for iphone.

For "secure" password login without using "social login" by the moment i just hash at application level the password, as by the moment i dont use ssh for requests. It doesnt make secure the system, but unless it doesnt exposes the original password in plain text.
It should be easy to improve the code to use a certificate and SSL sockets with some Qt magic.

Today i have a some social event... hope i can rip off that class and upload it tomorrow. The rest is ready

@bnogal Sounds great !
From my side, I couldn't progress at that rate...what I did is create the skeleton of a plugin (I was unfamiliar with the plugin mechanism) and request for a playground project from the developer mailing list. I have continued evaluation with Apigee candidate. But I feel I should restrain myself now and focus on a something simple that could be later on improved so I will start with the firebase plugin. Hopefully I should be able to publish in a couple of days.

Looking at people's attempts to get PouchDB working with Qt/QML, it doesn't seem very encouraging.

Do you have some links ?
There's a Python adapter project that uses Qt internally to wrap PouchDB, so a C++/QML adapter should be feasible. That said, maybe it's too much effort and we should to stick to the REST API.

It seems to be like most of the large JavaScript frameworks: it assumes it's running in either a browser or node.js, and QML's JavaScript doesn't provide either of those environments.

Yes.
There's another non-JS (Android) adapter that's using the same tactics as the PyQt adapter (hidden Webview), but it's also unmaintained.
What's strange is that IBM (Cloudant) is advertising PouchDB as the right solution for mobile apps sync, so they're focusing on "mobile" web apps maybe.

I think it would be helpful to create a Qt playground project so we could publish our backend wrappers in a commun repository. I have sent a request for creating the playground project to the development mailing list but I don't know whether it is the way to go...Do you know what is the process to follow to create a playground project ?

So far, I have designed a QML plugin which is a work in progress :

the plugin only offers for now a Firebase backend element which is quite limited : the only supported authorization method is the secret token id and at this time it is only a thin wrapper around the REST API (GET, DELETE, POST, PATCH and PUT are supported).

what remains to be done :

define and implement an API suitable for supporting multiple backend : one option could be to design an API similar to the former enginio (create, update, query, uploadFile, downloadURL, remove...) but this API relies on backend features that we would have to mimics (users and usergroups, fileref...) so I am not sure this would be the way to go...

create a model that could be nicely used from QML like the former enginio model

use the Android and iOS sdk depending on the deployed platform (and using the REST sdk otherwise)

implement other backend using the same API (cloudant, apigee, BaaSBox...)

I don't know if this does worth to publish this to github or to wait for the creation of a playground qt module...

I don't know if this does worth to publish this to github or to wait for the creation of a playground qt module...

I'd say, if you have some code, even WIP, publish it. When discussing specs in a group, actual code can help to keep everyone on the same page.

Given our limited resources, it would help a lot to have only a thin layer of code to add to another, wider audience project (at the moment Qt/QML on mobile is a small audience, sadly). I thinking specifically of Hood.ie. I don't know if it's technically feasible, but using Hoodie as a kind of middleware could save a lot of code; just mimicking its API could be useful too. Of course if (for instance) Hoodie is bound to the browser, we have the same problem as with PouchDB.

I just finished a pure Qt Quick CouchDB interface (no C++ or plugins) which meets my current needs for storage. It would probably be trivial to get it work with Cloudant, but I'm not interested in adopting yet another BaaS provider that's going to disappear in a month or a year like Enginio and Parse did.

I just finished a pure Qt Quick CouchDB interface (no C++ or plugins) which meets my current needs for storage. It would probably be trivial to get it work with Cloudant, but I'm not interested in adopting yet another BaaS provider that's going to disappear in a month or a year like Enginio and Parse did.

That's great! A CouchDB interface should be directly pluggable to the Cloudant service. See this. In fact, I gather that IBM is currently a main contributor to CouchDB, and was before the acquisition of Cloudant, which was originally an independant company. Think of Cloudant as CouchDB hosting + benefits.
The situation is different for Enginio because of the closed-source components. OTOH, CouchDB is an Apache project and Cloudant can be self-hosted if you don't like "public" clouds.

As an aside, why Enginio, or a least the main parts, can't be open-sourced is beyond me. Given the competitive nature of the Baas market, not having an "we-failed-commercially-but-here-is-the-source-code" exit plan is disappointing, given the Qt ties to the OSS/Free software world.

@Charby said:
Ok, I will wait a couple of days to see if we could get a playground project to collaborate, otherwise I will create a github (next week) with my project at its current state. I came back a shortwhile looking to Apigee and I think it could be the best candidate I have seen so far, I will keep Firebase in my project but will definetely add an Apigee wrapper in addition.

@gadlim You are right (sic) - Actually I missed that pricing page as I jumped directly to the console after having signed a trial free account. Apigee seems to match perfectly what I was looking for (100% RESTfull, push notifications, social login, already builtin collections for assets-users-group, console is complete and powerfull and documentation is comprehensive...). It is a very unclear what are the pricing plan for BaaS...but if it is really 300$/mo (or more) it will be a no go for me...

There's also Stamplay (https://stamplay.com/) that has some nice features, and others. The problem with these kind of vendors (Backendless, Stamplay, Backand, Firebase,...) is the lock-in. Like @xargs1, I'm wary of the future of the service, so I'm gravitating towards less black-boxy solutions, at least for the core DB services. For additional services like push, there's tons of options that can be plugged (and changed) separately if needed.
A tie-in to a particular technology (CouchDB or MongoDB for instance) is not ideal either, but it's a big plus if it's an OSS project IMO.

@gadlim I think the lock-in could be acceptable as long as we would use the backend services using a commun Qt API that would have a backup and reload function, don't you think ?
Regarding the push services, what would you advise as possible candidate ?
I have contacted Apigee to have a better understanding of their licence agreement, to summarize :

the free trial period is supposed to be limited to 30 days but they usually don't stop the trial period after 30 days

they do not propose a package containing only BaaS, so to get BaaS you need the enterprise-class Edge account which licence fee is around 200k$ yearly (!!!)...

FYI here is the transcript of the chat conversation :

guillaume: at 16:23:33
Hi, I am an independant mobile application developper looking for a backend solution in
replacement of engin.io cloud services (I am using Qt framework). I am evaluating apigee BaaS
which I find great but I am confused with the pricing...
guillaume: at 16:24:45
In the pricing table, the BaaS is checked only for the trial column. what happens after the 30 days
trial period ?
Dylan: at 16:24:51
Well BaaS is only available with our enterprise solution
Dylan: at 16:25:11
You can download the free trial and use as you please
Dylan: at 16:25:33
We are usually not to strict on ending people's trials at the 30 day mark
guillaume: at 16:27:26
Does that mean that I can continue using BaaS for unlimited period as long as I use the API within
the maximal API calls ?
Dylan: at 16:28:35
I wouldn't say unlimited but you can continue to use it and if you have any issues please reach out
to us
guillaume: at 16:29:44
What would be the price for using BaaS only ?
Dylan: at 16:30:38
You have to have our enterprise Edge version to use Apigee BaaS
guillaume: at 16:31:51
I haven't found the price of edge version, and I think that this package contains much more than
my current needs.
Dylan: at 16:32:02
We have Start Up and SMB Edge versions as well but they do not come with BaaS
Dylan: at 16:33:27
Enterprise Edge starts at 200K a year is that within your budget? If not, you are not going to be
able to use a paid version of Apigee BaaS.
guillaume: at 16:35:20
It is definitely not within my budget unfortunately...do you plan to propose a package for indie
(with BaaS only) that I could propose to my customer in addition with the mobile development ?
guillaume: at 16:37:19
My typical use case would be to develop a mobile application using apigee for a customer and to
sell the product. The customer would have to pay for Apigee services fee (+variable cost
depending of API usage)
guillaume: at 16:37:55
Is this kind of package offering already in your roadmap ?
Dylan: at 16:40:19
16/2/2016 about:blank
about:blank 2/2
Would you like to go to one of our Expert Sessions to learn more about our platform and see if we
could be a fit for your customers?
guillaume: at 16:42:27
If I have to invest time on Apigee I would need to ensure the business model is relevant for my
customers...
Dylan: at 16:44:34
Well if you need Apigee BaaS and you don't have a six figure budget then this platform is most
likely not going to suit your needs. If Apigee BaaS is a nice to have and you are interested in our
SMB or Start Up API Management solutions then I suggest attending a session.
guillaume: at 16:44:52
Depending of the project, I sell the development of mobile application several thousand of euros
(one shot). These application usually needs datastore, authentification and push notification. I
think most of my customers could afford to pay a couple of hundred dollars of reccurent fee for
the backend services
Dylan: at 16:45:56
Okay, we do have authentification and push notifications.
Dylan: at 16:46:26
http://apigee.com/about/products/api-management
guillaume: at 16:49:57
will that mean that my in my typical use case, let's say a company order a small application using
datastore, push notification, and authentification...It would cost let's say 5k$ for my development
but the customer would have to get a edge account (200k$ yearly) to continue using the app ?
Dylan: at 16:50:29
The Edge package that they need depends on their API call volume
Dylan: at 16:50:47
http://apigee.com/about/pricing/apigee-edge-pricing-features
guillaume: at 16:51:53
but they would need a Edge account (enterprise account) which you told me is around 200k$
yearly
Dylan: at 16:52:22
No, they only need an enterprise level account if they want BaaS
Dylan: at 16:52:42
BaaS is not available on the SMB or Start Up level subscriptions
Dylan: at 16:53:04
Without BaaS they could definitely choose one of the cheaper versions
guillaume: at 16:55:42
My understanding is that you are currently not proposing a BaaS only package (much cheaper
than edge) for mobile application...do you think this could be proposed in a near future ?
Dylan: at 16:56:29
I am not sure if we will offer that but yes we do not have a standalone BaaS offering at the
moment
guillaume: at 16:56:59
ok thanks.
Dylan: at 16:57:31
Have a nice day

To help synthesize the results of our evaluation of the different BaaS services, I propose to work on a commun document where we could all insert/edit the solutions we know (and also correct each other).
feel free to modify/update !

Follow-up on my Cloudant evaluation: I ran into problems trying to figure out how to encrypt user passwords.
I'll have a conversation on that subject with IBMers tomorrow.
I hope to also discuss the broader subject of whether Cloudant is a good fit for 2-tiers apps (no server code, or almost no server code).

Coming from Enginio (MongoDB), the migration can be painful. The main problem is that it's not as much "forget-there's-even-a-database" as MongoDB is. For instance you don't have the concept of the collection, so if you treat each collection as a separate DB in CouchDB, you have to manage different DBs, and for each DB you have to think ahead about the indexes you need (in MongoDB indexes are are not mandatory for queries).
In contrast, the Parse REST API is eerily similar to the Enginio API. There's not so many ways to implement a REST API over MongoDB, but still you have to wonder if Enginio used Parse.com as an inspiration.

In my case, I'm running out of time and have a fair amount of code to port, so my choice is made: I'll go with a Parse Server hosting. There's several options, I think I'll go with IBM Bluemix. I've followed the instructions to setup a Parse server, it involved a whole day of trial-and-error but it's working, and I've ported our main app in less than a day - it's just a matter of tweaking some parameters. It's not fully working yet because of the fact that Parse only sends back the ID of new objects instead of the whole objects, but it's still ten times easier than other options.

Porting the Enginio SDK to Parse Server or anything similar should be easy too.

@gadlim regarding the user passwords, If I am correct, there are 2 different methods, one can either base64 encode username:password to be passed as header (with "Basic " as a prefix) or you can post with something like this :

I don't really understand the 2-tiers issue as you can directly use Cloudant i.o Cloudant through BlueMix, what would be your concern in this case ?

I have spend the last days playing with BlueMix (as I need to set up Push Notification) but regarding Cloudant I came to same conclusions that this lack of collection principle might be an issue for porting enginio apps...Firebase seems to be a much better candidate in this field (and Firebase has a powerfull mecanism to notify asynchronously when new data comes in - in replacement of enginion web socket - and I haven't found something similar with Cloudant...that would be another issue when polling is not an option).

I am very interested in the Parse Server hosting solution...I thought Parse was a no go as they are ramping down the services this year...do FB plans to release Parse to opensource (or did they already do ?). BTW, do you know what are their reasons for stopping the service ?
-Edit - replying to myself ;-) - Parse.com did release to opensource the datastore which is great - but the not the complete Parse solution (social login, PN, auth...), anyhow Parse server is promising ! I think I will have a try...

@gadlim regarding the user passwords, If I am correct, there are 2 different methods, one can either base64 encode username:password to be passed as header (with "Basic " as a prefix) or

I'm refering to the way the password is stored in the DB, not how it's sent to the server. In Enginio it's hidden (and probably encrypted), in Parse it's encrypted, etc. For Cloudant you have to encrypt yourself in the client or in an "update function" you're setting on the server.

I don't really understand the 2-tiers issue as you can directly use Cloudant i.o Cloudant through BlueMix, what would be your concern in this case ?

Nothing in particular, except passwords, but there's a feeling you get from the sample apps + the opinion from others that used it that you're expected to write some server code for "serious" applications.

I have spend the last days playing with BlueMix (as I need to set up Push Notification) but regarding Cloudant I came to same conclusions that this lack of collection principle might be an issue for porting enginio apps...Firebase seems to be a much better candidate in this field (and Firebase has a powerfull mecanism to notify asynchronously when new data comes in - in replacement of enginion web socket - and I haven't found something similar with Cloudant...that would be another issue when polling is not an option).

I am very interested in the Parse Server hosting solution...I thought Parse was a no go as they are ramping down the services this year...do FB plans to release Parse to opensource (or did they already do ?). BTW, do you know what are their reasons for stopping the service ?
-Edit - replying to myself ;-) - Parse.com did release to opensource the datastore which is great - but the not the complete Parse solution (social login, PN, auth...), anyhow Parse server is promising ! I think I will have a try...

It's too early to know how the OSS Parse project will evolve, but it starts with a good momentum (around 600 000 apps), so it think it should carry on. For now, missing services are provided by the hosters, but for instance Push notifications have already been added.

What's interesting about Bluemix is that they have everything. But that's also a problem if you don't have a guide, because you're quickly lost in a sea of various tools and services. See for instance this , and that's just one single service.

Better late than never, I finally published a first draft of my work to design a Baas plugin.
The project can be found here : https://github.com/a-team-fr/QtQML-BaaS
It is really a Work In Progress stuff but things are slowly getting up...

For now, it is only supporting Parse Server, I used to start working on Firebase then Bluemix but I stopped and finally even dropped the code for now. I find Parse to be a much better candidate and I prefer first reach an operative state with Parse server only and then complete the plugin with others cloud services.

So far, the BaaS element can only manage users (told you it was an early phase!) and the BaaSModel can be used to get data and work as usual with a ListView for instance...I have implemented some of the feature for query but haven't tested yet.
The best (I think) is to build the whole project and see the sample project to see how to add new feature.
All comments are welcome ! I will try to improve the commenting and documenting...

I have an issue when deploying the plugin, the qmldir file is not copied to the plugin location - will investigate this later.

I think you're right, and that Parse Server is a good starting point. It may be even more than a starting point because it's evolving to be more modular and allow to plug many services, including other DBs.

I've (mostly) finished porting my app, it's working nicely. I had to patch the Parse Server code for a feature existing in Engin.io but missing in Parse (atomic updates of subdocuments), it wasn't that hard in spite of my lack of Node.js knowledge. I was pleasantly suprised by the size of the code (it's rather small, I was expecting a huge pile of exotic JS).
I've made a pull request for my patch, let's see how the discussion with the maintainers goes.

About your QML plugin, did you consider just forking/patching the Enginio C++/QML SDK instead ? If not, why ?

@gadlim I felt the same with Parse server, the project is rather small, simple and well written. Furthermore the project team seems very productive and I bet the project will become very popular.
I didn't fork Enginio for a couple of reasons :

I looked at the code and found it rather difficult to master - I still haven't understand the private class principle and I haven't found a design documentation. But maybe I should have spent more time to study and hopefully understand it...

I think forking the project would have been more difficult in the long term to main backward compatibility with existing API.

even if my project is not a fork from enginio, it is inspired of some enginio design principles - for instance I liked the enginio model, but rewriting a similar model was not a complex task (and it was interesting from the learning point of view)

I wanted the plugin to be as much as possible backend agnostic, that the reason why the main class is pure virtual and will be refined to ease integration of additional backend services.

I looked at the code and found it rather difficult to master - I still haven't understand the private class principle and I haven't found a design documentation.

Ditto, I don't fully understand it either (the websocket stuff in particular). The private classes are mainly for binary compatibility that Qt has to maintain for all the Qt5 line, it's much simpler without that burden.