Firefox 4: An early walk-through of IndexedDB

Web developers already have localStorage, which is used for client side storage of simple key-value pairs. This alone doesn’t address the needs of many web applications for structured storage and indexed data. Mozilla is working on a structured storage API with indexing support called IndexedDB, and we will have some test builds in the next few weeks. This can be compared to the WebDatabase API implemented by several browsers that uses a subset of the allowable language of SQLite. Mozilla has chosen to not implement WebDatabase for various reasons discussed in this post.

In order to compare IndexedDB and WebDatabase, we are going to show four examples that use most parts of the asynchronous APIs of each specification. The differences between SQL storage with tables (WebDatabase) and JavaScript object storage with indexes (IndexedDB) becomes pretty clear after reading the examples. The synchronous versions of these APIs are only available on worker threads. Since not all browsers currently implement worker threads, the synchronous APIs will not be discussed at this time. The IndexedDB code is based off a proposal that Mozilla has submitted to the W3C WebApps working group that has gotten positive feedback so far. The code for both APIs does not include any error handling (for brevity), but production code should always have it!

These examples are for the storage of a candy store’s sale of candy to customers, which we’ll refer to as kids. Each entry in candySales represents a sale of a specified amount of candy to a kid, specified by an entry in candy and kids respectively.

Example 1 – Opening and Setting Up a Database

This first example demonstrates how to open a database connection and create the tables or object stores if the version number is not correct. Upon opening the database, both examples check the version and create the necessary tables or object stores and then set the correct version number. WebDatabase is a bit stricter in how it handles versions by giving an error if the database version is not what the caller expects (this is specified by the second argument to openDatabase). IndexedDB simply lets the caller handle versioning as they see fit. Note that there is active discussion about how IndexedDB should handle version changes in the working group.

Example 2 – Storing Kids in the Database

This example stores several kids into the appropriate table or object store. This example demonstrates one of the risks that have to be dealt with when using WebDatabase: SQL injection attacks. In WebDatabase explicit transactions must be used, but in IndexedDB a transaction is provided automatically if only one object store is accessed. Transaction locking is per-object store in IndexedDB. Additionally, IndexedDB takes a JavaScript object to insert, whereas with WebDatabase callers must bind specific columns. In both cases you get the insertion id in the callback.

Example 3 – List All Kids

This example lists all of the kids stored in the kids table or the kids object store. WebDatabase uses a result set object which will be passed to the callback method provided after all rows have been retrieved. IndexedDB, on the other hand, passes a cursor to the event handler as results are retrieved. Results should come back faster, as a result. While not shown in this example, you can also stop iterating data with IndexedDB by simply not calling cursor.continue().

Example 4 – List Kids Who Bought Candy

This example lists all the kids, and how much candy each kid purchased. WebDatabase simply uses a LEFT JOIN query which makes this example very simple. IndexedDB does not currently have an API specified for doing a join between different object stores. As a result, the example opens a cursor to the kids object store and an object cursor on the kidId index on the candySales object store and performs the join manually.

IndexedDB generally simplifies the programming model for interacting with databases, and allows for a wide number of use cases. The working group is designing this API so it could be wrapped by JavaScript libraries; for instance, there’s plenty of room for a CouchDB-style API on top of our IndexedDB implementation. It would also be very possible to build a SQL-based API on top of IndexedDB (such as WebDatabase). Mozilla is eager to get developer feedback about IndexedDB, particularly since the specification has not been finalized yet. Feel free to leave a comment here expressing your thoughts or leave anonymous feedback through Rypple.

Keeping it civil is something but most people are sick of these over-paranoid responses these days. You expect someone to respect the content and ask meaningful questions. That one is not a good example. You explain a technology and the first commenter asks about how to disable it… How can we keep it civilized? I totally agree with Andy. You have to “think” before you say something. Like you do it when talking face-to-face.

You had me until the JOIN example… that’s a nightmare. Until there is some reasonable API for multi-table access no one is going to want to go through all that error-prone rigmarole just to do a simple join.

If you don’t provide a simple join function in the base API, you run the risk of nobody trying to grapple with the IndexedDB way of thinking for long enough to bother writing a simpler API over the top that contains what we actually want: i.e. joins!

A simple join function should be very easy to do, so why not just do it? Nested objects are great —awesome in fact — but they probably don’t cover even half of the cases in which people want to use joins; joins are more like cross-referencing documents rather than subsections in a document. (And that couchdb joins article shows just how illegible things can get.)

I think the overall approach and reasoning is good. Just join it with joins.

IndexDB is just a hash, which is conceptually simple, fast and all, but is useful only in very limited cases as a mechanism for storing and retrieving structured data. It’s fine for handling serialized objects.

A significant case for real storage is for mobile apps that should be able to work in the absence of a connection. It makes sense to have a subset of your server-based database stored locally and then synced at the next connection.

I am no fan of SQL (I have watched so-called developers abuse it for years), but it is a well-developed standard that solves a lot of problems that hashed databases don’t. The absence of a standard is a laughable reason to abandon the approach, and it’s classic bloat mentality that suggests we built yet another SQL engine on top of the IndexDB storage engine. SQLite is fine.

I have to agree that the code for IndexedDB, is far less attractive to me than the Web Database code. I’m a web developer, and insulated from most SQL work, so I would think that a methodology like IndexedDB would be more attractive to me, as it’s more familiar, but I have to go with the standard SQL query language here. I’m no DB admin, but that SQL code looks a lot more inviting.

I think what you lose in javascripty-ness with the SQL examples you gain in readability. I have to work hard to understand what you are doing in the IndexedDB examples (because although it may be more native to JavaScript, it’s new to us). Pretty much everyone knows SQL, so reading that is simple.

Also, just a question because it really bothers me, isn’t there a race condition when you say:

There is no race condition there because JavaScript will always run to completion. That is, your entire function will run before any callbacks or event handlers can possibly begin. We wouldn’t even try to dispatch success until your function has completed.

It saddens me to see Yet Another Asynchronous API that isn’t composable with all the other asynchronous APIs available to JS in a webpage (such as XmlHttpRequest, Web Workers, setTimeout and so forth). The best asynchronous-result model I’ve ever seen is Twisted Python’s “Deferred” class, neatly ported to JavaScript as part of the MochiKit framework:

By “composable” I don’t just mean “can be made to work with”, I mean “naturally slots in alongside”. XMLHttpRequest and IndexedDB both use an “event model”, but they use different events (“onreadystatechange” versus “onsuccess” and “onerror”).

How about this: imagine I have some code that currently retrieves various kinds of data with XMLHttpRequest, and I want to update it so that it first checks an IndexedDB cache before sending a request. If I want to implement this cache without changing all the code that calls it, my new code has to do all the IndexedDB stuff, then return an object that fakes the (somewhat complicated) XMLHttpRequest API.

Roughly translated, this says “Try to get the data from the database. If it’s there, it’s the result. If an error is encountered (such as ‘not found’), try to get the data from the web. If we get the data from the web succeeds, cache the found value in the DB and return that data. If we fail to get the data from the web, or fail to cache the data, return that error.

I’ll be the first to admit that Deferreds don’t make for the most clean and readable code, but asynchronous functions in a procedural langugage are never going to be clean and tidy – and Deferreds are far better than every asynchronous function having its own way of returning results and signalling errors.

You’re right, that Generator-based code does make the asynchronous nature look a lot flatter – I suspect it would get a bit uglier if your Generator-based asynchronous function wanted to call other Generator-based asynchronous functions, though.

Twisted has a few methods for integrating Python’s generators (which are very like the JS generators you talk about) with its Deferred system:

For what it’s worth, I agree with those that clamor a better way of doing asynchronous coding in JavaScript. There are several proposals out there, beside the already mentioned “Deferred” class in Twisted Python, Dojo has a similar Deferred module, and recent versions of Firefox has an experimental shallow continuations implementation, (see https://developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Iterators_and_Generators).

I would love to see the various asynchronous DOM APIs out there grow native support for one of these features. However trying to specify which model to use as part of a database specification would be a mistake. This could lead to disastrous results such as having XMLHttpRequest using one Defer solution, and IndexedDB using another.

I’d love to see a solution for asynchronous API developed for the web. We could then add support for that in loads of specifications, such as IndexedDB, WebWorkers, XMLHttpRequest, WebSocket, etc. However until we have such a standardized solution, I think it’s best to stick with the model that exists, namely DOM Events. That way libraries such as dojo.Deferred have a consistent API to work against to bridge the gap until we have something better.

As for joins. It was a tough call weather to include them or not in our examples. Proper use of IndexedDB would really be to put both the list of kids and the sales into one table. I.e. have objects like:

This completely removes the need for the join and makes things very easy.

You can still create an index which contains just the sales in case you need to do queries that way.

So really, object stores together with custom indexes removes much of the need for joins. The reason we still included them in our example was that we wanted to show an example of a complex query in order to spark imagination.

I’m glad you’re pushing for IndexedDB. Even though it looks harder to use from the onset, that’s where frameworks come in. The ability to have a low level API that you can use to build a user friendly database API (be i SQL based or whatever) makes a lot more sense than starting with a user fiendly API that gives you little flexibility.

That’s really what the working group is shooting for too. We are trying to create a low-level API that is powerful so that libraries can do some amazing things with it. It’s worked pretty well with the web so far, and the really good APIs have then been standardized and made faster by the browser vendors.

It sounds like what’s happening is that IndexedDB is creating a storage engine and punting on a well-established, highly functional, approach for managing the data.

The reason people hate SQL is that it addresses a fundamentally very hard problem, in particular enforcing data integrity. And having spent many years cleaning up bad data, I know that it’s a hard problem. Many databases get it wrong, some very wrong. SQL is hard, but in a certain way it’s still an elegant solution to a very hard problem. As I understand, this is not a problem that IndexedDB addresses, and that would be a serious loss.

The golden rule for a spec: simple things should be simple, complex things – possible.

In indexedDB even simplest things are horrendously complex. I agree that SQL syntax maybe should not be used in webapps, however it *should* be a better way to define queries in pure JS without writing hundred lines of code to do simple joins.

My verdict: IndexedDB is not ready for *any* production use at the moment.

Shawn,
There is a race condition in your code as the gentleman pointed out. You open the indexDB in one statment and then attempt to add a callback handler in the next statement. If the first call completes and the database open completes before the onsuccess handler is added, then the handler is never called.

If, on the other hand, the database open call does not complete before the onsuccess handler is invoked, then why not? The correct way is for the callback to be added at the same time the method is invoked so internally, the handler is added BEFORE the attempt to open the database occurs.

There isn’t actually a race condition in this code, as I indicated here. JavaScript is single threaded, so your asynchronous callback is not going to happen until your current code is finished executing. I’d be happy to go into more detail if needed here, but this is pretty trivial for implementers to ensure consumers don’t race.

In the above code (single threaded), statement 1 completes before statement 2 begins. Correct? When statement 1 completes, it can be true that the database is thus open. Typically, all the registered listeners are notified BEFORE statement 1 would complete and this is how a race condition can be avoided. Ergo, it is deterministic.

Line 2, however, now attempts to add a listener. What code, then, will be notifying the listener in line 2 of the open event, and whatever code does that, resides outside the completion scope of line 1. yes?

I think you are making the assumption that openDatabase is synchronous, but it is in fact asynchronous. No events will be dispatched until some time after openDatabase returns. I’m going to do a new post handling some of the more commonly asked questions with detailed answers where I’ll provide a much more detailed response.

Looks like a question whether we prefer CouchDB-like databases, or SQL-like ones.

The autor of the artice argues that we could implement both on top of IndexedDB, but that’s also true for any other database API. We could as well implement IndexedDB on top of WebDatabase. We could probably implement WebDatabase on top of local storage, so what?

Those implementations will be slower than a native mechanism, so in my opinion, browsers should implement the faster and more flexbible one.

About substituting joins with proper database structure — there are situations in which joins will still be necessary. For example, an app may have a timetable, or some kind of a calendar. And we want to check, whether there any two events at the same time occuring.

IndexedDB is essentially just exposing a B-tree API. CouchDB and SQL-based databases are all implemented on top of a B-tree. IndexedDB is really the lowest common denominator. The group working on the spec is designing this to be simple enough that people can use it without a library, but fully expect libraries to spring up and solve lots of pain points of a raw B-tree API.

As for performance, your argument doesn’t really apply. Disk I/O is going to be far slower than JavaScript parsing the data. Even then, JavaScript engines in modern browsers are not exactly slow anymore.

“I think you are making the assumption that openDatabase is synchronous, but it is in fact asynchronous.”

Shawn, this is why its a race condition. Maybe in your tests that amount of time to open the database consistently happens AFTER the result.onsuccess registration, but you cannot expect that behavior uniformly. Its nondeterministic, from a computational perspective.

To make it computationally reliable and deterministic, the handlers have to be registered prior to the open call or an exception should be thrown. Gambling that one call _should_ finish before a separate, asynchronous call is invoked leads to problems, IMO. Just my $0.02.

It’s true that opening the database is actually nondeterministic but the notification that it was opened is deterministic. The notification must be dispatched on the main thread per spec; this is the thread that the code creating the request is running on. Since JavaScript has run-to-completion semantics as Shawn mentioned earlier, the notification cannot be dispatched until the JS code that made the request is finished – this includes statement 2 and any future statements until control flow is returned to event loop. So in your annotated example, the notification could not possibly be dispatched before statement 2, even if the database was actually opened before then. Make sense?

you are still not getting what he is saying, the call to open the database is not done UNTIL the script exits. so if i sat there calculating the sum primes from 1 to a billion between those two statements, we still wouldnt have a problem, because until im done doing my calculations, the call to open the database has NOT happened. once its done, database is called, and the notifications sent.

So if I step through the code with FireBug, all the code in the database open call will complete and not block. yes? then the function completes in the main thread. yes? Ok, then what code that issues the notifies? Your are saying the execution now goes back to some previous code to complete? I don’t think so.

Where is that code? Back in the database open method? So then you are saying after the database open method, there are NO side effects? that’s complete nonsense from a programming point of view. But I will just wait for the code and step through it myself.

“So if I step through the code with FireBug, all the code in the database open call will complete and not block. yes?”

Yes.

“then the function completes in the main thread. yes?”

Yes. But note that the function does not complete by opening the database. It completes by queuing the execution of a synchronous open in a separate thread context and returns.

“Ok, then what code that issues the notifies?”

When the aforementioned synchronous open finishes on a separate thread, it queues some code to run in the main thread’s event loop. That code will run call onsuccess.

“Your are saying the execution now goes back to some previous code to complete? I don’t think so.”

Well I’m not sure why you think open() doesn’t return like a normal function. It returns alright but there’s no guarantee the database has been opened when it does. If the database is opened by the time it returns (or at any point after it queues the synchronous open), then there’s an event waiting in the main thread’s event queue to dispatch the notification. That event doesn’t get run until the current thread of JS execution finishes. This is what is meant by “run to completion” semantics.

“So then you are saying after the database open method, there are NO side effects?”

From the script point of view, no, not until it yields control back to its original caller or otherwise invokes the event loop.

From the browser’s point of view, there is of course a side effect – spawning the new thread of execution to open the DB.

I would love to see a native XML database using XQuery made accessible to websites (and potentially shareable across applications).

I just cannot understand why XQuery has not been embraced more widely except by some die-hard fans like myself, outside of venturing that it is because there is a lack of accessibility in common and accessible development environments like JavaScript or PHP.

It is such an elegant and expressive language, and to me fits in with the web model more than any other.

The examples above show the IndexDB as longer than the SQLite one. If the contrived examples are longer, that does not seem like a good sign, though I’ll admit I haven’t studied it carefully. But it doesn’t grab me as ultra-friendly like XQuery is.

Shouldn’t the web leverage existing understanding of X/HTML on which the most basic unit of the web is built (as XQuery so conveniently allows targeted inclusion of XML in the input and output–similar to how E4X does, though which unfortunately hasn’t been given attention in other browsers) and is also based on other already convenient, standard, and familiar technologies (as XQuery leverages XPath)?

Shouldn’t this new API work equally well for hierarchical data, as most web documents are?

Shouldn’t data table creators be able to use simple XHTML to build an XHTML table and have it both be directly viewable and editable as a document in its own right as well as queryable easily through XPath/XQuery?

If you want to work with tables, let people create the tables they’re most familiar with, . And for those who want to be able to work conveniently with hierarchical data (which might include for that matter) including via direct queries to a document or document collection, let us work with that…

I have no idea how well it stacks up, but there is already a very small XML database (Apache license), Sedna: http://modis.ispras.ru/sedna/ which might be usable as a base.

XHTML is dead/deprecated, and in my opinion we need popular standards, that most people understand, there is already too much stuff a web developer needs to learn.

This is why we need SQL, we need to minimize the amount of required knowledge.

For example, XUL may be better for expressing UIs than HTML, but the fact that it’s required for created simple Firefox extensions is awkward. I don’t want to learn a zillion new technologies, my clients don’t pay me for technologies, they pay me for features.
It is nice to create a simple browser-extension for a site, but in this scenario I don’t want to learn a new language just to be able to do it.

My example may be hard to understand, but here is a simplified version(not directly related to this comment):
Stop inventing new technologies!!! We don’t want new technologies, we want stuff that actually work.

At risk of distracting from my suggestion just now to simplify, I might add that there are XQuery-style interfaces to RDF as well (e.g., http://www.w3.org/2004/07/16-FA/ , http://www.fatdog.com/xsrql.html ), as might appeal to those who would go through the trouble of generating highly structured data (e.g., see http://dbpedia.org/ ), though I’m only suggesting this as a possible supplement to the far simpler XQuery+XML, not a replacement.

In response to the comment just now, XHTML is not dead, as HTML5 is itself allowing an XHTML serialization. If you know HTML, it will take you about 5 minutes to learn how to write XHTML: http://www.w3schools.com/xhtml/xhtml_html.asp . And it’s not inconceivable that an XQuery-based XML database could be made to work against HTML (especially now that the rules for parsing HTML are being spelled out explicitly in HTML5).

I did this with my add-on XqUSEme which uses Firefox’s own internal representation of HTML (which is usually close enough to XML) : https://addons.mozilla.org/en-US/firefox/addon/5515 , as apparently also does the interesting project XQIB in using Tidy: http://www.xqib.org/ ). But neither of these currently work with XML databases (only individual documents).

As far as the query language, SQL is more familiar (if not exactly standard), that is true, and if I had my way, I’d even advocate allowing it. But SQL is not set up to work with hierarchical data, while XML+XQuery can do either. In my view, SQL’s success is an accident of history, while if XQuery (on which the SQL co-creator himself worked) had been the first out of the gate, it would have taken over. I hope the next generation will have access to its intuitiveness and power, especially in combination with Full Text and Update functionality included.

Here are a few interesting use cases for XQuery against the Wikipedia database (using the very interest looking Sedna XML database I mentioned): http://wikixmldb.org/help/use-cases

Its easy to understand developers that can’t imagine nothing more than tables. But Web is a graph over data organized hierarchical (domains n documents), I don’t see any tables there. I welcome storage more adequate for semi-structural information. My question then, taking excellent example of native XML database http://exist-db.org, how easy will be to implement some XML indexing scheme ( http://atomic.exist-db.org/articles/wolfgangs_talk.pdf ) for IndexedDB?

Or if we adapted a library like jQuery to accept XPath (as should be reasonable to convert CSS Selectors to the more powerful XPath) and handle collection() and doc() as used in XQuery, we might use the same API as above to simplify to:

(1) It appears that you can have only one (primary key) index per table. As a result, many operations in a non-simple application will have to do table scans, and ordering/paging will be an issue. For example (for ordering/paging), suppose each company has zero or more employees. I want to write a query to display employees (ordered by last name, then first name) and for each employee, to display the employee name and company name. And in fact I want only from row M to N of that ordered result (to do paging). Now without an index on employee for lastname,firstname it seems unlikely to be possible to express this in any efficient way.

(2) I really think we need a synchronous database API. I like that with the Web SQL Database API, there are both synchronous and asynchronous APIs available. My concern with the lack of a synchronous API is that composition becomes a nightmare. Suppose I have a class Employee, I would like to be able to generate code so the user can write:

var e = … // get Employee from some query
e.setPhone(555 123-45567);
e.update();
… // do something after the update

If the generated Employee class has to use an asynchronous database API, then we cannot encapsulate the database access inside a synchronous update method. Now if update could take an unbounded amount of time, or for queries that may take a lot of time to execute (especially if there are insufficient indexes), I appreciate that an asynchronous API can be useful. But for well-designed queries/updates on tables with appropriate indexes, with the database API encapsulated inside generated classes, asynchronous is simple a non-starter.

My thoughts here are focused on building business applications for mobile devices (e.g. smartphones).

Yeah, we wrote this code long before we had builds working, which means it wasn’t ever tested (and may have bugs like what you found). I’m working on a new demo that actually runs in builds (and can be interacted with).

I was very happy to find out that IndexedDB made it into the first beta and decided to prepare a little demo/simple higher-level API with limited functionality so that you can just grab it and test it indexedDB for yourselves. I put it up on my company’s blog: http://antmicro.com/blog/2010/07/async-indexeddb-js-api-test/

Please let me know if it works for you :) of course it’s far from complete, plus I’m not really a js coder (we mostly do embedded systems) so do not hesitate to tell me if things could be improved.

I think this is permitted to store that as a value, but what happens if you create an index on the ‘address’ field? Is that allowed? Seems like it would then need to behave as a key, with the restriction to types that have a ‘natural order’.

Ideally I’d like to be able to create an index on a complex type and supply a comparison function for the engine to use when sorting the index. (or alternatively some kind of path syntax, eg an index on ‘address.postcode’)

I don’t like the IndexedDB approach and would highly appreciate if all major browsers would implement the same api instead.
As webkite and opera already support Web SQL (SQLLite) I think it is a very bad decision to take another route just for some academical reasons.
I am using GWT to implement my JavaScript apps so it would be possible to add a GWT layout on top of Web SQL to provide a cleaner API if you are not able to live without it.
But if choosing another approach it should at least support all the features of SQL Lite, otherwise it is worthless.
So please: Add support for SQLLite so that web developers are not forced to use Google Gears to be able to implement offline capable rich web applications that run on all major browsers!

Thanks for the info. I can probably track it down from that. I assume your ‘add kids’ code example above must have the same problem since neither of us are explicitly doing any transaction stuff? (the default is READ_ONLY then?)

Would be great to see some more explanation of the transactions. For example I’m not really clear why you used a transaction in the JOIN example since you’re only reading and not writing.

I already sussed that you have to use the moz_ prefix on indexedDB thanks to the Antmicro guys above…

Current thoughts in the working group is that it’s more useful to throw at the call site when we know something is going to fail immediately instead of doing it asynchronously with events. It’s easier to debug, and developers get immediate feedback.

If you have a write transaction, it will never bounce because you already have the lock needed for that object store. Locks do not automatically promote either, so if you open as READ_ONLY, it will always be READ_ONLY.

Ah! I finally realised that you can get your objectStore from the transaction object and start using it immediately… you don’t have to wait for the transaction.oncomplete event (that gets fired after you’ve added your objects to the store or whatever).

One thing that seems missing is the ability to delete a db? You can removeObjectStore but I don’t see the equivalent for a db.

I am testing IndexedDB. I can insert data in the object store. However, I can not update the data stored in the object store. I attempted to use “add” method with two arguments as following code.
————————-
transaction.objectStore(“my_object_store”).add(new_object, id).onsuccess = function(event) {
//some code
}
————————-
First argument was “object” and second argument was “id” tnat should be updated. And, it failed. If you know, please teach how to update it.

I suspect you have the same problem Anentropic has above you. You need to open the transaction as READ_WRITE in order to write to the object store. If you do not specify, the default is to open it as READ_ONLY.

Thank you for the answer. However, I was not able to solve this problem by your answer. Because I am not a native speaker of English, I might not be able to explain this problem well for you. I know that I must open the transaction as READ_WRITE in order to write to the object store as you say. Therefore, I can “insert” the data in the object store. But I cannot “update” the data that I inserted in the object store once. And I want to know how to “update” the data. I did not succeed though I executed the above-mentioned code to “update” the data that I inserted in the object store once. It would be greatly appreciated if this explanation is useful for you.

Thank you for your kindness!!! By reading test code that you showed me, I was able to understand how to update data that was inserted in a object store. By your kindness, I became to be able to use “modify” method and “addOrModify” method. I understand how to use them is related to how to create a object store. And, I will work to develop my web application by using indexedDB. I am looking forward to the completion of Firefox 4. Thank you really for this time.

I updated Firefox 4 from beta 4 to beta 5. And I noticed that ObjectStore object did not have “modify” method and “addOrModify” method for IndexedDB on beta 5. But ObjectStore object seems to have “put” method and “clear” method on beta 5. To update a existing record, we seem to need to “clear” the existing record once and then “put” the new record.

I wrote “To update a existing record, we seem to need to “clear” the existing record once and then “put” the new record.”. But, after that, I noticed that all records of the object store were cleared when “clear” method was executed with specifying the identification number of the existing record as the following code.
————————————————–
var request = my_object_store.clear(id);
————————————————–
Is my usage wrong or is that a bug?

I noticed that we should use “remove” method to delete the existing record as the following code.
———-
ver request = my_object_store.remove(id);
———-
I apologize for my offer of the erroneous information. So, to update a existing record on Firefox 4 Beta 6, we seem to need to “remove” the existing record once and then “put” the new record. But I hope that we can use “put” method without using “remove” method to update the existing record.

I have been working on developing a web application by using IndexedDB. When I created two indexes for one object store, the program did not work as expected. Cannot we use two indexes for one object store? I think that we might want to retrieve a object store by two or more indexes.

To explain the situation that I encountered, I wrote the sample code, and posted it yesterday. However, the post is not displayed today. Because the post was too long, that might not have been accepted by this comment system. I have the post now. I want to show it but I do not know how to show it for you.

Is IndexedDB portable between different operating systems? I am developing applications for ebooks which will save book contents in the database. It will be great for users if the database is portable between any operating systems.

The execution result of the above-mentioned program seems to show that having two indexes for one object store is impossible. Is that a specification of IndexedDB or is my implementation wrong? I want to know it.

I have to give favor to the WebKit SQLite side of client side db’s. SQL works for many other programming implementations, so why not in Javascript too. The point is, Mozilla (IMHO) should have done the same thing as WebKit SQL or whatever you want to call it, just to create a near de facto standard. The more browsers using the same client side db’s syntax and style, the better. A missed opportunity because programmers prefer to lead than to follow.

And that isn’t to say that IndexedDB isn’t good or better, just that having all future browsers using a common db strategy would have been wonderful (a guy can dream). IndexedDB could then become a leader in the next generation, additional db implementation for javascript. It would seem that IndexedDB has a place (document-centric or JSON based), but what we needed was a more common (dare I say standard) interface in javascript now.

Microsoft was very much against implementing WebDatabase, so even if Mozilla could ignore its issues with the specification, you still would not have had it all browsers. Contrast this with Mozilla, Webkit (via the chromium team), and Mozilla all committing to and working on the IndexedDB specification.

So what if Microsoft was against developing IE with an SQLite DB, Shawn? Is that so much of a surprise since they market relational DB’s and were never likely to come on-board anyway. Issue’s with “the specification” seems like a wafer thin excuse not to implement what is becoming a popular back-end persistence layer. Is anyone actually in charge at Mozilla anymore?

So,there’s no need to know what kind of database or where it located when we create a IndexedDB and use it?
I am not good to IndexedDB’s API.most of them still look strange to me.I’m not a skilled web coder yet.

How should we think about the security of indexedDB? I am working to develop small web application using indexedDB, then I am taking measures against cross site request forgeries similar to server application to my application. That is, I send a request of user to server once, and I verify it, and finally I send the instruction to write data to indexedDB of user’s browser. But because JavaScript has “same origine policy”, I think it may not be necessary that I send a request to server and verify it. What do you think of this idea?

I have just read this article, not the comments yet, so excuse me for rushing in to comment, if I make a fool of myself it’s only because i feel strongly about this situation. Isn’t this SVG vs VML all over again? From what I have read, it was egos and pride back then, and we all lost because of that. Had we have a agreed upon vector standard back then the web better today.

What i’m arguing for is get the guys from Google, Apple, MSFT, Mozilla and Opera in a room, NO ONE leaves until we have white smoke. MOVE THE WEB FWD FAST. Ration the pizza while locked there :-)

Sheesh, had JavaScript was to be created by a you guys, we would still be waiting for it today ;-) If there is a lesson to be learned i think it’s: choose something already, who cares if it’s less then perfect, no one is going to jail you for that.

Javascript was invented by a single person/corporation (Brendan Eich/Netscape) in the beginning. If it had to be invented and agreed upon by a committee from several competing corporations from day 1, I wouldn’t be surprised that we’d waited 5 years before they agreed on what functions should be in it and another 5 before it was ratified and recommended for use and finally another 5 before we all had something that worked more or less similarly.

The sad thing is, history has proven it again and again (VHS/BetaMax, HD/BluRay), we’ll all be forced to choose to learn one, or spend extra time on both until one finally wins in the market place.

Clients only care about the end result on time, I just want what works and if I already know it, even better. If I have to learn something new, I will. I just don’t want to have to learn TWO just to do the same thing on all browsers. This stinks like the whole cross-browser compatibility rubbish again.

I think exposing a B-Tree database as the foundation makes sense. I agree that using it directly looks overly complex, but we are really just a jQuery plugin away from elegance. You really can build a decent performing relational, hierarchal, or document oriented database atop this foundation.

I appreciate the efforts of those on the forefront of these decisions and discussions. It takes a lot to put your technical considerations and beliefs in front of everyone to ramble on about. It benefits us all greatly!

I personally want both. I am on the SQL side of the syntactic likability factor for storing relationally structured data but I also love the javascript methods and programming models.

I do see all kinds of situations, particularly on the web where I want/need to store objects. I want to persist them for later use in my client side applications.

So I want both types of storage available at my disposal.

What I come to at the end is if I use the SQL option, and need to use it or send it back home, I have a bit more Marshalling to do to use it on the client side or to get/send it over the http model. I have to pack it up at one end and unpack it at the other end.

I don’t know why that firefox 4.0b7 changed the indexedDB API,I made a demo before,and it works all nice in beta 2,but now it totally can’t work in beta 7.do you know anything about the method of indexedDB.open() ?

I think the way Mozilla are heading is correct. If we don’t get this right now, before web-application’s begin to supplant client-side apps, we’re just going to be in one whole kludge of a mess that can’t be undone easily.

Look at the trouble a single short-sighted product design has inflicted on the technological growth of the internet by not thinking of the long term implications. I’m talking about internet explorer.

While the guy several posts up with broken English was a little hard to understand, I agree that the well established model of thinking of data in terms of tables and columns is an overly restrictive and case-specific thing to force on the web as a lowest common denominator. More loosely defined object stores just make more sense here and as has been stated, more rigid table-like semantics can be implemented on top of this if required.

I do think that we definitely need a clear and concise syntax for querying these object stores. Again this, as stated, makes sense as a higher level implementation..

In terms of a simple, flexible, but structured and finitely defined querying approach, I can’t help but think of Microsoft’s LINQ – Or more specifically, The whole IEnumerable interface and accompanying LINQ methods.

To be clear for those who know LINQ, the object methods required to support LINQ querying are distinct from it’s SQL-like query syntax, which just compiles down to methods on return objects. You can still use LINQ without writing a string based query.

A quick google turns up a couple of Javascript implementations of this!, which excites me a little if only to have a play.

Because JS is a prototype based language, supporting the querying methods on enumerable objects (just arrays??) should be fairly easy to complete in full. This looks like the case with the links I posted.

Where the real guts would come in is the more lower level interaction between it and DBIndex, so that data iteration can happen lazily. But I’m sure it could be done!

As a web developer, the solution is simple from my perspective. Drop firefox support in my apps as so many did in the prior browser war, and handle IE users with Chrome Frame. This is really fairly simple since firefox is still a minority browser.

No offense to mozilla but I really have no intention of holding off on using key technologies that work in favor of someone else’s political games.

To me it would be better to declare support for SQL92 or draft a new SQL-like spec. If current implementations don’t comply, then it should be considered a bug.

Dude,
Do not use w3schools.com for web stats, although it doesn’t explicitly state it on their site I am convinced that the stats come from users of their site, so their stats have some considerable sample bias – quite simply web developers, and aspiring web developers are more likely to use FF or Chrome.
If you found some better, more representative, stats you’d probably find IE is still leading (it is the default option on Windows, and not that many will change), followed by FF, and then Chrome. Opera, Safari and Rockmelt might have a percent or two each.

@Paul, I can’t believe you consider yourself a web developer yet you would drop one of the fastest advancing and most popular web browsers to date.

The majority of mobile users are using webkit now (iOS and Android), which Firefox complies with almost entirely. I find very few things that don’t work among the most popular browsers: Chrome, Safari, and Firefox.

Firefox 4 is offering some amazing new technology that will push online applications and gaming into the next generation. Other browsers will follow suit, so if you expect to have a job as a web dev I highly recommend not dropping support for Firefox in your apps.

To be clear I’m not talking about dropping support for firefox alltogether, but for apps that would use offline storage.

I really have trouble seeing how stuffing a whole new query language into javascript will be anything but painful for someone who needs to deal with more than 3 or 4 “tables”. Especially considering that this solution has us going back to walking through btrees.

WebDB is an existing standard in production-level browsers. Offline apps can be developed against it without having to support / keep up with an experimental standard used exclusively in experimental versions of browsers.

There’s no way I’d be able to keep my job as a web developer if I had to try and justify developing anything against this.

I for one am not a fan of this IndexedDB thing….yet. For one, I am comfortable with tables and database terms. However, that is not a good enough reason to say we should use that style of storage access and development. There is something to be said about using the notion of “objects”, as that’s what we deal with in our everyday physical lives, and so is a good analogy in terms of computing and storage. What I didn’t see from the examples above, is more complicated relational logic…which is important for, well, relating objects to one another. Now, perhaps the examples above are overly simplistic in an attempt to only demonstrate IndexedDB and contrast it with the proposed WebDatabase proposal (which admittedly, is severly flawed in that as of right now, it primarily deals with SQLite).

But, before we go jumping on the Indexed DB bandwagon, let’s consider one alternative that could make WebDatabase work (or, even enhance IndexedDB in terms of using a object-relational model instead of a tabular-relational model–which if you think about it, isn’t really all that different when you consider that at the core, it’s all relational algebra). Anyway, the main problem is that the databases (e.g. MS SQL Server (or CompactSQL), SQLLite, etc.) al use a different “dialect” of SQL–to put it mildly. Since most of these “compact” and lite databases are offered for free, there really isn’t any reason why they each need to speak a different dialect of SQL. Perhaps there should be a single standard of SQL used for the Web only that all these different compact databases will use (or database engines created specifically for WebDatabase or IndexedDB) in order to facilitate cross-browser, cross-DB support? I know, seems novel…. I don’t understand why we can’t all get over ourselves and make this happen.

Anyway, just my $0.02… You don’t have to get mad at me. If you don’t like what I say, just move on and read something else or continue to do whatever you do currently.

Mostly, I just encourage everyone who’s making an effort to bring this to fruition to make the interface as simple as possible to query. Data is king, and many people without formal knowledge of database technology need to query and analyze that data–even creating “queries” on the fly.

Are you able to run example codes from this post in Firefox 4 beta 10 or 11?
My problem is that the success event triggered after calling mozIndexedDB.open() doesn’t have result property and I cannot get to the instance of the database? Is this a firefox bug or am I doing something wrong?

OK, I found the answer myself. In Firefox you have to read the result from the request object not the event object. Don’t understand why it is done that way, cause the Indexed DB draft says something else.

Those code samples read like advertisements for Web SQL. In every case, I would pay money to be allowed to use the Web SQL code rather than the Indexed DB code.

I just tried implementing a simple one-table, one-index setup in Firefox 4 using IndexedDB, and the code is so painful to write, and the performance so dismal (4 *seconds* to simply loop through 200 objects of about 1KB each with a cursor) that I’m ready to support *anything* other than this.

Just my 2 cents here but why didn’t Mozilla simply support WebDB as originally specified by WC3? Seriously? You have to write your own join strategies? It’s 2011, but Mozilla has implemented a storage strategy that basically throws away about 40 years of grounded relational theory (Codd, et al)? Even if you don’t like the proprietary(-ish) nature of Sqlite couldn’t they have come up with something that at least had SQL join syntax?

Grrrr. I’m currently doing a HTML5 audit of features for a new product I’m writing and because of this I’m seriously thinking about excluding Firefox as a supported browser. Go Chrome!

You guys have a major influence on these standards. Complaining that it’s deprecated is just a cop-out. Why can’t you just pick up the standard in whatever state it’s in, and fix whatever issues it has? You moan about the fact that the standard references a specific implementation and the consequent difficulties of defining the language independently of one, but most of the work has already been done for you. SQLite supports most of the SQL-92 syntax. So, as a starting point, Web SQL Database could define a subset of SQL-92 that happens to also be a subset of SQLite syntax.

Now, I’m not totally naïve. I’m sure it’s more complicated than that, but do you seriously expect us to believe that it would be more work than what you’re trying to do now?

Yes, the decision is pretty bad.
Yes, the excuses are laughable.
OK, let’s think for a minute.
Please correct me if I am wrong, but there should be some rationale behind this.
People at Mozilla are not idiots and morons (as per Kevin Layman).
So, who benefits most from hindering acceptance of common browser SQL DB standard?
Obviously, the company who owns the desktop.
Who loses most?
Obviously, the company who wants to move all apps to the net.
Is it counter intuitive that Mozilla took the side of the former, rather than the later?
Yes, it is, but we do not know everything behind the scenes.
Is it possible that some money changed hands or someone was offered high position at Microshaft?
Who knows?

Anyway, what would be the consequences?
Shall businesses start to move to webkit based browsers their intranet apps?
Shall Mozilla move be side steped with another Chrome frame?
Shall some new, killer app, based on WebDatabase, pop up and swipe Mozilla market share?

As I understand it, LINQ is an abstraction layer for SQL (there are even built-in utilities for viewing the SQL queries that are generated by LINQ). And while the syntax for LINQ is somewhat different, there are very few features that exist in SQL that cannot be found in LINQ.

This is clearly not the case with indexedDB, there is no concept of relationship between data entities, you must maintain that yourself. Regardless of what year it is, looping through lists of objects to filter them on your own is not only more work syntactically, but it more work computationally.

Any developer who’s been in the industry for more than a year or two should know by now that syntax is largely irrelevant as long as functionality is still there. I’m not married to the SQL language, i’m sure there are other methods for querying that are easier to read/maintain. But if we’re going to have an object relational model for persistent storage, then relational integrity must be guaranteed. This isn’t in indexedDB and until it is, it is an inferior standard when compared to webDB.

So while it may be a good start, I have trouble justifying the time and effort to build against it when I can use Chrome and Chrome Frame and end up with a stable, maintainable application.

LINQ is based on the IEnumerable interface. The concept is that any object that can be enumerated conforms to a single interface standard which can be operated on, queried and related using a unified model and syntax – with results being generated as implicit runtime types (js takes care of this by being prototype).
There are no pre-existing relations. LINQ is definitely not a SQL generator. LINQ to SQL is a LINQ implementation for querying SQL. Likewise LINQ to Object or LINQ to XML perform the same for other data store types.

The .NET Entity framework does exist to provide preset relations between objects, and it does integrate with LINQ but is not required. I believe this is where you are getting confused.

Also, iterating result sets is not implicitly more computationally expensive. I’m and not sure what would give you this idea?

Regardless of whether the API you are using does the iteration for you, or you do it yourself; it’s still being done somewhere.

With the case of indexedDB you are given the lower level control to perform this yourself. As has been indicated – another API can be built utilising indexedDB providing this relational ‘integrity’ in a packaged way.

This has development method has worked great for libraries like jquery

Point taken with LINQ, I had confused it with LINQ to SQL. But I have to say that if SQL databases loop through entire datasets, that’s considered a “full table scan” and is a sign that you at least need some indexes defined.

I think that you’ve hit the nail on the head about why there’s so much opposition to indexedDB. It’s a lower-level data storage solution, which means we have to go back to the drawing board on query tools, reporting tools, analysis tools, etc.

This isn’t a government project. I don’t think that it’s unreasonable to think that concurrent efforts are possible, in fact the web is made of a multitude of standards that serve the same purpose with different strengths and weaknesses, giving designers and architects choices on what’s best for each application.

The software industry has seen countless iterations of fundamental changes like this, as new technologies mature, they take replace the older technologies as they offer advantages in real-world applications. Beyond a more javascript-like syntax there is no advantage to indexedDB in terms of new features.

In fact it has fewer features than WebDB. The examples demonstrate that the programmers are on their own when it comes to joins. There are no demonstrations of aggregates, or protections from things like orphans.

I personally have no problem with working with new technologies, I find it strange when people in this industry do. But I also don’t understand the obsession so many in this industry have with syntax. I’ve worked with a lot of technologies and frankly I find that syntax rarely makes a difference. I especially don’t see the attraction to a new syntax when demonstrations like these show that the new syntax will result in MORE WORK.

I’m sure that indexedDB will eventually mature into something that provides advantages, or will be replaced with something that does. Until then I can’t justify supporting firefox in mobile applications when I have support for WebDB built into every other browser (including IE through Chrome Frame).

I just think that an object store model suits much more to a modern environment and what I hope is a coming age of ubiquity with web technologies. The fact that you can reference a store of like objects indexed by common properties is fairly powerful.

It fits the bill for javascript because it preserves the type-fluidity of prototype.

I think that if you’re trying to use an object store as a SQL store, then you are correct; its a long way around.

The power comes when you realise that you are not constrained in your storage to a table schema. All you must do is ensure that you set a meaningful key or key-property on the store value; the rest is free form.

Take for instance an object store named ‘Entity’
Now, ‘Entity’ stores records for both people, and companies.

Therefore when we store objects in this store, we ensure we assign a unique ‘EntityID’ as a key – but the VALUE has little similarity beyond the fact that it has a property named ‘Entity Type’ which is a hint for which other related properties to expect exist.

We may index ‘Entity Type’ to speed up querying, but essentially you have a SQL ‘table’ which has differing columns based on need.

In this context, a ‘Person’ record may have First Name, Last Name, Username, Password, History[], Friends[]

The latter two being arrays of more objects or EntityIDs

Personally I’m not a big believer of database level relational management. It just seems like a copout for bad coding of whatever is calling the SQL queries.

Don’t get me wrong, that’s obviously a broad sweeping statement to make – But I think you are right in that a DB systems primary function is indexed, timely access of information. Relation management comes second IMO and in this case I’d be more than happy to manage that in a higher layer.

There’s also the fact that SQL is essentially a glorified text store. Obviously we have numeric datatypes too, inclusive of date. You can store ‘blobs’, but they are dumb – the db engine cannot understand them.

What I’m not totally clear on here is that IndexedDB will take a ‘Value’ in it’s key/value pair as a standard javascript style object with properties – and allow QUERYING on the objects properties.

This is a major benefit given the time I spend serialising and deserialising data to text if I have to use a SQL DB

Perhaps I should wait a minute before posting so I can combine replies;

What we have here is a critical path in technology. There was an article recently positing that we only have the industry and reliance on expensive liquid rocket technology we have now due to a specific sequence of events including world war 2 and Hitlers fascination with the V2 rocket, and ensuing cold war and ICBM leap-frog development.

The end result is industry and ‘standardised’ technology relating to liquid rocket propulsion which may not necessarily be the best solution – but which we are stuck with because of the expense of ‘starting again’ and the investment and resistance of change of experience in the field.

This is equally true in this scenario. We are talking about the platform of the future here, so it makes sense to take stock of what technologies form the base.

IndexedDb is not working for me .. the code was simple, but its throwing an error while trying to open the database. Error: uncaught exception: [Exception… The operation failed for reasons unrelated to the database itself and not covered by any other error code. code: 1 nsresult: x80660001 (NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR) location: file:///D:/shiby/html5/IndexedDB/IndexedDbSampleFF.htm Line: 14]

50X Faster? “I never said the data in [table] cells had to be atomic” — Ed Codd, father of the relational database, 1999 Very Large Database Conference in San Francisco. I went to the conference just to hear him say it.

He was defending his invention and explaining why object stores are often 50X to 100X faster than comparable relational implementations. He explained how the industry went astray in the hands of the first relational DB implementors by recognizing nothing more complex than atomic values (strings, ints, floats, blobs, etc.) in the cells of a table.

Codd’s relational algebra showed how a complex real-world information structure can be normalized (decomposed) into a set of 2-dimensional tables and later reconstituted via the relational operators project, select, and join. The problem is that reconstituting from many smaller atomic parts takes more work than storing the same information in fewer compound pieces. This is why modern object stores, XML databases, object-oriented and the old hierarchical databases like MUMPS and IMS (ancient but they got it right) handle complex data orders of magnitude faster than relational. The only thing that allowed the relational database to survive and thrive these past 40 years was accelerating hardware following Moore’s Law.

Really? In 1984 I created both a relational and an XML-based (GEDCOM) object database for the LDS Church FamilySearch database, the largest genealogical database in the world at the time. (I know, XML didn’t exist yet). Both were implemented and deployed in production serving the same requirements. We did both at the same time out of group uncertainty. The relational version was carefully indexed and semi-normalized to optimize for performance. Notwithstanding, the XML/GEDCOM-based system was over 50X faster, not 50% but 50X! A whole family, its members, and details were stored in a single compressed physical chunk. Deployed on a lowly 1984 PC, it ran circles around our relational system on multi-million dollar mainframe. This was due to the real-world complexity of the data and the effects of breaking data up versus keeping them together. The mainframe ended up preparing data for deployment on CDs.

Learnings? IndexedDB’s ability to store a complex object can potentially yield much greater performance, if implemented to advantage. Lack of join and traversal semantics, bulky syntax, and other problems have been solved before in other settings; we just need to put our heads, best ideas, and data together.

There is a reason SQL doesn’t just have a ‘SELECT * FROM mytable’ command (which is essentially what IndexedDB seems to have, as far as I can see).

Anything that you can do in IndexedDB you can do with a similar level of difficulty in SQL, but things you can do easily in SQL look to be really complicated and/or inefficient in IndexedDB. Maybe there are some things which are easier in IndexedDB, but they’re not clear from these examples…

Saying that disk I/O will be slow compared to Javascript reading and filtering data, for instance, is very disingenous. This ignores the fact of caching. Consider doing a sequential scan of a 50MB database. If the database has to be read from disk, the disk speed will make the difference between JS and a compiled language have very small effect on the overall time. However, if the database has been loaded into memory (either in the DB engine, or just in the OS cache), then the difference between JS and the compiled language will be MUCH more significant.

Your comments ignore performance and suggest that code simplicity is paramount. When you have thousands of simultaneous transactions and terabytes of complex interrelated data, performance often determines what is possible.

I didn’t say the speedup is because of compiled code, it is because of the number of discrete physical blocks that have to be accessed in a relational implementation vs a hierarchical one. The speedup I reported was not theoretical, it was actual. It was measured on real optimized databases on real optimized hardware with real user loads. The speedup was 50X to 100X (not 50% but 50 to 100 times faster). Orders of magnitude performance gains are significant in big systems, don’t you think?

My measurements above included caching and were not disingenuous. The numbers are actual performance comparisons, not projected. They were measured on large memory supercomputers with huge caches. But there are limits to the problems caching can solve. Performance depends most on how many physical blocks have to be retrieved from disk to satisfy the query, and big databases don’t fit in cache, by definition. Relational implementations scatter the data you need across many blocks whereas a hierarchical implementation can deliver a complex bundle of data in a single physical read, depending on how it is organized and how it needs to be traversed.

This is why indexedDB should be able to run circles around a traditional relational implementation when complex related data can be grouped hierarchically corresponding to traversal requirements. And this is why IndexedDB is a big step in the right direction. If you don’t like the language syntax then suggest an alternative, but avoid confusing syntax with the amount of work the hardware has to do under the hood.

Your example would be 50 – 100 times faster because the data you are storing is highly hierarchical and therefore I’m guessing has only one relation defined, parent has children, that traverses the entire tree. If your system had more relations in it, I.E. parent has children and works for employer, you would see a very different result unless you store data multiple times in your XML db and that seems like you are asking for integrity issues and or circular reference problems.

How do you resolve circular references in your XML? I’m curious because you say that the entire family is stored as a single data block. I.E. if the unthinkable happened, (and unfortunately it does) and someone has a child with one of their descendants, how can you store the entirety of the family when you hit that descendant, the entire tree above them can’t be repeated infinitely right?

Specialized data can be sped up with a specialized storage mechanism, we are talking about (with a web standard) a generalized storage mechanism that should well for the widest variety of data with the lowest cost to maintenance (ease of use, difficulty in doing the “wrong” thing) while providing the highest performance possible. SQL and relational dbs have so far been the best compromise. There’s a reason that Google came up with BigTable for their proprietary DB, the data worked best with that structure, and performance outweighed the other items. Building a mechanical parts ordering system with BigTable is probably not a good use.

IndexedDB doesn’t seem to be a good general data storage mechanism. It sounds like it’s a little too low level for use. I.E. data integrity is the user’s responsibility entirely, the ability to store completely different data types in one store with ease frankly scares me based on the developers that I’ve seen out there in the last 30 yrs. Why do people think it’s a good thing to be able to store People and Companies in the same logical data store?

I have an existing desktop application I want to redevelop as a web app with off-line storage. It currently uses SQLite, has fourteen tables and builds its own dynamic queries based on user choices. Many of these queries contain multiple sub-queries and deeply nested WHERE clauses.

I can port this to the Web SQL DB without substantial difficulty. Even contemplating the task with Indexed DB at a theoretical level is impossible.

For some cases, Indexed DB may be a perfectly acceptable solution. But it’s not an appropriate tool for some of the work I hope to do. And I’m convinced that the web is the future of applications.

If you read the comments by supporters I think that the reason is obvious. It’s politics.

Supporters of KVP databases like Couch and Indexed DB (both mozilla pet projects) insist that we need a database that’s equally as scalable when it comes to read access when working with terabytes worth of information — inside a browser environment.

They are also convinced that this performance increase is worth shirking the responsibilities that relational/SQL databases have taken head-on such as data integrity and native support for aggregate functions. These solutions shift the responsibility for those things to the application developer to handle through disciplined use of good design patterns. In exchange they get to claim better I/O times for the database without ever having to show any actual metrics on the impact to practical overall application performance.

Supporters also insist that there is an additional advantage in keeping the SQL language from infecting web development…because when you’re working with the web, you should never have to work with multiple syntax structures and languages simultaneously.

These supporters may be right, however I’ve seen no recommendations for any frameworks, reporting tools, or other development tools available for these databases…because they don’t exist. It would be an understatement to say that Indexed DB is in its infancy. Compare this to WebDB which is based on SQLite, with a host of development, reporting and testing tools and for anyone who actually wants to do practical application development the advantage is obvious…there is no justified reason for Mozilla not to support WebDB (at least until Indexed DB is mature) except for politics…multiple standards can and should be supported with offline storage just as multiple standards for document layouts (XML, HTML, XHTML, etc) are supported.

With this MooTools-based Database.js script, you don’t need to worry about async if you don’t want to (it has built-in queuing of async DB requests), you don’t have to keep regenerating connections to the DB (does if for you), and you get a ton of chainable helper methods that allow you to rock socks. Oh and did I mention you can bootstrap a schema with a JSON object if you like?

What about an equivalent to the sqlite3 CLI? I located and opened the add-ons database for my Firefox with sqlite3, issued a “.schema”, followed by a “SELECT name FROM addon;” and, lo and behold, I saw a list of Firefox add-ons I currently have installed. Now, this isn’t WebSQL, so I did a further test…

I loaded http://project.mahemoff.com/sql.html in Chrome and added a couple of cities. I then loaded the corresponding database with sqlite3, confirmed that cities were stored with .schema then SELECT, added another city with INSERT and reloaded the web page. And there it was. The whole exercise took less than a minute, most of which was spent figuring out where Chrome stores its WebSQL databases.

Consider how this degree of flexibility can impact the development process, especially if you choose from the plethora of extant GUI clients instead of the crufty old sqlite3 CLI, or even build your own tools using one of the many languages with SQLite bindings. For instance, the following Python script strains a list of city populations out of another web page and adds them to the WebSQL demo at http://project.mahemoff.com/sql.html. After a reload, the demo displays the populations of 100 cities.

I am sure plenty of effort is going into IndexedDB to make it easier to work with. But when Paul indicated that there was no tooling, I don’t think that 235 lines of API wrapper — lovely as those lines may be — are what he had in mind.

I’m not sure you really understand the “API wrapper” you are seemingly dismissive of. First off, it isn’t just a wrapper. It offers convenient methods that people are use to working with when using JS natives like Array that don’t exist on the native IndexedDB object. Additionally it *eliminates entirely* the async handling for you and makes it so you *never* have to worry about whether your transaction objects are stale.

You tipped your hand when you called it a *wrapper* and offhandedly dismissed it. At that point I knew you didn’t really give it a decent look, much less try it out ;)

@Daniel: Firstly, I did have a good look at it before commenting, and no, I wasn’t trying to be dismissive by calling it a wrapper. Even the raw SQLite API is much more difficult to code against than the C++/Python/Javascript/etc. wrappers, which also do more than just “wrap” the SQLite API. If I came across as dismissive, you have my unreserved apology; that wasn’t my intent.

My point was, and still is, that a library that makes it easier to access the raw API — even one that improves async handling and adds support for storing additional data types — is a far cry from the wide range of tools, libraries and services that are available for SQLite. Do you dispute the claim that IndexedDB is in its infancy? That’s basically the claim Paul and I were making.

We choose to take the stance that if the user wants offline functionality in their app then they will have to use a webkit browser or IE w/ chrome frame. The server DB and iOS/Android apps are already using sql, writing the html5 app using a very simular database schema had many advantages. Sql works on almost all mobile devices and 80% of desktop browsers (using google frame)

My users have no problem using a more advanced web browser to get offline functionality, most of them move to Chrome and never look back.

[…] that the syntax was just awful compared to it’s SQL alternative. The code samples found at Mozilla Hacks show this pretty well (although it seems that the point of the post is supposed to sing the […]

[…] couple of articles from the Mozilla Hacks blog on web databases, the Road to IndexedDB and an introduction to IndexedDBMSDN’s introduction to IndexedDB (supported in IE10)An introduction to webStorage, also from MSDN4 […]

I always say “USE COMMON SENSE!”. Firefox is getting complicated, slow and outdated by the day. And when I see this attitude which say “we don’t want to support webdb for NO CONVINCING reason” then it is clear to me why Firefox will be history soon. Many people I know are moving to chrome nowadays because of its ease of use. Users and developers doesn’t give a damn about politics and some theoretical blah blah. What they always prefer is something that WORK with EASE. This is just common sense. And if you lose sight to the simple things that are absolutely obvious to any normal person then your done!

IndexedDB is terrible to implement when you’re dealing with complex data relationships. It is the biggest pain to try and join multiple related data stores when all it would have needed in sql was a few joins to get the job done.. and for data optimization I don’t see how you could even come close to leveraging the power of something like a CTE either.. As web developers we should be making tools easier to use.. not the other way around.

I created HTML5 pages that use IndexedDB. They are working fine with Google Chrome. But when running on Mozilla Firefox, IndexedDB just does NOT work. I am using Firefox 18.0. Does anyone have any idea why ? Does anyone have IndexedDB running successfully of any version of Firefox ?

A long time ago, a large company I was working for was looking at two competing database technologies. One was basically a hierarchical database manager – with a query language similar to VSAM. The other was a SQL RDBMS. We asked both companies to implement a series of CRUD-type operations using data that were typical for our company. Both were given the flexibility to structure the data however they chose, as long as the operations we required could be implemented.

SQL-guy created a set of tables, and wrote the seven queries, all in about 3 hours. Hierarchical-guy hand-generated highly optimized code to do the same things, and it took him (full time) seven weeks to do it.

Advantage Hierarchical-guy, right? But wait… SQL-guy took one look at the results, and said “Well, I guess I need to create an index or two.” An hour or so later, he had the average query time down to 0.8 seconds. The performance still lagged behind Hierarchical-guy’s, but users didn’t really notice the difference in performance.

Then we asked both companies to make a fairly significant change to one of the queries. (What? You’ve never had requirements change mid-stream?) SQL-guy shrugged, spent 30 minutes making the changes, and had the new version running with comparable performance to the old. Hierarchical guy pitched a hissy-fit, spent nearly a week working on it, and came back with a query that produced the results required, but performed very poorly… He said he would have to re-design the entire suite to resolve that performance issue.

Our company’s decision? We went with SQL. It was unquestionably true that a master programmer, spending an inordinately long time writing highly optimized code could make the hierarchical solution outperform the SQL database. But the cost in development time was 35X higher to code the hierarchical solution. And a (very common) requirements change mid-stream was roughly 70X the cost in development time.

So why bring up this ‘ancient history’ in this context? I’m hearing the same refrain here: A great programmer should be able to write faster, more elegant code using IndexedDB. (Quite possibly — maybe even probably — true.) But at what cost? And how about the vast majority of “real world” programmers, who fit into the “roughly average” skill category? (And let’s face it folks, fully HALF of ALL programmers are “below average” skill level! Think about it… Look to your left, look to your right, if you don’t see the “below average” programmer sitting there, you’re probably the one… ;-) For those folks, and even for those who truly are “above average”, WebSQL may be a better commercial solution. It probably won’t be as elegant, but it will likely cost less to write, maintain, and change those solutions over time.

My vote: provide both interfaces, and let the programmers decide when to “pay the piper” for the ultimate in flexibility and performance, and when to use a “good enough” solution.

It looks like it’ll be worth the effort to master Chrome’s technologies for my webapp. Moz has been going downhill these past couple years…. My theory is when Blake Ross took over he elevated some people with him who held on for a while and intimidated the crazy wrapper gurus who don’t know anything about design into being quiet, but now the Seamonkey people have taken back over and Moz is headed back to the nut house. This was a tough decision (maybe tougher than it should have been), but I’m dropping Firefox for Google pretty soon.

Wow, people sure are forgetting we’ve entered a consumer landscape dominated by client-driven apps not beholden to massive DBs and interactions with a back-end server. Obviously there are many apps that still rely on such models, but the IndexedDB API serves today’s common mobile app use-cases.

For each person on this thread up in arms over the inability transfer the SQL paradigm to the client, there will be ten more client-side developers, who when equipped with a slick client-side IndexedDB module/lib, will be happy everyday not to have to deal with SQL.

Perception is reality folks – you must realize that many of the apps commonly developed for phones, tablets, and…errr…glasses, are not cut from the same cloth as old school server-bound apps. Additionally, the developer pool is changing – it now includes pure-JS folks who will be making apps where almost all the code resides on an installed, on-device package. These developers don’t care hardly as much about some far-off relational database server or the millions of rows it houses. This point is underscored by the emergence of a whole class of auth, content/media, payment, and no-SQL storage services now available to web app developers who have grown tired of all the annoyances the traditional app-stack interaction/maintenance forces them to wade through.

Maybe it’s frustrating to those who are closely tied to older models of web development, but this is the evolving reality of where app development is clearly headed.

Daniel why not give people the choice? Let the consumers decide. And please, please make the file handle api something independent of indexedDB. We don’t want that. Our users don’t want it. They want data they can share with other users and between PCs. They need a conversion point between the browser and the file system. It just seems to me like this is a big bunch of baloney holding back people who like to code in Javascript. I love Javascript… it’s so easy to turn prototype ideas into working ideas with it… you can start with a goal but maybe not a plan, and just keep going until you’re there. Maybe your code won’t be interoperable with other scripts… but if you’re creating something that’s never been seen before, maybe it doesn’t need to be. I don’t make much use of libraries, but my code is lightweight and very easy to understand. That’s all I need from it… I can do by myself what it takes ten people to do in Java or C++, with Javascript. That and the ability to make use of all that browsers have to offer nowadays with HTML 5… you can’t hardly do that with either Java or C++ with the same simplicity (VB is the only thing that comes close). Cloud services are impractical for small developers because they are impossible to manage without spending thousands of dollars.

Personally tired of it. Just going to Chrome. At least they think of user convenience there. You might win me back, but you’ll have to work for it. And, you’ll have to win back the 8% of the market you’ve lost over the last two years… it’s because of them that I think it’s OK to leave. I think you’re doomed to become the Linux desktop of web browsers: 2% market share competing for a small hermit-like niche… surprise me. …You’ve made me so angry.

BTW… what’s wrong with IndexedDB? Why do so many people hate it? Because it was forced on them by a third party that has the leverage. That’s not something the creators of commercial databases had. You are presuming that users will force developers to go your way out of a sense of brand loyalty. You have assumed wrong.

In an age of ‘cloud’ this feature is irrelevant. Firefox suffers from too many chefs and bad design. For example, some screens are URL’s like ‘about:config’, yet others are old fashioned popup dialog wizards, and so on and so on….the lack of webkit compatibility leaves firefox’s engine in the stone age….I hate chrome, but its the best crap out there….

I’ve left this thread alone for ages because it’s so inflammatory, but sorry, have to comment:

“For each person on this thread up in arms over the inability transfer the SQL paradigm to the client, there will be ten more client-side developers, who when equipped with a slick client-side IndexedDB module/lib, will be happy everyday not to have to deal with SQL.”

Not sure whether this is a serious comment, but no one I’ve ever met, ever, in client side or serverside development has a problem dealing with SQL. Also, as a mobile developer, one of the reasons that I’ve dropped Mozilla is because it’s pretty much useless for mobile apps, especially given its lack of support for SQLite. Saw a report today that mentions that Firefox is avoiding iOS – good.