A closer look at IndexedDB

When you create a website or web application and you need to store information on the client because of performance reason or because the site should work offline you can choose between two built in options: Web Storage and IndexedDB. Web Storage consists of the sessionStorage and localStorage object. The browser destroys data stored in sessionStorage when the user closes the tab or browser, localStorage survives a browser restart. Web Storage only supports strings as keys and values. You can still save arrays and objects into Web Storage when you serialize them first (for example with JSON.stringify).

Web Storage is a very easy to use API, but when you have to store a lot of data or the application needs to query the data in different ways IndexedDB might be the more suitable database. In the following post I will take a closer look at IndexedDB.

W3C issued the final recommendation for IndexedDB in January 2015 and the browser support got better in the last couple of months. Safari 8 for example had a very buggy implementation that made it unusable in this browser. But today the support for IndexedDB looks very good except in the Microsoft browsers IE and Edge. Both have an IndexedDB implementation built in but it's not feature complete. For example, they don't support compound keys and multiEntry indexes.

There is a polyfill for IndexedDB when your application needs to support older browsers: IndexedDBShim. It brings support for compound keys to the Internet Explorer too. When you create Cordova applications and have to support older platforms this plugin could be useful: cordova-plugin-indexedDB.

When you are only interested in creating web sites and progressive web applications for mobile devices, the browser support for IndexedDB looks very good. Both Chrome and Safari have a complete implementation of IndexedDB version 1 and already support the upcoming version 2 of the standard. On the desktop you have Chrome, Safari, Firefox and Opera that support version 1 and version 2 of the specification. The Microsoft browsers do not support IndexedDB version 2 yet (September 2017).

See the current browser support for IndexedDB version 2 on Can I use....

IndexedDB is a database in the browser that can store a significant amount of data and supports indexes (like a SQL database) that allow an application to query the data in a performant way.

IndexedDB is very flexible and allows an application to store almost everything. IndexedDB databases (like Web Storage) are protected by the same-origin policy. An origin is defined by the scheme (http, https), host and port of an URL. A website from http://www.bar.com cannot access a database created by a website from http://www.foo.com.

Unlike a SQL database, which uses fixed column tables, IndexedDB is an object-oriented database. It let you store arbitrary JavaScript native types and objects that are mapped to a primary key.

An application needs to specify the database schema before it can store data. It has to create databases and object stores before it can store data. Each object store must have a name that is unique inside the database. Each object store has a primary key and optionally one or more indexes. The object store persists records that are key value pairs. The primary key can be of type string, date, float, binary blob and array. A value can be anything that JavaScript supports: boolean, number, string, date, object, array, regexp, undefined, null, Arraybuffer, types array objects and DataView.

Records within an object store are sorted according to the primary key in ascending order. Each object store can hold arbitrary objects but most applications group similar objects in one store (for example one store for users, one for tasks, one for notes and so on). An object store is like a table in a SQL database or a collection in MongoDB.

Web applications are not limited to one database, they can create as many databases as they want and each database can hold multiple object stores.

IndexedDB is a transactional database and every operation (read, create, update, delete) needs to be executed in a transaction. When something goes wrong IndexedDB automatically rolls back the changes and ensures the data integrity of the database.

Most methods of the IndexedDB API are asynchronous. Because the standard was defined before the age of Promises it uses DOM events that the application needs to listen on.

An experienced user can delete the database by opening the browser developer tools and manually delete them. For these reasons applications need to be aware and handle the situation that the database could be gone the next time they run.

In the future you could use the StorageManager Estimation API that tells the application if there is enough room to store the data. It's an experimental API and only available in the latest developer build of Chrome and Firefox (September 2017)

Here a few links that provide more detailed information about IndexedDB

You open a database with the open() method from the global indexedDB object.

const openRequest = indexedDB.open('TestDB',1);

Each database has a name that identifies it within the origin. The name can be any string. The second parameter specifies the current version of the database. When you omit the second parameter the method uses 1 as the default. To delete a database you can call the deleteDatabase() method:

indexedDB.deleteDatabase('TestDB');

open() is an asynchronous operation and emits three events: success, error and upgradeneeded.

If the database does not exist, open() creates it and triggers the upgradeneeded event. If the database already exists but the stored version number is less than the second parameter of the open() method the upgradeneeded event is emitted.

IndexedDB runs the upgradeneeded handler inside a transaction. IndexedDB automatically starts and commits this transaction. When the handler throws an error IndexedDB aborts the upgrade transaction and rolls back the changes.

When the upgradeneeded handler completes successfully the success event is triggered.

When the database exists and the version number of the open call, matches the current version of the stored database only the success event is emitted.

The version number must be a positive long number. The numbers don't need to be in consecutive order, but must be in ascending order. Most likely you use an order like 1, 2, 3, 4 but 10, 20, 30, 40 works as well.

When an application tries to open a database with a requested version number less than the current database version IndexedDB throws an error.

Because you can only change the database schema in the upgradeneeded event handler you need to trigger this event by increasing the version number every time you have to make changes to the structure of the database, like creating, deleting and renaming object stores and indexes.

Each asynchronous method of IndexedDB returns a request object. This object receives the success and error event and has onsuccess and onerror properties where your application can attach a handler. Alternatively an application can use addEventListener and removeEventListener to manage handler methods. The request object also has readyState, result and errorCode properties that give the application access to the current state of the request. The result property holds the result of the operation. For example for the open operation it references the database object, that is the main entry point for the following operations.

When you execute this code for the first time in a browser you see two console log entries "upgradeneeded" and "success". But when you refresh the browser and execute the program a second time you just see "success". When you increase the version number parameter in the open call and refresh the browser you see both strings again.

Error events bubble, like DOM events. First they try to call the error handler on the request object, then on the transaction and finally on the database. Therefore, you could install a global error handler on the database object.

db.onerror= event =>{
console.log(event);};

Success events do not bubble up.

To store data an application needs to create an object store first. As mentioned above, applications can only do that in the upgradeneeded event listener. In this example the application creates a store with the name users which has a primary key that references the field userName in the user object.

After the onupgradeneeded handler finishes successfully the success event handler is called. Before the application can insert and read data it needs to start a transaction. The following example inserts two user objects with the objectStore.add() and objectStore.put() method. add() is an insert only operation and does throw an error when the primary key already exists. put() is either an insert, when the primary key does not exist or an update when the record with that primary key exists.

objectStore.get() can fetch a record with the primary key. objectStore.getAll() returns all records of the store. The objectStore.delete() method deletes records with the primary key and objectStore.count() returns the number records in that store.

All these operations need to run inside a transaction and return a request object where the application can listen for the success and error event. All operations run asynchronously but are executed in the order in which they were made and the results are returned in the same order.

IndexedDB automatically commits transaction, there is no method to manually do that. But an application may abort a transaction at any time with the transaction.abort() method. This method rolls back all changes made during the transaction.

Browsers allow you to see and modify IndexedDB databases in the Developer Tools. In Chrome, you find the databases in the Application tab. Firefox lists the databases in the Storage tab.

In this section we will take a closer look at the definition of primary keys. Every record that is stored in an object store is referenced by a unique primary key.

IndexedDB supports two kinds of keys. in-line and out-of-line keys.

An in-line key is stored as part of the object. When the store uses an in-line key the application can only store JavaScript objects as values. An in-line key is created with the keyPath option.

An out-of-line key is stored separately from the value. It has the advantage that the application can store not only JavaScript objects but also primitive datatypes like string and numbers as the value of the record.

Both types of keys can be either provided by the application or auto generated by the database. In total, you have 4 different types of primary keys to choose from.

To create an out-of-line primary key you omit the second parameter of the db.createObjectStore() method. The following example maps a number to a string (1->"one", 2->"two", ...). To insert a record you can choose between objectStore.add() and objectStore.put(). Because the primary key is not auto generated the code has to provide it as the second parameter of the method call. add() can only insert records and throws an error when the primary already exists. put() is either an insert or update operation.

To create an auto increment primary key you provide an object with the autoIncrement field set to true as the second parameter of the db.createObjectStore(). IndexedDB uses a key generator that produces consecutive integer values starting with 1. The key generator does not reset when the application deletes records. With an autogenerated key there is no longer a need to provide a second parameter to the objectStore.add() and objectStore.put() methods, both behave the same and insert the record. The following code inserts three records and the database generates the keys 1, 2 and 3. When you run this code, a second time the object store holds six records with the keys 1 to 6.

You can still provide a value as second parameter, in that case IndexedDB does not auto generate the primary key for this particular record. Note that add() will fail when you try to insert a key that already exists and put() will update the record. With a new database and object store the following code assigns 1 to the first record, 10 to the second and 11 to the third. When you try to run this code a second time it will fail because the primary key 10 already exists.

To create an object store with an in-line primary key you need to provide the keyPath configuration option as second parameter to the db.createObjectStore() method. As usual objectStore.add() is an insert only operation and throws an error when the primary key already exists and objectStore.put() is either an insert or an update operation.

To create this kind of primary key you specify both keyPath and autoIncrement in the configuration object of the db.createObjectStore() method. IndexedDB then installs a key generator for this object store that creates consecutive integer values starting with 1. The property that holds the key should not be present in the object that you want to store. The database automatically adds the property to the object when the insert is successful.

When you use inline-keys you cannot provide the second parameter to the add() and put() method. This code throws an exception

objectStore.add({data: 2}, 20);

But you can save objects that already contain the property referenced in the keyPath. The following code insert the records with the primary keys 1, 20, 21 and 40. add() will fail when the key already exists and put() updates the record.

IndexedDB supports composite or array keys, a key comprises two or more fields. This could be useful when you have an object but none of the properties alone is unique. This is not a primary key the database can auto generate, the application has to provide the values.

Over the lifespan of an application requirements change and affect the structure of the database. With IndexedDB an application can only change the structure of a database in the upgradeneeded event handler. This handler is triggered every time the version number of the db.open() method is increased and is greater than the current version number of the database. We have already seen this behaviour in action in the first example. In this section we look at an example with multiple versions and how we can organise the migration steps.

The initial version of our example application stores contact objects that look like this.

For this application we create a database with the name MyApp, an initial version 1 and an object store contacts with an auto increment primary key mapped to the id field of the contact object. And we create indexes for the name and email fields. We know that in the scope of our application an email can only be referenced in one contact and is therefore unique.

Over time our application grows and we add a new function to store todo lists. For that we need a new object store. To trigger the upgrade we increase the version number to 2. We cannot simply add one db.createObjectStore() statement to the existing source code, because there are users with a version 1 database that contains the contacts object store and IndexedDB throws an error when the application calls createObjectStore with a name that already exists. And we cannot simply delete the code from version 1 and only add the db.createObjectStore() statement for the new store, because there will be users that open our web application for the first time and don't have any database yet.

One way to solve that is to differentiate first time users and recurring users. Fortunately the upgradeneeded event object contains a property oldVersion. For new users this property has the value 0 and for recurring 1. The code checks if the version is 0 and creates the contacts object store. When the database is at version 1, it only has to create the new store.

Another way to solve the problem is by checking if the object store already exists. The database has a read-only property objectStoreNames that contains the names of all object stores currently stored in the database.

We continue working on the application and doing some refactoring. We want to change the names from the object stores to singular because we no longer like the plural names and we want to change the postfix for the indexes from 'Ix' to 'Index'.

We have now to support 3 different cases. New users, users with a version 1 and users with a version 2 database. Here we do that with checking if the old object store exists. If not, we create the stores with the new names. If the old store exists we rename it. To rename a store we need to get a reference to it. This can be done by calling the objectStore() method on the transaction object (event.target.transaction). Then we can rename the store by assigning a new value to the name property. To rename the index we call the index() method from the object store object und assign a new name to the name property. Note that renaming is a feature from the IndexedDB version 2 standard.

In the next update we not only want to change the structure of the database but also need to migrate some data. In the contact object store we initially store the name in one field (John, Doe) first name and last name separated with a comma. We realise that this is a mistake because it makes finding a contact very difficult. For that reason we want to split this field into two fields, create an index for each field and delete the nameIndex index.

Now we have to manage 4 different cases. We adjust the code from the 3rd upgrade, because here we only have to create the object stores when the store with the oldname (contacts) and new name (contact) not already exists. Then we delete the index with contactStore.deleteIndex('nameIndex'), iterate over all the contact records with a cursor, split the name field into a firstName and lastName field, delete the name field, update the changed object in the database with cursor.update() and finally create an index for the two new fields.

You can always add, rename and delete indexes without losing any data.

The only way to change the primary key of an object store (for example not generated to auto generated or a new keyPath) is by creating a new store, iterate over the old store and copy each record to the new store.

We've seen transactions in action and mentioned that every read and modify operation has to run inside a transaction. To start a transaction you call database.transaction(). This method takes two parameters. The first parameter is mandatory and lists all the object stores you want to access inside the transaction. This is either one string or an array of strings. The optional second parameter specifies if the transaction should open in read only or read write mode. When this parameter is omitted IndexedDB starts a read only transaction.

Transactions emit three events: error, abort and complete. When an error occurs the error event is triggered and by default aborts the transaction and rolls back all the changes made during the transaction after that the abort event is emitted. You can override this behaviour in the error handler by calling event.stopPropagation() Otherwise when all pending requests have completed successfully the transaction is automatically committed and the complete event is fired.

The read only property transaction.objectStoreNames contains a list of object store names that are accessible in this transaction. The transaction.objectStore() method returns a reference to an object store. This store has to be one of the stores you provided to the db.transaction() method. The transaction throws an error when the application tries to access a store that was not listed.

Indexes are an important part of IndexedDB and we have already seen a few examples of the objectStore.createIndex() method. But so far I haven't discussed what indexes do and why an application needs them.

When the database would not support indexes the only way to access the records is either with the primary key or by iterating over all the records. Indexes provide an additional performant access path to the records.

To create an index an applications calls the objectStore.createIndex() method. This method takes three methods, though the last parameter is optional. The first parameter defines the name of the index. The second parameter is the keyPath that references a property in the object. Like the primary keys this can be an array of two and more properties. The last parameter is an object with configuration parameters.

This code creates an index with the name nameIx and references the field name in the value object.

objectStore.createIndex('nameIx', 'name');

Unique indexes do not allow multiple values, similar to the primary key. This adds a constraint to the database. The database throws an exception when the application tries to insert a record that violates the unique constraint.

objectStore.createIndex('emailIndex', 'email', { unique: true});

When a property is an array and you want IndexedDB to add each of the array elements to the index you need to set the multiEntry option to true.

objectStore.createIndex('hobbies', 'hobbies', { multiEntry: true});

You should not create an index for each property in your object. Only create indexes for properties you know will be accessed in your application. IndexedDB needs to store indexes in internal data structures and has to update them every time an application inserts, updates and deletes records. You can always add (and remove) indexes in future versions without losing any data.

To look at some queries the following example creates an object store with the name contacts and inserts 10 records.

After the insert operation completes successfully, the code calls the runQueries method. As always read operations have to run inside a transaction therefore the application starts a read only transaction that spans the contacts object store. Then it gets a reference to the index with the objectStore.index() method.

The index object provides several methods to access the records. Note that all of these methods are asynchronous and you always have to add a success handler to fetch the result. The property event.target.result holds the result of the query. Here is a basic code template with the index.get() method as example.

lastNameIndex.getAll() returns all records that are stored in this index. All index methods return the records sorted according to the index property in ascending order. Notice that this particular index only contains 9 of the 10 test records. Record 10 does not have a property lastName and is therefore not included in this index.

index.get() returns the first record that matches the provided index key. When the index is non unique it can hold multiple records with the same index key. In that case get() returns the record with the lowest primary key. get() is equivalent to getAll(..., 1).

An alternative to get() and getAll() are the methods getKey() and getAllKeys(). They do not return the whole record but only the primary key. If the application only needs the primary key of a record you should always use these methods, because they don't have to deserialize the record, which saves time and memory. getAllKeys() always returns an array and getKey() only returns one value.

All query methods mentioned in the previous section do not only support a value as parameter that has to match exactly the index key but also support an object of type IDBKeyRange. To create a range you call one of the provided methods of the IDBKeyRange object. By default the bounds are included but you can exclude them with the optional boolean parameters.

Range

Code

Index key <= x

IDBKeyRange.upperBound(x)

Index key < x

IDBKeyRange.upperBound(x, true)

Index key >= y

IDBKeyRange.lowerBound(y)

Index key > y

IDBKeyRange.lowerBound(y, true)

Index key >= x && <= y

IDBKeyRange.bound(x, y)

Index key > x &&< y

IDBKeyRange.bound(x, y, true, true)

Index key > x && <= y

IDBKeyRange.bound(x, y, true, false)

Index key >= x &&< y

IDBKeyRange.bound(x, y, false, true)

Index key === z

IDBKeyRange.only(z)

The IDBKeyRange object provides the includes() method that tests if a value is inside the range.

The query methods we have seen so far return the records in one batch. This could be a problem when you have to process thousands of records. getAll() has to deserialize all the objects at once and store them in memory. For processing a lot of records it's more memory efficient to use a cursor that process the records one by one.

The methods index.openCursor() and index.openKeyCursor() create a cursor. These two methods also exist on the objectStore object. When the application only needs access to the primary key you should always use the openKeyCursor() method, which does not has to deserialize the object from the database into memory.

Both methods take two optional parameters. First parameter is either the index key or an IDBKeyRange object. This works exactly the same as with all the get methods we have seen so far.

The second parameter provides a direction. This is a string constant and supports these four values

'next': The cursor shows all records and starts at the lower bound and moves upwards in the order of the index key. Default when the parameter is omitted.

'nextunique': The cursor starts at the lower bound and moves upwards but only shows unique values. This is useful for non unique indexes when you only want to see the unique values. When multiple records exist the cursor only processes the one with the lowest primary key and skips the others.

'prev': The cursor shows all records and starts at the upper bound and moves downwards.

'prevunique': Like the nextunique direction filters out duplicates but starts at the upper bound and moves downwards.

Working with cursors follows this basic pattern. As soon as the application opens the cursor the database calls the success handler with the first matched record. In the handler you have to test if the cursor object is null. The cursor is null when the query does not return any results or when all records were processed. If the cursor is not null you process the record, then you call the cursor.continue() method to move the cursor to the next record. This call will trigger the success handler again.

Each object already has a unique property (id) that the application defines as the primary key of the object store. The applications queries the type and weaknesses properties and therefore creates indexes for these fields. Both fields are arrays and to instruct IndexedDB to index each array element individually the multiEntry option is set to true.

After the database and object store is created the application starts a read only transaction to count the number of records in the object store. To save bandwidth the application fetches the JSON file only once when the object store is empty.

The insertData method fetches the JSON file from GitHub and inserts the records into the database. Here you see an example how you can wrap the IndexedDB callbacks in a Promise. The complete event is emitted as soon as all started asynchronous requests (store.put()) in the transaction are finished.

The queryData method shows you a few example queries. As usual, to query data, the application has to start a transaction.

store.count() returns the number of records in the store.

getAll() returns all the records that match the index key. In the first example these are Pokémons that are of type 'Grass', in the second query all Pokémons that are weak against 'Flying'.

In real world application you often have to combine queries. IndexedDB supports compound indexes that span multiple properties these are useful for AND combinations. Unfortunately the database does not support compound indexes together with the multiEntry option. But it's not that complicated to combine queries by executing multiple queries and then combine the results. The AND example searches for all Pokémons that are of type 'Bug' and are weak against 'Fire'. The application starts both queries with index.getAllKeys(), wraps each of the callbacks in a Promise and waits with Promise.all for both results. Then the code creates an intersection of the two result arrays, iterates over it and fetches each Pokémon individually with store.get().

The last example looks for all Pokémons that are weak against 'Ice' OR 'Flying'. This looks very similar to the previous query, but instead of an intersection it creates a union of the two result arrays. Set helps filtering out duplicate primary keys.

This example imports a larger dataset. A list of earthquakes that happened in the past 30 days. At the time of writing this post (September 2017) the file contains over 9500 earthquakes. The application follows the same pattern as the previous example and imports the data only if the database is empty. This saves some bandwidth because the JSON file is quite big. As usual the applications creates the object store and indexes in the upgradeneeded event handler. All the properties that the application is interested in are inside the embedded properties object. To reference them the application uses the dot notation. The magTsunami index is a compound index that spans two properties.

The program executes five queries. First it counts all the records in the object store and prints out the number.

Then it lists all earthquakes with a magnitude of 6 (inclusive) and higher. magIndex.getAll() with the IDBKeyRange.lowerBound range returns all the earthquakes in ascending order of the magnitude.

The next query is similar and lists all earthquakes from the past 6 hours. The earthquake dataset stores the time in milliseconds since 1970-1-1.

Next I wanted to fetch the earthquake with the highest magnitude. The code could very easy access the earthquake with the lowest magnitude with magIndex.getAll(null, 1). This works because the mag index is internally sorted by magnitude in ascending order. But getAll() does not give us the possibility to fetch records in descending order. For that reason the application uses a cursor with the direction prev. Because the application wants to stop the cursor after the first returned record it does NOT call cursor.continue().

The last examples accesses the compound index magTsunami and queries for all earthquakes that happened in or near an ocean and probably cause a tsunami (tsunami === 1) and have a magnitude between 5 (inclusive) and 5.5 (exclusive). Bounds are by default inclusive and you need to provide the optional boolean parameters to exclude them.