This gist lists challenges you run into when building offline-first applications based on IndexedDB, including open-source libraries like Firebase, pouchdb and AWS amplify (more).
Note that some of the following issues affect only Safari. Out of the major browsers, Chrome's IndexedDB implementation is the best.
When this bug occurs, every time you use the indexeddb, the WAL file grows. Garbage collection doesn't seem to be working, so after a while, you end up with gigabytes of data.
Sometimes Safari gets into a funky state, where exceptions like "Unable to open database file on disk" are thrown. Other exceptions related to indexeddb methods are also possible.
When that happens, incredibly, hostname resolution starts failing. This seems unrelated but occurs at the same time, for unknown reasons.
It's unclear what is causing this issue but I've been observing this problem mainly when there are many indexeddb databases for a given origin. The databases don't need to be opened for this bug to occur. You can fix the issue by deleting your local state.
You can work around this problem by only creating a small number of indexeddb databases.
After 7 days of inactivity, Safari deletes all browser storage, including cookies, localstorage, websql and indexeddb. Other browsers don't do this.
The feature, called Intelligent Tracking Prevention (ITP), is meant to prevent advertisers from abusing indexeddb as a way to track users. But it also has the consequence that purely in-browser apps like excalidraw cannot treat browser storage as durable storage for the user's data; they can only use it as a disposable cache.
The IsLoggedIn API might allow browsers to tell tracking from legitimate use of browser storage in the future but is still in the planning phase.
In our offline-first product, we've had reports from Safari users that sometimes data isn't persisted to disk even though no exceptions are printed to the console. The problem seems to be that in rare circumstances, the promise chain stalls, with no progress or error - the worst kind of failure mode, because changes are silently dropped.
At first we thought that the problem was in our code, but we discovered that similar problems in Safari have been reported in the Firebase SDK:
A few times, I've seen Firestore in a state where changes stop getting written to the cloud, or even IndexDB. There are no errors in the console, and only a refresh seems to fix it. Upon refreshing the page, the user's changes are simply gone.
Stepping through the code in this state, I can see that the write is enqueued in AsyncQueue, but never gets executed.
As far as we can tell, the issue seems to occur when the Safari tab is sent to or comes back from the background. We know that Safari throttles tab activity aggressively to safe energy. But of course, this shouldn't lead to indexeddb operations being delayed indefinitely.
IndexedDB has the concepts of transactions. Does this mean that IndexedDB is ACID compliant? Let's look at atomicity (A) and isolation (I) in particular:
-
IndexedDB provides atomic transactions. When multiple writes are wrapped in a transaction, IndexedDB guarantees that either all writes will go through or none of them will. This avoids inconsistencies due to writes interrupted mid-way.
-
IndexedDB does not provide transaction isolation. As far as I know, concurrent transactions (in multiple tabs or even in a single tab) are never rolled back because they touch the same object stores. The only exception, as far as I can tell, is exceptions thrown when a primary key constraint is violated. This can be used to achieve some level of transaction isolation. However, the guarantees are nowhere near as extensive as those provided by SQL databases.
There are semantic differences between how Safari interprets the IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB transactions auto-close more aggressively when nothing is being done to a transaction in a stackframe. This problem is apparent when using indexeddb with promises, because the use of Promise.resolve().then( () => ... ) can lead to the transaction being closed prematurely, causing an exception - especially in Safari and Firefox.
The idb promise wrapper contains an warning but transaction auto-commit semantics in combination with promises are still difficult to get right. For example, Firebase went so far as to implement their own version of Promise for the purpose of working around these issues.
Safari is most finicky, though operating within the spec. In any case the behavior differences compared to other implementations are confusing. In fact it's fair to say that it's hard to understand when transactions are supposed to auto-close, either as a application developer or browser implementer (see this bug report)
In general, asynchronous operations or promises mixed with IndexedDB transactions are fraught with peril. Consider the case of deleting a number of keys from an index. This will require two steps:
- Call
.getAllKeys()on the index - Iterate over all the keys and call
.delete()on the ObjectStore for each key
It's tempting to do all of this in a single transaction - after all, they two parts semantically belong together. However, the frist step is necessarily asynchronous. As a result, it's possible that once we get to step 2, the transaction has already auto-committed.
Unfortunately, this will result in a latent bug that only manifests itself in rare circumstances, depending on timing. For example, Firefox will sometimes throw an exception with the message Transaction is already committing or done, but not in a reproducible manner.
The solution is to perform all write operations (.put, .delete, etc) synchronously in a single stackframe, the same stackframe that created the transaction. In other words, it's not safe to perform steps 1 and 2 in a single transaction. Instead, each step should be a done in its own transaction (the first one being readonly, the second readwrite).
Safari is the only major browser that doesn't support IDBTransaction.commit()
The Web Platform lacks a standardized way - or any proper way, really - to lock a resource across multiple tabs of the same browser. Multiple tabs of an app using IndexedDB will invariably write to the same IndexedDb database. Without cross-tab locking, database corruption is hard to avoid.
Attempts have been made to builds locks on top of existing APIs like localStorage and postMessage, but clearly locking requires a new primitive and cannot be polyfilled. Proposed by Google, the Web Locks API adds the capabilities desparately needed to implement safe concurrent use of IndexedDB. However, while Chrome shipped Web Locks in 2018, Safari and Firefox haven't implemented the API yet. Similar things can be said about the new Atomics API.
Update 2024: all major browsers now support Web Locks.
Firefox is the only major browser without support for IndexedDB in Private Browsing mode. In a Private Browsing window (or if Private Browsing is enabled as a global setting), Firefox throws with a cryptic error when you call indexeddb.open:
A mutation operation was attempted on a database that did not allow mutations
Because there is no API to tell if Private Browsing is active, the only workaround is to try to open a test database and to regard the API as unavailable if this fails.
Browser vendors enforce qutoas for all browser storage, including IndexedDB. When the quota is exceeded, writes will fail wih QuotaExceededError or similar exceptions.
However, how quotas work is different from browser to browser. One surprising finding is that in Firefox, quotas seem to apply, not per subdomain, but to all subdomains belonging to a certain site name (more precisely, an eTLD+1) taken together. So for the purposes of quota, a.example.com and b.examples.com will be counted together. As a result, clearing storage data for the current subdomain may not prevent quota exceptions because the quota may be used up by other subdomains.
Before IndexedDB there was Web SQL, a thin wrapper around the legendary SQLite embedded database. Web SQL is more powerful (a proper superset of IndexedDB) and arguably better designed than IndexedDB. So why did browser vendors end up favoring the inferior IndexedDB standard?
As it turns out, Mozilla objected to the Web SQL interface as having no alternative implementation available. So IndexedDB, an API with less surface, was chosen instead. Ironically, IndexedDB is internally implemented based on SQLite in Firefox (Chrome uses the simpler LevelDB instead).
In a similar vein, wiki service Notion discusses problems with their IndexedDb backend:
Before SQLite, we relied on IndexedDB for client-side storage. But we encountered storage quotas, a number of bugs, and performance concerns on Windows machines in particular, which meant IndexedDB wouldn’t scale with Notion’s growing user base.
Their solution? To move to an SQLite db bundled with their Electron app running outside the browser context.
The Webkit team keeps shipping critical storage-related bugs into production, again and again. In May 2021, Safari 14.1.1 was released with a bug where IndexedDB databases would fail to open() when the browser is loaded for the first time (discussion). This came a month after the Safari 14.1 release, which fundamentally broke localStorage (discussion).
@pesterhazy Thank you for this wonderful summary. I shared it with my team.